expert

Firmware Testing

In this lesson, we'll walk you through how to extract, create a test driver, package, and test firmware binaries. These techniques are useful when presented with a Linux firmware with no available source code or documentation.

---

Estimated Time: 20 minutes

Objectives

By the end of this lesson, you will be able to:

1. Define the challenges regarding firmware testing.
2. Walk through a firmware testing example for the DGN2200v4 router firmware image.
3. Test and override functions with LD_PRELOAD.
4. Execute a Mayhem run on a target with the firmware test driver.

Prerequisites

You will need the following:

1. Mayhem and the Mayhem CLI.
2. Netgear N300.
3. MIPS firmware image.
4. Disassembler compatible with ForAllSecure's bncov (we will be using Binary Ninja)
5. Strong knowledge of reverse engineering. We will be looking at MIPS assembly code and Binary Ninja's HLIL
6. Docker

What's So Special about Firmware?

Firmware testing presents a specific set of challenges that are not often present together in other targets, thereby requiring an increased level of expertise and know-how to deal with efficiently.

In particular, the challenges firmware testing presents are...

1. Dependency on specific hardware features present on the physical device.
2. Non-x86 processor architecture.
3. Non-glibc C standard library.
4. Lack of available source code or documentation.

In this tutorial, we will cover how to deal with each one of these challenges in a firmware testing context.

Example: Netgear N300 a.k.a. DGN2200v4

For this example, we will be looking at the Netgear N300 (henceforth refered to as DGN2200v4) router firmware image. This is a good target to look at because while it is a Linux firmware binary, it presents all of the challenges listed above. Specifically, this firmware...

1. Relies on specific hardware features for synchronization used by its programs.
2. Is compiled for the MIPS architecture.
3. Uses uClibc instead of glibc C standard library.
4. Has no source code and very few debug symbols available in binaries of interest.

Now we will cover how to extract, create a test driver, package, and fuzz this firmware.

Environment Setup

For this tutorial, we have already set up a Docker image containing all the necessary tools and files. Therefore, it is recommended that you use the supplied Docker image.

Run the following to download and run the docker container:

docker pull forallsecure/fuzzing-firmware
docker run -ti forallsecure/fuzzing-firmware

Alternatively, you can also install the following tools manually:

• QEMU user static (ex: apt-get install -y qemu-user-static)
• Binwalk (ex: python -m pip install git+https://github.com/ReFirmLabs/binwalk)
• Jefferson (ex: python -m pip install git+https://github.com/sviehb/jefferson)
• MIPS cross compiler
• Mayhem CLI

Whether you are using Docker or the above manual setup environment, ensure you have access to Binary Ninja with the bncov plugin installed.

Extracting Firmware

Extracting firmware can sometimes be difficult due to custom firmware layouts and encryption. Luckily many firmwares, including this one, are just compressed file systems. This means that off-the-shelf tools such as binwalk can easily extract them. Run the following to extract:

$ binwalk -Me DGN2200v4-V1.0.0.126_1.0.126.chk
Scan Time:     2022-02-25 20:04:48
Target File:   /fuzzing-firmware/DGN2200v4-V1.0.0.126_1.0.126.chk
MD5 Checksum:  3b504ce5df0ea5cbd66b473fca1ec73f
Signatures:    410

DECIMAL       HEXADECIMAL     DESCRIPTION
--------------------------------------------------------------------------------
58            0x3A            JFFS2 filesystem, big endian

Note

The e option refers to extract and the M option refers to recursively extract if there are nested containers within the firmware.

After running binwalk, the extracted firmware will be in _DGN2200v4-V1.0.0.126_1.0.126.chk.extracted.

Before doing anything else, let's clean up the extracted firmware. In particular, binwalk created a lot of output but the only thing we need from it is the filesystem. Therefore, let's move the filesystem out of the extraction folder and rename it to root.

$ mv _DGN2200v4-V1.0.0.126_1.0.126.chk.extracted/jffs2-root/fs_1 root
$ rm -rf _DGN2200v4-V1.0.0.126_1.0.126.chk.extracted

Now that we have the filesystem in a root folder, we can investigate and test the binaries inside it.

Picking a target

Now that we have extracted the firmware, we need to identify a binary for test driving. Two good options for router targets are:

1. User facing web servers
2. Custom internal binaries such as database managers, etc.

We can also find interesting binaries by getting another similar firmware (such as a similar model by another manufacturer) and comparing which binaries are unique to each system with a script. While this can generate some noise (such as two routers using sqlite vs postgres), it helps massively narrow down the amount of binaries to look through and is a good first step.

For this tutorial, we will be looking at DGN2200v4's httpd webserver. Webservers on embedded systems (especially routers) are particularly interesting because they tend to control many functions besides just being a webserver including device bring-up, authentication, and process management. For this same reason, they can also be tricky to get running.

First look at httpd

First, let's figure out what kind of binary httpd is. Execute the following command to check the httpd binary within the newly created root folder:

$ file root/usr/sbin/httpd
root/usr/sbin/httpd: ELF 32-bit MSB executable, MIPS, MIPS32 version 1 (SYSV), dynamically linked, interpreter /lib/ld-uClibc.so.0, stripped

From this, we learn that this is a 32 bit MIPS binary using a non-standard libc (uClibc). Let's work past these roadblocks one at a time.

Since this is a MIPS binary and you're probably not doing this tutorial on a MIPS box, we will need to use QEMU to test out httpd or any other binaries we want to look at.

Additionally, to run these binaries in the proper environment, we will need to use chroot to "remount" the root directory to that of the extracted filesystem. The chroot command follows the form:

$ chroot <new-root-directory> <cmd>

Info

The chroot command above "remounts" new-root-directory as the root directory for the duration of the running cmd.

Because the invoked cmd only has access to files within new-root-directory, we will need to use statically linked QEMU (normal QEMU needs access to host shared libraries to run) in the chroot environment.

$ cp `which qemu-mips-static` root

Now that all the parts are in place, let's try actually running httpd:

$ chroot root /qemu-mips-static /usr/sbin/httpd
/usr/sbin/httpd: can't load library 'libssl.so.0.9.7'

Note

We use absolute paths for qemu-mips-static and httpd since our root directory has been changed to the root folder.

Uh oh. It looks like this target will need a little TLC before it will run happily.

Getting httpd running

Let's find the missing library and add it to the LD_LIBRARY_PATH environment variable.

Note

To proceed, you will need to remove the symbolic link for root/tmp and create an actual directory for root/tmp.

$ find root -name libssl.so.0.9.7
root/lib/public/libssl.so.0.9.7
$ ls -l root/tmp
lrwxrwxrwx 1 root root 8 Feb 25 20:18 root/tmp -> /var/tmp
$ unlink root/tmp
$ mkdir -p root/tmp
$ chroot root /qemu-mips-static -E LD_LIBRARY_PATH=/lib/public/ /usr/sbin/httpd
shm ID: 262152
Semaphore Create Failed.

Tip

Seeing a problem like the following?

root@f5c3f91750ae:/fuzzing-firmware# chroot root /qemu-mips-static -E LD_LIBRARY_PATH=/lib/public/ /usr/sbin/httpd
shm ID: 0
Get a correct Segment_ID: 0 and semaphore ID:0
Can't find handler for ASP command: usb_cgi_get_register_state();
Can't find handler for ASP command: usb_cgi_get_invite_state();
Can't find handler for ASP command: check_is_index()
Can't find handler for ASP command: wds_cgi_get_param("base_station_addr");
Can't find handler for ASP command: wds_cgi_get_param("link_rate");
Can't find handler for ASP command: wds_cgi_get_param("signal_strength");
Can't find handler for ASP command: wds_cgi_get_param("repeater1_addr");
Can't find handler for ASP command: wds_cgi_get_param("repeater2_addr");
Can't find handler for ASP command: wds_cgi_get_param("repeater3_addr");
Can't find handler for ASP command: wds_cgi_get_param("repeater4_addr");
root@f5c3f91750ae:/fuzzing-firmware# /var/run/httpd.pid: No such file or directory

If so, execute a rm root/tmp/shm_id and try again. This should resolve the issue and allow you to proceed!

Let's investigate with the -strace option on QEMU:

$ chroot root /qemu-mips-static -strace -E LD_LIBRARY_PATH=/lib/public/ /usr/sbin/httpd
...
271 open("/tmp/shm_id",O_WRONLY|O_CREAT|O_TRUNC,0666) = 3
271 ioctl(3,21517,2147481080,0,0,0) = -1 errno=89 (Function not implemented)
271 brk(0x0069b000) = 0x0069b000
271 write(1,0x7f52d2a8,15)shm ID: 229383
= 15
271 write(3,0x699070,6) = 6
271 close(3) = 0
271 ipc(21,229383,0,2147481280) = 0
271 ipc(2,123456,1,1974) = -1 errno=17 (File exists)
271 write(1,0x7f52d2a8,25)Semaphore Create Failed.
= 25
271 exit(1)

Based on the output from -strace, we can tell that the reason for the failure might be something to do with the IPC and IOCTL calls related to shared memory operations (shm = shared memory) – a poorly supported feature in QEMU.

The only way we can find out for sure is in a disassembler. Let's open root/usr/sbin/httpd and its dependency root/lib/libnvram.so in Binary Ninja and have a look.

Using high level IL view (HLIL) in main at 0x4130d4, we can see that it does indeed look like we are failing after a failed call to sub_408f78 which in turn calls semget.

*Subroutine at 0x408f78*

*Semget*

semget likely makes the failed ipc call (since these semaphores are used for interprocess communication) and we should therefore avoid or fix it. One may be tempted to immediately resort to binary patching but since this binary was actually dynamically linked, we can use LD_PRELOAD instead to influence the behavior of the program without modifying the binary at all.

Another thing we might notice from our disassembler is that this main function does a lot more than just serve http requests. It looks like it brings a lot of the router up as well!

*Router bring-up code*

This means that we might want to enter the program at a different place besides main. Thankfully, if we can find the HTTP parse code, LD_PRELOAD can also be used to directly test drive an internal function.

Test Drive functions with LD_PRELOAD

First, let's focus on directly targeting the parse http request function. Through reverse engineering, we can find that this function is located at 0x408f90. Additionally, through more reverse engineering, we can find that this functions signature looks something like this:

void parse_http_req(char *http_req, void *unk, int32_t inet_addr, int32_t out_fd)

Test driving an individual function in LD_PRELOAD is easy once you know the trick. We simply override the libc main (in our case __uClibc_main) by defining a function of the same name in our LD_PRELOAD test driver and calling the function interest inside it.

// harness.c

#include <fcntl.h>
#include <stdio.h>
#include <stdint.h>
#include <unistd.h>

// the real signature is longer but turns out it doesn't matter
void __uClibc_main(void *main, int argc, char** argv)
{
    char req[4096 + 1];
    int32_t in_addr;

    if (argc != 2) {
        printf("Usage: %s <fuzz-file>\n", argv[0]);
        exit(1);
    }

    char *fuzzfile = argv[1];
    int fd = open(fuzzfile, O_RDONLY);

    int n_read = read(fd, &in_addr, sizeof(in_addr));
    if (n_read != sizeof(in_addr)) exit(1);

    n_read = read(fd, req, sizeof(req) - 1);
    if (n_read < 0) exit(1);
    req[n_read] = 0;

    fprintf(stderr, "Request: %s\n\n", req);

    // declare a function pointer pointing to the real parse_http_req
    void (*parse_http_req)(char *, void *, int32_t, int) = (void *)0x408f90;

    // Skip a lot of device init and get right to the server setup and http handler
    parse_http_req(req, NULL, in_addr, STDERR_FILENO);

    // need to exit here b/c this function is expected to not return
    exit(0);
}

As you can see, this test driver is just like normal in almost every way. There are a couple differences:

1. Instead of using main, start at the libc main (the function that calls the real main).
2. Because of this we need to exit at the end instead of returning (libc main calls exit with what the real main returns).
3. Instead of being to call the function directly, we need to make a function pointer to it and call that.

A couple of notes about this test driver:

1. We are testing both the request and the connecting address. This will check if there are any special cases or mishandled addresses.
2. We print each request to stderr and pass stderr to parse_http_req as the output FD. This will allow us to view results visually on the commandline when testing and in Mayhem.

Since LD_PRELOAD works by overriding shared library loads with a provided shared object, we need to compile harness.c to a shared object as well:

$ /fuzzing-firmware/cross-compiler-mips/bin/mips-gcc harness.c -o root/harness.so -shared -fPIC

Now that we have our LD_PRELOAD test driver compiled, let's create a test file since our test driver takes in a file. Now we can run httpd as before except we add the LD_PRELOAD environment variable and the test.txt argument.

$ echo AAAABBBBCCCC > root/test.txt
$ chroot root /qemu-mips-static -E LD_PRELOAD=/harness.so -E LD_LIBRARY_PATH=/lib/public/ /usr/sbin/httpd
Usage: /usr/sbin/httpd <fuzz-file>
$ chroot root /qemu-mips-static -E LD_PRELOAD=/harness.so -E LD_LIBRARY_PATH=/lib/public/ /usr/sbin/httpd test.txt
Request: BBBBCCCC


qemu: uncaught target signal 11 (Segmentation fault) - core dumped
Segmentation fault

Darn. Still crashing. Let's see why using -strace again:

$ chroot root /qemu-mips-static -strace -E LD_PRELOAD=/harness.so -E LD_LIBRARY_PATH=/lib/public/ /usr/sbin/httpd test.txt &
...
97 ipc(23,1074866065,131072,438) = -1 errno=2 (No such file or directory)
97 ipc(21,-1,0,2147275888) = -1 errno=22 (Invalid argument)
97 ipc(2,1074866065,1,1974) = -1 errno=17 (File exists)
97 ipc(2,1074866065,1,512) = 0
97 rt_sigprocmask(SIG_BLOCK,0x7ffcd400,NULL) = 0
97 ipc(1,0,1,0) = 0
--- SIGSEGV {si_signo=SIGSEGV, si_code=1, si_addr=0x0000001f} ---
qemu: uncaught target signal 11 (Segmentation fault) - core dumped
Segmentation fault

Looks like it's still crashing in the IPC code. But where? The majority of calls out of parse_http_req appear to be related to acosNvramConfig_. Let's try using GDB to break on one and check if the crash is inside. We can use QEMU with gdb-multiarch using the -g <port> argument to QEMU.

$ chroot root /qemu-mips-static -g 1234 -E LD_PRELOAD=/harness.so -E LD_LIBRARY_PATH=/lib/public/ /usr/sbin/httpd test.txt &
$ gdb-multiarch -q root/usr/sbin/httpd
(gdb) target remote :1234
Remote debugging using :1234
...
(gdb) break acosNvramConfig_match
Breakpoint 1 at 0x4ade90
(gdb) continue
...
Breakpoint 1, 0x004ade90 in acosNvramConfig_match ()
(gdb) finish
...
Program received signal SIGSEGV, Segmentation fault.
0x7f6f5f90 in ?? ()

Since the crash happened somewhere in acosNvramConfig_match, overriding this function should fix the crash. Let's give it a shot.

Overriding functions with LD_PRELOAD

With LD_PRELOAD we can override any dynamically linked function. In this case, we found that our program crashes in acosNvramConfig_match. So if we override this function by skipping it or re-implementing it ourselves, we should be able to avoid the crash all together.

Let's add an override for the match function to our test driver harness.so:

int acosNvramConfig_match(char *key, char *value) {
    printf("acosNvramConfig_match(%s, %s)\n", key, value);
    return 0;
}

Next, let's try running the new test driver version:

$ /fuzzing-firmware/cross-compiler-mips/bin/mips-gcc harness.c -o root/harness.so -shared -fPIC
$ chroot root /qemu-mips-static -E LD_PRELOAD=/harness.so -E LD_LIBRARY_PATH=/lib/public/ /usr/sbin/httpd test.txt
...
acosNvramConfig_match(passwordrecovered_debug2, 1)
acosNvramConfig_match(passwordrecovered_debug, 1)
acosNvramConfig_match(http_rmenable, 1)
qemu: uncaught target signal 11 (Segmentation fault) - core dumped
Segmentation fault

Unfortunately it looks like we're still segfaulting. However, if we look at the strace output, it looks like we have the same call pattern as before. Let's add overrides for the rest of the acosNvramConfig_* family (left as an exercise to the reader).

$ chroot root /qemu-mips-static -E LD_PRELOAD=/harness.so -E LD_LIBRARY_PATH=/lib/public/ /usr/sbin/httpd test.txt
Request: BBBBCCCC
...
acosNvramConfig_get(local_ip9)
acosNvramConfig_get(local_ip10)
acosNvramConfig_match(rm_access, ip_single)
acosNvramConfig_match(rm_access, ip_range)
acosNvramConfig_match(rm_access, ip_list)
acosNvramConfig_match(rm_access, all)

Great! Now we have no crashes when we run httpd with our test driver. However, recall that even though we passed stderr as our output file descriptor, we see no output here. This must mean that parse_http_req is relying on the outputs of acosNvramConfig_*, which is an indication that we need to increase the fidelity of our overrides.

Right now we are just returning default values (0 or the empty string) for these functions. Instead, let's now modify our test driver to get these values out of /etc/nvram in our firmware image.

Finally, running our httpd with our test driver produces the output that we want:

$ chroot root /qemu-mips-static -E LD_PRELOAD=/harness.so -E LD_LIBRARY_PATH=/lib/public/ /usr/sbin/httpd test.txt
...
HTTP/1.0 401 Unauthorized
WWW-Authenticate: Basic realm="NETGEAR DGN2200v4"
x-frame-options: SAMEORIGIN
Set-Cookie: XSRF_TOKEN=1222440606; Path=/
Content-type: text/html

<html>
<head>
<meta http-equiv='Content-Type' content='text/html; charset=utf-8'>
<title></title></head>
<body><h1></h1>
<p></p></body>
</html>

It's now time to get our firmware and compiled code into Mayhem!

Getting it into Mayhem

Surprisingly, once we have the firmware running locally, getting it into Mayhem is easy! If you recall from previous tutorials, Mayhem expects a package to look like:

my-package/
    Mayhemfile
    root/
        usr/
        bin/
        etc/
        ...
    [tests/]
        ...

Luckily, our unpacked firmware already looks like the root folder (indeed the root folder in a Mayhem package is a mini-filesystem). So, to make our firmware + test driver into a Mayhem package we just need to add a Mayhemfile. Of course, we will need to specify the environment variables to make sure Mayhem knows to LD_PRELOAD the test driver and where to look for the libraries.

project: netgear-n300
target: parse_http_request

cmds:
- cmd: /usr/sbin/httpd @@
  env:
    LD_LIBRARY_PATH: /lib/public
    LD_PRELOAD: /harness.so

Now we can mayhem run from the directory containing our firmware root directory and Mayhemfile:

$ ls
Mayhemfile harness.c root
$ mayhem run .
/var/folders/4z/qn09fnw164dd4jrghzws9q1h0000gn/T/tmp4ra3bszk/testsuite.tgz 100% |########################| Time:  0:00:00   1.4 KiB/s
Syncing /var/folders/4z/qn09fnw164dd4jrghzws9q1h0000gn/T/tmppqpnt2py 100% |###########################| Time:  0:00:00
/var/folders/4z/qn09fnw164dd4jrghzws9q1h0000gn/T/tmpd7x8xrim/root.tgz 100% |##########################| Time:  0:00:01   1.4 MiB/s
Run started: netgear-n300/parse-http-request/1
Run URL: https://tutorial.forallsecure.com:443/mayhemuser/netgear-n300/parse-http-request/1

Tip

Having issues getting your Mayhem run to execute successfully? You may need to remove the remaining symbolic links from the root folder like so: find root -type l -delete. Once finished, re-execute the Mayhem run.

🔍 Review It! N300 Firmware Testing

Solution

Download and extract the n300-firmware-fuzzing.tgz file to obtain the correct set up for testing the n300 firmware in Mayhem.

✏️ Summary and Recap

In this lesson, you learned how to extract, create a test driver, package, and test firmware binaries!

---

I learned how to...

1. Define the challenges regarding firmware testing.

• In particular, the challenges firmware testing presents are...
  1. Dependency on specific hardware features present on the physical device.
  2. Non-x86 processor architecture.
  3. Non-glibc C standard library.
  4. Lack of available source code or documentation.

2. Walk through a firmware testing example for the DGN2200v4 router firmware image.

• For this example, we will be looking at the Netgear N300 (henceforth refered to as DGN2200v4) router firmware image. This is a good target to look at because while it is a Linux firmware binary, it presents all of the challenges listed above. Specifically, this firmware...
  1. Relies on specific hardware features for synchronization used by its programs.
  2. Is compiled for the MIPS architecture.
  3. Uses uClibc instead of glibc C standard library.
  4. Has no source code and very few debug symbols available in binaries of interest.

3. Test drive and override functions with LD_PRELOAD.

• Test driving an individual function in LD_PRELOAD is easy once you know the trick. We simply override the libc main (in our case __uClibc_main) by defining a function of the same name in our LD_PRELOAD test driver and calling the function interest inside it.

4. Execute a Mayhem run on a target with the firmware test driver.

• We will need to specify the environment variables to make sure Mayhem knows to LD_PRELOAD the test driver and where to look for the libraries.

  project: netgear-n300
  target: parse_http_request
  
  cmds:
  - cmd: /usr/sbin/httpd @@
  env:
      LD_LIBRARY_PATH: /lib/public
      LD_PRELOAD: /harness.so