expert

Hardware Testing

In this lesson, we'll walk you through how to create a test driver to test software with hardware dependencies.

---

Estimated Time: 10 minutes

Objectives

By the end of this lesson, you will be able to:

1. Explain cases in which hardware might talk to software.
2. Discuss strategies for test driving software with hardware dependencies.
3. Walk through an example hardware test driver.

When Software Meets Hardware

Earlier, we loosely defined a program as something that read input and produced output. Often, one or both of those will be handled by hardware.

Sometimes, software will read input from sensors such as:

1. A digital thermometer reading from temperature sensor
2. A Fitbit reading from a gyroscope
3. Pokémon GO reading from an iPhone’s GPS
4. Backup Alarm Systems in cars reading from LIDAR
5. Vessel Tracking Systems on ships reading from AIS transponders
6. Altitude Warning Systems on planes reading from an altimeter

Other times, software will read input from devices or peripherals such as:

1. Common USB devices

   1. Touch: keyboard, mouse, joypad
   2. Audio: microphone
   3. Video: webcam

2. The steering wheel in a vehicle or plane

And, sometimes, software will send output to hardware, too:

1. Displaying graphics on a screen
2. Lane Assist in cars
3. Auto-pilot in UAVs
4. Preventing energy spikes in power grids
5. Ensuring constant pressure in water treatment facilities
6. Controlling robotic arms in manufacturing plants

These systems tend to be critical and/or high-availability, so having a good set of test cases that ensures they won’t crash is important. In all of these cases, if we want to test the software, we’ll need to emulate or replace these hardware interfaces. Once we do, we’ll be able to analyze it with Mayhem.

Test Driver Strategies

When creating a test driver, remember that Mayhem (and other software testing tools) will assume the software under test has a single input. Our goal when test driving, therefore, is to find a way to represent hardware data as a single input file or network connection. Depending on how the software actually “speaks” with the hardware, this can be easier or harder to do.

Hardware and software can “talk” in a few major ways:

1. Through physical connections (JTAG, serial, USB, 802.11x, etc).
2. Through direct memory access (DMA).

Sometimes, these are direct links. For example, in certain embedded platforms (like the Nintendo Entertainment System), writing to a specific memory location in RAM will actually perform the DMA to an APU or GPU automatically. In most modern systems, however, these links are managed by the kernel itself. The kernel will then provide an interface to userland applications to abstract away device specifics and/or ensure proper management of resources.

When Mayhem analyzes software, it does so in an isolated environment that won’t have access to the hardware or these kernel interfaces. As mentioned previously, we can still test the software by emulating the hardware or replacing functionality with something similar. In both cases, we’ll generally require some actual captures of device data in order to create the test driver.

Dealing with Direct Memory Access

DMA is sometimes quite easy to deal with as you only need to create an internal representation of the address space for testing. Unfortunately, it’s also sometimes quite difficult because you’ll need to emulate changes to the memory regions being accessed that would occur under normal operation. As a result, the changes required for test driving tend to be less invasive than when dealing with other problems, but are often more complicated, too.

To emulate DMA, it’s sometimes possible to map a region of memory inside the process that will allow reads/writes to function normally. If, for example, you have a DMA buffer at addresses 0x005c0000-0x005fffff, you may be able to mmap (with MAP_FIXED) a region of memory there instead of initializing the actual DMA region. Populating the region with data you expect to be there during operation of the device is also important. Both of these should be done as early in the program’s execution as possible, prior to any accesses of the DMA region.

Since the device itself may modify certain addresses in memory to indicate state, you may need a separate function that will make those modifications as well. Identifying a good place to put this logic can be difficult. In general, a good place to perform the modifications is right at the beginning of an event loop (prior to the DMA region being accessed) or right at the end of an event loop (to generate new events and set the DMA region up for the next access).

Dealing with API Calls

Calls to APIs like ioctl on Linux and DeviceIoControl on Windows can often be replaced with a series of functions. In some cases, the result of the ioctl should be relatively static (e.g. getting a device name). In other cases, you’ll need to read from an input file that your testing (such as Mayhem) can control. In still other cases, you’ll need to emulate logic from the hardware and provide the expected response (either from an input file, a data file, or a hard-coded value).

Calls to APIs that deal with reading/writing serial or USB data are usually a little easier to test. The main reason is that they generally already take the form of reading/writing from a file or a network socket. The best strategy, therefore, is to turn those into actual file/network reads/writes. The hardest part is generally deciding if timing requirements (like baud rate) need to be enforced and, if so, how to enforce them.

Having actual data captures from the hardware to provide to Mayhem as input test cases is extremely important to ensure not only that the test driver does what is expected, but that Mayhem can quickly begin to produce useful analysis. Just remember that the captured data may need to have things added, removed, or changed depending on what edits were made to API calls to get around them.

Dealing with Other Issues

There are other problems that can crop up during testing as well. For example, many systems that interact with hardware care about device states or timing data. Providing general guidance on how to deal with these issues is difficult, but the best place to start is probably: “Can you encode it in your input stream?”

If, for example, the software expects to wait 2 or 3 “time steps” before receiving a certain message, is it possible to create a new message type that indicates “a time step has passed” (it's sometimes even better if this message can include an explicit relative timestamp, too.). The same goes for device state changes: Can a message be “sent” or “received” that encodes the state transition?

If so, then Mayhem should be able to modify the input to explore different state transitions that may or may not be expected to happen. This will ensure good test coverage of all possible states, rather than just the few that you expect (as is generally the case with hand-written unit tests).

An Example Test Driver

File: hid-example.c

Tip

The file can also be found in the Linux kernel source tree at samples/hidraw/hid-example.c.

Below, we’ll walk through the process of test driving the Linux kernel team’s example HIDRAW device code. Snippets of this file will be reproduced and modified as we step through the process of test driving it.

Identifying Input Sources

The first step is determining what our input sources are. In this example, there is a single source of input: /dev/hidraw0. It’s opened on line 53:

fd = open("/dev/hidraw0", O_RDWR|O_NONBLOCK);

if (fd < 0) {
    perror("Unable to open device");
    return 1;
}

Here, we need to make two changes:

1. We want to swap the input device /dev/hidraw0 out for an input file. This will enable us to feed symbolic input from Mayhem to the program during analysis.
2. Software testing systems like Mayhem often assume that the input won’t be modified during the test. Since this program also writes to the device, we’ll want to open up a separate file descriptor for output. Even if this isn’t required for the test, it’s generally good practice to split the two up as it makes things easier to debug for developers later if issues are found.

Here’s one way of making these modifications:

fd = open(argv[1], O_RDONLY);

if (fd < 0) {
    perror("Unable to open input file");
    return 1;
}

out = open("output", O_WRONLY);

if (out < 0) {
    perror("Unable to open output file");
    return 1;
}

This will give us a single, concrete input file (argument 1) and a separate output file (output).

Note

We removed O_NONBLOCK because, according to the Linux API manual, it doesn’t do anything for file access. We’ll also need to add the out file descriptor to the variable declaration on line 45.

Another consequence of test driving this way is that results of device queries will need to come from the same input file as device data reads. In reality, this may be a poor choice. It may be better to keep that information in separate data files (e.g. device_name.dat, device_desc.dat, etc.) and read each of those in separately.

Note

The decision to test drive this way for this example was made for ease of explanation. If you need more control over your application for testing, you should structure your changes more modularly as described above.

Fixing Includes

Since we’re stripping out all code related to hardware, we’ll also be disposing of the need for any #include directives that aren’t from a shared library like libc. For this example, this means removing the following lines (14-15) from the source code:

#include <linux/input.h>
#include <linux/hidraw.h>

We will need some information from these files, though. Specifically, we’ll need to know what some of the #define macros are from linux/hidraw.h in order to know how to replace code later:

struct hidraw_report_descriptor {
    __u32 size;
    __u8 value[HID_MAX_DESCRIPTOR_SIZE];
};

struct hidraw_devinfo {
    __u32 bustype;
    __s16 vendor;
    __s16 product;
};

/* ioctl interface */
#define HIDIOCGRDESCSIZE    _IOR('H', 0x01, int)
#define HIDIOCGRDESC        _IOR('H', 0x02, struct hidraw_report_descriptor)
#define HIDIOCGRAWINFO      _IOR('H', 0x03, struct hidraw_devinfo)
#define HIDIOCGRAWNAME(len)     _IOC(_IOC_READ, 'H', 0x04, len)
#define HIDIOCGRAWPHYS(len)     _IOC(_IOC_READ, 'H', 0x05, len)
/* The first byte of SFEATURE and GFEATURE is the report number */
#define HIDIOCSFEATURE(len)    _IOC(_IOC_WRITE|_IOC_READ, 'H', 0x06, len)
#define HIDIOCGFEATURE(len)    _IOC(_IOC_WRITE|_IOC_READ, 'H', 0x07, len)

#define HIDRAW_FIRST_MINOR 0
#define HIDRAW_MAX_DEVICES 64
/* number of reports to buffer */
#define HIDRAW_BUFFER_SIZE 64

We’ll need to copy the two struct definitions above into our program. The rest we can replace during the test driving process. In order to use the struct definitions, we’ll also need the value of HID_MAX_DESCRIPTOR_SIZE from hid.h:

#define HID_MAX_DESCRIPTOR_SIZE     4096

To understand the sizes and types being used for different ioctl calls, we’ll also need to look up some definitions from ioctl.h:

#define _IOC_NRBITS 8
#define _IOC_TYPEBITS   8
#define _IOC_SIZEBITS  14

#define _IOC_NRSHIFT    0
#define _IOC_TYPESHIFT  (_IOC_NRSHIFT+_IOC_NRBITS)
#define _IOC_SIZESHIFT  (_IOC_TYPESHIFT+_IOC_TYPEBITS)
#define _IOC_DIRSHIFT   (_IOC_SIZESHIFT+_IOC_SIZEBITS)

#define _IOC(dir,type,nr,size) \
    (((dir)  << _IOC_DIRSHIFT) | \
    ((type) << _IOC_TYPESHIFT) | \
    ((nr)   << _IOC_NRSHIFT) | \
    ((size) << _IOC_SIZESHIFT))

#define _IOC_READ  2U

#define _IOC_TYPECHECK(t) (sizeof(t))

#define _IOR(type,nr,size)  _IOC(_IOC_READ,(type),(nr),(_IOC_TYPECHECK(size)))

We’ll keep these handy for the next section.

Next, we’ll need to copy the four bus types out of input.h so that the bus_str function will compile correctly:

#define BUS_USB         0x03
#define BUS_HIL         0x04
#define BUS_BLUETOOTH   0x05
#define BUS_VIRTUAL     0x06

Lastly, we can remove lines 30-33 since we’ll be eliminating the need for them as part of the test driving process (they’re actually a duplicate from hidraw.h meant to provide redundancy for older systems, anyway).

Replacing API Calls

As explained earlier, in modern systems, the kernel generally manages hardware itself and simply provides an interface to userland code for interacting with it. In our example here, specific system calls (ioctl's) are being made to interact with the hardware. Since the isolated environment Mayhem will be running the software in won’t be able to use these ioctl's, we’ll need to swap them out for something it can use.

In most cases, the best thing to do is to turn them into file reads and file writes. Since we’ve already opened an actual file, rather than a device, this becomes fairly easy to do.

The following are all of the calls to ioctl in the program:

// line 65
res = ioctl(fd, HIDIOCGRDESCSIZE, &desc_size);

// line 73
res = ioctl(fd, HIDIOCGRDESC, &rpt_desc);

// line 84
res = ioctl(fd, HIDIOCGRAWNAME(256), buf);

// line 91
res = ioctl(fd, HIDIOCGRAWPHYS(256), buf);

// line 98
res = ioctl(fd, HIDIOCGRAWINFO, &info);

// line 114
res = ioctl(fd, HIDIOCSFEATURE(4), buf);

// line 122
    res = ioctl(fd, HIDIOCGFEATURE(256), buf);

Since a given ioctl may be a read, a write, or both, we’ll need to look at each one in context to understand how it should be replaced. To do this, we’ll need to look at the second argument to the ioctl and the information we pulled out of the header files above. The result of this lookup should be the following conclusions:

• Line 65: This is a read (_IOR) of size sizeof(int)
• Line 73: This is a read (_IOR) of size sizeof(struct hidraw_report_descriptor)
• Line 84: This is a read (_IOR) of size 256 (sizeof(buf))
• Line 91: This is a read (_IOC_READ) of size 256 (sizeof(buf))
• Line 98: This is a read (_IOC_READ) of size sizeof(struct hidraw_devinfo)
• Line 114: This is a read and a write (_IOC_READ|_IOC_WRITE) of size 4
• Line 122: This is a read and a write (_IOC_READ|_IOC_WRITE) of size 256 (sizeof(buf))

With those mappings, we should now be able to transform the ioctl calls into appropriate read and write calls instead:

// line 65
res = read(fd, &desc_size, sizeof(int));

// line 73
res = read(fd, &rpt_desc, sizeof(struct hidraw_report_descriptor));

// line 84
res = read(fd, buf, sizeof(buf));

// line 91
res = read(fd, buf, sizeof(buf));

// line 98
res = read(fd, &info, sizeof(struct hidraw_devinfo));

// line 114
write(out, buf, 4);
res = read(fd, buf, 4);

// line 122
write(out, buf, sizeof(buf));
res = read(fd, buf, sizeof(buf));

In addition to the calls to ioctl, there are also two other calls that result in hardware I/O that we’ll need to look at as well:

// line 136
res = write(fd, buf, 2);

// line 145
res = read(fd, buf, 16);

Since both of these are already file I/O, we simply need to make sure we’re supplying the right file descriptor in each case. In this example, we’ll only need to make a single change: Swapping fd with out on line 136:

// line 136
res = write(out, buf, 2);

And, with that, we’re done! The last finishing touches we’ll need to add are to close the out file descriptor at the end of main and remove sys/ioctl.h from the #include's at the top of the file.

Tip

Check your work with the hid-harness.c solution file.

Building the Test Driver

Now that we’ve made all of the changes necessary, we’ll need to build the test driver. Our example should be trivially compiled with the following:

gcc hid-harness.c -o hid-harness

Packaging the Application

Once the test driver is built, we can build a package for Mayhem:

mayhem package -o /tmp/hid-harness hid-harness

We’ll need to add a blank output file before using the package, though:

touch /tmp/hid-harness/root/path/to/output

Info

Replace /path/to in the above example with the path to your hid-harness executable.)

Then, we can upload the harness to begin testing!

mayhem run /tmp/hid-harness

✏️ Summary and Recap

In this lesson, you learned how to create a test driver to test software with hardware dependencies!

---

I learned how to...

1. Explain cases in which hardware might talk to software.

• Sometimes, software will read input from sensors such as:

  1. A digital thermometer reading from temperature sensor
  2. A Fitbit reading from a gyroscope
  3. Pokémon GO reading from an iPhone’s GPS
  4. Backup Alarm Systems in cars reading from LIDAR
  5. Vessel Tracking Systems on ships reading from AIS transponders
  6. Altitude Warning Systems on planes reading from an altimeter

• And, sometimes, software will send output to hardware, too:

  1. Displaying graphics on a screen
  2. Lane Assist in cars
  3. Auto-pilot in UAVs
  4. Preventing energy spikes in power grids
  5. Ensuring constant pressure in water treatment facilities
  6. Controlling robotic arms in manufacturing plants

2. Discuss strategies for testing software with hardware dependencies.

• When testing, remember that Mayhem (and other software testing tools) will assume the software under test has a single input. Our goal when testing, therefore, is to find a way to represent hardware data as a single input file or network connection. Depending on how the software actually “speaks” with the hardware, this can be easier or harder to do.
• Hardware and software can “talk” in a few major ways:
  1. Through physical connections (JTAG, serial, USB, 802.11x, etc).
  2. Through direct memory access (DMA).

3. Walk through an example hardware test driver.

• You'll need to perform the following operations:
  1. Identify input sources
  2. Remove any #include directives.
  3. Replace any existing API calls.
  4. Build the test driver.
  5. Package the application.