beginner

Testing a Non-Docker Target with Mayhem

Have a normal binary not contained within a Docker image? In this lesson, we'll walk you through how to package and fuzz your non-Docker targets with Mayhem packages!

---

Estimated Time: 30 minutes

Objectives

By the end of this lesson, you will be able to:

1. Package a target binary using the mayhem package command.
2. Execute a Mayhem run on the packaged testme target binary.
3. Set up and run regression testing via target replacement (no Mayhemfile modification).
4. Seed test case files for a packaged target.

Quickstart

Run through the lesson:

See prerequisites before beginning.

1. Package the testme binary using the mayhem package command:

   mayhem package testme -o /tmp/testme-pkg
   

2. Configure the Mayhemfile for the testme binary:

   project: testme
   target: testme
   duration: 90
   advanced_triage: true
   cmds:
     - cmd: /root/tutorial/testme/v1/testme @@
   

3. Execute the Mayhem run on the testme binary using the mayhem run command.

   mayhem run /tmp/testme-pkg
   

Prerequisites

You will need the following:

• The Mayhem CLI installed and authenticated with the Mayhem server.

• A Linux OS (macOS and Windows does not fully support mayhem package).

  For this reason we recommend using the forallsecure/tutorial Docker image as a means of using a Linux OS environment to follow along.

  docker pull forallsecure/tutorial:2.5
  docker run -ti --privileged --rm forallsecure/tutorial:2.5
  

Terminal Recording

Using the mayhem package Command

If Docker is unavailable, users can use the mayhem package command to automatically package an application binary by statically inferring all of its associated dependencies from the local filesystem and building a mini-root filesystem containing the application along with its determined dependencies.

Note

Any associated dependencies that remain undetected will have to be manually added to the mini-root filesystem by the user to ensure that the target application can execute properly. This is why we recommend packaging targets using Docker due to the fact that the target binary is already bundled with its necessary environment, thereby avoiding potential dependency issues.

To use the mayhem package command, you will need the following parameters:

mayhem package <path_to_binary> -o <output_path>

The general workflow for packaging and testing a non-Docker target in Mayhem is as follows:

1. Package the target binary using the mayhem package command.
2. Configure the generated Mayhemfile for the run.
3. Execute the mayhem run for the Mayhemfile with the mayhem run command.

Now that you're familiar with a brief overview of mayhem package, let's see how to use the mayhem package command for the testme application.

Packaging and Testing the testme Binary

In previous lessons we packaged and fuzzed the testme binary as a containerized application residing within a Docker container. However, if we were unable to use Docker but still had access to the testme binary, we would package and fuzz the testme binary as a Mayhem package instead.

Important

The mayhem package command only works within a Linux OS environment. Therefore macOS users should utilize the forallsecure/tutorial Docker image as a means of using a Linux OS environment to follow along. However, we also recommend existing Linux users to do the same as this lesson will assume you are working in the forallsecure/tutorial Docker image.

docker pull forallsecure/tutorial:2.5
docker run -ti --privileged --rm forallsecure/tutorial:2.5

For those without an Internet connection and therefore cannot pull the forallsecure/tutorial Docker image, download and extract the following: testme.zip

First, navigate to /root/tutorial/testme/v1 within the forallsecure/tutorial:2.5 Docker image.

1. Package the testme Binary

Next, we'll need to package the testme application binary. Execute the following in the current directory of the testme binary:

mayhem package ./testme -o /tmp/testme-pkg

This will create the following package and sub-contents located at /tmp/testme-pkg:

1. Mayhemfile: The configuration file for the Mayhem run
2. testsuite: The repository of test case files.
3. root: The mini-root filesystem containing the application binary and associated dependencies.

Then, you'll need to configure the Mayhemfile before you can execute a Mayhem run for your packaged target.

2. Configure the Mayhemfile

The Mayhemfile generated from the mayhem package command is automatically configured using what Mayhem inferred from the mayhem package command; however, users may need to configure their Mayhemfile with more details if necessary in order to get their applications to run properly in Mayhem.

Tip

You should keep these helpful tips in mind when configuring the Mayhemfile for more complex applications:

• Mayhem executes targets as an unprivileged user. If a target needs privileges, you should set the uid field as a top level configuration option in the Mayhemfile. For example, if your application needs to run as root, set uid: 0 in the Mayhemfile.

• Mayhem packages run inside of a Debian "Buster" Docker image by default. This means the package uses libc and other system libraries by Debian. You can change the base OS image with the image top-level directive. For example, if your application runs on Alpine Linux, use image: alpine.

• Mayhem runs packages with a very minimal user environment. If you need to set environment variables for a specific cmd, set the env as a list. For example, to put /usr/local/bin in your application PATH, set env: { "PATH": "/usr/local/bin:/usr/bin"}.

A mayhem package generated Mayhemfile will look similar to the following:

# Mayhem by https://forallsecure.com
# Mayhemfile: configuration file for testing your target with Mayhem
# Format: YAML 1.1

# Owned project name that the target belongs to
project: testme

# Target name (should be unique within the project)
target: testme

# Time to spend testing.
duration: 90

# Image to run the binary in.
# image: $MAYHEM_DOCKER_REGISTRY/forallsecure/debian-buster:latest

# Turns on extra test case processing (completing a run will take longer)
advanced_triage: false

# Configure tasks to be run. If null or missing, defaults to exploitability factors, regression testing, and behavior testing.
tasks:
  - name: exploitability_factors
  - name: regression_testing
  - name: behavior_testing

# List of commands used to test the target
cmds:

  # Command used to start the target, "@@" is the input file
  # (when "@@" is omitted Mayhem defaults to stdin inputs)
  - cmd: /root/tutorial/testme/v1/testme @@
    env: {}

Set the duration of the Mayhem run to 90 for the sake of this lesson (Mayhem runs are set to infinite duration by default). Here we see that the Mayhem run for the testme binary will take about 90 seconds and points to the location of the binary at /root/tutorial/testme/v1/testme, which corresponds to the root folder of the three assets (Mayhefile, testsuite, and root) that mayhem package creates. This Mayhem run uses a file input method for the testme binary indicated by @@.

Info

Your auto-generated cmd filepath may be differ from what you see above if you are not using the forallsecure/tutorial Docker image; the cmd filepath is auto-populated based on where the target application was originally packaged.

Once the Mayhemfile is properly configured, users can execute the mayhem run command pointing to the mayhem package directory containing the configured Mayhemfile.

3. Execute the Mayhem Run

Finally, execute the mayhem run command for the configured Mayhemfile:

mayhem run /tmp/testme-pkg

Info

You will need to use the mayhem login command to authenticate with the Mayhem server before you can execute your Mayhem run successfully. See Authenticating with the Mayhem Server for more information.

And that's it! Mayhem should be able to find the improper input validation defect for the testme binary in about 90 seconds. You can view the results in the Mayhem UI.

Regression Testing via Target Replacement

Previously, to perform regression testing you pointed a subsequent Mayhem run to a fixed/updated version of the testme binary via the cmd parameter and kept the same project and target Mayhemfile configuration values.

Note

Doing this effectively used previously generated test cases (from runs with the same project and target) in the new Mayhem run and compared previous (crashing) test cases with the updated (fixed) behavior of the new testme binary.

However, users can also perform regression testing by directly replacing the target binary specified in the Mayhemfile with an updated/fixed version of the application, thereby leaving the Mayhemfile configuration unmodified.

In particular there are two ways to set up regression tests for updated versions of a binary application:

1. Re-using project and target Mayhemfile parameters and pointing the cmd parameter to the location of an updated/fixed binary (seen here).
2. Replacing the previous binary with a fixed binary and keeping the Mayhemfile configuration the same.

Since we have direct access to the testme binary located within the /tmp/testme-pkg Mayhem package, let's see how we can set up regression testing doing a direct target replacement.

Info

Both Docker targets and non-Docker targets alike can use both methods for setting up regression tests; however, performing a direct replacement of the target binary will be more complicated for Docker targets as this requires a user to run the Docker image in interactive mode, replace the target binary, commit the change to the Docker image, and re-push the updated Docker image to either Docker Hub or the Mayhem Docker Registry. Hence, why the second method of target replacment for regression testing is shown in this lesson.

In the directory /root/tutorial/testme/v2, we've already provided a run.sh script to automatically replace the previous testme binary with the fixed version.

#!/bin/sh

# Copy over the new, fixed version into the package.
#
# Note that we are *overwriting* the old testme application.
# Overwriting an old version allows you to use your existing
# Mayhemfile
cp testme /tmp/testme-pkg/root/root/tutorial/testme/v1/testme

# Re-run mayhem. There is no need to edit the Mayhemfile.
# The run ID is saved to $id
id=$(mayhem run --regression /tmp/testme-pkg)

# Wait for the run to finish
mayhem wait $id --regression

# Sync the test suite to the "testsuite" directory.
mayhem sync /tmp/testme-pkg

Info

The --regression parameter to the mayhem wait command waits until the regression test phase completes.

Here we see that the updated testme binary replaces the previously packaged testme binary at /tmp/testme-pkg/root/root/tutorial/testme/v1/testme and then regression testing is executed for the packaged Mayhemfile.

Now just run the following script in the /root/tutorial/testme/v2 directory:

sh run.sh

Looking at the associated run in the Mayhem UI, we see that the regression testing results show that previously crashing test cases and their associated defects have been marked as fixed.

And that's it! Now you know how to perform the two different types of workflows for setting up regression testing!

Testing testme with a Seeded Test Suite

Prior to executing a Mayhem run, Mayhem will also scan for pre-existing test cases within the testsuite directory to use as a seed or "jumpstart" to improve the speed and coverage of the Mayhem run. Each file within the testsuite directory represents an individual input test case along with it's contents.

Info

Check out the testsuite parameter for more information on configuring a Mayhemfile to seed the corresponding Mayhem run with a designated test suite folder containing pre-existing test cases. Seeded test suites can be set for both Docker targets and non-Docker targets alike.

For example, let's take another look at the underlying testme code. For our Mayhem package located at /tmp/testme-pkg, if we were to create a seed.txt file and place it in the testsuite folder containing the following contents:

bu

Then the seeded test case would effectively optimize the performance of the Mayhem run due to the fact that the bu test case provides deeper initial coverage of the testme binary (rather than something like aa).

int fuzzme(char *buf)
{
  if(buf[0] == 'b')
    if(buf[1] == 'u')
      if(buf[2] == 'g') {
        printf("You've got it!");
        abort(); // Defect: SIGABRT.
      }
  return 0;
}

Therefore, place the seed.txt (containing the content bu) in the testsuite directory of our /tmp/testme-pkg folder:

cp seed.txt /tmp/testme-pkg/testsuite

Next, modify the Mayhemfile and change the target parameter value to testme-seed. This ensures that we start a fresh Mayhem run that will use our manually seeded test suite (Mayhem runs by default will seed the current run with the generated test suite from previous runs of the same <project>/<target>.)

project: testme
target: testme-seed
duration: 90
advanced_triage: true
cmds:
  - cmd: /root/tutorial/testme/v1/testme @@

Now re-execute the mayhem run command to use the seed test suite for the testme binary!

$ mayhem run /tmp/testme-pkg
/tmp/tmpo2ifgn0f/testsuite.tgz 100% |#############| Time:  0:00:00 224.7 B/s
Syncing /tmp/testme-pkg/testsuite 100% |####################| Time:  0:00:01
Run started: testme/testme-seed/1
testme/testme-seed/1

The Mayhem run for the testme binary should now find the underlying defect much quicker compared to the previous un-seeded run. This is because Mayhem had to do less exploration of the program as the seed test case bu provided a relevant starting point for covering the if statements checking for the input "bug" that leads to the improper input validation defect.

⚡ Real World Exercise: Seeding and Testing the testme-npd Binary

Now that you know how to package binaries and seed a test suite, let's see if you can do it for the testme-npd binary from the previous lesson.

Note

If you're using the forallsecure/tutorial Docker image, you may need to use the docker cp command to move the testme-npd.zip into your corresponding Docker container. Check out the official documentation on docker cp for more information.

Instructions:

1. Download and extract the following: testme-npd

2. Package the testme-npd binary using the mayhem package command.

3. Set the project and target values of the Mayhemfile to testme-npd. Also, set the duration of the to 60 seconds.

4. Provide a seed.txt test case file in the testsuite directory of the testme-npd package.

5. Execute the Mayhem run using the mayhem run command.

Hint

This Mayhem run is only 60 seconds long! You'll want to speed up the discovery of the null pointer dereference bug, therefore think about what text might improve coverage to the null pointer dereference bug.

🔍 Review It! Seeding and Testing the testme-npd Binary

Solution

First things first, you needed to package the testme-npd binary using the mayhem package command.

mayhem package testme-npd -o /tmp/testme-npd-pkg

Then, you needed to set the project, target, and duration parameter of the Mayhemfile to their corresponding values.

# Mayhem by https://forallsecure.com
# Mayhemfile: configuration file for testing your target with Mayhem
# Format: YAML 1.1

# Owned project name that the target belongs to
project: testme-npd

# Target name (should be unique within the project)
target: testme-npd

# Base image to run the binary in.
# image: $MAYHEM_DOCKER_REGISTRY/forallsecure/debian-buster:latest

# Time that analysis will run for in seconds - if absent, run forever
duration: 90

# Turns on extra test case processing (completing a run will take longer)
advanced_triage: false

# List of commands used to test the target
cmds:

  # Command used to start the target, "@@" is the input file
  # (when "@@" is omitted Mayhem defaults to stdin inputs)
  - cmd: /root/testme-npd/testme @@
    env: {}

    ...

Next, you should have created a seed.txt file with the content "nul" and placed it in the testsuite directory. The seed.txt test case file should speed up the discovery of the null pointer dereference bug by providing a relevant starting point for covering the if statements leading to the null pointer dereference code.

int testme(char *buf, unsigned len)
{
  if(buf[0] == 'b')
    if(buf[1] == 'u')
      if(buf[2] == 'g') {
        abort(); // Defect: SIGABRT.  
      }

  if(buf[0] == 'n')
    if(buf[1] == 'u')
      if(buf[2] == 'l')
        if(buf[3] == 'l') {
          int *p = NULL; 
          int y = *p; // Defect: null pointer dereference
        }

  return 0;
}

Lastly, upon executing the mayhem run command, you should have seen a run page similar to the following:

mayhem run /tmp/testme-npd-pkg

Well done! You're already building and optimizing fuzz targets all on your own!

✏️ Summary and Recap

In this lesson, you learned how to package up an application using the mayhem package command, configure the resulting Mayhemfile, execute your Mayhem run, and even seed a test suite to improve analysis and testing performance.

---

I learned how to...

1. Package a target binary using the mayhem package command.

• Mayhem can use the mayhem package command to automatically package up an application binary by statically inferring all of its associated dependencies and building a mini-root filesystem containing the application along with the determined dependencies.

• The mayhem package command follows the form:

  mayhem package <path_to_binary> -o <output_path>
  

• The package contents should look similar to the following:

  ├── /tmp/testme-pkg
      └── Mayhemfile - The configuration file for the Mayhem run
      └── testsuite - The repository of test case files.
      └── root - The mini-root filesystem containing the application binary.
  

2. Execute a Mayhem run on a packaged target.

• Upon packaging a target, users can use the mayhem run command for the resulting Mayhemfile located in the target package.

3. Set up and run regression testing via target replacement (no Mayhemfile modification).

• In particular there are two ways to set up regression tests for updated versions of a binary application:

  1. Re-using project and target Mayhemfile parameters and pointing the cmd parameter to the location of an updated/fixed binary (seen here).
  2. Replacing the previous binary with a fixed binary and keeping the Mayhemfile configuration the same.

• In the directory /root/tutorial/testme/v2, we've already provided a run.sh script to automatically replace the previous testme binary with the fixed version.

  #!/bin/sh
  
  # Copy over the new, fixed version into the package.
  #
  # Note that we are *overwriting* the old testme application.
  # Overwriting an old version allows you to use your existing
  # Mayhemfile
  cp testme /tmp/testme-pkg/root/root/tutorial/testme/v1/testme
  
  # Re-run mayhem. There is no need to edit the Mayhemfile.
  # The run ID is saved to $id
  id=$(mayhem run --regression /tmp/testme-pkg)
  
  # Wait for the run to finish
  mayhem wait $id --regression
  
  # Sync the test suite to the "testsuite" directory.
  mayhem sync /tmp/testme-pkg
  

4. Seed test case files for a packaged target.

• Prior to executing a Mayhem run, Mayhem will also scan for pre-existing test cases within the testsuite directory to use as a seed or "jumpstart" to improve the speed and coverage of the Mayhem run. Each file within the testsuite directory represents an individual input test case along with it's contents.

• For example, if we take a look at the underlying testme code. For our Mayhem package located at /tmp/testme-pkg, if we were to create a seed.txt file and place it in the testsuite folder containing the following contents:

  bu
  

• Then the seeded test case would effectively optimize the performance of the Mayhem run due to the fact that the bu test case provides deeper initial coverage of the testme binary (rather than something like aa).

  int fuzzme(char *buf)
  {
    if(buf[0] == 'b')
      if(buf[1] == 'u')
        if(buf[2] == 'g') {
          printf("You've got it!");
          abort(); // Defect: SIGABRT.
        }
    return 0;
  }