beginner

Code Testing in Your Browser¶

All systems go! In this lesson, we'll walk you through firing up your first Mayhem run via the Mayhem UI.

Estimated Time: 15 minutes

Objectives

By the end of this lesson, you will be able to:

Execute your first Mayhem run on a Docker image (containing the testme target) via the Mayhem UI.
Reproduce defects generated by crashing test cases.
Perform regression testing on a fixed/updated version of the testme target.
Execute a Mayhem run for the popular lighttpd web server via the Mayhem UI.

Quickstart

Run through the lesson:

To use the Mayhem UI to fuzz a containerized application residing in a Docker image, you will need to follow the process of:
1. Logging in to the Mayhem UI hosted on your Mayhem deployment.
2. Selecting a Docker Image for your new Mayhem run.
3. Configuring the Mayhemfile for the run.
4. Executing your new Mayhem run!

Create a new Mayhem run for the testme target residing in the forallsecure/tutorial Docker image using the Mayhem UI with the following Mayhemfile configuration:

image: forallsecure/tutorial:latest
duration: 90
project: forallsecure-tutorial
target: testme
cmds:
  - cmd: /root/tutorial/testme/v1/testme @@

Execute a subsequent Mayhem run for the updated testme target residing in the $MAYHEM_DOCKER_REGISTRY/forallsecure/tutorial Docker image using the Mayhem UI with the following Mayhemfile configuration (regression testing only):

image: $MAYHEM_DOCKER_REGISTRY/forallsecure/tutorial:latest
duration: 90
project: forallsecure-tutorial
target: testme
tasks:
  - name: regression_testing
cmds:
  - cmd: /root/tutorial/testme/v2/testme @@

Creating A New Mayhem Run from a Docker Image¶

Provided as an intuitive web interface, users can use the Mayhem UI to create, manage, and analyze Mayhem security testing for their applications or targets. From the Mayhem UI, users can initiate Mayhem runs for testing containerized applications residing within Docker images that have been uploaded to the public Docker Hub registry.

Provided as an intuitive web interface, users can use the Mayhem UI to create, manage, and analyze Mayhem security testing for their applications or targets. From the Mayhem UI, users can initiate Mayhem runs for testing containerized applications residing within Docker images that have either been uploaded to Mayhem's internal Docker Registry or hosted on the public Docker Hub registry.

How do you do this? It's as easy as:

Logging in to the Mayhem UI hosted on your Mayhem deployment.
Selecting a Docker Image for your new Mayhem run.
Configuring the options for the run.
Executing your new Mayhem run.

Let's see how this works in more detail. To get you quickly started, we've provided a button for you to create a new pre-configured run using a Docker image hosted on Docker Hub. For this exercise we will be executing a Mayhem run on the previously discussed testme application.

Let's see how this works in more detail. To get you quickly started, we've provided a button for you to create a new pre-configured run using a Docker image hosted on Mayhem's Docker Registry. For this exercise we will be executing a Mayhem run on the previously discussed testme application.

After clicking the button, you should be presented with the following screen:

Here you can create a new code testing project and execute a new Mayhem run. There are two main steps to executing a new Mayhem run:

Choose Image: Specify the Docker image that Mayhem will use for security testing a containerized target application. Choose an image residing on the public Docker Hub registry.
Configure and Run: Configure the options for the Mayhem run, then confirm your selections and execute the run.

Choose Image: Specify the Docker image that Mayhem will use for security testing a containerized target application. Choose an image residing on either the private Mayhem Docker Registry or from the public Docker Hub registry.
Configure and Run: Configure the options for the Mayhem run, then confirm your selections and execute the run.

Click the Show Mayhemfile link at the bottom to get a better look at the exact specifications for the Mayhem run dictated by the underlying Mayhemfile. You should see something similar to:

image: index.docker.io/forallsecure/tutorial:latest
duration: 90
project: forallsecure-tutorial
target: testme
cmds:
  - cmd: /root/tutorial/testme/v1/testme @@

image: $MAYHEM_DOCKER_REGISTRY/forallsecure/tutorial:latest
duration: 90
project: forallsecure-tutorial
target: testme
cmds:
  - cmd: /root/tutorial/testme/v1/testme @@

Info

The $MAYHEM_DOCKER_REGISTRY variable is used to represent the URL of the local Mayhem Docker registry (e.g. tutorial.forallsecure.com:5000); public Docker images on the other hand do not require such a URL and pull straight from Docker Hub. Check out the Mayhemfile Specification documentation for a full reference on each of the configuration options for a Mayhemfile.

Once you've confirmed your selections, click Create project and run to execute your first Mayhem run! Upon executing your Mayhem run, you should see the following run analysis page:

The run analysis page indicates the progress of the Mayhem run and will continue to auto-refresh until the end of its duration. In this time, Mayhem will test the target application and generate as many test cases as it can to detect potential security vulnerabilities in the target application. Test cases that result in crashes or defects will be marked accordingly for future reference.

In particular, if we scroll down to the bottom of the run analysis page, we can see more information about what defects Mayhem found when testing the containerized testme target and even the individual test cases that were used during testing.

For this particular testme target, there appears to be an underlying improper input validation defect.

Warning

Keep in mind that the exact hash/ID of your test case for your specific Mayhem run may differ from what is shown in this lesson.

And when we switch the pane's view from Defects to Test Cases, we see the individual breakdown of test cases generated during the Mayhem run; Mayhem generated two particular test cases for this run resulting in the specified defect.

Nice job executing your first Mayhem run and finding a defect for the testme target binary!

To confirm that the defect is indeed valid, we can reproduce the defective behavior of the testme binary by manually testing the testme binary with the test case that Mayhem generated and saved. Users can download their test cases for this very use case.

Reproducing Defects via Generated Test Cases¶

Click on the Defect ID (TDID-474151: Improper Input Validation) for one of the crashing test cases. The Mayhem UI should then reveal a screen that indicates generated test cases that resulted in the defect selected.

Here Mayhem also provides the necessary commands to re-execute the testme binary with the given input test case to reproduce the logged behavior.

./root/tutorial/testme/v1/testme <TESTCASE_ID>

Therefore, if we were to download and extract the testme binary as well as the crashing test case, and then execute the commands to reproduce the defect, we should see something similar to the following:

~/tutorial/testme/v1# ./testme 318d86b978cf3774c6d38c5265b0daf19ab17dc49b25ebea42de273e64d4a05c
Aborted

Info

Here we provide the testme binary for download, which requires a Linux OS to execute. If you have Docker installed, you can test the containerized testme target within the Docker container itself. We will show you how to do this in the next lesson: CLI Testing.

And there it is! We can see that the defect has been reproduced for the given input test case. Now, let's fix the underlying code, re-compile the testme binary, and execute another Mayhem run to confirm that the defect (and its associated test cases) are fixed!

Running Regression Testing and Confirming Fixes¶

When Mayhem generates test cases involved with testing a target application, it also saves the test cases for future Mayhem runs of the same target. This way, future Mayhem runs can utilize those previously generated test cases to confirm if the current behavior of the target application has changed (i.e. previous passing test cases now crash or previous crashing test cases now pass). This is called regression testing.

Info

Mayhem will re-use the same test suite for future Mayhem runs of a given <project>/<target> run of a particular owner. For this example, Mayhem will re-use the generated test suite for the run forallsecure-tutorial/testme owned by your Mayhem user account.

Let's see how this works in practice. Recall that we just fuzzed the testme application that was shown to have an improper input validation defect:

CWE-20: Improper input validation defect.

Now, we'll want to fix the underlying defect and execute a regression test to fuzz the target with the previously generated test cases and confirm that the found defect has been fixed.

Let's take a look at a fixed version of our testme application.

#include <stdio.h>
#include <string.h>

int fuzzme(char *buf)
{
  if(buf[0] == 'b')
    if(buf[1] == 'u')
      if(buf[2] == 'g') {
        return 0; // Fixed: No more defect.
      }
  return 0;
}

int main(int argc, char *argv[])
{
  FILE *f;
  char buf[12];

  if(argc != 2){
    fprintf(stderr, "Must supply a text file\n");
    return -1;
  }
  f = fopen(argv[1], "r");
  if(f == NULL){
    fprintf(stderr, "Could not open %s\n", argv[1]);
    return -1;
  }
  if(fgets(buf, sizeof(buf), f) == NULL){
    fprintf(stderr, "Could not read from %s\n", argv[1]);
    return -1;
  }
  fuzzme(buf);
  return 0;
}

Simply changing the abort(); to return 0; on line 9 should be enough to fix the improper input validation defect. Now, click on the button below to run a regression test for the fixed testme application!

Within the create new run flow, click the Show Mayhemfile link at the bottom of the page to confirm that your Mayhemfile looks similar to the following:

image: index.docker.io/forallsecure/tutorial:latest
duration: 90
project: forallsecure-tutorial
target: testme
tasks:
  - name: regression_testing
cmds:
  - cmd: /root/tutorial/testme/v2/testme @@

image: $MAYHEM_DOCKER_REGISTRY/forallsecure/tutorial:latest
duration: 90
project: forallsecure-tutorial
target: testme
tasks:
  - name: regression_testing
cmds:
  - cmd: /root/tutorial/testme/v2/testme @@

Once ready, go ahead and kick off the regression test for the recent fix of the testme application!

If we scroll down to the bottom of the Mayhem run page, we can see the results of the regression test on a per test case basis. Notice how previously crashing test cases and their associated defects have been marked as fixed.

And that's it! Well done. You've now not only used Mayhem to find defects for a target application, but also confirmed that the test cases relating to the defects have been resolved upon testing a fixed version of the target binary and it's underlying code!

Tip

Check out the Integrations page for more information on automating this testing and regression testing life cycle within a CI/CD pipeline.

⚡ Real World Exercise: Testing the `lighttpd` Docker Image¶

Lighttpd is an open-source web server optimized for speed with considerations for compliance, security, and flexibility. In the past, lighttpd 1.4.15 version had a few vulnerabilities that have since been patched.

Let's see if you can sniff out those bugs by creating a new Mayhem run for the forallsecure/tutorial/lighttpd:1.4.15 Docker image located in the Mayhem Docker registry!

Let's see if you can sniff out those bugs by creating a new Mayhem run for the forallsecure/lighttpd:1.4.15 Docker image located on Dockerhub!

Instructions:

Navigate to the Create New Run page.
Enter forallsecure/lighttpd:1.4.15 as the docker image you would like to test. Provide a project name for putting your new run in.

In the Command(s) section enter the value /usr/local/sbin/lighttpd -D -f /usr/local/etc/lighttpd.conf. Under the top-level tab Command (1) set Network Settings -> Network URL to tcp://localhost:80 and Network Settings -> Network Timeout to 2. You can also optionally change your test duration to 90 seconds. Your mayhem command at the bottom should look like the following:

mayhem run forallsecure/lighttpd:1.4.15 --owner myuser \
  --project forallsecure-tutorial \
  --target testme --docker --duration 90 \
  --cmd '/usr/local/sbin/lighttpd -D -f /usr/local/etc/lighttpd.conf' \
  --network-url tcp://localhost:80 --network-timeout 2

Info

Network targets such as lighttpd will need to have their network settings configured within the Mayhemfile for Mayhem to properly run and test the network application. Check out the Mayhemfile Specification documentation for more information on network settings.

Click Create project and run to execute a new Mayhem run for the lighttpd Docker image! Mayhem should be able to find the defect for the lighttpd target within 90 seconds.

Navigate to the Create New Run page.
Select Mayhem Docker Registry as your image source and search for the forallsecure/tutorial/lighttpd Docker repository. Select the 1.4.15 tagged Docker image and click Next to configure the corresponding Mayhemfile.

Verify that the cmds value is set to /usr/local/sbin/lighttpd -D -f /usr/local/etc/lighttpd.conf and that the Mayhemfile preview looks similar to the following:

image: $MAYHEM_DOCKER_REGISTRY/forallsecure/tutorial/lighttpd:1.4.15
duration: 90
project: forallsecure-tutoriallighttpd
target: '1415'
cmds:
  - cmd: /usr/local/sbin/lighttpd -D -f /usr/local/etc/lighttpd.conf
    network:
      url: tcp://localhost:80
      timeout: 2
      client: false

Info

Network targets such as lighttpd will need to have their network settings configured within the Mayhemfile for Mayhem to properly run and test the network application. Check out the Mayhemfile Specification documentation for more information on network settings.

Click Next until you reach the end of the create new run flow. When ready, confirm you selections and click Start Run to execute a new Mayhem run for the lighttpd Docker image! Mayhem should be able to find the defect for the lighttpd target within 90 seconds.

🔍 Review It! Testing the `lighttpd` Docker Image¶

Solution

If you were able to successfully execute a Mayhem run on the lighttpd:1.4.15 image, you should see a Run page similar to the following:

Let's now take a closer look at what Mayhem found! Scroll down to the Analysis and Test Cases/Defects pane.

Here we can view the following metrics:

Defects Found: 1
Crashing Test Cases: 21
Runtime Errors: 0
Tests Run: 2,082,039
Tests Run Per Second: 2,761.32
Test Suite Size: 420
Edges Covered: 5,343

Your Mayhem run results should look similar to but may not be exactly what is shown above. This is because when Mayhem tests a target binary, the random input from fuzzing can result in differing levels of coverage for runs of the same target (and duration).

In addition, if we navigate to the Test Cases pane, we can see that for the 21 crashing test cases and 1 unique defect that were found, additional detail is provided for each individual test case of the test suite:

Test Case ID: The unique identifier of the test case, accompanied by a link for further details and results of the individual test case.
Test Case Result: The outcome of the input test case.
Defect Count: The number of defects found per test case.
Defects: The number of defects found as a result, and the description of the defects found.
Time: The time of the resulting program crash during the Mayhem run.
Download: A download button to download and save each individual test case.

And that's it for now! Play around and see what else you can find about the resulting test cases for lighttpd and how they impacted the underlying code!

Nice job completing your first hands-on lesson!

✏️ Summary and Recap¶

In this lesson, you learned how to execute your first Mayhem run using the Mayhem UI!

The Mayhem UI makes it easy for users to ingest and execute runs on publicly available Docker images. Users simply need to navigate to the Create New Run page, select their Docker image source, and configure the Mayhemfile for the run before confirming their selections and executing the run!

The Mayhem UI makes it easy for users to ingest and execute runs on private or publicly available Docker images. Users simply need to navigate to the Create New Run page, select their Docker image source, and configure the Mayhemfile for the run before confirming their selections and executing the run!

I learned how to...

1. Execute my first Mayhem run on a Docker image (containing the testme target) via the Mayhem UI.

Within the Mayhem UI, users are provided an easy-to-use web interface for creating and managing Mayhem runs on containerized applications, or targets, residing within Docker images that have either been uploaded to Mayhem's internal Docker Registry or hosted on the public Docker Hub registry. Through the Mayhem UI, users can not only test their containerized applications but also view analytic results of their testing outcomes.
- There are three stages to the create new run process:
  1. Choose Image: Choose the Docker image that will be ingested into Mayhem for fuzz testing, and specify from where the Docker image will be ingested, either from the Mayhem private Docker Registry or from the public Docker Hub registry.
  2. Create Mayhemfile: Configure the specifications for the Mayhem run dictated by the generated Mayhemfile.
  3. Run Mayhem: Confirm your selections and execute the Mayhem run!

2. Reproduce defects generated by crashing test cases.

If we were to download and extract the testme binary as well as the crashing test case, and then execute the commands to reproduce the defect, we should see something similar to the following:
```
~/tutorial/testme/v1# ./testme d1134fb406ea8ded0fe8c19dfdfd63cecea25f7e1cd7292e789d0841adfa9f12
Floating point exception
```

3. Perform regression testing on a fixed version of the testme target.

When Mayhem generates test cases involved with testing a target application, it also saves the test cases for future use. This way, future Mayhem runs can utilize those same previously generated test cases to confirm if the current run's defects have been resolved. This is called regression testing.

4. Execute a Mayhem run for the popular lighttpd web server via the Mayhem UI.

Lighttpd is an open-source web server optimized for speed with considerations for compliance, security, and flexibility. In the past, lighttpd 1.4.15 version had a few vulnerabilities that have since been patched.

Verify that the cmds value is set to /usr/local/sbin/lighttpd -D -f /usr/local/etc/lighttpd.conf and make sure that the Mayhemfile preview looks similar to the following:

image: $MAYHEM_DOCKER_REGISTRY/forallsecure/tutorial/lighttpd:1.4.15
duration: 90
project: forallsecure-tutoriallighttpd
target: '1415'
cmds:
  - cmd: /usr/local/sbin/lighttpd -D -f /usr/local/etc/lighttpd.conf
    network:
      url: tcp://localhost:80
      timeout: 2
      client: false

Code Testing in Your Browser¶

Creating A New Mayhem Run from a Docker Image¶

Reproducing Defects via Generated Test Cases¶

Running Regression Testing and Confirming Fixes¶

⚡ Real World Exercise: Testing the lighttpd Docker Image¶

🔍 Review It! Testing the lighttpd Docker Image¶

✏️ Summary and Recap¶

⚡ Real World Exercise: Testing the `lighttpd` Docker Image¶

🔍 Review It! Testing the `lighttpd` Docker Image¶