beginner

Code Testing From Your Terminal¶

Let's dive deeper! In this lesson, we'll walk you through installing the Mayhem CLI so that you can test your targets from the command line.

Estimated Time: 20 minutes

Objectives

By the end of this lesson, you will be able to:

Download and install the Mayhem CLI.
Authenticate with Mayhem via the mayhem login command.
Execute a Mayhem run on a Docker image via the Mayhem CLI.
Reproduce defects generated by crashing test cases within Docker containers.
Create a Mayhem workflow using scripts.

Quickstart

Run through the lesson:

Download the Mayhem CLI and check to make sure it's installed correctly.
```
mayhem --version
2.5.0rc16+a374eca
```

Authenticate with the Mayhem server using the mayhem login command.

mayhem login <MAYHEM_HOST_URL> <USER_API_KEY>
Logged in successfully at '<MAYHEM_HOST_URL>:443' as '<USER>'.
Syncing default settings: /Users/andrew/.config/mayhem/mayhem.

mayhem login <MAYHEM_HOST_URL> <USER_API_KEY>
Logged in successfully at '<MAYHEM_HOST_URL>:443' as '<USER>'.
Syncing default settings: /Users/andrew/.config/mayhem/mayhem.
Logging into the Docker registry at <MAYHEM_HOST_URL>:5000
Successfully logged in to the remote Docker registry.

Execute a Mayhem run on a Docker target using the mayhem run command with the following Mayhemfile:

image: forallsecure/tutorial:2.5
duration: 90
project: forallsecure-tutorial
target: latest
cmds:
  - cmd: /root/tutorial/testme/v1/testme @@

image: $MAYHEM_DOCKER_REGISTRY/forallsecure/tutorial:2.5
duration: 90
project: forallsecure-tutorial
target: latest
cmds:
  - cmd: /root/tutorial/testme/v1/testme @@

Prerequisites

You will need the following:

A Linux or MacOS operating system.
Docker installed (for authenticating with the Mayhem Docker registry).
A valid internet connection (for testing publicly available Docker Hub images).

Terminal Recording¶

Executing a Mayhem run on a Docker image via the Mayhem CLI¶

Regardless of whether you're using the Mayhem UI or the Mayhem CLI to test target applications, a Mayhem run requires users to provide a configured Mayhemfile for their target. Therefore, when testing target binaries using the Mayhem CLI, users will need to configure the Mayhemfile for their Docker targets before they can execute the mayhem run command. Let's see how this is done!

Warning

Make sure that you have succesfully installed the Mayhem CLI and authenticated your credentials by checking out the Mayhem CLI Installation page.

First, create a new folder tutorial and navigate into it. Then, copy and paste the following configuration into a new Mayhemfile:

image: forallsecure/tutorial:2.5
duration: 90
project: forallsecure-tutorial
target: testme
cmds:
  - cmd: /root/tutorial/testme/v1/testme @@

image: $MAYHEM_DOCKER_REGISTRY/forallsecure/tutorial:2.5
duration: 90
project: forallsecure-tutorial
target: testme
cmds:
  - cmd: /root/tutorial/testme/v1/testme @@

Your resulting folder structure should look like the following:

├── tutorial
    └── Mayhemfile: The configuration file for the Mayhem run

Now that you have your Mayhemfile set, you can now run the mayhem run command. The mayhem run command follows the format:

mayhem run <mayhemfile_directory>

Therefore we can simply run the following to execute a Mayhem run in the tutorial folder where we just created our new Mayhemfile.

mayhem run .

If your run is successful, you should be able to see output similar to the following:

$ mayhem run .
Run started: forallsecure-tutorial/testme/4
Run URL: https://tutorial.forallsecure.com:443/mayhemuser/forallsecure-tutorial/testme/4

And the Mayhem UI should reflect the newly executed Mayhem run:

Fantastic! You just executed another Mayhem run using the Mayhem CLI!

Tip

To view your past run results for a given project and target, you can also use the mayhem show <project>/<target> command. For example, mayhem show forallsecure-tutorial/testme would look something like:

Name                            Status                                                                                 Reports    Crashes    Defects  Date
------------------------------  -----------------------------------------------------------------------------------  ---------  ---------  ---------  ------------------------------
forallsecure-tutorial/testme/3  dynamic_analysis:completed, regression_testing:completed, static_analysis:completed         30          7          1  Fri Jan 28 15:29:19 2022 +0000
forallsecure-tutorial/testme/2  regression_testing:completed                                                                24          0          0  Fri Jan 28 15:21:47 2022 +0000
forallsecure-tutorial/testme/1  dynamic_analysis:completed, regression_testing:completed, static_analysis:completed         24          1          1  Fri Jan 28 15:01:51 2022 +0000

Reproducing Defects for Containerized Target Binaries¶

The recommended packaging method for Mayhem targets is to containerize application binaries within Docker containers that can ultimately be shared via Docker images. This is so when users want to validate defects found for a fuzzed target binary, users can simply download the Docker image for the run and re-execute test cases for the target binary located within the container as proof.

Assuming a user has already executed a Mayhem run, a user must execute the following steps to reproduce defects for a containerized target binary:

Assuming a user has already executed a Mayhem run and is logged into the private Mayhem Docker Registry via the docker CLI on their local machine, a user must execute the following steps to reproduce defects for a containerized target binary:

Execute a docker pull of the Docker image for the run.
Execute a docker run in interactive mode for the newly downloaded Docker image.
Copy the test case into the running Docker container.
Execute the fuzzing steps for the given test case to reproduce the defect.

For example, for our recent Mayhem run, we can execute a docker pull for the value of the image found in the Mayhemfile configuration.

image: forallsecure/tutorial:2.5
duration: 90
project: forallsecure-tutorial
target: testme
cmds:
  - cmd: /root/tutorial/testme/v1/testme @@

image: $MAYHEM_DOCKER_REGISTRY/forallsecure/tutorial:2.5
duration: 90
project: forallsecure-tutorial
target: testme
cmds:
  - cmd: /root/tutorial/testme/v1/testme @@

$ docker pull forallsecure/tutorial:2.5
latest: Pulling from forallsecure/tutorial
0f7bc87f0b77: Already exists 
30cc39211964: Pull complete 
b2cea678fd2b: Pull complete 
980e4fca2c5e: Pull complete 
c95be0a0e427: Pull complete 
b2f764506b1d: Pull complete 
ab5ce6632a38: Pull complete 
c8f755707129: Pull complete 
362f2e212a33: Pull complete 
0c6af55db2b2: Pull complete 
4dac91f0ff83: Pull complete 
d223eb603e3e: Pull complete 
Digest: sha256:dfc613cb21f206027970363fec47a3c82280f0c6a3f46f29a089af5dae92dfd4
Status: Downloaded newer image for forallsecure/tutorial:2.5
docker.io/forallsecure/tutorial:2.5

$ docker pull tutorial.forallsecure.com:5000/forallsecure/tutorial:2.5
latest: Pulling from forallsecure/tutorial
0f7bc87f0b77: Already exists 
30cc39211964: Pull complete 
b2cea678fd2b: Pull complete 
980e4fca2c5e: Pull complete 
c95be0a0e427: Pull complete 
b2f764506b1d: Pull complete 
ab5ce6632a38: Pull complete 
c8f755707129: Pull complete 
362f2e212a33: Pull complete 
0c6af55db2b2: Pull complete 
4dac91f0ff83: Pull complete 
d223eb603e3e: Pull complete 
Digest: sha256:dfc613cb21f206027970363fec47a3c82280f0c6a3f46f29a089af5dae92dfd4
Status: Downloaded newer image for tutorial.forallsecure.com:5000/forallsecure/tutorial:2.5
tutorial.forallsecure.com:5000/forallsecure/tutorial:2.5

Once the Docker image has been downloaded, users can use the docker run -it command to run the forallsecure/tutorial Docker image interactively and execute commands within the container itself.

$ docker run -it forallsecure/tutorial:2.5
~/tutorial#

$ docker run -it tutorial.forallsecure.com:5000/forallsecure/tutorial:2.5
~/tutorial#

Crashing test cases can then be downloaded and copied over into the running Docker container using the docker ps command in conjunction with the docker cp command:

$ docker ps
CONTAINER ID   IMAGE                   COMMAND       CREATED         STATUS         PORTS     NAMES
abc183c111ff   forallsecure/tutorial   "/bin/bash"   6 seconds ago   Up 4 seconds             mystifying_hofstadter
$ docker cp aca42ffba79fea39a6b4876a7b6e0be67128f0aee2024b194e47a23cc3d08110 abc183c111ff:/root/tutorial/testme/v1

Lastly, defects can be reproduced for the containerized binary target by executing the given test case for the binary within the container itself:

$ ./root/tutorial/testme/v1/testme aca42ffba79fea39a6b4876a7b6e0be67128f0aee2024b194e47a23cc3d08110
Aborted

⚡ Real World Exercise: Automating Workflows with Mayhem CLI¶

One of the main benefits to using the Mayhem CLI is that it allows you to create scripts of underlying Mayhem commands. Combined with the other commands Mayhem CLI has to offer, users can create valuable workflows for testing their targets. For example, users can run the following mayhem commands in a script:

mayhem run <path_to_Mayhemfile> will execute the Mayhem run for the given Mayhemfile.
mayhem wait <id> will block until a run is complete. This is useful if you write a script that needs to wait until Mayhem completes an analysis.
mayhem sync <path_to_Mayhemfile> will synchronize the latest Mayhem state—including the test suite—to your local machine.

Hint

Use the syntax id=$(mayhem run <path_to_Mayhemfile>) to set the corresponding $id variable for the mayhem wait command.

Instructions:

Use your previously created Mayhemfile to create a set of scripts/workflows that execute a Mayhem run on the testme application contained in the forallsecure/tutorial Docker image.

You can access the container of the forallsecure/tutorial Docker image by running the following:

docker pull forallsecure/tutorial:2.5
docker run -ti --privileged --rm forallsecure/tutorial:2.5

Then within the forallsecure/tutorial container, create and run the following scripts:
1. A testing script that executes a Mayhem run, waits for the Mayhem run to complete, and then syncs the generated test cases from the Mayhem run to your local machine.
2. A regression testing script that executes a regression test, waits for the regression test to complete, and then syncs the test cases from the regression test to your local machine.
Note

Because we're working within the forallsecure/tutorial Docker container for this exercise, the synced test case files will be downloaded within the Docker container.

🔍 Review It! Automating Workflows with Mayhem CLI¶

Solution

For the first testing script, let's assume we have the following Mayhemfile.

image: $MAYHEM_DOCKER_REGISTRY/forallsecure/tutorial:2.5
duration: 90
project: forallsecure-tutorial
target: latest
cmds:
- cmd: /root/tutorial/testme/v1/testme @@

In the same directory as the Mayhemfile, you needed to create a simple script run.sh to execute a Mayhem run, wait for the run to finish, and download the generated test cases for the run to your local machine like so:

#!/usr/bin/env sh

# Run mayhem. The run ID is saved to $id
id=$(mayhem run .)

# Wait for the run to finish
mayhem wait $id

# Sync the test suite to the "testsuite" directory.
mayhem sync .

Then simply run the following to initiate the script:

$ sh run.sh
/tmp/tmpi2_eubh4/Mayhemfile 100% |#################| Time:  0:00:00   7.6 KiB/s
Run started: forallsecure-tutorial/testme/1
Run URL: https://tutorial.forallsecure.com:443/mayhemuser/forallsecure-tutorial/testme/1
Downloaded: Mayhemfile.
Downloading testsuite.tar: 138.0 KiB |#                 | Elapsed Time: 0:00:00 150.7 KiB/s
Extracting test 128 of 128 |#######################| Time:  0:00:00
Target synced at: '.'.

By this point, you should have files similar to the following:

Mayhemfile: The mayhem run configuration file.
run.sh: The testing script.
testsuite: The synced test cases from the Mayhem run.

And if you look within the testsuite directory, you can see the actual test cases that were used in testing the testme target. This is valuable as we now have access to all the test cases for the run including the crashing test cases resulting in the improper input validation defect.

~/testsuite# ls -l | head -n 10
total 448
-rw-r--r-- 1 root root 14 Jan  1  1970 03be4170751ccbfeb3a1e9ef2b7b4f32bb7dd259a4cc0744d0e4ab1182c30821
-rw-r--r-- 1 root root 14 Jan  1  1970 04520a13840f8a3d0d6393e4a3a27c28531deddd0fbb20be565a21aa74837f2a
-rw-r--r-- 1 root root  8 Jan  1  1970 054fb66d37ab1222584a8f5e40d1d1dd9b1a52bf278ed5b5e34305a502dddc93
-rw-r--r-- 1 root root 12 Jan  1  1970 08bab601cb90a24a8fface4991fb2bdd78073c249c881e321f3b4b1be37ac9b8
-rw-r--r-- 1 root root 16 Jan  1  1970 0909ed9a9705deb19525e5af04eb1f0fec67a00f17ef048f3077304844b265b2
-rw-r--r-- 1 root root  5 Jan  1  1970 0a679f190e7e16930e9ad1fd9e113015e7300e6ef141612f2c05175d0090b972
-rw-r--r-- 1 root root 12 Jan  1  1970 0af2a88ccebf716e022b634526cd09a1ef23ea17c2dbc36ebc486f586dcad9dc
-rw-r--r-- 1 root root  6 Jan  1  1970 0bffd96d0cbdf2a7881ae8514c720a2c42aaea3d3c9a4f4f46c149ed75fd0db2
-rw-r--r-- 1 root root 36 Jan  1  1970 0d98d3438376a8a5d4b678ffb9c3439a28c923ff661c3f5bb656a27405f7238c

Now, for the second regression testing script the process is much of the same. Let's assume we have the following Mayhemfile pointing to a fixed version of the testme target (improper input validation defect is fixed).

image: $MAYHEM_DOCKER_REGISTRY/forallsecure/tutorial:2.5
duration: 90
project: forallsecure-tutorial
tasks:
- name: regression_testing
target: latest
cmds:
- cmd: /root/tutorial/testme/v2/testme @@

And if we have the following regression testing script, we can simply run it and create an automated regression testing workflow.

#!/usr/bin/env sh

# Run Mayhem regression testing
# The run ID is saved to $id
id=$(mayhem run . --regression)

# Wait for the run to finish
mayhem wait $id --regression

# Sync the test suite to the "testsuite" directory.
mayhem sync .

$ sh regression_testing.sh
/tmp/tmp6t6ue4e9/Mayhemfile 100% |#################| Time:  0:00:00   7.6 KiB/s
Run started: forallsecure-tutorial/testme/2
Run URL: https://tutorial.forallsecure.com:443/andrewyang/forallsecure-tutorial/testme/2
Downloaded: Mayhemfile.
Downloading testsuite.tar: 122.0 KiB |#               | Elapsed Time: 0:00:00 223.6 KiB/s
Extracting test 112 of 112 |#####################| Time:  0:00:00
Target synced at: '.'.

And that's it! Congratulations on creating your first set of Mayhem workflows!

✏️ Summary and Recap¶

In this lesson, you learned how to install the Mayhem CLI and fuzz targets from the command line!

Upon installing the Mayhem CLI, users will have to authenticate with the Mayhem server using the mayhem login command before they can execute Mayhem runs using the mayhem run command for a given Mayhemfile. Additional Mayhem CLI commands such as mayhem wait and mayhem sync can also give users more control over their Mayhem runs.

I learned how to...

1. Download and install the Mayhem CLI.

Users can navigate to the Mayhem CLI Installation page located in the Mayhem UI and download the correct Mayhem CLI package for their OS (Linux or macOS).

2. Authenticate with Mayhem via the mayhem login command.

Users will need to authenticate their credentials with the Mayhem server using the mayhem login command before they can execute Mayhem runs.
Users can navigate to the Mayhem CLI Installation page located in the Mayhem UI to copy their user API token for the mayhem login command.

3. Execute a Mayhem run on a Docker image via the Mayhem CLI.

Users can use the mayhem run command for a given Mayhemfile. By setting the image parameter users can set the target for the run as a private Docker image or public Docker Hub image.

4. Reproduce defects generated by crashing test cases within Docker containers.

Assuming a user has already executed a Mayhem run and is logged into the private Mayhem Docker Registry via the docker CLI on their local machine, a user must execute the following steps to reproduce defects for a containerized target binary:
1. Execute a docker pull of the Docker image for the run.
2. Execute a docker run in interactive mode for the newly downloaded Docker image.
3. Copy the test case into the running Docker container.
4. Execute the fuzzing steps for the given test case to reproduce the defect.

5. Create a Mayhem workflow using scripts.

One of the main benefits to using the Mayhem CLI is that it allows you to create scripts of underlying Mayhem commands. Combined with the other commands Mayhem CLI has to offer, users can create valuable workflows for testing their targets, such as:

#!/usr/bin/env sh

# Run mayhem. The run ID is saved to $id
id=$(mayhem run .)

# Wait for the run to finish
mayhem wait $id

# Sync the test suite to the "testsuite" directory.
mayhem sync .

#!/usr/bin/env sh

# Run Mayhem regression testing
# The run ID is saved to $id
id=$(mayhem run . --regression)

# Wait for the run to finish
mayhem wait $id --regression

# Sync the test suite to the "testsuite" directory.
mayhem sync .