Executing and Downloading Coverage via Mayhem CLI

Coverage files can not only be downloaded via the target run page UI, but also directly from the Mayhem CLI as well.

Assuming a mayhem run was executed on a target using the --coverage parameter (or the --all parameter to execute all tasks, which includes coverage analysis), developers can use the mayhem sync or mayhem download commands upon run completion to download the associated target coverage files.

Performing a Mayhem Run with Coverage Analysis

Let's walk through this process in more detail. For this example we will be utilizing the 2.5 Tutorial Docker image.

docker pull forallsecure/tutorial:2.5
docker run -ti --privileged --rm forallsecure/tutorial:2.5

First, navigate to the testme target folder.

Then, we will need to package up the testme dependencies before executing the mayhem run command.

Tip

You may need to perform a mayhem login to connect to your Mayhem server.

# Package testme-v1 library dependencies.
mayhem package ./testme -o /tmp/testme-pkg

After the testme target has been packaged, we can execute the mayhem run command with the --coverage or --all parameters. In this case, we used the --all parameter to execute all mayhem run tasks.

mayhem run --all /tmp/testme-pkg

Your workflow up to this point should look like the following:

Sancov/Gcov Support

Mayhem supports generating coverage with sancov/gcov compatible binaries by using those tools when executing your target.\ For gcov specifically, you will need to provide the .gcno files generated by the compiler in the same directory as the target binary.\ For more information on how to layout and compile sancov/gcov binaries, check the sancov and gcov directories in the tutorial image.\ You can execute the same flow as for testme target above in order to perform a mayhem run for those targets.

Downloading Coverage Files with mayhem sync

Next, if we navigate to the /tmp/testme-pkg where our testme target has been packaged, we will notice that there are only three items within the directory:

1. Mayhemfile: The config file for the mayhem run command.
2. tests: The associated test cases generated.
3. root: The location of the packaged application.

However, when we run the mayhem sync command, the Mayhem CLI will download the associated coverage files for our testme target.

Important

You will need to have at least one completed run to download the coverage files.

mayhem sync /tmp/testme-pkg

A new folder called testme_coverage is now downloaded, and within it are three files:

1. block_coverage.drcov: Can be used with applications such as Binary Ninja (bncov), IDA Pro (Lighthouse), or Ghidra (Dragon Dance) to visualize the basic blocks (at the machine code level) covered by the test suite.
2. func_coverage.json: A parseable JSON file containing information related to function coverage.
3. line_coverage.lcov A lcov file describing which source code lines were covered by the test suite.

In case your target is sancov compatible, further files are included in the coverage folder: 1. coverage_points.sancov: Contains the coverage points as reported by the target binary in sancov format 2. symbolic_coverage.symcov: A parsable JSON file containing coverage points as well as location information. It is the same as if you run sancov -symbolize <sancov_files> <binary>. Can be used with tools such as coverage report server to navigate coverage for your target in the browser.

The workflow for downloading coverage files with mayhem sync should look like the following:

Downloading Coverage Files with mayhem download

The mayhem download command is very similar to the mayhem sync command, and for all intents and purposes, the two commands can be used interchangeably to download available coverage files.

However, the mayhem download command differs in that it also allows for an optional parameter to specify the output folder to where the target package and its associated coverage files will be stored. Let's see how this works.

mayhem download testme/testme -o testme_mayhem_download

The mayhem download command requires an argument in the form of <project>/<target> to pinpoint the target run, and uses the -o parameter to specify the location of the downloaded coverage files.

The workflow for downloading coverage files with mayhem download should look like the following:

Summary

The Mayhem CLI provides the options for developers to download coverage files using either the mayhem sync or mayhem download commands.

This can be particularly useful for developers who want to automate such operations via scripts to immediately download coverage files upon a mayhem run completion. For example, a complete workflow (such as a run.sh file for testme) could look like the following:

#!/bin/sh

# Package testme-v1 library dependencies.
mayhem package ./testme -o /tmp/testme-pkg

# Run mayhem. The run ID is saved to $id
id=$(mayhem run --all /tmp/testme-pkg)

# Wait for the run to finish
mayhem wait $id

# Sync the test suite to the "testsuite" directory.
mayhem sync /tmp/testme-pkg