Glossary¶

Mayhem (and our documentation for it) may use terms you're unfamiliar with (and/or ones you are familiar with, but in unfamiliar ways). Our definitions for these terms are outlined here.

Application¶

An application is user-defined; in some contexts, it may refer to a collection of software used to accomplish one purpose, like a "web app". In Mayhem, an application is a single program, such as "httpd" or "/bin/ls". Mayhem uses the terms application, binary, and executable interchangeably.

Behavior Testing¶

Behavior testing runs an application with different inputs and observes how program flow, code coverage, and application termination vary as the input changes. Abnormal observations are triaged to determine if a behavior is a vulnerability.

Control Flow Graph (CFG)¶

A control flow graph (CFG) is a directed graph where each node is a statement, and an edge exists between two nodes if there is a possible transfer of control.

Dependency¶

An application dependency includes any library or configuration files needed to run the application. Examples include libraries listed by ldd /usr/bin/program or the nginx.conf file for NGINX.

Dynamic Analysis¶

Dynamic analysis monitors the target program as it executes on a real input. For example, running valgrind on your program is a type of dynamic analysis that looks for memory errors.

Executable Program¶

An executable, also referred to as an application or binary in Mayhem, is a single machine-executable program on disk. Mayhem-compatible executables are written in C/C++, Go, Rust, or similar languages and compiled into an executable file. These executables may rely on external dependencies such as libraries or configuration files. Mayhem does not currently analyze programs written in interpreted languages like Python, Javascript, or bash.

Edges Covered¶

Mayhem uses the edge coverage metric, which measures how many control flow graph edges have been exercised. More formally, consider a control flow graph where each node is a program statement, and two nodes are connected only if there is potential control flow between them. Edge coverage measures the number of edges exercised by the test suite.

Fuzzing¶

Fuzzing is mutating a chosen input A to create a new input B and then running the application with input B. The term "fuzzing" is widely credited to have been coined by Bart Miller.

Mayhem is a portfolio fuzzer. At a high level, the family of fuzzing techniques can be broken down into:

Black-box techniques, which mutate the input using no knowledge of the program itself, typically at random. The Linux zzuff program is an example of a black-box fuzzer.
White-box fuzzers, which use information about the program to derive input B from A. Mayhem includes a white-box fuzzer based upon symbolic execution. Symbolic execution is a program analysis technique that uses formal computer science methods to derive the new input B by modeling how the program executes on A.
Gray-box fuzzers, which use instrumentation to derive each new input. The afl fuzzer is a gray-box fuzzer. Mayhem includes a gray-box fuzzer.
Portfolio fuzzers, which intelligently combine fuzzing techniques above to maximize coverage.

Harnessing¶

Harnessing consists of a user adding in new code to call specific routines they wish to target within the executable under test.

Example:

OpenSSL defines several harnesses in the fuzz directory.

Mayhemfile¶

A Mayhem configuration file.

mCoders¶

Any user of Mayhem, independent of their background—application security, dev(sec)ops, vulnerability research, or quality assurance—interested in improving the quality of their software!

Negative Testing¶

Testing for incorrect or undesired behavior as opposed to positive testing. Negative testing ensures that the target does not behave in an unexpected manner when incorrectly formed inputs are provided. For example, ensuring that your number parser can handle non-numeric string inputs gracefully would be a type of negative test.

Number of Tests Run¶

Total number of iterations with different inputs that ran against the program.

Note

Not every iteration results in a new test case and that's expected since we only keep test cases that identify new edge coverage. For instance, 60M test runs means that we attempted at least 60 million mutations of inputs and then sent those against the target.

Organizations¶

Organizations are the highest level structure for managing users (and consequently run access) within Mayhem. Organizations can contain members, projects, and teams.

Organization Owner¶

A user member who has admin privileges over an organization. This allows the user to manage members and teams inside the organization.

Organization Member¶

A user account listed as a member of the organization. A user must be a member of an organization to start a run within the respective organization; each user's API token is checked against the organization member list prior to initiating the run. Users are able to be members of multiple organizations.

Package¶

A package is a complete chroot environment for running the application plus a Mayhemfile configuration file specifying Mayhem parameters. A typical package will consist of:

A Mayhemfile for configuring the Mayhem run for a fuzz target.
A tests folder containing the compilation of test cases for the fuzz target.
A chroot environment containing any application-specific libraries and application-specific configuration files needed to run the application or compiled binary.

Note

Currently packaging is a catch-all concept that includes Docker images or chroot environments for users that do not use Docker. The purpose of the Mayhem package is to specify a full runtime environment for distributed fuzzing.

Positive Testing¶

Positive testing is a type of testing where valid data is sent as the input to a program to check for desired outcomes.

Portfolio Analysis¶

Running several different types of analyses, where the type signature of each analysis is the same. For example, symbolic execution and fuzzing both take a program and a seed input, and output a new test case. Symbolic execution and fuzzing can be run together as a portfolio analysis.

Project¶

A project is a collection of targets to analyze. The user can decide which targets to include in a project. One common idiom is to put all your targets for a single larger application into one project.

Project Contributor¶

A user or team account that has been added to a project with permissions of either read, write, or admin access. Only a contributor with admin privilege is able to delete a project. Any user account is able to create a project.

Run¶

A Mayhem run (also called a job) where Mayhem fuzz tests an application within its distributed cluster.

Target¶

A target is a compiled application along with the command line to run it.

For example, compiling OpenSSL produces the openssl executable. Here are three different targets for the one executable, where the symbol @@ represents the file to fuzz.

openssl cms -cmsout -inform DER -in @@
openssl sha @@
openssl seed -in @@ -out /tmp/file2 -k foobar

Team¶

A team is a way to bundle multiple user accounts into one unit to assign permissions for all users inside the team.

Test Case¶

A test case refers to a specific input or data that is generated and fed into a program or system to evaluate its behavior and resilience against unexpected or malicious inputs. Fuzzing test cases typically contain malformed or unexpected data in order to trigger potential vulnerabilities or uncover software bugs.

Test Coverage¶

Test coverage, also referred to as code coverage, measures how much of the underlying code is exercised for a program that is fuzzed with varying input test cases.

For example, imagine a program that reads in an input string. If a test case "welcome" is fed into the program, the program prints "hello!", otherwise the program does something else.

If we think of this program as a control flow graph, we can see that there are multiple paths that can be exercised based on the input test case.

graph TD
    1[1 - input] -- if input == 'welcome' --> 2[2 - print 'hello!'];
    1 -- else --> 3;
    2-->3[3 - do something];

The coverage for this program can be broken down into the following categories:

Node Coverage: Represents the aggregate of test paths that exercise every node in the graph. Node coverage only stipulates that every node is visited, even if paths between the nodes are not. For example, the test path [1, 2, 3] satisfies the requirement {1, 2, 3} because it hits every node.
- Test Requirement = {1, 2, 3}
- Test Paths:
  - [1, 2, 3]
Edge Coverage: Represents the aggregate of test paths that exercise every unique line, or edge, in the graph. Edge coverage requires that every path between two nodes is executed, so the above test path for node coverage would not be sufficient. We would need an additional test path [1, 3] to cover the edge directly between nodes 1 and 3.
- Test Requirement = {(1, 2), (1, 3), (2, 3)}
- Test Paths:
  - [1, 2, 3]
  - [1, 3]
Complete Path Coverage: Represents the test paths in which all linear paths in the program are executed at least once.
- Total Test Paths:
  - [1, 2, 3]
  - [1, 3]

Info

Complete path coverage can be different from edge coverage when considering loops in a program's control flow graph. Check out more about coverage analysis here!

Therefore, when fuzzing a target program, Mayhem generates random input test cases that exercise various test paths along the program control flow. Mayhem then tracks this data, such as edges covered, and produces coverage files that can be used for further coverage analysis.

Test Suite¶

A test suite refers to a collection or set of test cases that are designed to be executed together in order to evaluate the robustness and security of a program or system. A fuzzing test suite consists of multiple test cases, each containing different variations of inputs and data, with the goal of maximizing code coverage and detecting potential vulnerabilities or software flaws.

Test Suite Size¶

Number of test cases that were discovered by Mayhem, each one of which is contributing unique code coverage for the underlying target.

Note

If we dropped any of these test cases we would not exercise some of the already discovered functionality.