beginner

Testing a Docker Target with Mayhem

Have a containerized application packaged in a Docker image? In this lesson, we'll walk you through how to compile and upload your own Docker image so that the containerized application within can be tested using Mayhem.

---

Estimated Time: 30 minutes

Objectives

By the end of this lesson, you will be able to:

1. Build your own local Docker image.
2. Push a local Docker image to the Docker Hub registry.
3. Execute a Mayhem run on your uploaded Docker image via the Mayhem UI.
4. Execute a Mayhem run on your uploaded Docker image via the Mayhem CLI.

1. Build your own local Docker image.
2. Push a local Docker image to the Mayhem Docker registry.
3. Execute a Mayhem run on your uploaded Docker image via the Mayhem UI.
4. Execute a Mayhem run on your uploaded Docker image via the Mayhem CLI.

Quickstart

Run through the lesson:

See prerequisites before beginning.

1. Users should be aware of the following steps for testing Docker targets in Mayhem:

   1. A user builds a local Docker image with the docker build command.
   2. The user uploads their Docker image to the public Docker Hub registry (requires a Docker Hub account).
   3. The user executes a Mayhem run using either the Mayhem UI or Mayhem CLI, in which case the Mayhem API pulls the designated Docker image from the Docker Hub registry required for the Mayhem run.
   4. Mayhem executes the run and results are made available.
   1. A user builds a local Docker image with the docker build command.
   2. The user uploads their Docker image to either the public Docker Hub or the private Mayhem Docker Registry (requires authentication using mayhem login).
   3. The user executes a Mayhem run using either the Mayhem UI or Mayhem CLI, in which case the Mayhem API pulls the designated Docker image from its corresponding source required for the Mayhem run.
   4. Mayhem executes the run and results are made available.

2. Download testme-docker and extract the file to build and tag the testme-docker image using the docker build command and the following Dockerfile:

   FROM debian:buster-slim as builder
   RUN apt-get update && \
       apt-get install -y gcc make libc6-dbg && \
       rm -rf /var/lib/apt/lists/*
   WORKDIR /mayhem
   COPY src testme
   RUN cd testme && make
   

3. Re-tag and push the testme-docker image to the Mayhem Docker Registry using the docker tag and docker push commands:

   docker tag testme-docker <DOCKERHUB_USERNAME>/testme-docker
   docker push <DOCKERHUB_USERNAME>/forallsecure/testme-docker
   

   docker tag testme-docker $MAYHEM_DOCKER_REGISTRY/forallsecure/testme-docker
   docker push $MAYHEM_DOCKER_REGISTRY/forallsecure/testme-docker
   

4. Execute a Mayhem run on the testme-docker image using either the Mayhem UI or Mayhem CLI with the following Mayhemfile:

   image: $MAYHEM_DOCKER_REGISTRY/forallsecure/testme-docker:latest
   duration: 90
   project: forallsecure-testme-docker
   target: testme
   cmds:
     - cmd: /mayhem/testme/v1/testme @@
   

   image: $MAYHEM_DOCKER_REGISTRY/forallsecure/testme-docker:latest
   duration: 90
   project: forallsecure-testme-docker
   target: testme
   cmds:
     - cmd: /mayhem/testme/v1/testme @@
   

Prerequisites

You will need the following:

• Docker installed.
• The Mayhem CLI installed.
• Linux or MacOS operating system (if testing the Docker image via Mayhem CLI).

• Docker installed.
• The Mayhem CLI installed and authenticated with the private Mayhem Docker registry.
• Linux or MacOS operating system (if testing the Docker image via Mayhem CLI).

Terminal Recording

Workflow for Mayhem Docker Targets

Take a look at the following diagram to get a sense of the steps necessary for testing Docker targets in Mayhem (start to finish).

In particular, users should be aware of the following steps:

1. A user builds a local Docker image with the docker build command.
2. The user uploads their Docker image to the public Docker Hub.
3. The user executes a Mayhem run using either the Mayhem UI or Mayhem CLI, in which case the Mayhem API pulls the designated Docker image from Docker Hub for the Mayhem run.
4. Mayhem executes the run and results are made available.

1. A user builds a local Docker image with the docker build command.
2. The user uploads their Docker image to either the public Docker Hub or the private Mayhem Docker Registry (requires authentication using mayhem login).
3. The user executes a Mayhem run using either the Mayhem UI or Mayhem CLI, in which case the Mayhem API pulls the designated Docker image from its corresponding source required for the Mayhem run.
4. Mayhem executes the run and results are made available.

Let's start with the first step of building a local Docker image!

Building and Pushing Your Own Docker Image

With Docker, it's easy to build your own image containing your target binary. To do so, users will simply need to specify a Dockerfile and use the docker build command.

Download and extract the following to get started: testme-docker

You should see the corresponding extracted files:

├── testme-docker
    └── Dockerfile: Defines a set of Docker build commands to be executed.
    └── Makefile: Defines a set of build tasks to be executed.
    └── src: A folder containing the source files for the testme application.

A Dockerfile is a text-based script of instructions used to create a Docker image from an underlying Docker container. Within the Dockerfile, users are given several options to create and modify their Docker containers as they see fit. Once complete, users can use the docker build command to point to the Dockerfile and execute the underlying commands to build and tag their Docker container images.

Navigate to the testme-docker directory and open the included Dockerfile.

FROM debian:buster-slim as builder
RUN apt-get update && \
    apt-get install -y gcc make libc6-dbg && \
    rm -rf /var/lib/apt/lists/*
WORKDIR /mayhem
COPY src testme
RUN cd testme && make

Here we can see the following commands used in the Dockerfile:

• FROM: Initializes a new build stage and sets the Base Image for subsequent instructions. A valid Dockerfile must start with a FROM instruction.
• RUN: Executes any commands in a new layer on top of the current image and commits the results. The resulting committed image will be used for the next step in the Dockerfile.
• WORKDIR: Sets the working directory for any RUN, CMD, ENTRYPOINT, COPY and ADD instructions that follow it in the Dockerfile.
• COPY: Copies new files or directories from and adds them to the filesystem of the container at the path

Executing the following will use the Dockerfile to build a resulting Docker image:

docker build -f Dockerfile -t testme-docker .

• docker build is used to build a Docker image from a Dockerfile.
• -f parameter specifies the name of the Dockerfile (default is PATH/Dockerfile)
• -t parameter tags the resulting Docker image with a human readable name.
• . specifies the PATH of the “context” for the build and all files in the local directory are packaged (via tar) and sent to the Docker daemon.

Info

Check out the Dockerfile Reference and docker build examples for more information on setting up your Dockerfile and building your Docker images, respectively.

In the case of our testme-docker example, the Dockerfile sets the debian:buster-slim base image and downloads the necessary build tools such as gcc and make using the apt-get command. It then sets the working directory as /mayhem, copies the contents of the local src folder into a testme folder, and navigates to the testme folder to execute the make command for the Makefile within. The resulting Docker container is then built and tagged as a Docker image known as testme-docker.

If you're docker build was successful, you should see output similar to the following:

You can also confirm that the testme-docker image was successfully built by running docker images and piping the output to grep testme-docker to search for and output any matching cases of testme-docker:

$ docker images | grep testme-docker
testme-docker      latest        118f9b5b7874        5 minutes ago       201MB

Congrats! You just created your first local Docker image from scratch!

Now that you've successfully built your own local Docker image, it's time to upload it to Docker Hub so that Mayhem can ingest and fuzz the containerized testme application.

Now that you've successfully built your own local Docker image, it's time to upload it to Mayhem so that it can ingest and fuzz the containerized testme application. To do this we can push the local Docker image to Mayhem's private Docker registry, which is included on all Mayhem deployments.

Before we can do this, however, we'll need to use the mayhem login command from the Mayhem CLI which performs two functions:

1. Authenticates your user credentials with the Mayhem server.
2. Logs into the private Mayhem Docker registry.

$ mayhem login <MAYHEM_HOST_URL> <USER_API_KEY>
Logged in successfully at '<MAYHEM_HOST_URL>:443' as '<USER>'.
Syncing default settings: /Users/andrew/.config/mayhem/mayhem.
Logging into the Docker registry at <MAYHEM_HOST_URL>:5000
Successfully logged in to the remote Docker registry.

Therefore, make sure you are logged into your Mayhem server with mayhem login. Upon successfully logging into the Mayhem Docker registry, users can execute the docker tag and docker push commands in succession to re-tag their local Docker images intended for the Mayhem Docker registry and upload the corresponding Docker image to the registry.

The docker tag and docker push commands for uploading Docker images to the public Docker Hub registry follows the form:

The docker tag and docker push commands for uploading Docker images to a private Docker registry follows the form:

docker tag <DOCKER_IMAGE> <DOCKERHUB_USERNAME>/<DOCKER_IMAGE>
docker push <DOCKERHUB_USERNAME>/<DOCKER_IMAGE>

docker tag <DOCKER_IMAGE> <REGISTRY_HOST>/<DOCKER_REPO>/<DOCKER_IMAGE>
docker push <REGISTRY_HOST>/<DOCKER_REPO>/<DOCKER_IMAGE>

Warning

For this example, our Mayhem deployment is located at tutorial.forallsecure.com with the Mayhem Docker registry residing at port 5000; however, the location of your Mayhem deployment and private Docker registry may differ. Use the mayhem login or mayhem docker-registry commands to log into your Mayhem server and confirm the location of the private Docker registry, respectively, in order to properly push your Docker images.

Therefore, if we run the following, we will re-tag our local testme-docker image and push the newly tagged testme-docker image to the public Docker Hub registry:

Therefore, if we run the following, we will re-tag our local testme-docker image and push the newly tagged forallsecure/testme-docker image to our private Mayhem Docker registry located at tutorial.forallsecure.com:5000:

Important

For the following examples, I've used my andrew5194 Docker Hub repo; however, you will need to tag your local testme-docker image according to your specific Docker Hub account/repository name.

docker tag testme-docker docker.io/<DOCKERHUB_USERNAME>/testme-docker
docker push docker.io/<DOCKERHUB_USERNAME>/testme-docker

docker tag testme-docker tutorial.forallsecure.com:5000/forallsecure/testme-docker
docker push tutorial.forallsecure.com:5000/forallsecure/testme-docker

We can verify that our local Docker image was successfully uploaded to the Docker Hub registry by navigating navigating to Docker Hub and searching for your <DOCKERHUB_USERNAME>/testme-docker image.

We can verify that our local Docker image was successfully uploaded to the private Mayhem Docker registry by navigating to the Mayhem Docker Registry page of the Mayhem UI and searching for the forallsecure/testme-docker image.

Note

Navigating to the Info icon in the top-right corner of the Mayhem Docker Registry page will also confirm the location of your private Docker registry as well as bookmark some of the docker tag and docker push commands shown here.

Testing Your Docker Image via Mayhem UI

Now that we have our Docker image uploaded to the public Docker Hub registry, we can simply create a new Mayhem run from the Mayhem UI as we did before in previous lessons.

Now that we have our Docker image uploaded to the Mayhem Docker Registry, we can simply create a new Mayhem run from the Mayhem UI as we did before in previous lessons.

Select your source as the Docker Hub registry and search for your <DOCKERHUB_USERNAME>/testme-docker image that you just uploaded. Start a new Mayhem run and preview the associated Mayhemfile to ensure it looks similar to the following:

Select your source as the Mayhem Docker Registry and search for the forallsecure/testme-docker image that you just uploaded. Start a new Mayhem run and preview the associated Mayhemfile to ensure it looks similar to the following:

image: index.docker.io/<DOCKERHUB_USERNAME>/testme-docker:latest
duration: 90
project: forallsecure-testme-docker
target: testme
cmds:
  - cmd: /mayhem/testme/v1/testme @@

image: $MAYHEM_DOCKER_REGISTRY/forallsecure/testme-docker:latest
duration: 90
project: forallsecure-testme-docker
target: testme
cmds:
  - cmd: /mayhem/testme/v1/testme @@

Lastly, confirm your selections and click the Start Run button at the end of the Create New Run flow to execute your Mayhem run on your local Docker image!

Nice job! You just built a local Docker image from scratch, uploaded the Docker image to the public Docker Hub Registry, and successfully fuzzed the containerized target to detect the improper input validation defect!

Nice job! You just built a local Docker image from scratch, uploaded the Docker image to the Mayhem Docker Registry, and successfully fuzzed the containerized target to detect the improper input validation defect!

Testing Your Docker Image via Mayhem CLI

If you'd rather use the Mayhem CLI, you can also create a Mayhemfile and adjust the image key to point to your <DOCKERHUB_USERNAME>/testme-docker image residing in the Docker Hub registry like so:

image: <DOCKERHUB_USERNAME>/testme-docker:latest
duration: 90
project: forallsecure-testme-docker
target: testme
cmds:
  - cmd: /mayhem/testme/v1/testme @@

Info

The $MAYHEM_DOCKER_REGISTRY environment variable is set when you successfully authenticate your credentials with a Mayhem server and log into its private Docker registry using the mayhem login command.

Then, just run the mayhem run command in the same directory as the newly created Mayhemfile.

$ mayhem run .
Run started: forallsecure-testme-docker/testme/2
Run URL: https://tutorial.forallsecure.com:443/mayhemuser/forallsecure-testme-docker/testme/2

⚡ Real World Exercise: Building and Testing testme-npd

A segmentation fault is a specific kind of memory error caused by trying to read or write to an illegal memory location. One example is a null pointer dereference in which a pointer with a value of NULL is used as though it were pointed to a valid memory area.

int *p = NULL;
int y = *p;

Let's see if we can add such a null pointer dereference to our existing testme application, rebuild, and use Mayhem to fuzz the modified testme-npd application to detect our newly added bug!

Instructions:

1. Download and extract the following: testme-docker.zip

2. Append the testme.c file by adding an if-else statement that invokes a null pointer dereference bug when the test case containing null is input to the application.

3. Rebuild the Dockerfile using the docker build command and tag the resulting Docker image as testme-npd.

4. Push the testme-npd Docker image to the internal Mayhem Docker registry using the docker tag and docker push commands.

5. Fuzz the testme-npd Docker image using either the Mayhem UI or Mayhem CLI and make sure to set the associated Mayhemfile accordingly with a duration of 120 seconds to account for the extra time needed to find the additional defect. You should be able to find two defects for the testme-npd binary.

1. Download and extract the following: testme-docker.zip

2. Append the testme.c file by adding an if-else statement that invokes a null pointer dereference bug when the test case containing null is input to the application.

3. Rebuild the Dockerfile using the docker build command and tag the resulting Docker image as testme-npd.

4. Push the testme-npd Docker image to the internal Mayhem Docker registry using the docker tag and docker push commands.

5. Fuzz the testme-npd Docker image using either the Mayhem UI or Mayhem CLI and make sure to set the associated Mayhemfile accordingly with a duration of 120 seconds to account for the extra time needed to find the additional defect. You should be able to find two defects for the testme-npd binary.

🔍 Review It! Building and Testing testme-npd

Solution

Solved Files: testme-npd.zip

First things first, you needed to add the null pointer dereference code snippet to the testme function. This essentially makes it so that when the testme application is fuzzed, the input test case "null" will trigger the null pointer dereference error.

int testme(char *buf, unsigned len)
{
  unsigned ok;

  if(!ok) // Defect: uninitialized use of ok.
    ok = len;

  if(buf[0] == 'b')
    if(buf[1] == 'u')
      if(buf[2] == 'g') {
        return abort(); // Defect: SIGABRT.
      }

  if(buf[0] == 'n')
    if(buf[1] == 'u')
      if(buf[2] == 'l')
        if(buf[3] == 'l') {
          int *p = NULL; // Defect: null pointer dereference
          int y = *p;
        }

  return 0;
}

Then, you needed to run the docker build command in the same directory as the Dockerfile and proceed to tag the resulting Docker image as testme-npd:

docker build -f Dockerfile -t testme-npd .

Next, you had to push the testme-npd Docker image to the internal Mayhem Docker registry.

docker tag testme-npd tutorial.forallsecure.com:5000/forallsecure/testme-npd
docker push tutorial.forallsecure.com:5000/forallsecure/testme-npd

Here we uploaded the testme-npd Docker image to our tutorial.forallsecure.com Mayhem deployment, specifically at the internal Docker registry located at port 5000. For your Docker image to have successfully uploaded, you would have had to adjust the URL for your Mayhem deployment Docker registry accordingly.

Lastly, you could have executed a Mayhem run on the uploaded testme-npd Docker image using either the Mayhem UI or Mayhem CLI. As long as your Mayhemfile looked similar to the following:

image: $MAYHEM_DOCKER_REGISTRY/forallsecure/testme-npd:latest
duration: 120
project: forallsecure-testme-npd
target: testme-npd
cmds:
  - cmd: /mayhem/testme/v1/testme @@

Your final run page should look like the following:

Congratulations! Mayhem found the improper input validation defect as well as the null pointer dereference defect that you added! You just built a Docker image from scratch and added a vulnerability that was detected in Mayhem!

✏️ Summary and Recap

In this lesson, you learned how to build your own Docker image and push your Docker image to Docker Hub to be fuzzed!

In this lesson, you learned how to build your own Docker image and push your Docker image to Mayhem's private Docker registry to be fuzzed!

Docker images can be built using Dockerfiles and the docker build command. Once built, local Docker images can then be pushed to a Docker registry located on a deployed Mayhem instance. With the Docker image now available for ingestion into Mayhem, users can execute runs on the Docker image using either the Mayhem UI or the Mayhem CLI.

---

I learned how to...

1. Build my own local Docker image.

• With Docker, it's easy to build your own image containing your target binary. To do so, users will simply need to specify a Dockerfile and use the docker build command.

  FROM debian:buster-slim as builder
  RUN apt-get update && \
      apt-get install -y gcc make libc6-dbg && \
      rm -rf /var/lib/apt/lists/*
  WORKDIR /mayhem
  COPY src testme
  RUN cd testme && make
  

• Here we can see the following commands used in the Dockerfile:

  • FROM: Initializes a new build stage and sets the Base Image for subsequent instructions. A valid Dockerfile must start with a FROM instruction.
  • RUN: Executes any commands in a new layer on top of the current image and commits the results. The resulting committed image will be used for the next step in the Dockerfile.
  • WORKDIR: Sets the working directory for any RUN, CMD, ENTRYPOINT, COPY and ADD instructions that follow it in the Dockerfile.
  • COPY: Copies new files or directories from and adds them to the filesystem of the container at the path .

2. Push local Docker images to a Docker registry.

• Users can push their newly built Docker images to the public Docker Hub registry.
• Users can also verify that their Docker images have been successfully pushed to Docker Hub by visiting the website and searching for their image name.

• Users can push their newly built Docker images to Mayhem's internal Docker registry which is included on all Mayhem deployments.
• Users will need to first log into the Mayhem Docker registry using the mayhem login command and then can use the docker tag and docker push commands to name and upload their local Docker images into the private Mayhem Docker registry.
• Users can also verify that their Docker images have been successfully pushed to the Mayhem Docker registry by visiting the Mayhem Docker Registry in the Mayhem UI.

3. Execute a Mayhem run on your Docker image via the Mayhem UI.

• With the Docker image successfully uploaded to Mayhem's internal Docker registry, users can use the Create New Run flow in the Mayhem UI to fuzz their uploaded Docker images.

4. Execute a Mayhem run on your Docker image via the Mayhem CLI.

• With the Docker image successfully uploaded to Mayhem's internal Docker registry, users can configure a Mayhemfile and set the image parameter to point to the uploaded Docker image. Then, users can execute the mayhem run command via the Mayhem CLI to fuzz their uploaded Docker images.