Successful Coverage¶

Once authentication is configured, the next step is to ensure that Mayhem for API is able to successfully cover as much of your API as possible.

What is Successful Coverage?¶

Mayhem considers any 2xx response from the API server to indicate success, and a single successful response from a given endpoint is enough for us to consider it successfully covered.

If your API has some other criteria for success: we don't currently support it, but we are interested in this use case, so please reach out to us on Discord or by emailing support@forallsecure.com, and let's see what we can figure out!

Why is Successful Coverage Important?¶

The absence of findings, for an endpoint which is not successfully covered, is meaningless.

Mayhem API testing is designed around this idea: that the most interesting test results come from requests that are almost correct. Some of the payloads we generate are purely random—that's important, too!—but once we find a successful payload for a given endpoint, we use that as a starting point for many future requests.

Consider a single POST endpoint which executes an INSERT statement against a relational database on success. If every single request issued by Mayhem is invalid, the INSERT statement has never been executed; if that INSERT statement has a SQL injection vulnerability, it won't be found.

If, on the other hand, we pass a valid payload (causing the INSERT statement to be executed) and get a 200 from this endpoint, we'll then use the successful payload as the basis for many future tests. If that INSERT statement has a SQL injection bug, we'll eventually find it.

That's why we emphasize "successful coverage": successfully covering an endpoint massively increases our confidence that the absence of findings is meaningful.

Reasons For Lack of Successful Coverage¶

Authentication, which is documented separately, is often the first barrier to successful coverage.

Mayhem (and fuzzing in general) usually does just fine, without intervention, generating values that satisfy simple data validations—things like "the 'page' parameter must be an integer".

What are other reasons Mayhem might be failing to successfully cover an endpoint?

Incorrect URLs¶

Sometimes the simplest solution is hard to see! If you're getting unexpected failures on endpoints that should be easy to exercise successfully—especially if the failures are 404s!—double-check that Mayhem is hitting the correct URLs.

Complex Structural Validation¶

Info

If you start with thorough OpenAPI specs, you're less likely to run into this problem.

While we'll fairly quickly generate a random value that "must be an integer", it'll take quite a long time to generate a random value that, for example, "must be a JSON document with exactly one key, named 'page', whose value is an integer".

Stateful Validation¶

The trickiest validations are ones which are based on the current state of the service. For example, a value that "must be the unique id of an existing row in the 'pages' table" might take essentially forever to generate through pure randomness.

Mayhem has some tricks up its sleeves (such as trying values from the output of one request in the inputs of another) that help, but it's not perfect.

Whatever the reason, what follows are suggestions for diagnosing and fixing problems in successful coverage.

Diagnosing Problems in Successful Coverage¶

If you're still here, that means you have at least one endpoint that Mayhem can't yet successfully cover. So, how do you begin to figure out why that's happening?

It's worth noting that this may be extremely hard unless you either are (or working closely with someone who is) deeply familiar with the service being tested.

Step One: Focus¶

We strongly recommend that you dig into one endpoint (or a very small set of interrelated endpoints) at a time, when attempting to get successful coverage, and to keep the runs fairly short (60 seconds or less).

Filter your runs down via the --include-endpoint <endpoint> flag to the mapi run, or other filtering methods documented in more detail here.

Step Two: Logs¶

When looking at logs, what you're looking for are patterns: what kinds of requests is Mayhem making, and why are they failing? What kinds of requests is Mayhem not making, and how can it be coerced into making them?

Start by looking at your own service logs, since hopefully you're familiar with those!

If your own logs aren't illuminating enough, you can also capture the full request/response payloads of an entire API testing run using the --har <file> flag to mapi run. The output is in JSON HAR format.

Step Three: Reach Out!¶

Don't hesitate to reach out to us on Discord or by emailing support@forallsecure.com. Between your expertise with your service, and our expertise with Mayhem, we should be able to figure it out!

How To Improve Successful Coverage¶

Correct the URLs¶

Mayhem always constructs URLs by concatenating each endpoint's path with a common server URL.

The paths are taken directly from the API specification.

The server URL is given via the --url <url> argument to mapi run. (There are other ways, but we'll focus on this one, as it is the most explicit!)

Sometimes the specification elides a common prefix from all endpoints. Consider a specification:

  /version:
    get:
     ...
  /user:
    get:
      ...

If the complete URLs are supposed to be https://localhost/api/v2/version and https://localhost/api/v2/user, the common prefix (/api/v2) will need to be included in the server URL given to Mayhem:

mapi run ... --url 'https://localhost/api/v2'

Improve The Spec¶

Often, Mayhem just needs a bit more... specificity... in the specification, in order to successfully cover an endpoint.

Provide Schemas¶

If Mayhem is consistently unable to generate structurally valid payloads, this can usually be resolved by adding or refining schemas in the spec.

OpenAPI allows for parameters and request body types to be declared, in whatever level of detail is needed, using Schema Objects.

For example, just listing the required properties for an endpoint's request body makes Mayhem massively more likely to successfully cover that endpoint:

  /example:
    post:
      summary: example
      requestBody:
        required: true
        content:
          application/json:
            schema:
              type: object
+             properties:
+               page:
+                 type: integer

Use Consistent Names and Types¶

Mayhem will do a better job generating statefully valid payloads if stateful identifiers that are passed between endpoints have the same name and type.

For example, if you have an endpoint that lists resources and another that fetches a resource by id, make sure the identifiers share a name and type, so that Mayhem can more easily connect the dots:

  /example:
    get:
      summary: list of examples
      responses:
        '200':
          description: all the examples
          content:
            application/json:
              schema:
                type: array
                items:
                  type: object
                  properties:
                    example_id:
                      type: integer
- /example/{id}:
+ /example/{example_id}:
    get:
      summary: details of a single example
      parameters:
        - in: path
-         name: id
+         name: example_id
          schema:
-           type: string
+           type: integer

Info

The spec changes you make to help Mayhem exercise your API will also improve anything else—code, documentation, and so on—that you derive from your specs! Win-win!

Provide Examples in your Specification¶

There are situations where Mayhem really just needs a hint. For example, consider endpoints that include the username of the currently-authenticated user, like '/user/{username}/settings'.

Info

Examples are often needed when the service being tested has some pre-loaded state (like the user, for authentication!) that's difficult to naturally "discover" through fuzzing.

In this case, OpenAPI allows "example" values, and Mayhem will take the examples into account when generating request payloads. For the "{username}" parameterized endpoint, this might be as simple as:

  /user/{username}/settings:
    get:
      summary: settings
      parameters:
        - in: path
          name: username
          schema:
            type: string
+           example: mayhem4api-user

Resource Hints¶

Mayhem may need additional hints to make a successful requests to endpoints. Perhaps every request requires that a constant value is applied to every request body or path parameter. It is possible to provide hints for all generated requests by passing one or more --resource-hint options to the mapi run command or defining them in the mapi config file.\ See:

The .mapi configuration file section for more information on how to specify resource hints from the mapi configuration file
Implicit vs Explicit resource hint groups section for more information on the differences between resource hints provided via the cli (Implicit) vs resource hints provided via the .mapi configuration file (Explicit)

In order to understand which resources are applicable to the --resource-hint option, you should first list the available resources for your API specification with the mapi describe specfication command.

For example, let's list the resources of the petstore demo API:

mapi describe specification \
  "https://demo-api.mayhem.security/api/v3/openapi.json"

Spec path is https://demo-api.mayhem.security/api/v3/openapi.json
PUT /pet BODY category/id
PUT /pet BODY category/name
PUT /pet BODY id
PUT /pet BODY name
...
PUT /user/{username} BODY userStatus
PUT /user/{username} BODY username
DELETE /user/{username} PATH username

Every resource for the specification is flattened and listed out in the response. For example, the following represents a field value that is passed as part of a request body when generating a PUT request against /pet.

Method
|
|   Path
|   |
|   |    Request part (can be QUERY, PATH, HEADER or BODY)
v   v    v
PUT /pet BODY category/id
              ^
              Fully qualified path to request body

Another example is a path parameter:

Method
|
|       Path
|       |
|       |               Request part
v       v               v
DELETE /user/{username} PATH username
                             ^
                             Path parameter

The --resource-hint option to mapi run takes a tuple of resource path and value, separated by a colon.

For example, say we wanted to always use the same username value of foo for the DELETE method noted above:

mapi run \
     petstore auto "https://demo-api.mayhem.security/api/v3/openapi.json" \
     --url "https://demo-api.mayhem.security/api/v3/" \
     --resource-hint "DELETE /user/\{username\} PATH username:foo"

The --resource-hint specifies that any request generated for this DELETE resource will now use foo for the username parameter:

                 Resource Path                          split with ':'
                 v                                      v
--resource-hint "DELETE /user/\{username\} PATH username:foo"
                              ^                          |
                              must escape '{' for        |
                              valid regex                |
                                                         ^ Value to use

--resource-hint will accept a regular expression for matching resource path as well. For instance, say you wanted to substitute foo for EVERY username value:

                         Match ANY request that requires a 'username'
                         v
--resource-hint "username$:foo"

When multiple --resource-hint are specified that match the same resource path, Mayhem will randomly select one of the provided hints whenever it generates a request.

                Choose between 'foo'  ...       or 'bar'
                v                               v
--resource-hint "username$:foo" --resource-hint "username$:bar"

The .mapi configuration file¶

You can specify a collection of resource hint groups in the .mapi configuration file:

#
# Example .mapi configuration file
#
version: "1.0"

...

# You can specify different groups with different combinations of hints
resource_hints:
- name: Group Name A
  hints:
  - "username:foo"
  - "group:bar"
- name: Group Name B
  hints:
  - "username:baz"
  - "group:group-A"
  - "group:group-B" # You can also specify the same regular expression with different values

A resource hint group consists of a name field and a collection of tuples of resource path and value, separated by a colon.\ You can specify the same resource path multiple times with a different value each time. In this case, if this resource group is selected, a value will be selected in random.\ For each tuple (resource hint) the same rules apply as for the resource hints specified via the cli.

Implicit vs Explicit resource hint groups¶

With the introduction of resource hint groups, the value selection algorithm has changed a bit.\ We now have two categories of resource hints:

Implicit (resource hints specified via the cli)
Explicit (resource hints speicfied via the .mapi configuration file)

You can think all of the resource hints specified via the cli as a single implict group (without name).\ The major change with resource hint groups is that if mapi selects a resource hint from a group, then it will try to select all the other resource hints from that previously selected group (if any matches are found).\ This allows you to basically specify unique valid combinations for resources.

Let's have a look at an example:

Suppose we have the following resource:

PUT /user/{username} BODY userStatus

and the following resource hint group:

resource_hints:
- name: Group A
  hints:
  - "username$:foo"
  - "userStatus$:good"

Since mapi will match one of the resource hints in that group (e.g username$ or userStatus$), next time it will generate a request for that resource, it will try to select a resource hint from that group that has not been selected previously. In that way, mapi will be able to generate requests with combinations of values that make sense for your request.

Ignore Endpoints¶

If, for some reason, your endpoint cannot be covered successfully, consider skipping it during fuzzing (using the --ignore-endpoint <endpoint> argument to mapi run), because we have low confidence in lack of findings.

At this point, Mayhem is able to exercise your API quite thoroughly, and probably uncover all sorts of issues... and, more importantly in the long run, give confidence that your API is free of the issues we don't find!

The next section describes optional setup steps to maximize the specificity and detail of the issues Mayhem reports. You can do that now, or you can come back to it later.