Response Classify Plugins¶

Mayhem has support for response classification via plugins, which are standalone programs that implement a protobuf interface and expose it via gRPC.

The interface details are below. But first...

Do I Need This?¶

We've tried to make Mayhem work out of the box for as many use-cases as we can, and we'll keep adding more. These plugins are intended as a kind of escape-hatch for situations where an API has requirements that we can't currently meet out of the box.

Here are the cases we had in mind when building the feature:

Sensitive Data Leaking¶

If there are certain keys, values or patterns that you want to assert should never show up in your API output, regardless of inputs, Mayhem doesn't currently have a direct solution.

A response classify plugin can make these assertions.

Nonstandard Response Code Assertions¶

The rules Mayhem follows based on each endpoint's response codes are baked in, and based on standards. There may be specific endpoints where you want to apply more strict constraints—for example, an endpoint whose contract is that it never returns a 400 error.

A response classify plugin can be used to check for this kind of constraint.

Something else?¶

The classifier plugin interface is extremely open-ended, and can be used to check just about any aspect of a response.

If you're thinking about using a classify plugin for any reason, please get in touch. This is a very young, very raw feature, and we'd like to know how people are using it, and how we can make it better!

Interface¶

Invocation¶

To add a response classify plugin to an API testing run, use the --classify-plugin <path-to-plugin-executable> argument to mapi run.

Executable gRPC Server¶

mapi will run the executable with no command-line arguments, and expects it to be a long-lived process that: * listens for gRPC traffic on a localhost port * prints the port number, followed by a newline, to stdout as the first line

Beyond that, it can do whatever it wants. Additional stdout/stderr output from the plugin are forwarded to the mapi debug logs.

At the end of the testing run, mapi will attempt to kill the plugin process.

Protobuf¶

Response classify plugins implement the following protobuf interface:

syntax = "proto3";

package mapi.classify;

service ClassifyPlugin {
   rpc Classify(Response) returns (Issues);
}

message Response {
   message Header {
      bytes name = 1;
      bytes value = 2;
   }
   string request_url = 1;
   string final_url = 2;
   uint32 status = 3;
   repeated Header headers = 4;
   bytes body = 5;
}

message Issues {
   message Issue {
      string summary = 1;
   }
   repeated Issue issues = 1;
}

Example¶

Here's an example classifier plugin, which does... absolutely nothing, just acknowledging each response without flagging any issues.

First, generate all of the gRPC and protobuf boilerplate from the .proto above:

$ pip install grpcio-tools
$ python -m grpc_tools.protoc --python_out=. --grpc_python_out=. -I. response-classify-plugin.proto

Then create a python program which implements the ClassifyPluginServicer, stands up a gRPC server, and lets mapi know what port it's listening on:

#!/usr/bin/env python

import sys
from concurrent import futures
import grpc
import response_classify_plugin_pb2
import response_classify_plugin_pb2_grpc

## implement the ClassifyPlugin interface
class ClassifyPluginServicer(response_classify_plugin_pb2_grpc.ClassifyPluginServicer):
    def Classify(self, response, context):
        issues = response_classify_plugin_pb2.Issues()
        if something_went_wrong():
            issue = response_classify_plugin_pb2.Issues.Issue()
            issue.summary = "something went wrong!"
            issues.issues.append(issue)
        return issues

if __name__ == '__main__':
    ## boot up the gRPC server
    server = grpc.server(futures.ThreadPoolExecutor(max_workers=10))
    response_classify_plugin_pb2_grpc.add_ClassifyPluginServicer_to_server(
        ClassifyPluginServicer(), server)
    server.add_insecure_port('127.0.0.1:50051')
    server.start()

    ## inform mapi of the port we're listening on
    print("50051")
    sys.stdout.flush()

    server.wait_for_termination()

Make sure your .py file is executable, then pass it to mapi via --classify-plugin. That's it!

Caveats and Cautions¶

bytes, Not string!¶

API responses are not always representable as strings. Be careful!

Subject to Change¶

All of this is volatile and subject to change. If you're using this feature, please get in touch, so that we can keep you in the loop as we consider refinements and changes.