Instrumentation Overview

In this section, we will discuss the Filibuster instrumentation and how it is used for in fault injection testing. The focus of this presentation will be on our built-in HTTP instrumentation. However, the insights that we provide are valuable to anyone looking to instrument other libraries for use with Filibuster.


The goals of instrumenting libraries for use with Filibuster are the following:

  • Identify when and where remote calls are made between two different services.

  • Alter the responses of remote service calls: either by throwing an exception at the callsite or returning a response that indicates an error.

  • Propagating required instrumentation metadata from service to service as calls are made.

We discuss each goal below to outline the overall architecture of Filibuster.

Filibuster Architecture


Filibuster runs a server that is responsible for running the functional test for the application, generating additional test executions that try the various ways calls can fail between services, running those additional test executions, gathering coverage data and reporting the results back to the developer.

We now look at the concrete functionality we need to address the three goals:

  • In order for Filibuster to know what additional test executions to generate, the Filibuster server must be notified of the remote calls made between different services. For this, the Filibuster server exposes an HTTP API that instrumented libraries communicate with.

  • When executing subsequent test executions and injecting failures, Filibuster will notify the instrumentation through the responses to the HTTP API calls with information on whether or not a failure should be injected. It is up to the instrumentation to inject the appropriate failure based on the information returned by the Filibuster server.

  • Filibuster’s dynamic reduction algorithm requires that both requests between different services are able to be uniquely identified across executions (using execution indexes) and that we are able to understand the causal relationships between requests across service boundaries (using vector clocks.) This information needs to be propagated between requests through instrumentation.

Anatomy of a Request

To understand how Filibuster addresses the first two goals, we look at a diagram that shows when, and from where, the instrumentation calls are made.


To understand the data format and API of these instrumentation calls, we refer you to the Filibuster Server API section of the documentation.


Referencing the above diagram, instrumentation works as follows. First, before the target method is invoked, instrumented libraries should first issue an invocation call to Filibuster to notify it of the remote call that is about to be issued. The return value from this call will notify the instrumentation on whether or not a fault should be injected. Next, if the call is issued, the target of the call should notify the Filibuster server of the service that received the remote call. Finally, if the call is actually issued, the instrumentation should notify the Filibuster that the call has completed with the response value.

The table below and diagram describes where these instrumentation calls should be placed in the request path.

Request process with instrumentation endpoints and instrumentation types








Request originates at Service A



requests.get used by Service A to issue a request to B



PUT /filibuster/create


Notify Filibuster server invocation with call site and metadata



Issue call to Service B if fault should not be injected



POST /filibuster/update


Notify Filibuster server of invoked service



POST /filibuster/update


Notify Filibuster of return value when call completes

Metadata Propagation

We now look at how we address the third goal.

Filibuster requires that instrumentation metadata be propagated between services on nested service calls. To achieve this, our instrumentation automatically assigns these values to headers associated with each call and then parses these values when received with an incoming call.

We refer the reader to the following diagram for a depiction of how this occurs in the case of our examples written with Flask and the requests library.


Instrumentation calls made to the Filibuster server require that five values be present, we discuss these below.

Metadata propagated by instrumentation required in instrumentation calls



Data Type





A totally ordered identifier for this request in this test execution



VClock (Map)

Vector clock associated with this outgoing request



VClock (Map)

Vector clock associated with the request that originated this request



EI (String)

Encoded execution associated with this outgoing request




Unique identifier for the user request; used to assign a vector clock and execution index.