Using Failure Flags by proxy
Supported platforms:
Failure Flags supports proxy-based fault injection, letting you run experiments without making changes to your application code. This approach works by intercepting network traffic from your application and injecting failures at the network level. This is the recommended approach to using Failure Flags as it's scalable, minimizes performance impact, and requires less upfront development effort.
How Failure Flags by proxy works
Failure Flags by proxy routes network traffic from your application through a sidecar container, which then forwards the traffic to its destination. The sidecar intercepts this traffic for the sole purpose of orchestrating experiments that your team runs using Gremlin, without requiring code changes.
Setting up Failure Flags by proxy involves:
- Deploying the Failure Flags sidecar alongside your application.
- Configuring your desired proxy modes (
ingress,dependency, orlambda). - Configuring your environment to route traffic through the sidecar proxies.
- Generating network traffic to let Gremlin automatically create targets for scoping experiments.
- Creating and running experiments using Gremlin.
Proxy modes
The sidecar supports three proxy modes: ingress, dependency, and Lambda runtime API. These proxy modes are not mutually exclusive and can be used in tandem: for example, you can enable the ingress and dependency proxy to experiment on inbound and outbound traffic in an application.
Ingress proxy mode
The ingress proxy intercepts inbound network requests to your application, letting you simulate response delays, errors, or corrupted responses. It automatically creates two new Failure Flags in Gremlin named ingress and http-ingress.
Dependency proxy mode
The dependency proxy intercepts outbound HTTP/HTTPS calls to external dependencies, letting you simulate network failures, slow responses, or API errors. For each HTTP(S) dependency your application communicates with, Gremlin creates a Failure Flag named dependency-<dependency hostname>. Gremlin also creates a general Failure Flag named response for outbound traffic.
Lambda Runtime API proxy mode
The Lambda Runtime API proxy is specifically designed for AWS Lambda functions, as it intercepts Lambda runtime API calls. This simplifies deploying Failure Flags to AWS Lambda.
Enabling Failure Flags by proxy
Running Failure Flags experiments requires enabling one or more proxy modes, which are points in your application’s execution where you can inject faults. Gremlin automatically creates these modes, but you can enable or disable them independently.
Before following any of these steps, you should first follow the instructions for deploying Failure Flags to your platform of choice:
- Deploying Failure Flags on AWS Lambda
- Deploying Failure Flags on AWS ECS
- Deploying Failure Flags on Kubernetes
- Deploying Failure Flags on Istio via Envoy
- Deploying Failure Flags on Pivotal Cloud Foundry (PCF)
The sidecar supports three proxy modes: ingress, dependency, and Lambda runtime API (lambda for short). You can enable, disable, and configure each proxy mode using configuration file properties or environment variables. Proxy modes are not mutually exclusive and can be used in tandem. For example, you can enable the ingress and dependency modes simultaneously to experiment on inbound and outbound traffic.
Common proxy settings
These options apply to all proxy types:
Ingress proxy
The ingress proxy mode enables experiments on inbound network calls to your application. It automatically creates the ingress and http-ingress (for HTTP traffic) flags.
To enable ingress proxy mode experiments, set the following options:
Dependency proxy
The dependency proxy mode enables experiments on outbound network calls to external endpoints. For each dependency your application communicates with, Gremlin automatically creates a Failure Flag named dependency-<dependency name>. Gremlin also creates a generic response Failure Flag for all outbound traffic.
To enable dependency proxy mode experiments, set the following options:
AWS Lambda API proxy
The Lambda Runtime API (lambda for short) proxy is specifically designed for AWS Lambda functions, as it intercepts Lambda runtime API calls.
Your Lambda function(s) must have the AWS_LAMBDA_RUNTIME_API environment variable set to the port where the Lambda proxy is running. The Lambda Layer includes a bootstrap script that automatically sets this variable. You can optionally enable this script by setting the AWS_LAMBDA_EXEC_WRAPPER environment variable to /opt/bootstrap.
To enable Lambda API proxy experiments, set the following options:
Automatically-created Failure Flags
These flags are automatically created by Gremlin when certain proxy modes are enabled. You may not see all of these flags for all applications, depending on your configuration.
dependency-<name>
Gremlin can automatically identify HTTP/HTTPS endpoints that your application frequently communicates with. When the dependency proxy is enabled, each of these endpoints will be listed as its own Failure Flag named dependency-<hostname>.
egress
The egress Failure Flag is a superflag that represents all dependencies rather than a specific one. Selecting this lets you run experiments on all outbound traffic. Gremlin adds this flag when the dependency proxy mode is enabled.
http-ingress
The http-ingress Failure Flag is created when the sidecar detects inbound HTTP or HTTPS traffic to your application. This makes it easy to target HTTP traffic without using selectors. Gremlin adds this flag when the ingress proxy mode is enabled.
ingress
The ingress Failure Flag represents all inbound traffic. Gremlin adds this flag when the ingress proxy mode is enabled.
process
The process Failure Flag represents the Gremlin sidecar. Running experiments on this Failure Flag will apply the effects to the sidecar. Gremlin adds this flag for all applications.
response
The response Failure Flag represents traffic that your application sends in response to a request from another application.