Using Failure Flags by proxy
Supported platforms:
Failure Flags supports proxy-based fault injection, letting you run experiments without making changes to your application code. This approach works by intercepting network traffic from your application and injecting failures at the network level. This is the recommended approach to using Failure Flags as it's scalable, minimizes performance impact, and requires less upfront development effort.
How Failure Flags by proxy works
Failure Flags by proxy routes network traffic from your application through a sidecar container, which then forwards the traffic to its destination. The sidecar intercepts this traffic for the sole purpose of orchestrating experiments that your team runs using Gremlin, without requiring code changes.
Setting up Failure Flags by proxy involves:
- Deploying the Failure Flags sidecar alongside your application.
- Configuring your desired proxy modes (
ingress,dependency, orlambda). - Configuring your environment to route traffic through the sidecar proxies.
- Generating network traffic to let Gremlin automatically create targets for scoping experiments.
- Creating and running experiments using Gremlin.
Proxy modes
The sidecar supports three proxy modes: ingress, dependency, and Lambda runtime API. These proxy modes are not mutually exclusive and can be used in tandem. For example, you can enable the ingress and dependency proxy to experiment on inbound and outbound traffic in an application.
Ingress proxy mode
The ingress proxy intercepts inbound network requests to your service, letting you simulate response delays, errors, or corrupted responses. It automatically creates two new Failure Flags in Gremlin named ingress and http-ingress.
Dependency proxy mode
The dependency proxy intercepts outbound HTTP/HTTPS calls to external dependencies, letting you simulate network failures, slow responses, or API errors. For each HTTP(S) dependency your service communicates with, Gremlin creates a Failure Flag named dependency-<dependency hostname>. Gremlin also creates a general Failure Flag named response for outbound traffic.
Lambda Runtime API proxy mode
The Lambda Runtime API proxy is specifically designed for AWS Lambda functions, as it intercepts Lambda runtime API calls. This simplifies deploying Failure Flags to AWS Lambda.
Enabling Failure Flags by proxy
Running Failure Flags experiments requires enabling one or more proxy modes, which are points in your application’s execution where you can inject faults. Gremlin automatically creates these modes, but you can enable or disable them independently.
Before following any of these steps, you should first follow the instructions for deploying Failure Flags to your platform of choice:
- Deploying Failure Flags on AWS Lambda
- Deploying Failure Flags on AWS ECS
- Deploying Failure Flags on Kubernetes
- Deploying Failure Flags on Istio via Envoy
- Deploying Failure Flags on Pivotal Cloud Foundry (PCF)
The sidecar supports three proxy modes: ingress, dependency, and Lambda runtime API (lambda for short). You can enable, disable, and configure each proxy mode using configuration file properties or environment variables. Proxy modes are not mutually exclusive and can be used in tandem. For example, you can enable the ingress and dependency modes simultaneously to experiment on inbound and outbound traffic.
Common proxy settings
These options apply to all proxy types:
Ingress proxy
The ingress proxy mode intercepts inbound network requests to your service, letting you simulate response delays, errors, or corrupted responses. It automatically creates two new Failure Flags in Gremlin named ingress and http-ingress.
To enable ingress proxy mode experiments, set the following options:
Dependency proxy
The dependency proxy intercepts outbound HTTP/HTTPS calls to external endpoints and services. This lets you simulate network failures, slow dependencies, and API errors. For each HTTP(S) dependency your service communicates with, Gremlin automatically creates a Failure Flag named dependency-<dependency name>. Gremlin also creates a generic Failure Flag named response for all outbound traffic.
To enable dependency proxy mode experiments, set the following options:
AWS Lambda API proxy
The Lambda Runtime API (lambda for short) proxy is specifically designed for AWS Lambda functions, as it intercepts Lambda runtime API calls. This simplifies deploying Failure Flags to AWS Lambda services.
Your Lambda function(s) must have the AWS_LAMBDA_RUNTIME_API environment variable set to the port where the Lambda proxy is running. The Lambda Layer includes a bootstrap script that automatically sets this variable. You can optionally enable this script by setting the AWS_LAMBDA_EXEC_WRAPPER environment variable to /opt/bootstrap.
To enable Lambda API proxy experiments, set the following options:
Automatically-created Failure Flags
These flags are automatically created by Gremlin when certain proxies are enabled. You may not see all of these flags for all services, depending on your configuration.
egress
The egress Failure Flag is a superflag that represents all dependencies rather than a specific one. Selecting this lets you run experiments on all outbound traffic. Gremlin adds this flag when the dependency proxy mode is enabled.
http-ingress
The http-ingress Failure Flag is created when the sidecar detects inbound HTTP or HTTPS traffic to your service. This makes it easy to target HTTP traffic without using selectors. Gremlin adds this flag when the ingress proxy mode is enabled.
ingress
The ingress Failure Flag represents all inbound traffic. Gremlin adds this flag when the ingress proxy mode is enabled.
process
The process Failure Flag represents the Gremlin sidecar. Running experiments on this Failure Flag will apply the effects to the sidecar. Gremlin adds this flag for all services.
response
The response Failure Flag represents traffic that your service sends in response to a request from another service.