Using Failure Flags by proxy

Failure Flags supports proxy-based fault injection, letting you run experiments without making changes to your application code. This approach works by intercepting network traffic from your application and injecting failures at the network level. This is the most scalable approach to rolling our fault injection capabilities in any environment, while minimizing the performance impact to your architecture outside of the scope of an experiment.

‍

How Failure Flags by proxy works

Failure Flags by proxy routes network traffic from your application through a sidecar container, which then forwards the traffic to its destination. The sidecar intercepts this traffic for the sole purpose of orchestrating experiments that your team runs using the Gremlin web app (or API). Unlike the Failure Flags SDK, this lets you run experiments on your applications without changing a single line of code.

The process of setting up Failure Flags by proxy includes:

1. Deploying the Failure Flags sidecar alongside your application.
2. Configuring your desired proxy modes (ingress, dependency, or Lambda).
3. Configuring your environment to route traffic through the sidecar proxies.
4. Generating network traffic to let Gremlin automatically create targets for scoping experiments.
5. Creating and running experiments using Gremlin.

‍

Proxy modes

The sidecar supports three proxy modes: ingress, dependency, and Lambda runtime API. These proxy modes are not mutually exclusive and can be used in tandem. For example, you can enable the ingress and dependency proxy to experiment on inbound and outbound traffic in an application.

‍

Ingress proxy

The ingress proxy intercepts inbound network requests to your service, letting you simulate response delays, errors, or corrupted responses. It automatically creates two new Failure Flags in Gremlin named ingress and http-ingress.

‍

Dependency proxy

The dependency proxy intercepts outbound HTTP/HTTPS calls to external dependencies, letting you simulate network failures, slow responses, or API errors. For each HTTP(S) dependency your service communicates with, Gremlin creates a Failure Flag named dependency-<dependency hostname>. Gremlin also creates a general Failure Flag named response for outbound traffic.

‍

Lambda Runtime API proxy

The Lambda Runtime API proxy is specifically designed for AWS Lambda functions, as it intercepts Lambda runtime API calls. This simplifies deploying Failure Flags to AWS Lambda.

‍

Enabling Failure Flags by proxy

Before following any of these steps, you should first follow the instructions for deploying Failure Flags to your platform of choice:

• Deploying Failure Flags on AWS Lambda
• Deploying Failure Flags on AWS ECS
• Deploying Failure Flags on Kubernetes
• Deploying Failure Flags on Istio via Envoy
• Deploying Failure Flags on Pivotal Cloud Foundry (PCF)

Once you've enabled and authenticated the sidecar, follow the configuration instructions below for your platform. Failure Flags supports two configuration methods: using configuration files (in YAML format), and using environment variables.

‍

AWS Lambda API proxy

For Lambda functions, add the following properties to your configuration file:

• lambda_proxy_enabled: set to true to enable the proxy.
• lambda_proxy_port: set the IP address/hostname and port that the proxy will bind to. Defaults to localhost:5033.

Alternatively, you can use the GREMLIN_LAMBDA_PROXY_ENABLED and GREMLIN_LAMBDA_PROXY_PORT environment variables.

Additionally, your Lambda function must have the AWS_LAMBDA_RUNTIME_API environment variable set to the port where the Lambda proxy is running. This is handled automatically when you use the provided Lambda Layer.

‍

Dependency proxy

To enable experiments on outbound network calls, add the following properties to your configuration file:

• dependency_proxy_enabled: set to true to enable the proxy for dependency traffic.‍
• dependency_proxy_port: set the IP address/hostname and port that the proxy will bind to. Defaults to localhost:5034.

Alternatively, you can use the GREMLIN_DEPENDENCY_PROXY_ENABLED and GREMLIN_DEPENDENCY_PROXY_PORT environment variables.

‍

Ingress proxy

To enable experiments on inbound network calls, add the following properties to your configuration file:

• ingress_proxy_enabled: set to true to enable the proxy for ingress traffic.‍
• ingress_proxy_port: set the IP address/hostname and port that the proxy will bind to. Defaults to localhost:5035.‍
• ingress_proxied_endpoint: the URL of your application. This is required.
  
  • For AWS ECS, set this to http://localhost:9080.

Alternatively, you can use the GREMLIN_INGRESS_PROXY_ENABLED, GREMLIN_INGRESS_PROXY_PORT, and GREMLIN_INGRESS_PROXIED_ENDPOINT environment variables.

‍

Common proxy settings

These options apply to all proxy types:

• proxy_idle_connection_timeout: sets the idle timeout for proxied connections. Defaults to 2m (two minutes).
• proxy_read_timeout: sets the read timeout. Defaults to 2m (two minutes).
• proxy_write_timeout: Sets the write timeout. Defaults to 2m (two minutes).

Alternatively, you can use the GREMLIN_PROXY_IDLE_TIMEOUT, GREMLIN_PROXY_READ_TIMEOUT, and GREMLIN_PROXY_WRITE_TIMEOUT environment variables.