Failure Flags > Deploying Failure Flags on AWS Lambda

Deploying Failure Flags on AWS Lambda

On AWS Lambda, Failure Flags runs as a Lambda extension called Gremlin-Lambda, which you add to your function as a Lambda layer. The extension starts with your function and runs in the same execution environment. With Failure Flags by proxy, you don’t need to change your function’s code: the extension intercepts your function’s invocations and its outbound HTTP(S) calls, and Gremlin injects faults there. Gremlin-Lambda supports both x86_64 and arm64 functions. You can learn more about the AWS Lambda Extensions API in the AWS documentation.

‍

How Failure Flags by proxy works on Lambda

Lambda functions can use two proxy modes:

The Lambda Runtime API proxy sits between your function’s runtime and the Lambda Runtime API, on localhost:5033 by default. It sees every invocation and every response. The layer includes a wrapper script, /opt/bootstrap, which automatically configures your runtime to use the proxy.
The dependency proxy listens on localhost:5034 by default. Your function’s outbound HTTP and HTTPS calls reach it through the standard HTTP_PROXY and HTTPS_PROXY environment variables before continuing to their destination.

The ingress proxy mode used on ECS and Kubernetes isn’t used on Lambda. The Lambda Runtime API proxy covers inbound traffic.

‍

Prerequisites

Before you begin, you'll need:

Your Gremlin Team ID, team certificate, and team private key. You can retrieve these from Team Settings in the Gremlin web app.
A function on a Lambda managed runtime, such as Python, Node.js, Java, .NET, or Ruby, with permission to update your function’s configuration.

‍

Adding the Gremlin-Lambda Extension to your Lambda Function

Gremlin publishes a new layer version with each sidecar release, so the version number at the end of the ARN changes with each release. The Failure Flags sidecar release notes list the layer version for each release. If the version below is older than the latest release notes entry, use the version from the release notes.

The Extension's Amazon Resource Name (ARN) varies depending on where your Lambda Function is deployed and which architecture you're using. First, retrieve the following values:

The current Lambda layer version (32)
The Lambda layer ARN ID (044815399860)
The region where your Lambda function is deployed (e.g. us-east-1)
Your Lambda layer architecture (x86_64 or arm64)

Next, assemble the ARN by combining arn:aws:lambda:region:arnID:layer:gremlin-lambda-v2-architecture:version.

For example, to run version 32 of the Gremlin Lambda layer on an x86_64 function in ap-northeast-1, use the ARN arn:aws:lambda:ap-northeast-1:044815399860:layer:gremlin-lambda-v2-x86_64:32.

Once you have the correct ARN, add the extension to your Lambda Function by following the instructions in the AWS Lambda Developer Guide.

If you use the Lambda API proxy, set AWS_LAMBDA_EXEC_WRAPPER to /opt/bootstrap. If your function already has an exec wrapper from another layer (such as Dynatrace's /opt/dynatrace), move that value to GREMLIN_LAMBDA_NEXT_EXEC_WRAPPER so both wrappers run.

Lambda extensions can impact the size and performance of your function(s). Gremlin-Lambda's impact is minimal, but non-zero. See the AWS documentation on Lambda extensions for details.

‍

Building your own Lambda Layer

You may want to build your own Lambda Layer or incorporate the gremlin-lambda binary in a Lambda container image. Maybe you're at risk of using too many layers, or maybe your organization has a package caching policy. Either way you'll want to build your own asset. You can access all of the gremlin-lambda and failure-flags-sidecar builds via our website.

Files provided via assets.gremlin.com have the following format: https://assets.gremlin.com/packages/gremlin-lambda/{VERSION}/{ARCH}/{FILE}

You can reference the LATEST release at: https://assets.gremlin.com/packages/gremlin-lambda/latest/{ARCH}/{FILE}

ARCH is one of the following:

x86_64
arm64

Different releases may include different files, but the current FILE set includes:

gremlin-lambda-linux.tar.gz (x86_64)
layer-gremlin-lambda.zip
LICENSE

You can find a list of releases in our release notes, or in our VERSIONS file.

‍

Enabling and configuring Failure Flags for AWS Lambda

The Gremlin-Lambda Extension accepts the following options. When a variable references a file, use a fully qualified path from the root of the function’s file system, which is typically /var/task.

You can set options using environment variables (recommended) or a configuration file. Environment variables take precedence over configuration file values. When using a configuration file,

‍

Quick start

The following options will get you up and running quickly using the default settings:

Environment Variable	Example value	Description
`GREMLIN_LAMBDA_ENABLED`	`true`	Enables the Failure Flags Lambda extension.
`GREMLIN_TEAM_ID`	`11111111-1111-1111-1111-111111111111`	Your Gremlin Team ID. This and other credential details are available in the Gremlin web app.
`GREMLIN_TEAM_CERTIFICATE_FILE`	`file:///var/task/certificate_file.pem`	The path of your Gremlin team certificate file in your container.
`GREMLIN_TEAM_PRIVATE_KEY_FILE`	`file:///var/task/private_key.pem`	The path of your Gremlin team private key file.
`GREMLIN_LAMBDA_PROXY_ENABLED`	`true`	Enable the Lambda Runtime API proxy, which lets you run experiments on ingress traffic.
`GREMLIN_LAMBDA_PROXY_PORT`	`localhost:5033`	The IP address/hostname and port that the proxy will bind to.
`GREMLIN_DEPENDENCY_PROXY_ENABLED`	`true`	Enables the outbound proxy from your application to its dependencies, which lets you run experiments on traffic to your function's dependencies.
`GREMLIN_DEPENDENCY_PROXY_PORT`	`localhost:5034`	The IP address/hostname and port that the proxy will bind to.
`HTTP_PROXY`, `HTTPS_PROXY`	`http://localhost:5034`, `https://localhost:5034`	Where to route outbound traffic. This should match `GREMLIN_DEPENDENCY_PROXY_PORT`.

‍

Enabling Failure Flags

These options enable Failure Flags:

Environment Variable	Description
`GREMLIN_LAMBDA_ENABLED`	Set to `true` to enable the extension. Without it, the extension runs in no-op mode.
`FAILURE_FLAGS_ENABLED`	Set to `true` to enable the Failure Flags SDK. If you're not using the SDK and only want to use Failure Flags by proxy, you don't need to set this variable.
`GREMLIN_CONFIG_FILE`	If your config file is included in your function container, set this to the full path of the file. For example, `/var/task/config.yaml`.
`GREMLIN_CONFIG_ARN`	If your config file is hosted e.g. in AWS Secrets Manager, this is the Amazon Resource Name (ARN) of the resource containing your configuration file. If you must assume a different role at runtime to access that resource, you can specify that IAM role using `GREMLIN_CONFIG_ARN_ROLE`.
`AWS_LAMBDA_EXEC_WRAPPER`	Set to `/opt/bootstrap` to point your Lambda's runtime to the proxy automatically. If you already have a value for this that is not Gremlin, see `GREMLIN_LAMBDA_NEXT_EXEC_WRAPPER`
`GREMLIN_LAMBDA_NEXT_EXEC_WRAPPER`	Set to your existing or other Lambda layer wrapper, e.g. `/opt/otel-handler`. Gremlin's wrapper will automatically call this wrapper as well as its own.

‍

Customizing the service name and adding labels

Gremlin automatically detects the service name from your Lambda function's name and environment. If you want to override this, you can do so using the SERVICE_NAME option. You can also add metadata labels to uniquely identify your service deployments using GREMLIN_LABEL_[key], where [key] is the name of your custom label:

Environment Variable	Config file option	Description
`SERVICE_NAME`	`service_name`	Set to the name of the service as you want it to appear in Gremlin. Service names can only contain alphanumeric characters, hyphens, and underscores, and must be less than 64 characters long.
`GREMLIN_LABEL_*`	`labels.*`	Change `*` to the name of a custom label that you want to assign to this function. For example, `GREMLIN_LABEL_ENVIRONMENT=Production` creates a label named `Environment` with the value `Production`.

‍

Adding credentials

When adding credentials, you can choose to provide your Gremlin team certificate and private key directly, in a file in your Lambda function, or as an Amazon Resource Name (ARN) resource (e.g. for storing them in Secrets Manager). You can retrieve your credentials from the Gremlin web app.

Environment Variable	Config file option	Description
`GREMLIN_TEAM_ID`	`team_id`	Set to your Gremlin Team ID. This and other credential details are available in the Gremlin web app.
`GREMLIN_TEAM_CERTIFICATE`	`team_certificate`	Set to the contents of your Gremlin team certificate. Preserve newlines by adding the \nescape character at the end of each line, or by omitting them entirely.
`GREMLIN_TEAM_CERTIFICATE_FILE`	`team_certificate_file`	Set to the full path of your Gremlin team certificate file.
`GREMLIN_TEAM_CERTIFICATE_ARN`	`team_certificate_arn`	Set to the Amazon Resource Name (ARN) where your Gremlin team certificate is stored.
`GREMLIN_TEAM_PRIVATE_KEY`	`team_private_key`	Set to the contents of your Gremlin team private key. Preserve newlines by adding the \nescape character at the end of each line, or by omitting them entirely.
`GREMLIN_TEAM_PRIVATE_KEY_FILE`	`team_private_key_file`	Set to the full path of your Gremlin team private key file.
`GREMLIN_TEAM_PRIVATE_KEY_ARN`	`team_private_key_arn`	Set to the Amazon Resource Name (ARN) where your Gremlin team private key is stored.

‍

Enabling the Lambda API proxy

The Lambda Runtime API (lambda for short) proxy is specifically designed for AWS Lambda functions, as it intercepts Lambda runtime API calls.

To enable Lambda API proxy experiments, set the following options:

Environment Variable	Config file option	Description
`GREMLIN_LAMBDA_PROXY_ENABLED`	`lambda_proxy_enabled`	Set to `true` to enable the Lambda Runtime API proxy. This proxy lets you run experiments on ingress traffic.
`GREMLIN_LAMBDA_PROXY_PORT`	`lambda_proxy_port`	Set the IP address/hostname and port that the proxy will bind to. Defaults to `localhost:5033`.

‍

Enabling dependency proxy mode

The dependency proxy mode enables experiments on outbound network calls to external endpoints. For each dependency your application communicates with, Gremlin automatically creates a Failure Flag named dependency-. Gremlin also creates a generic egress Failure Flag for all dependency traffic.

To enable dependency proxy mode experiments, set the following options:

Environment variable	Description
`GREMLIN_DEPENDENCY_PROXY_ENABLED`	Set to `true` to proxy traffic from your application to its dependencies.
`GREMLIN_DEPENDENCY_PROXY_PORT`	Set the IP address/hostname and port that the proxy will bind to. Defaults to `localhost:5034`.
`HTTP_PROXY`, `HTTPS_PROXY`	Set to the address of `GREMLIN_DEPENDENCY_PROXY_PORT` to route outbound traffic from your function to the proxy. For example, `http://localhost:5034`.

‍

Disabling metrics

The Failure Flags sidecar can collect traffic metrics to make testing easier. This is used solely for Intelligent Health Checks, which monitor request throughput, latency, and error rate during testing to determine whether your functions pass or fail a test. Gremlin does not use these metrics for telemetry, nor does it collect sensitive details. For more information on our security practices, see our security page.

Environment variable	Config file option	Description
`GREMLIN_METRICS_OPT_OUT`	`metrics_opt_out`	Set to `true` if you do not want the sidecar to collect metrics.
`GREMLIN_METRICS_EXCLUDE_HEALTH_CHECK_PATHS`	`metrics_exclude_health_check_paths`	Provide a comma-separated list of URL paths that you want to exclude from metrics collection. This is primarily for excluding paths used for Intelligent Health Checks.

‍

Setting common proxy options

These options apply to all proxy types:

Environment variable	Config file property	Description
`GREMLIN_PROXY_IDLE_TIMEOUT`	`proxy_idle_connection_timeout`	Sets the idle timeout for proxied connections. Defaults to 2m (two minutes).
`GREMLIN_PROXY_READ_TIMEOUT`	`proxy_read_timeout`	Sets the read timeout. Defaults to 2m (two minutes).
`GREMLIN_PROXY_WRITE_TIMEOUT`	`proxy_write_timeout`	Sets the write timeout. Defaults to 2m (two minutes).

‍

AWS Lambda metadata discovered by Gremlin

Gremlin automatically detects the following environment variables for Lambda functions:

Environment variable	Description
`AWS_LAMBDA_FUNCTION_NAME`	The name of the function.
`AWS_LAMBDA_FUNCTION_MEMORY_SIZE`	The amount of memory available to the function in MB.
`AWS_LAMBDA_FUNCTION_VERSION`	The version of the function being executed.
`AWS_LAMBDA_INITIALIZATION_TYPE`	The initialization type of the function, which is `on-demand`, `provisioned-concurrency`, `snap-start`, or `lambda-managed-instances`.
`AWS_LAMBDA_RUNTIME_API`	The host and port of the runtime API.
`AWS_LAMBDA_EXEC_WRAPPER`	The file system path of an executable binary or script that runs when the Lambda function starts.
`AWS_REGION`	The AWS Region where the Lambda function is executed.

‍

Using the Lambda proxy with other exec wrappers

AWS Lambda only lets a function have one exec wrapper (AWS_LAMBDA_EXEC_WRAPPER). Some other tools ship their own wrapper as a Lambda layer, such as Dynatrace (/opt/dynatrace) or OpenTelemetry (/opt/otel-handler). If your function uses one of these, you can still use the Lambda API proxy by chaining the wrappers together:

Set AWS_LAMBDA_EXEC_WRAPPER to /opt/bootstrap.
Set GREMLIN_LAMBDA_NEXT_EXEC_WRAPPER to the value AWS_LAMBDA_EXEC_WRAPPER had before.

When the function starts, the Gremlin bootstrap script points AWS_LAMBDA_RUNTIME_API at the Lambda proxy. It then runs the next wrapper and passes along the arguments Lambda gave it. The other wrapper and your function's runtime both talk to Lambda through the proxy, so Failure Flags experiments and the other tool both keep working.

Chaining more than one wrapper

To chain several wrappers, separate their paths with :, for example /opt/dynatrace:/opt/otel-handler. They run in the order listed.

Each wrapper except the last one gets the next wrapper's path as its first argument, instead of the path to the runtime. Most wrappers don't care, because they just run whatever arguments they get (exec "$@"). Some wrappers read or change the runtime's arguments, such as wrappers that add a flag after the runtime executable. Put that kind of wrapper last in the list.

Requirements and behavior

Every chained wrapper must exec the arguments it receives, as standard Lambda exec wrappers do.
A chained wrapper must not overwrite AWS_LAMBDA_RUNTIME_API. If it does, the runtime skips the Lambda proxy and Failure Flags by proxy won't trigger.
If a path in GREMLIN_LAMBDA_NEXT_EXEC_WRAPPER is missing, is not a regular file, or is not executable, the bootstrap script skips it and logs [gremlin] skipping exec wrapper '<path>': not an executable file. The function still starts, but without that wrapper. If a path points back at /opt/bootstrap, it's skipped too.

‍

Verifying your Failure Flags Lambda deployment

After deploying the extension, confirm that Gremlin can see your function before you try to run an experiment.

Invoke your function at least once. Gremlin creates Failure Flags from observed activity, so nothing appears until the function runs.
Confirm the service in Gremlin. Open Failure Flags in the web app. Your function appears under its SERVICE_NAME, or under its auto-detected AWS_LAMBDA_FUNCTION_NAME if you didn't set one.
Confirm the flags. You should see the process flag, plus flags for each proxy mode you enabled.

If the service or its flags don't appear, see Troubleshooting Failure Flags.

‍

Configuring Failure Flags

Deploying Failure Flags on AWS ECS