Failure Flags > Deploying Failure Flags on AWS Lambda

Deploying Failure Flags on AWS Lambda

This document will walk you through setting up Failure Flags for AWS Lambda Functions. On Lambda, Failure Flags is deployed as a Lambda Extension called Gremlin-Lambda. This extension creates a sidecar container that runs alongside your function. Gremlin-Lambda supports both AMD64/x86_64 and ARM64 architectures. You can learn more about the AWS Lambda Extensions API in the AWS documentation.

‍

Configuring Failure Flags for AWS Lambda

The Gremlin-Lambda Extension accepts the following environment variables in addition to those defined in Configuring Failure Flags. When referencing files, the filename must be a fully qualified path from the root of the resulting Lambda Function file system (typically /var/task):

Environment variable	Description
`GREMLIN_LAMBDA_ENABLED`	Set to `true`, `yes`, or `1` to enable the Gremlin Lambda extension.
`FAILURE_FLAGS_ENABLED`	Set to `true`, `yes`, or `1`to enable the Failure Flags SDK in your application. If you’re using Failure Flags by proxy, you don't need to set this variable.
`GREMLIN_METRICS_EXCLUDE_HEALTH_CHECK_PATHS`	A comma-separated list of URL paths that you want to exclude from metrics collection. This is primarily for excluding paths used for Health Checks.
`GREMLIN_METRICS_OPT_OUT`	Set to `true` if you do not want the sidecar to collect application metrics (metrics are used for Intelligent Health Checks.
`GREMLIN_SIDECAR_ENABLED`	Set to `true`, `yes`, or `1`to enable the Failure Flags sidecar. If this is unset or set to any other value, the sidecar will operate in `NOOP` mode.
`GREMLIN_TEAM_ID`	Your Gremlin Team ID. This and other credential details are available in the Gremlin web app.
`GREMLIN_TEAM_CERTIFICATE`	Set to the contents of your Gremlin Team certificate. This and other credential details are available in the Gremlin web app. Preserve newlines by adding the `\n`escape character at the end of each line, or by omitting them entirely.
`GREMLIN_TEAM_PRIVATE_KEY`	Set to the contents or file path of your Gremlin Team private key. This and other credential details are available in the Gremlin web app. Preserve newlines by adding the `\n`escape character at the end of each line, or by omitting them entirely.
`SERVICE_NAME`	Set the name that you want to register the service as in Gremlin. Service names can only contain alphanumeric characters, hyphens, and underscores, and must be less than 64 characters long. If not set, Gremlin will auto-detect the service name from Lambda.

‍

AWS Lambda metadata discovered by Gremlin

Gremlin automatically detects the following environment variables for Lambda functions:

Environment variable	Description
`AWS_LAMBDA_FUNCTION_NAME`	The name of the function.
`AWS_LAMBDA_FUNCTION_MEMORY_SIZE`	The amount of memory available to the function in MB.
`AWS_LAMBDA_FUNCTION_VERSION`	The version of the function being executed.
`AWS_LAMBDA_INITIALIZATION_TYPE`	The initialization type of the function, which is `on-demand`, `provisioned-concurrency`, `snap-start`, or `lambda-managed-instances`.
`AWS_LAMBDA_RUNTIME_API`	The host and port of the runtime API.
`AWS_LAMBDA_EXEC_WRAPPER`	The file system path of an executable binary or script that runs when the Lambda function starts.
`AWS_REGION`	The AWS Region where the Lambda function is executed.

‍

Adding the Gremlin-Lambda Extension to your Lambda Function

Gremlin releases a new version of the Failure Flags Lambda layer with each sidecar release. This means the Lambda layer version (indicated by the number at the end of the ARN) will change with each release. We’ll include the corresponding Lambda layer number for each sidecar release in the Failure Flags sidecar release notes.

If the layer version shown below is older than the most recent entry in the Failure Flags sidecar release notes, use the version from the release notes.

‍

The Extension's Amazon Resource Name (ARN) varies depending on where your Lambda Function is deployed and which architecture you're using. First, retrieve the following values:

The current Lambda layer version (29)
The Lambda layer ARN ID (044815399860)
The region where your Lambda function is deployed (e.g. us-east-1)
Your Lambda layer architecture (x86_64 or arm64)

Next, assemble the ARN by combining arn:aws:lambda:region:arnID:layer:gremlin-lambda-v2-architecture:version.

For example, to run version 29 of the Gremlin Lambda layer on an x86_64 function in ap-northeast-1, use the ARN arn:aws:lambda:ap-northeast-1:044815399860:layer:gremlin-lambda-v2-x86_64:29.

‍

Once you have the correct ARN, add the extension to your Lambda Function by following the instructions in the AWS Lambda Developer Guide.

Lambda extensions can impact the size and performance of your function(s). Gremlin-Lambda's impact is minimal, but non-zero. See the AWS documentation on Lambda extensions for details.

‍

Building your own Lambda Layer

You may want to build your own Lambda Layer or incorporate the gremlin-lambda binary in a Lambda container image. Maybe you're at risk of using too many layers, or maybe your organization has a package caching policy. Either way you'll want to build your own asset. You can access all of the gremlin-lambda and failure-flags-sidecar builds via our website.

Files provided via assets.gremlin.com have the following format: https://assets.gremlin.com/packages/gremlin-lambda/{VERSION}/{ARCH}/{FILE}

You can reference the LATEST release at: https://assets.gremlin.com/packages/gremlin-lambda/latest/{ARCH}/{FILE}

ARCH is one of the following:

x86_64
arm64

Different releases may include different files, but the current FILE set includes:

gremlin-lambda-linux.tar.gz (x86_64)
layer-gremlin-lambda.zip
LICENSE

You can find a list of releases in our release notes, or in our VERSIONS file.

‍

Verifying your Failure Flags Lambda deployment

After deploying the extension, confirm that Gremlin can see your function before you try to run an experiment.

Enable debug logging. Set GREMLIN_DEBUG to true on your function. The extension writes to your function's CloudWatch log group, prefixed with [gremlin-lambda].
Invoke your function at least once. Gremlin creates Failure Flags from observed activity, so nothing appears until the function runs.
Confirm the service in Gremlin. Open Failure Flags in the web app. Your function appears under its SERVICE_NAME, or under its auto-detected AWS_LAMBDA_FUNCTION_NAME if you didn't set one.
Confirm the flags. You should see the process flag, plus flags for each proxy mode you enabled.

If the service or its flags don't appear, see Troubleshooting Failure Flags.

‍

Troubleshooting Failure Flags

This Lambda Extension provides debug logging when the GREMLIN_DEBUG environment variable is set to true . This information will be included with your Lambda logs and prefixed with [gremlin-lambda]. This additional information will include configuration details, registration status, connection tracing, any relevant errors encountered while interacting with the control plane, and requests for experiments from the Failure Flags SDKs.

‍

Configuring Failure Flags

Deploying Failure Flags on AWS ECS