Dashboard
Failure Flags

Deploying Failure Flags on AWS Lambda

This document will walk you through setting up the Failure Flags agent for Lambda Functions. The Failure Flags agent runs alongside your Function and is responsible for connecting your Function with Gremlin.

Note
The Failure Flags agents are not in critical path for your application logic or network. They are never exposed to sensitive customer data (encrypted or otherwise). They do not act as network proxies. They do periodically reach out to Gremlin to determine if there are any experiments targeting the attached application and cache those results for a short time.

Installing the Gremlin-Lambda agent on AWS Lambda

Gremlin-Lambda is a Lambda Extension that you can add to your Lambda Functions. Gremlin-Lambda supports both AMD64/x86_64 and ARM64 architectures. You can learn more about the AWS Lambda Extensions API in the AWS documentation.

Setting required environment variables

You can add the Gremlin-Lambda Extension to any Lambda Function without impacting your function availability or performance. But you do need to add configuration to your environment variables before Gremlin-Lambda will add any value. Configuration comes in via environment variables and or configuration files.

Get started quickly with environment variables only:

  1. FAILURE_FLAGS_ENABLED must be set to either true or yes or 1 to enable the Failure Flags SDK in your application.
  2. GREMLIN_LAMBDA_ENABLED must be set to either true or yes or 1 to enable Gremlin-Lambda. If unset or set to any other value Gremlin-Lambda will operate in NOOP mode.
  3. GREMLIN_TEAM_ID must be set to your Gremlin Team ID. This and other credential material is available through the Gremlin UI.
  4. GREMLIN_TEAM_CERTIFICATE must be set to your Gremlin Team certificate. Newlines may be preserved using the \n escape characters or omitted entirely. This and other credential material is available through the Gremlin UI.
  5. GREMLIN_TEAM_PRIVATE_KEY must be set to your Gremlin Team private key. Newlines may be preserved using the \n escape characters or omitted entirely. This and other credential material is available through the Gremlin UI.

If you want to use configuration files, you'll use environment variables to specify the configuration file:

  1. FAILURE_FLAGS_ENABLED must be set to either true or yes or 1 to enable the Failure Flags SDK in your application.
  2. GREMLIN_LAMBDA_ENABLED must be set to either true or yes or 1 to enable Gremlin-Lambda. If unset or set to any other value Gremlin-Lambda will operate in NOOP mode.
  3. GREMLIN_CONFIG_FILE must be set to the fully-qualified path of a file containing a Gremlin-Lambda configuration file.

Alternatively, you can load configuration files from AWS resources like secretsmanager secrets and ssm parameters:

  1. FAILURE_FLAGS_ENABLED must be set to either true or yes or 1 to enable the Failure Flags SDK in your application.
  2. GREMLIN_LAMBDA_ENABLED must be set to either true or yes or 1 to enable Gremlin-Lambda. If unset or set to any other value Gremlin-Lambda will operate in NOOP mode.
  3. GREMLIN_CONFIG_ARN must be set to an ARN of a resource containing a Gremlin-Lambda configuration file.
  4. GREMLIN_CONFIG_ROLE must be set to an ARN of a AWS IAM role to use when retrieving the configuration file.

Individual Configuration Values from Files or ARNs

You can configure individual configuration values like GREMLIN_TEAM_CERTIFICATE, GREMLIN_TEAM_PRIVATE_KEY, and GREMLIN_CUSTOM_ROOT_CERTIFICATE to retrieve values from files in your Lambda function or from AWS services using their ARNs. Instead of setting those environment values directly, use their _FILE or _ARN counterparts. Files must be fully qualified paths from the root of the resulting Lambda Function file system (typically something like /var/task/<path from your code root>). This project currently supports secretsmanager secret and ssm parameter ARNs.

When you add the Gremlin-Lambda Extension to your Lambda Function and configure the environment variables correctly, your application will be able to consult that extension for Gremlin experiment configuration. You will be able to find your Function in the Gremlin UI under Failure Flags > Services after you launch your app with the layer configured and you exercise the function.

Once you've added Gremlin-Lambda to your project you can use the Failure Flags library (Node, Python, Java, Go) from your Function code!

Creating a Configuration File with Examples

Gremlin-Lambda uses YAML for file-based configuration. These files contain sensitive credential material and should be stored appropriately. The following is a brief description of each property:

  1. team_id
  2. team_certificate
  3. team_private_key
  4. labels
  5. https_proxy
  6. ssl_cert
  7. debug
  8. api_endpoint_url
YAML
## Gremlin Team Id - you can find this value at https://app.gremlin.com/settings/teams
team_id: ffffffff-ffff-ffff-ffff-ffffffffffff

## Failure Flags Service Labels - Add labels to identify unique deployments.
labels:
    datacenter: corp-na1
    project: columbia

## Debug, set to true for enhanced debug logging to STDOUT
## Uncomment to enable debugging
#debug: true

## Gremlin Team Certificate - Paste certificate content here.
team_certificate: |
    -----BEGIN CERTIFICATE-----
    ExampleXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
    XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
    XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
    XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
    XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
    XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
    XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
    XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
    XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
    XXXXXXXX
    -----END CERTIFICATE-----

## Gremlin Team Certificate - Paste certificate content here.
team_private_key: |
    -----BEGIN EC PRIVATE KEY-----
    ExampleXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
    XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
    XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX==
    -----END EC PRIVATE KEY-----

## HTTPS Proxy, set this when routing outbound Gremlin HTTPS traffic through a proxy
#https_proxy: https://corp.proxy.internal:3128

## Custom CA Certificate, set this when using a https proxy with a self-signed certificate
## Paste certificate content here.
#ssl_cert: |
#    -----BEGIN CERTIFICATE-----
#    ExampleXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
#    XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
#    XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
#    XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
#    XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
#    XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
#    XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
#    XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
#    XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
#    XXXXXXXX
#    -----END CERTIFICATE-----

## API ENDPOINT, set this to the Gremlin API endpoint you want to interact with. The beta
## endpoint is https://beta.gremlin.com/v1. The GA endpoint is at https://api.gremlin.com/v1/ff.
#api_endpoint_url: https://beta.gremlin.com/v1

Adding the Gremlin-Lambda Extension to your Lambda Function

The Extension's Amazon Resource Name (ARN) varies depending on where your Lambda Function is deployed and which architecture you're using. Retrieve the correct ARN for your architecture and AWS region from this table:

Region x86_64 Arm64
ap-northeast-3 arn:aws:lambda:ap-northeast-3:044815399860:layer:gremlin-lambda-x86_64:13 arn:aws:lambda:ap-northeast-3:044815399860:layer:gremlin-lambda-arm64:13
ap-northeast-2 arn:aws:lambda:ap-northeast-2:044815399860:layer:gremlin-lambda-x86_64:13 arn:aws:lambda:ap-northeast-2:044815399860:layer:gremlin-lambda-arm64:13
ap-northeast-1 arn:aws:lambda:ap-northeast-1:044815399860:layer:gremlin-lambda-x86_64:13 arn:aws:lambda:ap-northeast-1:044815399860:layer:gremlin-lambda-arm64:13
ap-south-1 arn:aws:lambda:ap-south-1:044815399860:layer:gremlin-lambda-x86_64:13 arn:aws:lambda:ap-south-1:044815399860:layer:gremlin-lambda-arm64:13
ap-southeast-1 arn:aws:lambda:ap-southeast-1:044815399860:layer:gremlin-lambda-x86_64:13 arn:aws:lambda:ap-southeast-1:044815399860:layer:gremlin-lambda-arm64:13
ap-southeast-2 arn:aws:lambda:ap-southeast-2:044815399860:layer:gremlin-lambda-x86_64:13 arn:aws:lambda:ap-southeast-2:044815399860:layer:gremlin-lambda-arm64:13
ca-central-1 arn:aws:lambda:ca-central-1:044815399860:layer:gremlin-lambda-x86_64:13 arn:aws:lambda:ca-central-1:044815399860:layer:gremlin-lambda-arm64:13
eu-north-1 arn:aws:lambda:eu-north-1:044815399860:layer:gremlin-lambda-x86_64:13 arn:aws:lambda:eu-north-1:044815399860:layer:gremlin-lambda-arm64:13
eu-west-3 arn:aws:lambda:eu-west-3:044815399860:layer:gremlin-lambda-x86_64:13 arn:aws:lambda:eu-west-3:044815399860:layer:gremlin-lambda-arm64:13
eu-west-2 arn:aws:lambda:eu-west-2:044815399860:layer:gremlin-lambda-x86_64:13 arn:aws:lambda:eu-west-2:044815399860:layer:gremlin-lambda-arm64:13
eu-west-1 arn:aws:lambda:eu-west-1:044815399860:layer:gremlin-lambda-x86_64:13 arn:aws:lambda:eu-west-1:044815399860:layer:gremlin-lambda-arm64:13
eu-central-1 arn:aws:lambda:eu-central-1:044815399860:layer:gremlin-lambda-x86_64:13 arn:aws:lambda:eu-central-1:044815399860:layer:gremlin-lambda-arm64:13
sa-east-1 arn:aws:lambda:sa-east-1:044815399860:layer:gremlin-lambda-x86_64:13 arn:aws:lambda:sa-east-1:044815399860:layer:gremlin-lambda-arm64:13
us-east-1 arn:aws:lambda:us-east-1:044815399860:layer:gremlin-lambda-x86_64:13 arn:aws:lambda:us-east-1:044815399860:layer:gremlin-lambda-arm64:13
us-east-2 arn:aws:lambda:us-east-2:044815399860:layer:gremlin-lambda-x86_64:13 arn:aws:lambda:us-east-1:044815399860:layer:gremlin-lambda-arm64:13
us-west-1 arn:aws:lambda:us-west-1:044815399860:layer:gremlin-lambda-x86_64:13 arn:aws:lambda:us-west-1:044815399860:layer:gremlin-lambda-arm64:13
us-west-2 arn:aws:lambda:us-west-2:044815399860:layer:gremlin-lambda-x86_64:13 arn:aws:lambda:us-west-1:044815399860:layer:gremlin-lambda-arm64:13

Once you have the correct ARN, add the extension to your Lambda Function by following the instructions in the AWS Lambda Developer Guide.

Note
Lambda extensions can impact the size and performance of your function(s). Gremlin-Lambda's impact is minimal, but non-zero. See the AWS documentation on Lambda extensions for details.


Building your own Lambda Layer

You may want to build your own Lambda Layer or incorporate the gremlin-lambda binary in a Lambda container image. Maybe you're at risk of using too many layers, or maybe your organization has a package caching policy. Either way you'll want to build your own asset. You can access all of the gremlin-lambda and failure-flags-sidecar builds via our website.

Files provided via assets.gremlin.com have the following format:

https://assets.gremlin.com/packages/gremlin-lambda/{VERSION}/{ARCH}/{FILE}

You can reference the LATEST release at:

https://assets.gremlin.com/packages/gremlin-lambda/latest/{ARCH}/{FILE}

ARCH is one of the following:

  • x86_64
  • arm64

Different releases may include different files, but the current file set includes:

  • gremlin-lambda-linux.tar.gz
  • layer-gremlin-lambda.zip
  • LICENSE

All versions are listed in a file at: https://assets.gremlin.com/packages/gremlin-lambda/VERSIONS

No items found.
Previous
Next
Previous
This is some text inside of a div block.
Compatibility
Installing the Gremlin Agent
Authenticating the Gremlin Agent
Configuring the Gremlin Agent
Managing the Gremlin Agent
User Management
Integrations
Health Checks
Notifications
Command Line Interface
Updating Gremlin
Reliability Management (RM) Quick Start Guide
Services and Dependencies
Detected Risks
Reliability Tests
Reliability Score
Targets
Experiments
Scenarios
GameDays
Overview
Deploying Failure Flags on AWS Lambda
Deploying Failure Flags on AWS ECS
Deploying Failure Flags on Kubernetes
Classes, methods, & attributes
API Keys
Examples
Container security
General
Linux
Windows
Chao
Helm
Glossary
Additional Configuration for Helm
Amazon CloudWatch Health Check
AppDynamics Health Check
Blackhole Experiment
CPU Experiment
Certificate Expiry
Custom Health Check
Custom Load Generator
DNS Experiment
Datadog Health Check
Disk Experiment
Dynatrace Health Check
Grafana Cloud Health Check
Grafana Cloud K6
IO Experiment
Install Gremlin on Kubernetes manually
Install Gremlin on OpenShift 4
Installing Gremlin on AWS - Configuring your VPC
Installing Gremlin on Kubernetes with Helm
Installing Gremlin on Windows
Installing Gremlin on a virtual machine
Installing the Failure Flags SDK
Jira
Latency Experiment
Memory Experiment
Network Tags
New Relic Health Check
Overview
Overview
Overview
Overview
Overview
Packet Loss Attack
PagerDuty Health Check
Preview: Gremlin in Kubernetes Restricted Networks
Private Network Integration Agent
Process Collection
Process Killer Experiment
Prometheus Health Check
Configuring Role Based Access Control (RBAC)
Running Failure Flags experiments
Scheduling Scenarios
Shared Scenarios
Shutdown Experiment
Slack
Managing Teams
Time Travel Experiment
Troubleshooting Gremlin on OpenShift
User Authentication via SAML and Okta
Managing Users
Webhooks
Integration Agent for Linux
Test Suites
Restricting Testing Times
Reports
Process Exhaustion Experiment
Enabling DNS collection
Authenticating Users with Microsoft Entra ID (Azure Active Directory) via SAML
AWS Quick Start Guide