How to run a Chaos Engineering experiment on AWS Lambda using Golang and Failure Flags
In this tutorial, you’ll learn how to run a Chaos Engineering experiment on a Go application running on AWS Lambda using Failure Flags. Failure Flags is a Gremlin feature that lets you inject faults into applications and services running on fully managed serverless environments, such as AWS Lambda, Azure Functions, and Google Cloud Functions. With Failure Flags, you can:
- Add latency or errors to applications.
- Inject data into function calls without having to edit or re-deploy source code.
- Simulate partial outages behind API gateways or reverse proxies.
- Customize the behavior and impact of experiments.
This tutorial will focus on testing a Go application on AWS Lambda. You can learn about our other supported languages and platforms in the Failure Flags documentation.
Overview
This tutorial will show you how to:
- Install the Failure Flags Go SDK.
- Deploy an application and the Failure Flags agent to AWS Lambda.
- Run a latency experiment using Failure Flags.
Prerequisites
Before starting this tutorial, you’ll need:
- A Gremlin account (sign up for a free trial here).
- An AWS account with access to Lambda (you can use the lowest-tier x86 or Arm instance for this tutorial to save on costs).
- Go installed on your local machine. This tutorial was written using Go version 1.22.1.
Step 1 - Set up your Go application with Failure Flags
In this step, we’ll create a Go application and add a Failure Flag. This is a simple application that responds to HTTP requests with the current timestamp and the time taken to process the response.
First, initialize a new module. You can give it any name you’d like, or use the one provided here:
This module depends on two other modules: the AWS Lambda Go library, and of course, the Failure Flags Go library. Add them by opening the newly created go.mod
file and adding the following text:
Next, create a new file named main.go
. This will contain our application code. In this file, add the following code:
To break this down: this is a simple Lambda Go function that receives incoming HTTP requests, calculates the time taken to process the request, then responds with the processing time. The Failure Flag is declared on line 16, with gremlin.Invoke(gremlin.FailureFlag{})
. We set the name of this Failure Flag to “http-ingress-go”. We also pass the request method and path to the Failure Flag: this enables us to fine-tune our experiments to only impact requests matching specific patterns.
Now, we need to download the Lambda and Failure Flags Go modules. Run the following commands in your terminal:
Lastly, build the executable that we’ll deploy to Lambda. The executable will be named bootstrap
:
Step 2 - Download your client configuration file
Before you can deploy your application, you’ll need to authenticate it with Gremlin. Gremlin provides a downloadable client configuration file that you can use to authenticate any Gremlin agent, including Failure Flags agents. This file contains your Gremlin team ID and TLS certificates, but you can add additional labels like your application name, version number, region, etc.
- Download your client configuration file from the Gremlin web app and save it in the root directory of your project folder as
config.yaml
. - Optionally, add any labels to your configuration file. You can use these labels to identify unique deployments of this application, letting you fine-tune which deployments to impact during experiments. For example, you could add the following block to identify your function as being part of the
us-east-2
region and thefailure-flags-go
project, letting you target all functions running inus-east-2
or that belong to thefailure-flags-go
project:
The configuration file supports other options, but the defaults are all you need for this tutorial.
Step 3 - Deploy your Go application to Lambda
Next, let’s deploy our app to Lambda.
So far, we’ve configured our application and the Failure Flags SDK. The SDK is what’s responsible for injecting faults into our app, but it doesn’t handle communicating with Gremlin’s backend servers or orchestrating experiments. For that, we need the Failure Flags Lambda layer.
We’ll deploy the Lambda layer alongside our Go app. The specifics of deploying an app to Lambda go beyond the scope of this tutorial, so we’ll link to the AWS docs where necessary.
- Create a ZIP file containing your
bootstrap
andconfig.yaml
files. - Follow the instructions in Deploy Go Lambda functions with .zip file archives. Remember to download your Gremlin client configuration file to your project folder!
- Create a new Lambda function using the instructions in Creating and updating functions with .zip files using the console. For the runtime, select “Amazon Linux 2023” (or a similar OS-only runtime).
- Before deploying the function, we need to add some environment variables. These are necessary for enabling Failure Flags. In the AWS Console, select the Configuration tab, then select Environment Variables. Click Edit, then enter the following variables:some text
FAILURE_FLAGS_ENABLED=1
GREMLIN_LAMBDA_ENABLED=1
GREMLIN_CONFIG_FILE=/var/task/config.yaml
- Optionally, you can provide additional metadata via environment variables, including your Gremlin team credentials. This isn't necessary for this tutorial, since we're authenticating via a config file. See the Failure Flags installation docs for details.
- Click Test to confirm that your application can receive and process requests correctly.
- Now we need to add the Failure Flags Lambda layer. Select the Code tab, then scroll down to Layers and click Add a layer:some text
- Under Choose a layer, select Specify an ARN.
- Enter one of the ARNs presented in this link, depending on which region and architecture your function is running on. For example, if your Lambda is running in us-east-2 on x86, enter
arn:aws:lambda:us-east-2:044815399860:layer:gremlin-lambda-x86_64:13
- Click Verify to confirm that the ARN matches your region and architecture, then click Add.
- Publish your Lambda by scrolling to the top of the page, clicking Actions, then clicking Publish new version. Enter a name for this version, then click Publish to push your function live.
- Create a new Function URL by following the instructions in Creating and managing Lambda function URLs. Once the URL is created, click on the link to see your function's output in a new tab. It should look similar to the following:
Step 4 - Run an experiment
Now, let’s run an experiment on our Go application.
To set the context: Lambda functions are susceptible to changes in network throughput. In other words, if our network connection becomes congested or slow, we’ll likely see less throughput and slower response times. The question is: how does this added latency impact the overall performance of our Lambda function, as well as any other services that depend on it? To test this, we’ll run a latency experiment, add one full second of latency, and observe what happens to our application.
- In the Gremlin web app, select Failure Flags in the navigation pane (or click this link).
- Click + Experiment to create a new experiment.
- Enter a name for the new experiment.
- Under Failure Flag Selector, click the combo box to show a list of active applications with Failure Flags that Gremlin detected. If your app doesn't show up, confirm that it's finished deploying on Lambda and has responded to at least one request.
- Optionally, you can add any additional attributes, such as label selectors, in the Attributes box. You can ignore this field for this tutorial.
- In the Effects box, specify the impact that you want to have on your app. For example, say we want to add 1000 ms (one second) of latency to each call to this function. We can do this by adding the following JSON to this field:some text
{ "latency": 1000 }
- Set the Impact Probability percentage. For now, set it to 100% to ensure that every call to this function gets impacted.
- Optionally, change the Experiment Duration to your preferred time. For now, set it to 5 min so you have plenty of time to observe the impact. You can always stop the experiment using Gremlin's Halt button.
- Click Save & Run to start the experiment.
While the experiment runs, open your Lambda URL in a web browser or a performance testing tool. You should see the fields reflect the running experiment:
Consider how this might impact a real-world application. How is your function responding? Is the amount of latency more than you expected (longer than one second)? If so, why do you think that is? Imagine that this function is a dependency of another function: how might you rearchitect your other function so that it can tolerate latency in this function?
When you're finished making observations and want to stop the experiment, simply click Halt this experiment in the Gremlin web app to stop the experiment.
Conclusion
Congratulations on running a serverless Chaos Engineering experiment on AWS Lambda with Gremlin! Now that you have Failure Flags set up, try running different kinds of experiments. Add jitter to your network latency, impact a larger or smaller percentage of traffic, generate exceptions, or perform a combination of effects. For more advanced tests, you can even define your own experiments or inject data into your app. Failure Flags can also trigger panics, a feature that’s specific to Go.
If you'd like to try Failure Flags outside Lambda, we also have a sidecar for Kubernetes. Just deploy the sidecar, then define and run your experiment. Remember that Failure Flags has no performance or availability impacts on your application when not in use, so don't be afraid to add it to your applications. We also offer SDKs for Node.js, Java, and Python. These are all available on Github under an Apache-2.0 license.
Avoid downtime. Use Gremlin to turn failure into resilience.
Gremlin empowers you to proactively root out failure before it causes downtime. See how you can harness chaos to build resilient systems by requesting a demo of Gremlin.