How to use config values stored in AWS

Zach Nofzinger
Software Engineer
Last Updated:
December 17, 2023
This tutorial will walk through how to create and use values stored in AWS to configure the Gremlin agent running on your machines.


This tutorial will provide a walkthrough on setting up and using configuration values located in AWS for configuration in your Gremlin agent installation.


Before you begin this tutorial, you'll need the following:

Step 1a - Create config value in AWS SSM

To create a parameter in AWS SSM:

  1. Navigate to the SSM console (us-west-2 example).
  2. Navigate to 'Parameter Store' on the left side and click 'Create parameter'.
  3. Create a name for your parameter, and set the type to be either 'String' or 'SecureString', then enter your config value in the value field. When satisifed with the settings, create the parameter.

Step 1b - Create config value in AWS Secrets Manager

To create a secret in AWS Secrets Manager:

  1. Navigate to the Secrets Manager console (us-west-2 example).
  2. Click on "Store a new secret" on the right. When creating a secret, use the "Other type of secret" option, and use the "Plaintext" tab to enter the raw values you wish to use.
    • If you wish, you can use a custom KMS key to encrypt the secret.
  3. Click 'Next' and set the name and any optional fields, and keep going until the secret is created.

Step 2 - Setup permissions for credentials

Whatever credentials you will have the Gremlin agent use will need the corresponding IAM permissions associated with them.

  • For SSM parameters, the <span class="code-class-custom">ssm:GetParameter</span> action on the parameter resource is required.
  • For Secrets Manager secrets, the <span class="code-class-custom">secretsmanager:GetSecretValue</span> action on the secret resource is required.
  • If any of these values are encrypted with a KMS key, the <span class="code-class-custom">kms:Decrypt</span> action is required on the KMS key resource.

Step 3 - Set Gremlin config values to use ARN

Now in your gremlin configuration, you can set some of the configuration values to use the AWS ARN of the AWS resource you created. When the agent starts, it will reach out to AWS to retrieve the value stored there, and keeps it in memory. Here is an example configuration file with AWS ARNs used to store sensitive values:


## Gremlin Identifier
identifier: arn:aws:ssm:us-west-2:123412341234:parameter/gremlin-identifier

## Gremlin Team Id
team_id: arn:aws:ssm:us-west-2:123412341234:parameter/gremlin-team-id

## Gremlin Team Secret
#team_secret: arn:aws:secretsmanager:us-west-2:123412341234:secret:gremlin/team-secret-djfgdjkf

## Gremlin Team Certificate
team_certificate: arn:aws:secretsmanager:us-west-2:123412341234:secret:gremlin/team-cert-ovbndf

## Gremlin Team Certificate
team_private_key: arn:aws:secretsmanager:us-west-2:123412341234:secret:gremlin/team-key-asdgiovb

Note: When using the Gremlin Helm Chart, you can pass these values like you would with raw secret values (requires chart version 0.12.0 or later):


helm install gremlin gremlin/gremlin \
     --namespace gremlin \
     --set      gremlin.hostPID=true \
     --set      gremlin.hostNetwork=true \
     --set      gremlin.collect.processes=true \
     --set      gremlin.container.driver=any \
     --set      gremlin.secret.managed=true \
     --set      gremlin.secret.type=certificate \
     --set      gremlin.secret.clusterID=${GREMLIN_CLUSTER_ID} \
     --set      gremlin.secret.teamID=${GREMLIN_TEAM_ID} \
     --set      gremlin.serviceUrl=${GREMLIN_SERVICE_URL} \
     --set      gremlin.secret.certificate=arn:aws:secretsmanager:us-west-2:123412341234:secret:gremlin/team-cert-ovbndf \
     --set      gremlin.secret.key=arn:aws:secretsmanager:us-west-2:123412341234:secret:gremlin/team-key-asdgiovb \
     --set      gremlin.serviceAccount.annotations."eks\.amazonaws\.com\/role-arn"="arn:aws:iam::123412341234:role/K8sServiceAccountRole" \
     --set      chao.serviceAccount.annotations."eks\.amazonaws\.com\/role-arn"="arn:aws:iam::123412341234:role/ChaoK8sServiceAccountRole"


You've setup your Gremlin agent to use remote configuration values, increasing the security of your configuration! Refer to the agent configuration to read all the supported values you can use an AWS ARN for.

No items found.
Gremlin's automated reliability platform empowers you to find and fix availability risks before they impact your users. Start finding hidden risks in your systems with a free 30 day trial.
start your trial

Avoid downtime. Use Gremlin to turn failure into resilience.

Gremlin empowers you to proactively root out failure before it causes downtime. See how you can harness chaos to build resilient systems by requesting a demo of Gremlin.GET STARTED

Product Hero ImageShape