Installing Gremlin on Amazon ECS

Gremlin supports Amazon Elastic Container Service (ECS) using our container agent. This document will walk you through deploying the Gremlin container to your ECS deployment.

To get the most from this installation guide, you should also be familiar with installing Gremlin as a container. You can reference Installing Gremlin in a Docker Container for help.

‍

Step 1: Create the task definition

This task definition runs the Gremlin agent as a daemon container on each EC2 instance in your cluster. ECS reads your team ID, certificate and private key from AWS Secrets Manager and passes them to the agent as environment variables.

‍

Store your Gremlin credentials in Secrets Manager

1. In the Gremlin web app, open Team Settings and select Details. Copy your Team ID.
2. On the Certificates line, click Download (see Download a certificate pair). Unzip the file. The certificate is named <team-name>-client.pub_cert.pem and the private key is <team-name>-client.priv_key.pem.
3. Create a Secrets Manager secret with three keys: team_id, team_certificate and team_private_key. The certificate and key values must hold the full PEM text, including the BEGIN and END lines and the line breaks between them. The commands below build the secret from your files and keep the line breaks intact. They require the AWS CLI and jq 1.6 or later. Set GREMLIN_TEAM_NAME to the name of your Gremlin team.

Copy the ARN value from the command's output. It ends with a six-character suffix that AWS adds to the secret name, such as gremlin/ecs-agent-AbCdEf. You'll need the full ARN in the next step.

‍

Fill in the task definition

Copy the JSON task definition below into a text editor. Replace each placeholder, including its leading $, using the table that follows it.

To let the execution role read your secret, add this inline policy to it:

To use a team secret instead of a certificate, store a team_secret key in the secret. Then replace the two certificate entries in secrets with one entry named GREMLIN_TEAM_SECRET whose valueFrom ends in :team_secret::.

1. Review the memoryReservation value against your instance sizes.
2. In the AWS Management Console, navigate to Task Definitions and choose Create new task definition with JSON.
3. Paste your edited task definition into the JSON field, then click Create.

‍

Step 2: Create the daemon service definition

1. In the AWS Management Console, navigate to Clusters in ECS.
2. Select the cluster you want to deploy Gremlin to.
3. On the Services tab, click Create.

On the Configure Services page, set the parameters as follows:

1. Select the Launch Type compute option.
2. Select the EC2 launch type.
3. Select the Service application type.
4. Set Task Definition → Family to the family you created in Step 1 (gremlin-<your identifier>).
5. Set Task Definition → Revision to “latest”.
6. Set Service Type to DAEMON.
7. Click Create to create the service.

‍

Verifying the installation

To verify that Gremlin is properly installed and running:

1. In the AWS management Console, navigate to Clusters.
2. Select the cluster you just deployed Gremlin into.
3. On the Services tab, you should now see the Gremlin service.
4. Verify that Desired tasks matches the number of ECS hosts in your cluster
5. Verify that Running tasks matches the number of Desired tasks. Note that it can take several minutes for the ECS scheduler to launch Gremlin to full capacity.
6. Once the Gremlin service is running at full capacity, navigate to https://app.gremlin.com/configurations/agents. You can search via the tag platform=ecs to verify that the Gremlin control plane can see the freshly launched ECS daemons.
7. Navigate to https://app.gremlin.com/attacks/new and click Containers.
8. Verify that you are seeing the application containers and tags currently running on your ECS cluster.

Once your ECS agents show as Active, run your first experiment against them. See How to run an experiment.

‍

Additional ECS configuration options

There are some advanced options that change how Gremlin interacts with ECS:

• networkMode: This option determines which network space we would like to affect. For example, setting it to awsvpc means the task can only affect the awsvpc interface. Some other options are: host, bridge, or none. By default, this value is set to host. For more information, please consult the AWS guide on network mode.
• pidMode: This parameter allows you to configure the container to share their process ID with either the host or other containers in the task. It may prove useful when performing process killer attacks to set this parameter to host. By default, this value is set to host. For more information, please consult the AWS guide on PID mode.