How to install Gremlin on ECS

Philip Gebhardt

Software Engineer

Last Updated: August 28, 2023

Categories: Chaos Engineering

This advanced installation guide will walk you through installing Gremlin docker containers in your ECS environment, and verifying that you can run a CPU attack against the freshly installed Gremlin agents. In the verification steps we will be creating a container to run htop exposed as a web interface via port 8888, which will allow us to visualize changes in real time as a simple CPU attack is run against the container.

Prerequisites

• Functional ECS cluster, built in the region of your choice, utilizing EC2 backing instances. Fargate backed ECS is not currently supported.
• Private Subnet's in the ECS VPC that route through a NAT-GW. Gremlin will be deployed in those Private Subnet's
• Certificate based authentication should be used, and the certificates should be made available to the Gremlin daemon in /var/lib/gremlin.

Additionally, to get the most from this installation guide you should already be familiar with running Gremlin as a container. You can reference Install Gremlin in a Docker Container for help getting started with with Gremlin and Docker.

Step 1: Create the Task Definition

• Copy the provided JSON task definition into a text editor and supply vaulues for my-team-id, my-team-secret, my-aws-account. Additionally, review the Task Definition's limits and CPU architecture values to ensure they match your target environment.
• In the AWS management console navigate to Task Definitions the ECS service, and choose Create New Task Definition
• Select EC2 for the launch type compatibility and click Next Step
• Scroll down to the bottom of the page and click the button Configure via JSON
• Paste the edited task definition into the JSON text field and click the Save button

Step 2: Create the Daemon Service Definition

• In the AWS management console navigate to Clusters in the ECS service
• Select the cluster you want to deploy Gremlin into
• On the Services tab, click the Create button

On the Configure service page, set the parameters as follows:

• Select the Launch Type compute option.
• Select the EC2 launch type.
• Select the Service application tpye.
• Task Definition -> Family: gremlin
• Task Definition -> Revision: latest
• Service type: DAEMON
• The rest of the defaults are acceptable, click Create

Step 3: Verify the Installation

• In the AWS management console navigate to the Clusters in the ECS service
• Select the cluster you just deployed Gremlin into
• On the Services tab, you should now see the Gremlin service
• Verify that Desired tasks matches the number of ECS hosts in your cluster
• Verify that Running tasks matches the number of Desired tasks. Note that it can take several minutes for the ECS scheduler to launch Gremlin to full capacity
• Once the Gremlin service is running at full capacity, navigate to https://app.gremlin.com/clients/infrastructure
• You can search via the tag platform=ecs to verify that the Gremlin control plane can see the freshly launched ECS daemons
• Navigate to https://app.gremlin.com/attacks/new and click on the Containers tab
• Verify that you are seeing the application containers and tags currently running on your ECS cluster being imported into the Gremlin control plane

Step 4: Create a HTOP Elastic Container Repository with image

This will create a docker container that exposes htop via shellinaboxd on port 8888. htop is an interactive process viewer for Unix systems. We'll use htop as the target for an attack in step 8. Using htop isn't a requirement for installing Gremlin in Docker, but for this tutorial, using it makes it easier to see the impact of our attacks.

• In the AWS management console navigate to Repositories in the ECS service

• If you don't already have a repository, click Get started at the top; otherwise click New repository

• In Repository name type in htop, then click Next step

• Take note of the endpoint to push your docker image to, then click Done

• SSH into an instance in your AWS environment with the AWS command line tools and docker installed (e.g a jump box)

• Authenticate docker client against ECR: sudo $(aws ecr get-login --no-include-email --region us-east-1)

• Create and change directory to ~/docker-htop ; mkdir -p ~/docker-htop; cd ~/docker-htop

• Create the docker file

  1cat <<< 'FROM alpine:latest
  2RUN apk --no-cache add --update htop && rm -rf /var/cache/apk/*
  3RUN apk --no-cache add --repository http://dl-cdn.alpinelinux.org/alpine/edge/testing shellinabox
  4ENTRYPOINT ["shellinaboxd", "-t", "-p8888", "-s/:nobody:nogroup:/:htop"]' > Dockerfile

• Create the docker image sudo docker build -t htop .

• Tag the image to push to the repository, you'll need the end point details from creating the repository sudo docker tag htop:latest <<ACCNTID>>.dkr.ecr.us-east-1.amazonaws.com/htop:latest

• Push the container to ECR, again you'll need the end point details from creating the repository sudo docker push <<ACCNTID>>.dkr.ecr.us-east-1.amazonaws.com/htop:latest

Step 5: Create the HTOP Task Definition

• In the AWS management console navigate to Task Definitions the ECS service, and choose Create New Task Definition

• Select EC2 for the launch type compatibility and click Next Step

• On the Configure task and container definitions page, set the parameters as follows:

  • Task Definition Name: htop

  • Task Role: Leave blank

  • Network Mode: Leave as <default>

  • Task execution role: Leave as none

  • Task memory (MiB): 128

  • Task CPU (unit): 128

  • Click Add container, and in the Add container modal enter the following information, leaving defaults unless otherwise specified:

    • Container name: htop
    • Image: <<ACCNTID>>.dkr.ecr.us-east-1.amazonaws.com/htop:latest
    • Private repository authentication: Leave unchecked
    • Memory Limits (MiB): Hard limit 128
    • Port mappings: Host port: 8888; Container port: 8888
    • Scroll down to the Docker Labels section and enter appropriate key-value tags, at a minimum we suggest app:htop
    • Click the Add button

  • Scroll down and click the Create button

Step 6: Create a service definition for HTOP

• In the AWS management console navigate to the Clusters in the ECS service

• Select the cluster you just to deployed Gremlin into

• On the Services tab, click the Create button

• On the Configure service page, set the parameters as follows:

  • Launch type: EC2
  • Task Definition -> Family: htop
  • Task Definition -> Revision: latest
  • Cluster -> The cluster you wish to deploy into
  • Service name: htop
  • Service type: REPLICA
  • Number of tasks: 1

• Click Next step to bring you to the Set Auto Scaling page, and Next step again

• Review the service details to ensure accuracy, and if everything looks good click Create Service

Step 7: Open HTOP

• In the AWS management console navigate to the Clusters in the ECS service

• Select the cluster you just to deployed Gremlin into

• On the Services tab, click the htop service we just created

• Click on the Tasks tab

• Click on the task-ID for the running HTOP task

• Expand the htop container by clicking on the arrow next to the container name htop

• In the Network bindings section, click on the provided External Link

  • If the external link does not work, you may need to go into the security group associated with your ECS cluster and open port 8888

Congratulations, you should now see the htop interface in your web browser. Leave this open, as we'll be referring back to it in the next step.

Step 8: Run a test attack

• In a new browser window, open the link to the Gremlin Attacks UI: https://app.gremlin.com/attacks
• Click New Attack
• Select the Containers tab
• Select the HTOP container we created as your target by clicking the checkbox next to the container ID, you should be able to find this based on the Docker key-value pair we added, app:htop
• Click Choose a Gremlin
• Select the Resource category and CPU attack
• Default values should be fine, click Unleash Gremlin to launch the attack
• Observe in the open htop browser window that you can see the increased CPU load on the docker container.

Additional ECS Configurations

Now that you've ran a basic attack in ECS, there may be some advanced configuration that we want to make aware of:

• networkMode - This option determines which network space we would like to affect. In our example, we have it set to awsvpc which means the task can only affect the awsvpc interface. Some other options are: host, bridge, or none. For more information, please consult the AWS guide on Network mode.
• pidMode - This parameter allows you to configure the container to share their process ID with either the host or other containers in the task. By default, the setting is not stated. It may prove useful when performing process killer attacks to set this parameter to host. For more information, please consult the AWS guide on PID mode.

Conclusion

You now have Gremlin up and running in your ECS environment, and validated its functionality against a running htop container. For security, you should remove the htop container from your running cluster, as it's an unsecured metric view into your running environment.

Feel free to expand this to other ECS environments and have fun running Chaos Experiments!

Gremlin Task Definition JSON

1{
2   "family": "gremlin",
3   "containerDefinitions": [
4      {
5         "name": "gremlin",
6         "image": "gremlin/gremlin:latest",
7         "cpu": 0,
8         "portMappings": [],
9         "essential": true,
10         "entryPoint": [
11            "/entrypoint.sh"
12         ],
13         "command": [
14            "daemon"
15         ],
16         "environment": [
17            {
18               "name": "GREMLIN_TEAM_SECRET",
19               "value": "my-team-secret"
20            },
21            {
22               "name": "GREMLIN_TEAM_ID",
23               "value": "my-team-id"
24            },
25            {
26               "name": "GREMLIN_CLIENT_TAGS",
27               "value": "platform=ecs"
28            }
29         ],
30         "mountPoints": [
31            {
32               "sourceVolume": "runtime-runc",
33               "containerPath": "/run/docker/runtime-runc/moby",
34               "readOnly": false
35            },
36            {
37               "sourceVolume": "runtime-socket",
38               "containerPath": "/var/run/docker.sock",
39               "readOnly": false
40            },
41            {
42               "sourceVolume": "cgroup-root",
43               "containerPath": "/sys/fs/cgroup",
44               "readOnly": false
45            },
46            {
47               "sourceVolume": "gremlin-state",
48               "containerPath": "/var/lib/gremlin",
49               "readOnly": false
50            },
51            {
52               "sourceVolume": "gremlin-logs",
53               "containerPath": "/var/log/gremlin",
54               "readOnly": false
55            }
56         ],
57         "volumesFrom": [],
58         "linuxParameters": {
59            "capabilities": {
60               "add": [
61                  "KILL",
62                  "NET_ADMIN",
63                  "SYS_BOOT",
64                  "SYS_TIME",
65                  "SYS_ADMIN",
66                  "SYS_PTRACE"
67               ]
68            }
69         },
70         "logConfiguration": {
71            "logDriver": "awslogs",
72            "options": {
73               "awslogs-create-group": "true",
74               "awslogs-group": "/ecs/gremlin",
75               "awslogs-region": "us-west-1",
76               "awslogs-stream-prefix": "ecs"
77            }
78         }
79      }
80   ],
81   "executionRoleArn": "arn:aws:iam::my-aws-account:role/ecsTaskExecutionRole",
82   "networkMode": "host",
83   "volumes": [
84      {
85         "name": "runtime-runc",
86         "host": {
87            "sourcePath": "/run/docker/runtime-runc/moby"
88         }
89      },
90      {
91         "name": "runtime-socket",
92         "host": {
93            "sourcePath": "/var/run/docker.sock"
94         }
95      },
96      {
97         "name": "cgroup-root",
98         "host": {
99            "sourcePath": "/sys/fs/cgroup"
100         }
101      },
102      {
103         "name": "gremlin-state",
104         "host": {
105            "sourcePath": "/var/lib/gremlin"
106         }
107      },
108      {
109         "name": "gremlin-logs",
110         "host": {
111            "sourcePath": "/var/log/gremlin"
112         }
113      }
114   ],
115   "requiresCompatibilities": [
116      "EC2"
117   ],
118   "cpu": "1024",
119   "memory": "1024",
120   "pidMode": "host",
121   "runtimePlatform": {
122      "cpuArchitecture": "X86_64",
123      "operatingSystemFamily": "LINUX"
124   },
125   "tags": [
126      {
127         "key": "ecs:taskDefinition:createdFrom",
128         "value": "ecs-console-v2"
129      }
130   ]
131}