Failure Flags > Deploying Failure Flags on AWS ECS

Deploying Failure Flags on AWS ECS

This document will walk you through setting up Failure-Flags-Sidecar for your ECS Tasks. Gremlin provides a sidecar container called "Failure-Flags-Sidecar" that runs alongside your application. Failure Flags sidecar container images are available via DockerHub and support both AMD64/x86_64 and ARM64 architectures. These container images include a LICENSE file and a single binary program built for Linux. Alternatively, you can download the archives directly for arm64 or x86_64.

You can find a list of releases in our release notes, or in our VERSIONS file.

‍

Configuring Failure Flags for ECS

Failure Flags on ECS accepts the following environment variables:

Environment variable	Description
`ECS_SERVICE_NAME`	Set to the name of the service as you want it to appear in Gremlin. Service names can only contain alphanumeric characters, hyphens, and underscores, and must be less than 64 characters long.
`GREMLIN_SIDECAR_ENABLED`	Set to `true`, `yes`, or `1` to enable the Gremlin sidecar extension.
`FAILURE_FLAGS_ENABLED`	Set to `true`, or `1`to enable the Failure Flags SDK in your application. If you’re using Failure Flags by proxy, you don't need to set this variable.
`GREMLIN_INGRESS_PROXY_ENABLED`	Enable the reverse proxy and begin experimenting on inbound requests and responses to those requests without any additional changes to your application code.
`GREMLIN_INGRESS_PROXY_PORT`	Set to the port number that the ingress proxy will bind to (e.g., :5035).
`GREMLIN_INGRESS_PROXIED_ENDPOINT`	Required when the ingress proxy is enabled. The URL of your application inside the task, including the scheme.
`GREMLIN_TEAM_ID`	Your Gremlin Team ID. This and other credential details are available in the Gremlin web app.
`GREMLIN_TEAM_CERTIFICATE`	Set to the contents of your Gremlin Team certificate. This and other credential details are available in the Gremlin web app. Preserve newlines by adding the `\n`escape character at the end of each line, or by omitting them entirely.
`GREMLIN_TEAM_PRIVATE_KEY`	Set to the contents of your Gremlin Team private key. This and other credential details are available in the Gremlin web app. Preserve newlines by adding the `\n`escape character at the end of each line, or by omitting them entirely.
`SERVICE_NAME`	This variable is only read if ECS_SERVICE_NAME is unreadable. Set the name that you want to register the service as in Gremlin. Service names can only contain alphanumeric characters, hyphens, and underscores, and must be less than 64 characters long.
`GREMLIN_METRICS_EXCLUDE_HEALTH_CHECK_PATHS`	A comma-separated list of URL paths that you want to exclude from metrics collection. This is primarily for excluding paths used for Health Checks.
`GREMLIN_METRICS_OPT_OUT`	Set to `true` if you do not want the sidecar to collect application metrics (metrics are used for Intelligent Health Checks.

‍

Adding the sidecar to your container definitions

Adding the sidecar means including an additional task in any ECS application where you want to use Failure Flags. For example, this definition deploys an application named ff-nodejs-demoapp and the Gremlin sidecar container. This application will be listed in Gremlin with the name my-fargate-sidecar-demo (via the ECS_SERVICE_NAME environment variable):

JSON

{
    "containerDefinitions": [
        {
            "name": "app",
            "image": "samew/ff-nodejs-demoapp:amd64",
            "cpu": 0,
            "portMappings": [
                {
                    "name": "app-3000-tcp",
                    "containerPort": 3000,
                    "hostPort": 3000,
                    "protocol": "tcp",
                    "appProtocol": "http"
                }
            ],
            "essential": true,
            "environment": [
                {
                    "name": "FAILURE_FLAGS_ENABLED",
                    "value": "true"
                },
                {
                    "name": "HTTP_PROXY",
                    "value": "http://localhost:5034"
                },
                {
                    "name": "HTTPS_PROXY",
                    "value": "http://localhost:5034"
                },
                {
                    "name": "NO_PROXY",
                    "value": "localhost,127.0.0.1"
                }
            ],
            "logConfiguration": {
                "logDriver": "awslogs",
                "options": {
                    "awslogs-create-group": "true",
                    "awslogs-group": "/ecs/nodejs-ff-demoapp",
                    "awslogs-region": "us-east-2",
                    "awslogs-stream-prefix": "ecs"
                },
                "secretOptions": []
            }
        },
        {
            "name": "gremlin-sidecar",
            "image": "gremlin/failure-flags-sidecar:latest",
            "cpu": 0,
            "portMappings": [
                {
                    "name": "ff-ingress-5035-tcp",
                    "containerPort": 5035,
                    "hostPort": 5035,
                    "protocol": "tcp",
                    "appProtocol": "http"
                }
            ],
            "essential": true,
            "environment": [
                {
                    "name": "ECS_SERVICE_NAME",
                    "value": "my-fargate-sidecar-demo"
                },
                {
                    "name": "GREMLIN_DEBUG",
                    "value": "true"
                },
                {
                    "name": "GREMLIN_TEAM_CERTIFICATE",
                    "value": "-----BEGIN CERTIFICATE-----ExampleXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX-----END CERTIFICATE-----"
                },
                {
                    "name": "GREMLIN_SIDECAR_ENABLED",
                    "value": "true"
                },
                {
                    "name": "GREMLIN_INGRESS_PROXY_ENABLED",
                    "value": "true"
                },
                {
                    "name": "GREMLIN_INGRESS_PROXY_PORT",
                    "value": ":5035"
                },
                {
                    "name": "GREMLIN_INGRESS_PROXIED_ENDPOINT",
                    "value": "http://localhost:3000"
                },
                {
                    "name": "GREMLIN_TEAM_ID",
                    "value": "ffffffff-ffff-ffff-ffff-ffffffffffff"
                },
                {
                    "name": "GREMLIN_TEAM_PRIVATE_KEY",
                    "value": "-----BEGIN EC PRIVATE KEY-----ExampleXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX==-----END EC PRIVATE KEY-----"
                },
                {
                    "name": "GREMLIN_DEPENDENCY_PROXY_ENABLED",
                    "value": "true"
                }
            ],
            "logConfiguration": {
                "logDriver": "awslogs",
                "options": {
                    "awslogs-create-group": "true",
                    "awslogs-group": "/ecs/nodejs-ff-demoapp",
                    "awslogs-region": "us-east-2",
                    "awslogs-stream-prefix": "ecs"
                },
                "secretOptions": []
            }
        }
    ],
    "family": "nodejs-ff-demoapp",
    "executionRoleArn": "arn:aws:iam::999999999999:role/ecsTaskExecutionRole",
    "networkMode": "awsvpc",
    "volumes": [],
    "requiresCompatibilities": [
        "FARGATE"
    ],
    "cpu": "1024",
    "memory": "3072",
    "runtimePlatform": {
        "cpuArchitecture": "X86_64",
        "operatingSystemFamily": "LINUX"
    }
}

‍

ECS metadata discovered by Gremlin

Gremlin automatically detects the following environment variables for ECS functions:

Environment variable	Description
`ECS_CONTAINER_METADATA_URI`	The address of the task metadata version 3.
`ECS_CONTAINER_METADATA_URI_V4`	The address of the task metadata version 4.
`ECS_HOST_DATA_DIR`	The host path where the container’s agent state file is located.
`ECS_SERVICE_NAME`	The name of the Amazon ECS service to which the task belongs.

‍

Route Traffic

Inbound: in your ECS service's load balancer settings, set the target group to container gremlin-sidecar, port 5035, instead of app:3000. Health checks follow automatically.

Outbound: HTTP_PROXY/HTTPS_PROXY on the app container handle this for most runtimes. For Java, add -Dhttps.proxyHost=localhost -Dhttps.proxyPort=5034 (and the http. equivalents) to JAVA_TOOL_OPTIONS. For Node.js, configure your HTTP client's proxy agent.

‍

Verify

Deploy the new task definition revision and confirm both containers are RUNNING in the ECS console.
Check the sidecar's logs for successful registration: aws logs tail /ecs/nodejs-ff-demoapp --follow.
Send a few requests through the load balancer, including some that call a dependency.
In Gremlin, open Failure Flags. my-fargate-sidecar-demo should be listed with process, ingress, http-ingress, egress, and one dependency- flag per host it called.

If the service or its flags don't appear, see Troubleshooting Failure Flags.

‍

Privileges required

Privilege	Description
CLIENTS_READ	Allows reading all client information within the team
CLIENTS_WRITE	Allows editing all client information within the team

Deploying Failure Flags on AWS Lambda

Deploying Failure Flags on Kubernetes