Deploying Failure Flags on Pivotal Cloud Foundry (PCF)

Failure Flags lets you run Gremlin experiments on applications running on Pivotal Cloud Foundry (PCF). This includes adding latency to network calls, generating exceptions, and triggering custom behaviors.

‍

Adding Failure-Flags-Sidecar to your Deployment manifest

Failure Flags runs on Cloud Foundry using a sidecar. However, unlike the regular Failure Flags sidecar, PCF sidecars run in the same container as your application. To accomplish this, you must download and extract the Failure Flags sidecar to your project folder. This section will walk you through this process.

‍

Configuring the Failure Flags sidecar

Failure Flags on PCF uses the common options listed in Configuring Failure Flags. We recommend creating a new file (e.g. vars.yaml) containing your configuration details.

‍

Example vars.yaml file:

gremlin_team_id: "YOUR_GREMLIN_TEAM_ID"
gremlin_team_certificate: |-
  -----BEGIN CERTIFICATE-----
  ...
  -----END CERTIFICATE-----
gremlin_team_private_key: |-
  -----BEGIN EC PRIVATE KEY-----
  ...
  -----END EC PRIVATE KEY-----
...

‍

Configuring your deployment manifest

First, download and extract the archive from one of these links, depending on your architecture: arm64 or x86_64. These links will download the latest version of the archive. If you want to download a specific version, replace latest with a release version (e.g., for version 2.0.0, use https://assets.gremlin.com/packages/failure-flags-sidecar/v2.0.0/arm64/failure-flags-sidecar-linux.tar.gz). You can find a list of releases in our release notes, or in our VERSIONS file.

‍

Next, add the Gremlin sidecar to your deployment manifest. This example shows how to bundle Gremlin with a Python Flask application, including necessary environment variables. You can adjust the memory and disk space available to the sidecar.

---
applications:
  - name: python-flask-app                # App name in CF
    memory: 512M                          # Memory for the container
    disk_quota: 1G                        # Disk space for the container
    instances: 1                          # Number of instances

    buildpacks:
      - python_buildpack                  # Use Python buildpack for Flask

    path: .                               # Push contents of current directory

    command: null                         # Let the buildpack detect and use your Procfile

    env:
      DEBUG_MODE: "false"
      FAILURE_FLAGS_ENABLED: "true"
      GREMLIN_SIDECAR_ENABLED: "true"
      GREMLIN_TEAM_ID: ((gremlin_team_id))
      GREMLIN_TEAM_CERTIFICATE: ((gremlin_team_certificate))
      GREMLIN_TEAM_PRIVATE_KEY: ((gremlin_team_private_key))
      SERVICE_NAME: "python-flask-app"

    sidecars:
      - name: gremlin-sidecar
        process_types: ["web"]            # Attach sidecar to 'web' process
        memory: 256M
        disk_quota: 256M
        command: "./bin/failure-flags-sidecar-amd64-linux"  # Run Gremlin sidecar

‍

Finally, push your application to Cloud Foundry:

cf push --vars-file vars.yaml

‍

Troubleshooting

Unable to interpolate credhub refs

The following error indicates that the staging steps failed to connect to a dependency (e.g. CredHub):

[STG/0] [ERR] Unable to interpolate credhub refs: Unable to interpolate credhub references: Post "https://credhub:8844/api/v1/interpolate": proxyconnect tcp: dial tcp 127.0.0.1:5034: connect: connection refused

If this happens, configure the HTTP_PROXY/HTTPS_PROXY environment variables for the runtime container (not globally), as follows:

1. Create a file called .profile in the root directory of your application containing the following lines:
   
   1. ‍export HTTP_PROXY="http://localhost:5034"
   2. export HTTPS_PROXY="http://localhost:5034"‍
2. Remove HTTP_PROXY/HTTPS_PROXY as global variables from the env block in the manifest file:

env:
  # HTTP_PROXY: "http://localhost:5034"   # remove
  # HTTPS_PROXY: "http://localhost:5034"  # remove

‍