Failure Flags > Deploying Failure Flags on Google Cloud Run

Deploying Failure Flags on Google Cloud Run

This document will walk you through setting up Failure Flags for services running on Google Cloud Run. On Cloud Run, Failure Flags is deployed as a sidecar container: Gremlin's failure-flags-sidecar image runs alongside your application container within the same Cloud Run instance.

This guide uses Failure Flags by proxy. In this mode the sidecar intercepts both inbound and outbound HTTP(S) traffic and injects faults at the network level, so no code changes are required. The application container is configured to route its traffic through the sidecar via the HTTP_PROXY and HTTPS_PROXY environment variables.

Cloud Run's billing mode can affect the sidecar's ability to reach Gremlin's control plane. If your service uses request-based billing (the default), see Sidecar tuning options below.

‍

Configuring Failure Flags for Cloud Run

Cloud Run deployments use the same common configuration options as any other Failure Flags sidecar deployment, plus the proxy mode options below. There are no Cloud Run-specific environment variables required beyond these.

‍

Common and credential options (sidecar container)

Environment variable	Description
`GREMLIN_SIDECAR_ENABLED`	Set to `true`, `yes`, or `1` to enable the sidecar. If unset or set to any other value, the sidecar operates in `NOOP` mode.
`GREMLIN_TEAM_ID`	Your Gremlin Team ID. Available in the Gremlin web app.
`GREMLIN_TEAM_CERTIFICATE`	The contents of your Gremlin Team certificate. Preserve newlines by adding the `\n` escape character at the end of each line, or by omitting them entirely.
`GREMLIN_TEAM_PRIVATE_KEY`	The contents or file path of your Gremlin Team private key. Preserve newlines as above.
`SERVICE_NAME`	The name you want the service to register as in Gremlin. Alphanumeric, hyphens, and underscores only, under 64 characters. Set this explicitly, as in the example config below.
`GREMLIN_DEBUG`	Set to `true` to enable debug logging.
`GREMLIN_TRACE`	Set to `true` to enable additional debugging information on network requests from the sidecar.

‍

Treat GREMLIN_TEAM_CERTIFICATE and GREMLIN_TEAM_PRIVATE_KEY as sensitive values. In this example, we store them in Secret Manager and reference the secret from the service definition (valueFrom.secretKeyRef)

‍

Ingress proxy options (sidecar container)

The ingress proxy makes the sidecar the entry point for inbound traffic, forwarding it to your application after Gremlin has had a chance to inject a fault.

Environment variable	Description
`GREMLIN_INGRESS_PROXY_ENABLED`	Set to `true` to enable the reverse proxy for inbound traffic.
`GREMLIN_INGRESS_PROXY_PORT`	The port the ingress proxy binds to, e.g. `:5035`. This is the port Cloud Run should route inbound requests to.
`GREMLIN_INGRESS_PROXIED_ENDPOINT`	The URL of your application, e.g. `http://localhost:3000`. Required.

‍

Dependency proxy options (sidecar container)

The dependency proxy intercepts your application's outbound HTTP/HTTPS calls so you can inject faults into calls to downstream dependencies.

Environment variable	Description
`GREMLIN_DEPENDENCY_PROXY_ENABLED`	Set to `true` to enable the proxy for dependency (outbound) traffic.
`GREMLIN_DEPENDENCY_PROXY_PORT`	The address and port the dependency proxy binds to. Defaults to `localhost:5034`.

‍

Application container options

The application containers needs to be configured to route outbound calls through the sidecar's dependency proxy. The exact environment variables/method to accomplish this depends on your application's programming language and libraries, but for most deployments setting HTTP_PROXY/HTTPS_PROXY is sufficient:

Environment variable	Description
`HTTP_PROXY` / `HTTPS_PROXY`	Point these at the sidecar's dependency proxy address (e.g. `http://localhost:5034`) so your application's outbound HTTP(S) libraries route traffic through it. Whether your runtime honors these automatically depends on your language/HTTP client.
`NO_PROXY`	Exclude `localhost,127.0.0.1` so traffic to the sidecar itself (and any other localhost services) isn't recursively proxied.

‍

Please note that in some common application environments (Java, Node) HTTP_PROXY and HTTPS_PROXY environment variables are not read by default.

‍

Sidecar tuning options

Environment variable	Description
`GREMLIN_REQUEST_TIMEOUT`	How long the sidecar will wait for a response from Gremlin's control plane before timing out. See the note on billing settings below; this is a specific consideration on Cloud Run.
`GOMEMLIMIT`	A standard Go runtime variable that tells the sidecar's Go process its soft memory limit. Useful in memory-constrained environments (e.g. cgroup or Kubernetes/Cloud Run memory limits): set below the container's hard memory limit to help avoid OOMKills of the sidecar.

‍

Request-based vs. instance-based billing

Cloud Run services run under one of two billing settings:

Request-based billing (default): CPU is only allocated to the instance while it's starting up, shutting down, or actively processing a request. Between requests, the instance's access to CPU is severely limited.
Instance-based billing: CPU is allocated for the entire lifecycle of the instance, whether or not it's currently handling a request.

‍

For request-based billing, the sidecar may be CPU-throttled during idle periods between requests, which may impact sidecar communications with Gremlin's control plane. Increasing GREMLIN_REQUEST_TIMEOUT can mitigate this in request-based billing environments.

‍

Adding the sidecar to your Cloud Run service

Add the sidecar by declaring a second container in your Cloud Run service definition. Here is a yaml of an example application myapplication for deploying in Google Cloud Run:

‍

The sidecar (gremlin-sidecar), not the application, is the ingress container. The declared containerPort (5035) is what makes Cloud Run route inbound requests to it. It then forwards each request to the application via GREMLIN_INGRESS_PROXIED_ENDPOINT (http://localhost:3000).
Outbound traffic is routed back through the sidecar. myapplication sets HTTP_PROXY/HTTPS_PROXY to the sidecar's dependency-proxy address (http://localhost:5034), with NO_PROXY excluding localhost and 127.0.0.1 so the sidecar's own local traffic isn't recursively routed through itself.

‍

Troubleshooting Failure Flags

The sidecar provides debug logging when the GREMLIN_DEBUG environment variable is set to true. This information is included with your Cloud Run logs and prefixed with [gremlin-sidecar], and will include configuration details, registration status, connection tracing, and any relevant errors encountered while interacting with the control plane.

For additional Failure Flags network request debugging, also set GREMLIN_TRACE to true. Trace logging shows additional network request information, including HTTP requests to the Gremlin API, TLS handshake details, Network timeouts/retries, proxy interactions.

‍

Deploying Failure Flags on Pivotal Cloud Foundry (PCF)

Installing the Failure Flags SDK