Getting Started > Configuring the Gremlin Agent

Configuring the Gremlin Agent

This documentation page shows you how to configure the Gremlin Agent. You can configure Gremlin using either environment variables or a configuration file, although we recommend using the configuration file in most cases. This guide assumes you've already installed and authenticated the Agent.

‍

Using the configuration file

Gremlin's configuration file is located in a file named config.yaml. On Linux systems, this file is located at /etc/gremlin/config.yaml. On Windows, it's located at C:\ProgramData\Gremlin\Agent\config.yaml. Gremlin ships with a copy of this file with all available options commented out. Use this file or the following example to set up your configuration.

Any changes to config.yaml require a restart of the gremlind service.

‍

Accepted variables

Variable	Description
`identifier`	a string that uniquely identifies a Gremlin installation (defaults to the hosts IP address)
`team_id`	the ID of your Gremlin team
`tags`	key-value pairs that help you target this machine during experiments
`team_secret`	the secret value (used by secret authentication only)
`team_certificate`	the public portion of the certificate-pair (required for signature auth only)
`team_private_key`	the private portion of the certificate-pair (required for signature auth only)
`https_proxy`	the url of a proxy in between Gremlin and api.gremlin.com
`ssl_cert_file`	the ssl certificate for your https proxy
`push_metrics`	set to `false` if you would like to disable system metrics used for experiment charting (defaults to `true`)
`collect_dns`	set to `false` to stop sending DNS query data to Gremlin for dependency discovery (defaults to `true`). See Enabling DNS collection.
`iam_role`	role the agent uses to retrieve remote cloud config values
`service_url`	(for Gremlin Private Edition users) the URL of your private Gremlin API suffixed with `/v1`
`tls_identity_certificate`	(for mutual TLS users) A PEM-encoded certificate chain, starting with the leaf certificate, followed by its intermediates.
`tls_identity_private_key`	(for mutual TLS users) A PEM-encoded private key.
`push_pod_cidr_tags`	Set to `false` to stop the agent from publishing pod CIDR ranges as tags (defaults to `true`). See Using tags.
`push_zone_cidr_tags`	Set to `false` to stop the agent from publishing zone CIDR ranges as tags (defaults to `true`). See Using tags.

‍

Example: config.yaml

‍

Restarting the Gremlin Agent

To restart the Gremlin Agent and apply your new changes, run the following commands (depending on your environment):

‍

Linux

‍

Windows

‍

Docker

‍

Using environment variables

The daemon supports the following environment variables. For AWS users, Gremlin supports ARN values from AWS Secrets Manager or AWS Systems Manager Parameter Store.

Environment Variable	Description
`GREMLIN_TEAM_ID`	Your Team ID (required for authentication)
`GREMLIN_TEAM_SECRET`	Your Team Secret (should only require secret or PEM certificates, not both)
`GREMLIN_TEAM_PRIVATE_KEY_OR_FILE`	Your PEM-encoded private key or path to a file containing the private key (required for authentication). Paths should be prefixed with `file://` . For example, on Unix: `file:///var/lib/gremlin/key.pem` , on Windows: `file://C:/ProgramData/Gremlin/Agent/key.pem`
`GREMLIN_TEAM_CERTIFICATE_OR_FILE`	The PEM-encoded public-key certificate or path/filename to the file containing your PEM-encoded public-key certificate (required for authentication). Paths should be prefixed with `file://`. For example, on Unix: `file:///var/lib/gremlin/cert.pem` , on Windows: `file://C:/ProgramData/Gremlin/Agent/cert.pem`
`GREMLIN_IDENTIFIER`	Custom name for this client (default as the host's IP address)
`GREMLIN_CLIENT_TAGS`	Comma-separated list of custom tags to assign to this client. For example: `GREMLIN_CLIENT_TAGS="zone=us-east1,role=mysql,foo=bar"`
`GREMLIN_SERVICE_URL`	For Gremlin Private Edition users, this is the URL of your private Gremlin API instance, suffixed with `v1`. For example, `https://api.gremlin.example.com/v1`.

‍

The following are optional environment variables that may be required in containerized or cloud deployments:

Environment Variable	Description
`GREMLIN_BYPASS_USERNS_REMAP`	Indicates to run the Gremlin Daemon on the same Docker namespace as the host. By default, Gremlin does not launch with a namespace defined. To set this variable, change the value to `1`
`GREMLIN_DOCKER_IMAGE`	Indicates the location and version of the Gremlin Docker image to use. This variable can be used to indicate an internal repository. For example: `GREMLIN_DOCKER_IMAGE=registry.hub.docker.com/gremlin/gremlin:latest`
`GREMLIN_IAM_ROLE`	Indicates to run the Gremlin Daemon with the specified IAM role when retrieving remote config values.

‍

When running the Docker image, pass --net=host so network experiments affect the host network, and --pid=host so Process Killer can affect host processes. Both flags are included in the docker run command on the Linux installation page.

‍

In addition, the following standard Linux environment variables allow proxy configuration:

Environment Variable	Description
`https_proxy`	In the form `http[s]://[username:password@]address:port`
`SSL_CERT_FILE`	Indicates location of the proxy certificates used to authenticate traffic with the proxy. For Example: `SSL_CERT_FILE=/etc/ssl/certs/ca-example.crt`

‍

Configuration value priorities

If both the environment variables and the configuration file are used when configuring Gremlin, the environment variable values will take priority. For example if you have the GREMLIN_IDENTIFIER environment variable set to foo and in the config.yaml file the identifier field is set to bar, Gremlin will end up using foo as its identifier when registering with the API.

‍

Remote config values

The Gremlin agent supports using AWS ARNs for the following config values in either ENVVARs or config file:

identifier
team_id
team_secret
team_certificate
team_private_key
tls_identity_certificate
tls_identity_private_key

When remote configuration values are used, Gremlin does not store the corresponding configuration values on the hard disk of the system where it runs. When Gremlin is configured to use Secret Authentication option (i.e. using a team_secret), the agent must store temporary session tokens in the .credentials file, which is only readable by the Gremlin system user.

The following services are supported for reading values:

AWS Systems Manager (ssm)
AWS Secrets Manager

The agent requires some credentials to exist on the machine to be able to retrieve the remote values. The credentials preference follows the default AWS order.

‍

Passing environment variables to Systemd or SysV

While configuring the daemon with config.yaml is recommended, process managers like systemd and sysvinit can pass environment variables directly to the daemon process from /etc/default/gremlind. This file is not created during installation; copy and edit the example file from /etc/default/gremlind.example, or create a new file from the following options:

‍

You can set any of the environment variables listed in the previous section in the configuration file.

‍

DNS Collection

In order to run dependency tests using Gremlin Reliability Management (RM), the Gremlin agent must be configured to collect DNS data by enabling DNS collection. When DNS collection is enabled, the Gremlin Agent collects DNS information from hosts, containers, and Kubernetes objects, and uses it to detect dependencies.

As of Linux Agent version 2.42.0, Gremlin automatically enables DNS Collection.

To learn more about enabling or disabling DNS collection, see DNS collection.

‍

Client TLS Identity for Mutual TLS (mTLS)

The gremlind and gremlin-integrations agents support setting a TLS identity to enable mutual TLS. The following values are required for setting that up:

tls_identity_certificate: A PEM-encoded certificate chain, starting with the leaf certificate, followed by its intermediates.
tls_identity_private_key: A PEM-encoded private key.
service_url: The URL of your mTLS gateway.

Values can be set directly into config.yaml , referenced as paths on the file system, or ARNs to secrets in AWS (see Remote config values).

For Kubernetes users, upgrade to Helm Chart 0.25.0 and supply values under gremlin.tls.identity and chao.tls.identity. For example:

‍

Private Edition configuration

For Gremlin Private Edition users, you will need to change the URL that the agent uses to communicate with Gremlin from the public Gremlin API to your Private Edition instance. You can do this by setting the gremlin.serviceUrl option (or the GREMLIN_SERVICE_URL environment variable) to the URL of your instance, with the suffix v1. For example, this will configure the agent to send requests to a local Private Edition instance:

‍

Customize Gremlin's Linux user and group

By default, Gremlin installs a gremlin user and group with the necessary privileges to perform all Gremlin features. These defaults can be overridden at installation time by supplying environment variables to the installer. The full list of variables available at install time and their defaults are as follows:

Variable	Default	Description
`GREMLIN_INSTALL_USER`	`gremlin`	The Linux user to own all file installed by Gremlin. This user is created if it does not exist
`GREMLIN_INSTALL_GROUP`	`gremlin`	The Linux group to own all file installed by Gremlin. This group is created if it does not exist
`GREMLIN_INSTALL_BIN_MODE`	`0750`	The file mode for executable files installed by Gremlin. Gremlin versions before `2.33.0` default this value to `6111` instead.
`GREMLIN_INSTALL_BIN_CAPABILITIES`	unset	A boolean argument specifying whether Gremlin's required capabilities should be set on the binaries themselves. The default is false. Any value supplied to this variable evaluates to true.

‍

For example, to run Gremlin as root and restrict executable access to root:root, run:

Authenticating the Gremlin Agent

Onboarding services automatically