Dashboard
Getting Started

Authenticating the Gremlin Agent

cdBefore you can start using Gremlin, you need to authenticate the Agent with your Gremlin team. This documentation page covers the various methods of authenticating the Agent.

Note
If you've already authenticated the agent (e.g. by downloading the client configuration file during installation), you can skip this section. If you're not sure whether you've already authenticated, seec Verifying Gremlin Agent authentication to learn how to check.

Agent authentication methods

There are three methods for authenticating clients with Gremlin:

  1. (Recommended) The Client configuration file is a pre-generated configuration file containing everything needed to authenticate a Gremlin Agent, including a team ID.
  2. Signature authentication requires supplying a public and private ECDSA certificate-key pair to Gremlin clients. This certificate pair is created from the Gremlin web app, and can be downloaded again at any time, but expires 1 year from the date of creation.
  3. Secret authentication involves a secret string value that Gremlin clients use to prove their identity. Secrets are created from the Gremlin UI, and are like passwords. They can never be retrieved from Gremlin after creation; they can only be reset.

The recommended method is using the client configuration file, but the other authentication methods are available for teams that wish to use them.

Client Configuration File

Gremlin automatically generates a pre-formatted configuration file containing everything you need to authenticate, including your team ID. It uses signature-based authentication to generate a private and public key pair and embeds those keys into a YAML file, which you can use to authenticate your Agents.

To create a client configuration file:

  1. Access the Team Settings page in the Gremlin web app.
  2. Click the Configuration tab.
  3. On the Client Configuration File line, click Download to download the file. You'll receive a file named config.yaml.
  4. Optionally, make any additional configurations to the config.yaml file.
  5. Copy the contents of the file into the /etc/gremlin/config.yaml file (or simply overwrite the existing file).
  6. Restart the gremlind service.

Signature-based authentication

Download a certificate pair

To generate a certificate pair:

  1. Navigate to Team Settings in the Gremlin web app.
  2. Click on the Configuration tab.
  3. On the Certificates line, click Create to create a new certificate.
  4. Click Download to download your certificates. The downloaded zip file contains both a public-key certificate and a matching private key.
  5. Unzip the file, then read the next section to configure the clients.


Configure clients with certificate pair

To configure Gremlin clients for signature authentication, follow one of the advanced configuration methods:

  1. set team_id, team_certificate, and team_private_key in your config.yaml
YAML

#/etc/gremlin/config.yaml
team_id: 11111111-1111-1111-1111-111111111111
team_certificate: file:///var/lib/gremlin/cert.pem
team_private_key: file:///var/lib/gremlin/key.pem

  1. or pass environment variables GREMLIN_TEAM_ID, GREMLIN_TEAM_CERTIFICATE_OR_FILE, and GREMLIN_TEAM_PRIVATE_KEY_OR_FILE directly to the gremlind process
SHELL

# export these to the gremlind process or save them to /etc/default/gremlind (used by Systemd and SysV)
GREMLIN_TEAM_ID=11111111-1111-1111-1111-111111111111
GREMLIN_TEAM_CERTIFICATE_OR_FILE=file:///var/lib/gremlin/cert.pem
GREMLIN_TEAM_PRIVATE_KEY_OR_FILE=file:///var/lib/gremlin/key.pem

Rotating expiring certificates

You must be a Team or Company Manager of your Gremlin account to manage certificates.

All certificates expire one year after creation. Before the expiration date, you must create a new certificate, reconfigure all clients with it, and delete the old certificate. As the expiration date draws near, the Gremlin web app will display a warning about it.

To create a new certificate, simply click the Create New button next to your existing certificate. Now that you have two certificates, you cannot download the older certificate (though all clients still configured with it may still use it to authenticate), and you cannot create a third certificate. Make sure you move all clients to the new certificate before the old one expires.

Secret-based authentication

Create a secret

To create a new secret:

  • Navigate to Team Settings in the Gremlin web app.
  • Click on the Configuration tab.
  • On the Secret Key line, click Create to create a new secret key, then read the next section to configure your Agents.
If a secret already exists, you must reset the existing secret. If you reset the secret, you will need to apply the new secret to any existing clients.

Configure clients with secret

To configure Gremlin clients for secret authentication, follow one of the [advanced configuration] (/docs/getting-started/advanced-configuration/) methods:

  1. set team_id and team_secret in your config.yaml
YAML

team_id: 11111111-1111-1111-1111-111111111111
team_secret: 11111111-1111-1111-1111-111111111111

  1. or pass environment variables GREMLIN_TEAM_ID and GREMLIN_TEAM_SECRET directly to the gremlind process.
SHELL

# export these to the gremlind process or save them to /etc/default/gremlind (used by Systemd and SysV)
GREMLIN_TEAM_ID=11111111-1111-1111-1111-111111111111
GREMLIN_TEAM_SECRET=11111111-1111-1111-1111-111111111111

Additionally, secret authentication can also be set up via gremlin init

SHELL

gremlin init --tag service=my-api --tag service-version=1.0.0 --tag service-type=http


f secret-based auth is successful, the client will create a hidden file, .credentials in the gremlin user's home directory. This file contains an authentication token that gremlind uses to connect to the Gremlin Control Plane.

Verifying Gremlin Agent authentication

To verify that your Gremlin Agents are connected to the Gremlin Control Plane, open the Agents page in the Gremlin web app. If your system doesn't appear there, the next step is to ensure that the Gremlin Agent is running.

For Linux hosts, run the following command:

SHELL

sudo systemctl status gremlind

For Windows hosts, run the following command:

POWERSHELL

Get-Service gremlind

The output from either of these commands should show the service as Running.

Next, use gremlin check auth to check the Gremlin Agent's authentication status:

SHELL

gremlin check auth

If the Gremlin Agent authenticated successfully, the output will be similar to the following:

auth
====================================================
Auth Input Type                      : Certificate
API Response                         : OK

If not, the output will explain why the Gremlin Agent was unable to authenticate:

auth
====================================================
Auth Input Type                      : No valid auth found
========= Identification ============:
Team ID Source                       : Team ID not found
Gremlin Identifier                   : [Host identifier]
Gremlin Identifier Source            : Not supplied explicitly, using the default
========= Secret/Token Info =========:
Team Secret Source                   : Team Secret not found
.credentials valid                   : /var/lib/gremlin/.credentials file not found
========= Certificate Info ==========:
Team Certificate Source              : Team Certificate not found
========= Private Key Info ==========:
Team Private Key Source              : Team Private Key not found

If you're having trouble authenticating, see the troubleshooting section below.

Releasing client certificates

There are three different methods you can use to release client certificates:

  • Gremlin Logout command - From the host, you can log out of the client using the command "gremlin logout". This method allows you to remove the authorization from our server but keep the binaries in place for future use. Once logged out, the license should be released within a few minutes for consumption by other clients.
  • WebUI deactivation option - From the WebUI, you can deactivate the client to release the license. This will prevent the host from being used in future experiments until it is reactivated.
  • Uninstalling Gremlin client from the host - This method is similar to option 1. By removing the binaries, the host will no longer communicate with our service and the license will be released for future consumption.

Once any of these options have been performed, it may take a few minutes for new clients to use the newly released entitlements. You can view the details of which teams have active clients for a given month by navigating to Gremlin Reports within the Gremlin web app. The Active Clients metric on this page is cumulative for any active clients that may have authenticated.

Gremlin .credentials file

When secret-based authentication is used and the Gremlin agent successfully authenticates against the Gremlin Control Plane, the Gremlin agent creates a new file named /var/lib/gremlin/.credentials (C:\ProgramData\Gremlin\Agent\.credentials on a Windows-based install). This file stores temporary session and authorization data. Sessions are valid for twelve hours. After twelve hours (or if the session is ever invalidated), the agent creates a new .credentials file in the same location. It also assigns the file to user ID and group ID (uid:gid) gremlin:gremlin and sets the file's permissions to 660.

Note that this file isn't created when using certificate-based authentication.

Troubleshooting

If you're having problems authenticating the agent, see the Authentication FAQ in the Gremlin Knowledge Base.

Configuring Gremlin

Once you've successfully authenticated, continue to Configuring Gremlin to complete Agent setup.

No items found.
Previous
Next
Previous
This is some text inside of a div block.
Compatibility
Installing the Gremlin Agent
Authenticating the Gremlin Agent
Configuring the Gremlin Agent
Managing the Gremlin Agent
User Management
Integrations
Health Checks
Notifications
Command Line Interface
Updating Gremlin
Quick Start Guide
Services and Dependencies
Detected Risks
Reliability Tests
Reliability Score
Targets
Experiments
Scenarios
GameDays
Overview
Deploying Failure Flags on AWS Lambda
Deploying Failure Flags on AWS ECS
Deploying Failure Flags on Kubernetes
Classes, methods, & attributes
API Keys
Examples
Container security
General
Linux
Windows
Chao
Helm
Glossary
Alfi
Additional Configuration for Helm
Amazon CloudWatch Health Check
AppDynamics Health Check
Application Level Fault Injection (ALFI)
Blackhole Experiment
CPU Experiment
Certificate Expiry
Custom Health Check
Custom Load Generator
DNS Experiment
Datadog Health Check
Disk Experiment
Dynatrace Health Check
Grafana Cloud Health Check
Grafana Cloud K6
IO Experiment
Install Gremlin on Kubernetes manually
Install Gremlin on OpenShift 4
Installing Gremlin on AWS - Configuring your VPC
Installing Gremlin on Kubernetes with Helm
Installing Gremlin on Windows
Installing Gremlin on a virtual machine
Installing the Failure Flags SDK
Jira
Latency Experiment
Memory Experiment
Network Tags
New Relic Health Check
Overview
Overview
Overview
Overview
Overview
Packet Loss Attack
PagerDuty Health Check
Preview: Gremlin in Kubernetes Restricted Networks
Private Network Integration Agent
Process Collection
Process Killer Experiment
Prometheus Health Check
Role Based Access Control
Running Failure Flags experiments
Scheduling Scenarios
Shared Scenarios
Shutdown Experiment
Slack
Teams
Time Travel Experiment
Troubleshooting Gremlin on OpenShift
User Authentication via SAML and Okta
Users
Webhooks
Integration Agent for Linux
Test Suites
Restricting Testing Times
Reports
Process Exhaustion Experiment
Enabling DNS collection