Gremlin’s “Failure as a Service” makes it easy to find weaknesses in your system before they cause problems for your customers. Gremlin is a simple, safe, and secure way to use Chaos Engineering to improve system resilience.
Gremlin attacks are generated on the control plane. Clients make outbound SSL calls to poll for attacks. Gremlin provides secure command execution, security auditing, multi-factor authentication (MFA), and SAML SSO.
Gremlin does not require root privileges to any machines in your infrastructure. When installed on the host, gremlin operations are run via a
gremlin user created with default Linux privileges.
Gremlin does not need root privileges because the installed binary is granted the following capabilities to carry out attacks:
||used by shutdown to shutdown (and optionally reboot) your hosts|
||used by time travel to move your hosts forward and backward through time|
||used by the network gremlins for all network attacks|
||used by process killer to kill requested process(es)|
Gremlin never intercepts the content or payload of any network traffic. Gremlin only looks at routing information in order to apply its impact to the intended network traffic.
All communication between the Gremlin daemon and our service is initiated by the Gremlin daemon. For this reason, the daemon must have an outbound network path to the Gremlin service (
api.gremlin.com). Additionally, since all connections from the daemon are established outbound, it is not necessary to open ports in your security groups or firewall to allow inbound communications to the daemon.
The Gremlin client supports http/https proxies via the Linux environment variables
https_proxy. These should be set to use a proxy server via HTTP and HTTPS traffic, respectively. Values used should be of the form
http[s]://[username:password@]address:port, such as
export https_proxy=https://proxy.your_company.com:8080 or
The Gremlin daemon, which is typically run as a service, requires these environment variables be set in
echo "https_proxy=https://localhost:8888" | sudo tee -a /etc/default/gremlind sudo systemctl restart gremlind
Note that the Gremlin Service will only function via encrypted communication (HTTPS), attempts to connect to it via unencrypted channels (HTTP) will be denied.
The Gremlin daemon periodically communicates with our service over a TLS-protected channel which is authenticated using your organization credentials. Once authenticated, the daemon sends heartbeat messages to the service and receives instructions from the service as responses to the heartbeat messages. If an attack has been scheduled, the daemon receives the instructions for executing that attack. Each instruction action is pre-defined within the daemon and arbitrary instructions cannot be executed.
The service API only supports TLSv1.2 connections.
The Gremlin client, daemon, API, and website undergo regular security auditing, including penetration testing, by an external security auditor, Bishop Fox. All identified vulnerabilities are remediated promptly and confirmed via remediation testing by our auditors. We can provide a Letter of Assessment from our auditors outlining our most recent audit findings and remediation results upon request.
Gremlin offers Two Factor Authentication. See User Management.
Gremlin supports SAML SSO. See User Management.
Gremlin currently uses the host's file system to store temporary log and state information about attacks that have been run on the system. When running Docker with user namespace remapping (
userns-remap), Gremlin needs to assume the user namespace of the host. This applies for both the gremlin daemon container as well as when running
gremlin attack-container. Note that by assuming the user namespace of the host, we are creating an exception to backspace isolation for the Docker containers running Gremlin.
docker run -d \ --userns-remap=host \ -e GREMLIN_BYPASS_USERNS_REMAP=1 \ -v /var/lib/gremlin:/var/lib/gremlin \ -v /var/log/gremlin:/var/log/gremlin \ gremlin/gremlin daemon
echo "GREMLIN_BYPASS_USERNS_REMAP=1" | sudo tee -a /etc/default/gremlind sudo systemctl restart gremlind
export GREMLIN_BYPASS_USERNS_REMAP=1 gremlin attack-container 38dbd9016529 cpu