Container security

Gremlin containers run as root

When Gremlin runs within a container, <span class="code-class-custom">gremlin</span> processes run as <span class="code-class-custom">root</span>. This is because there are several file resources that Gremlin must mount from the host which are owned by <span class="code-class-custom">root</span> on the host machine:

  • <span class="code-class-custom">
  • /var/lib/gremlin</span> and <span class="code-class-custom">/var/log/gremlin</span>: These are mounted from the host into the Gremlin agent container and sidecar experiments. By default, container runtimes will set the ownership of these resources to <span class="code-class-custom">root:root</span>
  • Container runtime sockets (e.g. <span class="code-class-custom">/var/run/docker.sock</span>), which by default will be owned by <span class="code-class-custom">root</span>
  • <span class="code-class-custom">
  • /proc/sysrq-trigger</span> and <span class="code-class-custom">/sys/fs/cgroup</span> for attack capabilities, which by default will be owned by root

Mitigating the privileges of root

Though Gremlin runs as <span class="code-class-custom">root</span> within the container in which it runs, Gremlin is restricted by the specific capabilities it requests, despite running as <span class="code-class-custom">root</span>. Here are some other mitigations system administrators can make to reduce the impact of Gremlin's usage of <span class="code-class-custom">root</span>:

  • While Docker's default seccomp profile is already fairly restrictive, you can remove any undesired syscalls by overriding this default, or provide a more restrictive seccomp profile to Gremlin containers.
  • Restrict Gremlin's resource access on the host using AppArmor and the Gremlin Agent AppArmor Profile.
  • Avoid running Gremlin within the network or process namespace of the host. For Docker, avoid <span class="code-class-custom">docker run --net=host --pid=host</span>. Note that this will disable Gremlin's ability to run host-level Network and Process Killer experiments.
  • Consider an extra layer of safety by enabling SELinux to harden all containers running on your systems


If you do not wish to run Gremlin as a <span class="code-class-custom">root</span> user, even with the restrictions described above, consider installing Gremlin directly onto the host where it runs under a dedicated Linux user: <span class="code-class-custom">gremlin</span>.

Docker (Linux)

User namespace isolation

Gremlin currently uses the host's file system to store temporary log and state information about experiments. When running Docker with user namespace remapping (<span class="code-class-custom">userns-remap</span>), Gremlin needs to assume the user namespace of the host. This applies for both the gremlin daemon container as well as when running <span class="code-class-custom">gremlin experiment-container</span>. Note that by assuming the user namespace of the host, we are creating an exception to backspace isolation for the Docker containers running Gremlin.

For running the Gremlin daemon in a container


docker run -d \
    --userns-remap=host \
    -v /var/lib/gremlin:/var/lib/gremlin \
    -v /var/log/gremlin:/var/log/gremlin \
    gremlin/gremlin daemon

For running the Gremlin daemon on the host


echo "GREMLIN_BYPASS_USERNS_REMAP=1" | sudo tee -a /etc/default/gremlind
sudo systemctl restart gremlind

For running a Gremlin experiment from the command line


gremlin attack-container 38dbd9016529 cpu

SELinux and Gremlin in Containers

Gremlin performs some actions that are not allowed by the default SELinux process label for containers (<span class="code-class-custom">container_t</span>):

  • Install and manipulate files on the host: <span class="code-class-custom">/var/lib/gremlin</span>, <span class="code-class-custom">/var/log/gremlin</span>
  • Load kernel modules for manipulating network transactions during network experiments, such as <span class="code-class-custom">net_sch</span>
  • Communicate with the container runtime socket (e.g. <span class="code-class-custotm">/var/run/docker.sock</span>) to launch containers that carry out experiments
  • Read files in <span class="code-class-custom">/proc</span>

Bypass container_t restrictions

It is possible to alleviate these restrictions on <span class="code-class-custom">container_t</span> by installing the following policy. However, this grants the privileges required by Gremlin to all other containers on your system that use <span class="code-class-custom">container_t</span>.

This is not ideal for secure environments. Gremlin recommends setting up a new process label type for Gremlin containers and granting privileges to this type only. You can find more information, including steps to configure this, at github.com/gremlin/selinux-policies..

If you wish to run Gremlin with the <span class="code-class-custom">container_t</span> process label, and bypass its restrictions, supply the following type enforcement rules into a new SELinux policy:

# WARNING: This policy adds capabilities to all containers run under the default type: container_t
# Gremlin needs access to /var/log/gremlin
allow container_t container_log_t:dir { read write create getattr setattr unlink link add_name remove_name rmdir open };
allow container_t container_log_t:file { read write create getattr setattr append unlink link open };
allow container_t var_log_t:dir { write add_name };

# Gremlin needs access to /var/run/docker.sock
allow container_t container_runtime_t:unix_stream_socket connectto;

# Gremlin needs access to /var/lib/gremlin
allow container_t container_var_lib_t:dir { read write create getattr setattr unlink link add_name remove_name rmdir open };
allow container_t container_var_lib_t:file { read write create getattr setattr append unlink link open };

# Gremlin needs to load the kernel modules: net_sch
allow container_t kernel_t:system module_request;

No items found.
This is some text inside of a div block.
Installing the Gremlin Agent
Authenticating the Gremlin Agent
Configuring the Gremlin Agent
Managing the Gremlin Agent
User Management
Health Checks
Command Line Interface
Updating Gremlin
Quick Start Guide
Services and Dependencies
Detected Risks
Reliability Tests
Reliability Score
Deploying Failure Flags on AWS Lambda
Deploying Failure Flags on AWS ECS
Deploying Failure Flags on Kubernetes
Classes, methods, & attributes
API Keys
Container security
Additional Configuration for Helm
Amazon CloudWatch Health Check
AppDynamics Health Check
Application Level Fault Injection (ALFI)
Blackhole Experiment
CPU Experiment
Certificate Expiry
Custom Health Check
Custom Load Generator
DNS Experiment
Datadog Health Check
Disk Experiment
Dynatrace Health Check
Grafana Cloud Health Check
Grafana Cloud K6
IO Experiment
Install Gremlin on Kubernetes manually
Install Gremlin on OpenShift 4
Installing Gremlin on AWS - Configuring your VPC
Installing Gremlin on Kubernetes with Helm
Installing Gremlin on Windows
Installing Gremlin on a virtual machine
Installing the Failure Flags SDK
Latency Experiment
Memory Experiment
Network Tags
New Relic Health Check
Packet Loss Attack
PagerDuty Health Check
Preview: Gremlin in Kubernetes Restricted Networks
Private Network Integration Agent
Process Collection
Process Killer Experiment
Prometheus Health Check
Role Based Access Control
Running Failure Flags experiments
Scheduling Scenarios
Shared Scenarios
Shutdown Experiment
Time Travel Experiment
Troubleshooting Gremlin on OpenShift
User Authentication via SAML and Okta
Integration Agent for Linux
Test Suites
Restricting Testing Times
Enabling DNS collection