Gremlin agent resource usage
The Gremlin Agent is a lightweight binary: at idle it uses under 0.1% CPU, roughly 17–23 MiB of memory, and about 0.12 KB/s of combined network bandwidth on a 1 vCPU / 1 GiB host. Usage rises slightly during experiments (about 0.75 KB/s of bandwidth). Resource experiments run inside the target's cgroup, so they are bounded by the target container's limits. The Failure Flags sidecar is similarly light at idle but adds per-request memory and CPU when used as a proxy.
Idle agent resource usage
This table lists agent resource usage when no experiments are running. On Docker hosts, usage depends on how many containers the host runs and whether their processes are long or short-lived.
DNS collection
When DNS collection is enabled, the Gremlin Agent will send additional data and the bandwidth consumed will depend on how many dependencies are discovered. The information is gzip compressed in order to minimize network consumption. To measure the actual bandwidth consumed by Gremlin for your particular installation, we recommend using a tool such as iptraf or nethogs.
Measuring agent resource usage yourself
If you want to monitor agent usage on your systems, you can do so using these commands.
On Windows, typeperf prints every second. Use the -si option to average over longer time periods. For example, -si 29:00 averages over 29 minutes. You can include multiple typeperf counters by separating them with spaces.
Agent resource usage during experiments
Resources usage increases only slightly during an experiment, mainly due to more frequent communication between the agent and the Gremlin Control Plane.
Container resource sharing
During container and Kubernetes experiments, Gremlin runs inside the target's cgroup, so the target's CPU and memory limits also apply to Gremlin. When an experiement starts, the agent injects fault into the target container's cgroup, sharing its resource limits and namespace. When the experiment ends, the Gremlin sidecar is removed and its resources are freed.
With the systemd driver and the legacy Docker container driver, Kubernetes experiments still respect pod limits, but Gremlin's usage won't appear in cAdvisor metrics (e.g. metrics-server) and may not trigger autoscaling. Docker-only containers may fall under the host's system.slice cgroup instead of container limits.
CPU experiment resource allocation
CPU experiments size themselves to the CPU allocated to the container(s). CPU experiments are additive: a single-core host at 20% CPU running an experiment that uses 25% CPU rises to about 45%.
For containers with CPU limits, the percentage is calculated against the limit and rounded up to a whole CPU. For example, a 3500m limit becomes 4 CPUs, so a 50% experiment will use two cores.
As of agent 2.58.0, a container's available cores reflects its cgroup limits, still rounded up (i.e. a 500m limit counts as 1 CPU).
Memory experiment resource allocation
Memory experiments read the cgroup's limit and current usage to decide how much extra memory to consume.
OOMKillers can terminate container processes, including Gremlin, when they compete for resources. On Linux agent 2.13 and later, Kubernetes may evict targeted pods that exhaust their limits, deleting Gremlin with them; this can show as a Failed experiment, and Gremlin warns when targets are eviction candidates. Linux agent 2.59.0 changed memory-based reliability tests so Out of Memory (OOM) events trigger on the target application's processes, not just the Gremlin process.
Failure Flags sidecar resource usage
The Failure Flags sidecar is largely IO-bound and very light at idle. Running it as a proxy slightly increases usage.
Avoiding OOM kills and throttling
If the Failure Flags agent is being terminated by the out-of-memory (OOM) killer, or is being throttled by your platform, perform these common troubleshooting steps. Additional troubleshooting is available in Troubleshooting Failure Flags.
GOMEMLIMIT
Where memory limits are enforced, set the Go soft memory limit at or below the sidecar's memory limit to help prevent OOM kills. On Kubernetes, the docs map it to the sidecar's requests.memory with resourceFieldRef.
Pre-emptive termination
If the sidecar isn't connecting, check resource requests and limits so the container isn't being terminated early.
Google Cloud Run CPU throttling
On Google Cloud Run, under request-based billing (the default), CPU is severely limited between requests, which can disrupt sidecar communication with the control plane. Raise GREMLIN_REQUEST_TIMEOUT or use instance-based billing.
Network bandwidth and log storage
The agent needs only an outbound, TLS-encrypted connection to the Gremlin Control Plane; no inbound ports. Bandwidth rises during experiments because the agent stays in constant contact with the Control Plane. These bandwidth totals were measured over a 5 minute period:
Accessing logs
Agent logs are stored in /var/log/gremlin/daemon.log. Each fault injection is logged under /var/log/gremlin/executions by execution ID. You can check disk usage with du -sh /var/log/gremlin. On Systemd-based Linux distributions, you can also access logs using journalctl -xeu gremlind (you may need elevated privileges).