Fault Injection

Latency Experiment

The Latency experiment injects latency into IP packets at the transport layer, targeted by supplied port and host arguments. Gremlin injects latency on a per-packet basis on layer 3 of the OCI model.


The Latency experiment uses existing Quality of Service (QOS) and Differentiated Services (diffserv) facilities in the Linux kernel to emulate packet latency.

This experiment does not interact with <span class="code-class-custom">iptables</span>, and so it does not interfere with any existing <span class="code-class-custom">iptables</span> rulesets.

This experiment requires the <span class="code-class-custom">NET_ADMIN</span> capability, which is enabled by default at installation time. See capabilities(7)


The Latency experiment uses a custom driver to emulate packet latency.

This experiment was introduced with Windows agent 1.0.11.


IP Addresses-i IP addressFalse0.0.1Only impact traffic to these IP addresses. Also accepts CIDR values (i.e.,).
Device-d interfacesFalseDevice discovery0.0.1Impact traffic over these network interfaces. Comma separated lists and multiple arguments supported. Linux-only: You can define multiple interfaces starting with agent version 2.30.0.
Hostnames-h hostnamesFalse^api.gremlin.com0.0.1Only impact traffic to these hostnames.
Remote Ports-p port numbersFalse^530.0.1Only impact outgoing traffic to these destination ports. Also accepts port ranges (e.g. 8080-8085).
Local Ports-s port numbersFalse0.0.1Only impact outgoing traffic from these source ports. Also accepts port ranges (e.g. 8080-8085).
MS-m intFalse1000.0.1How long to delay egress packets (millis).
Protocol-P {TCP, UDP, ICMP}Falseall1.5.3Only impact a specific protocol.
ProvidersWebUI and API OnlyFalse0.0.1External service providers to affect.
TagsWebUI and API OnlyFalse0.0.1Only impact traffic to hosts running Gremlin clients associated with these tags.
Length-l intFalse600.0.1The length of the experiment (seconds).


Why is the observed latency much higher than what's configured in the latency experiment?

You may sometimes observe a higher latency impact on your application than you configured in your latency experiment. Gremlin injects latency on a per-packet basis on layer 3 of the OSI model. This is representative of common networking failure modes, such as queuing at network switches or latency due to increased route distance. This contributes to higher observed latency primarily in two ways.

First, for TCP connections, if your request or response doesn't fit into a single congestion window, then latency will be added for each round trip. Applications that reuse large connection pools on reliable networks typically have very large congestion windows, so the latency is applied only once. However, if the latency increase causes requests to queue and overwhelm the existing pool, new connections may be created with new congestion windows. Each of these new connections will have to reopen the connection window over multiple round trips, applying the latency each time.

Second, applications may make multiple requests to the same dependency in serial. For example, if you send two queries to the same database, adding 100ms of latency will increase the application request time by 200ms (100ms for each request).

In both cases, the experiment is working as intended and revealed a reliability risk in the design of the application being tested.

No items found.
This is some text inside of a div block.
Installing the Gremlin Agent
Authenticating the Gremlin Agent
Configuring the Gremlin Agent
Managing the Gremlin Agent
Health Checks
Command Line Interface
Updating Gremlin
Reliability Management (RM) Quick Start Guide
Services and Dependencies
Detected Risks
Reliability Tests
Reliability Score
Deploying Failure Flags on AWS Lambda
Deploying Failure Flags on AWS ECS
Deploying Failure Flags on Kubernetes
Classes, methods, & attributes
API Keys
Container security
Additional Configuration for Helm
Amazon CloudWatch Health Check
AppDynamics Health Check
Blackhole Experiment
CPU Experiment
Certificate Expiry
Custom Health Check
Custom Load Generator
DNS Experiment
Datadog Health Check
Disk Experiment
Dynatrace Health Check
Grafana Cloud Health Check
Grafana Cloud K6
IO Experiment
Install Gremlin on Kubernetes manually
Install Gremlin on OpenShift 4
Installing Gremlin on AWS - Configuring your VPC
Installing Gremlin on Kubernetes with Helm
Installing Gremlin on Windows
Installing Gremlin on a virtual machine
Installing the Failure Flags SDK
Latency Experiment
Memory Experiment
Network Tags
New Relic Health Check
Packet Loss Attack
PagerDuty Health Check
Preview: Gremlin in Kubernetes Restricted Networks
Private Network Integration Agent
Process Collection
Process Killer Experiment
Prometheus Health Check
Configuring Role Based Access Control (RBAC)
Running Failure Flags experiments
Scheduling Scenarios
Shared Scenarios
Shutdown Experiment
Time Travel Experiment
Troubleshooting Gremlin on OpenShift
User Authentication via SAML and Okta
Managing Users and Teams
Integration Agent for Linux
Test Suites
Restricting Testing Times
Process Exhaustion Experiment
Enabling DNS collection
Authenticating Users with Microsoft Entra ID (Azure Active Directory) via SAML
AWS Quick Start Guide
Installing Gremlin on Amazon ECS