Release Notes


May 1, 2024
  • Fix Improved logging in situations where gremlind cannot open or parse configuration and certificate files.
  • Info Updated dependencies
April 12, 2024
  • InfoAdd logging for troubleshooting Windows validation errors.
  • InfoBetter error message for failing to load certificates.
  • InfoUpdated dependencies.
April 3, 2024
  • NewWindows support for experiment Process Exhaustion, a way to consume processes to identify limits within the target system.
  • NewAdded DNS dependency feature to Windows installer. It will be enabled automatically if a valid Windows Pcap library is already installed.
  • FixRolling back network attacks no longer considers missing network devices as a critical error. This accounts for failure modes where the network device is torn down externally.
February 28, 2024
  • NewDuring a rollback, the gremlind process tells the associated attack told to tear down before it attempts to remove any remaining impact.
  •    InfoImproved logging in daemon.log when attacks are rolled back.
  •    InfoRaised the TCP connect timeout for API requests that transition attacks between stages from 1 second to 5 seconds.  
February 27, 2024
  • InfoAdded log messages to suggest turning on DNS collection.
February 15, 2024
  • FixAddressed an issue where rollback would fail when no teardown was required.
February 14, 2024
  • FixInstall the right Windows driver that matches the KMDF (Kernel-Mode Driver Framework) version.
February 7, 2024
  • NewAdded a new DNS-based dependency collection feature. Learn more about this feature here.
  • FixPrint full error on rollback failures.
  • InfoUpdated dependencies.
January 16, 2024
  • Fix: Apply latency with high accuracy on Windows Server 2016
January 10, 2024
  • Fix: Resolved latency issues on ESXi virtual machines
  • Info: Updated dependencies
December 7, 2023
  • New Provided Gremlin has access to a valid AWS credentials chain, it now interprets AWS ARN values in GREMLIN_TEAM_ID, GREMLIN_TEAM_SECRET, GREMLIN_TEAM_CERTIFICATE_OR_FILE, GREMLIN_TEAM_PRIVATE_KEY_OR_FILE. Gremlin supports ARN values from AWS Secrets Manager or AWS Systems Manager Parameter Store. Gremlin can optionally be supplied with GREMLIN_IAM_ROLE to specify a role to assume for the strict purpose of fetching secret values.
  • Fix More context is added to various error messages
  • Fix: Regression introduced in 1.11.0 where attacks with invalid arguments would end up Lost Communication instead of Failed
  • Info Updated dependencies
November 14, 2023
  • New Attacks can sometimes fail to notify the Gremlin Control Plane when its connection is impacted by the attack itself. The Gremlin agent now tolerates these failures more often and attempts to resend failed notifications. This fixes attacks that end up in the HaltFaled stage that would otherwise finish in the Successful stage.
November 7, 2023
  • New Improved the output of the Gremlin Agent validation routine that happens on startup. When validation fails, details about the failure are written to daemon.log.
October 20, 2023
  • Fix Fixed an issue where attacks were incorrectly labeled HaltFailed when Gremlin fails to notify api.gremlin.com during teardown of the network impact.
  • New For users running Gremlin on AWS, more error information is printed to the log file when AWS metadata cannot be retrieved.
  • Info Updated dependencies
July 7, 2023
  • Fix Errors related to spawning subprocesses now have more detailed information useful for troubleshooting.
June 30, 2023
  • Fix For hostnames supplied to network attacks, Gremlin delegates DNS queries to the operating system. When this query fails, Gremlin now attemps to resolve the name completely within the running process in an attempt to overcome operating system failures. This allows Gremlin network attacks to continue in the face of failed DNS processing.
June 9, 2023
  • Fix Fixed a bug where Latency attacks would attempt to target multiple network interfaces, which is not yet supported. An error is now returned if --device is used to specify multiple network interfaces for a Latency attack. If no --device is specified, Gremlin will choose the highest priority device it finds.
  • Info Updated dependencies.
May 10, 2023
  • Fix Gremlin now tears down the TCP connection pool with api.gremlin.com after successive timeout failures.
  • Fix Gremlin includes the name of the targeted network interface in execution log events related to applying network impact.
  • Info Updated dependencies.
March 29, 2023
  • Info Added support for tag values to be any simple YAML datatype (boolean, integer, float, string). Previously only strings were supported.
  • Info Updated dependencies.
March 23, 2023
  • Fix Fixed an issue that prevented Gremlin from ingesting Azure Tags.
  • Fix Fixed an issue that made Gremlin validation unreliable.
  • Info Updated dependencies.
March 15, 2023
  • New Gremlin's version command now prints more build information.
  • Info Updated dependencies.
March 14, 2023
  • New Multiple network interface attacks are now supported. Details are available in Network device selection.
  • Info Updated dependencies.
January 5, 2023
  • Fix Fix a bug in collect_certs when the target dropped the network connection before completing the TLS setup.
  • Fix Only consider IPv4 addresses for the default Gremlin Identifier.
  • Info Updated help URLs.
  • Info Updated dependencies.
November 22, 2022
  • Fix Fix a bug that prevented collect_certs from working when run against a container.
  • Info Updated dependencies.
November 21, 2022
  • New Add a short argument (-n) for the not_less_than option.
  • Info Updated dependencies.
November 16, 2022
  • New Introduce Certificate Expiry test for Reliability Management.
  • Info Updated dependencies.
October 28, 2022
  • Fix Fixed a bug where Gremlin would not properly launch attacks that resolve to a large amount of IP addresses / blocks.
August 31, 2022
  • Fix: Fixed a bug where Gremlin would not properly include swap in free memory calculations, leading to incorrect attack results.
  • Info: Updated dependencies.
July 29, 2022
  • Fix: Fixed a bug where Gremlin would attempt to allocate more memory than was available when the cgroup attribute memory.limit_in_bytes was higher than available system memory.
  • Info: Updated dependencies.
July 26, 2022
  • New: Gremlin's Memory attack now has a new argument: --allocation-strategy (-s), which informs Gremlin on how to interpret other memory consumption arguments: --percent, --mb, and --gb. See more at Memory: Options
  • Info: Updated dependencies.
May 23, 2022
  • Fix: Correctly handle proxy usernames and passwords that contain special characters. Special characters must be percent-escaped. For example, %5C is used in place of a backslash. Details are available here.
  • Info: Updated dependencies.
May 9, 2022
  • New: Disk attack improvements. The Disk attack is much faster, more accurate, and safer.
  • New: Gremlin now reads custom Azure tags associated with the machine and makes those tags available for targeting.
  • Fix: Fixed a bug where Gremlin would fail validation if the DNS lookup of api.gremlin.com failed. This was likely to be a problem in high security environments.
  • Fix: Fixed a bug where Gremlin would crash on some operating systems when process collection was enabled. Gremlin avoids crashing and disables process collection when errors are detected.
  • Fix: Fixed a bug where Gremlin used to skip tag configuration even when it had a valid session. Gremlin now always configures tags on startup as long as it can communicate with the Gremlin control plane.
  • Fix: Some automatic Azure tags were not being correctly read. The azEnvironment, location, name, osType, privateIpAddress, publicIpAddress, sku, vmId, vmSetScaleName, and zone tags are all automatically read and available for targeting.
  • Info: Removed NTP timestamp from gremlin check os.
  • Info: Updated dependencies.
January 21, 2022
  • Info: Updated Rust version used to build agent in response to CVE-2022-21658
December 3, 2021
  • New: On startup, the Gremlin agent now performs some validation on its ability to run a CPU and Latency attack. Validation results are accessible through the Clients API.
  • Info: Updated dependencies
November 16, 2021
  • Info: Updated dependencies
October 25, 2021
  • New: The Disk attack has been significantly improved. In most cases it is much faster, more accurate, and safer. It also uses significantly less CPU and RAM when filling disk volumes. The improved version is used when the environment variable GREMLIN_DEUCHAINN_EN1023 is set to true; all other values are treated as false. This environment variable may be ignored or removed in a future version without notice.
October 20, 2021
  • Fix: Fixed bug with Gremlin's IO attack cleanup when --mode r or --mode w was used. Previously, Gremlin would try to tear down files that did not exist, leading to attack failures.
  • Info: Improve messages reported by the Gremlin IO attack, when file-creation errors occur.
October 8, 2021
  • Info: Changed the way PUSH_METRICS boolean configuration variable is evaluated. Previously, any non-empty value other than "0" would evaluate to true (e.g. PUSH_METRICS=false would evaluate to true). This has been changed to provide expected outcomes: the only values that evaluate true are now "1", "true", and "TRUE", leaving all other values to evaluate to false.
  • Info: Updated dependencies
September 27, 2021
  • Fix: Fixed a bug where the Gremlin agent does not properly roll back time travel attacks with an offset of 5 seconds or less.
September 10, 2021
  • New: The percent argument for Disk attacks now accepts real numbers. For example, --percent 27.5 was previously unsupported.
  • Fix: Gremlin now correctly determines the local hostname making the automatic local-hostname available for targetting.
  • Fix: API interactions made by the Gremlin agent now always send the appropriate Content-Type header value.
  • Info: Updated dependencies
August 19, 2021
  • Fix: Fixed a bug where the CPU attack would not affect all processors on systems with more than 64 processors.
  • Info: Updated dependencies
August 2, 2021
  • Info: Updated dependencies
July 15, 2021
  • Fix: This update fixes Memory attack bugs. Previously, the amount of memory consumed would be limited to RAM. Memory attacks with this update include swap space / all virtual memory.
  • New: The Memory attack is more "aggressive" in the sense that the memory allocated by Gremlin during the attack is more difficult to swap to disk.
June 29, 2021
  • Info: Updated dependencies.
June 15, 2021
  • New: The Gremlin CLI now has a gremlin check daemon subcommand which reports on the status of any running Gremlin agent.
  • Info: Updated dependencies.
April 29, 2021
  • Info Updated dependencies.
April 26, 2021
  • Info Updated dependencies.
April 14, 2021
  • Fix While never observed, according to the Windows API documentation, getting the current username can fail. If that happened the Gremlin Client would fail to run an attack. Instead, this version resorts to using "unknown" if the username cannot be determined.
April 5, 2021
  • New Daemon log file management improvements. Previously, the log file was truncated at midnight. That made troubleshooting difficult. The log file is now rolled when it reaches approximately 1 MiB. Ten compressed log files are kept. With this update the current log file typically captures several days and the compressed log files typically capture a few weeks at a modest cost of approximately 2 MiB of disk space.
March 17, 2021
  • Fix Fix a bug in Gremlin's argument parsing for the hostnames and ipaddresses arguments for network attacks.
March 12, 2021
  • Fix Improve command-line argument parsing by providing better error messages and catching more edge cases related to illegal inputs.
March 4, 2021
  • Fix Patch a vulnerability in a 3rd party library that posed a variety of memory corruption scenarios, most likely use-after-free.
  • Info Improve error messages among network attacks when an invalid network device is supplied. Error message now includes all valid devices.
  • Info Drop invalid targeting tags with a warning.
February 18, 2021
  • New The daemon version is included in the gremlin check report.
  • Fix Occasionally the Docker version was incorrectly parsed which would result in the classic driver being used for container attacks.
February 8, 2021
  • New Some agent API traffic is now consolidated into fewer requests.
January 27, 2021
  • New Some agent API traffic is now gzip-compressed, reducing network overhead on machines where Gremlin is installed.
January 12, 2021
  • Fix Patch a vulnerability in a 3rd party library that posed a potential buffer overflow scenario
  • Fix Patch a vulnerability in a 3rd party library that posed a potential scenario to operate on dangling memory references
December 11, 2020
  • New You can now specify the SSL_CERT_FILE variable via the config.yml file. See the advanced configuration page for details on how to use it.
December 7, 2020
  • Fix Gremlin now properly interprets escaped newline characters \n for values of the GREMLIN_SSL_CERT environment variable.
  • Info Gremlin now reports container and process data at a slower rate, down from every 5 seconds during active attacks (and every 10 seconds otherwise) to every 30 seconds. We've found that this data changes much less frequently than is justified for a 5-10 second interval. This should result in significantly reduced network overhead required to run Gremlin.
  • Info Updated dependencies
November 20, 2020
  • Fix The Gremlin agent now writes a message to daemon.log when attacks finish. This provides observers of this log with an approximation on when attacks have ended.
  • Info Updated dependencies
November 6, 2020
  • New Gremlin correctly reports Windows 2019 (and later)
  • Fix Gremlin can be removed if the kernel driver is stopped or removed by hand
This is some text inside of a div block.
Installing the Gremlin Agent
Authenticating the Gremlin Agent
Configuring the Gremlin Agent
Managing the Gremlin Agent
User Management
Health Checks
Command Line Interface
Updating Gremlin
Quick Start Guide
Services and Dependencies
Detected Risks
Reliability Tests
Reliability Score
Deploying Failure Flags on AWS Lambda
Deploying Failure Flags on AWS ECS
Deploying Failure Flags on Kubernetes
Classes, methods, & attributes
API Keys
Container security
Additional Configuration for Helm
Amazon CloudWatch Health Check
AppDynamics Health Check
Blackhole Experiment
CPU Experiment
Certificate Expiry
Custom Health Check
Custom Load Generator
DNS Experiment
Datadog Health Check
Disk Experiment
Dynatrace Health Check
Grafana Cloud Health Check
Grafana Cloud K6
IO Experiment
Install Gremlin on Kubernetes manually
Install Gremlin on OpenShift 4
Installing Gremlin on AWS - Configuring your VPC
Installing Gremlin on Kubernetes with Helm
Installing Gremlin on Windows
Installing Gremlin on a virtual machine
Installing the Failure Flags SDK
Latency Experiment
Memory Experiment
Network Tags
New Relic Health Check
Packet Loss Attack
PagerDuty Health Check
Preview: Gremlin in Kubernetes Restricted Networks
Private Network Integration Agent
Process Collection
Process Killer Experiment
Prometheus Health Check
Role Based Access Control
Running Failure Flags experiments
Scheduling Scenarios
Shared Scenarios
Shutdown Experiment
Time Travel Experiment
Troubleshooting Gremlin on OpenShift
User Authentication via SAML and Okta
Integration Agent for Linux
Test Suites
Restricting Testing Times
Process Exhaustion Experiment
Enabling DNS collection