Release Notes


Gremlin Release Notes

Below you'll find notes describing what's new with the Gremlin suite of products. Get product announcements and release notes in your inbox by joining the Gremlin Announce group.


** Host (daemon) & Client (CLI) [2.14.3]**

  • Note that 2.14.2 was never officially released
  • Minor fix: The daemon was not correctly handling the case when it started up in an un-authed state and relied on gremlin init being run to provide the .credentials file. In particular, it was missing some critical metadata which charting relied upon.
  • Minor fix: Added subheaders to gremlin check auth to better categorize the output
  • Minor fix: Read tags supplied in config.yaml
  • New feature: Ship example config.yaml to RPM/DEB packages
  • New feature: Auto-initialize daemon if secret is present and credentials are not present


Host (daemon) & Client (CLI) [2.14.1]

  • Minor fix: gremlin measure $TYPE now accepts TYPE in uppercase or lowercase (e.g. gremlin measure cpu). It previously only accepted uppercase.


Host (daemon) & Client (CLI) [2.14.0]

  • Minor fix: Kubernetes Pod eviction events triggered by Gremlin resource attacks no longer produce Failed attack states. There is now additional information when Gremlin is killed, and the steps it took to clean up.
  • Minor fix: Gremlin agents installed into Azure now properly set the publicIpAddress metadata tag (erroneously named publicpAddress in prior versions).
  • New feature: Gremlin now pushes CPU metrics for active attacks. These metrics will be used in charting features that allow you to see Gremlin's effect on your machines in real time. To disable this functionality, add PUSH_METRICS=0 to the configuration for gremlind. No data is collected when attacks aren't running, and only data relevant to the attack is collected:

    • CPU: statistics for CPU load
    • Memory: statistics for used and available memory
    • Disk: statistics for used and available space
    • IO: statistics for device IOPS
    • Shutdown: machine uptime
    • TimeTravel: system time


Host (daemon) & Client (CLI) [2.13.0]

  • Major fix: Gremlin can now compete with the resources dedicated to a container, instead of taking free resources from the host. (See more about Gremlin and Cgroups)
  • Major fix: Running attacks are now halted when the gremlind service is told to shut down from process managers
  • Minor fix: Memory Gremlin more aggressively touches memory it consumes to better ensure that operating systems don't try to reuse some of it
  • New feature: os-name tag added to clients by default; this value, in combination with os-type, now make up the full os description of the machine (i.e. os-type=Linux + os-name=Ubuntu)


Host (daemon) & Client (CLI) [2.12.27]

  • Major fix: Memory leak collecting measurement data
  • Minor fix: Ensure capabilities are correctly applied during a rollback
  • Minor improvements to I/O and Disk attack targetting
  • Improve capabilities handling overall
  • Better local IP address determination when automatically setting GREMLIN_IDENTIFIER
  • Improved shutdown handling (SIGINT, SIGTERM, and attack halt)
  • Updated dependencies


Host (daemon) & Client (CLI) [2.12.26]

  • Major fix: There was a regression in 2.12.25 where host attacks that required capabilities did not properly rollback. This release fixes that.
  • There was a regression in 2.12.23 where the value of SSL_CERT_FILE was not added to the trust store. That is properly wired into the trust store again
  • Minor fix: /var/log/gremlin/executions/{guid} was not being cleared on halts - now it is
  • Minor fix: Shutdown container attack showed an error in the logs, now this case is handled more gracefully
  • Minor fix: gremlin status was displaying UnknownVariantError in some cases
  • Updated dependencies


Host (daemon) & Client (CLI) [2.12.25]

  • Updated dependencies


Host (daemon) & Client (CLI) [2.12.24]

  • Major fix: Better handling for the case when a stateful attack doesn't get a chance to clean up properly within a container

Host (daemon) & Client (CLI) [2.12.23]

  • Major fix: address startup errors referencing number too large to fit in target type, which happens under certain configurations of the target machine's CPU.
  • Minor fix: signal handling improvements
  • Minor fix: process killer signaling improvements


Host (daemon) & Client (CLI) [2.12.22]

  • Minor fix: make file management for Gremlin logs more operating system agnostic
  • Minor fix: improve capabilities checking
  • Minor fix: update dependencies


ALFI Java Client Library [0.7.3]

  • Major fix: Change the payload of the authorization header sent to Gremlin API to resolve HTTP 401s from a server-side change that does extra certificate validation
  • Added support for HTTP proxy. Set http_proxy environment variable, and ALFI traffic to Gremlin API will use the specified proxy URL.


Host (daemon) & Client (CLI) [2.12.21]

  • Minor security fix: prevent non-privileged users from acquiring Gremlin secrets if they have command-line access to linux hosts while a container attack is running
  • Minor fix: dependency updates
  • Minor fix: supply the correct DOCKER_API_VERSION to container attacks
  • New feature: new os_type tag added to all new Gremlin clients (e.g. os_type:Linux)


Host (daemon) & Client (CLI) [2.12.20]

  • Fixed bug that caused Network Gremlins to fail when attacking two or more processes (including containers) when they shared a network interface.


Host (daemon) & Client (CLI) [2.12.19]

  • Improved memory attack performance by as much as four times while limiting the CPU impact.
  • Recover gracefully from operating system out-of-memory errors.
  • Minor status message improvements for the memory attack.


Host (daemon) & Client (CLI) [2.12.17]

  • Fixed a bug where launching a container attack was not respecting the GREMLIN_BYPASS_USERNS_REMAP environment variable. This should get set only when the Docker namespace remapping feature is being used.


Host (daemon) & Client (CLI) [2.12.16]

  • Fixed a bug where Memory Gremlin puts unnecessary strain on getrandom and therefore system entropy.


Host (daemon) & Client (CLI) [2.12.15]

  • Updated dependencies


Host (daemon) & Client (CLI) [2.12.14]

  • Fixed bug where Gremlin (in Docker only) would log errors about missing directories until it received an attack to run


Host (daemon) & Client (CLI) [2.12.13]

  • Fixed a bug where the Gremlin CPU attack would leave too much CPU in the idle and sy states. The CPU attack will now consume the requested amount, using us instead.


Host (daemon) & Client (CLI) [2.12.12]

  • Minor improvements to gremlin check functionality


Host (daemon) & Client (CLI) [2.12.11]

  • Fixed bug where Gremlin would fail attacks due to a closed HTTP stream
  • Fixed bug where Gremlin would fail to load attacks under certain circumstances


Host (daemon) & Client (CLI) [2.12.10]

  • Improved error messaging around loading authentication configuration
  • New command gremlin check for diagnostics, check out the docs


Host (daemon) & Client (CLI) [2.12.9]

  • Dependency updates


Host (daemon) & Client (CLI) [2.12.8]

  • Improve help text for Blackhole Gremlin arguments about ports
  • Dependency updates


Host (daemon) & Client (CLI) [2.12.7]

  • Fix bug where Gremlin would create /var/lib/gremlin/.credentials with permissions from the OS umask. Gremlin would then change the mode of the created file before writing to it. Now, Gremlin creates the file with proper permissions, without having to change mode later.
  • Remove world-readable bit from the /var/log/gremlin directory


Host (daemon) & Client (CLI) [2.12.6]

  • Dependency updates


Host (daemon) & Client (CLI) [2.12.5]

  • Bugfix to Memory Gremlin running in containers - we were allowing the Gremlin to allocate more memory than was given to the target container


Host (daemon) & Client (CLI) [2.12.4]

  • Bugfix to Memory Gremlin - we were letting the --percentage option consume more memory than was available


Host (daemon) & Client (CLI) [2.12.3]

  • Fewer writes by the client to the filesystem, reducing the chance that a Disk Gremlin fails


Host (daemon) & Client (CLI) [2.12.2]

  • Dependency updates


Host (daemon) & Client (CLI) [2.12.1]

  • Explicitly track tearing down successful attacks, so we don't halt attacks too early in the case teardown takes a material amount of time.


Host (daemon) & Client (CLI) [2.11.17]

  • Ensure Gremlin sidecars launched in a container have the same GREMLIN_IDENTIFIER as the daemon.
  • Dependency updates


Host (daemon) & Client (CLI) [2.11.16]

  • Make the Memory attack track its allocation time in the Initializing state.


Host (daemon) & Client (CLI) [2.11.15]

  • Dependency updates - no functional change


Host (daemon) & Client (CLI) [2.11.14]

  • Dependency updates - no functional change


Host (daemon) & Client (CLI) [2.11.12]

  • Build and CI updates - no functional change


Host (daemon) & Client (CLI) [2.11.11]

  • Build and CI updates - no functional change


Control Plane API

  • No updates

Web App

  • When running an attack against AWS instances, the attack details page has a link created automatically to the instance in the Cloudwatch metrics dashboard
  • Enter notes, observations, and a metrics link on the attack details page once an attack has completed
  • Enter a reason when When halting all or a single attack and view it in the attack details view


Host (daemon) & Client (CLI) [2.11.10]

  • Bugfix for gremlin attack-container CLI command


Host (daemon) & Client (CLI) [2.11.9]

  • Fix handling of GREMLIN_CLIENT_TAGS, which were ignored starting in 2.11.6.
  • Added more trust-store file locations


Host (daemon) & Client (CLI) [2.11.8]

  • Build and CI updates - no functional change


Host (daemon) & Client (CLI) [2.11.7]

  • Build and CI updates - no functional change


Host (daemon) & Client (CLI) [2.11.6]

  • Automatically populate client tags when running in Microsoft Azure or Google Cloud


Host (daemon) & Client (CLI) [2.11.4]

  • Bugfix for halted attacks which ended in a Lost Communication state (introduced in 2.11.2)


Host (daemon) & Client (CLI) [2.11.3]

  • Dependency updates - no functional change


Host (daemon) & Client (CLI) [2.11.2]

  • Dependency updates - no functional change intended but regression introduced where halted attacks ended in a Lost Communication state


Host (daemon) & Client (CLI) [2.11.1]

  • Automatically populate client tags with instance-id when running on AWS EC2.
  • Dependency updates


Control Plane API [1.6.47]

  • No updates

Web App

  • Resolved an issue where switching teams would not update the ALFI attacks view


Host (daemon) & Client (CLI) [2.11.0]

  • Resource CPU Attacks can now impact All cores and can consume a percentage of CPU capacity
  • Network DNS attacks now cache the IP address of the Gremlin Control Plane to avoid the attack from halting prematurely
  • Proxy details are now hidden in the attack logs on successful calls


ALFI Java Client Library [0.7.2]

  • Bugfix: Allow certificate parsing to work properly on Windows
  • Internal dependency refresh


ALFI Java Client Library [0.7.1]

  • Much friendlier error messages when installation/setup is unsuccessful


ALFI Java Client Library [0.7.0]

  • Addition of Inbound HTTP injections points, both for javax.servlet Filters and JAX-RS Filters


Host (daemon) & Client (CLI) [2.9.0]

  • No updates

Control Plane API [1.6.27]

  • No updates

Web App

  • Resolved an issue where client certificates couldn't be deleted
  • Deleting the current team is no longer possible


Host (daemon) & Client (CLI) [2.8.30]

  • No updates

Control Plane API [1.6.25]

  • When creating a Blackhole attack, port 53 is whitelisted by default
  • Resolved an issue where the Shutdown attack would fail with a start delay of 2 or more minutes

Web App

  • Auto-add feature has been deprecated and removed


Host (daemon) & Client (CLI) [2.8.29]

  • Resolved an issue when the Time Travel attack is halted and the time was not reverted accurately on the host

Control Plane API [1.6.20]

  • Resolved an issue where logins would fail if a user was a member of 10 or more teams


Host (daemon) & Client (CLI) [2.8.28]

  • Syscheck has been deprecated and removed from the client

Control Plane API [1.6.17]

  • Resolved an issue where duplicate Company names were shown on login

Web App

  • No updatets


Host (daemon) & Client (CLI) [2.8.26]

  • Resolved an issue with the Disk attack, where the client will not crash when the disk reaches 100% capacity

Control Plane API [1.6.14]

  • Resolved an issue where an SSO login without an account would show an infinite spinner

Web App

  • No updates


ALFI Java Client Library [0.6.1]

  • Internal dependency refresh


ALFI Java Client Library [0.6.0]

  • Allow chaining of property sources, so that a failure to lookup in Parameter Store still allows a lookup from environment variables


Host (daemon) & Client (CLI) [2.8.26]

  • Improved error messaging when running syscheck

Control Plane API [1.5.39]

  • No updates

Web App

  • Resolved an issue when re-running a Time Travel attack, where the time period to advance was not always accurate
  • Drop down menus in the Create Attack view are now searchable
  • Invited users who have not signed up can now be deleted
  • A notification is shown 7 days before a trial ends


ALFI Java Client Library [0.5.3]

  • Release process changes only


ALFI Java Client Library [0.5.2]

  • Change artifact location to


Host (daemon) & Client (CLI) [2.8.21]

  • Resolved an issue when running syscheck on RHEL 6.7, which no longer relies on using top to minimize dependencies

Control Plane API [1.4.38]

  • When using containers with Datadog integration, container labels are now available

Web App

  • When configuring a memory attack, provide the amount of memory to consume in either MB or GB, but not both
  • Resolved an issue where at times the number of users within a company was reported incorrectly
  • Resolved an issue with re-running time trial attacks, where the time to advance the clock was not always accurate
  • Drop downs in the create attack view are now searchable
  • The Choose a Gremlin section of the create attack view has been updated to show the categories and attack types more clearly


Host (daemon) & Client (CLI) [2.8.17]

  • While a Time Travel Attack is underway, if an NTP server futher updates the clock, the attack will not make subsequent changes and when complete the clock will be reset to the accurate value
  • Installing with RPM no longer requires sudo

Control Plane API [1.3.25]

  • Registering client tags now clears existing tags instead of appending to existing tags
  • The use of TLS 1.0 for API access is no longer allowed, TLS 1.1/1.2 are fully supported
  • Increased the allowed skew between control plane and daemon clocks from 15 seconds to 1 minute

Web App

  • Client list now includes a tab for registered ALFI clients
  • Execution errors are now available in the Attack Details view


Host (daemon) & Client (CLI) [2.8.14]

  • Resolved an issue where syscheck on a container would hang if the NET_ADMIN capability wasn't present, a relevant error message is now returned

ALFI Java Client Library [0.5.1]

  • The GREMLIN_ALFI_IDENTIFIER is required (previously was optional) when authenticating your application with Gremlin

Control Plane API [1.3.21]

  • Creating an ALFI attack that overlaps with an existing attack will fail
  • Datadog integration now supports ALFI

Web App

  • Resolved an issue where hostnames and IP addresses were shown incorrectly when cloning an attack
  • When creating an ALFI attack, previously used Application Type values will be available and searchable
  • To simplify attack creation, advanced fields will be initially hidden


Host (daemon) & Client (CLI) [2.8.13]

  • Signature-based auth using certificates is the expected authentication method, supported for hosts and containers
  • Syscheck now works with Docker containers

ALFI Java Client Library [0.5.0]

  • Install with Maven now available
  • Client library modules available individually

    • alfi-core Core library required for all ALFI functionality
    • alfi-aws [Optional] AWS integration, providing coordinate discovery for AwsLambda and AwsEc2
    • alfi-apache-http-client [Optional] ALFI injection points for Apache HTTP Client
    • alfi-aws-dynamodb-client [Optional] ALFI injection points for DynamoDB
  • AWS Parameter Store can be used for configuration

Control Plane API [1.3.6]

  • Slack integration issue resolved which was causing a timeout on registration
  • ALFI issue resolved where adding a key to ApplicationCoordinates prevented registeration

Web App

  • Resolved an issue where an attack end time was not displayed accurately
  • Certificates available for download on the team settings page for signature-based auth
  • The ALFI attack details will auto refresh until the attack is complete