Release Notes

Overview


Gremlin Release Notes

Below you’ll find notes describing what’s new with the Gremlin suite of products. Get product announcements and release notes in your inbox by joining the Gremlin Announce group.

2020-07-15

Linux Host (daemon) & Client (CLI) [2.15.3] and Windows Host (daemon) & Client (CLI) [1.0.7]

  • Minor fix: Improve error messaging when Gremlin fails to find an IP address for a hostname supplied with the --hostname argument, which can be passed to any network attack. Error message now mentions failures due to specifying a hostname that maps to an invalid DNS record type, such as NS.

    Linux Host (daemon) & Client (CLI) [2.15.3]

  • Major fix: Gremlin was not correctly using the SSL_CERT_FILE environment variable when running attacks against containers. As a result, Gremlin would only properly trust intermediate SSL proxies if the file referenced in SSL_CERT_FILE had a path within /var/lib/gremlin. Now, this file can live anywhere on the file system, so long as Gremlin has access to it.

2020-07-07

ALFI Java Client Library [0.7.4]

  • Major fix: If the gremlin.properties file was on the classpath, Gremlin was not properly using it when resolving configuration.

2020-07-01

Linux Host (daemon) & Client (CLI) [2.15.2] and Windows Host (daemon) & Client (CLI) [1.0.6]

  • Security fix: Patch a vulnerability in a 3rd party library that posed a potential denial of service to Gremlin’s outbound https connections. In practice this is 100% mitigated unless connecting Gremlin through a malicious SSL proxy
  • Updated dependencies

2020-06-30

Linux Host (daemon) & Client (CLI) [2.15.1]

  • Major fix: Gremlin was not using the custom TLS trust store (specified by the SSL_CERT_FILE environment variable) when carrying out attacks against containers. This resulted in a failure to launch container attacks for users that rely on this configuration.
  • Minor fix: Improve accuracy of latency measurement when checking Gremlin’s connectivity to the control plane using gremlin check api. This measurement now omits the time it takes to initialize the HTTP client used to test connectivity.

2020-06-10

Linux Host (daemon) & Client (CLI) [2.15.0]

  • Gremlin can be installed with a custom group, user, and/or binary mode. The three optional environment variables GREMLIN_INSTALL_GROUP, GREMLIN_INSTALL_USER, GREMLIN_INSTALL_BIN_MODE are set before running the install to establish the security context. The defaults are unchanged: gremlin, gremlin, 6111.

    Windows Host (daemon) & Client (CLI) [1.0.5]

  • Add the DNS attack to Gremlin for Windows

2020-06-02

Linux Host (daemon) & Client (CLI) [2.14.16] and Windows Host (daemon) & Client (CLI) [1.0.4]

  • Minor fix: Added more detail to error messages that occur when Gremlin fails to do a DNS lookup of a hostname. Previously the error message did not include the reason for the lookup failure. An example of the new detail we’ve added is: failed to lookup address information: Name does not resolve.

2020-05-27

Linux Host (daemon) & Client (CLI) [2.14.15]

  • Major fix: Fixed a bug where Time travel attacks were not blocking the NTP port of the target, even when told to do so. Now, specifying --ntp, or checking the Block NTP box in the UI, correctly blocks all traffic to outbound NTP servers. Omitting this option still correctly allows NTP traffic on the target.

2020-05-20

Linux Host (daemon) & Client (CLI) [2.14.14]

  • Major fix: Fixed a bug where container attacks (including Kubernetes) were not properly setting attacks to ClientAborted when Gremlin’s target is killed. This fix includes displaying more information about Gremlin’s status after the target is killed.

    Windows Host (daemon) & Client (CLI) [1.0.3]

  • Major fix: Fixed a bug where the Gremlin Disk attack would not clean up the impact files it created if it was halted from the UI.

2020-05-19

Linux Host (daemon) & Client (CLI) [2.14.13]

  • Major fix: Fixed a bug in how the Gremlin Agent reports attack status when Gremlin attacks exit abnormally. In many instances, attacks were incorrectly labeled as LostCommunication when they instead failed to start (Failed), or were killed mid-attack (ClientAborted).
  • Major fix: Fixed a bug where the Gremlin Disk attack would not clean up the impact files it created if it was halted from the UI.
  • Major fix: Changed the way Debian and RPM installation scripts handle failures when adding Gremlin to the Docker Linux group. Previously, Gremlin would fail and terminate the installation if a docker Linux group was found, but could not add Gremlin to it. Now, a warning is printed instead.

2020-05-11

Windows Host (daemon) & Client (CLI) [1.0.2]

  • Minor fix: added additional validity checks on data returned from the Windows API. This improves error handling when the target system does not behave as Gremlin expected.

    Linux Host (daemon) & Client (CLI) [2.14.12]

  • Minor fix: improved the safety guarantees of the Gremlin Agent when loading attacks from the filesystem. Now, if the Gremlin Agent fails to load any attack state due to IO errors, all attacks will be halted immediately to prevent any unexpected behavior.

2020-05-06

Linux Host (daemon) & Client (CLI) [2.14.11] and Windows Host (daemon) & Client (CLI) [1.0.1]

  • We now collect an approximate host boot time, this will aid Gremlin to better recognize unique hosts on your team.
  • Minor fix: Select a default network interface in more cases (also used when Gremlin identifier isn’t specified).

2020-04-30

Windows Host (daemon) & Client (CLI) [1.0.0]

  • Initial release of Gremlin for Windows, allowing attacks to be run on Windows environments (see more about running Gremlin on Windows). Supported attack types:

    • Blackhole attack
    • Shutdown Attack
    • CPU Attack
    • Disk Attack
    • IO Attack
    • Memory Attack

    Linux Host (daemon) & Client (CLI) [2.14.10]

  • Major fix: Fix bug that prevented the Gremlin agent from reading attack state for attacks created via the CLI. This was preventing users from halting such attacks from the UI, as well as reading logs from the attack details page.

  • Minor fix: Remove attack.log files associated with attacks that get rolled back from the CLI through gremlin rollback as well as signals, such as from a Ctrl-C.

2020-04-29

Linux Host (daemon) & Client (CLI) [2.14.9]

  • Minor fix: Immediately halt and mark the attack as “Initialization Failed” if a Disk Gremlin encounters an IO error while writing the desired amount of bytes.

2020-04-27

Linux Host (daemon) & Client (CLI) [2.14.8]

  • Minor fix: update dependencies

2020-04-14

Linux Host (daemon) & Client (CLI) [2.14.7]

2020-04-08

Linux Host (daemon) & Client (CLI) [2.14.6]

  • Major fix: cap the --workers (-w) argument for Disk and IO attacks to a maximum value equal to the number of CPUs available to Gremlin. This ensures Gremlin is always busy, and not generating more threads than can be fully utilized by the machine on which Gremlin runs. This also eliminates the possibility that Gremlin will exhaust all threads available to Gremlin, which was observed with very large values supplied to --workers (1024 or higher)
  • Minor fix: update dependencies

2020-03-27

Linux Host (daemon) & Client (CLI) [2.14.5]

  • Minor fix: Improved handling of invalid auth when running gremlin attack-container
  • New feature: Better organization of output of gremlin check auth, including more information in both the success/error cases

2020-03-26

Linux Host (daemon) & Client (CLI) [2.14.4]

  • Major fix: The daemon was not properly halting attacks when it did not have access to it’s library directory: /var/lib/gremlin, even though it would allow attacks to run. Attacks are now properly halted.
  • Minor fix: users can now supply push_metrics inside config.yaml. This attribute is a boolean value that defaults to true, and is equivalent to the environment variable PUSH_METRICS=1

2020-03-20

Linux Host (daemon) & Client (CLI) [2.14.3]

  • Note that 2.14.2 was never officially released
  • Minor fix: The daemon was not correctly handling the case when it started up in an un-authed state and relied on gremlin init being run to provide the .credentials file. In particular, it was missing some critical metadata which charting relied upon.
  • Minor fix: Added subheaders to gremlin check auth to better categorize the output
  • Minor fix: Read tags supplied in config.yaml
  • New feature: Ship example config.yaml to RPM/DEB packages
  • New feature: Auto-initialize daemon if secret is present and credentials are not present

2020-03-11

Linux Host (daemon) & Client (CLI) [2.14.1]

  • Minor fix: gremlin measure $TYPE now accepts TYPE in uppercase or lowercase (e.g. gremlin measure cpu). It previously only accepted uppercase.

2020-03-06

Linux Host (daemon) & Client (CLI) [2.14.0]

  • Minor fix: Kubernetes Pod eviction events triggered by Gremlin resource attacks no longer produce Failed attack states. There is now additional information when Gremlin is killed, and the steps it took to clean up.
  • Minor fix: Gremlin agents installed into Azure now properly set the publicIpAddress metadata tag (erroneously named publicpAddress in prior versions).
  • New feature: Gremlin now pushes CPU metrics for active attacks. These metrics will be used in charting features that allow you to see Gremlin’s effect on your machines in real time. To disable this functionality, add PUSH_METRICS=0 to the configuration for gremlind. No data is collected when attacks aren’t running, and only data relevant to the attack is collected:
    • CPU: statistics for CPU load
    • Memory: statistics for used and available memory
    • Disk: statistics for used and available space
    • IO: statistics for device IOPS
    • Shutdown: machine uptime
    • TimeTravel: system time

2020-02-28

Linux Host (daemon) & Client (CLI) [2.13.0]

  • Major fix: Gremlin can now compete with the resources dedicated to a container, instead of taking free resources from the host. (See more about Gremlin and Cgroups)
  • Major fix: Running attacks are now halted when the gremlind service is told to shut down from process managers
  • Minor fix: Memory Gremlin more aggressively touches memory it consumes to better ensure that operating systems don’t try to reuse some of it
  • New feature: os-name tag added to clients by default; this value, in combination with os-type, now make up the full os description of the machine (i.e. os-type=Linux + os-name=Ubuntu)

2020-02-26

Linux Host (daemon) & Client (CLI) [2.12.27]

  • Major fix: Memory leak collecting measurement data
  • Minor fix: Ensure capabilities are correctly applied during a rollback
  • Minor improvements to I/O and Disk attack targetting
  • Improve capabilities handling overall
  • Better local IP address determination when automatically setting GREMLIN_IDENTIFIER
  • Improved shutdown handling (SIGINT, SIGTERM, and attack halt)
  • Updated dependencies

2020-02-17

Linux Host (daemon) & Client (CLI) [2.12.26]

  • Major fix: There was a regression in 2.12.25 where host attacks that required capabilities did not properly rollback. This release fixes that.
  • There was a regression in 2.12.23 where the value of SSL_CERT_FILE was not added to the trust store. That is properly wired into the trust store again
  • Minor fix: /var/log/gremlin/executions/{guid} was not being cleared on halts - now it is
  • Minor fix: Shutdown container attack showed an error in the logs, now this case is handled more gracefully
  • Minor fix: gremlin status was displaying UnknownVariantError in some cases
  • Updated dependencies

2020-02-04

Linux Host (daemon) & Client (CLI) [2.12.25]

  • Updated dependencies

2020-01-09

Linux Host (daemon) & Client (CLI) [2.12.24]

  • Major fix: Better handling for the case when a stateful attack doesn’t get a chance to clean up properly within a container

Linux Host (daemon) & Client (CLI) [2.12.23]

  • Major fix: address startup errors referencing number too large to fit in target type, which happens under certain configurations of the target machine’s CPU.
  • Minor fix: signal handling improvements
  • Minor fix: process killer signaling improvements

2020-01-02

Linux Host (daemon) & Client (CLI) [2.12.22]

  • Minor fix: make file management for Gremlin logs more operating system agnostic
  • Minor fix: improve capabilities checking
  • Minor fix: update dependencies

2019-12-23

ALFI Java Client Library [0.7.3]

  • Major fix: Change the payload of the authorization header sent to Gremlin API to resolve HTTP 401s from a server-side change that does extra certificate validation
  • Added support for HTTP proxy. Set http_proxy environment variable, and ALFI traffic to Gremlin API will use the specified proxy URL.

2019-12-02

Linux Host (daemon) & Client (CLI) [2.12.21]

  • Minor security fix: prevent non-privileged users from acquiring Gremlin secrets if they have command-line access to linux hosts while a container attack is running
  • Minor fix: dependency updates
  • Minor fix: supply the correct DOCKER_API_VERSION to container attacks
  • New feature: new os-type tag added to all new Gremlin clients (e.g. os_type:Linux)

2019-11-21

Linux Host (daemon) & Client (CLI) [2.12.20]

  • Fixed bug that caused Network Gremlins to fail when attacking two or more processes (including containers) when they shared a network interface.

2019-11-12

Linux Host (daemon) & Client (CLI) [2.12.19]

  • Improved memory attack performance by as much as four times while limiting the CPU impact.
  • Recover gracefully from operating system out-of-memory errors.
  • Minor status message improvements for the memory attack.

2019-10-29

Linux Host (daemon) & Client (CLI) [2.12.17]

  • Fixed a bug where launching a container attack was not respecting the GREMLIN_BYPASS_USERNS_REMAP environment variable. This should get set only when the Docker namespace remapping feature is being used.

2019-10-23

Linux Host (daemon) & Client (CLI) [2.12.16]

  • Fixed a bug where Memory Gremlin puts unnecessary strain on getrandom and therefore system entropy.

2019-10-21

Linux Host (daemon) & Client (CLI) [2.12.15]

  • Updated dependencies

2019-10-17

Linux Host (daemon) & Client (CLI) [2.12.14]

  • Fixed bug where Gremlin (in Docker only) would log errors about missing directories until it received an attack to run

2019-10-14

Linux Host (daemon) & Client (CLI) [2.12.13]

  • Fixed a bug where the Gremlin CPU attack would leave too much CPU in the idle and sy states. The CPU attack will now consume the requested amount, using us instead.

2019-10-09

Linux Host (daemon) & Client (CLI) [2.12.12]

  • Minor improvements to gremlin check functionality

2019-10-01

Linux Host (daemon) & Client (CLI) [2.12.11]

  • Fixed bug where Gremlin would fail attacks due to a closed HTTP stream
  • Fixed bug where Gremlin would fail to load attacks under certain circumstances

2019-09-27

Linux Host (daemon) & Client (CLI) [2.12.10]

  • Improved error messaging around loading authentication configuration
  • New command gremlin check for diagnostics, check out the docs

2019‑09‑16

Linux Host (daemon) & Client (CLI) [2.12.9]

  • Dependency updates

2019-09-09

Linux Host (daemon) & Client (CLI) [2.12.8]

  • Improve help text for Blackhole Gremlin arguments about ports
  • Dependency updates

2019-09-05

Linux Host (daemon) & Client (CLI) [2.12.7]

  • Fix bug where Gremlin would create /var/lib/gremlin/.credentials with permissions from the OS umask. Gremlin would then change the mode of the created file before writing to it. Now, Gremlin creates the file with proper permissions, without having to change mode later.
  • Remove world-readable bit from the /var/log/gremlin directory

2019-09-04

Linux Host (daemon) & Client (CLI) [2.12.6]

  • Dependency updates

2019-08-28

Linux Host (daemon) & Client (CLI) [2.12.5]

  • Bugfix to Memory Gremlin running in containers - we were allowing the Gremlin to allocate more memory than was given to the target container

2019-08-23

Linux Host (daemon) & Client (CLI) [2.12.4]

  • Bugfix to Memory Gremlin - we were letting the --percentage option consume more memory than was available

2019-08-21

Linux Host (daemon) & Client (CLI) [2.12.3]

  • Fewer writes by the client to the filesystem, reducing the chance that a Disk Gremlin fails

2019-08-21

Linux Host (daemon) & Client (CLI) [2.12.2]

  • Dependency updates

2019-08-05

Linux Host (daemon) & Client (CLI) [2.12.1]

  • Explicitly track tearing down successful attacks, so we don’t halt attacks too early in the case teardown takes a material amount of time.

2019-07-31

Linux Host (daemon) & Client (CLI) [2.11.17]

  • Ensure Gremlin sidecars launched in a container have the same GREMLIN_IDENTIFIER as the daemon.
  • Dependency updates

2019-07-26

Linux Host (daemon) & Client (CLI) [2.11.16]

  • Make the Memory attack track its allocation time in the Initializing state.

2019-07-25

Linux Host (daemon) & Client (CLI) [2.11.15]

  • Dependency updates - no functional change

2019-07-18

Linux Host (daemon) & Client (CLI) [2.11.14]

  • Dependency updates - no functional change

2019-07-11

Linux Host (daemon) & Client (CLI) [2.11.12]

  • Build and CI updates - no functional change

2019-07-10

Linux Host (daemon) & Client (CLI) [2.11.11]

  • Build and CI updates - no functional change

2019-07-09

Control Plane API

  • No updates

Web App

  • When running an attack against AWS instances, the attack details page has a link created automatically to the instance in the Cloudwatch metrics dashboard
  • Enter notes, observations, and a metrics link on the attack details page once an attack has completed
  • Enter a reason when When halting all or a single attack and view it in the attack details view

2019-07-08

Linux Host (daemon) & Client (CLI) [2.11.10]

  • Bugfix for gremlin attack-container CLI command

2019-07-02

Linux Host (daemon) & Client (CLI) [2.11.9]

  • Fix handling of GREMLIN_CLIENT_TAGS, which were ignored starting in 2.11.6.
  • Added more trust-store file locations

2019-06-27

Linux Host (daemon) & Client (CLI) [2.11.8]

  • Build and CI updates - no functional change

2019-06-27

Linux Host (daemon) & Client (CLI) [2.11.7]

  • Build and CI updates - no functional change

2019-06-25

Linux Host (daemon) & Client (CLI) [2.11.6]

  • Automatically populate client tags when running in Microsoft Azure or Google Cloud

2019-06-21

Linux Host (daemon) & Client (CLI) [2.11.4]

  • Bugfix for halted attacks which ended in a Lost Communication state (introduced in 2.11.2)

2019-06-18

Linux Host (daemon) & Client (CLI) [2.11.3]

  • Dependency updates - no functional change

2019-06-12

Linux Host (daemon) & Client (CLI) [2.11.2]

  • Dependency updates - no functional change intended but regression introduced where halted attacks ended in a Lost Communication state

2019-06-10

Linux Host (daemon) & Client (CLI) [2.11.1]

  • Automatically populate client tags with instance-id when running on AWS EC2.
  • Dependency updates

2019-05-30

Control Plane API [1.6.47]

  • No updates

Web App

  • Resolved an issue where switching teams would not update the ALFI attacks view

2019-05-29

Linux Host (daemon) & Client (CLI) [2.11.0]

  • Resource CPU Attacks can now impact All cores and can consume a percentage of CPU capacity
  • Network DNS attacks now cache the IP address of the Gremlin Control Plane to avoid the attack from halting prematurely
  • Proxy details are now hidden in the attack logs on successful calls

2019-04-24

ALFI Java Client Library [0.7.2]

  • Bugfix: Allow certificate parsing to work properly on Windows
  • Internal dependency refresh

2019-04-11

ALFI Java Client Library [0.7.1]

  • Much friendlier error messages when installation/setup is unsuccessful

2019-04-02

ALFI Java Client Library [0.7.0]

  • Addition of Inbound HTTP injections points, both for javax.servlet Filters and JAX-RS Filters

2019-03-29

Linux Host (daemon) & Client (CLI) [2.9.0]

  • No updates

Control Plane API [1.6.27]

  • No updates

Web App

  • Resolved an issue where client certificates couldn’t be deleted
  • Deleting the current team is no longer possible

2019-03-22

Linux Host (daemon) & Client (CLI) [2.8.30]

  • No updates

Control Plane API [1.6.25]

  • When creating a Blackhole attack, port 53 is excluded by default
  • Resolved an issue where the Shutdown attack would fail with a start delay of 2 or more minutes

Web App

  • Auto-add feature has been deprecated and removed

2019-03-08

Linux Host (daemon) & Client (CLI) [2.8.29]

  • Resolved an issue when the Time Travel attack is halted and the time was not reverted accurately on the host

Control Plane API [1.6.20]

  • Resolved an issue where logins would fail if a user was a member of 10 or more teams

2019-03-01

Linux Host (daemon) & Client (CLI) [2.8.28]

  • Syscheck has been deprecated and removed from the client

Control Plane API [1.6.17]

  • Resolved an issue where duplicate Company names were shown on login

Web App

  • No updatets

2019-02-22

Linux Host (daemon) & Client (CLI) [2.8.26]

  • Resolved an issue with the Disk attack, where the client will not crash when the disk reaches 100% capacity

Control Plane API [1.6.14]

  • Resolved an issue where an SSO login without an account would show an infinite spinner

Web App

  • No updates

2019-02-21

ALFI Java Client Library [0.6.1]

  • Internal dependency refresh

2019-02-12

ALFI Java Client Library [0.6.0]

  • Allow chaining of property sources, so that a failure to lookup in Parameter Store still allows a lookup from environment variables

2019-01-25

Linux Host (daemon) & Client (CLI) [2.8.26]

  • Improved error messaging when running syscheck

Control Plane API [1.5.39]

  • No updates

Web App

  • Resolved an issue when re-running a Time Travel attack, where the time period to advance was not always accurate
  • Drop down menus in the Create Attack view are now searchable
  • Invited users who have not signed up can now be deleted
  • A notification is shown 7 days before a trial ends

2019-01-22

ALFI Java Client Library [0.5.3]

  • Release process changes only

2019-01-10

ALFI Java Client Library [0.5.2]

  • Change artifact location to maven.gremlin.com

2018-12-20

Linux Host (daemon) & Client (CLI) [2.8.21]

  • Resolved an issue when running syscheck on RHEL 6.7, which no longer relies on using top to minimize dependencies

Control Plane API [1.4.38]

  • When using containers with Datadog integration, container labels are now available

Web App

  • When configuring a memory attack, provide the amount of memory to consume in either MB or GB, but not both
  • Resolved an issue where at times the number of users within a company was reported incorrectly
  • Resolved an issue with re-running time trial attacks, where the time to advance the clock was not always accurate
  • Drop downs in the create attack view are now searchable
  • The Choose a Gremlin section of the create attack view has been updated to show the categories and attack types more clearly

2018-11-15

Linux Host (daemon) & Client (CLI) [2.8.17]

  • While a Time Travel Attack is underway, if an NTP server futher updates the clock, the attack will not make subsequent changes and when complete the clock will be reset to the accurate value
  • Installing with RPM no longer requires sudo

Control Plane API [1.3.25]

  • Registering client tags now clears existing tags instead of appending to existing tags
  • The use of TLS 1.0 for API access is no longer allowed, TLS 1.1/1.2 are fully supported
  • Increased the allowed skew between control plane and daemon clocks from 15 seconds to 1 minute

Web App

  • Client list now includes a tab for registered ALFI clients
  • Execution errors are now available in the Attack Details view

2018-10-23

Linux Host (daemon) & Client (CLI) [2.8.14]

  • Resolved an issue where syscheck on a container would hang if the NET_ADMIN capability wasn’t present, a relevant error message is now returned

ALFI Java Client Library [0.5.1]

  • The GREMLIN_ALFI_IDENTIFIER is required (previously was optional) when authenticating your application with Gremlin

Control Plane API [1.3.21]

  • Creating an ALFI attack that overlaps with an existing attack will fail
  • Datadog integration now supports ALFI

Web App

  • Resolved an issue where hostnames and IP addresses were shown incorrectly when cloning an attack
  • When creating an ALFI attack, previously used Application Type values will be available and searchable
  • To simplify attack creation, advanced fields will be initially hidden

2018-10-11

Linux Host (daemon) & Client (CLI) [2.8.13]

  • Signature-based auth using certificates is the expected authentication method, supported for hosts and containers
  • Syscheck now works with Docker containers

ALFI Java Client Library [0.5.0]

  • Install with Maven now available
  • Client library modules available individually
    • alfi-core Core library required for all ALFI functionality
    • alfi-aws [Optional] AWS integration, providing coordinate discovery for AwsLambda and AwsEc2
    • alfi-apache-http-client [Optional] ALFI injection points for Apache HTTP Client
    • alfi-aws-dynamodb-client [Optional] ALFI injection points for DynamoDB
  • AWS Parameter Store can be used for configuration

Control Plane API [1.3.6]

  • Slack integration issue resolved which was causing a timeout on registration
  • ALFI issue resolved where adding a key to ApplicationCoordinates prevented registeration

Web App

  • Resolved an issue where an attack end time was not displayed accurately
  • Certificates available for download on the team settings page for signature-based auth
  • The ALFI attack details will auto refresh until the attack is complete