Search documentation
Dashboard
Reliability Management

Reliability Tests

Reliability tests test a specific behavior of your service, such as autoscaling CPU and memory, zone and host redundancy, and dependency failures. While a test is running, Gremlin continuously monitors your service's state using its Health Checks. If any of your Health Checks become unhealthy during a test, then the test is immediately halted and marked as a failure. Otherwise, it's marked as passed.

Gremlin provides the following tests, organized by category:

  • Scalability
    • CPU: Tests that your service scales as expected when CPU capacity is limited. Gremlin will consume CPU in 3 stages (50%, 75%, 90%). Estimated test length: 15 minutes.
    • Memory: Tests that your service scales as expected when memory is limited. Gremlin will increase the memory utilization of your system in 3 stages (50%, 75%, 90%). Estimated test length: 15 minutes.
  • Redundancy
    • Host: Tests resilience to host failures by immediately shutting down a randomly selected host or container. Estimated test length: 5 minutes.
    • Zone: Tests your service's availability when a randomly selected zone is unreachable from the other zones. The Gremlin zone tag is required for this test. Estimated test length: 5 minutes.
  • Dependencies
    • Failure Test: Drops all network traffic to a specific dependency. Estimated test length: 5 minutes.
    • Latency Test: Delays all network traffic to this dependency by 100ms. Estimated test length: 5 minutes.
    • Certificate Expiry Test: Opens a secure connection to your dependency, retrieves the certificate chain, and validates that no certificates expire in the next 30 days. If there is no secure connection available, and therefore no certificates, this test will pass. Estimated test length: 1 minute.

Running reliability tests

To run a reliability test, first click on the service you wish to test to open the Service Details page. From there, find the test that you wish to run and click Run. A modal window will appear asking you to confirm. To run the test, click Run again. The test will start and Gremlin will display details about the test along with its current status. On this screen, you can monitor the progress of the test and drill down into its execution details. If the test fails, you can see the cause of the failure. If the failure was caused by a Health Check, you can see which of the Health Checks triggered the failure.

Screenshot of a completed CPU reliability test

To run the full suite of tests, click the Run All button at the top of the service overview page, then click Run All Tests to confirm. Gremlin will run each test sequentially. The page will automatically refresh to show the current running test and the results of completed tests.

Running dependency tests

When you define your service in Gremlin and select its process name, Gremlin uses network traffic data to identify network resources that your service communicates with. It then lists these resources in the Dependencies section. For each dependency, Gremlin automatically creates three tests:

  • The Failure Test drops all network traffic to the dependency.
  • The Latency Test delays all network traffic to this dependency by 100ms.
  • The Certificate Expiry Test opens a secure connection to your dependency, retrieves the certificate chain, and validates that no certificates expire in the next 30 days. If there is no secure connection available, and therefore no certificates, this test will pass.

You can run these tests for each dependency and they will contribute to the service's reliability score.

A list of dependencies for a service created in the Gremlin web app.

Running a reliability test on a dependency works the same way as running a reliability test on a service. Simply click the Run button and click Run again to confirm.

Autoscheduling Reliability Tests

A consistent testing schedule is key to improving a Service's Reliability Score. You can schedule Reliability Tests to run automatically during a weekly testing window. Gremlin will run as many eligible Reliability Tests as possible during the specified window and track the scores over time so you can see how it improves with regular testing.

You can set up a schedule to run:

  • All Reliability Tests
  • Any Reliability Tests that have passed at least once
  • Any Reliability Tests that have been run at least once

To schedule Reliability Tests for a Service:

  • On the Service page, click Autoschedule (or Settings and then Scheduling).
  • Select the test that you want to schedule:
    • All Reliability Tests
    • Any Reliability Tests that have passed at least once
    • Any Reliability Tests that have been run at least one
  • Under Test Window, specify the parameters for the test window:
    • Day
    • Start hour
    • Length of window (must be at least 2 hours)
  • Click Save.