Reliability Score
Supported platforms:
A reliability score is a value between 0 and 100 that indicates how well your services are meeting your team's reliability standards. The reliability score isn’t a measure of uptime or availability, but instead provides a standard way of measuring reliability across all teams and services in your organization.
How the Reliability Score is calculated
Gremlin automatically calculates reliability scores for your services based on the following:
- The number of Detected Risks discovered on the service.
- The outcomes of reliability tests run on the service.
- Successful Extra Credit Scenarios ran on the service (if Extra Credit is enabled for your company).
How reliability test scores are calculated
A Reliability Score is a value between 0 and 100 that indicates how well a service is meeting the team's reliability standards. The reliability score is calculated using the following method:
- Each reliability test is given a score between 0 and 100 based on whether your service passed or failed.
- Tests are grouped into different categories. Each category has an average score based on the tests in the category.
- The scores from each category are averaged to provide the service's overall reliability score.
Each reliability test can have one of four scores depending on whether the test was successful, failed, not yet run, or if it has expired (i.e., hasn't been run within the past week). Tests that were not run successfully (i.e., Attempted runs) don't count towards the score. The value of these scores is shown here:
Updating expired tests
If a test is marked Expired, this means it passed at one point, but hasn't been run in at least one month. To update this, you will need to re-run and pass the test. If the test fails or no action is taken, the score will remain Expired.
Test categories
Reliability tests are grouped into the following categories:
- Redundancy
- Scalability
- Dependencies
- Other
The individual test scores from each category are added up and averaged to provide the total Reliability Score. Note that certain categories may not appear if Gremlin detects that it's not relevant to the service (e.g. if it has no dependencies or no relevant detected risks). Categories that aren't listed have no impact on the final score.
Extra Credit
Extra Credit allows teams to increase a service’s weekly reliability score by running additional Scenarios that contain Health Checks on the service. Teams can earn up to ten additional points on their weekly reliability score, potentially improving their scores above 100%. Extra Credit points are additive only and will not lower or subtract from the reliability score.
To earn Extra Credit points, you must meet the following requirements:
- Your company has Extra Credit enabled (see below).
- You’re running a Scenario targeting a service.
- The Scenario includes a Health Check.
- The Scenario finishes successfully, and your service passes (i.e., no Health Checks fire).
Each successful, unique run earns one point, up to a maximum of ten points per service. Extra Credit points reset weekly.
Enabling Extra Credit
To enable Extra Credit:
- Log in to the Gremlin web app.
- Navigate to Company Settings and select Options.
- Toggle Allow Extra Credit on Reliability Scores to On.
- Click Save.
Please note that this is a company-wide change that applies to all services and will take effect immediately after saving. Extra Credit points will appear automatically in weekly scores under a new category called “Extra Credit.”
Frequently Asked Questions
Does Extra Credit replace reliability tests?
No. Reliability tests are still required. Extra Credit is additive only.
Can Extra Credit lower my score?
No. It can only increase your score.
I ran a Scenario, but didn’t earn any Extra Credit.
Ensure the Scenario has Health Checks configured. Health Checks are required to ensure meaningful validation.
Is Extra Credit available for Failure Flag services?
Failure Flag runs appear in Service History, but only qualifying scenario runs with Health Checks can earn Extra Credit.