Dynatrace Integration

Gremlin offers two ways of integrating with Dynatrace:

1. The Gremlin Dynatrace integration sends Dynatrace Events whenever you run an experiment, Scenario, or reliability test.
2. The Gremlin Dynatrace app links your Gremlin services to your Dynatrace services, letting you run reliability tests, view reliability scores, and identify problems, all within the Dynatrace UI.

‍

Dynatrace integration

Gremlin's Dynatrace integration sends Dynatrace Events whenever you run an experiment, Scenario, or reliability test. This lets you overlay test events on top of your dashboards to pinpoint exactly how and when Gremlin experiments occur, and how they impact your metrics.

‍

Data transmitted

The Dynatrace integration only sends events to Dynatrace when experiments are started or stopped in Gremlin.‍

Events are of type CUSTOM_INFO and are attached (via entitySelector) to either Kubernetes objects or direct EntityIds on UserDefinedServices. To tag UserDefinedServices with a Dynatrace EntityId: add a tag with key dynatrace-entity-id and value of the EntityId (eg. CLOUD_APPLICATION-B034359570C760BD).

An example of the data in Dynatrace:

{
   "correlationId" : "d336fad89390f85a",
   "endTime" : 1757008702661,
   "entityId" : {
      "entityId" : {
         "id" : "CLOUD_APPLICATION-D55F3C830196F516",
         "type" : "CLOUD_APPLICATION"
      },
      "name" : "my-example-deployment"
   },
   "eventId" : "-7216144416740554839_1757007801591",
   "eventType" : "CUSTOM_INFO",
   "properties" : [
      {
         "key" : "dt.event.title",
         "value" : "Gremlin Attack"
      },
      {
         "key" : "gremlin-event-type",
         "value" : "START"
      },
      {
         "key" : "gremlin-experiment-id",
         "value" : "e6522650-89b6-12f0-8ca8-262b5758a7e4"
      },
      {
         "key" : "gremlin-impact-type",
         "value" : "CPU"
      },
      {
         "key" : "gremlin-team-id",
         "value" : "a7352a6b-a9a0-513c-81e4-980f680a70c1"
      }
   ],
   "startTime" : 1757007801593,
   "status" : "CLOSED",
   "title" : "Gremlin Attack",
}

‍

‍

How to set up the integration

To enable the Dynatrace integration:

1. Create a valid Dynatrace access token or a valid personal access token (see Dynatrace API - Tokens and Authentication for more information). This token must include the Read Events permission to allow Gremlin to query existing event data.
2. Log into Gremlin and navigate to your Company Settings.
3. Under Integrations, click Add next to the Dynatrace icon.
4. Enter your access token in the Dynatrace API key field.
5. Enter the hostname (and protocol) in the Dynatrace Url field (should look like: https://{your-environment-id}.live.dynatrace.com/api/v2 or https://{your-activegate-domain}:9999/e/{your-environment-id}/api/v2/)
6. Click Save.

‍

Dynatrace app

The Gremlin Dynatrace app links your Gremlin services to your Dynatrace services so you can interact with your Gremlin environment directly from Dynatrace. With the app, you can:

• View your Gremlin services next to their corresponding Dynatrace services.
• See each service’s reliability score and test history.
• View test results side-by-side with Dynatrace metrics such as response time, request volume, and failure rate.
• Identify Problems and their potential impact on service reliability

The Gremlin app is distributed as a Dynatrace community app, which means it is not visible in the Dynatrace Hub by default. You make it visible by adding a Hub subscription to your environment, then installing it like any other app. Because the app calls the Gremlin API, you also need to add Gremlin’s API host to your environment’s outbound allowlist.

‍

Setting up the Dynatrace app

Before you begin, you will need:

• A Dynatrace user account with read and write access to Dynatrace settings. This is needed to add the Hub subscription and to edit the external requests allowlist.
• An active Gremlin account and API key.
• A Gremlin and DynatraceOne agent deployed to the infrastructure you want to test.

The setup process will involve:

1. Adding the Community apps Hub subscription.
2. Installing theGremlin Reliability Management app from the Hub.
3. Allowing outbound connections to api.gremlin.com.
4. Connecting the app with a Gremlin API key.

‍

1. Subscribing to the Community apps Hub

The app is distributed through the Dynatrace Community Hub, which requires a Hub subscription channel. If you aren't already subscribed to the Community Hub, follow these steps:

1. Open the Settings app from the Dynatrace launcher.
2. Navigate to General > Hub subscriptions.
3. Select Add subscription.
4. In the Name of subscription field, enter "Communtiy apps."
5. In the Channel ID field, enter 0ddb5501-3bed-44e1-9ac5-04926f228ac0.
6. Optionally, add a description, such as: "Used to install community apps such as Gremlin Reliability Management. For more information visit https://dt-url.net/community-apps."
7. Select Save.

‍

2. Install the Gremlin app from the Hub

To add the app to your Dynatrace workspace:

1. Log into Dynatrace.
2. Open the Hub app.
3. Search for “Gremlin." If multiple options appear, select Gremlin Reliability Management.
4. Click Install.

‍

3. Allow outbound access to the Gremlin API

Dynatrace blocks external requests from app functions by default as a security measure. The Gremlin app’s functions call the Gremlin API, so those calls are blocked until you add Gremlin’s API to your environment’s allowlist. For the full Dynatrace reference on this setting, see Allow outbound connections.

‍

To allow outbound connections to the Gremlin API:

1. Open the Settings app in your Dynatrace workspace.
2. Select General in the side menu. Under Environment management, select External requests.
3. Add the following hosts:
   
   1. ‍api.gremlin.com
   2. www.dynatrace.com
4. Save your changes.

‍

4. Connect the app to your Gremlin account

The next step is to connect the app to Gremlin using your Gremlin API key. If you don't yet have an API key, follow the instructions in Creating a new API key.

1. Open Gremlin Reliability Management from the Dynatrace launcher.
2. Enter your Gremlin credentials on the settings tab.
3. Save.

‍

5. Verify the installation

As an optional last step, verify that the installation was successful by running a single low-impact reliability test (such as a CPU test) from within Dynatrace. Confirm the result appears alongside your Dynatrace metrics.

‍

Using the Dynatrace app

When you open the app, Gremlin shows a list of your services along with their name, reliability score, Problems detected in the past 30 days, and application ID. Clicking on the Actions button lets you open a link to the service in Gremlin, run a reliability test, or run all tests in your Test Suite.

‍

Viewing service details

Clicking on a service’s name reveals more detail about it. Directly below the service name are details about its deployment and its reliability score. From here, there are four tabs you can select:

• Issues lists any problems identified with this service.
• Problems uses Dynatrace’s Problems app to list recent problems. You can use the drop-down to specify a date range for the list.
• Fault Injection lists the ten most recent reliability tests performed on this service along with their start time, duration, and status (e.g. “Passed” or “Failed”).
• Metrics shows key performance charts for this service, such as CPU usage, memory usage, Pod count (for Kubernetes services), response time, requests per minute, and request failure rate.

‍

Troubleshooting

Dynatrace application ID shows “N/A”

If the Gremlin service could not be linked to a Dynatrace application, the application ID will show as “N/A.” There are several reasons why this might be the case:

• The Dynatrace connection is missing or failing. Follow the instructions in the Dynatrace integration documentation to double-check your settings.
• The service isn’t a Kubernetes target. If the service targets AWS resources, is a host or Failure Flags service, or uses application selectors, no application ID will be available.
• Gremlin hasn’t discovered the cluster yet.
• The workload name Gremlin targets doesn’t match the entity name Dynatrace has stored for it. If automatic resolution doesn’t work for your service, you can override the Dynatrace entity ID Gremlin uses by adding a dynatrace-entity-id tag to the service’s metadata.

‍

Privileges Required