Troubleshooting
This page contains troubleshooting instructions for errors you might encounter. If you can't find the answer to your question, check the Gremlin knowledge base for additional information.
Gremlin Agent
Unhealthy State
Check that you are only running attacks on active Gremlin agents. It's possible to run an attack on an agent in an unhealthy state but the attack may not complete.
An unhealthy
state indicates that there was an issue with the installation or configuration of the Gremlin agent. If you see a Gremlin agent in an unhealthy state or you are experiencing problems running attacks, such as receiving "Attack Interrupted" errors, refer to the Gremlin knowledge base for more information.
LostCommunication
There are several reasons an agent can lose communication to the Gremlin Control Plane. Common examples include
- Running a network based attack that affected the traffic. Ensure both
api.gremlin.com
and DNS are white-listed. - Running a CPU attack has starved
Gremlin
of the ability to compute API encryption. This is rare but it does happen.
In the event of a LostCommunication error, The Gremlin
agent will trigger its dead-man switch and cease all attacks.
Tc Error: RTNETLINK answers: File exists
This can occur on a host when running a network attack, when a previous network attack had been run and the Gremlin
agent was killed mid attack by the user, system, or other tool which did not allow Gremlin
to run garbage collection.
To solve, run gremlin rollback
.
Failed to parse execution attribute ‘pid’ for execution < HASH_STRING >
There are two non-exclusive modes of failure that can occur with this error message:
- The running version of
Gremlin
is several versions out of date- Update the
Gremlin
agent or docker image
- Update the
/var/lib/gremlin/executions
has become corrupt- Delete the file
/var/lib/gremlin/executions
- Delete the file
Kubernetes
Run Chao in debug mode
Chao supports the GODEBUG
environment variable, which can be used to enable debug features such as verbose logging of HTTP activity. You can enable verbose HTTP logs by adding the following variable to the environment
section of the Chao deployment.
NOTE: Verbose logging prints sensitive information like HTTP request and response bodies. This configuration is intended to be a troubleshooting measure only, and should be removed when no longer needed.
1- name: GODEBUG2 value: http2debug=2
Chao's logs will now contain verbose logs for http requests.
Run Gremlin checks
You can run Gremlin's check
subcommand on Kubernetes clusters to troubleshoot common configuration or compatibility issues with the environment. The following is an example Job that you can run to get gremlin check
output.
1apiVersion: batch/v12kind: Job3metadata:4 name: gremlin-check5 namespace: gremlin6 labels:7 k8s-app: gremlin8 version: v19spec:10 template:11 metadata:12 labels:13 app.kubernetes.io/name: gremlin-check14 spec:15 restartPolicy: Never16 containers:17 - name: gremlin18 image: gremlin/gremlin19 # You can also pass subcommands (like `proxy` to check only proxy information)20 args: [ "check" ]21 env:22 # # Pass the same environment you would pass to the Gremlin DaemonSet, including secrets, and proxy information23 - name: GREMLIN_TEAM_PRIVATE_KEY_OR_FILE24 value: file:///var/lib/gremlin/cert/gremlin.key25 - name: GREMLIN_TEAM_CERTIFICATE_OR_FILE26 value: file:///var/lib/gremlin/cert/gremlin.cert27 - name: GREMLIN_IDENTIFIER28 valueFrom:29 fieldRef:30 fieldPath: spec.nodeName31 # # Example proxy configuration32 # - name: https_proxy33 # value: http://my-proxy:312834 # - name: SSL_CERT_FILE35 # value: /etc/gremlin/ssl/proxy-ca.pem36 # - name: GREMLIN_TEAM_ID37 # value: my-team-id38 volumeMounts:39 - name: docker-sock40 mountPath: /var/run/docker.sock41 - name: gremlin-state42 mountPath: /var/lib/gremlin43 - name: gremlin-logs44 mountPath: /var/log/gremlin45 - name: gremlin-cert46 mountPath: /var/lib/gremlin/cert47 readOnly: true48 # # Example proxy configuration49 # - name: proxy-ca50 # mountPath: /etc/gremlin/ssl51 volumes:52 - name: docker-sock53 hostPath:54 path: /var/run/docker.sock55 - name: gremlin-state56 hostPath:57 path: /var/lib/gremlin58 - name: gremlin-logs59 hostPath:60 path: /var/log/gremlin61 - name: gremlin-cert62 secret:63 secretName: gremlin-secret64 # # Example proxy configuration65 # - name: proxy-ca66 # configMap:67 # name: proxy-ca68 backoffLimit: 4
Once deployed, you can get the output of gremlin check
by pulling the logs of the Pod associated with the Job:
1kubectl logs --follow \2 --namespace gremlin \3 $(kubectl get pods --namespace gremlin --selector=job-name=gremlin-check --output=jsonpath='{.items[*].metadata.name}')
1proxy2====================================================3https_proxy : http://proxy.local:31284http_proxy : (unset)5SSL_CERT_FILE : /etc/gremlin/ssl/proxy-ca.pem6Service Ping : OK
Docker
Non-zero exit code (137)
Docker has killed the container via kill -9
. This is often attributed to OOM issues, and is most often seen when
running a memory attack. Allocating more RAM to Docker usually solves the issue.
Non-zero exit code (1)
Unable to find local credentials file:
Gremlin
is not configured to point to the correct credentials file, usually located in/var/lib/gremlin
. Ensure the credentials file(s), either certificates of API keys, exists andGremlin
has read+write access.Permission denied (os error 13): The
Gremlin
container does not have proper filesystem permissions.Gremlin
requires write access to/var/lib/gremlin
, including the ability to create new files. Check permission on the host, and ensure write access is being passed through via docker when running theGremlin
container.
OS Error 1
This is often observed in the context of Capabilities: Unable to inherit one or more required capabilities: cap_net_admin, cap_net_raw
Solution: Add the missing required capabilities to that Docker container (full list here: https://help.gremlin.com/security/#linux-capabilities)
Example: docker run -it --cap-add=NET_ADMIN --cap-add=KILL --cap-add=SYS_TIME gremlin/gremlin syscheck
API Return codes
401
The Gremlin
agent is unable to authenticate against the API. Causes of this error are usually due to bad or missing
credentials files or certificates, or a revocation issued against the client.
Examples:
401 Unauthorized - Authorization header is missing or malformed
Client has been revoked (401 Unauthorized)
AUTH_RENEW: 401 Unauthorized
Solution:
- Ensure you have valid credentials (Certificates or API keys) in a location that
Gremlin
can read from. - Ensure
Gremlin
has proper read+write access to/var/lib/gremlin
- Remove the file
/var/lib/gremlin/.credentials
if it exists - Rerun
gremlin init
This error can also be the result of a race condition when Gremlin
daemon is being started prior to the environment
variables being exported.
In some specific cases, this error can also occur when multiple hosts or agents are configured with the same GREMLIN_IDENTIFIER
.
Common places this can occur:
- Improperly configured ECS/Kubrenettes/Mesosphere where multiple
Gremlin
agents are assigned the same virtual IP - Missing HOST meta data on AWS/GCP/Azure which causes
Gremlin
to revert to the default localhost Identifier
402
The client limit for your company or team has been reached, Gremlin
does not have a license to apply to the client.
You may terminate or revoke existing clients, or contact Gremlin Sales to increase the client limit.
403
The account, most likely trial account, has expired. Please contact Gremlin Sales to extend the trial.
408
This is most often attributed to a host having bad time data. Verify the system clock of the host and try again. If this problem persists after validating your host's system clock, contact Gremlin Support.
409
An error code of 409
indicates there is a conflicting attack running on the host. This is most often seen in the
case of one network attack running (for example, a blackhole attack) and attempting to launch a second network attack. However,
this can also occur when trying to run two concurrent network or state attacks against the same target.