Infrastructure Layer

Kubernetes


Gremlin allows targeting objects within your Kubernetes clusters. After selecting a cluster, you can filter the visible set of objects by selecting a namespace. Select any of your Deployments, ReplicaSets, StatefulSets, DaemonSets, or Pods. When one object is selected, all child objects will also be targeted. For example, when selecting a DaemonSet, all of the pods within will be selected.

Installation

In addition to the Gremlin client that is installed on the host, or node, of a Kubernetes cluster, you must also install the Gremlin Kubernetes client (Chao) to the cluster. The Kubernetes client can be installed either using kubectl or helm. Both methods are outlined here.

Create a Kubernetes secret from Gremlin certificates

(Skip this step if you are using secret-based authentication)

  • Download the Gremlin certificates (you need at least team manager access)

  • Unzip certificates.zip

  • Rename the files in the certificates folder. Team Name.pub_cert.pem becomes gremlin.cert. Team Name.priv_key.pem becomes gremlin.key.

  • Create a gremlin namespace: kubectl create namespace gremlin

  • Create a kubernetes secret by running the following:

    bash
    1kubectl -n gremlin create secret generic gremlin-team-cert --from-file=/path/to/gremlin.cert --from-file=/path/to/gremlin.key

kubectl

Download and apply the Gremlin configuration manifest
  • Download the Gremlin configuration manifest by running the following:

    bash
    1wget https://k8s.gremlin.com/resources/gremlin-conf.yaml
  • Open the file and update the following:

  • Replace the following line with your team ID: “YOUR TEAM ID GOES HERE”

  • Replace the following line with your team secret: “YOUR TEAM SECRET GOES HERE”

    (If you are using certificate-based authentication, remove this line.)

  • Replace the following line with a string that you will use to identify your cluster: “YOUR UNIQUE CLUSTER NAME GOES HERE”

  • Apply the manifest with this command: kubectl apply -f /path/to/gremlin-conf.yaml

Download and apply the Gremlin client manifest

If you are using certificate-based authentication:

  • Download and apply the gremlin client manifest for your kubernetes cluster by running the following:

    bash
    1kubectl apply -f https://k8s.gremlin.com/resources/gremlin-client.yaml

If you are using secret-based authentication:

  • Download and apply the gremlin client manifest for your kubernetes cluster by running the following:

    bash
    1kubectl apply -f https://k8s.gremlin.com/resources/gremlin-client-secret.yaml
Enabling Gremlin on the Kubernetes Master

Most Kubernetes deployments configure master nodes with the node-role.kubernetes.io/master:NoSchedule taint. You can run the following command to see if any of your nodes have this taint:

shell
1kubectl get no -o=custom-columns=NAME:.metadata.name,TAINTS:.spec.taints
2NAME TAINTS
3kube-01 [map[effect:NoSchedule key:node-role.kubernetes.io/master]]
4kube-02 <none>

If you wish to install Gremlin on a Kubernetes master that has been tainted, add a tolerations section to the PodSpec of the Gremlin Client Manifest.

yaml
1tolerations:
2 - key: node-role.kubernetes.io/master
3 operator: Exists
4 effect: NoSchedule

You will need to reapply the Gremlin client manifest after making this change.

Download and apply the K8s client (Chao) manifest

If you are using certificate-based authentication:

  • Download and apply the k8s client manifest by running:

    bash
    1kubectl apply -f https://k8s.gremlin.com/resources/gremlin-chao.yaml

If you are using secret-based authentication:

  • Download and apply the k8s client manifest by running:

    bash
    1kubectl apply -f https://k8s.gremlin.com/resources/gremlin-chao-secret.yaml

Proxy Configuration

Both Gremlin and Chao can be configured to use a proxy for outgoing HTTP traffic. The conventional https_proxy and no_proxy variables can be passed as environment variables for this purpose.

Proxy Certificate Authorities

When proxies support HTTPS communication, or are otherwise configured with a TLS certificate, it can be necessary to configure Gremlin to trust the proxy’s certificate authority. This is done by passing the SSL_CERT_FILE environment variable where the value is a path on the file system to a PEM encoded file containing the certificate authority’s certificate.

Confguring Gremlin

yaml
1- name: https_proxy
2 value: http://proxy.local:3128
3# Pass SSL_CERT_FILE when the proxy requires trusting a TLS certificate
4- name: SSL_CERT_FILE
5 value: /etc/gremlin/ssl/proxy-ca.pem

Configuring Chao

Because the Gremlin Kubernetes Client (Chao) communicates with the local Kubernetes ApiServer in addition to the internet, it’s important to bypass internet proxies for traffic bound to apiserver

yaml
1- name: https_proxy
2 value: http://proxy.local:3128
3- name: no_proxy
4 value: $(KUBERNETES_SERVICE_HOST):$(KUBERNETES_SERVICE_PORT)
5# Pass SSL_CERT_FILE when the proxy requires trusting a TLS certificate
6- name: SSL_CERT_FILE
7 value: /etc/gremlin/ssl/proxy-ca.pem
8# Pass SSL_CERT_DIR when SSL_CERT_FILE contains only the proxy certificate. This will ensure Chao trusts api.gremlin.com
9# The value of SSL_CERT_DIR varies depending on the operating system on which the cluster hosts run
10# See https://www.gremlin.com/docs/infrastructure-layer/kubernetes/#ssl_cert_dir
11- name: SSL_CERT_DIR
12 value: /etc/ssl/
SSL_CERT_DIR

Supplying SSL_CERT_DIR ensures Chao is still configured with the necessary certificate authories to trust api.gremlin.com. However it is not needed for most Gremlin installations because Chao will trust Gremlin servers by default. This variable is only required for Chao deployments when both of the following conditions are true:

  • Chao is configured with https_proxy and this proxy is configured to accept TLS connections
  • Chao is also configured with SSL_CERT_FILE, and the file it points to contains only the certificate authority for the https proxy

The value of SSL_CERT_DIR should point to the root of the certificate authority directory for the operating system on which Chao runs.

PathOS
/etc/ssl/certs/Debian/Ubuntu
/etc/pki/tls/Fedora/RHEL 6/OpenELEC
/etc/ssl/OpenSUSE / Alpine Linux
/etc/pki/ca-trust/extracted/pem/CentOS/RHEL 7

Helm

Let Gremlin know your Gremlin team ID and your Kubernetes cluster name

bash
1GREMLIN_TEAM_ID="changeit"
2GREMLIN_CLUSTER_ID="changeit"

Add the Gremlin helm chart

bash
1helm repo remove gremlin
2helm repo add gremlin https://helm.gremlin.com

Create a namespace for the Gremlin Kubernetes client

bash
1kubectl create namespace gremlin

Install the Gremlin Kubernetes client

shell
1helm install gremlin/gremlin \
2 --name gremlin \
3 --namespace gremlin \
4 --set gremlin.secret.managed=true \
5 --set gremlin.secret.type=secret \
6 --set gremlin.secret.teamID=$GREMLIN_TEAM_ID \
7 --set gremlin.secret.clusterID=$GREMLIN_CLUSTER_ID \
8 --set gremlin.secret.teamSecret=$GREMLIN_TEAM_SECRET

Verify your Installation

Last you need to check that Gremlin is installed properly

bash
1kubectl get pods -n gremlin

This should list a Gremlin agent per node (physical/virtual machine in your cluster) plus one for chao

Example

shell
1kubectl get pods -n gremlin
2
3NAME READY STATUS RESTARTS AGE
4chao-78bbc7cbf6-9hn7q 1/1 Running 0 5d20h
5gremlin-9r4t7 1/1 Running 0 5d20h
6gremlin-bwmtz 1/1 Running 1 126d
7gremlin-bx6dn 1/1 Running 0 5d20h

Pending Pods

If any pods are pending this means your installation is incomplete and you should contact your cluster administrator to debug why you are unable to run gremlin on those nodes

shell
1kubectl get pods -n gremlin
2
3NAME READY STATUS RESTARTS AGE
4chao-78bbc7cbf6-9hn7q 1/1 Running 0 5d20h
5gremlin-c25ld 0/1 Pending 0 112d
6gremlin-n5gt7 0/1 Pending 0 112d
7gremlin-zn4kq 1/1 Running 0 126d

Running an attack

Once you select the Kubernetes objects to be targeted, select and configure your desired Gremlin attack. When the attack is run, the underlying containers within the objects selected will be impacted.

Troubleshooting

Run Chao in Debug Mode

Chao supports the GODEBUG environment variable, which can be used to enable debug features such as verbose logging of HTTP activity. You can enable verbose http logs by adding the following variable to the environment section of the Chao deployment.

NOTE: Verbose logging prints sensitive information like HTTP request and response bodies. This configuration is intended to be a troubleshooting measure only, and should be removed when unused.

yaml
1- name: GODEBUG
2 value: http2debug=2

Chao’s logs will now contain verbose logs for http requests.

Run Gremlin Checks

Gremlin’s check subcommand can be run on Kubernetes clusters in order to troubleshoot common issues with configuration or compatibility with the environment. The following is an example Job that can be run to get gremlin check output

yaml
1apiVersion: batch/v1
2kind: Job
3metadata:
4 name: gremlin-check
5 namespace: gremlin
6 labels:
7 k8s-app: gremlin
8 version: v1
9spec:
10 template:
11 metadata:
12 labels:
13 app.kubernetes.io/name: gremlin-check
14 spec:
15 restartPolicy: Never
16 containers:
17 - name: gremlin
18 image: gremlin/gremlin
19 # You can also pass subcommands (like `proxy` to check only proxy information)
20 args: [ "check" ]
21 env:
22 # # Pass the same environment you would pass to the Gremlin DaemonSet, including secrets, and proxy information
23 - name: GREMLIN_TEAM_PRIVATE_KEY_OR_FILE
24 value: file:///var/lib/gremlin/cert/gremlin.key
25 - name: GREMLIN_TEAM_CERTIFICATE_OR_FILE
26 value: file:///var/lib/gremlin/cert/gremlin.cert
27 - name: GREMLIN_IDENTIFIER
28 valueFrom:
29 fieldRef:
30 fieldPath: spec.nodeName
31 # # Example proxy configuration
32 # - name: https_proxy
33 # value: http://my-proxy:3128
34 # - name: SSL_CERT_FILE
35 # value: /etc/gremlin/ssl/proxy-ca.pem
36 # - name: GREMLIN_TEAM_ID
37 # value: my-team-id
38 volumeMounts:
39 - name: docker-sock
40 mountPath: /var/run/docker.sock
41 - name: gremlin-state
42 mountPath: /var/lib/gremlin
43 - name: gremlin-logs
44 mountPath: /var/log/gremlin
45 - name: gremlin-cert
46 mountPath: /var/lib/gremlin/cert
47 readOnly: true
48 # # Example proxy configuration
49 # - name: proxy-ca
50 # mountPath: /etc/gremlin/ssl
51 volumes:
52 - name: docker-sock
53 hostPath:
54 path: /var/run/docker.sock
55 - name: gremlin-state
56 hostPath:
57 path: /var/lib/gremlin
58 - name: gremlin-logs
59 hostPath:
60 path: /var/log/gremlin
61 - name: gremlin-cert
62 secret:
63 secretName: gremlin-secret
64 # # Example proxy configuration
65 # - name: proxy-ca
66 # configMap:
67 # name: proxy-ca
68 backoffLimit: 4

Once deployed, you can get the output of gremlin check by pulling the logs of the Pod associated with the Job:

shell
1kubectl logs --follow \
2 --namespace gremlin \
3 $(kubectl get pods --namespace gremlin --selector=job-name=gremlin-check --output=jsonpath='{.items[*].metadata.name}')
1proxy
2====================================================
3https_proxy : http://proxy.local:3128
4http_proxy : (unset)
5SSL_CERT_FILE : /etc/gremlin/ssl/proxy-ca.pem
6Service Ping : OK