How to Install and Use Gremlin with Kubernetes

Last Updated March 15th, 2018

Kubernetes is a container management system which is built with reliability in mind.
Architecture is commonly 1 master and 2 or more nodes which are replicated from the master. When the master dies the nodes are ready to replace it. When one node dies another will be ready to replace it.

To create a Kubernetes cluster follow our guide on “How to Use and Install Kuberenetes with Weave Net”.

Prerequisites

  • A Kubernetes cluster with 1 master and 2+ nodes
  • A Gremlin account

Step 1 - Create a Kubernetes DaemonSet

A Kubernetes DaemonSet will enable you to spin up a Gremlin pod on your Kubernetes master to perform your Chaos Engineering attacks.
The Gremlin pod will then be replicated to your nodes.

To create the DaemonSet use the following, replace the following:

  • namespace: choose the namespace where you would like to run the attack, for example this would be sock-shop if you are using the Gremlin guide to creating a Kubernetes cluster on Ubuntu 16.04 with Weave Net.
  • Value for GREMLIN_TEAM_ID, obtain this in Settings within the Gremlin Control Panel
  • Value for GREMLIN_TEAM_SECRET, obtain this in Settings within the Gremlin Control Panel

Create the following:

vim daemonset.yaml
            
apiVersion: extensions/v1beta1
            kind: DaemonSet
            metadata:
              name: gremlin
              namespace: <namespace where you want to run an attack>
              labels:
                k8s-app: gremlin
                version: v1
            spec:
              template:
                metadata:
                  labels:
                    k8s-app: gremlin
                    version: v1
                spec:
                  containers:
                  - name: gremlin
                    image: gremlin/gremlin
                    args: [ "daemon" ]
                    imagePullPolicy: Always
                    securityContext:
                      capabilities:
                        add:
                          - NET_ADMIN
                          - SYS_BOOT
                          - SYS_TIME
                          - KILL
                    env:
                      - name: GREMLIN_TEAM_ID
                        value: <YOUR TEAM ID GOES HERE>
                      - name: GREMLIN_TEAM_SECRET
                        value: <YOUR SECRET GOES HERE>
                      - name: GREMLIN_IDENTIFIER
                        valueFrom:
                          fieldRef:
                            fieldPath: spec.nodeName
                    volumeMounts:
                      - name: docker-sock
                        mountPath: /var/run/docker.sock
                      - name: gremlin-state
                        mountPath: /var/lib/gremlin
                      - name: gremlin-logs
                        mountPath: /var/log/gremlin
                  volumes:
                    # Gremlin uses the Docker socket to discover eligible containers to attack,
                    # and to launch Gremlin sidecar containers
                    - name: docker-sock
                      hostPath:
                        path: /var/run/docker.sock
                    # The Gremlin daemon communicates with Gremlin sidecars via its state directory.
                    # This should be shared with the Kubernetes host
                    - name: gremlin-state
                      hostPath:
                        path: /var/lib/gremlin
                    # The Gremlin daemon forwards logs from the Gremlin sidecars to the Gremlin control plane
                    # These logs should be shared with the host
                    - name: gremlin-logs
                      hostPath:
                        path: /var/log/gremlin
            

Run the following command:

kubectl create -f daemonset.yaml
            

You will see the following result:

daemonset "gremlin" created
            

Step 2 - Verify the creation of the Gremlin Kubernetes DaemonSet

Run the following command:

kubectl get pods --namespace sock-shop
            

You will see the following result:

NAME                            READY     STATUS    RESTARTS   AGE
            carts-74f4558cb8-h9924          1/1       Running   0          4d
            carts-db-7fcddfbc79-v64fw       1/1       Running   0          4d
            catalogue-676d4b9f7c-55n4g      1/1       Running   0          4d
            catalogue-db-5c67cdc8cd-hvk96   1/1       Running   0          4d
            front-end-977bfd86-hq9x9        1/1       Running   0          4d
            gremlin-wkvv8                   1/1       Running   0          5m
            gremlin-x44l4                   1/1       Running   0          5m
            orders-787bf5b89f-xfdl6         1/1       Running   0          4d
            orders-db-775655b675-gv456      1/1       Running   0          4d
            payment-75f75b467f-4zzqs        1/1       Running   0          4d
            queue-master-5c86964795-t8sjg   1/1       Running   0          4d
            rabbitmq-96d887875-lf46w        1/1       Running   0          4d
            shipping-5bd69fb4cc-vprmp       1/1       Running   0          4d
            user-5bd9b9c468-4rms8           1/1       Running   0          4d
            user-db-5f9d89bbbb-r69pd        1/1       Running   0          4d
            

Step 3 - Creating attacks using the Gremlin Control Panel

Example: Creating a CPU Attack against a Kubernetes node using the Gremlin Control Panel

You can use the Gremlin Control Panel or the Gremlin API to trigger Gremlin attacks. You can view the available range of Gremlin Attacks in Gremlin Help.

The “Hello World” of Chaos Engineering is the CPU Resource Attack. To create a CPU Resource Attack select “Resource” and then “CPU” in the dropdown menu.

newcpu

The CPU Resource Attack will consume CPU resources based on the settings you select. The most popular default settings for a CPU Resource Attack are pre-selected, a default attack will utilize 1 core for 60 seconds.

Click Exact and select one of your Kubernetes nodes.

When your attack is finished it will move to Completed Attacks in the Gremlin Control Panel. To view the logs of the Attack, click on the Attack in Completed Attacks then click to the arrow to view the logs.

Conclusion

You’ve installed Gremlin on a server running Kubernetes and validated that Gremlin works by running the “Hello World” of Chaos Engineering for Kubernetes, the CPU Resource attack. You now possess tools that make it possible for you to explore additional Gremlin Attacks with Kubernetes.