How to install and use Gremlin locally with Kubernetes

This tutorial explains how to install and use Gremlin locally with Kubernetes. You will run a Chaos Engineering experiment using Gremlin to inject packet loss to a Kubernetes pod running the frontend of a microservices e-commerce store.

Step 1.0 – Install Docker For Mac

First you will need to install Docker For Mac if you do not yet have it on your local computer, follow the instructions provided by Docker. Next enable Kubernetes, by clicking Enable Kubernetes and Show system containers (advanced). Then click apply:

install k8s

Step 2.0 – Confirm your local Kubernetes cluster has been created

Run the following command:

kubectl cluster-info

You will see the following output:

Kubernetes master is running at https://localhost:6443

KubeDNS is running at https://localhost:6443/api/v1/namespaces/kube-system/services/kube-dns:dns/proxy

Step 3.0 – Create a local Gremlin directory

Next create a folder on your Desktop to store files you will need for Kubernetes and Gremlin authentication.

$ cd Desktop
$ mkdir gremlin
$ cd gremlin

Step 4.0 – Setup a Kubernetes Add-On for Networking Features and Policy

Save the Weave Net yaml file to your Gremlin directory:

$ curl -o weave.yaml https://cloud.weave.works/k8s/v1.8/net.yaml

Apply the Kubernetes daemonset by running the following command:

$ kubectl apply -f weave.yaml

You will see the following result:

serviceaccount "weave-net" created
clusterrole "weave-net" created
clusterrolebinding "weave-net" created
role "weave-net" created
rolebinding "weave-net" created
daemonset "weave-net" created

It may take a minute or two for DNS to be ready, run the following command to check for DNS to be ready:

kubectl get pods --all-namespaces

The successful result will look like this, every container should be running:

NAMESPACE     NAME                              READY     STATUS    RESTARTS   AGE
kube-system   etcd-kube-01                      1/1       Running   0          5m
kube-system   kube-apiserver-kube-01            1/1       Running   0          6m
kube-system   kube-controller-manager-kube-01   1/1       Running   0          5m
kube-system   kube-dns-6f4fd4bdf-whbhd          3/3       Running   0          6m
kube-system   kube-proxy-2hdhk                  1/1       Running   0          6m
kube-system   kube-proxy-tvhjk                  1/1       Running   0          5m
kube-system   kube-proxy-wspmv                  1/1       Running   0          5m
kube-system   kube-scheduler-kube-01            1/1       Running   0          6m
kube-system   weave-net-9ghn5                   2/2       Running   1          5m
kube-system   weave-net-lh8tq                   2/2       Running   0          5m
kube-system   weave-net-qhr25                   2/2       Running   0

Congratulations, now your Kubernetes cluster running on Ubuntu 16.04 is up and ready for you to deploy a microservices application.

Step 5.0 - Deploying the Weaveworks Microservices Sock Shop

First you will need to download the Weaveworks Microservices Sock Shop demo app to your Gremlin folder, run the following command:

git clone https://github.com/microservices-demo/microservices-demo.git

Create a namespace for your Sock Shop demo app:

kubectl create namespace sock-shop

You will see the following result:

namespace "sock-shop" created

Navigate to the microservices-demo/deploy/kubernetes folder:

cd microservices-demo/deploy/kubernetes

Next apply the demo to your kubernetes cluster:

kubectl apply -f complete-demo.yaml

Check to confirm that all the Sock Shop pods are now running:

kubectl get pods --namespace sock-shop

You will see the following result when all pods are ready, they will have the status of β€œRunning”:

NAMESPACE     NAME                                         READY     STATUS    RESTARTS   AGEdefault       carts-db-784446fdd6-kp7sm                    1/1       Running   0          1mdefault       gremlin-8xbnm                                1/1       Running   0          1mdocker        compose-74649b4db6-xdkwf                     1/1       Running   0          1mdocker        compose-api-6ff6b7fb4f-g29km                 1/1       Running   0          1mkube-system   etcd-docker-for-desktop                      1/1       Running   2          1mkube-system   kube-apiserver-docker-for-desktop            1/1       Running   2          1mkube-system   kube-controller-manager-docker-for-desktop   1/1       Running   2          1mkube-system   kube-dns-86f4d74b45-4n8b6                    3/3       Running   3          1mkube-system   kube-proxy-dsqqf                             1/1       Running   1          1mkube-system   kube-scheduler-docker-for-desktop            1/1       Running   2          1mkube-system   weave-net-wsbk9                              2/2       Running   3          1msock-shop     carts-6cd457d86c-x6vjs                       1/1       Running   0          1msock-shop     carts-db-784446fdd6-dxkvx                    1/1       Running   0          1msock-shop     catalogue-779cd58f9b-hk4vr                   1/1       Running   0          1msock-shop     catalogue-db-6794f65f5d-7bzr4                1/1       Running   0          1msock-shop     front-end-679d7bcb77-m2995                   1/1       Running   0          1msock-shop     orders-755bd9f786-w46z4                      1/1       Running   0          1msock-shop     orders-db-84bb8f48d6-nfzlq                   1/1       Running   0          1msock-shop     payment-674658f686-6br5w                     1/1       Running   0          1msock-shop     queue-master-5f98bbd67-gfhxx                 1/1       Running   0          1msock-shop     rabbitmq-86d44dd846-sqt7f                    1/1       Running   0          1msock-shop     shipping-79786fb956-z8xxg                    1/1       Running   0          1msock-shop     user-6995984547-lk9dg                        1/1       Running   0          1msock-shop     user-db-fc7b47fb9-xsqzw                      1/1       Running   0          1m

Visit http://localhost:30001/ to see the Sock Shop running:

sock shop

Step 6.0 – Set up your Gremlin credentials

After you have created your Gremlin account (sign up here) you will need to get your Gremlin Daemon credentials. Login to the Gremlin App using your Company name and sign-on credentials. These details were emailed to you when you signed up to start using Gremlin. Navigate to Company Teams Settings and click on your Team. Click the blue Download button to get your Team Certificate. The downloaded certificate.zip contains both a public-key certificate and a matching private key.

Unzip the certificate.zip and save it to your gremlin folder on your desktop. Rename your certificate and key files to gremlin.cert and gremlin.key.

gremlin directory

Next you will need to create the Kubernetes Daemonset that will be used to deploy your Gremlin pods.

Now you will need to create a secret which will be used to deploy your Gremlin pods, run this command from gremlin folder on your Desktop:

kubectl create secret generic gremlin-team-cert --from-file=./gremlin.cert --from-file=./gremlin.key

Navigate to your Gremlin Team Settings and copy your Team ID, you will need it for the next step.

gremlin settings

Create a gremlin.yaml file:

vim gremlin.yaml

Add the following information to your gremlin.yaml file, you will only need to change your Gremlin Team ID below. Replace Bd80e6da-8266-5534-80c1-ab7dce8dcc17 with your own Gremlin Team ID.

apiVersion: extensions/v1beta1
kind: DaemonSet
metadata:
  name: gremlin
  namespace: default
  labels:
    k8s-app: gremlin
    version: v1
spec:
  template:
    metadata:
      labels:
        k8s-app: gremlin
        version: v1
    spec:
      # If you want to enable host-level process-killing, add this flag:
      #hostPID: true
      # If you want to enable host-level network attacks, add this flag:
      #hostNetwork: true
      containers:
      - name: gremlin
        image: gremlin/gremlin
        args: [ "daemon" ]
        imagePullPolicy: Always
        securityContext:
          capabilities:
            add:
              - NET_ADMIN
              - SYS_BOOT
              - SYS_TIME
              - KILL
        env:
          - name: GREMLIN_TEAM_ID
            value: bd80e6da-8266-5534-80c1-ab7dce8dcc17
          - name: GREMLIN_TEAM_CERTIFICATE_OR_FILE
            value: file:///var/lib/gremlin/cert/gremlin.cert
          - name: GREMLIN_TEAM_PRIVATE_KEY_OR_FILE
            value: file:///var/lib/gremlin/cert/gremlin.key
          - name: GREMLIN_IDENTIFIER
            valueFrom:
              fieldRef:
                fieldPath: spec.nodeName
        volumeMounts:
          - name: docker-sock
            mountPath: /var/run/docker.sock
          - name: gremlin-state
            mountPath: /var/lib/gremlin
          - name: gremlin-logs
            mountPath: /var/log/gremlin
          - name: shutdown-trigger
            mountPath: /sysrq           
          - name: gremlin-cert
            mountPath: /var/lib/gremlin/cert
            readOnly: true
      volumes:
        # Gremlin uses the Docker socket to discover eligible containers to attack,
        # and to launch Gremlin sidecar containers
        - name: docker-sock
          hostPath:
            path: /var/run/docker.sock
        # The Gremlin daemon communicates with Gremlin sidecars via its state directory.
        # This should be shared with the Kubernetes host
        - name: gremlin-state
          hostPath:
            path: /var/lib/gremlin
        # The Gremlin daemon forwards logs from the Gremlin sidecars to the Gremlin control plane
        # These logs should be shared with the host
        - name: gremlin-logs
          hostPath:
            path: /var/log/gremlin
        # If you want to run shutdown attacks on the host, the Gremlin Daemon requires a /proc/sysrq-trigger:/sysrq mount
        - name: shutdown-trigger
          hostPath:
            path: /proc/sysrq-trigger
        # Distribute the certificate and key
        - name: gremlin-cert
          secret:
            secretName: gremlin-team-cert

Deploy the Gremlin pod by running the following commands:

kubectl apply -f gremlin.yaml

Your Gremlin pod will now be running in the default namespace, run the following command to confirm:

kubectl get pods -n default

You will see the following output:

NAME                        READY     STATUS    RESTARTS   AGE
gremlin-tj6wl               1/1       Running   0          7m

Step 7.0 - Installing the Datadog agent using a Kubernetes Daemonset

To install Datadog in a Kubernetes pod you can use the Datadog Kubernetes easy one-step install. It will take a few minutes for Datadog to spin up the Datadog container, collect metrics on your existing containers and display them in the Datadog App.

datadog api key

You will simple copy the Kubernetes DaemonSet, save it as datadog-agent.yaml and then run the following command:

kubectl apply -f datadog-agent.yaml

To confirm that the Datadog agent pod is now up, run the following command:

kubectl get pods -n default

You will see the following output:

NAME                        READY     STATUS    RESTARTS   AGE
datadog-agent-4kbq8         1/1       Running   0          1m
gremlin-tj6wl               1/1       Running   0          17m

Step 8.0 - Performing a Packet Loss Attack using Gremlin

Now you are ready to start performing your Chaos Engineering experiments. The first experiment we will run will be a packet loss attack on the front-end pod for the Sock Shop.

gremlin pod attack

Next, choose a 60% packet loss attack by modifying the percent in the Gremlin App.

gremlin percent

Now refresh the Sock Shop at http://localhost:30001/ and see the impact to the UI. You will notice that none of the items in the store will load.

sock shop no items

This type of Chaos Engineering experiment enables you to see how your application handles packet loss. It also enables you to view the experience of your customer.

Conclusion

You have now successfully run a Chaos Engineering experiment using Gremlin which injected packet loss to a Kubernetes pod running the frontend of a microservices e-commerce store. Join the Chaos Engineering Slack Community to discuss how Chaos Engineering can be practiced on Kubernetes.

Avoid downtime. Use Gremlin to turn failure into resilience.

Gremlin empowers you to proactively root out failure before it causes downtime. Try Gremlin for free and see how you can harness chaos to build resilient systems.