This tutorial explains how to install and use Gremlin locally with Kubernetes. You will run a Chaos Engineering experiment using Gremlin to inject packet loss to a Kubernetes pod running the frontend of a microservices e-commerce store.
Before you begin this tutorial, you'll need the following:
- An Ubuntu server
- A Gremlin account (sign up here)
First you will need to install Docker For Mac if you do not yet have it on your local computer, follow the instructions provided by Docker. Next enable Kubernetes, by clicking Enable Kubernetes and Show system containers (advanced). Then click apply:
Run the following command:
You will see the following output:
Kubernetes master is running at https://localhost:6443 KubeDNS is running at https://localhost:6443/api/v1/namespaces/kube-system/services/kube-dns:dns/proxy
Next create a folder on your Desktop to store files you will need for Kubernetes and Gremlin authentication.
$ cd Desktop $ mkdir gremlin $ cd gremlin
Save the Weave Net yaml file to your Gremlin directory:
$ curl -o weave.yaml https://cloud.weave.works/k8s/v1.8/net.yaml
Apply the Kubernetes daemonset by running the following command:
$ kubectl apply -f weave.yaml
You will see the following result:
serviceaccount "weave-net" created clusterrole "weave-net" created clusterrolebinding "weave-net" created role "weave-net" created rolebinding "weave-net" created daemonset "weave-net" created
It may take a minute or two for DNS to be ready, run the following command to check for DNS to be ready:
kubectl get pods --all-namespaces
The successful result will look like this, every container should be running:
NAMESPACE NAME READY STATUS RESTARTS AGE kube-system etcd-kube-01 1/1 Running 0 5m kube-system kube-apiserver-kube-01 1/1 Running 0 6m kube-system kube-controller-manager-kube-01 1/1 Running 0 5m kube-system kube-dns-6f4fd4bdf-whbhd 3/3 Running 0 6m kube-system kube-proxy-2hdhk 1/1 Running 0 6m kube-system kube-proxy-tvhjk 1/1 Running 0 5m kube-system kube-proxy-wspmv 1/1 Running 0 5m kube-system kube-scheduler-kube-01 1/1 Running 0 6m kube-system weave-net-9ghn5 2/2 Running 1 5m kube-system weave-net-lh8tq 2/2 Running 0 5m kube-system weave-net-qhr25 2/2 Running 0
Congratulations, now your Kubernetes cluster running on Ubuntu 16.04 is up and ready for you to deploy a microservices application.
First you will need to download the Weaveworks Microservices Sock Shop demo app to your Gremlin folder, run the following command:
git clone https://github.com/microservices-demo/microservices-demo.git
Create a namespace for your Sock Shop demo app:
kubectl create namespace sock-shop
You will see the following result:
namespace "sock-shop" created
Navigate to the microservices-demo/deploy/kubernetes folder:
Next apply the demo to your kubernetes cluster:
kubectl apply -f complete-demo.yaml
Check to confirm that all the Sock Shop pods are now running:
kubectl get pods --namespace sock-shop
You will see the following result when all pods are ready, they will have the status of “Running”:
NAMESPACE NAME READY STATUS RESTARTS AGEdefault carts-db-784446fdd6-kp7sm 1/1 Running 0 1mdefault gremlin-8xbnm 1/1 Running 0 1mdocker compose-74649b4db6-xdkwf 1/1 Running 0 1mdocker compose-api-6ff6b7fb4f-g29km 1/1 Running 0 1mkube-system etcd-docker-for-desktop 1/1 Running 2 1mkube-system kube-apiserver-docker-for-desktop 1/1 Running 2 1mkube-system kube-controller-manager-docker-for-desktop 1/1 Running 2 1mkube-system kube-dns-86f4d74b45-4n8b6 3/3 Running 3 1mkube-system kube-proxy-dsqqf 1/1 Running 1 1mkube-system kube-scheduler-docker-for-desktop 1/1 Running 2 1mkube-system weave-net-wsbk9 2/2 Running 3 1msock-shop carts-6cd457d86c-x6vjs 1/1 Running 0 1msock-shop carts-db-784446fdd6-dxkvx 1/1 Running 0 1msock-shop catalogue-779cd58f9b-hk4vr 1/1 Running 0 1msock-shop catalogue-db-6794f65f5d-7bzr4 1/1 Running 0 1msock-shop front-end-679d7bcb77-m2995 1/1 Running 0 1msock-shop orders-755bd9f786-w46z4 1/1 Running 0 1msock-shop orders-db-84bb8f48d6-nfzlq 1/1 Running 0 1msock-shop payment-674658f686-6br5w 1/1 Running 0 1msock-shop queue-master-5f98bbd67-gfhxx 1/1 Running 0 1msock-shop rabbitmq-86d44dd846-sqt7f 1/1 Running 0 1msock-shop shipping-79786fb956-z8xxg 1/1 Running 0 1msock-shop user-6995984547-lk9dg 1/1 Running 0 1msock-shop user-db-fc7b47fb9-xsqzw 1/1 Running 0 1m
Visit http://localhost:30001/ to see the Sock Shop running:
After you have created your Gremlin account (sign up here) you will need to get your Gremlin Daemon credentials. Login to the Gremlin App using your Company name and sign-on credentials. These details were emailed to you when you signed up to start using Gremlin. Navigate to Company Teams Settings and click on your Team. Click the blue Download button to get your Team Certificate. The downloaded certificate.zip contains both a public-key certificate and a matching private key.
Unzip the certificate.zip and save it to your gremlin folder on your desktop. Rename your certificate and key files to gremlin.cert and gremlin.key.
Next create your secret as follows:
kubectl create secret generic gremlin-team-cert --from-file=./gremlin.cert --from-file=./gremlin.key
Before installing with Helm, be sure to configure your team secret as described in the section above.
The simplest way to install the Gremlin client on your Kubernetes cluster is to use helm. If you do not already have Helm installed, go here to get started. Once helm is installed and configured, add the gremlin repo and install the client:
helm repo add gremlin https://helm.gremlin.com
helm install --set gremlin.teamID=YOUR-TEAM-ID gremlin/gremlin
For more information on the Gremlin Helm chart, including more configuration options, check out the chart on Github.
By definition, containers of a Kubernetes Pod all share a network interface. This means when Gremlin applies a network impact to one container within a Kubernetes pod, the impact will be observed for all containers in the Pod. Note that this does not apply to containers in Pod replicas. If you attack a specific Pod replica, the effect applies to containers within that replica only, and does not apply to the rest of the replicas.
It is always recommended to target only a single container of a Pod. If you wish to exclude some containers from the network impact, reduce your blast radius by specifying ports relevant to the containers you wish to see impact.
To install Datadog in a Kubernetes pod you can use the Datadog Kubernetes easy one-step install. It will take a few minutes for Datadog to spin up the Datadog container, collect metrics on your existing containers and display them in the Datadog App.
You will simple copy the Kubernetes DaemonSet, save it as datadog-agent.yaml and then run the following command:
kubectl apply -f datadog-agent.yaml
To confirm that the Datadog agent pod is now up, run the following command:
kubectl get pods -n default
You will see the following output:
NAME READY STATUS RESTARTS AGE datadog-agent-4kbq8 1/1 Running 0 1m gremlin-tj6wl 1/1 Running 0 17m
Now you are ready to start performing your Chaos Engineering experiments. The first experiment we will run will be a packet loss attack on the front-end pod for the Sock Shop.
Next, choose a 60% packet loss attack by modifying the percent in the Gremlin App.
Now refresh the Sock Shop at http://localhost:30001/ and see the impact to the UI. You will notice that none of the items in the store will load.
This type of Chaos Engineering experiment enables you to see how your application handles packet loss. It also enables you to view the experience of your customer.
You have now successfully run a Chaos Engineering experiment using Gremlin which injected packet loss to a Kubernetes pod running the frontend of a microservices e-commerce store. Join the Chaos Engineering Slack Community to discuss how Chaos Engineering can be practiced on Kubernetes.