Kubernetes is a container management system which is built with reliability in mind. Architecture is commonly 1 master and 2 or more nodes which are replicated from the master. When the master dies the nodes are ready to replace it. When one node dies another will be ready to replace it.
To create a Kubernetes cluster follow our guide on "How to Use and Install Kuberenetes with Weave Net".
- A Kubernetes cluster with 1 master and 2+ nodes
- A Gremlin account
After you have created your Gremlin account (sign up here) you will need to find your Gremlin Daemon credentials. Login to the Gremlin App using your Company name and sign-on credentials. These were emailed to you when you signed up to start using Gremlin.
Navigate to Team Settings and click on your Team. Click the blue Download button to save your certificates to your local computer. The downloaded certificate.zip contains both a public-key certificate and a matching private key.
Unzip the downloaded certificate.zip on your laptop and copy the files to the server you will be using with a Linux file transfer tool such as rsync, sftp or scp. Alternatively, you can store these certificates in a storage service such as AWS S3. For example:
rsync -avz /Users/tammybutow/Desktop/tammy-client.pub_cert.pem email@example.com:/var/lib/gremlin
rsync -avz /Users/tammybutow/Desktop/tammy-client.priv_key.pem firstname.lastname@example.org:/var/lib/gremlin
A Kubernetes DaemonSet will enable you to spin up a Gremlin pod on your Kubernetes master to perform your Chaos Engineering attacks. The Gremlin pod will then be replicated to your nodes.
Create the following, replacing the following items in the daemonset:
Create the following:
apiVersion: extensions/v1beta1 kind: DaemonSet metadata: name: gremlin namespace: <namespace where you want to run an attack> labels: k8s-app: gremlin version: v1 spec: template: metadata: labels: k8s-app: gremlin version: v1 spec: containers: - name: gremlin image: gremlin/gremlin args: [ "daemon" ] imagePullPolicy: Always securityContext: capabilities: add: - NET_ADMIN - SYS_BOOT - SYS_TIME - KILL env: - name: GREMLIN_TEAM_ID value: <YOUR TEAM ID GOES HERE> - name: GREMLIN_TEAM_PRIVATE_KEY_OR_FILE value: <YOUR PRIVATE KEY OR FILE LOCATION GOES HERE> - name: GREMLIN_TEAM_CERTIFICATE_OR_FILE value: <YOUR CERTIFICATE OR FILE LOCATION GOES HERE> - name: GREMLIN_IDENTIFIER valueFrom: fieldRef: fieldPath: spec.nodeName volumeMounts: - name: docker-sock mountPath: /var/run/docker.sock - name: gremlin-state mountPath: /var/lib/gremlin - name: gremlin-logs mountPath: /var/log/gremlin volumes: # Gremlin uses the Docker socket to discover eligible containers to attack, # and to launch Gremlin sidecar containers - name: docker-sock hostPath: path: /var/run/docker.sock # The Gremlin daemon communicates with Gremlin sidecars via its state directory. # This should be shared with the Kubernetes host - name: gremlin-state hostPath: path: /var/lib/gremlin # The Gremlin daemon forwards logs from the Gremlin sidecars to the Gremlin control plane # These logs should be shared with the host - name: gremlin-logs hostPath: path: /var/log/gremlin
Run the following command:
kubectl create -f daemonset.yaml
You will see the following result:
daemonset "gremlin" created
Run the following command, replacing default with your namespace
kubectl get pods --namespace default
You will see the following result:
NAME READY STATUS RESTARTS AGE carts-74f4558cb8-h9924 1/1 Running 0 4d carts-db-7fcddfbc79-v64fw 1/1 Running 0 4d catalogue-676d4b9f7c-55n4g 1/1 Running 0 4d catalogue-db-5c67cdc8cd-hvk96 1/1 Running 0 4d front-end-977bfd86-hq9x9 1/1 Running 0 4d gremlin-wkvv8 1/1 Running 0 5m gremlin-x44l4 1/1 Running 0 5m orders-787bf5b89f-xfdl6 1/1 Running 0 4d orders-db-775655b675-gv456 1/1 Running 0 4d payment-75f75b467f-4zzqs 1/1 Running 0 4d queue-master-5c86964795-t8sjg 1/1 Running 0 4d rabbitmq-96d887875-lf46w 1/1 Running 0 4d shipping-5bd69fb4cc-vprmp 1/1 Running 0 4d user-5bd9b9c468-4rms8 1/1 Running 0 4d user-db-5f9d89bbbb-r69pd 1/1 Running 0 4d
The “Hello World” of Chaos Engineering is the CPU Resource Attack. To create a CPU Resource Attack select “Resource” and then “CPU” in the dropdown menu.
The CPU Resource Attack will consume CPU resources based on the settings you select. The most popular default settings for a CPU Resource Attack are pre-selected, a default attack will utilize 1 core for 60 seconds.
Click Exact and select one of your Kubernetes nodes.
When your attack is finished it will move to Completed Attacks in the Gremlin App. To view the logs of the Attack, click on the Attack in Completed Attacks then click to the arrow to view the logs.
You've installed Gremlin on a server running Kubernetes and validated that Gremlin works by running the “Hello World” of Chaos Engineering for Kubernetes, the CPU Resource attack. You now possess tools that make it possible for you to explore additional Gremlin Attacks with Kubernetes.