Infrastructure Layer

Kubernetes


Gremlin allows targeting objects within your Kubernetes clusters. After selecting a cluster, you can filter the visible set of objects by selecting a namespace. Select any of your Deployments, ReplicaSets, StatefulSets, DaemonSets, or Pods. When one object is selected, all child objects will also be targeted. For example, when selecting a DaemonSet, all of the pods within will be selected.

Installation

In addition to the Gremlin client that is installed on the host, or node, of a Kubernetes cluster, you must also install the Gremlin Kubernetes client to the cluster. The Kubernetes client can be installed either using kubectl or helm. Both methods are outlined here.

Create a Kubernetes secret from Gremlin certificates

(Skip this step if you are using secret-based authentication)

  • Download the Gremlin certificates (you need at least team manager access)

  • Unzip certificates.zip

  • Rename the files in the certificates folder. Team Name.pub_cert.pem becomes gremlin.cert. Team Name.priv_key.pem becomes gremlin.key.

  • Create a gremlin namespace: kubectl create namespace gremlin

  • Create a kubernetes secret by running the following:

    kubectl -n gremlin create secret generic gremlin-team-cert --from-file=/path/to/gremlin.cert --from-file=/path/to/gremlin.key

kubectl

Download and apply the Gremlin configuration manifest
  • Download the Gremlin configuration manifest by running the following:

    wget https://k8s.gremlin.com/resources/gremlin-conf.yaml

  • Open the file and update the following:

  • Replace the following line with your team ID: “YOUR TEAM ID GOES HERE”

  • Replace the following line with your team secret: “YOUR TEAM SECRET GOES HERE”

    (If you are using certificate-based authentication, remove this line.)

  • Replace the following line with a string that you will use to identify your cluster: “YOUR UNIQUE CLUSTER NAME GOES HERE”

  • Apply the manifest with this command: kubectl apply -f /path/to/gremlin-conf.yaml

Download and apply the Gremlin client manifest

If you are using certificate-based authentication:

  • Download and apply the gremlin client manifest for your kubernetes cluster by running the following:

    kubectl apply -f https://k8s.gremlin.com/resources/gremlin-client.yaml

If you are using secret-based authentication:

  • Download and apply the gremlin client manifest for your kubernetes cluster by running the following:

    kubectl apply -f https://k8s.gremlin.com/resources/gremlin-client-secret.yaml

Enabling Gremlin on the Kubernetes Master

Most Kubernetes deployments configure master nodes with the node-role.kubernetes.io/master:NoSchedule taint. You can run the following command to see if any of your nodes have this taint:

shell
1$ kubectl get no -o=custom-columns=NAME:.metadata.name,TAINTS:.spec.taints
2NAME TAINTS
3kube-01 [map[effect:NoSchedule key:node-role.kubernetes.io/master]]
4kube-02 <none>

If you wish to install Gremlin on a Kubernetes master that has been tainted, add a tolerations section to the PodSpec of the Gremlin Client Manifest.

yaml
1tolerations:
2 - key: node-role.kubernetes.io/master
3 operator: Exists
4 effect: NoSchedule

You will need to reapply the Gremlin client manifest after making this change.

Download and apply the K8s client manifest

If you are using certificate-based authentication:

  • Download and apply the k8s client manifest by running:

    kubectl apply -f https://k8s.gremlin.com/resources/gremlin-chao.yaml

If you are using secret-based authentication:

  • Download and apply the k8s client manifest by running:

    kubectl apply -f https://k8s.gremlin.com/resources/gremlin-chao-secret.yaml

Helm

Let Gremlin know your Gremlin team ID and your Kubernetes cluster name

bash
1GREMLIN_TEAM_ID="changeit"
2GREMLIN_CLUSTER_ID="changeit"

Add the Gremlin helm chart

bash
1helm repo remove gremlin
2helm repo add gremlin https://helm.gremlin.com

Create a namespace for the Gremlin Kubernetes client

bash
1kubectl create namespace gremlin

Install the Gremlin Kubernetes client

shell
1helm install gremlin/gremlin \
2 --name gremlin \
3 --namespace gremlin \
4 --set gremlin.secret.managed=true \
5 --set gremlin.secret.type=secret \
6 --set gremlin.secret.teamID=$GREMLIN_TEAM_ID \
7 --set gremlin.secret.clusterID=$GREMLIN_CLUSTER_ID \
8 --set gremlin.secret.teamSecret=$GREMLIN_TEAM_SECRET

Verify your Installation

Last you need to check that Gremlin is installed properly

bash
1kubectl get pods -n gremlin

This should list a Gremlin agent per node (physical/virtual machine in your cluster) plus one for chao

Example

shell
1$ kubectl get pods -n gremlin
2
3NAME READY STATUS RESTARTS AGE
4chao-78bbc7cbf6-9hn7q 1/1 Running 0 5d20h
5gremlin-9r4t7 1/1 Running 0 5d20h
6gremlin-bwmtz 1/1 Running 1 126d
7gremlin-bx6dn 1/1 Running 0 5d20h

Pending Pods

If any pods are pending this means your installation is incomplete and you should contact your cluster administrator to debug why you are unable to run gremlin on those nodes

shell
1$ kubectl get pods -n gremlin
2
3NAME READY STATUS RESTARTS AGE
4chao-78bbc7cbf6-9hn7q 1/1 Running 0 5d20h
5gremlin-c25ld 0/1 Pending 0 112d
6gremlin-n5gt7 0/1 Pending 0 112d
7gremlin-zn4kq 1/1 Running 0 126d

Running an attack

Once you select the Kubernetes objects to be targeted, select and configure your desired Gremlin attack. When the attack is run, the underlying containers within the objects selected will be impacted.