Install Gremlin on OpenShift 3.x

Install Gremlin on OpenShift 3.x

This install is composed of 3 parts:

PartDescription
Custom Gremlin SELinux PolicyA policy that extends the standard container SELinux policy, and adds allowances for Gremlin attacks and state management
Gremlin Agent DaemonsetAn agent process that reports containers on each host, and dispatches attacks
Kubernetes Agent DeploymentAn agent process that reports Kubernetes pods and binds them with containers

Prerequisites

It is recommended to install Gremlin into its own namespace (e.g. gremlin)

shell
1oc create namespace gremlin

You will also need to download your Gremlin certificate key-pair for authenticating to your Gremlin team. The they key pair extracted to your local system, create a Kubernetes secret containing the key-pair and your team ID.

shell
1oc create secret generic gremlin-secret \
2 --namespace gremlin \
3 --from-literal=GREMLIN_TEAM_ID=$GREMLIN_TEAM_ID \
4 --from-literal=GREMLIN_CLUSTER_ID=$GREMLIN_CLUSTER_ID \
5 --from-file=gremlin.cert=$PATH_TO_CERTIFICATE \
6 --from-file=gremlin.key=$PATH_TO_PRIVATE_KEY

Install SELinux Policy

The following creates a new gremlin SELinux module

shell
1curl -fsSL https://github.com/gremlin/selinux-policies/releases/download/v0.0.1/selinux-policies-v0.0.1.tar.gz -o selinux-policies-v0.0.1.tar.gz
2tar xzf selinux-policies-v0.0.1.tar.gz
3sudo semodule -i selinux-policies-v0.0.1/gremlin-openshift3.cil

Install Gremlin Agent Daemonset

Install ServiceAccount and SecurityContextConstraint

Create a gremlin service account in the gremlin namespace

shell
1oc create serviceaccount gremlin -n gremlin

The following SecurityContextConstraint is used to allow the gremlin service account verious privileges and access to the newly installed gremlin SELinux module

yaml
1# gremlin-scc.yaml
2---
3apiVersion: security.openshift.io/v1
4allowHostDirVolumePlugin: true
5allowHostIPC: false
6allowHostNetwork: false
7allowHostPID: false
8allowHostPorts: false
9allowPrivilegeEscalation: true
10allowPrivilegedContainer: false
11allowedCapabilities: null
12defaultAddCapabilities: null
13fsGroup:
14 type: RunAsAny
15groups: []
16kind: SecurityContextConstraints
17metadata:
18 annotations:
19 kubernetes.io/description: 'gremlin provides all the features of the
20 restricted SCC but allows host mounts, any UID by a pod, and forces
21 the process to run as the gremlin.process SELinux type. This is intended
22 to be used solely by Gremlin. WARNING: this SCC allows host file
23 system access as any UID, including UID 0. Grant with caution.'
24 name: gremlin
25priority: null
26readOnlyRootFilesystem: false
27requiredDropCapabilities:
28- MKNOD
29runAsUser:
30 type: RunAsAny
31seLinuxContext:
32 seLinuxOptions:
33 type: gremlin.process
34 type: MustRunAs
35supplementalGroups:
36 type: RunAsAny
37volumes:
38- configMap
39- emptyDir
40- hostPath
41- persistentVolumeClaim
42- secret
shell
1oc create -f gremlin-scc.yaml -n gremlin
2oc adm policy add-scc-to-user gremlin -z gremlin -n gremlin

Install Daemonset

yaml
1# gremlin-daemonset.yaml
2---
3apiVersion: apps/v1
4kind: DaemonSet
5metadata:
6 name: gremlin
7 namespace: gremlin
8spec:
9 selector:
10 matchLabels:
11 app.kubernetes.io/name: gremlin
12 template:
13 metadata:
14 labels:
15 app.kubernetes.io/name: gremlin
16 spec:
17 serviceAccountName: gremlin
18 containers:
19 - name: gremlin
20 image: gremlin/gremlin:latest
21 args: [ "daemon" ]
22 env:
23 - name: GREMLIN_TEAM_ID
24 valueFrom:
25 secretKeyRef:
26 name: gremlin-secret
27 key: GREMLIN_TEAM_ID
28 - name: GREMLIN_IDENTIFIER
29 valueFrom:
30 fieldRef:
31 fieldPath: spec.nodeName
32 - name: GREMLIN_TEAM_CERTIFICATE_OR_FILE
33 value: file:///var/lib/gremlin/cert/gremlin.cert
34 - name: GREMLIN_TEAM_PRIVATE_KEY_OR_FILE
35 value: file:///var/lib/gremlin/cert/gremlin.key
36 - name: GREMLIN_CLIENT_TAGS
37 value:
38 - name: GREMLIN_DOCKER_IMAGE
39 value: gremlin/gremlin:latest
40 volumeMounts:
41 - name: docker-sock
42 mountPath: /var/run/docker.sock
43 - name: gremlin-state
44 mountPath: /var/lib/gremlin
45 - name: gremlin-logs
46 mountPath: /var/log/gremlin
47 - name: gremlin-cert
48 mountPath: /var/lib/gremlin/cert
49 readOnly: true
50 volumes:
51 - name: docker-sock
52 hostPath:
53 path: /var/run/docker.sock
54 - name: gremlin-state
55 hostPath:
56 path: /var/lib/gremlin
57 - name: gremlin-logs
58 hostPath:
59 path: /var/log/gremlin
60 - name: gremlin-cert
61 secret:
62 secretName: gremlin-secret
shell
1oc create -f gremlin-daemonset.yaml -n gremlin

Install Kubernetes Agent Deployment

Install ServiceAccount, ClusterRole, and ClusterRoleBinding

yaml
1# chao-service-account.yaml
2---
3apiVersion: v1
4kind: ServiceAccount
5metadata:
6 name: chao
7 namespace: gremlin
8---
9apiVersion: rbac.authorization.k8s.io/v1
10kind: ClusterRole
11metadata:
12 name: gremlin-watcher
13rules:
14 - apiGroups: ["apps"]
15 resources: ["replicasets", "deployments", "statefulsets", "daemonsets"]
16 verbs: ["get", "watch", "list"]
17 - apiGroups: [""]
18 resources: ["pods", "nodes"]
19 verbs: ["get", "watch", "list"]
20---
21apiVersion: rbac.authorization.k8s.io/v1
22kind: ClusterRoleBinding
23metadata:
24 name: chao
25subjects:
26 - kind: ServiceAccount
27 name: chao
28 namespace: gremlin
29roleRef:
30 apiGroup: rbac.authorization.k8s.io
31 kind: ClusterRole
32 name: gremlin-watcher
shell
1oc create -f chao-service-account.yaml -n gremlin

Install Deployment

yaml
1---
2apiVersion: apps/v1
3kind: Deployment
4metadata:
5 name: chao
6 namespace: gremlin
7spec:
8 replicas: 1
9 template:
10 selector:
11 matchLabels:
12 app.kubernetes.io/name: chao
13 template:
14 metadata:
15 labels:
16 app.kubernetes.io/name: chao
17 spec:
18 serviceAccountName: chao
19 containers:
20 - image: gremlin/chao:latest
21 env:
22 - name: GREMLIN_TEAM_ID
23 valueFrom:
24 secretKeyRef:
25 name: gremlin-secret
26 key: GREMLIN_TEAM_ID
27 - name: GREMLIN_CLUSTER_ID
28 valueFrom:
29 secretKeyRef:
30 name: gremlin-secret
31 key: GREMLIN_CLUSTER_ID
32 args:
33 - "-cert_path"
34 - "/var/lib/gremlin/cert/gremlin.cert"
35 - "-key_path"
36 - "/var/lib/gremlin/cert/gremlin.key"
37 imagePullPolicy: Always
38 name: chao
39 volumeMounts:
40 - name: gremlin-cert
41 mountPath: /var/lib/gremlin/cert
42 readOnly: true
43 volumes:
44 - name: gremlin-cert
45 secret:
46 secretName: gremlin-secret
shell
1oc create -f chao-deployment.yaml -n gremlin

Run Attacks

You can now run attacks on your cluster, including Kubernetes attacks.

Other Considerations

Running Gremlin in Privileged Mode

Gremlin does not recommend using OpenShift’s privileged SecurityContextConstraint (SCC) because it grants more privileges to Gremlin than necessary (including all Linux capabilities). However if you do not wish to install the gremlin-openshift3 SELinux module and gremlin scc, you will instead need to run Gremlin under the privileged SCC, and configure Gremlin’s seLinuxOptions to use the spc_t process label.

Set up Gremlin to use the privileged SCC

shell
1oc create serviceaccount gremlin -n gremlin
2oc adm policy add-scc-to-user privileged -z gremlin -n gremlin

Configure Gremlin to use the spc_t process label

The following Daemonset manifest is similar to the recommended installation manifest, but with a securityContext that runs Gremlin under the spc_t process label.

yaml
1# gremlin-daemonset.yaml
2---
3apiVersion: apps/v1
4kind: DaemonSet
5metadata:
6 name: gremlin
7 namespace: gremlin
8spec:
9 selector:
10 matchLabels:
11 app.kubernetes.io/name: gremlin
12 template:
13 metadata:
14 labels:
15 app.kubernetes.io/name: gremlin
16 spec:
17 serviceAccountName: gremlin
18 securityContext:
19 seLinuxOptions:
20 type: spc_t
21 containers:
22 - name: gremlin
23 image: gremlin/gremlin:latest
24 args: [ "daemon" ]
25 env:
26 - name: GREMLIN_TEAM_ID
27 valueFrom:
28 secretKeyRef:
29 name: gremlin-secret
30 key: GREMLIN_TEAM_ID
31 - name: GREMLIN_IDENTIFIER
32 valueFrom:
33 fieldRef:
34 fieldPath: spec.nodeName
35 - name: GREMLIN_TEAM_CERTIFICATE_OR_FILE
36 value: file:///var/lib/gremlin/cert/gremlin.cert
37 - name: GREMLIN_TEAM_PRIVATE_KEY_OR_FILE
38 value: file:///var/lib/gremlin/cert/gremlin.key
39 - name: GREMLIN_CLIENT_TAGS
40 value:
41 - name: GREMLIN_DOCKER_IMAGE
42 value: gremlin/gremlin:latest
43 volumeMounts:
44 - name: docker-sock
45 mountPath: /var/run/docker.sock
46 - name: gremlin-state
47 mountPath: /var/lib/gremlin
48 - name: gremlin-logs
49 mountPath: /var/log/gremlin
50 - name: gremlin-cert
51 mountPath: /var/lib/gremlin/cert
52 readOnly: true
53 volumes:
54 - name: docker-sock
55 hostPath:
56 path: /var/run/docker.sock
57 - name: gremlin-state
58 hostPath:
59 path: /var/lib/gremlin
60 - name: gremlin-logs
61 hostPath:
62 path: /var/log/gremlin
63 - name: gremlin-cert
64 secret:
65 secretName: gremlin-secret

Related

Avoid downtime. Use Gremlin to turn failure into resilience.

Gremlin empowers you to proactively root out failure before it causes downtime. See how you can harness chaos to build resilient systems by requesting a demo of Gremlin.

Get started
  • TechCrunch
  • Forbes
  • Business Insider
  • VentureBeat


© 2020 Gremlin Inc. San Jose, CA 95113