Infrastructure Layer

Preview Gremlin on OpenShift 4


Pre-requisites

This guide assumes you will be installing Gremlin into its own namespace. You can start a new project and namespace with the following command. All subsequent oc create commands in this guide leave out the --namespace argument, assuming that you wish to install Gremlin in the current OpenShift project.

shell
1oc new-project gremlin

You will also need to download your Gremlin certificate key-pair for authenticating to your Gremlin team. With the key pair extracted to your local system, create a Kubernetes secret containing the key-pair and your team ID.

shell
1oc create secret generic gremlin-secret \
2 --from-literal=GREMLIN_TEAM_ID=$GREMLIN_TEAM_ID \
3 --from-literal=GREMLIN_CLUSTER_ID=$GREMLIN_CLUSTER_ID \
4 --from-file=gremlin.cert=$PATH_TO_CERTIFICATE \
5 --from-file=gremlin.key=$PATH_TO_PRIVATE_KEY

Install SELinux Policy

Install SELinux Policy The following creates a new gremlin SELinux module

shell
1curl -fsSL https://github.com/gremlin/selinux-policies/releases/download/v0.0.2/selinux-policies-v0.0.2.tar.gz -o selinux-policies-v0.0.2.tar.gz
2tar xzf selinux-policies-v0.0.2.tar.gz
3sudo semodule -i selinux-policies-v0.0.2/gremlin-openshift4.cil

Install Gremlin Agent Daemonset

Install ServiceAccount and SecurityContextConstraint

Create a gremlin service account in the gremlin namespace

shell
1oc create serviceaccount gremlin

The following SecurityContextConstraint is used to allow the gremlin service account various privileges and access to the newly installed gremlin SELinux module

yaml
1# gremlin-scc.yaml
2---
3apiVersion: security.openshift.io/v1
4allowHostDirVolumePlugin: true
5allowHostIPC: false
6allowHostNetwork: false
7allowHostPID: true
8allowHostPorts: false
9allowPrivilegeEscalation: false
10allowPrivilegedContainer: false
11allowedCapabilities:
12 - ALL
13defaultAddCapabilities: null
14fsGroup:
15 type: RunAsAny
16groups: []
17kind: SecurityContextConstraints
18metadata:
19 annotations:
20 kubernetes.io/description: 'gremlin provides all the features of the
21 restricted SCC but allows host mounts, any UID by a pod, and forces
22 the process to run as the gremlin.process SELinux type. This is intended
23 to be used solely by Gremlin. WARNING: this SCC allows host file
24 system access as any UID, including UID 0. Grant with caution.'
25 name: gremlin
26priority: null
27readOnlyRootFilesystem: false
28requiredDropCapabilities: []
29runAsUser:
30 type: RunAsAny
31seLinuxContext:
32 seLinuxOptions:
33 type: gremlin.process
34 type: MustRunAs
35seccompProfiles:
36- unconfined
37supplementalGroups:
38 type: RunAsAny
39volumes:
40- configMap
41- emptyDir
42- hostPath
43- persistentVolumeClaim
44- secret
shell
1oc create -f gremlin-scc.yaml
2oc adm policy add-scc-to-user gremlin -z gremlin

Install Daemonset

yaml
1# gremlin-daemonset.yaml
2---
3apiVersion: apps/v1
4kind: DaemonSet
5metadata:
6 name: gremlin
7 namespace: gremlin
8spec:
9 selector:
10 matchLabels:
11 app.kubernetes.io/name: gremlin
12 template:
13 metadata:
14 labels:
15 app.kubernetes.io/name: gremlin
16 spec:
17 serviceAccountName: gremlin
18 hostPID: true
19 containers:
20 - name: gremlin
21 image: gremlin/gremlin:latest
22 securityContext:
23 capabilities:
24 add:
25 - ALL
26 args: [ "daemon" ]
27 env:
28 - name: GREMLIN_TEAM_ID
29 valueFrom:
30 secretKeyRef:
31 name: gremlin-secret
32 key: GREMLIN_TEAM_ID
33 - name: GREMLIN_IDENTIFIER
34 valueFrom:
35 fieldRef:
36 fieldPath: spec.nodeName
37 - name: GREMLIN_TEAM_CERTIFICATE_OR_FILE
38 value: file:///var/lib/gremlin/cert/gremlin.cert
39 - name: GREMLIN_TEAM_PRIVATE_KEY_OR_FILE
40 value: file:///var/lib/gremlin/cert/gremlin.key
41 - name: GREMLIN_CLIENT_TAGS
42 value:
43 - name: GREMLIN_DOCKER_IMAGE
44 value: gremlin/gremlin:latest
45 volumeMounts:
46 - name: gremlin-state
47 mountPath: /var/lib/gremlin
48 readOnly: false
49 - name: gremlin-logs
50 mountPath: /var/log/gremlin
51 readOnly: false
52 - name: host-proc
53 mountPath: /host/proc
54 readOnly: true
55 - name: sys-fs-cgroup
56 mountPath: /sys/fs/cgroup
57 readOnly: true
58 - name: resolv-conf
59 mountPath: /run/systemd/resolve/resolv.conf
60 readOnly: true
61 - name: containers-policy
62 mountPath: /etc/containers/policy.json
63 readOnly: true
64 - name: runtime-sock
65 mountPath: /var/run/crio/crio.sock
66 readOnly: true
67 - name: runtime-runc
68 mountPath: /run/runc
69 readOnly: false
70 - name: gremlin-cert
71 mountPath: /var/lib/gremlin/cert
72 readOnly: true
73 volumes:
74 - name: host-proc
75 hostPath:
76 path: /proc
77 - name: sys-fs-cgroup
78 hostPath:
79 path: /sys/fs/cgroup
80 - name: resolv-conf
81 hostPath:
82 path: /run/systemd/resolve/resolv.conf
83 - name: containers-policy
84 hostPath:
85 path: /etc/containers/policy.json
86 - name: runtime-sock
87 hostPath:
88 path: /var/run/crio/crio.sock
89 - name: runtime-runc
90 hostPath:
91 path: /run/runc
92 - name: gremlin-state
93 hostPath:
94 path: /var/lib/gremlin
95 - name: gremlin-logs
96 hostPath:
97 path: /var/log/gremlin
98 - name: gremlin-cert
99 secret:
100 secretName: gremlin-secret
shell
1oc create -f gremlin-daemonset.yaml

Install Kubernetes Agent Deployment

Install ServiceAccount, ClusterRole, and ClusterRoleBinding

yaml
1# chao-service-account.yaml
2---
3apiVersion: v1
4kind: ServiceAccount
5metadata:
6 name: chao
7 namespace: gremlin
8---
9apiVersion: rbac.authorization.k8s.io/v1
10kind: ClusterRole
11metadata:
12 name: gremlin-watcher
13rules:
14 - apiGroups: ["apps"]
15 resources: ["replicasets", "deployments", "statefulsets", "daemonsets"]
16 verbs: ["get", "watch", "list"]
17 - apiGroups: [""]
18 resources: ["pods", "nodes"]
19 verbs: ["get", "watch", "list"]
20---
21apiVersion: rbac.authorization.k8s.io/v1
22kind: ClusterRoleBinding
23metadata:
24 name: chao
25subjects:
26 - kind: ServiceAccount
27 name: chao
28 namespace: gremlin
29roleRef:
30 apiGroup: rbac.authorization.k8s.io
31 kind: ClusterRole
32 name: gremlin-watcher
shell
1oc create -f chao-service-account.yaml

Install Deployment

yaml
1# chao-deployment.yaml
2---
3apiVersion: apps/v1
4kind: Deployment
5metadata:
6 name: chao
7 namespace: gremlin
8spec:
9 replicas: 1
10 template:
11 selector:
12 matchLabels:
13 app.kubernetes.io/name: chao
14 template:
15 metadata:
16 labels:
17 app.kubernetes.io/name: chao
18 spec:
19 serviceAccountName: chao
20 containers:
21 - image: gremlin/chao:latest
22 env:
23 - name: GREMLIN_TEAM_ID
24 valueFrom:
25 secretKeyRef:
26 name: gremlin-secret
27 key: GREMLIN_TEAM_ID
28 - name: GREMLIN_CLUSTER_ID
29 valueFrom:
30 secretKeyRef:
31 name: gremlin-secret
32 key: GREMLIN_CLUSTER_ID
33 args:
34 - "-cert_path"
35 - "/var/lib/gremlin/cert/gremlin.cert"
36 - "-key_path"
37 - "/var/lib/gremlin/cert/gremlin.key"
38 imagePullPolicy: Always
39 name: chao
40 volumeMounts:
41 - name: gremlin-cert
42 mountPath: /var/lib/gremlin/cert
43 readOnly: true
44 volumes:
45 - name: gremlin-cert
46 secret:
47 secretName: gremlin-secret
shell
1oc create -f chao-deployment.yaml

Run Attacks

You can now run attacks on your cluster, including Kubernetes attacks.