Chaos Monkey Alternatives

OpenShift

  • 2 min read
  • Last Updated October 17, 2018

Monkey-Ops

Monkey-Ops is an open-source Chaos Monkey implementation written in Go and designed to be deployed alongside an OpenShift application. Monkey-Ops will randomly perform one of two possible attacks:

You can install Monkey-Ops either via Docker or as a separate OpenShift project.

Docker Installation

Create a Docker container with the following command. Be sure to replace TOKEN with your own OpenShift auth token and PROJECT_NAME with the appropriate value.

bash
1docker run joseangelsilvag/monkey-ops /monkey-ops \
2 --TOKEN="<TOKEN>" \
3 --PROJECT_NAME="chaos-demo" \
4 --API_SERVER="https://api.starter-us-west-2.openshift.com:443" \
5 --INTERVAL=30 \
6 --MODE="background"

This will randomly execute one of the two possible attacks every INTERVAL seconds. If you wish to have more control over attacks, change MODE to "rest" and use the /chaos REST API to launch an attack.

OpenShift Installation

Installing Monkey-Ops as an OpenShift project is a bit more complex.

  • Clone the Git repo to a local directory.

    bash
    1git clone https://github.com/joseangelsilvag/monkey-ops.git
  • Create a monkey-ops.json file and paste the following, which will be used to create a Service Account.

    json
    1{
    2 "apiVersion": "v1",
    3 "kind": "ServiceAccount",
    4 "metadata": {
    5 "name": "monkey-ops"
    6 }
    7}
  • Create the OpenShift Service Account using the OpenShift CLI and grant it privileges for your project (e.g. chaos-demo).

    bash
    1oc create -f monkey-ops.json && oc policy add-role-to-user edit system:serviceaccount:chaos-demo:monkey-ops
  • Now create a new pod using the monkey-ops-template.yaml found in the Monkey-Ops project.

    bash
    1oc create -f ./openshift/monkey-ops-template.yaml -n chaos-demo
  • Finally, create a new app called monkey-ops and pass appropriate values for each PARAM indicating when and how attacks will be executed.

    bash
    1oc new-app \
    2 --name=monkey-ops \
    3 --template=monkey-ops \
    4 --param APP_NAME=monkey-ops \
    5 --param INTERVAL=30 \
    6 --param MODE=background \
    7 --param TZ=America/Los_Angeles \
    8 --labels=app_name=monkey-ops -n chaos-demo

Engineering Chaos In OpenShift with Gremlin

Gremlin Free simplifies your Chaos Engineering workflow for OpenShift by making it safe and effortless to execute Chaos Experiments across all application containers. As a distributed architecture OpenShift is particularly sensitive to instability and unexpected failures. Gremlin Free can perform shutdown and CPU attacks on your OpenShift applications.

Check out this tutorial for installing Gremlin on CentOS or this guide for installing Gremlin on OpenShift via a Kubernetes DaemonSet to get started!

Pumba

As discussed in the Chaos Monkey Alternatives - Docker chapter, Pumba is a Chaos injection tool primarily built for Docker. However, it can also be deployed on Kubernetes and, by extension, on OpenShift using a DaemonSet. Pumba can stop, pause, kill, and remove containers, which means it works fairly well with OpenShift pods that are made up of one or more containers.

  • To deploy Pumba in OpenShift nodes using a DaemonSet you must first add a security policy to allow the OpenShift developer user to administer Kubernetes clusters.

    bash
    1oc adm policy --as system:admin add-cluster-role-to-user cluster-admin developer
  • Add the privileged security context restraint to the default user for your project.

    bash
    1oc adm policy add-scc-to-user privileged system:serviceaccount:<project>:default
  • Set the allowHostDirVolumePlugin option to true in the restricted security restraint, which will allow OpenShift to connect to the Docker container.

    bash
    1oc edit scc restricted
    bash
    1# Please edit the object below. Lines beginning with a '#' will be ignored,
    2# and an empty file will abort the edit. If an error occurs while saving this file will be
    3# reopened with the relevant failures.
    4#
    5allowHostDirVolumePlugin: true
    6allowHostIPC: false
    7allowHostNetwork: false
    8allowHostPID: false
    9allowHostPorts: false
    10allowPrivilegedContainer: false
    11allowedCapabilities: null
    12apiVersion: security.openshift.io/v1
    13# [...]
  • Download the pumba_openshift.yml file and modify it as necessary. By default every 30 seconds it will kill a container within a pod containing the string "hello" in its name.

    bash
    1curl -O https://raw.githubusercontent.com/alexei-led/pumba/master/deploy/pumba_openshift.yml
    yaml
    1apiVersion: extensions/v1beta1
    2kind: DaemonSet
    3metadata:
    4 name: pumba
    5spec:
    6 template:
    7 metadata:
    8 labels:
    9 app: pumba
    10 name: pumba
    11 spec:
    12 containers:
    13 - image: gaiaadm/pumba:master
    14 imagePullPolicy: Always
    15 name: pumba
    16 command: ['pumba']
    17 args:
    18 [
    19 '--random',
    20 '--debug',
    21 '--interval',
    22 '30s',
    23 'kill',
    24 '--signal',
    25 'SIGKILL',
    26 're2:.*hello.*',
    27 ]
    28 securityContext:
    29 runAsUser: 0
    30 volumeMounts:
    31 - name: dockersocket
    32 mountPath: /var/run/docker.sock
    33 volumes:
    34 - hostPath:
    35 path: /var/run/docker.sock
    36 name: dockersocket
  • Finally, create the DaemonSet from the pumba_openshift.yml.

    bash
    1oc create -f pumba_openshift.yml
    2daemonset.extensions "pumba" created

That’s it. Now just add some pods to your project that match the regex used in the DaemonSet, if any, and Pumba should pick up on them and start killing them off. Check out this handy video tutorial for all the details.

Β© 2020 Gremlin Inc.
All rights reserved.
Privacy Policy

Download PDF