Tools for Creating Chaos Outside of AWS
Chaos Monkey serves a singular purpose -- to randomly terminate instances. As discussed in Chaos Monkey and Spinnaker and The Pros and Cons of Chaos Monkey, additional tools are required when using Chaos Monkey, in order to cover the broad spectrum of experimentation and failure injection required for proper Chaos Engineering.
In this chapter, we'll explore a wide range of tools and techniques -- regardless of the underlying technologies -- that you and your team can use to intelligently induce failures while confidently building toward a more resilient system.
Perhaps the most prominent fault-tolerant tool for Apache is Cassandra, the NoSQL, performant, highly-scalable data management solution. Check out this talk by Christos Kalantzis, Netflix's Director of Engineering for a great example of how Chaos Engineering can be applied within Cassandra.
Hadoop's unique Distributed File System (HDFS) requires that the FileSystem Java API and shell access allow applications and the operating system to read and interact with the HDFS. Therefore, most of the Chaos Engineering tools that run on underlying systems can also be used for injecting failure into Hadoop. Check out the Linux section in particular.
Anarchy Ape is an open-source tool primarily coded in Java that injects faults into Hadoop cluster nodes. It is capable of corrupting specific files, corrupting random HDFS blocks, disrupting networks, killing DataNodes or NameNodes, dropping network packets, and much more.
Anarchy Ape is executed via the <span class="code-class-custom">ape</span> command line tool. Check out the official repository for more details on installing and using Anarchy Ape on your Hadoop clusters.
Hadoop Killer is an open-source tool written in Ruby that kills random Java processes on a local system. It can be installed using RubyGems and is configured via a simple YAML syntax.
- <span class="code-class-custom">target</span>: The name of the process to kill.
- <span class="code-class-custom">max</span>: Maximum number of times to kill the process.
- <span class="code-class-custom">probability</span>: Percentage probability to kill the process during each attempt.
- <span class="code-class-custom">interval</span>: Number of seconds between attempts.
Have a look at the GitHub repository for the basic info on using Hadoop Killer.
The primary fault injection tool explicitly built for Kafka is its built-in Trogdor test framework. Trogdor executes fault injection through a single-coordinator multi-agent process. Trogdor has two built-in fault types.
- <span class="code-class-custom">ProcessStopFault</span>: Stops the specified process by sending a <span class="code-class-custom">SIGSTOP</span> signal.
- <span class="code-class-custom">NetworkPartitionFault</span>: Creates an artificial network partition between nodes using <span class="code-class-custom">iptables</span>.
Each user agent can be assigned to perform a Task, which is an action defined in JSON and includes the full Java <span class="code-class-custom">class</span>, along with <span class="code-class-custom">startMs</span> and <span class="code-class-custom">durationMs</span>, which indicate the milliseconds since the Unix epoch for when to start and how long to run the Task, respectively. All additional fields are customized and Task-specific.
Here's a simple Task to trigger a <span class="code-class-custom">NetworkPartitionFault</span> by creating a network partition between <span class="code-class-custom">node1</span> and <span class="code-class-custom">node2</span>.
Check out the wiki documentation for more details on using Trogdor to inject faults and perform tests in your Kafka system.
Chaos Engineering with Spark comes down to the underlying platforms on which your application resides. For example, if you're using the Tensorframes wrapper to integrate Spark with your TensorFlow application then Gremlin's Failure as a Service solution can help you inject failure and learn how to create a more resilient application.
Spark Streaming applications also include built-in fault-tolerance. Each Resilient Distributed Dataset (RDD) that Spark handles is subject to loss prevention policies defined in the Fault-Tolerance Semantics documentation.
Lastly, Spark's built-in integration tests include a handful of fault injections like the NetworkFaultInjection.
There are an abundance of Chaos Monkey alternatives for container-based applications. Browse through the Chaos Monkey Alternatives - Docker and Chaos Monkey Alternatives - OpenShift chapters for many great solutions.
Check out Chaos Monkey Alternatives - Docker for details on using Pumba, Gremlin, Docker Chaos Monkey, and Docker Simian Army to inject chaos into your Docker containers.
Head over to the Chaos Monkey Alternatives - OpenShift chapter for information on utilizing Monkey Ops, Gremlin, and Pumba to run Chaos Experiments in OpenShift distributions.
Have a look at the official GitHub repository for more info on using Chaos Spawn to inject Chaos in your Elixir applications.
There are dozens of alternative tools to Chaos Monkey available for the most popular infrastructure technologies and platforms on the market. Have a look through Chaos Monkey Alternatives - Azure, Chaos Monkey Alternatives - Google Cloud Platform, and Chaos Monkey Alternatives - Kubernetes for many great options.
Read through our Chaos Monkey Alternatives - Azure chapter for guidance on how the Azure Search team created their own Search Chaos Monkey, along with implementing your own Chaos Engineering practices in Azure with Gremlin, WazMonkey, and Azure's Fault Analysis Service.
Check out Chaos Monkey Alternatives - Google Cloud Platform for details on using the simple Google Cloud Chaos Monkey tool, Gremlin's Failure as a Service, and the open-source Chaos Toolkit for injecting failure into your own Google Cloud Platform systems.
A quick read of our Chaos Monkey Alternatives - Kubernetes chapter will teach you all about the Kube Monkey, Kubernetes Pod Chaos Monkey, Chaos Toolkit, and Gremlin tools, which can be deployed on Kubernetes clusters to execute Chaos Experiments and create more resilient applications.
In addition to the many tools features in the Azure, Google Cloud Platform, Kubernetes, Private Cloud, and VMware sections we're looking at a few network manipulation tools for injecting failure in your on-premise architecture.
Blockade is an open-source tool written in Python that creates various network failure scenarios within distributed applications. Blockade uses Docker containers to perform actions and manage the network from the host system.
We can see that the network has been slowed significantly, causing (relatively) massive delays to the majority of our ping requests.
Toxiproxy is an open-source framework written in Go for simulating network conditions within application code. It is primarily intended for testing, continuous integration, and development environments, but it can be customized to support randomized Chaos Experiments. Much of Toxiproxy's integration comes from open-source client APIs, which make it easy for Toxiproxy to integrate with a given application stack.
As an API within your application Toxiproxy can accomplish a lot of network simulations and manipulations, but this example shows how to use Toxiproxy to disconnect all Redis connections during a Rails application test.
As software for data center management OpenStack can universally employ virtually any infrastructure-based Chaos tool we've already covered. Additionally, the OS-Faults tool is an open-source library developed by the OpenStack team and written in Python that is designed to perform destructive actions within an OpenStack cloud. Actions are defined using technology- and platform-specific drivers. There are currently drivers for nearly every aspect of OpenStack's architecture including drivers for the cloud, power management, node discovering, services, and containers.
While using OpenStack will depend heavily on your particular use case, below is one example showing the basics to get started.
That's all there is to it. OS-Faults is capable of many types of Chaos Experiments including disconnecting nodes, overloading IO and memory, restarting and killing services/nodes, and much more. The official documentation has more details.
Take a look at the Chaos Monkey Alternatives - Private Cloud chapter to see how to begin Chaos Engineering within your own private cloud architecture using GomJabbar, Gremlin, and Muxy.
Check out the Chaos Monkey Alternatives - VMware chapter to learn about the Chaos Lemur tool and Gremlin's own Failure as a Service solution, both of will inject failure into your VMware and other BOSH-managed virtual machines with relative ease.
The tools you choose for implementing Chaos Engineering in Java will primarily be based on the technologies of your system architecture. However, in addition to the Maven and Spring Boot tools discussed below, you may also consider Namazu, which is an open-source fuzzy scheduler for testing distributed system implementations. Via a pseudo-randomized schedule, Namazu can attack the filesystem and IO, network packets, and Java function calls.
Namazu can be installed locally or via a Docker container. Additionally, the Namazu Swarm plugin allows multiple jobs to be paralleled via Docker swarms or Kubernetes clusters.
Check out the official GitHub repository for more details on Chaos Engineering your Java applications with Nazamu.
Since Maven is a build automation tool for Java applications performing Chaos Experiments in Maven-based projects is as easy as utilizing one of the Java-related Chaos Engineering tools we've detailed elsewhere. Check out our guides for Chaos Monkey for Spring Boot, Gremlin, Fabric8, and Chaos Lemur for some ways to inject failure in Maven- and Java-based systems.
Exploring our Chaos Monkey Alternatives - Spring Boot chapter will teach you about how to use Chaos Monkey for Spring Boot, Gremlin, and Fabric8 to execute Chaos Experiments against your Spring Boot applications.
Regardless of the operating system you're using there is a slew of Chaos Monkey alternative technologies. Check out the notable selections in the Linux and Windows sections below for some quick tools to get your Chaos Engineering kicked into high gear.
Performing Chaos Engineering on Linux is a match made in heaven -- virtually every tool we've explored here in the Alternatives chapter and listed in our Resources - Tools section was designed for if not built with Unix/Linux.
Like Linux, Windows Chaos Engineering depends on the platforms and architecture your organization is using. Much of the software we've covered in Resources - Tools and this entire Alternatives chapter can be applied to Windows-based systems.
Some particularly useful Chaos Monkey alternatives for Windows are the Fault Analysis Service for Azure Service Fabric, Gremlin's Failure as a Service for Docker, and Muxy for private cloud infrastructures.