Client-side chaos: Making your front end more reliable

The concept of Chaos Engineering is most often applied to backend systems, but for teams building websites and web applications, this is only half of the story. While it’s important to maintain reliable backend infrastructure and applications, there’s a lot that can go wrong even after we’ve fulfilled the user’s request. Web pages can load slowly, images and scripts might fail to download, and large geographic distances can add significant latency. These can have a big impact on the user experience and can happen even when our backend systems are working perfectly.

Ensuring reliability when modernizing financial applications

For decades, information technology in the financial services industry meant deploying bulky applications onto monolithic systems like mainframes. These systems have a proven track record of reliability, but don’t offer the flexibility and scalability of more modern architectures such as microservices and cloud computing. During periods of unexpectedly high demand, this inflexibility can cause technical issues for organizations ranging from personal trading platforms to major banks. Likewise, periods of low demand result in unused computing resources, costing these same organizations money.

Understanding your application’s critical path

It’s 3 a.m. You’re lying comfortably in bed when suddenly your phone starts screeching. It’s an automated high-severity alert telling you that your company’s web application is down. Exhausted, you open the website on your phone and do some basic tests. Everything looks ok at first, but then you realize customers can’t log into their accounts or make purchases. What happened, and what does this mean for your application?

Looking back on Chaos Conf 2020

It’s already been a week since we closed our third annual Chaos Conf! While we were forced to take the conference online, this meant that more of you could join us. Over 3,500 people signed up to help make this the world’s largest Chaos Engineering conference. That’s 5x more than 2019, and nearly 10x more than 2018! This is a testament to the growth of Chaos Engineering as a practice across many different industries and around the world.

Is your microservice a distributed monolith?

Your team has decided to migrate your monolithic application to a microservices architecture. You’ve modularized your business logic, containerized your codebase, allowed your developers to do polyglot programming, replaced function calls with API calls, built a Kubernetes environment, and fine-tuned your deployment strategy. But soon after hitting deploy, you start noticing problems. Services take a long time to start, failures cascade from one container to the next, and small changes involve redeploying the entire application. Weren’t microservices supposed to solve these problems?

Knowing your systems and how they can fail: Twilio and AWS talk at Chaos Conf 2020

This year’s Chaos Conf was packed full of incredible talks from some of the industry’s foremost experts on Chaos Engineering. Two of these talks were particularly interesting to me: “The More You Know: A Guide to Understanding your Systems” by Twilio’s Tyler Wells, and “Failing Over without Falling Over” by AWS’ Adrian Cockroft. Both talks set out to answer an important question: what are we aiming to accomplish with Chaos Engineering, and how do we do it thoughtfully?

Tyler Wells on building a culture of reliability at Twilio

What does reliability look like at a company that has thousands of employees and provides critical communication services to over 150,000 customers?

What is fault injection?

When reading about Chaos Engineering, you’ll likely hear the terms “fault injection” or “failure injection.” As the name suggests, fault injection is a technique for deliberately introducing stress or failure into a system in order to see how the system responds. But what exactly does this mean, and how does this relate to Chaos Engineering? In this post, we’ll look at the history of fault injection, how it’s evolved over time, and how it contributed to Chaos Engineering as we know it today.

Validating the resilience of your API gateway with Chaos Engineering

API gateways are a critical component of distributed systems and cloud-native deployments. They perform many important functions including request routing, caching, user authentication, rate limiting, and metrics collection. However, this means that any failures in your API gateway can put your entire deployment at risk. How confident are you that your gateway will be resilient to common production conditions such as backend outages, poor network performance, and sudden traffic surges?

Announcing the availability of Gremlin using AWS CloudFormation Public Registry

We’re excited to announce that Gremlin is available on AWS CloudFormation Public Registry. CloudFormation Public Registry is a new searchable collection of extensions that lets customers easily discover, provision, and manage resource types (provisioning logic) and modules published by AWS Partner Network (APN) Partners and the developer community. We’ve collaborated with CloudFormation Public Registry to enable you to easily deploy Gremlin and run Chaos Engineering experiments on your AWS deployments.

Self-service reliability with Internal Developer Platforms and Chaos Engineering

Up until the early 2000s, developers and Ops (at the time IT) had separate and often competing objectives, separate department leadership, separate key performance indicators by which they were judged, and often worked on separate floors or even separate buildings. The result was siloed teams concerned only with their own priorities, long hours, botched releases and unhappy customers.

Getting started with Memory attacks

Memory (or RAM, short for random-access memory) is a critical computing resource that stores temporary data on a system. Memory is a finite resource, and the amount of memory available determines the number and complexity of processes that can run on the system. Running out of RAM can cause significant problems such as system-wide lockups, terminated processes, and increased disk activity. Understanding how and when these issues can happen is vital to creating stable and resilient systems.

Getting started with CPU attacks

The CPU attack is one of the most common attack types run by Gremlin users. CPU attacks let you consume CPU capacity on a host, container, Kubernetes resource, or service. This might sound like a trivial exercise, but consuming even small amounts of CPU can reveal unexpected behaviors on our systems. These behaviors can manifest as poor performance, unresponsiveness, or instability.

Getting started with Disk attacks

Persistent storage is one of the more difficult aspects of managing distributed systems. When we attach a storage device to a host—whether it’s flash storage, network attached storage (NAS), or old fashioned spinning disks—we generally don’t give it much thought until we start running distributed applications or need to increase capacity. But there’s more that can go wrong with storage, and this can have unexpected consequences for our systems, services, and applications.

Getting started with IO attacks

Storage devices remain one of the most significant bottlenecks in modern systems. CPU and RAM speed seems to increase exponentially year over year, and although there have been large improvements in IO performance with solid state (SSD) and NVMe drives, moving data to and from persistent storage is still orders of magnitude slower than moving it to and from memory. In scalable cloud applications, this slowness can have a major impact on performance, latency, and the user experience. To replicate this effect ourselves, we can use the IO attack.

Reconnecting at AWS re:Invent 2021

AWS re:Invent 2021 was an especially unique event, as it was the industry’s first big in-person event since the start of the pandemic. After nearly two years of virtual events, being able to walk the expo halls in-person was an adjustment to say the least. We can’t understate how meaningful it was to see colleagues, customers, partners, and friends again, and while there’s still a ways to go before in-person events feel normal again, this was a huge step towards showing that we can do them safely. Thanks to everyone who helped make this year’s re:Invent a safe, comfortable, and fun week!

Getting started with Process Killer attacks

Modern applications come in a variety of forms–monoliths, microservices, serverless functions, and containers to name a few–but at the heart of all of these are processes. Processes are the fundamental unit of execution that we use to run programs, and although we need processes to run our applications, software engineers rarely think about them. We leave it to the operating system to manage them for us, and rather than monitor individual processes for performance and availability, we monitor services as a whole. This doesn’t mean we shouldn’t care about them, as even one failed process can make an entire system unstable.

Join Gremlin for AWS re:Invent 2021

It’s that time again! re:Invent is less than a month away, and this year is going to be one for the books. Not only is re:Invent celebrating its 10th anniversary, but it will also be one of the first major in-person events since 2019! After nearly two years of virtual conferences, you can bet the Gremlin team is excited to meet up with you in-person again.

Getting started with Blackhole attacks

In today’s distributed, cloud-native world, network connectivity is just as important as reliable hardware. The migration from monoliths to microservices and on-prem datacenters to public clouds means our applications are much more dependent on healthy networks. But as systems become more distributed and network diagrams become more complex, so does the risk of failure. If a network card, switch, router, API gateway, firewall, or ethernet cable fails, it could take our entire application offline. We need to design applications and services to be resilient to these kinds of failures, and the Blackhole attack can help.

Getting started with Shutdown attacks

For many years, system uptime was the primary measure of reliability, especially when the most popular method of running software was on bare metal, on-premises servers. If a server was shutdown, rebooted, or otherwise became unavailable, downtime was expected until a system administrator could manually restart it. The introduction of virtualization in the early 2000s—followed by the rise of public cloud platforms—made it easier to automatically detect and reboot shutdown systems, but this didn't address the core problem of reliance on uptime. Applications were—and often still are—designed with the expectation that the underlying systems will have unlimited uptime, and this simply isn't realistic. We need to design our applications with the expectation that systems will shutdown, reboot, and fail suddenly due to power outages or other unexpected state changes.

If you're adopting Kubernetes, you need Chaos Engineering

When Ticketmaster started their Kubernetes migration, they had to address a huge problem: whenever ticket sales opened for a popular event, as many as 150 million visitors flooded their website, effectively causing distributed denial of service (DDoS) attacks. With new events happening every 20 minutes and $7.6 billion in revenue at stake, outages could mean hundreds of thousands in lost sales.

Getting started with Time Travel attacks

It's the middle of the night when your phone goes off. You rub your eyes and unlock the screen to see a SEV 1 alert from your incident management tool. The application is down, multiple cloud server instances are offline, and the remaining instances are being overwhelmed by the sudden increase in demand.

Getting started with Latency attacks

As the world becomes more dependent on cloud-native systems, the tolerance for slow services is decreasing. Users expect instantaneous access to services, whether it's for work, entertainment, or even cloud infrastructure. Even small amounts of latency can significantly decrease user satisfaction: nearly half of all users expect web pages to load in under two seconds, and as many as 28% of users will permanently abandon a slow site. The problem is, how can we test and verify that our applications and services will perform well even when network conditions are less than ideal?

Getting started with Packet Loss attacks

Imagine this: you're in the middle of an important presentation when all of a sudden your video feed starts to stutter. You hear other people speaking, but their words are choppy. A message comes through Slack from one of your co-workers: "I think your connection cut out." You scramble to try different solutions—restarting your videoconferencing application, checking your Internet connection, switching to your phone—but ultimately, your presentation gets cut short.

How to train your engineers in Chaos Engineering

Adopting a Chaos Engineering tool is a great step towards improving reliability, but a tool is only useful if you know how to utilize it effectively.

The first 4 chaos experiments to run on Apache Kafka

Apache Kafka is an open source distributed messaging platform that can deliver data at high throughput and with low latency. At scale, it can handle trillions of records per day while also providing fault tolerance, replication, and automatic disaster recovery.

Testing disaster recovery with Chaos Engineering

COVID-19 has changed daily life for everyone around the world. Social events were cancelled, governments issued stay-at-home orders, tens of thousands of businesses locked their doors, and telecommuting became the new normal for millions of workers.

How to map out your application’s critical path

In a recent blog post, we explained how every application has a critical path, which is the set of components that are essential to the application’s operation. A failure along the critical path makes your application unavailable, which means unhappy customers, reduced revenue, and a hit to your company’s reputation. For these reasons, we need to focus on making our critical path as reliable as possible. But to do that, we first need to know which components are part of it. We need to consider:

Chaos Engineering with Minecraft

For over a decade, Minecraft has brought over 100 million players together to explore, build, and fight monsters. But as fun as it is to explore an infinite world with your friends, there’s nothing fun about losing your progress because the server crashed or you were disconnected. If you’re hosting your own server, an outage is one of the fastest ways to lose your player base. To avoid this, we need to make sure our systems are reliable as possible so that our players have the best possible experience.

Running Chaos Experiments on Confluent Platform and Apache Kafka

Apache Kafka is a data streaming pipeline popular for its low-latency, high-throughput, distributed architecture. With systems becoming more decentralized and applications handling ever-increasing amounts of data, Kafka has become a crucial backbone for many organizations. And while Kafka is designed to withstand common production failures like node outages, we can’t be absolutely confident in its resilience unless we have a way to test it.