Featured Blogs

Migrating to the Cloud Is Chaotic. Embrace It.

March 11, 2019
-
10 min read

Updated January 24, 2020

Avoiding Problems When the Clocks Change

March 6, 2019
-
2 min read

The season is upon us when clocks change for Daylight Savings Time. Sure, distributed systems are frequently set by default to use UTC as the basis for all time-related settings and those systems then perform any needed locale-specific adjustments in the agent. However, this is not true for everyone and many of us are responsible for systems that rely on timestamps in a locale that adjusts the time twice a year. For the rest of us, even when using UTC, things can get complicated with leap years and the occasional leap second.

What is Chaos Engineering? SREs and Leaders Define the Practice & Where It's Going

March 22, 2019
-
7 min read

Chaos Engineering is a practice that is growing in implementation and interest. What is it and why are some of the most successful companies in the world adopting it?

Chaos Engineering and Add-To-Cart

July 8, 2019
-
5 min read

An E-Commerce Example of Preparing for Black Friday.

Incremental Reliability Improvement

August 22, 2019
-
9 min read

If you improved reliability by just 1% each day, how long would it take for you to get that “extra 9”? That’s an interesting question. This article begins by exploring the potential for small improvements to add up like compound interest. It concludes with a list of practical steps we can take that improve system reliability.

After the Retrospective: The 2017 Amazon S3 Outage

September 16, 2019
-
9 min read

Systems fail. Even Amazon breaks. Despite our best efforts, technology is never perfect. In this first post of a series, we look at the Amazon S3 outage of 2017 in a blameless way, seeking to learn from it things we can do to enhance the reliability of our own systems.

After the Retrospective: Heroku Incident #1892

October 8, 2019
-
8 min read

All of us use various upstream dependencies to streamline our work efforts. We must also think about and plan for what happens with our system if there is a problem with an upstream dependency. In this second post of a series, we take a blameless look at the Heroku outage from 2019 where an upstream dependency on AWS affected the availability of core functionality as well as the ability to restore access quickly, seeking to learn from it things we can do to enhance the reliability of our own systems.

How to Safely Manage Change in a CI/CD World

November 6, 2019
-
10 min read

Change management exists because it ensures the attention of many eyes and that much care is taken before modifying production systems, hopefully creating some reliability. The reason that change management processes don’t exist in modern systems is that our systems change so rapidly that no review panel could possibly keep up. This article proposes a way to keep the reliability factor alive while moving to a new, better methodology.

After the Retrospective: Dyn DDoS

October 28, 2019
-
6 min read

What happens when the DNS service we use is suddenly unavailable? Users of Dyn found out when the company was attacked by a Distributed Denial of Service (DDoS). In this third post of a series, we take a blameless look at the Dyn outage, seeking to learn from it things we can do to enhance the reliability of our own systems.

Updating the Industry's Reliability Practices

October 25, 2019
-
5 min read

Companies will continue to struggle to implement good reliability practices if the only opportunity to improve are adjustments made after production failures.

Ensuring Runbooks are Up-to-Date

October 9, 2019
-
5 min read

One thing that all technical documentation for software that is still in active development has in common is that if it is not already outdated, it will be. We must be intentional if we want information to stay current. This includes runbooks.

Using Chaos Engineering to Demonstrate Regulatory Compliance

December 2, 2019
-
10 min read

Chaos Engineering is a powerful tool that can help you prove that your systems are compliant with regulations and standards surrounding risk management and disaster recovery in your enterprise IT systems.

Bring Chaos Engineering to your CI/CD pipeline

January 27, 2020
-
9 min read

Chaos Engineering is all the rage these days. So many Site Reliability Engineering (SRE) teams have started performing chaos experiments in QA testing or stage environments, while moving toward limited and expanding testing in production systems. The industry is just beginning to think about the benefits of chaos testing earlier in the process, in the CI/CD pipeline. But we expect that to change because reliability and resiliency need to shift left and start at the beginning.

Preparing for traffic spikes because more people are working remotely

May 12, 2020
-
4 min read

Networks are being stressed in unprecedented ways now that workers are connecting remotely to do their jobs. Video calls add to the network traffic spike. How do we cope effectively? How do we maintain the ability for our employees and customers to stay productive in the new paradigm?

Design thinking leads to Chaos Engineering

April 8, 2020
-
4 min read

In the Double Diamond Framework for Innovation from the Design Council in the United Kingdom, there are four defined stages in the process of creating a good design. They illustrate those stages using a diagram like this (ours is simplified slightly from theirs).

Prepare your team to handle incidents remotely

June 4, 2020
-
5 min read

Use remote FireDrills to test your team’s incident response preparedness and ensure business continuity.

Achieving FMEA goals faster with Chaos Engineering

June 11, 2020
-
8 min read

Failure mode and effects analysis (FMEA) is a decades-old method for identifying all possible failures in a design, a manufacturing or assembly process, or a product or service. In the last few years it has begun to be used by companies looking to make their computer systems better. While FMEA is not officially an ISO standard procedure like ISO 31000 Risk Management, there are ISO implementations specific to certain applications. Instead, it is broader and able to be applied to a wide range of needs.

Chaos Engineering and Windows: Mitigating common Windows failure scenarios

June 18, 2020
-
7 min read

Microsoft Windows is a popular operating system for many enterprise applications, such as Microsoft SQL Server clusters and Microsoft Exchange Servers. About 30% of the world’s web application hosting systems are running Windows, making it an important part of every enterprise’s plans to prevent outages and enhance reliability.

Technology Business Management and Chaos Engineering

September 18, 2020
-
8 min read

Technology Business Management (TBM) is a decision-making tool that helps organizations maximize the business value of information technology (IT) spending by adjusting management practices. With TBM, IT is transformed to run like a business instead of merely a cost center. Decisions are made based on overall business needs like reliability and customer satisfaction.

Featured Tutorials

No tutorials found.