How to Use Gremlin Scenarios to Reproduce the AWS S3 Outage

How to Use Gremlin Scenarios to Reproduce the AWS S3 Outage

Overview

Gremlin is a simple, safe and secure service for performing Chaos Engineering experiments through a SaaS-based platform.

This tutorial will show you how to use Gremlin Scenarios to reproduce the AWS S3 Outage.

If you are interested in learning more about the outage, we have shared a detailed analysis of the 2017 S3 Outage on our Gremlin Blog. A good question to ask yourself at every postmortem is “how do we ensure this never happens again?”

  • Step 1 - Create a Sample App
  • Step 2 - Build and run your Sample app using Docker and NGINX
  • Step 3 - View your Sample app in your browser
  • Step 4 - View your Sample app via VNC
  • Step 5 - Running the Gremlin Unavailable Dependencies Scenario to reproduce the S3 outage
  • Step 6 - Viewing the result of the Gremlin Unavailable Dependencies Scenario

Prerequisites

Step 1 - Create a Sample App

Connect to your host with ssh and create a Dockerfile:

bash
1ssh username@your_server_ip
2
3vim Dockerfile
4
5FROM nginx:alpine
6COPY index.html /usr/share/nginx/html/index.html

Create the following index.html file:

bash
1vim index.html
html
1<html lang="en">
2 <head>
3 <title>Mythical Mysfits</title>
4 <meta charset="utf-8" />
5 <meta name="viewport" content="width=device-width, initial-scale=1" />
6 <script src="https://ajax.googleapis.com/ajax/libs/jquery/3.3.1/jquery.min.js"></script>
7 <script src="https://ajax.googleapis.com/ajax/libs/angularjs/1.5.6/angular.min.js"></script>
8 <script src="https://cdnjs.cloudflare.com/ajax/libs/popper.js/1.14.3/umd/popper.min.js"></script>
9 <script src="https://stackpath.bootstrapcdn.com/bootstrap/4.1.1/js/bootstrap.min.js"></script>
10 <link
11 rel="stylesheet"
12 href="https://stackpath.bootstrapcdn.com/bootstrap/4.1.1/css/bootstrap.min.css"
13 />
14 </head>
15
16 <body ng-app="mysfitsApp">
17 <style>
18 @media (max-width: 800px) {
19 img {
20 max-width: 300px;
21 }
22 }
23 </style>
24
25 <div style="text-align: center;">
26 <img
27 src="https://www.mythicalmysfits.com/images/aws_mythical_banner.png"
28 width="800px"
29 align="center"
30 />
31 </div>
32
33 <div class="container" ng-controller="mysfitsFilterController">
34 <div id="filterMenu">
35 <ul class="nav nav-pills">
36 <li
37 class="nav-item dropdown"
38 ng-repeat="filterCategory in filterOptionsList.categories"
39 >
40 <a
41 class="nav-link dropdown-toggle"
42 data-toggle="dropdown"
43 href="#!"
44 role="button"
45 aria-haspopup="true"
46 aria-expanded="false"
47 >
48 {{filterCategory.title}}
49 </a>
50
51 <div class="dropdown-menu">
52 <button
53 class="dropdown-item"
54 ng-repeat="filterCategorySelection in filterCategory.selections"
55 ng-click="queryMysfits(filterCategory.title, filterCategorySelection)"
56 >
57 {{filterCategorySelection}}
58 </button>
59 </div>
60 </li>
61
62 <li class="nav-item ">
63 <button
64 type="button"
65 class="btn btn-success"
66 ng-click="removeFilter()"
67 >
68 View All
69 </button>
70 </li>
71 </ul>
72 </div>
73 </div>
74
75 <br />
76
77 <div class="container">
78 <div id="mysfitsGrid" class="row" ng-controller="mysfitsListController">
79 <div
80 class="col-md-4 border border-warning"
81 ng-repeat="mysfit in mysfits"
82 >
83 <br />
84
85 <p align="center">
86 <strong> {{mysfit.name}}</strong>
87
88 <br />
89
90 <img src="{{mysfit.thumbImageUri}}" alt="{{mysfit.Name}}" />
91 </p>
92
93 <p>
94 <br />
95 <b>Species:</b> {{mysfit.species}}
96 <br />
97 <b>Age:</b> {{mysfit.age}}
98 <br />
99 <b>Good/Evil:</b> {{mysfit.goodevil}}
100 <br />
101 <b>Lawful/Chaotic:</b> {{mysfit.lawchaos}}
102 </p>
103 </div>
104 </div>
105 </div>
106
107 <p>
108 <br />
109 <br />
110 This site was created for use in the AWS Modern Application Workshop.
111 <a href="https://github.com/aws-samples/aws-modern-application-workshop"
112 >Please see details here.</a
113 >
114 </p>
115 </body>
116
117 <script>
118 var mysfitsApiEndpoint =
119 'http://mysfits-nlb-9c8e61c17ef3cd1d.elb.us-east-1.amazonaws.com';
120 var app = angular.module('mysfitsApp', []);
121 var gridScope;
122 var filterScope;
123
124 app.controller('clearFilterController', function($scope) {});
125
126 app.controller('mysfitsFilterController', function($scope) {
127 filterScope = $scope;
128
129 // The possible options for Mysfits to populate the dropdown filters.
130 $scope.filterOptionsList = {
131 categories: [
132 {
133 title: 'Good/Evil',
134 selections: ['Good', 'Neutral', 'Evil'],
135 },
136 {
137 title: 'Lawful/Chaotic',
138 selections: ['Lawful', 'Neutral', 'Chaotic'],
139 },
140 ],
141 };
142
143 $scope.removeFilter = function() {
144 allMysfits = getAllMysfits(applyGridScope);
145 };
146
147 $scope.queryMysfits = function(filterCategory, filterValue) {
148 var filterCategoryQS = '';
149
150 if (filterCategory === 'Good/Evil') {
151 filterCategoryQS = 'GoodEvil';
152 } else {
153 filterCategoryQS = 'LawChaos';
154 }
155
156 var mysfitsApi =
157 mysfitsApiEndpoint +
158 '/mysfits?' +
159 'filter=' +
160 filterCategoryQS +
161 '&value=' +
162 filterValue;
163
164 $.ajax({
165 url: mysfitsApi,
166
167 type: 'GET',
168
169 success: function(response) {
170 applyGridScope(response.mysfits);
171 },
172
173 error: function(response) {
174 console.log('could not retrieve mysfits list.');
175 },
176 });
177 };
178 });
179
180 app.controller('mysfitsListController', function($scope) {
181 gridScope = $scope;
182
183 getAllMysfits(applyGridScope);
184 });
185
186 function applyGridScope(mysfitsList) {
187 gridScope.mysfits = mysfitsList;
188
189 gridScope.$apply();
190 }
191
192 function getAllMysfits(callback) {
193 var mysfitsApi = mysfitsApiEndpoint + '/mysfits';
194
195 $.ajax({
196 url: mysfitsApi,
197
198 type: 'GET',
199
200 success: function(response) {
201 callback(response.mysfits);
202 },
203
204 error: function(response) {
205 console.log('could not retrieve mysfits list.');
206 },
207 });
208 }
209 </script>
210</html>

Save the index.html file.

Step 2 - Build and run your Sample app using Docker and NGINX

Next, build the Dockerfile by running the following:

bash
1docker build -t simple-nginx .

Now we can run our image by using

bash
1docker run -d -p 8080:80 simple-nginx

Step 3 - View your Sample app

Now you can see your sample running @ your_server_ip:8080

s3

Step 4 - View your Sample app via VNC

The Gremlin Unavailable Dependencies Scenario uses a Blackhole attack. We will be using this Blackhole attack to disallow images stored in S3 from loading. To see the results of this Blackhole Network attack we will be using a service called VNC.

On your host, install the Xfce and TightVNC packages:

bash
1sudo apt-get update
2
3sudo apt install xfce4 xfce4-goodies tightvncserver

To complete the VNC installation run the following command, you will be prompted to enter a password:

bash
1vncserver

Next, test the VNC connection on your local computer. Run the following command which uses port forwarding :

bash
1ssh -L 5901:127.0.0.1:5901 -N -f -l username server_ip_address

Now you can use a VNC client to connect to the VNC server at localhost:5901. You’ll be prompted to authenticate. Use the password you set up earlier. You can use the built-in program for Mac called Screen Sharing or VNC Viewer to view your Xfce Desktop.

On your host you will need to ensure you have a browser installed, install Firefox by running the following command:

bash
1apt-get install firefox

Before you move onto the next step, ensure that you are able to view the sample app using VNC. Connect to localhost:5901 and click on Applications > Internet > Firefox Web Browser:

S3

Using Firefox, navigate to localhost:8080, you should see the following:

S3

Now we’re ready to run the Gremlin Unavailable Dependency Scenario to reproduce the S3 Outage.

Step 5 - Running the Gremlin Unavailable Dependencies Scenario to reproduce the S3 outage

First, navigate to Recommended Scenarios within the Gremlin UI and choose the Unavailable Dependency Scenario:

S3

Next, select Add Targets and run. Then select your host using the local-hostname option:

S3

Then click customize:

S3

Next, you will click Add Attacks, this will take you to the Gremlin Attack configuration. To reproduce the S3 outage modify the default scenario to include 1 x Blackhole attack impacting AWS S3 us-east-1. You will need to make these changes in the Providers section of the Blackhole attack, see the screenshot below:

S3

Next, click Unleash Scenario. Your Gremlin Scenario will now be running and it will begin to reproduce the S3 Outage:

S3

Step 6 - Viewing the result of the Gremlin Unavailable Dependencies Scenario

To view the S3 Outage being reproduced open your VNC viewer and reload your Firefox tab, you will notice that the images stored in us-east-1 on S3 no longer load:

S3

Conclusion

This tutorial has demonstrated how you can use the Gremlin Recommended Scenario “Unavailable Dependency” to reproduce the AWS S3 Outage. This demonstrates that our sample app is not resilient to this outage.

To ensure we could reliably handle this scenario, we could run this Gremlin Scenario again after rectifying this situation with one of many options:

  • S3 failover
  • Multi-cloud storage
  • Multi-CDN
  • Handling image errors in the browser using React

Related

Avoid downtime. Use Gremlin to turn failure into resilience.

Gremlin empowers you to proactively root out failure before it causes downtime. See how you can harness chaos to build resilient systems by requesting a demo of Gremlin.

Get started
  • TechCrunch
  • Forbes
  • Business Insider
  • VentureBeat


© 2020 Gremlin Inc. San Jose, CA 95113