How to Use Gremlin Scenarios to Reproduce the AWS S3 Outage

How to Use Gremlin Scenarios to Reproduce the AWS S3 Outage
Tammy Butow
Tammy Butow
Principal SRE

Overview

Gremlin is a simple, safe and secure service for performing Chaos Engineering experiments through a SaaS-based platform.

This tutorial will show you how to use Gremlin Scenarios to reproduce the AWS S3 Outage.

If you are interested in learning more about the outage, we have shared a detailed analysis of the 2017 S3 Outage on our Gremlin Blog. A good question to ask yourself at every postmortem is “how do we ensure this never happens again?”

  • Step 1 - Create a Sample App
  • Step 2 - Build and run your Sample app using Docker and NGINX
  • Step 3 - View your Sample app in your browser
  • Step 4 - View your Sample app via VNC
  • Step 5 - Running the Gremlin Unavailable Dependencies Scenario to reproduce the S3 outage
  • Step 6 - Viewing the result of the Gremlin Unavailable Dependencies Scenario

Prerequisites

Step 1 - Create a Sample App

Connect to your host with ssh and create a Dockerfile:

bash
1ssh username@your_server_ip
2
3vim Dockerfile
4
5FROM nginx:alpine
6COPY index.html /usr/share/nginx/html/index.html

Create the following index.html file:

bash
1vim index.html
html
1<html lang="en">
2  <head>
3    <title>Mythical Mysfits</title>
4    <meta charset="utf-8" />
5    <meta name="viewport" content="width=device-width, initial-scale=1" />
6    <script src="https://ajax.googleapis.com/ajax/libs/jquery/3.3.1/jquery.min.js"></script>
7    <script src="https://ajax.googleapis.com/ajax/libs/angularjs/1.5.6/angular.min.js"></script>
8    <script src="https://cdnjs.cloudflare.com/ajax/libs/popper.js/1.14.3/umd/popper.min.js"></script>
9    <script src="https://stackpath.bootstrapcdn.com/bootstrap/4.1.1/js/bootstrap.min.js"></script>
10    <link
11      rel="stylesheet"
12      href="https://stackpath.bootstrapcdn.com/bootstrap/4.1.1/css/bootstrap.min.css"
13    />
14  </head>
15
16  <body ng-app="mysfitsApp">
17    <style>
18      @media (max-width: 800px) {
19        img {
20          max-width: 300px;
21        }
22      }
23    </style>
24
25    <div style="text-align: center;">
26      <img
27        src="https://www.mythicalmysfits.com/images/aws_mythical_banner.png"
28        width="800px"
29        align="center"
30      />
31    </div>
32
33    <div class="container" ng-controller="mysfitsFilterController">
34      <div id="filterMenu">
35        <ul class="nav nav-pills">
36          <li
37            class="nav-item dropdown"
38            ng-repeat="filterCategory in filterOptionsList.categories"
39          >
40            <a
41              class="nav-link dropdown-toggle"
42              data-toggle="dropdown"
43              href="#!"
44              role="button"
45              aria-haspopup="true"
46              aria-expanded="false"
47            >
48              {{filterCategory.title}}
49            </a>
50
51            <div class="dropdown-menu">
52              <button
53                class="dropdown-item"
54                ng-repeat="filterCategorySelection in filterCategory.selections"
55                ng-click="queryMysfits(filterCategory.title, filterCategorySelection)"
56              >
57                {{filterCategorySelection}}
58              </button>
59            </div>
60          </li>
61
62          <li class="nav-item ">
63            <button
64              type="button"
65              class="btn btn-success"
66              ng-click="removeFilter()"
67            >
68              View All
69            </button>
70          </li>
71        </ul>
72      </div>
73    </div>
74
75    <br />
76
77    <div class="container">
78      <div id="mysfitsGrid" class="row" ng-controller="mysfitsListController">
79        <div
80          class="col-md-4 border border-warning"
81          ng-repeat="mysfit in mysfits"
82        >
83          <br />
84
85          <p align="center">
86            <strong> {{mysfit.name}}</strong>
87
88            <br />
89
90            <img src="{{mysfit.thumbImageUri}}" alt="{{mysfit.Name}}" />
91          </p>
92
93          <p>
94            <br />
95            <b>Species:</b> {{mysfit.species}}
96            <br />
97            <b>Age:</b> {{mysfit.age}}
98            <br />
99            <b>Good/Evil:</b> {{mysfit.goodevil}}
100            <br />
101            <b>Lawful/Chaotic:</b> {{mysfit.lawchaos}}
102          </p>
103        </div>
104      </div>
105    </div>
106
107    <p>
108      <br />
109      <br />
110      This site was created for use in the AWS Modern Application Workshop.
111      <a href="https://github.com/aws-samples/aws-modern-application-workshop"
112        >Please see details here.</a
113      >
114    </p>
115  </body>
116
117  <script>
118    var mysfitsApiEndpoint =
119      'http://mysfits-nlb-9c8e61c17ef3cd1d.elb.us-east-1.amazonaws.com';
120    var app = angular.module('mysfitsApp', []);
121    var gridScope;
122    var filterScope;
123
124    app.controller('clearFilterController', function($scope) {});
125
126    app.controller('mysfitsFilterController', function($scope) {
127      filterScope = $scope;
128
129      // The possible options for Mysfits to populate the dropdown filters.
130      $scope.filterOptionsList = {
131        categories: [
132          {
133            title: 'Good/Evil',
134            selections: ['Good', 'Neutral', 'Evil'],
135          },
136          {
137            title: 'Lawful/Chaotic',
138            selections: ['Lawful', 'Neutral', 'Chaotic'],
139          },
140        ],
141      };
142
143      $scope.removeFilter = function() {
144        allMysfits = getAllMysfits(applyGridScope);
145      };
146
147      $scope.queryMysfits = function(filterCategory, filterValue) {
148        var filterCategoryQS = '';
149
150        if (filterCategory === 'Good/Evil') {
151          filterCategoryQS = 'GoodEvil';
152        } else {
153          filterCategoryQS = 'LawChaos';
154        }
155
156        var mysfitsApi =
157          mysfitsApiEndpoint +
158          '/mysfits?' +
159          'filter=' +
160          filterCategoryQS +
161          '&value=' +
162          filterValue;
163
164        $.ajax({
165          url: mysfitsApi,
166
167          type: 'GET',
168
169          success: function(response) {
170            applyGridScope(response.mysfits);
171          },
172
173          error: function(response) {
174            console.log('could not retrieve mysfits list.');
175          },
176        });
177      };
178    });
179
180    app.controller('mysfitsListController', function($scope) {
181      gridScope = $scope;
182
183      getAllMysfits(applyGridScope);
184    });
185
186    function applyGridScope(mysfitsList) {
187      gridScope.mysfits = mysfitsList;
188
189      gridScope.$apply();
190    }
191
192    function getAllMysfits(callback) {
193      var mysfitsApi = mysfitsApiEndpoint + '/mysfits';
194
195      $.ajax({
196        url: mysfitsApi,
197
198        type: 'GET',
199
200        success: function(response) {
201          callback(response.mysfits);
202        },
203
204        error: function(response) {
205          console.log('could not retrieve mysfits list.');
206        },
207      });
208    }
209  </script>
210</html>

Save the index.html file.

Step 2 - Build and run your Sample app using Docker and NGINX

Next, build the Dockerfile by running the following:

bash
1docker build -t simple-nginx .

Now we can run our image by using

bash
1docker run -d -p 8080:80 simple-nginx

Step 3 - View your Sample app

Now you can see your sample running @ your_server_ip:8080

s3

Step 4 - View your Sample app via VNC

The Gremlin Unavailable Dependencies Scenario uses a Blackhole attack. We will be using this Blackhole attack to disallow images stored in S3 from loading. To see the results of this Blackhole Network attack we will be using a service called VNC.

On your host, install the Xfce and TightVNC packages:

bash
1sudo apt-get update
2
3sudo apt install xfce4 xfce4-goodies tightvncserver

To complete the VNC installation run the following command, you will be prompted to enter a password:

bash
1vncserver

Next, test the VNC connection on your local computer. Run the following command which uses port forwarding :

bash
1ssh -L 5901:127.0.0.1:5901 -N -f -l username server_ip_address

Now you can use a VNC client to connect to the VNC server at localhost:5901. You’ll be prompted to authenticate. Use the password you set up earlier. You can use the built-in program for Mac called Screen Sharing or VNC Viewer to view your Xfce Desktop.

On your host you will need to ensure you have a browser installed, install Firefox by running the following command:

bash
1apt-get install firefox

Before you move onto the next step, ensure that you are able to view the sample app using VNC. Connect to localhost:5901 and click on Applications > Internet > Firefox Web Browser:

S3

Using Firefox, navigate to localhost:8080, you should see the following:

S3

Now we’re ready to run the Gremlin Unavailable Dependency Scenario to reproduce the S3 Outage.

Step 5 - Running the Gremlin Unavailable Dependencies Scenario to reproduce the S3 outage

First, navigate to Recommended Scenarios within the Gremlin UI and choose the Unavailable Dependency Scenario:

S3

Next, select Add Targets and run. Then select your host using the local-hostname option:

S3

Then click customize:

S3

Next, you will click Add Attacks, this will take you to the Gremlin Attack configuration. To reproduce the S3 outage modify the default scenario to include 1 x Blackhole attack impacting AWS S3 us-east-1. You will need to make these changes in the Providers section of the Blackhole attack, see the screenshot below:

S3

Next, click Unleash Scenario. Your Gremlin Scenario will now be running and it will begin to reproduce the S3 Outage:

S3

Step 6 - Viewing the result of the Gremlin Unavailable Dependencies Scenario

To view the S3 Outage being reproduced open your VNC viewer and reload your Firefox tab, you will notice that the images stored in us-east-1 on S3 no longer load:

S3

Conclusion

This tutorial has demonstrated how you can use the Gremlin Recommended Scenario “Unavailable Dependency” to reproduce the AWS S3 Outage. This demonstrates that our sample app is not resilient to this outage.

To ensure we could reliably handle this scenario, we could run this Gremlin Scenario again after rectifying this situation with one of many options:

  • S3 failover
  • Multi-cloud storage
  • Multi-CDN
  • Handling image errors in the browser using React

Related

Chaos Engineering
Start

4 Chaos Experiments to Start With

The transition from learning about Chaos Engineering to practicing it can be difficult. Chaos Engineering as a concept…
Matt Jacobs
Matt Jacobs
Software Engineer

Avoid downtime. Use Gremlin to turn failure into resilience.

Gremlin empowers you to proactively root out failure before it causes downtime. See how you can harness chaos to build resilient systems by requesting a demo of Gremlin.

Get started
Company
  • Team
    Join us
  • Product
  • Contact
  • Press
  • Privacy
© 2021 Gremlin Inc. San Jose, CA 95113