4 Chaos Experiments to Start With
The transition from learning about Chaos Engineering to practicing it can be difficult. Chaos Engineering as a concept…
Matt Jacobs
Software Engineer
Shutdown Experiment Pack
Description
This Gremlin Free Shutdown Experiment Pack shares how you can utilise the Gremlin Shutdown attack. Shutdown is available with your Gremlin Free account.
What’s Included
- Ability to shutdown cloud infrastructure hosts & containers on AWS, Azure, GCP & more
- Ability to shutdown containers running locally with Docker
What we’ll break
With Gremlin Free, you have the ability to shutdown any host or container wherever it may reside.
This pack includes 4 x 5 minute experiments:
- Experiment 1: Shutdown a cloud infrastructure host using Gremlin Free
- Experiment 2: Shutdown a cloud infrastructure Docker container using Gremlin Free
- Experiment 3: Shutdown a Kubernetes pod using Gremlin Free
- Experiment 4: Shutdown a local Docker container using Gremlin Free
- Experiment 5: Shutdown a cloud infrastructure host using the Gremlin Free API
What you’ll need
- A Gremlin Free account (sign up here)
- Your Gremlin Free daemon credentials
- Cloud Infrastructure hosts (e.g. an AWS EC2 instance)
- Cloud Infrastructure containers (e.g. A DigitalOcean Docker Droplet)
- A Kubernetes cluster with access to run a daemonset (e.g. Azure AKS)
- Docker running locally (e.g. Docker for Mac)
Get ready to unleash chaos, get your credentials!
After you have created your Gremlin Free account (sign up here) you will need to get your Gremlin Free Daemon credentials.
Login to the Gremlin App using your Company name and sign-on credentials. These details were emailed to you when you signed up to start using Gremlin. Make a note of your Team ID and Secret.
Experiment 1: Shutdown a cloud infrastructure host using Gremlin Free
Step 1.0 - Installing the Gremlin Daemon and CLI
First, ssh into your server and add the Gremlin Debian repository:
bash
1echo "deb https://deb.gremlin.com/ release non-free" | sudo tee /etc/apt/sources.list.d/gremlin.list
Import the repo’s GPG key:
bash
1sudo apt-key adv --keyserver keyserver.ubuntu.com --recv-keys C81FC2F43A48B25808F9583BDFF170F324D41134 9CDB294B29A5B1E2E00C24C022E8EF3461A50EF6
Then install the Gremlin daemon and CLI:
bash
1sudo apt-get update && sudo apt-get install -y gremlind gremlin
Step 1.1 - Register your server to the Gremlin control plane
Using your Gremlin login credentials (which were emailed to you when you created your account), log in to the Gremlin App. Open Settings and copy your Team ID and Secret.
SSH into your server:
bash
1ssh -i your-key.pem ubuntu@server-ip
Initialise Gremlin by running the following command and follow the prompts to enter your Gremlin Team ID and Secret
bash
1gremlin init
Now you’re ready to run attacks using the Gremlin Free .
Step 1.2 - Run a Shutdown Attack Using Gremlin Free
First click Create Attack.
First choose your target by selecting the host you registered with Gremlin.
Next we will use the Gremlin App to create a Shutdown Attack. Choose the State Category and Select the Shutdown Attack. If you like, you can change the delay to 0 and turn reboot off to trigger an immediate full shutdown.
Click Unleash Gremlin and the Gremlin Free Shutdown Attack will shutdown your host.
Experiment 2 - Shutdown a cloud infrastructure container using Gremlin Free
Step 2.0 - Install Docker
In this step, you’ll install Docker.
Add Docker’s official GPG key:
bash
1curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo apt-key add -
Use the following command to set up the stable repository.
bash
1sudo add-apt-repository "deb [arch=amd64] https://download.docker.com/linux/ubuntu $(lsb_release -cs) stable"
Update the apt package index:
bash
1sudo apt-get update
Make sure you are about to install from the Docker repo instead of the default Ubuntu 16.04 repo:
bash
1apt-cache policy docker-ce
Install the latest version of Docker CE:
bash
1sudo apt-get install docker-ce
Docker should now be installed, the daemon started, and the process enabled to start on boot. Check that it’s running:
bash
1sudo systemctl status docker
Make sure you are in the Docker usergroup, replace $USER with your username:
bash
1sudo usermod -aG docker $USER
Log out and back in for your permissions to take effect, or type the following:
bash
1su - ${USER}
Step 2.1 - Create an htop container for monitoring
Htop is an interactive process viewer for UNIX. We’ll use it to monitor the progress of our attacks.
First create the Dockerfile for your htop container:
bash
1vim Dockerfile
Add the following to the Dockerfile:
1FROM alpine:latestRUN apk add --update htop && rm -rf /var/cache/apk/*ENTRYPOINT ["htop"]
Build the Dockerfile and tag the image:
bash
1sudo docker build -t htop .
Run htop inside a container, this will monitor the host:
bash
1sudo docker run -it --rm --pid=host htop
To exit htop, enter q.
Next we will create an NGINX container and monitor it directly by joining the container’s pid namespace.
Step 2.2 - Create an NGINX container to attack
First we will create a directory for the html page we will serve using nginx:
bash
1mkdir -p ~/docker-nginx/html
bash
1cd ~/docker-nginx/html
Create a simple HTML page:
bash
1vim index.html
Paste in this content:
1<html> <head> <title>Docker nginx tutorial</title> <link rel="stylesheet" href="https://maxcdn.bootstrapcdn.com/bootstrap/4.0.0/css/bootstrap.min.css" integrity="sha384-Gn5384xqQ1aoWXA+058RXPxPg6fy4IWvTNh0E263XmFcJlSAwiGgFAW/dAiS6JXm" crossorigin="anonymous"> </head> <body> <div class="container"> <h1>Hello it is your container speaking</h1> <p>This nginx page was created by your Docker container.</p> <p>Now it’s time to create a Gremlin attack.</p> </div> </body></html>
Create a container using the nginx Docker image:
bash
1sudo docker run -l service=nginx --name docker-nginx -p 80:80 -d -v ~/docker-nginx/html:/usr/share/nginx/html nginx
Make sure the docker-nginx container is running:
bash
1sudo docker ps
1CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES352609a67e95 nginx "nginx -g 'daemon of…" 33 seconds ago Up 32 seconds 0.0.0.0:80->80/tcp docker-nginx
Step 2.3 - Set up your Gremlin client credentials
Using your Gremlin login credentials (which were emailed to you when you created your account), log in to the Gremlin App. Open Settings and copy your Team ID and Secret.
Set the following export variables:
bash
1export GREMLIN_TEAM_ID=your_team_id
bash
1export GREMLIN_TEAM_SECRET=your_team_secret
Step 2.4 - Run the Gremlin Free Daemon in a Container
Use docker run to pull the official Gremlin Free Docker image and run the Gremlin Free daemon:
bash
1sudo docker run -d \ --net=host \ --pid=host \ --cap-add=NET_ADMIN \ --cap-add=SYS_BOOT \ --cap-add=SYS_TIME \ --cap-add=KILL \ -e GREMLIN_TEAM_ID="${GREMLIN_TEAM_ID}" \ -e GREMLIN_TEAM_SECRET="${GREMLIN_TEAM_SECRET}" \ -v /var/run/docker.sock:/var/run/docker.sock \ -v /var/log/gremlin:/var/log/gremlin \ -v /var/lib/gremlin:/var/lib/gremlin \ gremlin/gremlin daemon
Make sure to pass in the three environment variables you set in Step 4. If you don’t, the Gremlin daemon cannot connect to the Gremlin backend.
Use docker ps to see all running Docker containers:
bash
1sudo docker ps
1CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES7167cacb2536 gremlin/gremlin "/entrypoint.sh daem…" 40 seconds ago Up 39 seconds practical_benzfb58b77e5ef8 nginx "nginx -g 'daemon of…" 10 minutes ago Up 10 minutes 0.0.0.0:80->80/tcp docker-nginx
Jump into your Gremlin container with an interactive shell (replace 7167cacb2536 with the real ID of your Gremlin container):
bash
1sudo docker exec -it 7167cacb2536 /bin/bash
From within the container, check out the available attack types:
bash
1gremlin help attack-container
1Usage: gremlin attack-container CONTAINER TYPE [type-specific-options]Type "gremlin help attack-container TYPE" for more details: blackhole # An attack which drops all matching network traffic cpu # An attack which consumes CPU resources io # An attack which consumes IO resources latency # An attack which adds latency to all matching network traffic memory # An attack which consumes memory packet_loss # An attack which introduces packet loss to all matching network traffic shutdown # An attack which forces the target to shutdown dns # An attack which blocks access to DNS servers time_travel # An attack which changes the system time. disk # An attack which consumes disk resources process_killer # An attack which kills the specified process
Then exit the container.
Step 2.5 - Run a Shutdown Attack against the NGINX container from a Gremlin Container
In this step we will run gremlin attack-container to target the NGINX container by its ID and run a Shutdown Attack against it.
Before running the Shutdown attack, use htop to monitor the docker-nginx container (replace f291a040a6aa with your docker-nginx container ID):
bash
1sudo docker run -it --rm --pid=container:f291a040a6aa htop
You will see the following:
11 [ 0.0%] Tasks: 3, 0 thr; 1 running 2 [| 0.7%] Load average: 0.72 0.41 0.21 Mem[||||||||||||||||||||||||| 141M/3.86G] Uptime: 00:30:34 Swp[ 0K/0K] PID USER PRI NI VIRT RES SHR S CPU% MEM% TIME+ Command 47 root 20 0 4488 2236 932 R 0.0 0.1 0:00.07 htop 1 root 20 0 32428 5180 4504 S 0.0 0.1 0:00.03 nginx: master process nginx -g daemon off; 8 101 20 0 32900 2476 1448 S 0.0 0.1 0:00.00 nginx: worker process
Run the following to create the Shutdown container attack against the container (replace f291a040a6aa with your docker-nginx container ID):
bash
1sudo docker run -d -it \ --cap-add=NET_ADMIN \ -e GREMLIN_TEAM_ID="${GREMLIN_TEAM_ID}" \ -e GREMLIN_TEAM_CERTIFICATE_OR_FILE="${GREMLIN_TEAM_CERTIFICATE_OR_FILE}" \ -e GREMLIN_TEAM_PRIVATE_KEY_OR_FILE="${GREMLIN_TEAM_PRIVATE_KEY_OR_FILE}" \ -v /var/run/docker.sock:/var/run/docker.sock \ gremlin/gremlin attack-container f291a040a6aa shutdown
Experiment 3 - Shutdown a Kubernetes pod using Gremlin Free
Kubernetes is a container management system which is built with reliability in mind. Architecture is commonly 1 master and 2 or more nodes which are replicated from the master. When the master dies the nodes are ready to replace it. When one node dies another will be ready to replace it.
To create a Kubernetes cluster follow our guide on "How to Use and Install Kuberenetes with Weave Net".
Step 3.0 - Retrieve Your Team ID and Secret Key
To install the Gremlin agent and Kubernetes client, you will need your Gremlin Team ID and Secret Key. If you don’t know what your Team ID and Secret Key are, you can get them from the Gremlin web app.
Visit the Teams page in Gremlin, and then click on your team’s name in the list.
On the Teams screen click on Configuration.
Make a note of your Team ID.
If you don’t know your Secret Key, you will need to reset it. Click the Reset button. You’ll get a popup reminding you that any running clients using the current Secret Key will need to be configured with the new key. Hit Continue.
Next you’ll see a popup screen that will show you the new Secret Key. Make a note of it.
Step 3.1 - Install the Gremlin Agent with Helm
The simplest way to install the Gremlin agent on your Kubernetes cluster is to use Helm. If you do not already have Helm installed, go here to get started. Once Helm is installed and configured, the next steps are to add the Gremlin repo and install the client.
Add the Gremlin Helm chart:
bash
1helm repo add gremlin https://helm.gremlin.com
Create a namespace for the Gremlin Kubernetes client:
bash
1kubectl create namespace gremlin
Next you will run the helm command to install the Gremlin client. In this command there are three placeholder variables that you will need to replace with real data. Replace $GREMLIN_TEAM_ID with your Team ID from step1, and replace $GREMLIN_TEAM_SECRET with your Secret Key from step 1. Replace $GREMLIN_CLUSTER_ID with a name for the cluster.
If you are using Helm v3, run this command:
bash
1helm install gremlin gremlin/gremlin \2 --namespace gremlin \3 --set gremlin.secret.managed=true \4 --set gremlin.secret.type=secret \5 --set gremlin.secret.teamID=$GREMLIN_TEAM_ID \6 --set gremlin.secret.clusterID=$GREMLIN_CLUSTER_ID \7 --set gremlin.secret.teamSecret=$GREMLIN_TEAM_SECRET
For older versions of Helm, use the --name option:
bash
1helm install gremlin/gremlin \2 --name gremlin \3 --namespace gremlin \4 --set gremlin.secret.managed=true \5 --set gremlin.secret.type=secret \6 --set gremlin.secret.teamID=$GREMLIN_TEAM_ID \7 --set gremlin.secret.clusterID=$GREMLIN_CLUSTER_ID \8 --set gremlin.secret.teamSecret=$GREMLIN_TEAM_SECRET
If you’re not sure which version of Helm you’re using, run this command:
bash
1helm version
For more information on the Gremlin Helm chart, including more configuration options, check out the chart on Github.
Step 3.2 - Creating attacks using the Gremlin App
Example: Creating a Shutdown Attack against a Kubernetes node using the Gremlin App
You can use the Gremlin App or the Gremlin API to trigger Gremlin attacks. You can view the available range of Gremlin Attacks in Gremlin Help.
To create a Shutdown Attack click Attacks in the left Navigation bar and New Attack.
Host targeting should be selected by default. Click on the Exact button to expand the list of available hosts, and select one of them. You’ll see the Blast Radius for the attack is limited to 1 host.
Click “Choose a Gremlin,” and then select State and Shutdown.
Leave the Delay set to 1 minute. This is the delay before the attack begins. Leave the Reboot radio button set to On. This will start the host up again after the shutdown.
When your attack is finished it will move to Completed Attacks in the Gremlin App. To view the logs of the Attack, click on the Attack in Completed Attacks then click to the arrow to view the logs.
Experiment 4 - Shutdown a local Docker container using Gremlin Free
Step 4.0 - Install Docker For Mac
First you will need to install Docker For Mac if you do not yet have it on your local computer, follow the instructions provided by Docker.
Step 4.1 - Set up your Gremlin credentials
The Gremlin daemon (gremlind) connects to the Gremlin backend and waits for attack orders from you. When it receives attack orders, it uses the CLI (gremlin) to run the attack.
bash
1export GREMLIN_TEAM_ID=your_team_id
bash
1export GREMLIN_TEAM_SECRET=your_team_secret
Step 4.2 - Create a Gremlin Free Docker container
Use docker pull to pull the official Gremlin Free Docker image:
bash
1docker pull gremlin/gremlin
Step 4.3 - Create an NGINX container to attack
First we will create a directory for the html page we will serve using nginx:
bash
1mkdir -p ~/docker-nginx/html2cd ~/docker-nginx/html
Create a simple HTML page:
bash
1vim index.html
Paste in this content:
1<html> <head> <title>Docker nginx tutorial</title> <link rel="stylesheet" href="https://maxcdn.bootstrapcdn.com/bootstrap/4.0.0/css/bootstrap.min.css" integrity="sha384-Gn5384xqQ1aoWXA+058RXPxPg6fy4IWvTNh0E263XmFcJlSAwiGgFAW/dAiS6JXm" crossorigin="anonymous"> </head> <body> <div class="container"> <h1>Hello it is your container speaking</h1> <p>This nginx page was created by your Docker container.</p> <p>Now it's time to create a Gremlin attack.</p> </div> </body></html>
Create a container using the nginx Docker image:
bash
1sudo docker run -l service=nginx --name docker-nginx -p 80:80 -d -v ~/docker-nginx/html:/usr/share/nginx/html nginx
Make sure the docker-nginx container is running:
bash
1sudo docker ps
1CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES7167cacb2536 gremlin/gremlin "/entrypoint.sh daem…" 40 seconds ago Up 39 seconds practical_benzfb58b77e5ef8 nginx "nginx -g 'daemon of…" 10 minutes ago Up 10 minutes 0.0.0.0:80->80/tcp docker-nginx
Step 4.4 - Run A Gremlin Free Shutdown Attack
Now use the Gremlin CLI (gremlin) to run a Shutdown attack from within a Gremlin container:
bash
1sudo docker run -i \ --cap-add=NET_ADMIN \ -e GREMLIN_TEAM_ID="${GREMLIN_TEAM_ID}" \ -e GREMLIN_TEAM_SECRET="${GREMLIN_TEAM_SECRET}" \ -v /var/run/docker.sock:/var/run/docker.sock \ gremlin/gremlin attack-container docker-nginx shutdown
This attack will shutdown your nginx container.
Experiment 5: Shutdown a cloud infrastructure host using the Gremlin Free API
It is also possible to run attacks programmatically using the Gremlin Free API. We will run these attacks from a Mac OS laptop.
Step 5.0 - Installing the Gremlin Daemon and CLI
First, ssh into your server and add the Gremlin Debian repository:
bash
1echo "deb https://deb.gremlin.com/ release non-free" | sudo tee /etc/apt/sources.list.d/gremlin.list
Import the repo’s GPG key:
bash
1sudo apt-key adv --keyserver keyserver.ubuntu.com --recv-keys C81FC2F43A48B25808F9583BDFF170F324D41134 9CDB294B29A5B1E2E00C24C022E8EF3461A50EF6
Then install the Gremlin daemon and CLI:
bash
1sudo apt-get update && sudo apt-get install -y gremlind gremlin
Step 5.1 - Register your server to the Gremlin control plane
Using your Gremlin login credentials (which were emailed to you when you created your account), log in to the Gremlin App. Open Settings and copy your Team ID and Secret.
Initialise Gremlin by running the following command and follow the prompts to enter your Gremlin Team ID and Secret:
bash
1gremlin init
Now you’re ready to obtain your Gremlin Free API token.
Step 5.2 - Obtain your Gremlin Free API token
First access your Gremlin Free API token by providing your gremlin username and password to /users/auth, replacing your email, password and company name in the curl request below:
bash
1curl -X POST --header 'Content-Type: application/x-www-form-urlencoded' \ --data-urlencode 'email=name@youremail.com' \ --data-urlencode 'password=changeit' \ --data-urlencode 'companyName=changeit' \ 'https://api.gremlin.com/v1/users/auth'
Your API token will be displayed on the screen. The example below shows "Bearer Y4MGE3MGMzOmdyZW1saW5AZ3JlbWxpbmluYy5jb206NWNhMWFiMWU" as the API token that will be used for interacting with the Gremlin Free API:
1[ { "expires_at": "2099-01-00T00:00:00.000Z", "header": "Bearer Y4MGE3MGMzOmdyZW1saW5AZ3JlbWxpbmluYy5jb206NWNhMWFiMWU", "identifier": "yourname@email.com", "org_id": "e7352a6b-a9a0-513c-8000-980f680a70c3", "org_name": "My Org (Production)", "renew_token": "5ca1ab1e-ffff-0000ffff0001", "role": "USER", "token": "5ca1ab1e-ffff-0000ffff0000" },]
Step 5.3 - Store your Gremlin Free API token
On your local computer (Mac OS), store your Gremlin Free API token as an environment variable:
bash
1export bearertoken="Bearer Y4MGE3MGMzOmdyZW1saW5AZ3JlbWxpbmluYy5jb206NWNhMWFiMWU"
Check to ensure you have set your token correctly:
bash
1echo $bearertoken
Your API token will be displayed on your terminal screen:
1Bearer Y4MGE3MGMzOmdyZW1saW5AZ3JlbWxpbmluYy5jb206NWNhMWFiMWU
Now you’re ready to run attacks using the Gremlin Free API .
Step 5.4 - Run a Shutdown Attack Using The Gremlin Free API
Now you’re ready to run a shutdown attack using the Gremlin Free API . Run the following command on your local computer (Mac OS):
bash
1curl --header "Content-Type: application/json" \ --header "Authorization: $bearertoken" \ https://api.gremlin.com/v1/attacks/new \ --data ' { "command": { "type": "shutdown" }, "target": { "type": "Random" } }'
View the attack in progress using the Gremlin Free API:
bash
1curl -X GET "https://api.gremlin.com/v1/attacks/active" -H "Authorization: $bearertoken" -H "accept: application/json"
Your current attack will be displayed in terminal:
1[ { "target_type": "Host", "targets": [ "docker-for-desktop" ], "org_id": "3f242793-018a-5ad5-8000-fb958f8dc084", "args": [ "shutdown" ], "created_at": "2019-03-07T18:56:24.232Z", "create_source": "Api", "stage": "Pending", "stage_lifecycle": "Active", "guid": "b452e2a6-410a-11e9-95a4-0242cbd04b28", "start_time": "2019-03-07T18:56:24.232Z", "create_user": "youremail@email.com", "updated_at": "2019-03-07T18:57:31.191Z", "kind": "Api" }
Further Attacks
Gremlin free unlocks the ability to perform Shutdown and CPU attacks. To unlock further attacks upgrade your Gremlin account by contacting our team sales@gremlin.com.
Related
A Primer on Automating Chaos
Automation is a must when operating and scaling cloud-based systems. The more servers and services there are to manage…
Mathias Lafeldt
Infrastructure Developer
Chaos Engineering Adoption Guide
Introduction Welcome to Chaos Engineering with Gremlin! This document will serve as your guide to implementing Chaos…
Andre Newman
Technical Writer
Avoid downtime. Use Gremlin to turn failure into resilience.
Gremlin empowers you to proactively root out failure before it causes downtime. See how you can harness chaos to build resilient systems by requesting a demo of Gremlin.
Get started© 2021 Gremlin Inc. San Jose, CA 95113