Daemon & Client Overview

Architecture

Client <--> Daemon --> Control Plane

The Gremlin daemon, gremlind is a binary installed on the operating system or available inside the Gremlin container. It heartbeats with the Gremlin Control Plane to let Gremlin know that the host is active and able to receive attack orders. It only communicates outbound with the Gremlin Control Plane. All traffic is encrypted.

The Gremlin client, gremlin, refers to the Gremlin command line interface that is responsible for creating the local impact within the host.

The daemon bundled with the command line interface as a unit is referred to as a targetable Client to the platform.

Client Lifecycle

Gremlin clients (infrastructure and application) that have been authenticated to the Gremlin Control Plane appear in the infrastructure clients and application clients lists. You can only run attacks on "active" clients. A client goes into an "idle" state if there is no activity for the past 5 minutes. You cannot run or schedule attacks on idle clients. If Gremlin does not hear from these idle clients for a period of 24 hours, the clients are removed from the list. However, if a client starts communicating with Gremlin again while still within the 24 hour idle window, the client is reactivated and returned to the "active" state.

Logs

Logs can be found under the /var/log/gremlin directory.

Daemon log entries can be found in the daemon.log file. Log entries in this file may indicate events where the daemon is not able to communicate with the Control Plane and eventually trigger the Dead Man Switch.

Each attack on the host is logged under /var/log/gremlin/executions using its unique attack execution ID.

Log Size

To see how much disk space is being used by logs, run the du utility on the /var/log/gremlin directory:

1du -sh /var/log/gremlin

Bandwidth Usage

Idle State

The daemon uses very little bandwidth in its idle state. In testing over a 15 minute period the daemon used only an average of 1.15 KB/sec over any given 10 seconds.

1Inbound bandwidth average, zeros dropped
21.98918829974
3
4Outbound bandwidth average, zeros dropped
50.55621495
6
7Aggregate bandwidth average, zeros dropped
82.51990083564
9
10Aggregate bandwidth average over testing period
111.15347573462

Attack State

While testing, there is a slight increase in overall bandwidth consumption during attacks. While attacks are being executed, the daemon stays in constant communication with the control plane as it checks for the abort condition to be executed. Regardless of attack being run, the attack state behavior looks the same. Provided are two data sets, one from a CPU Resource attack, and one from a Latency Network attack.

1Inbound bandwidth average, zeros dropped
22.71233697984
3
4Outbound bandwidth average, zeros dropped
50.87034858716
6
7Aggregate bandwidth average, zeros dropped
83.56685480642
9
10Aggregate bandwidth average over testing period
113.0556056175

1Inbound bandwidth average, zeros dropped
22.64944416154
3
4Outbound bandwidth average, zeros dropped
50.917963469407
6
7Aggregate bandwidth average, zeros dropped
83.55221005449
9
10Aggregate bandwidth average over testing period
113.11712392366

There is no statistically significant difference between the two attacks. Despite a nearly 3X rise in aggregate average traffic, the 3 KB/sec bandwidth utilization is still very low on a per client basis.