Netflix released Chaos Monkey as open-source software in 2012. The tool randomly terminates virtual servers and containers in production during weekday office hours. The terminations test a straightforward operational question: when a production instance abruptly disappears, can the surrounding service continue delivering video without interruption?
The philosophy behind the tool was to expose failure intolerance deliberately rather than wait for an inconvenient outage. In large-scale cloud environments, server instances inevitably fail. Instead of hoping instances would survive, engineers built Chaos Monkey to induce failures repeatedly during business hours, encouraging developers to design services that tolerate instance loss.
Testing Recovery in Daylight
The most distinctive constraint of Chaos Monkey is when it chooses to act. According to the original Simian Army documentation, the service operates during business hours, excluding weekends and holidays. Scheduling disruptions during working hours is intended to expose problems while engineering teams are available to investigate and respond.
The alternative is discovering the same weakness through an emergency page outside working hours. If an application cannot survive losing an instance, uncovering that flaw during the day allows engineers to analyze system behavior in real time and repair the underlying issue.
Controlled Randomness
Despite its provocative name, Chaos Monkey does not practice indiscriminate destruction. In its original open-source configuration within the Netflix Simian Army suite, safety controls governed every potential termination. In the default configuration, the system was enabled but explicitly “leashed”: it scanned instance groups and selected candidates for termination, but did not actually shut them down.
With default settings, permitting actual shutdowns required setting the leash property to false and opting in the target Auto Scaling Groups, though teams could also configure group participation as opt-out. The framework also enforced rate limiting, defaulting to a maximum of one termination per day for any given group. In the modern version integrated with Spinnaker, the tool schedules terminations on weekdays between 9 AM and 3 PM, using a probabilistic check to decide whether a termination happens. These constraints limit when disruptions occur and which groups they affect, but do not guarantee that an experiment will avoid an outage.
The Failure Modes Under Scrutiny
By regularly terminating individual instances, Chaos Monkey exercises several critical operational assumptions that passive monitoring cannot verify. First, it tests whether load balancers correctly detect failures and route requests around them.
Second, it verifies that the infrastructure can reliably rebuild and initialize replacement instances from automated configurations. In fast-moving engineering environments, an engineer might apply a temporary patch directly to a live server during an emergency and forget to commit the change to source control. Terminating that server can expose a discrepancy between the running system and its recorded configuration when a replacement is rebuilt.
The Boundaries of Chaos Engineering
Understanding what Chaos Monkey tests also requires understanding what it leaves untouched. The utility specifically targets virtual machine instances and containers. It does not simulate regional data center failures, inject network packet latency, or test database storage corruption—scenarios that require distinct testing mechanisms.
Furthermore, as Google’s Site Reliability Engineering book notes in its discussion of statistical testing, passing a series of random failure tests does not constitute mathematical proof of complete reliability. Because random terminations may not exercise every code path or race condition, testing builds empirical confidence rather than absolute certainty. Formal chaos engineering principles emphasize minimizing and containing customer impact, while allowing for some short-term negative effects.
Designing for Inevitable Loss
For engineers, Chaos Monkey suggests a practical distinction between keeping a particular server alive and keeping a service available. Repeated instance failures encourage teams to build services that tolerate the loss of individual machines, testing whether the remaining system can handle requests and whether a replacement can start cleanly from the recorded configuration.
The useful question is the one a disappearing server makes concrete: can the service keep running when that machine is gone?
Related stories
Sources
- Netflix releases Chaos Monkey as open source, Le Monde Informatique, 1 August 2012
- Chaos Monkey, Netflix SimianArmy Wiki
- Chaos Settings, Netflix SimianArmy Wiki
- Chaos Monkey Repository, Netflix
- Termination Behavior, Chaos Monkey Documentation
- Principles of Chaos Engineering
- Testing for Reliability, Google Site Reliability Engineering
- Featured image: OboeBlanket; Wikimedia Commons; CC0 (adapted from the original)
SIGNAL TITANS
Leave a Reply