Chaos engineering

Short description: In software engineering, experimenting with the product to test extreme situations

Not to be confused with The Chaos Engine.

Chaos engineering is the discipline of experimenting on a system in order to build confidence in the system's capability to withstand turbulent conditions in production.[1]

## Contents

* 1 Concept
* 2 Operational readiness using chaos engineering
* 3 History
* 4 Chaos engineering tools
* 4.1 Chaos Monkey
* 4.1.1 Simian Army
* 4.2 Other
* 5 See also
* 6 Notes and references
* 7 External links

## Concept

In software development, the ability of a given software to tolerate failures while still ensuring adequate quality of service—often termed resilience—is typically specified as a requirement. However, development teams may fail to meet this requirement due to factors such as short deadlines or lack of domain knowledge. Chaos engineering encompasses techniques aimed at meeting resilience requirements.

Chaos engineering can be used to achieve resilience against infrastructure failures,[2] network failures,[3] and application failures; the latter includes resilience to unhandled exceptions.[4]

## Operational readiness using chaos engineering

Calculating how much confidence we have in the interconnected complex systems that are put into production environments requires operational readiness metrics. Operational readiness can be evaluated using chaos engineering simulations. Solutions for increasing the resilience and operational readiness of a platform include strengthening the backup, restore, network file transfer, failover capabilities and overall security of the environment.

An evaluation to induce chaos in a Kubernetes environment terminated random pods receiving data from edge devices in data centers while processing analytics on a big data network. The pods' recovery time was a resiliency metric that estimated the response time.[5][6]

## History

1983 – Apple

While MacWrite and MacPaint were being developed for the first Apple Macintosh computer, Steve Capps created "Monkey", a desk accessory which randomly generated user interface events at high speed, simulating a monkey frantically banging the keyboard and moving and clicking the mouse. It was promptly put to use for debugging by generating errors for programmers to fix, because automated testing was not possible; the first Macintosh had too little free memory space for anything more sophisticated.[7]

1992 – Prologue While ABAL2 and SING were being developed for the first graphical versions of the PROLOGUE operating system, Iain James Marshall created "La Matraque", a desk accessory which randomly generated random sequences of both legal and invalid graphical interface events, at high speed, thus testing the critical edge behaviour of the underlying graphics libraries. This program would be launched prior to production delivery, for days on end, thus ensuring the required degree of total resilience. This tool was subsequently extended to include the Database and other File Access instructions of the ABAL language to check and ensure their subsequent resiliance. A variation of this tool is currently employed for the qualification of the modern day version known as OPENABAL.

2003 – Amazon

While working to improve website reliability at Amazon, Jesse Robbins created "Game day",[8] an initiative that increases reliability by purposefully creating major failures on a regular basis. Robbins has said it was inspired by firefighter training and research in other fields lessons in complex systems, reliability engineering.[9]

2006 – Google

While at Google, Kripa Krishnan created a similar program to Amazon's Game day (see above) called "DiRT" (Disaster Recovery Testing).[9][10][11] Jason Cahoon, a Site Reliability Engineer [12] at Google, contributed a chapter on Google DiRT [13] in the "Chaos Engineering" book [14] and described the system at the GOTOpia 2021 conference.[15]

2011 – Netflix

While overseeing Netflix's migration to the cloud in 2011 Nora Jones, Casey Rosenthal, and Greg Orzell [14][16][17] expanded the discipline while working together at Netflix by setting up a tool that would cause breakdowns in their production environment, the environment used by Netflix customers. The intent was to move from a development model that assumed no breakdowns to a model where breakdowns were considered to be inevitable, driving developers to consider built-in resilience to be an obligation rather than an option:

> "At Netflix, our culture of freedom and responsibility led us not to force engineers to design their code in a specific way. Instead, we discovered that we could align our teams around the notion of infrastructure resilience by isolating the problems created by server neutralization and pushing them to the extreme. We have created Chaos Monkey, a program that randomly chooses a server and disables it during its usual hours of activity. Some will find that crazy, but we could not depend on the random occurrence of an event to test our behavior in the face of the very consequences of this event. Knowing that this would happen frequently has created a strong alignment among engineers to build redundancy and process automation to survive such incidents, without impacting the millions of Netflix users. Chaos Monkey is one of our most effective tools to improve the quality of our services."[18]

By regularly "killing" random instances of a software service, it was possible to test a redundant architecture to verify that a server failure did not noticeably impact customers.

The concept of chaos engineering is close to the one of Phoenix Servers, first introduced by Martin Fowler in 2012.[19]

## Chaos engineering tools

### Chaos Monkey

Chaos Monkey is a tool invented in 2011 by Netflix to test the resilience of its IT infrastructure.[16] It works by intentionally disabling computers in Netflix's production network to test how the remaining systems respond to the outage. Chaos Monkey is now part of a larger suite of tools called the Simian Army designed to simulate and test responses to various system failures and edge cases.

The code behind Chaos Monkey was released by Netflix in 2012 under an Apache 2.0 license.[20][21]

The name "Chaos Monkey" is explained in the book Chaos Monkeys by Antonio Garcia Martinez:[22]

> Imagine a monkey entering a 'data center', these 'farms' of servers that host all the critical functions of our online activities. The monkey randomly rips cables, destroys devices and returns everything that passes by the hand [i.e. flings excrement]. The challenge for IT managers is to design the information system they are responsible for so that it can work despite these monkeys, which no one ever knows when they arrive and what they will destroy.

#### Simian Army

The Simian Army[21] is a suite of tools developed by Netflix to test the reliability, security, or resilience of its Amazon Web Services infrastructure and includes the following tools:[23]

* At the very top of the Simian Army hierarchy, Chaos Kong drops a full AWS "Region".[24] Though rare, loss of an entire region does happen and Chaos Kong simulates a systems response and recovery to this type of event.
* Chaos Gorilla drops a full Amazon "Availability Zone" (one or more entire data centers serving a geographical region).[25]

### Other

Voyages-sncf.com's 2017 "Day of Chaos"[26] gamified simulating pre-production failures[27] to present at the 2017 DevOps REX conference.[28] Founded in 2019, Steadybit popularized pre-production chaos and reliability engineering.[29] Its open-source Reliability Hub extends Steadybit.[30][31]

Proofdock can inject infrastructure, platform, and application failures on Microsoft Azure DevOps.[29] Gremlin is a "failure-as-a-service" platform.[32] Facebook's Project Storm simulates datacenter failures for natural disaster resistance.[33]

## See also

* Data redundancy
* Error detection and correction
* Fail-fast system
* Fail fast (business), a related subject in business management
* Fall back and forward
* Fault injection
* Fault tolerance
* Fault-tolerant computer system
* Grease (networking)
* Resilience (network)
* Robustness (computer science)
* Fuzzing

## Notes and references

1. ↑ "Principles of Chaos Engineering". https://principlesofchaos.org/.
2. ↑ Basiri, Ali; Behnam, Niosha; de Rooij, Ruud; Hochstein, Lorin; Kosewski, Luke; Reynolds, Justin; Rosenthal, Casey (2016). "Chaos Engineering". IEEE Software 33 (3): 35–41. doi:10.1109/ms.2016.60. Bibcode: 2016ISoft..33c..35B.
3. ↑ Heorhiadi, Victor; Rajagopalan, Shriram; Jamjoom, Hani; Reiter, Michael K.; Sekar, Vyas (2016). "Gremlin: Systematic Resilience Testing of Microservices". 2016 IEEE 36th International Conference on Distributed Computing Systems (ICDCS). pp. 57–66. doi:10.1109/icdcs.2016.11. ISBN 978-1-5090-1483-5.
4. ↑ Zhang, Long; Morin, Brice; Haller, Philipp; Baudry, Benoit; Monperrus, Martin (2021). "A Chaos Engineering System for Live Analysis and Falsification of Exception-Handling in the JVM". IEEE Transactions on Software Engineering 47 (11): 2534–2548. doi:10.1109/tse.2019.2954871. Bibcode: 2021ITSEn..47.2534Z.
5. ↑ Siwach, Gautam (29 November 2022). "Evaluating operational readiness using chaos engineering simulations on Kubernetes architecture in Big Data" (pdf). 2022 International Conference on Smart Applications, Communications and Networking (SmartNets). Botswana. pp. 1–7. https://ieeexplore.ieee.org/document/9993998. Retrieved 3 January 2023.
6. ↑ "Machine Learning Podcast Host and Technology Influencer: Gautam Siwach". LA Weekly. October 7, 2022. https://www.laweekly.com/machine-learning-podcast-host-and-technology-influencer-gautam-siwach/.
7. ↑ Hertzfeld, Andy. "Monkey Lives". https://www.folklore.org/StoryView.py?project=Macintosh&story=Monkey_Lives.txt.
8. ↑ "Game day". Amazon. 31 December 2020. https://wa.aws.amazon.com/wat.concept.gameday.en.html.
9. ↑ 9.0 9.1 Limoncelli, Tom (September 13, 2012). "Resilience Engineering: Learning to Embrace Failure". ACM Queue 10 (9). https://queue.acm.org/detail.cfm?id=2371297.
10. ↑ Krishnan, Kripa (September 16, 2012). "Weathering the Unexpected". ACM Queue 10 (9): 30–37. doi:10.1145/2367376.2371516. https://queue.acm.org/detail.cfm?id=2371516.
11. ↑ Krishnan, Kripa (8-13 November 2015). "10 Years of Crashing Google" (html). 2015 Usenix LISA. Washington DC. https://www.usenix.org/conference/lisa15/conference-program/presentation/krishnan. Retrieved 25 February 2024.
12. ↑ Beyer, Betsy; Jones, Chris (2016). Site Reliability Engineering (1st ed.). O'Reilly Media. ISBN 9781491929124. OCLC 1291707340.
13. ↑ "Chapter 5. Google DiRT: Disaster Recovery Testing". O'Reilly Media. 30 April 2020. https://learning.oreilly.com/library/view/chaos-engineering/9781492043850/ch05.html.
14. ↑ 14.0 14.1 Jones, Nora; Rosenthal, Casey (2020). Chaos Engineering (1st ed.). O'Reilly Media. ISBN 9781492043867. OCLC 1143015464.
15. ↑ Cahoon, Jason (June 2, 2021). "WATCH: The DiRT on Chaos Engineering at Google" (in en) (video). GOTO Conferences. https://youtube.com/v/6I8Ag9mCEpY.
16. ↑ 16.0 16.1 "The Netflix Simian Army". Medium. 19 July 2011. https://medium.com/netflix-techblog/the-netflix-simian-army-16e57fbab116.
17. ↑ Orzell, Gregory S. & Yury Izrailevsky, "Validating the resiliency of networked applications", US patent 20120072571, published 2012-03-22
18. ↑ "Netflix Chaos Monkey Upgraded". Medium. 2016-10-19. https://medium.com/netflix-techblog/netflix-chaos-monkey-upgraded-1d679429be5d.
19. ↑ "PhoenixServer". Martin Fowler (software engineer). 10 July 2012. https://martinfowler.com/bliki/PhoenixServer.html.
20. ↑ "Netflix libère Chaos Monkey dans la jungle Open Source" (in fr). Le Monde Informatique. https://www.lemondeinformatique.fr/actualites/lire-netflix-libere-chaos-monkey-dans-la-jungle-open-source-49940.html.
21. ↑ 21.0 21.1 "SimianArmy: Tools for your cloud operating in top form. Chaos Monkey is a resiliency tool that helps applications tolerate random instance failures". Netflix, Inc.. 20 October 2017. https://github.com/Netflix/SimianArmy.
22. ↑ "Mais qui sont ces singes du chaos ?" (in fr-FR). 25 July 2017. https://15marches.fr/business/singes-du-chaos.
23. ↑ SemiColonWeb (8 December 2015). "Infrastructure : quelles méthodes pour s'adapter aux nouvelles architectures Cloud ? - D2SI Blog" (in fr-FR). D2SI Blog. http://blog.d2-si.fr/2015/12/08/architecture-cloud/.
24. ↑ (in en) Chaos Engineering Upgraded, 19 April 2017, https://netflixtechblog.com/chaos-engineering-upgraded-878d341f15fa, retrieved 10 April 2020
25. ↑ (in en) The Netflix Simian Army, https://medium.com/netflix-techblog/the-netflix-simian-army-16e57fbab116, retrieved 12 December 2017
26. ↑ "Days of Chaos" (in fr-FR). http://days-of-chaos.com/.
27. ↑ "DevOps: feedback from Voyages-sncf.com" (in fr-FR). 17 March 2017. https://www.blogdumoderateur.com/devops-retour-experience-voyages-sncf-com/.
28. ↑ devops REX (3 October 2017). "[devops REX 2017 Days of Chaos : le développement de la culture devops chez Voyages-Sncf.com à l'aide de la gamification"]. https://fr.slideshare.net/devopsrex/days-of-chaos-le-dveloppement-de-la-culture-devops-chez-voyagessncfcom-laide-de-la-gamification-80396202.
29. ↑ 29.0 29.1 Miller, Ron (22 September 2022). "Steadybit wants developers involved in chaos engineering before production". https://techcrunch.com/2022/09/22/steadybit-wants-developers-involved-in-chaos-engineering-before-production/.
30. ↑ steadybit/reliability-hub-db, Steadybit, 2024-08-26, https://github.com/steadybit/reliability-hub-db, retrieved 2024-08-26
31. ↑ "Home" (in en). https://hub.steadybit.com/.
32. ↑ "Gremlin raises $18 million to expand 'failure-as-a-service' testing platform". 28 September 2018. https://venturebeat.com/business/gremlin-raises-18-million-to-expand-failure-as-a-service-testing-platform/.
33. ↑ Hof, Robert (11 September 2016). "Interview: How Facebook's Project Storm Heads Off Data Center Disasters" (in en). https://www.forbes.com/sites/roberthof/2016/09/11/interview-how-facebooks-project-storm-heads-off-data-center-disasters/.

## External links

* Principle of Chaos Engineering – The Chaos Engineering manifesto
* Chaos Engineering – Adrian Hornsby
* How Chaos Engineering Practices Will Help You Design Better Software – Mariano Calandra

Template:Netflix

0.00

(0 votes)

Original source: https://en.wikipedia.org/wiki/Chaos engineering. Read more

|

Retrieved from "https://handwiki.org/wiki/index.php?title=Chaos_engineering&oldid=4418339"