# Chaos engineering

> Breaking your own system on purpose, carefully, to find weak spots before outages do.

- Level: Advanced
- Topics: Cloud, Software
- URL: https://tidbit.legios.com.ar/chaos-engineering/
- In Spanish: [Ingeniería del caos](https://tidbit.legios.com.ar/es/chaos-engineering/)

**Chaos engineering** is the practice of running controlled experiments that inject failures into a system, such as killing a server, adding network delay, filling a disk or making a dependency return errors, to see whether it keeps working. The idea is that in large distributed systems, failures will happen anyway; better to discover how your system reacts on a Tuesday afternoon with the team watching than at 3 a.m. during a real outage.

An experiment follows a scientific shape. First you define the **steady state**: a measurable sign that things are normal, like orders per minute or error rate. Then you form a hypothesis ("if one database replica dies, checkout keeps working"), inject the fault, and look for a difference. Good practice keeps the **blast radius** small, starting with a fraction of traffic or a test environment, and always has a stop button that rolls everything back.

Netflix popularized the idea with Chaos Monkey, a tool that randomly shut down its production servers so engineers had to build services that survive it. Today there are open source tools like Chaos Mesh and LitmusChaos for Kubernetes, and managed ones like **AWS Fault Injection Service**. Chaos engineering only makes sense once you have solid observability; otherwise you break things and cannot tell what happened.

## An analogy

A fire drill, but with a real small fire in a controlled corner of the building, so you learn whether the sprinklers, the alarms and the exits actually work before the day you need them.

## Keep in mind

- Controlled experiments that inject failures to test resilience.
- Define a steady state, form a hypothesis, inject a fault, compare.
- Keep the blast radius small and always have a way to stop.
- Requires good observability to learn anything from the experiment.

## Go deeper

- [Principles of Chaos Engineering](https://principlesofchaos.org/)
- [Chaos engineering on Wikipedia](https://en.wikipedia.org/wiki/Chaos_engineering)
- [What is AWS Fault Injection Service? (official docs)](https://docs.aws.amazon.com/fis/latest/userguide/what-is.html)

## Related terms

- [Server](https://tidbit.legios.com.ar/server/): A computer whose job is to answer requests from other computers, all day, every day.
- [Database](https://tidbit.legios.com.ar/database/): An organized store of data that programs can search and update safely.
- [Netflix](https://tidbit.legios.com.ar/netflix/): A streaming service, and the engineering it built to deliver video to the whole world.
- [Open source](https://tidbit.legios.com.ar/open-source/): Software whose code anyone can read, use, change and share.
- [Kubernetes](https://tidbit.legios.com.ar/kubernetes/): The system that runs and babysits containers across many machines.
- [Observability](https://tidbit.legios.com.ar/observability/): Being able to tell what a running system is doing, and why, from what it emits.