Is your redundancy real? A data-centre fire says check

What happened

In 2023, three major cloud providers had outages within a few months. The Pragmatic Engineer newsletter compared them.

• Google Cloud, April 25. A water leak led to a fire in a battery room at a Paris data centre. The europe-west9 region was down for roughly 14 hours. The region-wide impact traced to a regional database, Spanner, with two of its three replicas in the building that lost power, so it lost quorum.
• AWS, June 13. An overloaded capacity management subsystem for Lambda degraded about 104 services in us-east-1 for around three hours. Engineers the author spoke to said a load test caused the overload.
• Azure, July 5. A storm in the Netherlands brought down a tree that cut a fibre path in West Europe. That removed 25% of the network links between two data centres, and because the rest was already running close to its target utilization, packets were dropped. Some customers were affected for around eight hours.

So what for your business

Look at what each one had in common. In every case, something designed to be redundant turned out to share a weakness. Two zones in one building. Replicas that all depended on the same power. Network links with no headroom left when one was cut.

You may not be running a cloud, but you likely have the same pattern in your own setup: two servers on the same host, a primary and a replica in the same availability zone, backups stored in the same account as the data they protect.

Redundancy that was never tested is a belief, not a control.

What to do this month

1. Draw your failure domains. For each critical service, list what it shares: zone, account, provider, power, DNS, a single person.
2. Check replicas and backups. Are they in a different zone, a different account and, for the data that matters most, a different provider?
3. Look at your headroom. If you lost a quarter of your capacity right now, would the rest cope?
4. Run a game day. Turn off one piece in a safe window and see what happens.
5. Read the provider's postmortems. They show how a platform fails in practice.