Skip to content

Mitigating Single Points of Failure

What could go wrong usually does.

Murphy still wins: anything that can go wrong will. Availability fails across people, process, and technology - not only servers.

Redundancy is how you buy time. How much redundancy depends on what the outage costs.

Single point of failure

One component. It fails, you feel it.

Single Point of Failure
One failure, whole service down.
  • People - the only person who can approve a release is on leave
  • Process - one brittle checklist nobody else knows
  • Technology - one server, one CDN, one DNS zone

High availability

A second identical path that can take over immediately - often automatically.

High Availability (HA)
Two paths live; one can fail without taking the service down.

Two guards at the door. Two servers behind a load balancer. Active-active (both working) costs more and is harder. Active-passive (hot standby) is cheaper and may lose a little state on failover.

Disaster recovery

Not always-on redundancy. A rehearsed plan you invoke when the primary site or system is gone.

Disaster Recovery (DR)
Primary fails; DR path starts - often with some downtime.

On-call cover when someone is sick. Traffic police when the lights die. Bring up the secondary region when the primary datacentre is dark. Expect longer recovery and possible data loss versus HA.

Business continuity

Everything else failed. You keep a degraded business running anyway.

Business Continuity Planning (BCP)
Ugly mode: cash only, mobiles instead of phones, self-guided tours.

BCP is for earthquakes, long power loss, pandemics - not for "the CDN had a bad afternoon." I wrote about that gap after Azure Front Door in Cloud Reliability.

Pick controls that pay

Start from assets and inherent risk. Then:

Control value = risk reduction / control cost

A $100k build plus $50k/year ops to avoid a $200k loss that might hit once a decade is bad maths. Match HA vs DR vs BCP to the actual blast radius - not to the vendor's brochure.

HA: automatic failover, less data loss, more cost. DR: cheaper, manual steps, more downtime. BCP: last resort when the service is allowed to look ugly.

Olivier Reuland