Cluster DR vs Regional HA: The Four Levels of Database Availability Every Platform Engineer Must Understand
"Our database is highly available" means four different things, and they protect against four different failures. A ladder for forcing that conversation to be specific — plus why the money most teams spend on availability goes to the failure modes they face least.
Cluster DR vs Regional HA: The Four Levels of Database Availability Every Platform Engineer Must Understand
"Our database is highly available."
Almost every production system makes this claim. And almost every engineering team means something slightly different when they say it. One team means: if the primary dies, a replica takes over in 30 seconds. Another means: if a data center burns down, we fail over to another region. A third means: we have read replicas, so we have HA.
These aren't the same thing. They protect against completely different failure modes. And confusing them is how you end up with a system that survives every failure you tested for, until the one you didn't test for takes everything down.
The problem is that "HA" and "DR" are marketing words as much as technical ones, and every vendor draws the line in a slightly different place. So here is a four-level ladder I use to force the conversation to be specific. It is my framing, not any vendor's taxonomy — if you go looking for these exact terms in a product's docs you won't find them. What you will find is that most managed database offerings sit at level 2, describe themselves as "highly available," and leave levels 3 and 4 as your problem.
The useful thing about the ladder isn't the labels. It's that each rung protects against a failure mode the rung below it cannot touch, and the failures that actually take teams down live at the top.
Level 1: Node HA
What it protects against: A single database server dying.
How it works: Multiple database instances form a consensus group (Raft, Paxos, or traditional primary/replica). When the primary fails, an election happens and a replica is promoted.
What it looks like in practice:
What it doesn't protect against: A bug in your database software that crashes all nodes simultaneously. A bad config push that affects all members of the cluster. A Raft split-brain where consensus is lost. An operator running DROP TABLE on the primary (it replicates to all replicas).
Node HA is the most commonly implemented level, and it's genuinely useful. But it's the level most teams stop at and mistake for "full HA."
Level 2: Cluster HA (Regional, Multi-AZ)
What it protects against: An availability zone going offline within a region.
How it works: Cluster members are spread across multiple AZs. If AZ-A loses power, the members in AZ-B and AZ-C still have quorum and can continue operating without AZ-A's members.
What it looks like in practice:
What it doesn't protect against: A regional cloud provider outage that takes all AZs in a region offline simultaneously. A software bug in the database version running across all nodes. A corrupted Raft log that all nodes replicate faithfully. An operator-level mistake that gets applied to the entire cluster.
Multi-AZ is what most managed database services offer, but check whether yours is on by default — the answers differ and the difference is expensive:
- MongoDB Atlas: on by default. Every cluster is a replica set of at least three nodes distributed across separate availability zones, where the region has three.
- Google Cloud Spanner: on by default. Any regional configuration maintains three read-write replicas in three zones.
- Amazon RDS: not on by default. Multi-AZ is an opt-in deployment option —
MultiAZdefaults to false onCreateDBInstance, and if you didn't tick the box you have a single-AZ database. (Aurora spans AZs inherently, which is a different architecture.)
That RDS default has surprised a lot of teams during their first AZ event. It's worth checking your own console rather than assuming.
Level 3: Regional HA (Active-Active or Active-Passive Cross-Region)
What it protects against: An entire cloud region becoming unavailable.
How it works: A second cluster in a separate geographic region receives replication from the primary region. On regional failure, traffic is rerouted to the secondary region and the secondary cluster is promoted.
What it looks like in practice:
What it doesn't protect against: the scenario that's actually most common.
Regional HA is expensive (you're paying for a full second cluster) and designed for a rare event (full regional outage). The problems you'll actually face are:
- A bad deployment corrupts cluster state
- An operator mistake that's harder to reverse than it looks
- A software bug that crashes the process on all nodes within minutes of a rolling deploy
- A security incident requiring you to isolate and rebuild the cluster
Regional HA doesn't help with any of these. The problem isn't where the cluster is running — it's how the cluster is operating.
Level 4: Cluster DR
What it protects against: Application-layer failures that affect the entire cluster regardless of geography.
How it works: A completely separate cluster — independent of the primary cluster's consensus group — receives replication and can be promoted if the primary cluster becomes operationally compromised. The key distinction from regional HA: this is a separate cluster, not just separate nodes within the same cluster.
The one-line version of the distinction, and the reason this rung exists: regional redundancy protects against where your database runs failing. Cluster-level redundancy protects against how it is operating going wrong. A second region running the same corrupted Raft log, the same broken config, and the same crashing binary is not a recovery plan — it's the same failure, twice, with better latency.
The failure modes an independent cluster addresses that regional redundancy misses:
Raft consensus corruption: If a bad operator command or software bug corrupts the Raft log on all nodes, regional HA won't save you — all nodes faithfully replicate the corruption. An independent consensus group may still have clean state from before the corruption. Note the "may": if the corruption arrives through the replication stream rather than through the consensus layer, an independent cluster inherits it too. This is why replication lag on a DR cluster is sometimes a feature.
Config push that breaks all nodes: A misconfigured TLS cert, a bad secrets policy, or an invalid config file that kills the process on startup — this affects every node in the cluster simultaneously. Your multi-AZ setup has three broken nodes instead of one. An independent cluster is one where the bad config hasn't propagated yet.
Rolling deploy of a broken version: You push a database software version with a critical bug. It rolls out to all nodes. They crash. Your multi-AZ setup just made the problem worse by distributing the broken version faster. An independent cluster on the last known good version gives you somewhere to fail over to.
Security isolation: If you need to isolate a compromised cluster for forensics, an independent cluster lets you continue operations on the clean replica while the primary is locked down for investigation.
A caution on shopping for this. Level 4 is a property of an architecture, not a checkbox you can buy — and vendors use "DR" for level 3 far more often than for level 4. When a managed service advertises "cross-region disaster recovery," that is usually a geographically separate copy fed by the same replication stream, which puts it at level 3 on this ladder: it will faithfully replicate your corruption. Ask two questions. Does the standby have an independent consensus group? And can it be pinned to a different software version than the primary? If the answer to either is no, you have redundancy against infrastructure loss, not against your own mistakes.
In practice, most teams reach level 4 not with a second live cluster but with tested, independently-stored snapshots and a rehearsed restore path — which is cheaper and covers most of the same failure modes, at the cost of a much worse RTO.
The Availability Matrix
Here's how the four levels map to failure modes:
| Failure Mode | Node HA | Multi-AZ | Regional HA | Cluster DR |
|---|---|---|---|---|
| Single node failure | ✓ | ✓ | ✓ | ✓ |
| AZ outage | ✗ | ✓ | ✓ | ✓ |
| Full regional outage | ✗ | ✗ | ✓ | ✓ |
| Software bug across all nodes | ✗ | ✗ | ✗ | ✓ |
| Operator-caused cluster corruption | ✗ | ✗ | ✗ | ✓ |
| Security isolation requirement | ✗ | ✗ | ✗ | ✓ |
The hard truth is in the shape of that table: the bottom three rows — software bugs, operator mistakes, and security incidents — are the failure modes you're most likely to face, and they're the ones only the rightmost column covers. Node failures and AZ outages are real but comparatively infrequent in modern cloud infrastructure. Software and operational failures happen on every team, at every scale, and no amount of geographic redundancy touches them.
Which produces the uncomfortable conclusion: the money most teams spend on availability is spent on the columns they need least.
Why Most Teams Stop at Level 2
Cost and complexity. Each additional level adds infrastructure cost (another cluster), operational cost (more failover procedures to maintain), and complexity (more things that can go wrong with the DR system itself).
The risk assessment is genuinely contextual:
- A startup's primary database: Node HA + multi-AZ is probably fine. A 30-minute regional outage is survivable. Level 4 isn't worth the cost — but tested backups are, and those are cheap.
- A secrets management or identity platform: level 4 makes sense. If your secrets store is down, every service that depends on dynamic credentials or encryption is down with it, and the blast radius dwarfs the cost of a second cluster. This is the category where vendors have actually built cross-region DR features, and for good reason.
- A financial core ledger or authentication system: level 4 is likely worth it. These are systems where a two-hour outage causes direct revenue loss and customer trust damage.
The pattern across all three: the level you need is set by blast radius, not by how much you like your database. A system that other systems cannot start without belongs a rung higher than its own traffic volume would suggest.
Designing Your Availability Strategy
When evaluating availability requirements for a system, work through the matrix explicitly:
Define your blast radius
If this system goes down, what else stops working? A database that other teams' services call has larger blast radius than an isolated data store.
Classify failure probability
Go and count, from your own incident history over the last two years, rather than reasoning from intuition. Most teams who do this are surprised by the shape: hardware and AZ events are rarer than they assumed, and self-inflicted incidents — bad deploys, bad migrations, bad config — are the clear majority. Your numbers are the only ones that should drive this decision, and you already have them.
Calculate the cost of downtime
For each failure mode, what's the revenue and trust cost of a 1-hour outage? A 4-hour outage? This number justifies the infrastructure spend at each level.
Test your failover paths
An untested failover path is not a failover path. Cluster DR only works if you've actually tested promotion, verified replication lag is acceptable, and documented the runbook. Build regular DR drills into your operational calendar.
The "our database is highly available" claim is only meaningful when you can answer: available against what failure modes, within what RTO/RPO? The four levels give you the vocabulary to answer that question precisely — and to know which level you actually need.
Comments (0)
No comments yet. Be the first to share your thoughts!