What Are Database Failover Mechanisms and How Do They Work?
Learn how database failover works, from health checks to promotion and fencing, and how to avoid split-brain during a failure.
Expected Interview Answer
Database failover is the mechanism by which a standby or replica database automatically (or manually) takes over as the primary when the original primary becomes unreachable, so writes and reads can resume with minimal downtime.
A failover system relies on continuous health checks against the primary, an agreed detection threshold to avoid reacting to a brief network blip, and a promotion procedure that elevates a chosen replica to primary status. Once promoted, the new primary must accept writes, other replicas must re-point to it, and clients must be redirected via DNS, a virtual IP, a proxy, or a service-discovery layer. The hardest part is guaranteeing the old primary cannot keep accepting writes after a new primary is chosen, since two writable nodes at once corrupts data.
- Reduces downtime after a primary failure
- Keeps the system available without manual intervention
- Protects against single points of failure
- Supports business continuity and SLA targets
AI Mentor Explanation
Think of a team's designated vice-captain who has trained on every tactical call the captain makes, so if the captain is suddenly injured mid-over, the vice-captain steps in immediately instead of the team standing idle waiting for a decision. The umpires and scorers must all be told who is now captain before play resumes, exactly like clients being redirected to the new primary. If the injured captain somehow kept shouting instructions from the boundary, confusion and wrong calls would follow, which is why failover systems must firmly retire the old primary.
Step-by-Step Explanation
Step 1
Continuously monitor the primary
A health-check agent or orchestrator polls the primary for liveness, replication status, and responsiveness.
Step 2
Detect failure past a threshold
After a configured number of missed checks (to avoid false positives from transient blips), the primary is declared down.
Step 3
Promote a replica
The most up-to-date replica is elected and promoted to accept writes as the new primary.
Step 4
Redirect clients and fence the old primary
DNS, a virtual IP, or a proxy reroutes traffic to the new primary, while the old primary is fenced off (STONITH) to prevent split-brain writes.
What Interviewer Expects
- Understanding of health checks and detection thresholds
- Knowledge of promotion and client redirection mechanics
- Awareness of the split-brain risk and fencing (STONITH)
- Distinction between automatic and manual failover triggers
Common Mistakes
- Assuming failover is instantaneous with zero data loss
- Forgetting to mention fencing the old primary
- Not distinguishing detection time from promotion time
- Ignoring how clients discover the new primary
Best Answer (HR Friendly)
“Database failover is the process where, if the main database goes down, a backup database automatically takes over so the application keeps working. It involves detecting the failure, promoting a replica to be the new primary, and making sure everyone points to that new primary while safely shutting off the old one to avoid conflicting writes.”
Code Example
-- On the standby server, promote it to become the new primary
-- (run as the postgres OS user, or via pg_ctl)
SELECT pg_promote();
-- Confirm the server is no longer in recovery mode (i.e. is now primary)
SELECT pg_is_in_recovery();
-- returns false once promotion completes
-- Application connection strings then point at the new primary's
-- host/IP, typically via a proxy or updated DNS record.Follow-up Questions
- What is the difference between hot, warm, and cold standby?
- How does STONITH (Shoot The Other Node In The Head) prevent split-brain?
- What is RTO and RPO, and how do they relate to failover design?
- How do connection poolers or proxies handle failover transparently?
MCQ Practice
1. What is the primary purpose of fencing the old primary during failover?
Fencing ensures the old primary cannot accept writes after a new primary is promoted, preventing two nodes from diverging.
2. Why do failover systems use a detection threshold instead of failing over on a single missed health check?
A threshold avoids triggering costly failovers due to transient network issues rather than a genuine primary failure.
3. Which mechanism is commonly used to redirect application traffic to a newly promoted primary?
Clients typically discover the new primary through DNS changes, a floating virtual IP, or a proxy/service-discovery layer.
Flash Cards
What is database failover? — The process of a standby database automatically taking over as primary when the original primary fails.
What is fencing (STONITH)? — Forcibly isolating the old primary so it cannot accept writes after a new primary is promoted.
Why use a detection threshold? — To avoid triggering failover on brief, non-fatal network blips.
How do clients find the new primary? — Via DNS updates, a virtual IP, or a proxy/service-discovery layer.