Error Handling & Fault-Tolerant Systems: A Mental Model for Backend Engineers
Your code will fail in production. The only question is how you handle your errors. That's the entire job of error handling.
This is part 12 of my backend-from-first-principles series. Every previous topic — HTTP, routing, databases, caching, queues, search — assumed the happy path. This one is about the unhappy path, which is the path your code will spend most of its real-world life on.
The shift is mental, not technical. Junior engineers ask "why did this break?" Senior engineers ask "when this breaks, will the system survive?" Once you internalize that the database will drop connections, third parties will rate-limit you, and users will send malformed JSON, your job changes. You stop trying to prevent failure. You start designing for it.
The Mental Model: The Pilot's Mindset
Aviation is the safest mode of transport in human history. Not because planes never fail — they fail constantly. Engines lose pressure. Hydraulics leak. Birds hit windshields. The reason aviation is safe is that the entire industry is designed around the assumption that something will go wrong.
Look at how it's structured:
- Pre-flight checks — every system is verified before takeoff, not mid-flight.
- Redundant systems — two engines, dual hydraulics, backup electronics. One goes down, the plane still flies.
- Checklists — for every conceivable failure mode, there's a documented procedure. Pilots don't improvise during emergencies.
- Black box — every action is recorded, so post-incident analysis can prevent the next one.
- Co-pilot — a second pair of eyes ready to take over instantly.
- Graceful degradation — a plane on one engine doesn't crash. It descends, declares emergency, and lands at the nearest airport.
That's exactly the architecture of a fault-tolerant backend:
Pre-flight checks → Config validation + deep health checks Redundant systems → Fallbacks, replicas, circuit breakers Checklists → Runbooks / playbooks for incidents Black box → Structured logs + observability Co-pilot → Global error handler catching what slipped through Graceful degradation → Disable non-essential features, keep the core alive
Carry this analogy through the rest of the article. Every concept maps to it.
The Five Failure Categories
You can't design for failures you haven't named. Backend errors fall into five distinct categories, and each one demands a different defensive strategy.
1. Logic Errors — The Silent Killers
The most dangerous failures are the ones that don't look like failures. The code runs, returns 200 OK, logs nothing alarming — and silently produces the wrong result.
The classic example: an e-commerce app applies a percentage discount, then applies a coupon code that also discounts. Two discounts stacked. Shipping cost goes negative. The order completes. Revenue leaks for weeks before anyone notices.
Logic errors aren't bugs in the technical sense — they're bugs in the intent. They come from misread requirements, edge cases nobody whiteboarded, or implementations that match the spec but not the reality. They don't throw stack traces; they corrupt your business slowly.
Defense: not try/catch. The defense against logic errors is testing (unit + integration), business-metric monitoring (sudden drops in conversion, average order value spikes), and code review by people who understand the domain. The system can't catch what it doesn't know is wrong.
2. Database Errors — The System Stoppers
Most backends are conversations with a database. When that conversation breaks, everything stops.
- Connection errors — your app can't reach the database. Network partition, DB restart, or — most commonly — your connection pool is exhausted. Every TCP handshake costs time, so we pool connections; when the pool runs out, requests queue and time out.
- Constraint violations — the database refuses a write because it would break its own rules. Unique constraint: creating a user with an email that already exists. Foreign key: inserting an order for a customer ID that doesn't exist.
- Query errors — malformed SQL, typos in table names (
SELECT * FROM custumers), missing columns. Caught at runtime if you're unlucky, at deploy time if you're using a typed query builder. - Deadlocks — two transactions each holding a lock the other one needs. They wait forever. The database eventually picks one as the loser and rolls it back.
Defense: connection pool tuning, retry on transient errors with backoff, treat constraint errors as expected business outcomes (translate "unique violation" into a clean 409 Conflict response, don't 500), and isolate transactions to minimize deadlock surface area.
3. External Service Errors — The Things You Don't Control
Modern backends are integrators. Stripe processes payments, Resend sends emails, S3 stores files, Auth0 handles login. Every external dependency is a failure point you cannot fix.
- Network failures — DNS resolution fails, the connection times out, a packet drops mid-request.
- Authentication failures — the API key got rotated, the OAuth token expired, the IAM permissions changed.
- Rate limiting (HTTP 429) — you exceeded your quota. The provider returns "Too Many Requests" and tells you to slow down.
- Service outages — the provider is just down. AWS us-east-1 has gone fully dark on multiple occasions. There is nothing you can do to bring it back.
Defense for 429s: exponential backoff with jitter. Don't retry immediately — wait 1s, then 2s, then 4s, then 8s, doubling each time. Add a random offset (jitter) so you don't synchronize retries with every other client and create a thundering herd against the provider. (Same pattern we covered in the Task Queues article — it shows up everywhere.)
Defense for outages: graceful fallback. If Redis is down, your cache layer should return cache-miss instead of throwing — the database takes the load, slower but alive. If your image CDN is down, serve a placeholder. If Stripe is throwing 503s, queue the payment for retry and tell the user "we're processing your order." The core feature stays alive even when a dependency dies.
4. Input Validation Errors — The Easy Ones
Users send bad data. Sometimes by accident (typo in their email), sometimes intentionally (a bot probing your API). This is the easiest error category to handle, and the one that should never reach your business logic.
Catch at the door. Reject with 400 Bad Request. We covered this in depth in Validation & Transformations (part 5) — the rule is simple: validate at the entry point, reject early, never trust input.
5. Configuration Errors — The Boot-Time Bombs
You deploy to production. The build succeeds. The container starts. The HTTP server binds to its port. Then a user calls the AI endpoint, and the server throws 500: OPENAI_API_KEY is undefined. The variable was missing the whole time — you just didn't notice until the first request that needed it.
This is the worst possible failure pattern: silent at deploy, loud at runtime.
The fix is one of the highest-leverage habits in backend engineering: validate config at boot, before the server accepts traffic. If a required env var is missing, intentionally crash the process before the HTTP listener starts.
// At the top of your entry file, before anything else
const required = ['DATABASE_URL', 'OPENAI_API_KEY', 'JWT_SECRET'];
for (const key of required) {
if (!process.env[key]) {
throw new Error(`Missing required env: ${key}`);
}
}
// Only now do we start the server
app.listen(3000);
Why is this so important? Because in a blue-green deployment, the new ("green") version must pass health checks before traffic shifts to it. If the green deploy fails to start due to a missing env var, the load balancer keeps routing to the healthy "blue" version. Users see nothing. You see a broken deployment in your dashboard. That's the right outcome.
The rule: it is always better for a deployment to fail to start than for the application to randomly throw 500s to real users at 3 AM.
Proactive Detection: Catch Failures Before They Cause Damage
Pilots don't wait for an engine to die before checking it. They run pre-flight inspections. Backend systems need the same instinct.
Health Checks: Shallow vs. Deep
The naive health check is a route that returns 200 OK if the HTTP server is alive:
GET /health → 200 OK
This tells you almost nothing. Your HTTP server can be perfectly responsive while the database is down, the cache is unreachable, and email delivery is broken. A green health check with a broken backend is worse than a red one — it lies.
A deep health check verifies actual functionality:
GET /health/deep
{
"http": "ok",
"database": "ok (avg query: 4ms)",
"redis": "ok (ping: 1ms)",
"email": "ok (test send succeeded)",
"auth": "ok (token verify succeeded)"
}
Run a representative SQL query. Ping Redis. Send a test email to a sink address. Verify a known JWT. If any of these fails, return 503 Service Unavailable — the load balancer pulls this instance out of rotation, and the on-call engineer gets paged before users notice.
Monitoring: Performance and Business Metrics
Error rates aren't the only signal. Often by the time error rates spike, the system is already in trouble. The early warning signs are quieter:
- Performance degradation — average response time creeps from 50ms to 200ms to 500ms. No errors yet, but the system is straining. Something will snap soon.
- Business metrics — successful checkouts drop by 30% in the last hour. No exceptions in the logs. But your conversion funnel just collapsed. Something is wrong — maybe a payment provider is silently rejecting cards, maybe a UI bug is hiding the buy button. Logical failures show up here before they show up as exceptions.
Structured JSON logging makes all of this searchable. Don't log "User 42 failed to checkout: card declined" as a plain string. Log:
{
"level": "warn",
"event": "checkout_failed",
"user_id": 42,
"reason": "card_declined",
"provider": "stripe",
"correlation_id": "req_a1b2c3"
}
Now Grafana, Loki, or Datadog can aggregate, filter, and graph that field across millions of events. You stop reading logs and start querying them.
Recoverable vs. Non-Recoverable: Knowing Which Is Which
The instant an error fires, the system has to make a decision. Most failures fall into one of two categories, and the response is completely different for each.
Recoverable Errors → Retry
The error is transient. The thing that failed will probably work if you ask again in a moment. Examples: network timeout, database connection dropped, third-party API returned 503.
The strategy: exponential backoff with jitter, capped at a maximum number of retries. Don't retry forever — at some point, give up and treat it as non-recoverable.
Non-Recoverable Errors → Contain and Degrade
The error won't fix itself. Examples: malformed payload, unique constraint violation, business rule violation, downstream service that's confirmed dead. Retrying does nothing except waste resources.
The strategy: graceful degradation. Disable the affected feature, switch to a fallback, return a clear error to the user. The plane lands on one engine — slower, less convenient, but everyone walks away.
Concretely:
- Recommendation engine is down? Show a "popular products" list from a static cache instead of a personalized one.
- Email service is down? Queue the email for later delivery, return success to the user.
- Payment processor is down? Disable checkout, show "payments temporarily unavailable", keep browsing alive.
The rule: never let a failure in a non-essential feature take down an essential one. Recommendations failing should not break product browsing. Analytics being down should not break checkout. Isolate dependencies so one bad node can't bring down the cluster.
Error Propagation: The Bubble-Up
Where should errors be handled — at the spot they happen, or somewhere higher?
The amateur instinct is to try/catch everywhere — wrap every database call, every external API call, every parse. The result: 200 lines of error handling for every 50 lines of business logic. Every layer makes its own decisions about HTTP status codes, response shapes, and logging. Nothing is consistent.
The professional pattern is the opposite: let errors bubble up. Throw at the source, catch at the boundary.
Repository: throws raw "no rows" or "constraint violation" errors
↑ (does not know HTTP exists)
Service: may add business context, otherwise re-throws
↑ (does not know HTTP exists)
Controller: may catch a known business error, otherwise lets it through
↑
[GLOBAL ERROR HANDLER] ← catches everything that escaped
↓
Translates to HTTP response, logs, sends to monitoring
This works because lower layers don't have the context to decide the right response. The repository doesn't know whether "no user found" should be a 404 (the user looked up a missing record) or a 500 (an internal lookup that should never miss). That decision belongs higher up, where the request's intent is known.
The Global Error Handler: The Final Safety Net
Every fault-tolerant backend has one piece of code that's worth more than any other: the global error handler. It's the co-pilot that takes over when something the captain didn't plan for happens.
Architecturally, it's a single middleware registered at the very end of the request pipeline. Every error thrown anywhere — controller, service, repository, third-party SDK — bubbles up to it. It inspects the error, decides what HTTP response is appropriate, logs it, and sends a clean response to the client.
Worked Example: A Books API
Picture a GoodReads-style app with the standard layered architecture: Router → Controller → Service → Repository → Database.
Three different errors, three different correct responses, all flowing through the same handler:
| Scenario | What the DB throws | What the user should see |
|---|---|---|
| GET /books/123 — book doesn't exist | NoRowsError | 404 Not Found |
| POST /books — title already exists | UniqueViolationError | 409 Conflict |
| POST /books — author_id doesn't exist | ForeignKeyError | 400 Bad Request |
| POST /books — title > 500 chars | ValidationError | 400 Bad Request |
| GET /books — Redis down, fallback fails | InternalError | 500 + generic message |
The handler is the only place this translation lives. The controller doesn't try/catch for unique violations. The service doesn't have if-error-is-foreign-key-then. The repository just throws and forgets.
// Express-style global error handler (registered LAST)
app.use((err, req, res, next) => {
if (err instanceof ValidationError) return res.status(400).json({ error: err.message });
if (err instanceof NoRowsError) return res.status(404).json({ error: "Not found" });
if (err instanceof UniqueViolationError)return res.status(409).json({ error: "Already exists" });
if (err instanceof ForeignKeyError) return res.status(400).json({ error: "Invalid reference" });
// Unknown error — log it, return generic message
logger.error({ err, correlation_id: req.id });
return res.status(500).json({ error: "Something went wrong" });
});
What this gives you:
- One source of truth for HTTP error responses across the entire codebase.
- Zero repetition — no
try/catchboilerplate in 200 controllers. - Forgiveness for forgotten edge cases — if a developer forgets to handle a specific error, the handler catches it and returns a sane response instead of a raw stack trace.
- Centralized logging — every error gets logged once, with full context (correlation ID, user ID, request path).
The litmus test: a new junior on the team should be able to write a handler that throws raw errors freely, knowing the global handler will translate them correctly. The architecture protects them from making the codebase worse.
Security: What Your Errors Reveal to Attackers
Error messages and logs are intelligence sources. The wrong message in the wrong place is a gift to anyone probing your system.
1. Don't Leak Internals
Never send raw database errors to the client. A response like relation "user_credentials" does not exist tells an attacker your table names, your schema layout, and that you're using PostgreSQL. From there, they craft targeted SQL injection attempts.
The global handler should catch every unhandled error and return a generic "Something went wrong" message to the user. The detailed error goes to your logs, where only your team can read it.
2. Prevent Account Enumeration
This is the classic mistake on login forms. The user types an email and a wrong password. What does your API return?
BAD: "No user with that email exists."
→ Attacker writes a script that tries 10,000 emails.
Now they know exactly which ones are real accounts.
GOOD: "Invalid email or password."
→ Attacker can't tell which part was wrong.
No information leaked.
Same rule applies to password reset, signup ("email already in use" can be a leak), and any endpoint where the response shape changes based on whether an account exists. Reference the OWASP Authentication Cheat Sheet for the full pattern.
3. Sanitize Logs (PII)
Logs are your debugging gold. They're also a regulatory liability waiting to happen. If your logging infrastructure (Loki, Datadog, CloudWatch) is breached, every plain-text password and credit card you logged is now in the wild.
Rules:
- Never log passwords. Not even on failure. Not even "for debugging".
- Never log raw API keys, JWTs, or secrets. If you must, log only the first 4 and last 4 characters.
- Mask PII fields. Email, phone, address, credit card — replace with redacted versions:
j***@example.com,****-****-****-1234. - Use stable identifiers instead. Log
user_id: 42, notemail: "john@example.com". The user ID lets you correlate without exposing identity. - Use correlation IDs. Generate a UUID per request, attach it to every log line in that request's lifecycle. Bug reports become "search for ID
req_a1b2c3" instead of grepping by username.
The Mental Model Summary
Carry this forward:
- The pilot's mindset = Don't ask "what if it fails?" Ask "when it fails, will the system survive?" Aviation is safe because it assumes failure; backends should too.
- Five failure categories = Logic (silent), Database (constraints + connections + deadlocks), External (rate limits + outages), Input (validate at the door), Configuration (fail at boot, not runtime).
- Logic errors = The dangerous ones. They don't crash; they corrupt. Defense is testing + business-metric monitoring, not
try/catch. - Fail-fast on config = Validate env vars before the server accepts traffic. A failed deploy is infinitely better than runtime 500s.
- Deep health checks = Verify the database, cache, and dependencies — not just that the HTTP server is responding. A green-but-broken health check lies.
- Recoverable vs. non-recoverable = Retry the transient (with exponential backoff + jitter), degrade gracefully on the permanent. Never let a non-essential feature take down an essential one.
- Bubble errors up = Throw at the source, catch at the boundary. Lower layers don't have the context to choose HTTP status codes.
- Global error handler = One middleware translates every error into the right response. Eliminates
try/catchboilerplate, guarantees consistency, catches what developers forget. - Security in errors = Never leak internals. Use generic auth failure messages to prevent enumeration. Sanitize logs of PII; use user IDs and correlation IDs instead.
- The mindset shift = "Why did this break?" → "I am prepared for the worst." Production lives on the unhappy path. Design for it.
Fault tolerance isn't a feature you bolt on at the end — it's the architectural posture you start from. Every layer assumes the layer below it might fail, and prepares accordingly. Done right, the user submits an order, three retries happen behind the scenes, a Redis fallback kicks in, an external API returns 503 and gets queued for retry, the email delivery is async — and the user sees a clean 201 Created in 80ms. That's the entire job. Make the chaos invisible.
This is part 12 of my "Backend from First Principles" series. Everything in this article comes from what I learned watching the Backend from First Principles playlist by Sriniously — specifically the video on error handling and building fault-tolerant systems. I'm not claiming original research. This is me taking what I studied, internalizing it, and presenting it as a mental model that made the concepts stick for me. If it helps you too, even better.
Follow along for the next one.