Logging, Monitoring & Observability: A Mental Model for Backend Engineers
Monitoring tells you something is broken. Observability tells you what. The whole topic is about understanding the inside of a system from the outside — because in production, you don't get to do anything else.
This is part 14 of my backend-from-first-principles series. Modern backends don't run on one machine — they're spread across services, regions, and clouds. You can't ssh into "the server" anymore, because there is no the server. The only way to understand what your system is doing is to ask it to tell you.
The Setup: It's 3 AM
You're on call. A page comes in: error rate just hit 80%. You open Slack, then your dashboard. A red line is climbing. You have no idea which service is at fault, which region is affected, or what kind of error is firing.
You can't read the logs of every container by hand — there are hundreds. You can't spot-check a database — there are six. You have a few minutes before customers start tweeting. Where do you look?
Everything in this article is the answer to that question.
The Mental Model: The ER Doctor
An unconscious patient is wheeled into the ER. The doctor has 60 seconds to figure out what's wrong with a body they cannot open. They use exactly three tools, and each one answers a different question:
🩺 Vitals monitor → Is something wrong? (METRICS) 📋 Patient's chart → What happened? (LOGS) 🧬 CT scan + contrast → Where exactly is it? (TRACES)
The vitals monitor beeps when blood pressure drops below a threshold. That's the alert. The chart tells the doctor what events the patient has been through — meds, prior conditions, what the paramedics saw. The CT scan injects a traceable dye and follows it through the body, lighting up the exact location of the blockage.
None of these alone is enough. Vitals tell you something is wrong but not what. The chart tells you what happened but not where in the body. The scan pinpoints the location but tells you nothing about when or why. Together, they save the patient.
That's exactly the structure of observability: metrics + logs + traces. Three different signals, each answering a different question, useless without the others.
Logging: What Happened
Logging is the patient's chart — a journal of every event your application thought was worth remembering. A user signed up. A payment was processed. A database query failed. A request came in from an unfamiliar IP.
Good logs are not just text. They carry metadata: the user ID, the request latency, the route, the function that ran. That metadata is what makes a log queryable instead of merely readable.
The Five Log Levels
debug— Granular trace of internal state. Local development only; never enable in production, the noise will drown everything else.info— Successful, expected events. "User created a new to-do item." The default heartbeat of a healthy system.warn— Something unexpected, but recoverable. Wrong password attempt, deprecated API call, automatic retry succeeded on the second try.error— An operation actually failed. A database query returned an error, validation rejected a request, an external API call timed out.fatal— The process is going down. Unrecoverable startup failure, the kind of crash where the next line of code will never run.
Structured vs. Unstructured
In local development you want logs to look pretty — colored, readable, scannable by a human eye:
[INFO] user_signup id=42 email=jane@acme.com latency=120ms
In production, throw all of that away. Use structured JSON:
{
"level": "info",
"event": "user_signup",
"user_id": 42,
"email": "jane@acme.com",
"latency_ms": 120,
"request_id": "req_a1b2c3"
}
The reason is brutal and simple: humans don't read production logs. Machines do. Datadog, Loki, CloudWatch — they all parse JSON instantly and let you query across millions of events ("show me every error from user 42 in the last hour"). Pretty colored text is unreadable to them.
Monitoring: Is Something Wrong?
Monitoring is the vitals monitor. It continuously samples a small set of metrics — concrete numbers like CPU usage, memory pressure, open DB connections, requests per second, error rate. Most monitoring systems sample every 10–15 seconds; faster than that wastes resources, slower than that misses incidents.
The goal isn't to look at metrics manually. The goal is to set alerts — pre-configured rules that fire automatically when a metric crosses a threshold. "If error rate > 5% for 2 minutes, page on-call." The system watches itself; the engineer only gets pulled in when the system can't fix itself.
The Limitation Nobody Tells You
Monitoring is reactive by design. It tells you something is wrong — error rate spiked, latency doubled, memory is climbing — but it cannot tell you what is wrong or why. The vitals monitor beeps "BP is critical" — it does not say "internal bleeding in the spleen at coordinates X, Y."
For that, you need the next pillar.
Observability: Why Is It Wrong?
A system is "observable" if you can determine its internal state by looking only at its external outputs. Observability is the discipline of making sure your system can be questioned — and answering, in detail, when you ask.
It's built on three pillars: logs, metrics, and a third one that ties everything together — traces.
What a Trace Actually Is
A trace follows a single request — one user click, one API call — through the entire backend, recording every step it touches and how long each step took:
Trace: req_a1b2c3 total: 312ms
[ Load Balancer ] 4ms
└─ [ Auth Middleware ] 18ms
└─ [ Bookings Service ] 280ms
├─ [ Validation ] 6ms
├─ [ DB: SELECT ] 42ms
└─ [ DB: INSERT ] ❌ 230ms ← failed here
That tree is the CT scan. It tells you the exact function, in the exact service, on the exact database call where the request died — and how long every other step took along the way. No grepping. No guessing. The path lights up like contrast dye in a vein.
Traces are what turn "we have a 500 error somewhere" into "the bookings service crashes on INSERT when the user has more than 10 active reservations." Logs and metrics on their own can't give you that — you need the path.
The 3 AM Workflow
This is what the three pillars look like when they're wired together. A real incident, end to end:
- The Alert (Monitoring). Error rate crosses 5%. Datadog pages Slack. You wake up.
- The Dashboard (Metrics). You open the dashboard. Errors started spiking 30 minutes ago, isolated to one service:
bookings. - The Lookup (Logs). The metric panel links straight to the matching log stream. You see hundreds of
500 Internal Server Errorentries, all from the same endpoint. - The Diagnosis (Traces). You click one log line. It opens the trace. The request passed validation, hit the database, and crashed on
INSERT INTO bookingswith a constraint violation. You read the payload. The bug is obvious.
Four clicks, ninety seconds, root cause identified. Not because the on-call engineer is brilliant — because the system was built to be questioned.
The litmus test of a well-instrumented system: when an incident fires, you should never have to guess. You should be able to follow a trail.
The Tooling Map
Two camps, and most teams pick one.
The open-source stack — pick a specialized tool per pillar and wire them together yourself:
- Logs → Loki, or the ELK stack (Elasticsearch + Logstash + Kibana)
- Metrics → Prometheus for collection, Grafana for dashboards
- Traces → Jaeger, Tempo, or Zipkin
The all-in-one platforms — one product handles all three pillars in a single dashboard:
- Datadog, New Relic, Grafana Cloud, Honeycomb
Open-source gives you total control and almost-zero licensing cost — at the price of running, scaling, and on-calling for the observability stack itself. Managed platforms give you one dashboard and one bill, at the price of vendor lock-in and per-GB ingest fees that grow with your traffic.
Smaller teams almost always start with a managed platform — observability that you have to babysit during an outage is observability you don't have. Larger teams with dedicated SRE eventually move to open-source for cost.
Instrumentation: How the Data Gets There
None of this works unless your code emits the signals. The act of adding logging, metrics, and trace calls to your code is called instrumentation.
The modern standard is OpenTelemetry (OTel) — an open-source spec with SDKs for every major language (Node, Python, Go, Java, Rust). Instrument once with OTel, and you can ship the data to Datadog today and Grafana Cloud tomorrow without rewriting a line.
The pattern in practice:
- A request enters the server. A piece of middleware creates a trace context object — a unique ID plus initial metadata (IP, user agent, route).
- That context is passed through every layer via the request's context object (
req.ctx, Go'scontext.Context, Python'scontextvars). - Each downstream function — service, repository, external API call — pulls the context, adds its own span (function name, duration, success/failure, custom attributes), and passes it further down.
- When the request ends, the full trace tree is shipped to your collector.
The result: every log line, metric, and trace span from that request shares one ID. In the dashboard, "show me everything that happened during request req_a1b2c3" becomes a single query.
The Mental Model Summary
Carry this forward:
- The ER doctor = vitals (metrics) + chart (logs) + CT scan (traces). Each one answers a different question; you need all three.
- Logs = the chart. What happened. Use levels to filter severity, structured JSON in production so machines can read them.
- Monitoring = the vitals monitor. Is something wrong? Sampled metrics + alerts on thresholds. Reactive by design.
- Observability = the discipline of designing a system that can be questioned. Three pillars: logs, metrics, traces.
- Traces = the missing piece. They follow a single request through every layer of the system, lighting up the exact place it failed.
- The workflow = alert → dashboard → logs → trace. Four clicks from "page" to "root cause" in a well-instrumented system.
- The tooling = open-source (Prometheus + Grafana + Loki + Jaeger) or managed (Datadog, New Relic, Honeycomb). Pick based on team size and budget; both work.
- Instrumentation = the act of making your code emit signals. OpenTelemetry is the modern standard. Trace context is passed through every layer so all signals share one ID.
The whole topic comes down to one shift: stop treating production as a black box. A well-instrumented backend doesn't just run — it narrates itself. When something breaks, you don't dig. You read.
This is part 14 of my "Backend from First Principles" series. Everything in this article comes from what I learned watching the Backend from First Principles playlist by Sriniously — specifically the video on logging, monitoring, and observability. I'm not claiming original research. This is me taking what I studied, internalizing it, and presenting it as a mental model that made the concepts stick for me. If it helps you too, even better.
Follow along for the next one.