Caching: A Mental Model for Backend Engineers
The fastest database query is the one you never make. That's caching in one sentence.
This is the ninth topic in my backend-from-first-principles series. Caching sounds simple — store things for later, serve them faster. But in production, it introduces some of the hardest problems in software engineering. The trick is understanding not just how to cache, but when, where, and what happens when the cache lies.
The Mental Model: A Library and Your Desk
Imagine a massive university library. Every book ever published is in there, perfectly catalogued across three floors of shelves. That's your database — vast, reliable, the single source of truth.
But every time you need to look up a definition, you have to stand up, walk to the third floor, find the right shelf, pull the book, flip to the page, note it down, and walk back. That trip takes 10 minutes.
So you start keeping the 5 books you reference most often stacked on your desk.
That stack is your cache. A small, fast copy of the data you use frequently, placed right where you need it. The library is still the source of truth — your desk is just a shortcut.
Two things happen when you reach for your desk:
- Cache hit — The book is on your desk. Flip it open. Done in seconds.
- Cache miss — It's not there. Walk to the library, grab it, bring it back to your desk for next time, then use it.
Now notice how this simple analogy naturally reveals every hard problem in caching:
- Your desk is small — you can only keep 5-6 books. When it's full and you need a new one, you have to remove one. Which one? The one you haven't touched in weeks (LRU). That's eviction.
- The library might release a new edition of a book while you're still using the old one on your desk. Your desk copy is now stale. That's the cache invalidation problem.
- You only bring books to your desk when you actually need them — you don't pre-copy the entire library. That's lazy loading.
Every caching system in existence — from CPU registers to CDNs to Redis — follows this exact pattern. The complexity is always in keeping the desk and the library in sync.
The Three Layers of Caching
Caching happens at multiple levels in a system. As a backend engineer, you'll interact with all three:
1. Network-Level: CDNs and DNS
CDN (Content Delivery Network) — Your origin server is in Virginia. A user in Tokyo requests an image. Without a CDN, the request crosses the Pacific Ocean. With a CDN, the image is cached at an edge server in Tokyo — the data travels 5 miles instead of 5,000.
Flow: User requests resource → DNS routes to nearest edge server → Cache hit? Serve instantly. Cache miss? Fetch from origin, cache at edge, serve. The TTL (Time to Live) defines how long the edge keeps the copy before checking the origin again.
DNS Caching — Even resolving a domain name (api.example.com → IP address) is cached at multiple levels: browser, operating system, and ISP resolver. Each layer avoids a full recursive lookup through root servers, TLD servers, and authoritative nameservers.
2. Hardware-Level: The CPU Memory Hierarchy
This is the original cache. Your CPU has a hierarchy of storage, each layer trading capacity for speed:
L1 Cache → ~1 ns | tiny | per-core L2 Cache → ~4 ns | small | per-core L3 Cache → ~10 ns | medium | shared RAM → ~100 ns | large | volatile Disk/SSD → ~10,000 ns| massive | persistent
You don't control this directly, but understanding it explains why in-memory caches like Redis are fast: they operate at the RAM level (~100ns) instead of the disk level (~10,000ns). That's a 100x speed difference.
3. Software-Level: Redis and Application Caching
This is where backend engineers spend most of their caching time. Tools like Redis or Memcached provide an in-memory key-value store that sits between your application and your database.
Client → Server → Check Redis (1ms) → Cache hit? Return instantly.
→ Cache miss? Query Postgres (50ms) → Store in Redis → Return.
Redis stores data in RAM: no disk seeks, no query parsing, no index traversal. Just a key lookup. This is why cached responses return in 1-2ms while database queries take 10-100ms+.
When to Cache (and When Not To)
Caching is not free. It introduces complexity, staleness risk, and memory costs. Use it when all three conditions are met:
- Expensive to compute or fetch. A complex SQL query joining 5 tables. An external API call with 200ms latency. A heavy algorithm computing trending topics.
- Accessed frequently. If only 3 users per day hit this endpoint, caching wastes memory. If 10,000 users per second hit it, caching saves your database.
- Doesn't change constantly. A user's profile changes rarely — cache it. A real-time stock price changes every millisecond — caching it means serving stale data.
The mental shortcut: read-heavy, write-light data is a caching goldmine. Write-heavy, read-light data is a caching nightmare.
Caching Strategies: How Data Flows
There are two fundamental patterns for how your application interacts with the cache. Each makes a different tradeoff between consistency and complexity.
Cache-Aside (Lazy Loading)
The application manages the cache directly. The cache doesn't know about the database; the database doesn't know about the cache.
Read: 1. Check cache → hit? Return. 2. Miss → query database 3. Store result in cache (with TTL) 4. Return to client Write: 1. Write to database 2. Invalidate (delete) the cached entry → Next read will trigger a fresh cache fill
Why it's popular: Simple. Your application has full control. Only data that's actually requested gets cached (no wasted memory on unused data).
The risk: Between the time data changes in the database and the cache entry expires or is invalidated, the cache serves stale data. For most applications, a few seconds of staleness is acceptable. For banking transactions, it's not.
Write-Through
Every write goes to the database AND the cache simultaneously.
Write: 1. Write to database 2. Write to cache → Cache is always fresh Read: 1. Check cache → always a hit (for written data)
Why it exists: Cache is always consistent with the database. No staleness window.
The cost: Every write is now two operations instead of one. Higher write latency. And you're caching data that might never be read — wasting memory.
The choice: Cache-Aside for most applications. Write-Through when consistency is worth the write overhead.
Cache Invalidation: The Hard Part
There's a famous quote in computer science: "There are only two hard things: cache invalidation and naming things."
The problem: your database has the truth. Your cache has a copy. The moment the truth changes and the copy doesn't, your application is lying to users.
Three strategies to keep them in sync:
- TTL-based expiry — Every cached entry has a timer. After 60 seconds (or 5 minutes, or 1 hour), the entry expires automatically and the next read fetches fresh data. Simple, but you accept staleness up to the TTL duration.
- Explicit invalidation — When your application writes to the database, it immediately deletes the corresponding cache key. The next read triggers a fresh cache fill. Precise, but you must remember to invalidate everywhere you write.
- Event-driven invalidation — The database publishes change events (via Change Data Capture), and a listener automatically invalidates affected cache entries. Most sophisticated, eliminates manual tracking, but requires infrastructure.
The production pattern: Most teams use TTL as the safety net + explicit invalidation on writes. TTL catches any invalidation you forgot. Explicit invalidation handles the common cases immediately.
Cache Eviction: When Memory Runs Out
RAM is finite. Your cache will eventually fill up. When a new entry needs to be stored and there's no room, the cache must decide which existing entry to remove. This is eviction.
- LRU (Least Recently Used) — Evict the entry that hasn't been accessed in the longest time. The most common policy. The logic: if nobody has asked for it recently, it's probably safe to remove.
- LFU (Least Frequently Used) — Evict the entry with the lowest total access count. Good when some data is consistently popular (trending topics) and shouldn't be evicted just because it wasn't accessed in the last 10 seconds.
- TTL-based — Entries self-destruct after a fixed time. Not strictly eviction, but it naturally frees memory.
- No eviction — Cache rejects new writes when full. Useful when you'd rather fail than serve stale data.
Redis defaults to no-eviction but supports all policies via configuration. For most backend applications, LRU with TTL is the right combination.
The Thundering Herd: Caching's Worst Day
This is the production scenario that catches teams off guard.
Imagine 40,000 users are hitting your cached trending topics endpoint. The TTL expires. All 40,000 requests simultaneously discover a cache miss. All 40,000 hit your database at the same instant. Your database, designed for 800 queries/second, receives 40,000. It collapses.
This is called a cache stampede (or thundering herd).
The fix: singleflight / mutex locking. When multiple requests discover a cache miss concurrently, only the first one actually queries the database. The rest wait for that single query to complete, then all share the result. One database hit instead of 40,000.
This is one of those problems you never think about until it takes down production at 2 AM.
Practical Use Cases for Backend Engineers
Here's where caching shows up in everyday backend work:
- Database query caching — Cache the result of expensive joins or aggregations. Key:
users:123:profile. Value: the JSON result. TTL: 5 minutes. - Session storage — Store authenticated session data in Redis instead of hitting the database on every request. We covered this in the auth article — sessions in Redis are caching in action.
- External API responses — Calling a weather API that allows 100 requests/hour? Cache the response for 10 minutes. 1,000 users get weather data from one API call instead of 1,000.
- Rate limiting — Track request counts per IP in Redis:
INCR rate:192.168.1.1with a 60-second TTL. If the count exceeds 50, return429 Too Many Requests. The counter auto-expires, resetting the window. - Computed results — Trending topics, recommendation scores, leaderboard rankings. Computed once, served to millions, refreshed periodically.
Cache Key Design: The Detail That Matters
A poorly designed cache key causes subtle, infuriating bugs. The key must encode every dimension that makes the response unique.
BAD: cache key = "users"
→ Returns the same cached list for every user, every page, every filter.
GOOD: cache key = "users:org:5:page:2:limit:10:sort:name:filter:active"
→ Each unique query gets its own cached result.
If two different queries map to the same cache key, one user gets another user's data. Include all variable dimensions: user/org ID, pagination, sorting, filters.
The Mental Model Summary
Carry this forward:
- Cache = A fast shortcut. Small copy of data stored closer to where it's needed. Kitchen counter, not the freezer.
- Cache hit = Data found in cache. Fast path (~1ms).
- Cache miss = Not in cache. Slow path (database → store in cache → return).
- Three layers = Network (CDN, DNS), Hardware (CPU L1/L2/L3, RAM, Disk), Software (Redis, Memcached).
- When to cache = Expensive + frequent + doesn't change constantly. Read-heavy, write-light data.
- Cache-Aside = App manages cache. Check → miss → fetch → store. Simple, lazy, most common.
- Write-Through = Write to DB and cache simultaneously. Always consistent, higher write cost.
- Invalidation = The hard problem. TTL as safety net + explicit deletion on writes.
- Eviction = LRU for most use cases. When memory is full, remove what's least recently used.
- Thundering herd = TTL expires, thousands of requests stampede the database. Fix: singleflight (one fetch, shared result).
- Key design = Encode every query dimension. Bad keys = data leaks between users.
Caching is the single highest-leverage performance optimization in backend engineering. One Redis key can save millions of database queries. But it comes with a fundamental tradeoff: speed for truthfulness. Every cache is a bet that the data hasn't changed since you stored it. Managing that bet — through TTLs, invalidation, and careful key design — is what separates a system that's fast from one that's fast and correct.
This is part 9 of my "Backend from First Principles" series. Everything in this article comes from what I learned watching the Backend from First Principles playlist by Sriniously. I'm not claiming original research. This is me taking what I studied, internalizing it, and presenting it as a mental model that made the concepts stick for me. If it helps you too, even better.
Follow along for the next one.