Backend Scaling & Performance Engineering: A Mental Model for Backend Engineers
Performance engineering is not "make it fast." It is "find exactly where time is being spent, then remove the right bottleneck with the least complexity possible."
This is part 17 of my backend-from-first-principles series. Scaling sounds glamorous — load balancers, Redis, replicas, serverless, microservices. But the senior move is usually boring: measure first, identify the bottleneck, fix the simplest thing that actually matters, then measure again.
The Mental Model: City Traffic Engineering
Imagine a growing city. Commute times are getting worse. People are angry. The mayor wants the traffic team to "fix congestion."
A bad traffic engineer immediately proposes flyovers everywhere. A good one starts with questions:
- Which roads are actually slow?
- What is the commute time for most drivers, and for the worst 1%?
- Which intersection creates the backup?
- Are roads full all day, or only during rush hour?
- Would a traffic light fix it, or do we need another lane?
- Can some trips be avoided entirely with local stores or delivery hubs?
That is backend performance engineering. Your users are the cars. Requests are trips through the city. Databases, caches, APIs, queues, and servers are roads and intersections. Scaling is not one trick — it is traffic design.
Latency → commute time Throughput → cars per minute Utilization → how full the roads are Bottleneck → the intersection where everything backs up Cache → local store near the user Queue → service lane for work that can happen later Load balancer → traffic controller CDN / edge → neighborhood depot
The rule: don't build a flyover until you know which intersection is blocked.
Step 1: Measure the Right Things
Before optimizing anything, define what "fast" means.
Latency: Use Percentiles, Not Averages
Average latency lies. If 99 users get a 100ms response and 1 user waits 10 seconds, the average still looks acceptable. But that one slow user might be the person checking out with a large cart.
Use percentiles:
- P50 — the median user experience. Half of requests are faster than this.
- P90 — the slower edge of normal traffic.
- P99 — the painful tail. If P99 is 2 seconds, 1% of users are waiting at least that long.
P99 matters because the tail often contains your most complex and valuable requests: purchases, large dashboards, account exports, bulk operations.
Throughput: How Many Requests Can You Handle?
Throughput is requests per second, jobs per minute, messages per second — how much work the system can process in a time window.
Latency and throughput are connected. As traffic rises, latency stays flat for a while. Then the system crosses a threshold and latency spikes hard. That spike is the city gridlocking.
Utilization: Leave Headroom
Utilization is how much capacity is actively being used — CPU, memory, database connections, worker slots, queue consumers.
The dangerous instinct is to run everything near 100% because it "uses resources efficiently." In reality, 100% utilization means no room for bursts. Requests start waiting in queues, and wait time grows exponentially, not linearly.
Most production systems should live around 60% to 80% utilization. The empty space is not waste. It is shock absorption.
Step 2: Find the Bottleneck
Never guess. Guessing is how teams add Redis to hide a missing database index, or split a monolith when one endpoint has an N+1 query.
Most backend slowness falls into two categories:
- I/O-bound — waiting on database queries, Redis, external APIs, disk, network.
- CPU-bound — spending compute on JSON serialization, compression, image processing, encryption, parsing, business logic.
Use the right tool for the bottleneck:
- Distributed tracing follows one request through every service and shows how many milliseconds each step added. This is how you catch "the API is slow because one database query takes 850ms."
- Profilers and flame graphs show where CPU time is spent inside the process. This is how you catch "60% of CPU is being burned serializing a giant JSON response."
Observability is not optional for performance work. Without traces and profiles, optimization is astrology.
The Database: Usually the First Traffic Jam
Databases are common bottlenecks because they do the hardest job: durable storage, concurrency control, indexes, locks, transactions, disk I/O, and query planning.
Fix N+1 Queries
N+1 means you fetch a list, then loop and query once per item:
1 query: fetch 100 posts 100 queries: fetch author for each post Total: 101 queries
The fix is to fetch in bulk: SQL joins, batched queries, or ORM primitives like select_related, includes, or eager loading. Turning 101 queries into 2 queries is often a bigger win than adding a cache.
Add Indexes Carefully
An index is like a sorted lookup map. Instead of scanning millions of rows, the database jumps directly to the matching range, usually through a B-tree.
But indexes are not free:
- They consume storage.
- They speed up reads.
- They slow down writes, because every insert, update, and delete must update the index too.
Use EXPLAIN ANALYZE to see whether the database is using your index or still doing a sequential scan. Trust the query plan, not vibes.
Use Connection Pooling
Opening a TCP connection for every query is expensive. During traffic spikes, it can also exhaust the database's hard connection limit.
A connection pool keeps a small set of open connections ready to borrow and return. When you horizontally scale application servers, use an external pooler like PgBouncer, otherwise each app instance may create its own pool and overwhelm the database together.
Caching: Avoid the Trip Entirely
If the database is a crowded central office, caching is putting a local store in every neighborhood. Users get the answer faster, and fewer trips hit the core system.
Common patterns:
- Cache-aside — check cache first. On miss, query the database, store the result, return it. Simple and common.
- Write-through — write to cache and database together. Better consistency, slower writes.
- Write-behind — write to cache immediately, update the database asynchronously. Fast, but risky if durability matters.
The hard part is invalidation. Time-based invalidation uses TTLs: this value expires after 5 minutes. Event-based invalidation deletes or updates the cache when the underlying record changes.
You can also layer caches:
- Local memory cache — fastest, but each server may have different data.
- Distributed cache — Redis or Memcached. Slight network cost, but shared across servers.
Caching trades freshness for speed. Make that trade deliberately, not accidentally.
Scaling Servers: Up vs Out
Once the software bottlenecks are fixed and traffic still grows, add capacity.
Vertical scaling means a bigger machine: more CPU, more RAM, faster disk. It is simple, requires almost no architecture change, and should not be dismissed. Many teams should scale up before they scale out.
The limits: hardware has a ceiling, one machine is a single point of failure, and users far away still pay network latency.
Horizontal scaling means multiple app servers behind a load balancer. It gives redundancy, capacity, and geographic options — but it demands statelessness.
Statelessness means no server owns user-specific state in local memory. If request one lands on Server A and request two lands on Server B, both must work.
Externalize state:
- Sessions → Redis or database
- Uploads → S3, Cloudflare R2, or object storage
- Relational data → PostgreSQL / MySQL
- Background work → queue + workers
If your app server can disappear and another one can continue serving users, you are ready to scale horizontally.
Load Balancers: The Traffic Controller
A load balancer sits in front of multiple servers and decides where each request goes.
Two common strategies:
- Round robin — send requests to servers in rotation. Simple, but blind. If some requests take 2 seconds and others take 200ms, one unlucky server can get overloaded.
- Least connections — send the next request to the server with the fewest active connections. Better for mixed workloads because heavy requests naturally keep a server "busy" longer.
Health checks are non-negotiable. The load balancer should continuously test every server. If one stops returning healthy responses, it is removed from rotation until it recovers.
Database Scaling: State Is Harder Than Servers
Application servers are easy to copy because they should be stateless. Databases are harder because they hold the truth.
Read Replicas
Most systems are read-heavy. If 70% to 90% of traffic is reads, route writes to the primary database and route reads to replicas.
The tradeoff is replication lag. A user updates their profile on the primary, then immediately reads from a replica that is 200ms behind and sees old data. That is not a bug in the replica. It is the consistency cost you chose.
Sharding
Sharding splits a huge dataset across multiple database instances using a sharding key: orders by region, users by ID range, events by date.
It increases capacity and reduces query scope, but it complicates everything: migrations, joins, transactions, backups, and rebalancing.
Distributed Databases
Modern systems often use managed distributed databases like PlanetScale, Neon, or CockroachDB to reduce the operational burden. They do not remove the hard problems, but they package many of them behind better tooling.
CDNs and Edge Computing: Move Work Closer
Some latency cannot be optimized away by better code. Physics is real. A request from Tokyo to a server on the US East Coast has a minimum round-trip cost before your app does any work.
A CDN solves this by caching content near users:
- Images
- JavaScript bundles
- CSS
- HTML
- Cacheable API responses
Instead of crossing the planet, the user hits a nearby edge node. Latency drops and your origin servers handle less traffic. CDNs also absorb large DDoS attacks before they reach your infrastructure.
Edge computing goes one step further: run lightweight code at the edge. Validate a token, redirect a user, personalize a small response, reject unauthorized traffic before it reaches your origin.
The tradeoff: edge runtimes are constrained. Limited CPU, memory, filesystem access, and execution time. Great for lightweight decisions. Bad for heavy business logic.
Async Processing: Don't Make Users Wait
Not everything needs to happen before the response. If a signup takes 100ms to write the user and 300ms to send an email, don't make the user wait 400ms.
Request: 1. Write user to database 2. Push "send email" job to queue 3. Return 201 Created Worker: 4. Send email later
Queues are perfect for email, image resizing, video processing, webhooks, reports, cleanup jobs, and bulk deletion. They reduce perceived latency and make failures retryable outside the request path.
Monoliths, Microservices, and Serverless
A monolith is one deployable application. It is simple to build, test, debug, and deploy. Start here unless there is a strong reason not to.
Microservices are not primarily a machine-performance tool. They are a team-scaling tool. When 100+ developers work in one codebase, independent deployability matters. Teams can own services, scale resource-heavy parts separately, and choose specialized languages where needed.
But the cost is huge: function calls become network calls, debugging requires distributed tracing, and data consistency across services becomes a real systems problem.
Serverless removes capacity planning. You provide functions and events; the provider runs them on demand and bills per execution time. It is great for spiky workloads and small teams.
The tradeoffs: cold starts, strict execution limits, statelessness, limited long-lived connections, and provider-specific constraints. Serverless is not "no servers." It is "someone else manages the servers, and you accept their rules."
The Core Rules
- Measure before optimizing. Use metrics, traces, profiles, and logs. Guessing creates expensive wrong fixes.
- Prefer simple solutions. A database index is better than a Redis layer if the index solves the problem. A bigger server is better than distributed complexity if it buys enough time.
- Scale for the real problem. Don't design for a million users on day one. Build for your current scale with reasonable headroom.
- Keep systems observable. If you can't see latency, throughput, utilization, and bottlenecks, you can't safely improve them.
- Complexity is a cost. Every cache, replica, queue, shard, service, and edge function becomes something you must debug at 3 AM.
The Mental Model Summary
Carry this forward:
- Performance engineering = measure, locate the bottleneck, apply the smallest effective fix, measure again.
- The city traffic model = requests are cars, latency is commute time, throughput is cars per minute, utilization is road fullness, bottlenecks are intersections.
- Percentiles beat averages = P99 tells you about the painful tail that averages hide.
- Utilization needs headroom = 100% utilization creates queues and exponential wait time.
- Trace I/O, profile CPU = distributed tracing finds slow dependencies; flame graphs find hot functions.
- Database first = fix N+1 queries, add the right indexes, use connection pooling before reaching for heavier architecture.
- Caching = avoid expensive trips, but manage freshness intentionally.
- Horizontal scaling requires statelessness = sessions, files, and state must live outside app servers.
- Load balancers and health checks = route traffic only to healthy capacity.
- CDNs and edge = move simple work closer to users because physics has a latency floor.
- Queues = return fast and do non-critical work later.
- Microservices and serverless = useful tools, not default destinations. They solve specific problems and introduce new ones.
The beginner scales by adding technology. The engineer scales by removing the actual bottleneck. Sometimes that means Redis, replicas, queues, serverless, or microservices. Sometimes it means one missing index. The only way to know is to measure.
This is part 17 of my "Backend from First Principles" series. Everything in this article comes from what I learned watching the Backend from First Principles playlist by Sriniously — specifically the video on backend scaling and performance engineering. I'm not claiming original research. This is me taking what I studied, internalizing it, and presenting it as a mental model that made the concepts stick for me. If it helps you too, even better.
Follow along for the next one.