Concurrency & Parallelism: I/O Bound vs CPU Bound: A Mental Model for Backend Engineers
Concurrency is not "doing everything at once." It is "not wasting the CPU while one task is waiting."
This is part 18 of my backend-from-first-principles series. Concurrency and parallelism are often used like synonyms, but they solve different problems. The difference matters because most backend systems are not CPU-bound most of the time. They are I/O-bound: waiting on databases, caches, networks, queues, and external services.
The Mental Model: A Restaurant Kitchen
Imagine a busy restaurant kitchen. Orders are coming in. Food has to be chopped, cooked, baked, plated, and served.
A bad kitchen has one chef start rice, then stare at the pot until it finishes. Then they start the next item. Then they wait again.
A good kitchen works differently:
- Start the rice.
- While it cooks, chop vegetables.
- While bread bakes, plate another dish.
- When the timer rings, return to the waiting item.
That is backend concurrency. The chef is the CPU. The oven, rice cooker, supplier, and delivery counter are I/O. A good system keeps the CPU doing useful work while other operations wait.
CPU-bound work → chopping, kneading, plating I/O-bound work → baking, boiling, delivery waiting Concurrency → one chef switching between dishes Parallelism → many chefs cooking at the same time Race condition → two chefs changing the same order ticket
The rule: don't make the chef stare at the oven.
Step 1: Know Whether You Are Waiting or Computing
Backend work usually falls into two buckets.
I/O-bound means the request spends most of its time waiting outside the CPU: database queries, Redis calls, HTTP APIs, file reads, message queues, email providers, payment gateways.
CPU-bound means the request spends most of its time calculating: JSON parsing, validation, encryption, compression, image processing, video encoding, large transformations, ML inference.
The numbers explain why this matters:
Local database query ~1-2 ms Different availability zone ~20-30 ms Different region ~90-100 ms Simple validation ~1-2 ms
A mid-level API call might make five network operations. If each takes around 50ms, the request spends 250ms waiting. If the CPU only spends 10ms doing real work, the CPU is idle for roughly 95% of that request.
A modern CPU can execute roughly 3 billion instructions per second. That is about 3 million instructions per millisecond. Waiting 100ms for a database response can waste the opportunity to do hundreds of millions of instructions if the server has nothing else to run.
Most backend performance work starts by asking: is the system waiting, or is it calculating?
Step 2: Use Concurrency for Waiting
Concurrency means managing many tasks over the same time period. It does not require multiple CPU cores. One core can do it by pausing a task that is waiting and running another task that is ready.
This is perfect for I/O-bound systems. While request A waits for the database, the server can start request B, resume request C, and finish request D.
The old way to get concurrency was one OS thread per request. That works, but it gets expensive:
- An OS thread stack on Linux can reserve up to 8MB of virtual memory.
- Even 500KB to 1MB of physical memory per thread becomes huge at 10,000 requests.
- Creating a thread can take microseconds to milliseconds.
- Context switching can cost 1 to 10 microseconds per switch.
At small scale, that is acceptable. At high concurrency, thousands of mostly waiting threads become memory pressure and scheduler overhead.
Event Loops: One Chef, Many Timers
Event-loop runtimes like Node.js solve the waiting problem with a different model. Instead of creating one heavy thread per request, the main thread starts I/O, registers interest in the result, and moves on.
When the database responds, the event loop resumes the waiting callback or promise. This is why one Node.js process can keep many I/O-bound requests in flight without thousands of OS threads.
But the tradeoff is strict: do not block the event loop.
If a CPU-heavy task takes 100ms on the event loop, everything else waits too. One expensive image resize, PDF render, or encryption loop can freeze progress for unrelated requests.
Event loops are excellent at waiting. They are fragile when you make them do heavy CPU work.
Step 3: Use Parallelism for Compute
Parallelism means executing multiple tasks at the exact same time. That requires hardware support: at least two CPU cores.
If work is CPU-bound, concurrency alone does not create more compute power. It only slices the same CPU into smaller time chunks. For heavy computation, use worker threads, separate processes, background workers, more cores, or more machines.
In kitchen terms, one chef can switch between many waiting dishes. But if four dishes all need active chopping right now, switching faster does not help. You need more chefs.
Goroutines and Virtual Threads: Lightweight Tickets
Languages and runtimes like Go use lightweight concurrency units managed by the runtime instead of mapping every task directly to a native OS thread.
A Go application might run thousands or millions of goroutines over a small number of OS threads. The runtime scheduler decides which goroutine runs on which thread, and switching between goroutines is much cheaper than switching between heavy native threads.
This gives a useful programming model: code can look blocking, while the runtime still multiplexes many waiting tasks efficiently.
The Race Condition: The Shared Order Ticket
Concurrency is not only a performance topic. It is also a correctness topic.
Imagine two chefs reading the same order ticket. Both see "one dessert left." Both decide they can serve it. Both update the ticket. Now the kitchen has promised two desserts when only one existed.
That is the lost update problem:
balance = 100 Request A checks: 100 >= 100 Request B checks: 100 >= 100 Request A subtracts 100 -> 0 Request B subtracts 100 -> -100
Both requests made a locally valid decision. Together, they produced invalid state.
The fix is not "never use concurrency." The fix is to protect shared state: database transactions, row locks, atomic updates, mutexes, queues, idempotency keys, or ownership boundaries.
The Core Rules
- If the system is waiting, use concurrency. Keep other requests moving while I/O is in flight.
- If the system is calculating, use parallelism. Add cores, workers, processes, or machines for CPU-heavy work.
- Do not block the event loop. Move heavy CPU work out of the main request loop.
- Do not create unbounded threads. Threads have memory and scheduling costs.
- Protect shared state. Concurrency without synchronization creates race conditions.
The Mental Model Summary
Carry this forward:
- I/O-bound work = the chef is waiting on the oven, supplier, or delivery counter.
- CPU-bound work = the chef's hands are actively busy.
- Concurrency = one chef switching between many dishes so waiting time is not wasted.
- Parallelism = many chefs working at the same time.
- OS threads = powerful but heavy at very high concurrency.
- Event loops = great for I/O-bound APIs, dangerous when blocked by CPU-heavy work.
- Goroutines and virtual threads = lightweight tasks scheduled by the runtime over fewer OS threads.
- Race conditions = two tasks touching shared state without coordination.
The beginner asks, "How do I make this run at the same time?" The engineer asks, "Is this waiting, computing, or sharing state?" That question decides whether the answer is async I/O, more CPU workers, a queue, a lock, a transaction, or a different design entirely.
This is part 18 of my "Backend from First Principles" series. Everything in this article comes from what I learned watching the Backend from First Principles playlist by Sriniously — specifically the video on concurrency, parallelism, I/O-bound work, and CPU-bound work. I'm not claiming original research. This is me taking what I studied, internalizing it, and presenting it as a mental model that made the concepts stick for me. If it helps you too, even better.
Follow along for the next one.