The foundational system-design mechanism everything else in this domain builds on: why a single server always hits a ceiling, the concrete difference between scaling vertically (a bigger box) and horizontally (more boxes), and the load-balancing algorithms and health-check machinery that make 'more boxes' actually work as one coherent service instead of a pile of independent servers a client has to pick between.
1. What is the core difference between vertical and horizontal scaling?
Interview Questions
2 questions
Cheatsheet
Download-ready reference
Cheatsheet
VERTICAL SCALING (scale up): bigger single machine.
+ No application changes needed.
- Hard ceiling (biggest instance available).
- Still a single point of failure.
HORIZONTAL SCALING (scale out): more machines running the same app.
+ No hard ceiling, removes single point of failure.
- Introduces the load-balancing problem: which server
handles each request?
LOAD BALANCING STRATEGIES:
Round robin — cycles servers evenly by REQUEST COUNT.
Assumes uniform request cost & server capacity.
Least connections — routes to server with FEWEST active connections.
Naturally adapts to uneven request cost/capacity.
Weighted — manual traffic-share ratio, for mixed fleets.
IP hash/consistent
hashing — same client → same server (session affinity).
HEALTH CHECKS: periodic probes to a lightweight endpoint (e.g.
/health). Failing checks → server pulled from rotation automatically.
Without health checks, a load balancer keeps sending traffic to
DEAD servers — it's just a splitter, not fault tolerance.
STATE PROBLEM: horizontal scaling breaks anything relying on
server-local memory (e.g. in-memory sessions).
Fix 1: STICKY SESSIONS (same client → same server, via LB config).
Downside: losing that server loses pinned sessions.
Fix 2: EXTERNALIZE STATE (Redis/DB shared by all servers).
Preferred — preserves true server interchangeability.
STATELESS service: any instance can handle any request. Scales
horizontally with near-zero friction.
STATEFUL service (DB, WebSocket server, in-memory cache): requires
a dedicated scaling strategy (replication, sharding, etc.) —
covered in later topics.
CONNECTION DRAINING: let in-flight requests finish before pulling a
server from rotation during deploys — avoids cutting off active users.
RULE OF THUMB: scale vertically first (cheap, fast, zero architecture
change) to buy time; use that time to build toward a stateless
architecture that can scale horizontally without a ceiling.
Every system-design conversation — whether it's a whiteboard interview or a real incident review — eventually comes back to the same question: what happens when there's more traffic than one machine can handle? Scalability is the discipline of answering that question deliberately, before it gets answered for you at 2am by an outage.
Vertical scaling: the easy first move that always runs out of road#
The simplest way to handle more load is to make the single server bigger — more CPU cores, more RAM, faster disks. This is vertical scaling (scaling up). It's attractive because it requires zero application changes: your app doesn't know or care that it's running on a bigger box. But it has two hard limits. First, there's a physical ceiling — at some point you're renting the largest instance a cloud provider sells, and there's nowhere bigger to go. Second, and often more urgent in practice, a single machine is a single point of failure: no matter how big it is, if that one machine goes down, the entire service goes down with it.
Horizontal scaling: trading a ceiling for a coordination problem#
Horizontal scaling (scaling out) means running the same application on multiple servers instead of one bigger one. This removes the hard ceiling — you can, in principle, keep adding servers — and removes the single point of failure, since one server dying leaves the others still serving traffic. But it introduces a new problem that vertical scaling never had to deal with: now there are N servers capable of handling a request, and something has to decide which one handles each incoming request. That "something" is a load balancer.
Vertical scaling hits a ceiling; horizontal scaling trades that ceiling for a new problem: distributing load
step 1 / 4
// Traffic grows: 1,000 req/s on a single 4-core, 8GB server
// Response times start climbing
Single server
waiting
4 vCPU / 8GB — saturating
Every scaling story starts here: one server, growing traffic, and CPU or memory approaching 100%.
A load balancer sits in front of a pool of application servers and distributes incoming requests across them, using one of a few common strategies:
Round robin: request 1 → server A, request 2 → server B, request 3 → server C, request 4 → server A, ... Simple, even distribution — assumes all requests and all servers are roughly equal cost.Least connections: route to whichever server currently has the FEWEST active connections. Better than round robin when request costs vary wildly (some requests are cheap, some are expensive long-running ones).Weighted: like round robin or least-connections, but servers get a weight (e.g. a bigger box gets 2x the traffic share of a smaller one) — useful in a mixed fleet.IP hash / consistent hashing: route based on a hash of the client's IP (or another key), so the SAME client tends to land on the SAME server repeatedly — relevant for session affinity (see "sticky sessions" below).
Just as important as distributing traffic is not sending traffic to a server that's unhealthy. Load balancers run health checks — periodic requests to a lightweight endpoint (often /health or /healthz) — and automatically stop routing to any server that fails enough consecutive checks, then resume once it starts passing again. This is what actually makes horizontal scaling fault-tolerant in practice: a crashed instance isn't just "one of several servers" in the abstract, it's a server the load balancer has concretely detected and stopped sending traffic to, usually within seconds.
Round-robin routing works cleanly when every request is independent. It breaks down the moment a server holds state a later request needs — the classic example is a login session stored only in that one server's memory. If request 1 hits server A and logs the user in, but request 2 gets round-robined to server B, server B has never heard of this session and the user appears logged out.
There are two common fixes, and they represent a real architectural fork:
Sticky sessions — configure the load balancer to route the same client to the same server every time (often via IP hash or a cookie). This works but partially defeats the purpose of load balancing: if that one server goes down, every session pinned to it is lost, and load can become uneven if some clients are much "heavier" than others.
Externalize the state — move session data out of any individual server's memory into a shared store (Redis, a database) that every server can read from. Now any server can handle any request for any user, because the state itself isn't tied to a server anymore. This is the approach most modern systems converge on, because it preserves the actual benefit of horizontal scaling — genuine interchangeability of servers — rather than working around its loss.
This state question — "does this server need to remember anything between requests?" — is the single most important design decision in making a service horizontally scalable, and it resurfaces directly in the caching, database, and real-time topics later in this domain.
A stateless service is one where any instance can handle any request, because no server holds request-relevant data that isn't also available elsewhere (a database, a cache, the request itself). Stateless services scale horizontally with almost no friction — add more instances, the load balancer spreads traffic, done. A stateful service (a database, a WebSocket server holding open connections, anything doing in-memory session storage) requires much more careful design to scale out, because you can no longer treat instances as interchangeable. Most real systems aim to push as much state as possible out of the "easy to scale" application tier and into a small number of purpose-built, carefully-scaled stateful tiers (a database, a cache, a message broker) — which is exactly why those get their own dedicated topics in this domain.
You're running a checkout API on 3 servers behind a load balancer using round robin. One of the 3 servers is noticeably underpowered (half the CPU of the other two). Traffic is otherwise uniform. What will you observe, and what's the fix?
Solution
What you'll observe: the underpowered server will fall behind — its requests will take longer, and its queue of in-flight connections will grow relative to the other two, even though round robin is sending it an equal count of requests. Round robin balances request count, not request cost or server capacity — it has no concept of "this server is slower."
The fix is one of:
Weighted round robin — configure the load balancer to send the weaker server a smaller share (e.g. half as many requests as each of the other two), matching traffic share to actual capacity.
Least-connections routing — switch strategy entirely; because the weak server will naturally accumulate more in-flight connections as it falls behind, least-connections will automatically route it less traffic without needing a manually-tuned weight.
The most durable fix, if practical, is simply not running a mixed-capacity fleet — homogeneous instances make every other load-balancing decision simpler, which is why many real infrastructures standardize on one instance type per tier.
A minimal round-robin and least-connections load balancer — the actual selection logic real load balancers run per-request, stripped down to its core:
class RoundRobinBalancer { constructor(servers) { this.servers = servers; // e.g. ["A", "B", "C"] this.index = 0; } next() { const server = this.servers[this.index]; this.index = (this.index + 1) %
This is genuinely how production load balancers reason, just with far more machinery around it: real health checks, weighting, connection draining during deploys, and TLS termination sit on top of the exact same core selection logic shown here.
The health-check loop a load balancer runs is conceptually similar to how any long-lived server process manages its own event handling — see The Node.js Event Loop for how a single Node process services many concurrent connections without blocking, which is part of why a single well-tuned instance can absorb more concurrent load than intuition suggests before horizontal scaling becomes necessary at all.
1. Storing session state in server memory, then adding a load balancer#
// ❌ In-memory session store — breaks the moment there's more than one serverconst sessions = new Map();app.post("/login", (req, res) => { sessions.set(req.body.userId, { loggedIn: true }); res.sendStatus(200);});
The moment a second server is added, roughly half of all "am I logged in" checks will hit a server that never saw the original login — because that server's sessions Map is a separate object in a separate process. The fix is externalizing session state to Redis or a database so every server reads from the same source of truth.
2. Treating vertical scaling as a permanent fix instead of a stopgap#
"Traffic doubled — let's just upgrade to the next instance size." // ❌ works once, then again, then you're on the largest SKU with nowhere left to go
Vertical scaling is a legitimate short-term move (it's genuinely simpler and requires no architecture changes), but relying on it as the only scaling strategy means eventually hitting a hard ceiling with no next step, usually under time pressure during a traffic spike rather than calmly in advance.
3. Assuming a load balancer removes the need for health checks#
// ❌ Load balancer configured with a static server list, no health checksservers: ["10.0.0.1", "10.0.0.2", "10.0.0.3"]// If .2 crashes, it silently keeps receiving 1/3 of all traffic and erroring
A load balancer without health checks is just a traffic splitter, not a fault-tolerance mechanism — it will happily keep routing requests to a server that's crashed, is out of memory, or is returning 500s, because it has no signal telling it to stop. Health checks are what make "one server can die without taking down the service" actually true in practice.
Design for statelessness in the application tier from the start — it's far cheaper to build stateless from day one than to retrofit it once sticky sessions are load-bearing in production.
Externalize session/state to a shared store (Redis, a database) rather than relying on sticky sessions as a permanent architecture, reserving sticky sessions for cases where the cost of externalizing genuinely outweighs the benefit.
Always pair horizontal scaling with health checks — more servers only improves fault tolerance if the load balancer can actually detect and route around a failed one.
Prefer a homogeneous fleet (same instance type per tier) unless there's a concrete reason not to — it keeps round-robin-style balancing accurate without needing manually-tuned weights.
Scale vertically first when it's cheap and horizontally when it's necessary — vertical scaling buys time with zero architectural cost; use that time to build the statelessness horizontal scaling actually requires.
Least-connections routing generally outperforms round robin whenever request costs vary significantly (e.g. a mix of cheap reads and expensive writes) — round robin's assumption of uniform request cost breaks down exactly there.
Connection draining (letting in-flight requests finish before removing a server from rotation during a deploy) avoids dropping active user requests mid-flight — a load balancer that simply yanks a server instantly during rollout will cut off whoever was mid-request.
Health check intervals are a real tradeoff: too slow and a dead server keeps receiving traffic for longer after failing; too fast and normal transient blips (a slow GC pause, a brief network hiccup) can cause healthy servers to be needlessly pulled from rotation.
this
.servers.
length
;
return server;
}
}
const rr = new RoundRobinBalancer(["A", "B", "C"]);