# Spike

A load balancer under a traffic spike: every request is a bead of light through a miniature data hall, with queues, health checks, outlier ejection, autoscaling and load shedding on a live NOC wall

By Numa

Topic: Computing, AI & Electronics

[Open the interactive Jig](https://ask.jig.com/j/xqd8x65ww3Q2)

## Guide

### 1. Steady state

One load balancer, four backends, round robin\. Every bead of light is one request, slowed ten times so you can follow it: in through the balancer, down an arc into a backend's queue, through one of two workers, and back as a green response\. Utilisation sits near 45%, and p99 is calm\.

The wall is computed from the beads you see: RPS, p99, error rate and utilisation\.

### 2. The spike

Someone just linked to you\. Send the spike and arrivals triple in two seconds while capacity stays put\. Once arrivals pass what the workers can serve, ρ = λ/cμ crosses 1, the queues stop draining, and p99 rides the hockey stick straight up\.

In an M/M/1 queue, going from 90% to 99% utilisation multiplies mean latency by ten, and p99 sits about 4\.6 times above the mean\.

### 3. One slow backend

Backend 3 is stuck in long GC pauses and serves four times slower\. Round robin hands it an equal share anyway, so its queue towers while its neighbours idle\. Random is no better\. Two choices samples two backends and sends the request to the shorter queue, and the slow one simply stops winning\.

Envoy's LEAST\_REQUEST samples two hosts by default\. With two choices, the worst queue grows like ln ln n instead of ln n / ln ln n \(Mitzenmacher\)\.

### 4. Ejected

Break a backend, or tap a rack\. A bad deploy answers 5xx in a few milliseconds, so it looks like the fastest host in the pool and two choices feeds it more\. Passive outlier detection counts consecutive 5xx and ejects it at five\. Probes keep pinging it, and once the rollback passes them it rejoins on slow start\.

Envoy defaults: 5 consecutive 5xx, 30 s base ejection that grows with each repeat, at most 10% of hosts ejected at once\.

### 5. Scale out

The spike is back, and this time target tracking is on\. Utilisation is far above its 60% target, so new backends boot, pass their first health check, and ramp in on slow start so a cold cache never takes a full share at once\. The queues drain and p99 comes back down\.

Real scale\-out takes minutes, not seconds\. The spike always lands first, which is why the next step matters\.

### 6. Shed load, not latency

Capacity is fixed at four and demand is not\. With unbounded queues, every request waits behind the overload, clients time out, and their retries add even more load: a retry storm, with grey zombies still being served long after anyone wants them\. Turn on load shedding and the balancer answers the excess with an instant 503\.

The SRE book's advice: retry at most 3 times per request, and keep retries under 10% of all requests\. A 503 can carry Retry\-After\.

### 7. Your incident

Everything is live now: two choices, health checks, outlier ejection, autoscaling and shedding\. Drag the traffic, tap racks to break or fix them, and watch the balancer, the probes and the autoscaler fight it out on the wall\.

Little's law: requests in flight = arrival rate × latency\. 1,000 req/s at 50 ms is 50 in flight\.