How to design Netflix's video delivery (CDN)

Requirements

• Functional: a viewer presses play and the video starts quickly, then keeps playing without rebuffering; quality adapts to the viewer's connection.
• Non-functional: start-up time of a couple of seconds, very low rebuffering, and a system that survives a launch where a large share of viewers start the same title in the same hour.
• Scope the question: delivery of video bytes, not recommendations, billing or the upload pipeline - unless the interviewer asks.

Capacity estimates

The numbers make the architecture obvious: no central data centre can push 50 Tbit/s to the world. Bytes must be served from caches close to viewers, and the origin should see only a small fraction of requests.

How video is served

• Encode each title into a ladder of resolutions and bitrates, then split each encoding into small segments of a few seconds.
• The player fetches a manifest (HLS or DASH) listing the segments, then downloads segments one by one over plain HTTP.
• Adaptive bitrate: the player measures throughput and picks the next segment's quality, so a slow connection drops to a lower bitrate instead of stalling.
• Because segments are immutable files fetched over HTTP, they are perfect for caching at the edge with long TTLs.

High-level design

The control plane and the data plane are separate. The control plane (API servers in a cloud region) handles login, browsing and "play", and returns a manifest whose segment URLs point at the best edge cache for that viewer. The data plane is the CDN: edge servers inside ISPs and internet exchange points that serve the segments.

• Viewer → API (control plane) → manifest with edge URLs.
• Viewer → nearest edge cache → on miss, a regional cache or origin shield → origin storage.
• Netflix runs its own CDN, Open Connect, placing cache appliances inside ISP networks; many other services use commercial CDNs.

Where it breaks: the cold-cache stampede

A cache only protects the origin when it already holds the content. When a new title launches, the edge caches are cold: every viewer's first request misses, and all of those misses go to the origin at the same moment. In the simulation above, switching the CDN to a cold cache sends the full 20,000 requests per second back to an API tier that can serve 8,000, and 60% of requests fail.

The same thing happens after a cache flush, a deploy that changes cache keys, or a TTL that expires for a hot object at the same instant everywhere.

Preventing the stampede

• Pre-position content: push a title to edge caches before launch, during off-peak hours, based on predicted popularity. Netflix's Open Connect fills its appliances this way.
• Origin shield: put a regional caching tier between the edges and the origin, so a thousand edge misses for the same segment become one origin request.
• Request coalescing: when many requests miss on the same object at once, let one fetch it and the rest wait for that result.
• Staggered TTLs: add jitter to expiry times so hot objects don't all expire at the same moment.
• Multiple CDNs and steering: route viewers to another CDN or edge when one is overloaded or down.

Hit ratio is the lever

Every point of cache hit ratio removes load from everything behind the edge. At a 70% hit ratio the origin serves 30% of requests; at 95% it serves 5% - six times less. Viewing is highly concentrated on a small share of popular titles, which is what makes very high hit ratios achievable for streaming. In the simulation, pre-warming the CDN from 70% to 95% cuts origin traffic from 6,000 to 1,000 requests per second.

What interviewers look for

• You estimated egress bandwidth and concluded that edge caching is mandatory.
• You separated the control plane from the data plane.
• You explained segments, manifests and adaptive bitrate.
• You raised the cold-cache stampede and gave several defences: pre-positioning, origin shield, coalescing and TTL jitter.
• You reasoned about hit ratio quantitatively.