Designing ChatGPT for 800M Users — on One Primary Database
A system design walkthrough in 17 diagrams: scoping, capacity, architecture, the read/write split that keeps OpenAI on a single unsharded Postgres primary, and where the $58.6M monthly bill actually goes.
Most people assume ChatGPT runs on a sharded database. It does not. OpenAI serves 800 million weekly users from a single unsharded Postgres primary, and the reason is a ratio almost everyone gets backwards.
This is that design, in 17 diagrams. Each one gets a few sentences — enough to
say what it decides and why. Every diagram is a generated .excalidraw file you
can open and edit.
1. Scope it before you draw anything
Nine questions. Each answer removes work from the design — the third one is the only one that changes the architecture.
2. Stream, or the product feels broken
Same tokens, same total generation time. The only variable is when the user sees the first one.
3. SSE, not WebSockets
Chat streaming is one-directional. WebSockets buy you a duplex channel you never use, and charge you 20M sockets held open while people read.
4. What streaming looks like on the wire
One HTTP request, one long-lived response body, one event per token. The send button stays disabled until the stream closes.
5. Requirements worth writing down
The right-hand column is the one that matters. Every number in it becomes a sizing constraint later.
6. Capacity — five chains, five numbers
Start from stated assumptions and multiply. Nothing here is clever; it just has to be written down before you size anything.
7. The architecture
Four layers. The Message Service is the orchestrator — it is the only component that talks to everything else.
8. One prompt, end to end
Count the arrows. Nine reads before the LLM is even called; the two writes happen after the response has already reached the user.
9. Why one primary is enough
This is the whole answer. 2.5B prompts a day looks write-heavy. Sending one message triggers roughly nine reads and one write, so the system is ~90% reads — and reads scale by adding replicas.
10. The schema
Three tables. Users have conversations, conversations have messages. That is the entire domain.
11. Three indexes, three queries
One index per query that runs on every page load. Column order and sort direction come from the query, not from habit.
12. Four candidates, one brief
554 TB in year one, ~90% reads, zero message loss, strict ordering. The cons column decides it.
13. And when each one is right
The jump that hurts is stage 3 to stage 4. Everything before it is configuration; sharding is a rewrite of every endpoint that touches the database.
14. The API surface
Nine endpoints. Only one of them is unusual: the message endpoint returns
text/event-stream instead of JSON.
15. Two limits, because there are two abuses
One huge prompt burns the budget without tripping a request limit. A flood of tiny prompts trips nothing at all. Neither counter catches the other's failure.
16. Where the money goes
$58.6M a month, and 89% of it is one line. Every other component is rounding error — which tells you exactly where to spend optimisation effort.
17. How those numbers were reached
Each fleet size traces back to the capacity estimate. Two assumptions carry most of the weight: 250 tokens per request, and 10 reads per message.
The three things worth remembering
- Count reads before you call something write-heavy. 2.5B prompts a day is ~90% reads once you count what each one loads. That single ratio is why a single primary survives 800M users.
- Stream because of latency, not throughput. Total generation time is unchanged. Time-to-first-token is everything.
- Optimise the 89% line. Prompt caching beats query tuning by three orders of magnitude here.
The diagrams
All 17 are generated .excalidraw scenes — Python scripts that emit the JSON,
plus an exporter that produces the light and dark SVG on this page. Sources and
generators are on GitHub:
packages/system_design_how_to_design_chatgpt.
There is also a combined canvas that stacks all 17 into named Excalidraw frames, if you would rather pan around one file than open seventeen.
git clone https://github.com/henryonai/blog-code.henryonai.com
cd blog-code.henryonai.com/packages/system_design_how_to_design_chatgpt
for f in scripts/generate-*-excalidraw.py; do python3 "$f"; done
python3 scripts/build-combined-excalidraw.pyBased on "Design ChatGPT Like a Senior Engineer" by Hayk Simonyan. Scale figures are OpenAI's published numbers as of January 2026; the cost model is an estimate built on Azure list pricing, not OpenAI's actual bill.