← All posts

Designing ChatGPT for 800M Users — on One Primary Database

A system design walkthrough in 17 diagrams: scoping, capacity, architecture, the read/write split that keeps OpenAI on a single unsharded Postgres primary, and where the $58.6M monthly bill actually goes.

Henry Hoang10 min read

Most people assume ChatGPT runs on a sharded database. It does not. OpenAI serves 800 million weekly users from a single unsharded Postgres primary, and the reason is a ratio almost everyone gets backwards.

This is that design, in 17 diagrams. Each one gets a few sentences — enough to say what it decides and why. Every diagram is a generated .excalidraw file you can open and edit.

1. Scope it before you draw anything

Nine questions. Each answer removes work from the design — the third one is the only one that changes the architecture.

Nine clarifying questions paired with the scope decision each one locks in: integrate an existing LLM, stream responses, 800-900M weekly actives, accounts with private history, English only, limit requests and tokens, text only, basic moderation.
Ask, then write down what the answer just ruled out.

2. Stream, or the product feels broken

Same tokens, same total generation time. The only variable is when the user sees the first one.

A frontend decision splitting into two paths. Option 1: send HTTP POST, wait for full response, show spinner, user waits 10-30 seconds, poor experience. Option 2: open persistent connection, stream tokens as they arrive, display progressively, user sees immediate progress, great experience.
Option 1 cannot meet a 2-second first token — nothing renders until the last one.

3. SSE, not WebSockets

Chat streaming is one-directional. WebSockets buy you a duplex channel you never use, and charge you 20M sockets held open while people read.

WebSocket path requiring protocol upgrade and a bidirectional channel held open, with use cases listed as collaborative editing, typing indicators and multiplayer. SSE path over standard HTTP with server-streamed tokens and the browser EventSource API reconnecting automatically.
WebSockets solve a problem this system does not have.

4. What streaming looks like on the wire

One HTTP request, one long-lived response body, one event per token. The send button stays disabled until the stream closes.

Sequence diagram between user, browser and server. The browser sends one message request, the server opens an SSE connection and streams tokens one at a time, each displayed to the user as it arrives, until a stream-complete signal closes the connection.
The user reads token 1 while the server is still generating token N.

5. Requirements worth writing down

The right-hand column is the one that matters. Every number in it becomes a sizing constraint later.

Two panels. Functional requirements: registration, conversation management, streaming responses via SSE, full history, token-by-token streaming, rate limiting, moderation. Non-functional: 99.9% availability, first token under 2 seconds, 200M DAU with 20M concurrent, zero message loss, cost efficiency, encryption.
Zero message loss rules out eventual consistency before you have picked a database.

6. Capacity — five chains, five numbers

Start from stated assumptions and multiply. Nothing here is clever; it just has to be written down before you size anything.

Five lanes of capacity estimation. Traffic: 1B weekly actives to 200M DAU to 2.5B messages per day, peaking at 69,444 messages per second and 20M concurrent conversations. Conversation storage: 600 bytes per exchange growing to 548 TB annual. Metadata storage: 6 TB, giving a year-one total of 554 TB. Connection overhead: 20M concurrent SSE connections. Bandwidth: 750 MB per second, 6 Gbps peak outbound.
554 TB in year one, 69,444 messages/second at peak, 20M concurrent connections.

7. The architecture

Four layers. The Message Service is the orchestrator — it is the only component that talks to everything else.

High-level design across four layers. Users reach a CDN and load balancer, which route to an API gateway and the Message Service. The Message Service calls auth, rate limiting, moderation, conversation and LLM gateway services, which read and write Redis, a Postgres primary with 20 read replicas, cold storage and a GPU fleet.
Everything fans out from one orchestrator, and the stream comes back the same way.

8. One prompt, end to end

Count the arrows. Nine reads before the LLM is even called; the two writes happen after the response has already reached the user.

Sequence diagram across ten participants. The message service validates the user, checks rate limits, checks content safety, fetches conversation context, sends the prompt to the LLM gateway, streams tokens back via SSE, then writes the messages to Postgres and the rate-limit counter to Redis in parallel.
Green is a read, red is a write, blue is the token stream.

9. Why one primary is enough

This is the whole answer. 2.5B prompts a day looks write-heavy. Sending one message triggers roughly nine reads and one write, so the system is ~90% reads — and reads scale by adding replicas.

Message and conversation services sending about 10% writes at 30,000 per second to a single Postgres primary, and reading about 90% of traffic from 20 read replicas fed by streaming replication. Read operations listed include loading accounts, quotas, rate counters, moderation rules, conversation metadata and recent messages; write operations are only the prompt, the AI response and the rate-limit counter.
30,000 writes/second is inside what one tuned primary handles. The reads are the part that grows.

10. The schema

Three tables. Users have conversations, conversations have messages. That is the entire domain.

Entity relationship diagram with three tables. Users has id, email, password hash, tier and timestamps. Conversations has id, user id, title and timestamps. Messages has id, conversation id, role, content, token count and created at. One user has many conversations; one conversation has many messages.
A natural hierarchy — exactly what a relational database is good at.

11. Three indexes, three queries

One index per query that runs on every page load. Column order and sort direction come from the query, not from habit.

Three indexes shown next to the query each serves: an index on users email for login, an index on conversations user id and created at descending for the recent-conversations sidebar, and an index on messages conversation id and created at ascending for reading a conversation in order.
Every index is paid for on every insert, which is why there are three and not thirty.

12. Four candidates, one brief

554 TB in year one, ~90% reads, zero message loss, strict ordering. The cons column decides it.

Four database options with pros and cons. PostgreSQL with a single primary and read replicas. MongoDB. Neon serverless Postgres. Sharded PostgreSQL. Each lists advantages in green and disqualifying trade-offs in red.
MongoDB loses on message ordering, not on scale. Sharding loses on migration cost, not on theory.

13. And when each one is right

The jump that hurts is stage 3 to stage 4. Everything before it is configuration; sharding is a rewrite of every endpoint that touches the database.

Four stages by scale. Zero to 5M daily users: Neon serverless Postgres. 5M to 100M: managed Postgres with read replicas. 100M to 200M: a tuned primary with 20 replicas, where OpenAI sits today. Over 200M: offload write-heavy workloads, and shard only if forced.
Pick the cheapest thing that still works, and move when writes — not reads — run out.

14. The API surface

Nine endpoints. Only one of them is unusual: the message endpoint returns text/event-stream instead of JSON.

Sequence diagram of client, auth service, API gateway, database and AI service, alongside the request and response shapes for the authentication, conversation and message endpoints, including the SSE streaming response with token and done events.
Call order on the left, wire format on the right.

15. Two limits, because there are two abuses

One huge prompt burns the budget without tripping a request limit. A flood of tiny prompts trips nothing at all. Neither counter catches the other's failure.

Two abuse scenarios. A single large request with low request count but massive token usage burns the budget in one shot without token limits. Many tiny messages with high request volume and low token count cause system spam without request limits. Below, a two-tier design: free tier at 25 requests and 50k tokens per day, paid tier at 500 requests and 2M tokens per day.
Counters live in Redis, keyed by user id, and the tier column decides which pair applies.

16. Where the money goes

$58.6M a month, and 89% of it is one line. Every other component is rounding error — which tells you exactly where to spend optimisation effort.

Monthly infrastructure cost breakdown at 200M daily active users. GPU infrastructure $52.4M at 89%, compute $4.6M, bandwidth $915k, monitoring $150k, database $138k, Redis $128k, load balancers $75k, CDN $75k, moderation $15k. Total approximately $58.6 million per month.
Caching 30% of prompts saves more than every database optimisation combined.

17. How those numbers were reached

Each fleet size traces back to the capacity estimate. Two assumptions carry most of the weight: 250 tokens per request, and 10 reads per message.

Sizing derivations for six cost lines: 17,500 H100 GPUs from token throughput and utilisation targets, 8,000 application servers from concurrent SSE connections, 20 read replicas from read throughput, 88 Redis nodes from peak operations per second, 19.4 PB of bandwidth from stream size and duration, and 10 T4 GPUs for moderation.
Change an assumption and the whole chain moves — which is the point of writing it out.

The three things worth remembering

  1. Count reads before you call something write-heavy. 2.5B prompts a day is ~90% reads once you count what each one loads. That single ratio is why a single primary survives 800M users.
  2. Stream because of latency, not throughput. Total generation time is unchanged. Time-to-first-token is everything.
  3. Optimise the 89% line. Prompt caching beats query tuning by three orders of magnitude here.

The diagrams

All 17 are generated .excalidraw scenes — Python scripts that emit the JSON, plus an exporter that produces the light and dark SVG on this page. Sources and generators are on GitHub: packages/system_design_how_to_design_chatgpt.

There is also a combined canvas that stacks all 17 into named Excalidraw frames, if you would rather pan around one file than open seventeen.

git clone https://github.com/henryonai/blog-code.henryonai.com
cd blog-code.henryonai.com/packages/system_design_how_to_design_chatgpt
for f in scripts/generate-*-excalidraw.py; do python3 "$f"; done
python3 scripts/build-combined-excalidraw.py

Based on "Design ChatGPT Like a Senior Engineer" by Hayk Simonyan. Scale figures are OpenAI's published numbers as of January 2026; the cost model is an estimate built on Azure list pricing, not OpenAI's actual bill.