Streaming in FastAPI: SSE vs WebSockets vs Polling

Every team building an LLM-backed feature hits the same decision point: the model streams tokens as they’re generated, and the frontend needs to show them appearing one at a time instead of waiting for the whole response. The reflexive answer is usually “WebSockets, because streaming” — and then three weeks later, someone’s debugging why the chat UI works perfectly on localhost and arrives in one giant chunk in production, because an nginx proxy or a corporate load balancer buffered the entire response before forwarding a single byte of it.

The actual decision isn’t “streaming vs not streaming” — it’s picking the right transport for a specific data flow, and WebSockets is frequently the wrong answer for what is, underneath the excitement about real-time UIs, a one-directional stream of text from server to client. Server-Sent Events (SSE) exist specifically for that shape of problem, ride on plain HTTP, and sidestep an entire category of infrastructure complexity WebSockets introduces — but SSE has its own sharp edges around proxy buffering that catch teams by surprise the first time they deploy behind anything more than a bare Uvicorn process. This post lays out when each of the three transports is actually correct, what breaks streaming in production infrastructure, and working FastAPI code for both SSE and WebSocket token streaming.

You’ll learn:

• Why polling is still the right choice for a meaningful slice of “real-time-ish” features, and when it stops being enough
• How Server-Sent Events work over plain HTTP, and why they fit LLM token streaming better than most teams initially assume
• When you genuinely need WebSockets — the specific bidirectional, low-latency cases SSE can’t cover
• The nginx buffering, gzip, and proxy configuration that silently breaks streaming responses in production
• Working FastAPI code for StreamingResponse-based SSE and for a WebSocket endpoint
• A decision framework you can apply directly instead of defaulting to whichever transport is trendiest

Table of Contents

1. The Basics
2. Polling: Still Sometimes Right
3. Server-Sent Events for Token-by-Token Output
4. WebSockets: When You Actually Need Bidirectional
5. Handling Disconnects and Mid-Stream Errors
6. Structuring Streaming Endpoints in a Larger App
7. The Decision Framework
8. Buffering and Proxy Pitfalls That Break Streaming
9. Code Examples
10. Common Pitfalls
11. Production Best Practices

The Basics

Three Transports, Three Different Shapes of Problem

Polling is the client repeatedly asking “anything new?” on a timer. Server-Sent Events is the server keeping one HTTP connection open and pushing text events down it as they happen, one direction only, server to client. WebSockets is a full-duplex connection where both sides can send messages at any time, independent of a request/response cycle at all.

The mistake most teams make is treating this as a difficulty ladder — polling is basic, SSE is better, WebSockets is best — when it’s actually a shape decision. If your data only ever flows one way (LLM tokens streaming to a chat UI, a progress bar, a live log tail), WebSockets solves a problem you don’t have while adding infrastructure you do have to maintain: connection state, reconnection logic, and a protocol that doesn’t fit as cleanly behind standard HTTP tooling.

Why This Decision Matters More With LLM Features Specifically

Token-by-token LLM output is, by its nature, one-directional and text-based — the exact shape SSE was designed for. It’s also usually not latency-critical in the sub-50ms sense WebSockets exist to serve (a token arriving 100ms later than theoretically possible is imperceptible against the multi-second span of full generation); it’s volume and reliability that matter — many tokens, arriving in order, over a connection that survives a brief network hiccup without the whole exchange failing.

Polling: Still Sometimes Right

Polling gets dismissed too quickly. For status that changes infrequently — a background job’s completion state, a document processing pipeline’s stage — a client polling every few seconds is simpler to build, simpler to debug, and works through every proxy, firewall, and CDN without special configuration, because it’s just repeated plain HTTP requests.

@router.get("/jobs/{job_id}/status")
async def get_job_status(job_id: str):
    job = await fetch_job(job_id)
    return {"status": job.status, "progress": job.progress}

const interval = setInterval(async () => {
  const res = await fetch(`/jobs/${jobId}/status`);
  const data = await res.json();
  updateProgressBar(data.progress);
  if (data.status === "complete") clearInterval(interval);
}, 2000);

Polling stops being enough once the update frequency needed is faster than it’s reasonable to poll for (sub-second), or once the volume of polling clients makes the wasted “nothing changed” requests a real cost — both are exactly the situation with token-by-token LLM output, where you’d need to poll multiple times per second per active user just to approximate what a stream gives you for free.

Server-Sent Events for Token-by-Token Output

SSE is plain HTTP with a specific response content type (text/event-stream) and a simple text format the browser’s native EventSource API (or a fetch + ReadableStream read loop) knows how to parse incrementally as bytes arrive, instead of waiting for the connection to close:

data: {"token": "The"}

data: {"token": " quick"}

data: {"token": " brown"}

Each data: line (blank line terminated) is one event. Because it’s just HTTP, it works through standard load balancers and CDNs with the right buffering configuration, doesn’t require a protocol upgrade the way WebSockets does, and the browser’s built-in EventSource API handles automatic reconnection with the last-event-id for you — a feature you’d otherwise have to hand-roll for WebSockets.

FastAPI serves SSE via StreamingResponse wrapping an async generator — each yield sends a chunk down the open connection immediately:

from fastapi.responses import StreamingResponse
import json

async def token_stream(prompt: str):
    async for chunk in llm_client.stream(prompt):
        yield f"data: {json.dumps({'token': chunk})}\n\n"
    yield "data: [DONE]\n\n"

@router.get("/chat/stream")
async def chat_stream(prompt: str):
    return StreamingResponse(
        token_stream(prompt),
        media_type="text/event-stream",
        headers={"X-Accel-Buffering": "no", "Cache-Control": "no-cache"},
    )

The generator’s async for cooperates naturally with the event loop — each yield is a natural suspension point, so a slow LLM provider doesn’t block other requests the way a synchronous blocking call would (the same event loop cooperation covered in Why Your FastAPI Endpoint Blocks the Event Loop). The X-Accel-Buffering: no header is a preview of the next section — without it, this entire endpoint can appear to work locally and then deliver its output as one giant chunk in production.

WebSockets: When You Actually Need Bidirectional

WebSockets earn their complexity when the client needs to send data back over the same open connection, at arbitrary times, without the overhead of a new HTTP request per message — a collaborative editor broadcasting keystrokes, a multiplayer game’s position updates, a chat feature where the client can interrupt or send follow-up messages mid-stream and the server needs to react immediately.

from fastapi import WebSocket, WebSocketDisconnect

@router.websocket("/ws/chat")
async def chat_ws(websocket: WebSocket):
    await websocket.accept()
    try:
        while True:
            prompt = await websocket.receive_text()
            async for chunk in llm_client.stream(prompt):
                await websocket.send_json({"token": chunk})
            await websocket.send_json({"done": True})
    except WebSocketDisconnect:
        pass

This looks similar to the SSE example, but the connection now genuinely supports the client sending a new prompt mid-session without opening a new HTTP request — that’s the actual capability being purchased, at the cost of needing your own reconnection logic (browsers don’t auto-reconnect WebSockets the way EventSource does), your own message framing if you need more structure than raw text/JSON, and infrastructure that correctly proxies the WebSocket upgrade handshake, which not every load balancer or CDN configuration supports out of the box.

For most “stream an LLM response to a chat UI” features, the client doesn’t actually need to send anything mid-stream — the next user message is a new request, not a message over the same open connection — which is precisely the case where SSE covers the requirement with meaningfully less infrastructure risk.

Handling Disconnects and Mid-Stream Errors

A streaming response has a longer failure surface than a normal request/response — the client can navigate away mid-stream, the LLM provider can error out after sending a partial response, and both need explicit handling or you end up with orphaned generation work or a stream that hangs instead of failing cleanly.

Detecting client disconnects. FastAPI’s Request object exposes is_disconnected(), which is worth checking inside a long-running generator so you stop doing (and paying for) work nobody is going to receive:

async def token_stream(request: Request, prompt: str):
    async for chunk in llm_client.stream(prompt):
        if await request.is_disconnected():
            break  # client navigated away — stop generating
        yield f"data: {json.dumps({'token': chunk})}\n\n"

Without this check, an abandoned browser tab doesn’t stop the underlying LLM call — you keep paying for and generating tokens nobody will ever see, which adds up quickly at any real traffic volume.

Surfacing mid-stream errors to the client. Once the first token has been sent, you can no longer change the HTTP status code — the response has already started. Errors that occur mid-generation need to be sent as a data event the frontend explicitly checks for, rather than relying on an HTTP error status:

async def token_stream(prompt: str):
    try:
        async for chunk in llm_client.stream(prompt):
            yield f"data: {json.dumps({'token': chunk})}\n\n"
    except LLMProviderError as exc:
        yield f"data: {json.dumps({'error': str(exc)})}\n\n"
    finally:
        yield "data: [DONE]\n\n"

The frontend then needs to check every incoming event for an error key rather than assuming every event is a token — a detail that’s easy to skip in a first implementation and only surfaces the first time the LLM provider actually fails mid-response in production.

Structuring Streaming Endpoints in a Larger App

Streaming routes tend to accumulate their own concerns — provider-specific retry logic, prompt construction, disconnect handling — that don’t belong mixed into a router alongside ordinary CRUD endpoints. In a domain-driven layout like the one covered in FastAPI Project Structure That Survives Growth, a chat or completions domain with its own router.py, service.py, and a dedicated streaming.py for the generator functions keeps this logic isolated and testable independently of the transport — the same token_stream generator can be driven by an SSE route, a WebSocket handler, or a test that just iterates it directly, without duplicating the LLM-calling logic three times across each entry point.

The Decision Framework

If the client never needs to send data mid-connection, start with SSE — it’s the shape LLM streaming actually is, and the operational cost is lower. Reach for WebSockets specifically when the interaction is genuinely two-way in real time, not because it sounds more sophisticated for a feature that’s actually one-directional.

Buffering and Proxy Pitfalls That Break Streaming

This is where SSE deployments actually go wrong, and it’s almost never in the FastAPI code — it’s in whatever sits between Uvicorn and the client.

nginx response buffering. By default, nginx buffers proxied responses before sending them to the client, which defeats streaming entirely — the client receives the whole response at once, whenever the LLM call finishes, not incrementally. The fix is disabling buffering for the streaming route specifically:

location /chat/stream {
    proxy_pass http://fastapi_upstream;
    proxy_buffering off;
    proxy_cache off;
    proxy_set_header Connection '';
    proxy_http_version 1.1;
    chunked_transfer_encoding off;
}

The X-Accel-Buffering: no response header shown in the FastAPI example above is nginx-specific and achieves the same effect without touching nginx config directly — useful when you don’t control the proxy config but do control the app’s response headers.

gzip/compression middleware. Response compression needs to buffer the full output to compress it effectively, which is directly at odds with incremental delivery. Exclude streaming routes from any GZipMiddleware or reverse-proxy compression configuration — compressing a stream of small SSE chunks individually also adds overhead with little benefit given how small each token typically is.

Load balancer idle timeouts. A managed load balancer (an AWS ALB, for instance) has a default idle timeout — often 60 seconds — that will close a connection with no new bytes sent for that long. A slow LLM response, or a pause between tokens longer than the timeout, silently kills the stream. Either send periodic SSE comment-lines (: keep-alive\n\n) to keep the connection active, or raise the load balancer’s idle timeout for streaming routes specifically.

Uvicorn worker count and long-held connections. Each open SSE or WebSocket connection occupies a worker’s attention for its entire duration. Running Uvicorn with too few workers relative to concurrent streaming connections means new, unrelated requests queue behind long-lived streams — size your worker count (or move to a process manager like Gunicorn with the Uvicorn worker class) with your expected concurrent stream count in mind, not just your average request rate.

Client-side buffering. Some browsers and HTTP clients buffer small response chunks before delivering them to JavaScript. Sending a minimum chunk size (some implementations pad early events) or explicitly flushing after each yield avoids the first few tokens appearing to “stick” before the stream visibly starts.

Code Examples

A complete SSE endpoint with keep-alive pings to survive load balancer idle timeouts, and a matching frontend consumer:

import asyncio
import json

async def token_stream(prompt: str):
    last_sent = asyncio.get_event_loop().time()
    async for chunk in llm_client.stream(prompt):
        yield f"data: {json.dumps({'token': chunk})}\n\n"
        last_sent = asyncio.get_event_loop().time()
    yield "data: [DONE]\n\n"

@router.get("/chat/stream")
async def chat_stream(prompt: str):
    return StreamingResponse(
        token_stream(prompt),
        media_type="text/event-stream",
        headers={
            "X-Accel-Buffering": "no",
            "Cache-Control": "no-cache",
            "Connection": "keep-alive",
        },
    )

const evtSource = new EventSource(`/chat/stream?prompt=${encodeURIComponent(prompt)}`);
evtSource.onmessage = (event) => {
  if (event.data === "[DONE]") { evtSource.close(); return; }
  const { token } = JSON.parse(event.data);
  appendToken(token);
};

For a genuinely bidirectional need — the client interrupting generation mid-stream — the WebSocket handler shown earlier extends naturally by checking for an incoming “stop” message concurrently with the generation loop, using asyncio.wait on both the receive coroutine and the generation coroutine so either can complete first.

Common Pitfalls

Mistake: reaching for WebSockets by default for LLM streaming. Most chat-style features are one-directional per turn and don’t need the added infrastructure. Solution: default to SSE unless the client genuinely needs to send data over the same open connection mid-stream.

Mistake: not disabling proxy buffering. This is the single most common cause of “streaming works locally, arrives as one chunk in production.” Solution: explicitly disable buffering (proxy_buffering off, X-Accel-Buffering: no) for every streaming route, and verify it with a real proxy in staging, not just against a bare Uvicorn process.

Mistake: forgetting keep-alives against load balancer idle timeouts. A stream that pauses longer than the LB’s idle timeout gets silently killed mid-response. Solution: send periodic comment-line pings, and know your load balancer’s idle timeout value explicitly rather than assuming it’s generous.

Mistake: compressing a streaming response. Compression middleware buffers to do its job effectively, defeating incremental delivery. Solution: exclude streaming routes from gzip/Brotli middleware entirely.

Mistake: under-provisioning workers for long-held connections. Each open stream occupies a worker for its full duration; too few workers means new requests queue behind active streams. Solution: size worker count against expected concurrent streaming connections, not average request throughput.

Production Best Practices

• Default to SSE for one-directional streaming, and reserve WebSockets for genuinely bidirectional, low-latency interaction.
• Test streaming behind your real production proxy stack in staging, not just against a bare Uvicorn process — buffering bugs are invisible until there’s a proxy in the path.
• Explicitly disable buffering and compression on streaming routes, both in the reverse proxy config and via response headers.
• Send keep-alive pings on long streams to survive load balancer and intermediate proxy idle timeouts.
• Size Uvicorn/Gunicorn worker count for concurrent open streams, not just requests per second — a long-held connection is a different capacity cost than a fast round trip.

Wrapping Up

The right streaming transport is determined by the actual shape of your data flow, not by which one sounds most impressive in an architecture doc — and for the overwhelming majority of LLM token-streaming features, that shape is one-directional, which is exactly what SSE was built for at a fraction of the infrastructure cost of WebSockets. The code that streams tokens correctly is usually the easy part; the proxy and load balancer configuration standing between your FastAPI app and the browser is where production streaming actually breaks, so test it there before you trust it.

Has your streaming endpoint actually been tested behind your production proxy, or only against a bare Uvicorn dev server?