LLM Inference #8: Chunked Prefill
Three users are streaming answers. A fourth sends a long prompt, and their streams pause. How chunked prefill shares each iteration, what the token budget buys, and how vLLM implements it.
Three users are streaming answers. A fourth sends a long prompt, and their streams pause. How chunked prefill shares each iteration, what the token budget buys, and how vLLM implements it.
Every request sends the same 4,000-token system prompt, and the GPU prefills it every time. How block hashing turns the KV free list into a cache, what it will not match, and why one salted block isolates a whole tenant.
An LLM server stops admitting requests while a third of its KV memory looks free. Where that memory goes, how block tables get it back, and three places where vLLM no longer matches the PagedAttention paper.
Why batching makes LLM decoding cheaper, why static batching wastes most of the gain, and how continuous batching fixes it, explained with diagrams instead of code.
Understand tensor, pipeline, and expert parallelism through small worked examples, GPU placement diagrams, and a clear explanation of what each GPU computes and communicates.
A visual introduction to Mixture of Experts: follow a token through routing, blend expert outputs, and understand the compute, memory, and communication tradeoffs.
How CPUs and GPUs cooperate, how grids, blocks, warps, and tiles execute, and why memory movement matters for LLM inference.
You send a prompt to an LLM endpoint. The server spends a moment working without returning text, then words begin to stream. The first word can take much longer than the gaps between later words. I...
Part 2 of the Lucene internals series. Part 1 — Deep Dive into Apache Lucene’s Core: From In-Memory Buffers to Commits followed documents into an index. This one follows a query out of it. Part 1 ...
Apache Lucene is a highly optimized, full-featured search engine library written in Java. While developers frequently interact with its high-level objects like IndexWriter and Document, what happen...