Prefix Cache vs KV Cache in LLM Inference

Problem

KV cache and prefix caching are often discussed together, leading to the mistaken expectation that enabling prefix reuse reduces every phase of generation.

Conclusion

KV cache stores previously computed key and value state. Prefix caching reuses eligible cached state when a new request shares a previously processed prefix. These describe related but different responsibilities.

Environment

Record the engine and version, model revision, prompt tokenization, prefix-caching settings, cache state and workload order. Use both repeated-prefix and unrelated-prefix requests.

Symptoms

Repeated long prefixes may benefit while unrelated prompts do not. A warm-cache test and a cold-cache test can produce different results despite using the same visible request text.

Cause

Reuse avoids recomputing eligible shared-prefix state. It does not eliminate generation of new output tokens. Cache capacity, eviction and exact prefix matching affect observed behavior.

Solution

Design two workload groups. One reuses a stable system prompt or document prefix; the other uses distinct prefixes. Keep output-length controls and arrival conditions consistent.

# Workload sketch: no performance measurements are implied.
shared_prefix = "A stable document or system prompt\n"
questions = ["Summarize the constraints.", "List the assumptions."]
requests = [shared_prefix + question for question in questions]

Measure the first cold request separately from subsequent warm requests. Capture TTFT and decode metrics independently. Repeat with a working set larger than the retained cache to understand eviction behavior.

Verification

Benchmark data pending real-world testing.

A useful report states whether cache state was warm or cold, which prefixes were repeated and which phase improved. Do not attribute every throughput change to prefix reuse.

Caveats

Exact cache matching rules and isolation behavior belong to the serving engine. Applications with multiple tenants should review the engine's cache-isolation controls and their security requirements.

References

• vLLM automatic prefix caching
• vLLM prefix caching design