Prompt Caching
Prompt caching lets a model service reuse work for the same beginning of an input. It may reduce repeat processing, but it does not reuse the old answer.
stable prefix → cache write/read → usage4,096 stable input tokensA support app sends the same rules and tool list before each new question. The service reuses that prefix; a changed rule starts new work.prefix_42stable | variablewrite → readstable input prefix → cache write/read → usage recordOPSAnother example
A support app sends the same rules and tool list before each new question. The service reuses that prefix; a changed rule starts new work.
Main parts
- 01Reusable input prefix
- 02Match or breakpoint rule
- 03Cache write
- 04Cache hit or miss
- 05Lifetime and invalidation
- 06Usage, latency, and cost
Use it when
Use it when many requests share a long, stable prefix and the chosen model service supports caching.
Do not use it when
Do not put private user data in a shared prefix. Do not confuse prompt caching with saved answers or HTTP caching.
Name used in code
stable input prefix → cache write/read → usage record Before you ship
Check the model and prompt versions, source data, versioned evaluation set, and measures. Verify links to evidence, tool permissions, privacy, common failures, decline-to-answer and fallback behavior, human review, monitoring, cost, speed, and rollback.
Outcome: Use or evaluate Prompt Caching to make AI behavior measurable and tied to evidence. User context: A support app sends the same rules and tool list before each new question. The service reuses that prefix; a changed rule starts new work. AI method or concept: Prompt Caching. Why it fits: Use it when many requests share a long, stable prefix and the chosen model service supports caching. Do not use it when: Do not put private user data in a shared prefix. Do not confuse prompt caching with saved answers or HTTP caching. AI requirements: Define the input and source evidence. Set model and tool permissions. Use versioned evaluation data and measures. Define failure, decline-to-answer, privacy, speed, and cost limits. Operational safeguards: Keep a clear trace and hide sensitive log data. Show users a safe fallback. Mark steps that need human review. Define how to roll back the model or prompt. Acceptance criteria: Record baseline and target measures on a versioned evaluation set. Test edge cases and hostile inputs. Verify fallback, monitoring, permissions, and rollback. Evidence and limits (evidence boundary): It is official for that source. Other platforms or teams may use the term in another way. Unknowns to confirm: Target task, model and version, evaluation owner, source data, risk limit, tool permissions, and production fallback.
Check this request
Review the current use of Prompt Caching. Definition: Prompt caching lets a model service reuse work for the same beginning of an input. It may reduce repeat processing, but it does not reuse the old answer. Release checks: Check the model and prompt versions, source data, versioned evaluation set, and measures. Verify links to evidence, tool permissions, privacy, common failures, decline-to-answer and fallback behavior, human review, monitoring, cost, speed, and rollback. Before changing code, report the evidence you found, gaps, severity, and the smallest safe fix.