Prompt 缓存

概述

Prompt caching lets providers reuse a previously processed prefix of your prompt (typically a long system message or repeated context) instead of re-processing the full input on every request. When a cache read hits, you pay a reduced rate for those tokens.

Boundless supports prompt caching on providers and models that expose it upstream — most notably OpenAI GPT and Claude models. You do not need a separate API flag in most cases; caching is handled by the upstream provider when your prompt structure qualifies.

支持的模型

Caching availability follows the upstream provider. As of our current catalog, caching is most relevant for:

See the full list on the Models page. If a model does not support caching upstream, Boundless cannot enable it.

Boundless 如何处理缓存

1. Pass-through. When you call POST /v1/chat/completions, Boundless routes to an upstream provider that supports the model. If that provider applies prompt caching, cache metadata is returned in the usage object and reflected in your bill.
2. Sticky routing. For cache hits to accumulate, Boundless keeps you on the same upstream provider for a given model where possible. Switching providers or model IDs between requests resets the cache.
3. No extra configuration. In most Claude and GPT flows, caching works automatically when your prompt has a cacheable prefix. You do not need a Boundless-specific header.

感谢这个技术问题。我们在上游支持 prompt caching 的模型上透传该能力（通常 OpenAI 和 Claude 会自动启用）。为最大化 cache hit，我们会尽量保持在同一上游 Provider 路由；你可以在用量日志里查看 cache write / cache read。

— Boundless 技术支持说明（beta onboarding）

计费

When caching is active, usage may include separate token buckets beyond standard input/output:

Boundless rates

Cache-related charges follow the same discount rules as the model:

• GPT & Claude models — Boundless bills at 50% of official list rates (input, output, and applicable cache tiers)
• Other models — 25% off official where upstream cache pricing exists

Official upstream cache price tables are set by OpenAI and Anthropic. Boundless applies the same percentage discount as standard input/output. See Models for per-model rates.

Example mental model for claude-sonnet-4-6 (official input $3/1M, Boundless input $1.50/1M): a cache read on 10K tokens costs roughly half of what 10K uncached input tokens would cost — exact ratio depends on upstream cache-read pricing.

用量日志

Inspect caching in Dashboard → Activity (or your usage export). Each row should show, when applicable:

• Model ID
• Input / output token counts
• Cache write and cache read token counts (if the provider reported them)
• USD cost for the request

What healthy caching looks like

If you only ever see cache writes and never reads across many requests, your prompt prefix is likely changing every call — see 故障排查.

提升缓存命中率

1. Stabilize the system prompt. Put static instructions in a fixed system message. Avoid embedding timestamps, random IDs, or per-request metadata in the cached prefix.
2. Keep tool schemas stable. Changing JSON tool definitions between agent steps invalidates the cache.
3. Reuse the same model ID within a session (e.g. stay on claude-sonnet-4-6, don't alternate models).
4. Stay on one provider route. Boundless routes consistently, but switching between different model families resets caching entirely.
5. Append, don't prepend. Add new user/assistant turns at the end. Rewriting the entire message array from scratch each time reduces hit rate.
6. Agent loops: the system prompt + tool definitions are the best cache candidates; user messages and tool outputs usually sit after the cached prefix.

Example: cache-friendly message structure

messages = [
  {"role": "system", "content": "<LONG STATIC SYSTEM PROMPT + TOOL RULES>"},
  {"role": "user", "content": "Step 1 user input"},
  # ... append new turns each step; do not mutate the system block
]

故障排查

Many cache writes, zero cache reads

Common causes:

• System prompt changes every request (dynamic date/time, session ID in system block)
• Tool schema or function list changes between agent steps
• Model ID changes between requests
• Provider routing changed due to failover (rare — contact support if persistent)
• Prefix too short — upstream providers require a minimum cacheable length

Fix: Run two identical requests back-to-back with the same model and a long static system prompt. You should see cache writes on the first call and cache reads on the second. If not, email support@boundlessapi.com with request IDs from Activity.

Costs higher than expected despite caching

• Output tokens dominate agent workloads — caching only reduces input cost on the cached prefix
• First request in every session pays cache write cost
• Very short prompts may not qualify for caching upstream

How to report an issue

Include:

1. Model ID
2. Two request timestamps from Activity
3. Screenshot or export showing cache write/read columns
4. Whether the system prompt was identical between calls

FAQ

Do I need to enable caching in my API call?

Usually no for Claude and GPT — upstream providers enable it when the prompt qualifies. Boundless does not require a separate Boundless-specific parameter.

Does caching work with streaming?

Yes. Usage (including cache token breakdown when reported) is available on the final chunk or via Dashboard logs.

Does caching work across different API keys?

No. Caches are scoped to the upstream provider account route, not your Boundless key display name. Use one key per workload during testing.

Will failover break my cache?

Failover to a different upstream path can reset cache state. For maximum cache benefit, use stable models with low failover during a session.

相关文档

Questions? support@boundlessapi.com