Prompt Caching

Overview

Prompt caching lets providers reuse a previously processed prefix of your prompt (typically a long system message or repeated context) instead of re-processing the full input on every request. When a cache read hits, you pay a reduced rate for those tokens.

Boundless supports prompt caching on providers and models that expose it upstream — most notably OpenAI GPT and Claude models. You do not need a separate API flag in most cases; caching is handled by the upstream provider when your prompt structure qualifies.

Supported models

Caching availability follows the upstream provider. As of our current catalog, caching is most relevant for:

See the full list on the Models page. If a model does not support caching upstream, Boundless cannot enable it.

How Boundless handles caching

1. Pass-through. When you call POST /v1/chat/completions, Boundless routes to an upstream provider that supports the model. If that provider applies prompt caching, cache metadata is returned in the usage object and reflected in your bill.
2. Sticky routing. For cache hits to accumulate, Boundless keeps you on the same upstream provider for a given model where possible. Switching providers or model IDs between requests resets the cache.
3. No extra configuration. In most Claude and GPT flows, caching works automatically when your prompt has a cacheable prefix. You do not need a Boundless-specific header.

Thanks for this technical question. We support prompt caching on providers/models that expose it (usually OpenAI and Claude enable it automatically). We stick to the same provider to maximize cache hits and you can inspect that from the usage logs.

— Boundless support response pattern (beta onboarding)

Billing

When caching is active, usage may include separate token buckets beyond standard input/output:

Boundless rates

Cache-related charges follow the same discount rules as the model:

• GPT & Claude models — Boundless bills at 50% of official list rates (input, output, and applicable cache tiers)
• Other models — 25% off official where upstream cache pricing exists

Official upstream cache price tables are set by OpenAI and Anthropic. Boundless applies the same percentage discount as standard input/output. See Models for per-model rates.

Example mental model for claude-sonnet-4-6 (official input $3/1M, Boundless input $1.50/1M): a cache read on 10K tokens costs roughly half of what 10K uncached input tokens would cost — exact ratio depends on upstream cache-read pricing.

Usage logs

Inspect caching in Dashboard → Activity (or your usage export). Each row should show, when applicable:

• Model ID
• Input / output token counts
• Cache write and cache read token counts (if the provider reported them)
• USD cost for the request

What healthy caching looks like

If you only ever see cache writes and never reads across many requests, your prompt prefix is likely changing every call — see Troubleshooting.

Maximize cache hits

1. Stabilize the system prompt. Put static instructions in a fixed system message. Avoid embedding timestamps, random IDs, or per-request metadata in the cached prefix.
2. Keep tool schemas stable. Changing JSON tool definitions between agent steps invalidates the cache.
3. Reuse the same model ID within a session (e.g. stay on claude-sonnet-4-6, don't alternate models).
4. Stay on one provider route. Boundless routes consistently, but switching between different model families resets caching entirely.
5. Append, don't prepend. Add new user/assistant turns at the end. Rewriting the entire message array from scratch each time reduces hit rate.
6. Agent loops: the system prompt + tool definitions are the best cache candidates; user messages and tool outputs usually sit after the cached prefix.

Example: cache-friendly message structure

messages = [
  {"role": "system", "content": "<LONG STATIC SYSTEM PROMPT + TOOL RULES>"},
  {"role": "user", "content": "Step 1 user input"},
  # ... append new turns each step; do not mutate the system block
]

Troubleshooting

Many cache writes, zero cache reads

Common causes:

• System prompt changes every request (dynamic date/time, session ID in system block)
• Tool schema or function list changes between agent steps
• Model ID changes between requests
• Provider routing changed due to failover (rare — contact support if persistent)
• Prefix too short — upstream providers require a minimum cacheable length

Fix: Run two identical requests back-to-back with the same model and a long static system prompt. You should see cache writes on the first call and cache reads on the second. If not, email support@boundlessapi.com with request IDs from Activity.

Costs higher than expected despite caching

• Output tokens dominate agent workloads — caching only reduces input cost on the cached prefix
• First request in every session pays cache write cost
• Very short prompts may not qualify for caching upstream

How to report an issue

Include:

1. Model ID
2. Two request timestamps from Activity
3. Screenshot or export showing cache write/read columns
4. Whether the system prompt was identical between calls

FAQ

Do I need to enable caching in my API call?

Usually no for Claude and GPT — upstream providers enable it when the prompt qualifies. Boundless does not require a separate Boundless-specific parameter.

Does caching work with streaming?

Yes. Usage (including cache token breakdown when reported) is available on the final chunk or via Dashboard logs.

Does caching work across different API keys?

No. Caches are scoped to the upstream provider account route, not your Boundless key display name. Use one key per workload during testing.

Will failover break my cache?

Failover to a different upstream path can reset cache state. For maximum cache benefit, use stable models with low failover during a session.

Related

Questions? support@boundlessapi.com