Prompt Caching Guide

Reduce latency and cost with prompt caching.

Table of Contents

• Overview
• Structuring Prompts for Caching
• How it Works
• GPT-5.6 Cache Writes and Breakpoints
• Prompt Cache Routing and Retention
• Responses and Chat Completions Patterns
• Requirements
• Measure Cache Performance
• What Can Be Cached
• Best Practices
• Cache Miss Troubleshooting
• Gemini Context Caching
  • Implicit vs Explicit Caching
  • AvalAI Support Status
  • Implicit Caching Details
  • Checking Cache Hits in Gemini Responses
  • Explicit Caching (Not Supported)
• Frequently Asked Questions
• Related Resources

Overview

Model prompts often contain repetitive content, like system prompts and common instructions. Underlying model providers (like OpenAI) may route API requests to servers that recently processed the same prompt, making it potentially cheaper and faster than processing a prompt from scratch. OpenAI documents Prompt Caching as an automatic optimization that can reduce latency by up to 80% and input token cost by up to 90% on eligible traffic; through AvalAI, treat those figures as provider/model dependent and verify the actual usage fields for your route. Cache writes have no additional fee on OpenAI models before the GPT-5.6 family. GPT-5.6 cache writes are billed separately at 1.25 times the uncached input rate.

Note: Availability and specific discounts depend on the underlying model provider and the specific model used. Please refer to the model provider's documentation for exact details.

On AvalAI, prompt caching can apply to OpenAI-compatible models that expose provider-side caching behavior, including gpt-5.6-sol, gpt-5.6-terra, gpt-5.6-luna, gpt-5.5, gpt-5.4, gpt-5.4-pro, gpt-5.2, gpt-5.1, gpt-5-mini, gpt-4.1, gpt-4o, o3, and o4-mini. Availability, request controls, retention behavior, and discounts still depend on the underlying provider and route; check the response usage fields for the request you are actually sending.

This guide describes how prompt caching generally works, so that you can optimize your prompts for potentially lower latency and cost.

Structuring Prompts for Caching

Cache hits are typically only possible for exact prefix matches within a prompt. To maximize potential caching benefits, place static content like instructions and examples at the beginning of your prompt, and put variable content, such as user-specific information, at the end. This principle also applies to images and tools, which usually must be identical between requests for the prefix to match.

(Image Source: OpenAI)

How it Works

Caching might be enabled automatically by the model provider for prompts exceeding a certain token length (for OpenAI, 1024 tokens or more). When you make an API request via AvalAI to such a model:

1. Cache Routing: The provider routes the request based on a hash of the prompt prefix. OpenAI documents that this prefix hash commonly starts from the first 256 tokens, though the exact length can vary by model. When supported, prompt_cache_key is combined with that prefix hash to improve cache locality for repeated traffic.
2. Cache Lookup: The provider checks whether the initial portion (prefix) of your prompt already exists in cache on the selected machine.
3. Cache Hit: If a matching prefix is found, the provider reuses the cached prefix. This can significantly decrease latency and reduce costs for cached input tokens.
4. Cache Miss: If no matching prefix is found, the provider processes the full prompt and may cache the prefix for future requests.

Cached prefixes generally remain active for a period of inactivity, though this varies by provider and retention policy. OpenAI's in-memory cache is typically active for 5 to 10 minutes of inactivity, with an upper bound around one hour; AvalAI routes can differ when the underlying provider is not OpenAI.

GPT-5.6 Cache Writes and Upstream Controls

GPT-5.6 changes the upstream cost and control model. OpenAI bills prompt tokens written to cache at 1.25 times the uncached input rate, reports writes in cache_write_tokens, and reports reads in cached_tokens. Implicit caching remains the supported AvalAI behavior.

OpenAI defines the following controls for direct upstream use of GPT-5.6 and later model families. AvalAI does not currently support explicit caching or explicit cache breakpoints, so treat these fields as upstream reference only and do not rely on them through AvalAI:

• prompt_cache_options.mode: "implicit" keeps the automatic breakpoint on the latest message and also uses explicit breakpoints.
• prompt_cache_options.mode: "explicit" disables the automatic breakpoint. With no explicit breakpoint, the request does not read or write prompt cache entries.
• prompt_cache_options.ttl: "30m" sets the minimum cache lifetime. 30m is currently the only supported value and the default.
• prompt_cache_breakpoint: {"mode": "explicit"} marks the exact end of a reusable prefix on a supported content block.

AvalAI's model catalog confirms prompt-caching capability and cache-write pricing for its GPT-5.6 routes, but not explicit cache control. Omit prompt_cache_options, prompt_cache_breakpoint, and provider-specific explicit cache markers. Use automatic implicit caching and verify the response usage fields.

This Responses request shape uses explicit-only caching. The stable content before the marker must render to at least 1,024 tokens before it is eligible for caching:

{
  "model": "gpt-5.6-luna",
  "prompt_cache_key": "tenant-acme-support-policy-v3",
  "prompt_cache_options": {
    "mode": "explicit",
    "ttl": "30m"
  },
  "input": [
    {
      "role": "developer",
      "content": [
        {
          "type": "input_text",
          "text": "Long, stable support policy and examples...",
          "prompt_cache_breakpoint": {
            "mode": "explicit"
          }
        }
      ]
    },
    {
      "role": "user",
      "content": "Draft a reply for the current ticket."
    }
  ]
}

The equivalent Chat Completions shape puts the marker on a supported content block:

{
  "model": "gpt-5.6-luna",
  "prompt_cache_key": "tenant-acme-support-policy-v3",
  "prompt_cache_options": {
    "mode": "explicit",
    "ttl": "30m"
  },
  "messages": [
    {
      "role": "system",
      "content": [
        {
          "type": "text",
          "text": "Long, stable support policy and examples...",
          "prompt_cache_breakpoint": {
            "mode": "explicit"
          }
        }
      ]
    },
    {
      "role": "user",
      "content": "Draft a reply for the current ticket."
    }
  ]
}

Each request can create up to four new cache writes. In implicit mode the automatic latest-message breakpoint consumes one write slot, leaving up to three new explicit writes; explicit mode can write the latest four explicit breakpoints. OpenAI currently considers up to the latest 50 breakpoints for reads and uses the longest matching prefix. Responses supports markers on input_text, input_image, and input_file; Chat Completions supports text, image_url, input_audio, file, and refusal blocks. Older models reject the new controls and should keep their existing automatic caching behavior.

Prompt Cache Routing and Retention

AvalAI is an aggregator: one public model ID may be served by several providers or infrastructure pools. Previously, two matching requests could land on different infrastructure, which could produce fewer provider-side implicit or in-memory cache hits than calling one provider directly.

The smart router now tracks successful routing at a per-user, per-model, and per-infrastructure level. It prefers the last-known-good infrastructure that successfully handled that user's request for the model. This sticky affinity rolls forward for 15 minutes after the latest successful request; every success renews the 15-minute window. This has drastically improved cache locality, but affinity is best effort, not a guarantee: health, capacity, failover, model availability, and provider routing can still move a request. The sticky window is a routing preference and does not extend or guarantee the provider's own cache lifetime.

AvalAI also needs time to propagate or synchronize user cache keys and model/infrastructure affinity. Routing was close to immediate for deepseek-v4-flash: a 2026-08-13 paired benchmark reached the same warm cache-hit ratio as the official DeepSeek endpoint after a 15-second post-prime wait. Propagation is model- and infrastructure-dependent, however. An initial deepseek-v4-pro run remained at 0.0% after that wait, while a subsequent run shortly afterward reached 99.9% versus DeepSeek's 98.6%. Treat 15 seconds as one observed result, not a universal activation deadline. Your application must remain correct on a cache miss and must not use cache affinity as conversation state.

For repeated OpenAI-compatible traffic through AvalAI, keep a stable prefix and, where supported, send a stable prompt_cache_key for a workload, tenant, or assistant configuration. Avoid using a key that groups too much traffic: OpenAI documents that if the same prefix and cache key exceed roughly 15 requests per minute, some requests may overflow to additional machines and reduce cache effectiveness.

Treat prompt_cache_key as a routing label, not user data. Prefer opaque buckets such as support-policy-v3, tenant-acme-chat-v2, or invoice-extractor-schema-v1. Do not put raw emails, phone numbers, user IDs, ticket IDs, API keys, or secrets in the key; place per-request identifiers near the end of the prompt instead so they do not break the shared prefix.

Retention controls differ by generation. For GPT-5.6 and later families, use prompt_cache_options.ttl; prompt_cache_retention is deprecated for those models. For pre-GPT-5.6 models, prompt_cache_retention remains the maximum-retention policy where supported. The following gpt-5.5 example is intentionally a legacy retention example.

import os
from openai import OpenAI

client = OpenAI(
    api_key=os.environ["AVALAI_API_KEY"],
    base_url="https://api.avalai.ir/v1",
)

response = client.responses.create(
    model="gpt-5.5",
    instructions="You are the BrightCart support assistant. Follow the fixed policy below...",
    input="Draft a two-sentence reply for ticket ORDER-8831.",
    prompt_cache_key="brightcart-support-policy-v1",
    prompt_cache_retention="24h",
)

cached = response.usage.input_tokens_details.cached_tokens
print(f"cached prompt tokens: {cached}")
print(response.output_text)

Use prompt_cache_retention="24h" only with models and providers that support extended retention. OpenAI documents short in-memory retention for many cacheable models (typically 5 to 10 minutes of inactivity, with an upper bound around one hour) and extended retention up to 24 hours on supported models. OpenAI also documents that prompt-cache pricing is the same for in-memory and extended retention; through AvalAI, still verify the selected route/model price and usage fields. For providers or models that do not expose this parameter through AvalAI, omit it and rely on automatic in-memory caching.

Extended retention is still a performance feature, not application state. OpenAI documents that extended caching may persist key/value tensors derived from customer content in GPU-local storage, not the final response; most usage expires after 1 to 2 hours and the maximum retention is 24 hours. The exact privacy, residency, and retention behavior remains provider- and route-dependent when you call through AvalAI.

OpenAI currently lists extended prompt-cache retention for these model families: gpt-5.5, gpt-5.5-pro, gpt-5.4, gpt-5.2, gpt-5.1-codex-max, gpt-5.1, gpt-5.1-codex, gpt-5.1-codex-mini, gpt-5.1-chat-latest, gpt-5, gpt-5-codex, and gpt-4.1. Treat this as an upstream capability list, not an AvalAI guarantee; AvalAI support depends on the exact route, account, and model alias you call.

Use this retention checklist before setting the parameter:

• For pre-GPT-5.6 models such as gpt-5.5 that expose extended caching through AvalAI, use prompt_cache_retention="24h" only when AvalAI passes the parameter through. Do not infer support for GPT-5.6 or later, which use prompt_cache_options.ttl instead.
• For older OpenAI models that support both in_memory and 24h, choose deliberately based on your data-retention requirements instead of relying on defaults, because defaults can differ when Zero Data Retention is enabled.
• For Zero Data Retention, residency, or regulated workloads, prefer omitting retention parameters until you have verified the exact provider route and policy.
• Never use prompt caching as a substitute for storing conversation state or business records in your own application database.

Responses and Chat Completions Patterns

For new AvalAI apps, prefer the Responses API because it aligns with stateful, multimodal, tool-using workflows. For existing v1/chat/completions apps, keep the route when migration would be risky; the same caching principles apply when the long system/developer prefix, tool schemas, image order, and structured output schema stay stable.

When migrating a cached Chat Completions workload to v1/responses, preserve the stable prefix and only change the endpoint shape:

• Move long, shared system instructions into instructions or the first stable input item.
• Keep dynamic user text, retrieved snippets, timestamps, and request IDs near the end.
• Reuse the same prompt_cache_key bucket for the same assistant, tenant, policy, or schema.
• Inspect usage.input_tokens_details.cached_tokens for Responses and usage.prompt_tokens_details.cached_tokens for Chat Completions; cache hits are optimization evidence, not correctness evidence.

# Responses API — recommended for new AvalAI work
response = client.responses.create(
    model="gpt-5.5",
    instructions="You are the BrightCart support assistant. Follow the fixed policy below...",
    input="Draft a two-sentence reply for ticket ORDER-8831.",
    prompt_cache_key="brightcart-support-policy-v1",
    prompt_cache_retention="24h",
)
print(response.usage.input_tokens_details.cached_tokens)

# Chat Completions API — keep for existing apps
chat_response = client.chat.completions.create(
    model="gpt-5.5",
    messages=[
        {
            "role": "system",
            "content": "You are the BrightCart support assistant. Follow the fixed policy below...",
        },
        {
            "role": "user",
            "content": "Draft a two-sentence reply for ticket ORDER-8831.",
        },
    ],
    prompt_cache_key="brightcart-support-policy-v1",
    prompt_cache_retention="24h",
)
print(chat_response.usage.prompt_tokens_details.cached_tokens)

Requirements

Caching availability and behavior depend on the provider and model. Requests below a minimum are processed normally without an error, but no cache read occurs.

Provider or model family	Minimum input tokens
OpenAI	1,024
Anthropic Claude 3.x	1,024
Anthropic Claude Sonnet/Opus 4.x	2,048
Anthropic Claude Haiku 4.5+ and Opus 4.5+	4,096
Bedrock Claude 3.5/3.7	1,024
Bedrock Claude Sonnet 4.x	2,048
Google Gemini implicit caching	1,024

Anthropic-family thresholds vary by exact release and route. Current platform-specific examples include 512 tokens for Claude Opus 5, Fable 5, and Mythos 5; 1,024 for Claude Opus 4.8, Sonnet 5, Sonnet 4.6/4.5/4, Opus 4.1/4; 2,048 for Claude Mythos Preview, Opus 4.7, and Haiku 3.5; and 4,096 for Claude Opus 4.6/4.5 and Haiku 4.5. Check the selected route's current model documentation rather than inferring one threshold for every alias.

Some API responses include details about cached tokens in the usage object:

{
  "usage": {
    "prompt_tokens": 2006,
    "completion_tokens": 300,
    "total_tokens": 2306,
    "prompt_tokens_details": {
      "cached_tokens": 1920,
      "cache_write_tokens": 0
    },
    "completion_tokens_details": {
      "reasoning_tokens": 0
    }
  }
}

In this example, 1920 prompt tokens were read from cache and no prompt tokens were written. cache_write_tokens is returned for GPT-5.6 and later model families when the selected route exposes the field.

For Responses API calls, the same idea appears under usage.input_tokens_details.cached_tokens. For Chat Completions, inspect usage.prompt_tokens_details.cached_tokens on the chat response. Provider-native APIs, such as Gemini, may use different names.

Cache-aware routing benchmark

On 2026-08-13, a paired streaming benchmark sent one byte-identical generated prefix and the same generation seed to AvalAI and the official DeepSeek endpoint. It primed both routes, waited 15 seconds, then ran 10 sequential rounds with deepseek-v4-flash:

python -m tests.benchmarks.test_cache_hit_ratio \
  --model deepseek-v4-flash \
  --rounds 10 \
  --prefix-tokens 2000

Metric	AvalAI	Official DeepSeek
Round 1	0/1,948 cached (0.0%)	1,920/1,948 cached (98.6%)
Warm mean, rounds 2–10	98.6%	98.6%
Warm-cache delta	+0.0 percentage points	Baseline
Successful requests	10/10	10/10

AvalAI first recorded a hit in round 2, 2.683 seconds into the measured rounds. DeepSeek unexpectedly reported a hit in round 1 despite the random unique prefix, so its first response is not evidence of a clean official cold miss. The defensible comparison is warm rounds 2–10, where both endpoints returned 1,920 cached tokens from each 1,948-token prompt.

This single-model run shows near-immediate routing progress after the measured 15-second wait; it does not prove zero-delay synchronization or guarantee a 98.6% hit rate. A separate deepseek-v4-pro test initially produced 0.0% AvalAI warm hits, then a later run reached 99.9%, showing that initial affinity can take longer for some models or infrastructure. Once established, successful routes renew the rolling 15-minute sticky preference. Results vary by exact prompt, model, provider, load, and routing state. See the full methodology, charts, latency/TTFT results, and limitations on AvalAI Performance.

Measure Cache Performance

Treat prompt caching as an observable optimization. Add lightweight metrics before and after you restructure prompts:

cache_hit_ratio = cached_tokens / input_or_prompt_tokens
uncached_tokens = input_or_prompt_tokens - cached_tokens

Track these fields by route, model, prompt_cache_key, and stable prompt version:

• input/prompt tokens, cached_tokens, cache_write_tokens when returned, output tokens, total tokens, latency p50/p95, and request count.
• Cache hit ratio for the second and later request in a repeated workload; the first request is usually a warm-up miss.
• Error or fallback rates when prompt_cache_retention is rejected by a route that does not support extended retention.

For GPT-5.6 traffic, calculate read and write cost separately. A cache write is worthwhile only when later discounted reads recover its higher write cost. Compare cache_write_tokens across warm-up requests with cached_tokens across later requests instead of treating every nonzero cache field as savings.

Use the metrics to decide whether to split or rotate cache buckets. A shared prompt_cache_key can improve locality for common prefixes, but if one key mixes too much traffic, provider routing may overflow and reduce hit rates. Keep correctness checks separate: cached tokens can reduce cost and latency, but the full prompt still counts toward rate limits and the response is generated fresh each time.

What Can Be Cached

The following content can contribute to a cacheable exact prefix when the provider supports it:

• Tool definitions and their stable order.
• System messages and long policy text.
• Text blocks in messages.
• User images and documents, with identical bytes, order, and detail settings.
• Tool-use and tool-result blocks from previous turns.
• Previous assistant thinking blocks only when replayed alongside cacheable prior-turn content; thinking blocks cannot be directly marked with cache_control.

Citation sub-blocks cannot be cached directly; cache the top-level document block. Empty text blocks are not cacheable. Explicit cache_control markers remain unsupported on AvalAI.

Invalidation hierarchy

Cache invalidation follows the prompt hierarchy tools → system → messages:

• Changing tool definitions invalidates tools, system, and messages.
• Web-search, citation, or speed-mode changes can invalidate system and messages.
• Changes to tool_choice, images, and many thinking/effort controls invalidate messages.
• Thinking-related invalidation of tools or system content can be model-specific.

Best Practices

• Structure prompts with static content first, dynamic content last.
• Monitor API response usage details for cached_tokens, latency, and cache-hit rate.
• Use prompt_cache_key consistently for repeated prefixes, but choose a granularity that stays below provider routing limits.
• Keep tool definitions, image order, structured output schemas, and long policy text stable between requests.
• Put timestamps, user IDs, retrieved snippets, and per-request facts near the end so they do not break the shared prefix.
• Treat caching as an optimization, not a correctness feature; your application should work even when a request is a cache miss.

Cache Miss Troubleshooting

If cached_tokens stays at 0 for traffic that should repeat, check these causes before changing models:

Symptom	Likely cause	Fix
Short prompts never cache	The request is below the provider's minimum cacheable length	Combine stable instructions, schemas, or reference context until the repeated prefix is long enough to cache.
Long prompts miss after every request	Dynamic values appear too early	Move user-specific text, timestamps, retrieved snippets, and request IDs after the stable prefix.
Tool-heavy requests miss unexpectedly	Tool schemas or ordering changed	Keep tool names, descriptions, schemas, and order stable between requests.
Image requests miss	Image order, URL/base64 bytes, or detail changed	Reuse the same image representation and detail value for the shared prefix.
Cache hit rate drops under load	One prompt_cache_key groups too much traffic	Split keys by assistant, tenant, or workload so each shared prefix/key pair remains reasonably bounded.
Only one provider misses	The selected route/model does not expose caching details through AvalAI	Confirm the provider page, omit unsupported retention parameters, and rely on normal request behavior.
AvalAI hit rate remains materially below direct-provider use	Affinity is still propagating, failover moved the request, or the provider uses different cache behavior	Retry after the propagation window, capture model, request timestamps and usage, then Create a support ticket.

For Anthropic-style responses, inspect both cache_creation_input_tokens and cache_read_input_tokens. If both are zero, caching did not occur for that request.

Gemini Context Caching

Gemini models have their own context caching mechanism with two distinct types: Implicit Caching and Explicit Caching.

Implicit vs Explicit Caching

Feature	Implicit Caching	Explicit Caching
Activation	Automatic	Manual (developer-controlled)
Cost Savings	Not guaranteed	Guaranteed
Setup Required	None	Create/manage cached content
TTL Control	No	Yes (configurable)
AvalAI Support	⚠️ Possible but not guaranteed	❌ Not currently supported

AvalAI Support Status

Important

AvalAI's smart router now tracks the last-known-good route per user, model, and infrastructure for 15 minutes after the latest successful request. Every success renews the sticky window. This drastically improves locality but remains best effort; failover or capacity can still move a request.

Caching Type	Status	Notes
Implicit Caching	⚠️ Best effort	15-minute rolling per-user, per-model, per-infrastructure preference
Explicit Caching	❌ Not supported	May be added in future updates

Implicit Caching Details

Implicit caching is enabled by default on Gemini models. When your request hits a cache, Google automatically passes on cost savings.

Minimum Token Requirements for Gemini

Model	Minimum Token Count
Gemini 3 Flash Preview	1,024 tokens
Gemini 3 Pro Preview	4,096 tokens
Gemini 2.5 Flash	1,024 tokens
Gemini 2.5 Pro	4,096 tokens

Tips to Increase Cache Hit Probability

1. Place large, static content at the beginning of your prompt

   • System instructions
   • Reference documents
   • Few-shot examples

2. Send similar requests in quick succession

   • Requests with the same prefix sent close together have higher cache hit probability

3. Keep dynamic content at the end

   • User-specific information
   • Variable queries
   • Timestamps and unique identifiers

Checking Cache Hits in Gemini Responses

When using the native Gemini API, you can check for cache hits in the response:

Python

import os
from google import genai

client = genai.Client(
    api_key=os.environ["AVALAI_API_KEY"],
    http_options={"base_url": "https://api.avalai.ir"},
)

response = client.models.generate_content(
    model="gemini-2.5-flash", contents="Your prompt here with substantial context..."
)

# Check usage metadata for cache information
if hasattr(response, "usage_metadata"):
    usage = response.usage_metadata
    print(f"Prompt tokens: {usage.prompt_token_count}")
    print(f"Cached tokens: {getattr(usage, 'cached_content_token_count', 0)}")
    print(f"Output tokens: {usage.candidates_token_count}")

JavaScript

import { GoogleGenAI } from "@google/genai";

const client = new GoogleGenAI({
  apiKey: process.env.AVALAI_API_KEY,
  httpOptions: { baseUrl: "https://api.avalai.ir" }
});

const response = await client.models.generateContent({
  model: "gemini-2.5-flash",
  contents: "Your prompt here with substantial context..."
});

// Check usage metadata for cache information
if (response.usageMetadata) {
  console.log(`Prompt tokens: ${response.usageMetadata.promptTokenCount}`);
  console.log(`Cached tokens: ${response.usageMetadata.cachedContentTokenCount || 0}`);
  console.log(`Output tokens: ${response.usageMetadata.candidatesTokenCount}`);
}

cURL

curl -X POST "https://api.avalai.ir/v1beta/models/gemini-2.5-flash:generateContent" \
  -H "Authorization: Bearer $AVALAI_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "contents": [{
      "parts": [{"text": "Your prompt here with substantial context..."}]
    }]
  }'

# Response will include usage_metadata with cached token counts

Explicit Caching (Not Supported)

Explicit caching allows you to manually cache content and reference it in subsequent requests using cachedContent. This guarantees cost savings but requires additional setup.

Note

Explicit caching endpoints (/cachedContents) are not currently supported on AvalAI. We may add support in future updates.

Why Explicit Caching Is Not Supported

As an API aggregator, AvalAI routes requests across multiple infrastructures for reliability and performance. Explicit caching requires:

• Persistent storage tied to specific infrastructure
• Consistent routing of all related requests to the same servers
• Direct management of cache lifecycle

These requirements conflict with AvalAI's distributed architecture.

Use Cases Best Suited for Caching

Context caching is particularly beneficial for:

1. Chatbots with extensive system instructions - Large personality definitions, company knowledge bases
2. Document analysis applications - Recurring queries against the same documents
3. Code analysis tools - Repository analysis, bug fixing across similar codebases
4. Video/Audio analysis - Multiple questions about the same media file

Frequently Asked Questions (Based on OpenAI's Implementation)

1. How is data privacy maintained for caches? Prompt caches are typically not shared between organizations. Only members of the same organization can benefit from caches of identical prompts submitted by that organization. AvalAI acts as a proxy, so caching benefits are tied to the underlying provider's handling of requests from AvalAI's infrastructure or potentially your specific organization if applicable provider features are used.
2. Does Prompt Caching affect the final response? No. Prompt Caching should not influence the generation of output tokens or the final response. Only the prompt processing is potentially optimized; the response is computed anew based on the full (potentially partially cached) prompt.
3. Is there a way to manually clear the cache? Manual cache clearing is generally not available. Caches are typically cleared automatically after periods of inactivity.
4. Is there an extra cost for Prompt Caching? Cache writes have no additional fee on OpenAI models before GPT-5.6. GPT-5.6 and later cache writes are billed at 1.25 times the uncached input rate, while later cache reads use the discounted cached-input rate. Inspect both cache_write_tokens and cached_tokens before claiming a net saving.
5. Do cached prompts contribute to TPM rate limits? Yes, the full prompt tokens (cached + non-cached) usually count towards rate limits like Tokens Per Minute (TPM). Caching affects cost and latency, not rate limit calculation.
6. Is discounting for Prompt Caching available everywhere? Availability depends on the provider and specific service tiers (e.g., OpenAI offers it on standard APIs and Scale Tier, but not the Batch API).
7. Does Prompt Caching work with Zero Data Retention (ZDR)? Provider behavior depends on the model and cache-retention mode. OpenAI documents in-memory and extended cache retention separately; for AvalAI, confirm the selected route before promising a ZDR-style guarantee.
8. Does Prompt Caching affect data residency? Treat residency as provider- and route-specific. OpenAI documents that in-memory caching does not store prompt data to disk and that extended caching must remain in-region when regional inference is used. Through AvalAI, verify the selected provider route, region, and retention behavior before making residency commitments to customers.
9. Should prompt_cache_key identify individual users? Usually no. Use stable, opaque workload, tenant, assistant, policy, or schema buckets and keep raw personal identifiers out of the cache key. Rotate the key when the long-lived policy, tool schema, or prompt prefix changes.
10. Can I rely on cached prompts for correctness or continuity? No. Cache hits only optimize repeated prompt processing. They do not replace sending the full required context, they do not store conversation state for your app, and cached prompt tokens can still count toward TPM limits.

Related Resources

• Model Selection Guide
• Latency Optimization
• Data Controls
• Pricing
• OpenAI Models
• Google Models
• Native GenAI SDK (v1beta)
• Official OpenAI prompt-caching guide

Adapted from the official OpenAI Cookbook and openai/openai-cookbook, with AvalAI endpoint, API key, model, and support-boundary changes.