Context Compaction

Long-running agents and chat workflows eventually accumulate more context than the model needs. Compaction reduces old turns into a smaller state package while preserving the facts, tool results, decisions, and open tasks required for the next turn.

OpenAI documents server-side and standalone compaction for the Responses API. In AvalAI, treat those hosted compaction controls as route-dependent: use them only when the selected model and account explicitly support context_management, compact_threshold, or /v1/responses/compact. The portable pattern below works with the standard AvalAI /v1/responses route.

What to Preserve

Compact old context into a durable handoff:

• Goal: the current user objective and success criteria.
• State: stable facts, IDs, user preferences, and constraints.
• Actions: completed tool calls, side effects, and external records changed.
• Evidence: citations, filenames, request IDs, or returned object IDs the next turn needs.
• Blockers: unresolved questions, failed calls, retries, or safety constraints.
• Next step: the concrete action the model should take now.

Do not compact away recent tool outputs, safety decisions, or permission checks that the next call must reason over exactly.

Choose a Strategy

Strategy	Use when	AvalAI note
App-managed summary	You need portable behavior across providers or strict control over stored state.	Works anywhere /v1/responses works, but the summary is only as good as your prompt and validation.
Server-side compaction	The selected route supports OpenAI-style context_management and compact_threshold.	Treat the returned compaction item as opaque; append it unchanged in stateless chaining.
Standalone compact endpoint	You want explicit control before the next turn and the route supports /v1/responses/compact.	Pass the returned compacted window to the next request as-is; do not prune it.
Manual truncation	You only need to drop irrelevant old chat turns.	Keep the latest user request, tool outputs, IDs, policy constraints, and human approvals verbatim.

App-Managed Compaction

Use a normal /v1/responses request to create a compact state object, store it in your app, and send it back with the next user turn.

Python

import json
import os
from openai import OpenAI

client = OpenAI(
    api_key=os.environ["AVALAI_API_KEY"],
    base_url="https://api.avalai.ir/v1",
)

transcript = [
    {"role": "user", "content": "Help me debug this billing integration..."},
    {"role": "assistant", "content": "First I checked the webhook logs..."},
    {"role": "user", "content": "The failed request ID is req_123."},
]

compact = client.responses.create(
    model="gpt-5.5",
    instructions=(
        "Compact the conversation into JSON with keys: goal, facts, "
        "decisions, completed_actions, blockers, next_step. Preserve IDs."
    ),
    input=json.dumps(transcript, ensure_ascii=False),
    store=False,
)

state = compact.output_text

next_response = client.responses.create(
    model="gpt-5.5",
    instructions="Use the compacted state as prior context. Do not invent missing details.",
    input=[
        {"role": "developer", "content": f"Compacted prior state:\n{state}"},
        {"role": "user", "content": "Now draft the fix plan."},
    ],
    store=False,
)

print(next_response.output_text)

JavaScript

import OpenAI from "openai";

const client = new OpenAI({
  apiKey: process.env.AVALAI_API_KEY,
  baseURL: "https://api.avalai.ir/v1",
});

const transcript = [
  { role: "user", content: "Help me debug this billing integration..." },
  { role: "assistant", content: "First I checked the webhook logs..." },
  { role: "user", content: "The failed request ID is req_123." },
];

const compact = await client.responses.create({
  model: "gpt-5.5",
  instructions:
    "Compact the conversation into JSON with keys: goal, facts, decisions, completed_actions, blockers, next_step. Preserve IDs.",
  input: JSON.stringify(transcript),
  store: false,
});

const nextResponse = await client.responses.create({
  model: "gpt-5.5",
  instructions: "Use the compacted state as prior context. Do not invent missing details.",
  input: [
    { role: "developer", content: `Compacted prior state:\n${compact.output_text}` },
    { role: "user", content: "Now draft the fix plan." },
  ],
  store: false,
});

console.log(nextResponse.output_text);

Hosted Compaction Boundary

When a route supports OpenAI-style hosted compaction:

• server-side compaction can run inside responses.create after a configured compact_threshold;
• standalone compaction can return a compacted context window for the next /v1/responses call;
• encrypted compaction items are opaque, so pass them forward as returned instead of editing them;
• if you use previous_response_id, do not manually prune the server-managed chain.

If those controls are not available on your AvalAI route, keep using app-managed compaction and Conversation State.

Hosted compacted output is not a human summary. Treat it as machine state for the next model call: store it only if your retention policy allows it, pass it forward unchanged, and keep any human-readable audit summary as a separate app-managed artifact.

Hosted Compaction Semantics

Use these rules when adapting OpenAI's hosted compaction pattern to AvalAI:

• Server-side compaction runs inside responses.create after the rendered token count crosses compact_threshold; you do not call a separate compact endpoint in that mode.
• The server may emit an encrypted compaction item in response.output or in the response stream. Treat that item as opaque model state.
• For stateless input-array chaining, append all returned output items to the next input. After testing, you may drop items before the newest compaction item to reduce request size and long-tail latency.
• For previous_response_id chaining, pass only the new user input and avoid manual pruning; the server-managed chain is responsible for carrying the compacted state.
• For standalone /v1/responses/compact, the returned output is the canonical next context window. Pass it into the next /v1/responses call as-is and do not prune the compact output.

Server-Side Compaction Shape

Use this shape only after confirming the route supports context_management. The threshold should be below the model's context window and leave room for output plus reasoning tokens.

Python

conversation = [
    {
        "type": "message",
        "role": "user",
        "content": "Start a long support investigation.",
    }
]

response = client.responses.create(
    model="gpt-5.5",
    input=conversation,
    store=False,
    context_management=[{"type": "compaction", "compact_threshold": 200_000}],
)

# Append output items, including any encrypted compaction item.
conversation.extend(response.output)

JavaScript

const conversation = [
  {
    type: "message",
    role: "user",
    content: "Start a long support investigation.",
  },
];

const response = await client.responses.create({
  model: "gpt-5.5",
  input: conversation,
  store: false,
  context_management: [
    { type: "compaction", compact_threshold: 200000 },
  ],
});

// Append output items, including any encrypted compaction item.
conversation.push(...response.output);

Standalone Compact Shape

When /v1/responses/compact is available, compact the current window before adding the next user message. The window you send to the compact endpoint must still fit within the selected model's context window.

Python

compacted = client.responses.compact(
    model="gpt-5.5",
    input=long_input_items,
)

next_input = [
    *compacted.output,
    {
        "type": "message",
        "role": "user",
        "content": "Continue from the compacted state.",
    },
]

next_response = client.responses.create(
    model="gpt-5.5",
    input=next_input,
    store=False,
)

JavaScript

const compacted = await client.responses.compact({
  model: "gpt-5.5",
  input: longInputItems,
});

const nextInput = [
  ...compacted.output,
  {
    type: "message",
    role: "user",
    content: "Continue from the compacted state.",
  },
];

const nextResponse = await client.responses.create({
  model: "gpt-5.5",
  input: nextInput,
  store: false,
});

Best Practices

• Compact before requests approach the model context limit, not after failures start.
• Keep the most recent user request and critical tool outputs verbatim.
• Validate compacted JSON before storing it or sending it into the next request.
• Track token usage before and after compaction to measure cost and latency impact.
• If you use stateless input-array chaining, you may drop items before the newest hosted compaction item only after testing that the next turn still has the required state.
• If you use previous_response_id, let the server-managed chain carry state and avoid manual pruning.
• Pair compaction with Token Counting and Prompt Caching for production workflows.