Predicted Outputs

Predicted Outputs reduce latency when most of a text response is already known. The common case is regenerating a text or code file after a small edit: you send the existing file as prediction.content, and the model can reuse matching output tokens instead of regenerating every token from scratch.

OpenAI's official guide documents Predicted Outputs for Chat Completions through the prediction request parameter. In AvalAI, availability depends on the upstream provider and model. For OpenAI-family traffic, prefer models that OpenAI lists for this feature, such as gpt-4.1, gpt-4.1-mini, gpt-4.1-nano, gpt-4o, and gpt-4o-mini, when they are available in your AvalAI account.

When to Use

Use Predicted Outputs when:

• you regenerate a code file, markdown document, config file, or template;
• most of the final answer should match the original content;
• you can provide the full expected text as a prediction;
• lower latency matters more than the risk of paying for rejected prediction tokens.

Avoid it when the answer is mostly new, uses tools, needs audio, or may produce multiple alternatives.

Prediction vs. Prompt Caching

Predicted Outputs and prompt caching solve different latency problems:

Technique	Speeds up	Best fit
Predicted Outputs	Output generation when most completion tokens are already known	Regenerating code, Markdown, config, templates, or other text files after a small edit
Prompt caching	Repeated input prefixes	Assistants with stable instructions, JSON schemas, policy text, or repeated RAG setup

For editing workflows, use both when supported: keep stable instructions and schemas at the front of the prompt for caching, then send the current file as prediction.content so unchanged output can be accepted quickly.

Chat Completions Example

This example asks the model to replace username with email in a TypeScript class. The current file is sent both as input and as the predicted output.

Bash

CODE_CONTENT=$(
  cat <<'EOF'
class User {
  firstName: string = "";
  lastName: string = "";
  username: string = "";
}

export default User;
EOF
)

curl https://api.avalai.ir/v1/chat/completions \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer $AVALAI_API_KEY" \
  -d "$(jq -n --arg code "$CODE_CONTENT" '{
    model: "gpt-4.1",
    messages: [
      {
        role: "user",
        content: "Replace the username property with an email property. Respond only with code, with no markdown formatting."
      },
      {
        role: "user",
        content: $code
      }
    ],
    prediction: {
      type: "content",
      content: $code
    }
  }')"

Python

import os
from openai import OpenAI

client = OpenAI(
    api_key=os.environ["AVALAI_API_KEY"],
    base_url="https://api.avalai.ir/v1",
)

code = """
class User {
  firstName: string = "";
  lastName: string = "";
  username: string = "";
}

export default User;
""".strip()

completion = client.chat.completions.create(
    model=os.getenv("AVALAI_MODEL", "gpt-4.1"),
    messages=[
        {
            "role": "user",
            "content": "Replace the username property with an email property. Respond only with code, with no markdown formatting.",
        },
        {"role": "user", "content": code},
    ],
    prediction={"type": "content", "content": code},
)

print(completion.choices[0].message.content)
print(completion.usage.completion_tokens_details)

JavaScript

import OpenAI from "openai";

const client = new OpenAI({
  apiKey: process.env.AVALAI_API_KEY,
  baseURL: "https://api.avalai.ir/v1",
});

const code = `
class User {
  firstName: string = "";
  lastName: string = "";
  username: string = "";
}

export default User;
`.trim();

const completion = await client.chat.completions.create({
  model: process.env.AVALAI_MODEL ?? "gpt-4.1",
  messages: [
    {
      role: "user",
      content:
        "Replace the username property with an email property. Respond only with code, with no markdown formatting.",
    },
    { role: "user", content: code },
  ],
  prediction: {
    type: "content",
    content: code,
  },
});

console.log(completion.choices[0].message.content);
console.log(completion.usage?.completion_tokens_details);

Responses API version (without prediction)

OpenAI documents prediction for Chat Completions, not as a Responses API parameter. When migrating this non-streaming workflow to /v1/responses, remove prediction and use the Responses shape (instructions, input, and response.output_text). Keep Chat Completions when accepted/rejected prediction token accounting is required.

Python

response = client.responses.create(
    model=os.getenv("AVALAI_MODEL", "gpt-5.6-luna"),
    instructions="Return only the complete updated TypeScript file. Do not use markdown.",
    input=[
        {
            "role": "user",
            "content": [
                {
                    "type": "input_text",
                    "text": "Replace the username property with an email property in this file.",
                },
                {"type": "input_text", "text": code},
            ],
        }
    ],
    store=False,
)

print(response.output_text)

JavaScript

const response = await client.responses.create({
  model: process.env.AVALAI_MODEL ?? "gpt-5.6-luna",
  instructions: "Return only the complete updated TypeScript file. Do not use markdown.",
  input: [
    {
      role: "user",
      content: [
        {
          type: "input_text",
          text: "Replace the username property with an email property in this file.",
        },
        { type: "input_text", text: code },
      ],
    },
  ],
  store: false,
});

console.log(response.output_text);

Bash

curl https://api.avalai.ir/v1/responses \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer $AVALAI_API_KEY" \
  -d "$(jq -n --arg code "$CODE_CONTENT" '{
    model: "gpt-5.6-luna",
    instructions: "Return only the complete updated TypeScript file. Do not use markdown.",
    input: [
      {
        role: "user",
        content: [
          { type: "input_text", text: "Replace the username property with an email property in this file." },
          { type: "input_text", text: $code }
        ]
      }
    ],
    store: false
  }')"

Migration checklist:

• messages → input
• system/developer prompt → instructions
• choices[0].message.content → response.output_text
• prediction → no direct Responses equivalent; use Chat Completions if prediction is required
• accepted_prediction_tokens / rejected_prediction_tokens → no equivalent in Responses usage

Read Usage Details

When the provider returns detailed usage, inspect:

• accepted_prediction_tokens: prediction tokens that matched the final output and helped reduce latency;
• rejected_prediction_tokens: prediction tokens that did not match the final output.

Rejected prediction tokens may still be billed as completion tokens. If rejected_prediction_tokens is consistently high, remove prediction for that workload or make the prediction more similar to the expected final answer.

Where Prediction Text Can Match

The predicted text does not need to be one continuous block at the start of the answer. It can match content before and after the new text the model inserts. For example, when adding one route to a server file, the unchanged imports, existing routes, and startup code can all count as accepted prediction tokens even if the new route appears in the middle.

Use this behavior for patch-style tasks:

• send the full current file as prediction.content;
• ask for the complete updated file, not a diff;
• keep formatting, comments, and surrounding text stable when possible;
• inspect rejected tokens to catch prompts that cause unnecessary rewrites.

Streaming with Predictions

Predicted Outputs can be useful with streaming because the matching parts of the answer may arrive faster.

Python

stream = client.chat.completions.create(
    model=os.getenv("AVALAI_MODEL", "gpt-4.1"),
    messages=[
        {
            "role": "user",
            "content": "Replace the username property with an email property. Respond only with code.",
        },
        {"role": "user", "content": code},
    ],
    prediction={"type": "content", "content": code},
    stream=True,
)

for chunk in stream:
    delta = chunk.choices[0].delta.content
    if delta:
        print(delta, end="")

JavaScript

const stream = await client.chat.completions.create({
  model: process.env.AVALAI_MODEL ?? "gpt-4.1",
  messages: [
    {
      role: "user",
      content: "Replace the username property with an email property. Respond only with code.",
    },
    { role: "user", content: code },
  ],
  prediction: { type: "content", content: code },
  stream: true,
});

for await (const chunk of stream) {
  process.stdout.write(chunk.choices[0]?.delta?.content ?? "");
}

Responses API streaming version (without prediction)

For the streaming example, migrate the transport to /v1/responses with stream: true / stream=True. This does not reproduce prediction-token acceleration, but it gives a developer-friendly streaming path for models and routes that support Responses.

Python

stream = client.responses.create(
    model=os.getenv("AVALAI_MODEL", "gpt-5.6-luna"),
    instructions="Return only the complete updated TypeScript file. Do not use markdown.",
    input=[
        {
            "role": "user",
            "content": [
                {
                    "type": "input_text",
                    "text": "Replace the username property with an email property in this file.",
                },
                {"type": "input_text", "text": code},
            ],
        }
    ],
    store=False,
    stream=True,
)

for event in stream:
    if event.type == "response.output_text.delta":
        print(event.delta, end="")

JavaScript

const stream = await client.responses.create({
  model: process.env.AVALAI_MODEL ?? "gpt-5.6-luna",
  instructions: "Return only the complete updated TypeScript file. Do not use markdown.",
  input: [
    {
      role: "user",
      content: [
        {
          type: "input_text",
          text: "Replace the username property with an email property in this file.",
        },
        { type: "input_text", text: code },
      ],
    },
  ],
  store: false,
  stream: true,
});

for await (const event of stream) {
  if (event.type === "response.output_text.delta") {
    process.stdout.write(event.delta);
  }
}

Bash

curl https://api.avalai.ir/v1/responses \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer $AVALAI_API_KEY" \
  -d "$(jq -n --arg code "$CODE_CONTENT" '{
    model: "gpt-5.6-luna",
    instructions: "Return only the complete updated TypeScript file. Do not use markdown.",
    input: [
      {
        role: "user",
        content: [
          { type: "input_text", text: "Replace the username property with an email property in this file." },
          { type: "input_text", text: $code }
        ]
      }
    ],
    store: false,
    stream: true
  }')"

Streaming migration checklist:

• messages → input
• system/developer prompt → instructions
• Chat stream chunks → Responses stream events such as response.output_text.delta
• prediction → no direct Responses equivalent; use Chat Completions if prediction is required
• prediction-token usage fields → no equivalent in Responses usage

Limitations

Predicted Outputs are provider- and model-specific. OpenAI documents these limitations for its Chat Completions implementation:

• only text output is supported;
• n greater than 1 is not supported;
• logprobs is not supported;
• positive presence_penalty and frequency_penalty values are not supported;
• audio inputs/outputs and modalities are not compatible;
• max_completion_tokens is not supported with predictions;
• tool/function calling is not currently supported with predictions.

Related Guides

• Chat Completions API
• Responses API
• Responses vs. Chat Completions
• Streaming Responses
• Prompt Caching