Latency Optimization

This guide covers core principles for improving latency across a wide variety of LLM-related use cases. These techniques are derived from working with a wide range of customers and developers on production applications.

Seven Principles for Latency Optimization

1. Process tokens faster
2. Generate fewer tokens
3. Use fewer input tokens
4. Make fewer requests
5. Parallelize
6. Make your users wait less
7. Don't default to an LLM

AvalAI Latency Checklist

Use these checks before you rewrite a working integration:

Lever	Use it when	AvalAI guidance
Smaller model	The task is bounded, repetitive, or easy to verify	Try gpt-5.4-mini, gpt-5.4-nano, or a fast provider-specific model before sending every step to a flagship model.
Output budget	The response can be brief or structured	Use text.verbosity and max_output_tokens on /v1/responses; use max_completion_tokens on /v1/chat/completions; keep legacy max_tokens only for older examples.
Reasoning effort	A reasoning model is spending too long thinking	GPT-5.5 defaults to medium; test low before none, and raise to high/xhigh only when evals prove the quality gain.
Prompt cache	Many requests share the same instructions, schema, or RAG prefix	Keep shared prompt text first, put dynamic content later, and use prompt_cache_key only when the route/model supports it.
Service tier	You are choosing cost vs. speed	Use service_tier: "default" for latency-sensitive production calls. Use flex only for cost-sensitive jobs that can tolerate slower or unavailable capacity.
Streaming	Users can consume partial output	Stream to reduce time-to-first-token and show real progress instead of waiting for the full completion.

Measure both time to first token and time to final token. A change that improves full completion latency may still feel slow if the UI does not stream or show progress.

For GPT-5.5 and other reasoning-capable OpenAI models, treat model choice, reasoning.effort, and text.verbosity as separate knobs. A high-reasoning request with a concise final answer can still be useful, but it costs more hidden reasoning tokens and time. Start from medium for quality baselines, compare low for interactive flows, and keep none for simple classification, retrieval, or formatting tasks that do not need multi-step planning.

Optimize in this order for most AvalAI apps:

1. Reduce output tokens first. Visible output and reasoning tokens usually dominate latency more than prompt tokens.
2. Use the smallest model that passes your evals. Add clearer instructions, few-shot examples, or fine-tuning before defaulting every step to a flagship model.
3. Combine or parallelize independent calls. Sequential API round trips add user-visible delay.
4. Cache stable prefixes. Put shared instructions, tool schemas, and policy text first; keep dynamic RAG snippets and user state later.
5. Avoid the LLM when deterministic code is better. Hard-code confirmations, use normal search/filtering for simple lookups, and render structured data with UI components instead of generated prose.

Measure Before You Tune

Do one baseline trace before changing prompts or models. For each request, log:

• avalai-request-id, endpoint, provider, model, service tier, and whether the request streamed;
• time to first byte, time to first token, and time to final token;
• input tokens, output tokens, reasoning tokens, and cached-token counts when available;
• retry count, rate-limit errors, provider fallback, and final status;
• user-visible wait time, including queueing, retrieval, rendering, and client-side buffering.

Then compare one optimization at a time. A smaller model may improve final-token latency, while streaming may improve perceived latency without reducing total compute time. Treat both as useful, but measure them separately.

Bottleneck-to-Lever Map

Use your baseline trace to identify the slowest layer before changing prompts. This keeps latency work focused and prevents swapping models when the real issue is retrieval, rendering, or sequential orchestration.

Bottleneck	Common signal	First AvalAI lever
Model compute is slow	Long final-token time, high reasoning-token count, or a flagship model on a simple task	Try a smaller model, lower reasoning.effort, or split easy classification from hard generation.
Output is too long	High output tokens or verbose JSON/function arguments	Lower text.verbosity, cap max_output_tokens/max_completion_tokens, shorten field names, or return structured IDs instead of prose.
Prompt is too large	High input tokens with low cached_tokens	Trim RAG context, clean HTML, keep static prefix first, and add prompt_cache_key only when the route supports it.
Too many round trips	Several sequential avalai-request-id values for one user action	Combine steps into one structured response, parallelize independent calls, or use speculative execution for likely-safe branches.
UI feels blank	Slow time-to-first-token or no progress during tool/retrieval work	Stream responses, show tool/retrieval steps, and chunk backend post-processing before forwarding to the UI.
LLM is unnecessary	Same constrained output repeats across requests	Hard-code confirmations, precompute variants, use search/filtering, or render metrics with UI components.

Process Tokens Faster

Inference speed is the rate at which the LLM processes tokens, often measured in tokens per minute (TPM) or tokens per second (TPS).

The main factor influencing inference speed is model size – smaller models usually run faster (and cheaper), and when used correctly can even outperform larger models. To maintain high quality performance with smaller models:

• Use a longer, more detailed prompt
• Add more few-shot examples
• Consider fine-tuning / distillation

For example, you might choose gpt-5.4-mini or claude-haiku-4-5 for faster responses when appropriate for the task.

Predicted Outputs can also improve inference time when most of a text response is already known, such as file edits. Use the Predicted Outputs guide when you can provide a close draft of the expected output.

# Using a smaller model with a more detailed prompt
response = client.chat.completions.create(
    model="gpt-5.4-mini",
    messages=[
        {
            "role": "system",
            "content": "You are a helpful assistant that provides concise, accurate answers.",
        },
        {
            "role": "user",
            "content": "Explain quantum computing in simple terms, focusing on qubits and superposition. Include an analogy that makes it easy to understand.",
        },
    ],
)

Responses API version

Use this version when the selected model supports /v1/responses. messages moves to input, and the final text is read from response.output_text.

import os
from openai import OpenAI

client = OpenAI(
    api_key=os.environ["AVALAI_API_KEY"],
    base_url="https://api.avalai.ir/v1",
)

response = client.responses.create(
    model="gpt-5.4-mini",
    instructions="You are a helpful assistant.",
    input="Explain quantum computing in simple terms, focusing on qubits and superposition. Include an analogy that makes it easy to understand.",
)

print(response.output_text)

• messages → input
• system message → instructions or a developer item
• choices[0].message.content → response.output_text
• for tools and multimodal output, inspect response.output by item type.

Generate Fewer Tokens

Generating tokens is typically the highest latency step when using an LLM. As a general rule, cutting 50% of your output tokens may cut ~50% of your latency. For reasoning models, remember that hidden reasoning tokens also consume output budget and time, even when they are not shown to the user.

To reduce output size:

• For natural language, ask the model to be concise ("under 20 words" or "be very brief")
• For structured output, minimize your output syntax: shorten function names, omit named arguments, coalesce parameters
• Use max_output_tokens on /v1/responses, max_completion_tokens on /v1/chat/completions, or stop/stop sequences where supported to end generation early

For latency-sensitive intermediate JSON, every field name is output. Keep public API contracts readable, but make internal reasoning or routing fields compact: message_is_conversation_continuation can become cont, response_requirements can become reqs, and explanations can live in the prompt or schema comments instead of being regenerated in every response. Validate the compact schema with evals before using it in a customer-facing contract.

# Requesting a concise response
response = client.chat.completions.create(
    model="gpt-5.5",
    messages=[
        {
            "role": "system",
            "content": "You are a helpful assistant that provides very brief answers, under 50 words.",
        },
        {"role": "user", "content": "Explain the theory of relativity"},
    ],
    max_completion_tokens=100,
)

Responses API version

Use this version when the selected model supports /v1/responses. messages moves to input, and the final text is read from response.output_text.

import os
from openai import OpenAI

client = OpenAI(
    api_key=os.environ["AVALAI_API_KEY"],
    base_url="https://api.avalai.ir/v1",
)

response = client.responses.create(
    model="gpt-5.5",
    instructions="You are a helpful assistant that provides very brief answers, under 50 words.",
    input="Explain the theory of relativity",
    max_output_tokens=100,
)

print(response.output_text)

• messages → input
• system message → instructions or a developer item
• choices[0].message.content → response.output_text
• for tools and multimodal output, inspect response.output by item type.

Use Fewer Input Tokens

While reducing input tokens does result in lower latency, the impact is less significant – cutting 50% of your prompt may only result in a 1-5% latency improvement. Consider these techniques when working with large contexts:

• Fine-tune the model to replace lengthy instructions/examples
• Filter context input (prune RAG results, clean HTML)
• Maximize shared prompt prefix by putting dynamic portions later in the prompt

For repeated production traffic, stable prefixes matter more than aggressively trimming every word. Put fixed instructions, JSON schemas, and policy text before dynamic chat history or retrieval snippets so provider-side caching can reuse the prefix. If your route supports it, add a stable prompt_cache_key per workload or assistant configuration; do not use raw user identifiers as cache keys.

# Example of filtering context input
def filter_relevant_context(query, documents, max_tokens=2000):
    # Sort documents by relevance to query
    sorted_docs = sort_by_relevance(query, documents)

    # Take only the most relevant documents up to max_tokens
    filtered_docs = []
    token_count = 0

    for doc in sorted_docs:
        doc_tokens = count_tokens(doc)
        if token_count + doc_tokens <= max_tokens:
            filtered_docs.append(doc)
            token_count += doc_tokens
        else:
            break

    return filtered_docs

Make Fewer Requests

Each API request incurs round-trip latency. Instead of sequential requests, consider combining multiple steps into a single prompt:

# Instead of separate requests for summarization and translation
response = client.chat.completions.create(
    model="gpt-5.5",
    messages=[
        {
            "role": "system",
            "content": "You will perform two tasks: 1) Summarize the text, and 2) Translate the summary to Spanish. Return results in JSON format with fields 'summary' and 'translation'.",
        },
        {"role": "user", "content": "Text to process: " + long_article},
    ],
)

Responses API version

Use this version when the selected model supports /v1/responses. messages moves to input, and the final text is read from response.output_text.

import os
from openai import OpenAI

client = OpenAI(
    api_key=os.environ["AVALAI_API_KEY"],
    base_url="https://api.avalai.ir/v1",
)

response = client.responses.create(
    model="gpt-5.5",
    instructions="You will perform two tasks: 1) Summarize the text, and 2) Translate the summary to Spanish. Return results in JSON format with fields 'summary' and 'translation'.",
    input="Text to process: " + long_article,
)

print(response.output_text)

• messages → input
• system message → instructions or a developer item
• choices[0].message.content → response.output_text
• for tools and multimodal output, inspect response.output by item type.

Parallelize

For non-sequential steps, parallelize API calls:

In production, pair parallelism with a concurrency limit and retries. AvalAI rate-limit tiers vary by model and endpoint, so bounded parallelism is safer than launching every document at once. For offline work, consider the Batch API when throughput matters more than interactive latency.

import asyncio
import os
from openai import AsyncOpenAI


async def process_documents(documents):
    client = AsyncOpenAI(
        api_key=os.environ["AVALAI_API_KEY"], base_url="https://api.avalai.ir/v1"
    )

    async def process_document(doc):
        response = await client.chat.completions.create(
            model="gpt-5.4-mini",
            messages=[
                {"role": "system", "content": "Summarize the following document:"},
                {"role": "user", "content": doc},
            ],
        )
        return response.choices[0].message.content

    # Process all documents in parallel
    tasks = [process_document(doc) for doc in documents]
    return await asyncio.gather(*tasks)

Responses API version

Use this version when the selected model supports /v1/responses. messages moves to input, and the final text is read from response.output_text.

import asyncio
import os
from openai import AsyncOpenAI


async def process_documents(documents):
    client = AsyncOpenAI(
        api_key=os.environ["AVALAI_API_KEY"], base_url="https://api.avalai.ir/v1"
    )

    async def process_document(doc):
        response = await client.responses.create(
            model="gpt-5.4-mini",
            instructions="Summarize the following document:",
            input=doc,
        )
        return response.output_text

    tasks = [process_document(doc) for doc in documents]
    return await asyncio.gather(*tasks)

• messages → input
• system message → instructions or a developer item
• choices[0].message.content → response.output_text
• for tools and multimodal output, inspect response.output by item type.

For sequential steps, consider speculative execution when one outcome is more likely:

1. Start step 1 & step 2 simultaneously (e.g., input moderation & story generation)
2. Verify the result of step 1
3. If the result was not as expected, cancel step 2 (and retry if necessary)

When you use speculative execution in production, record both request IDs and make the cancellation/discard path explicit. For example, if an input-moderation request fails, discard the already-started generation, do not stream it to the user, and count the wasted tokens in your cost dashboard.

Make Your Users Wait Less

The difference between waiting and watching progress is significant:

• Streaming: Immediately start showing the response as it's generated
• Chunking: Process output in chunks for real-time display
• Show your steps: Surface multi-step processes to users
• Loading states: Use spinners and progress bars

import os
import sys
from openai import OpenAI

client = OpenAI(
    api_key=os.environ["AVALAI_API_KEY"],
    base_url="https://api.avalai.ir/v1",  # AvalAI API endpoint
)

# Stream the response to the user
response = client.chat.completions.create(
    model="gpt-5.5",
    messages=[
        {"role": "user", "content": "Write a short story about a time traveler."}
    ],
    stream=True,
)

for chunk in response:
    if chunk.choices[0].delta.content:
        sys.stdout.write(chunk.choices[0].delta.content)
        sys.stdout.flush()

Responses API version

Use this version when the selected model supports /v1/responses. messages moves to input, and the final text is read from response.output_text.

import os
import sys
from openai import OpenAI

client = OpenAI(
    api_key=os.environ["AVALAI_API_KEY"],
    base_url="https://api.avalai.ir/v1",
)

stream = client.responses.create(
    model="gpt-5.5",
    input="Write a short story about a time traveler.",
    stream=True,
)

for event in stream:
    if event.type == "response.output_text.delta":
        sys.stdout.write(event.delta)
        sys.stdout.flush()

• messages → input
• system message → instructions or a developer item
• choices[0].message.content → response.output_text
• for tools and multimodal output, inspect response.output by item type.

Don't Default to an LLM

LLMs are versatile but not always the most efficient solution. Consider these alternatives:

• Hard-coding: For constrained outputs like confirmation messages
• Pre-computing: For limited input scenarios
• Leveraging UI: For summarized metrics or search results
• Traditional optimization: Binary search, caching, hash maps, etc.

# Example of using a cache for common queries
response_cache = {}


def get_response(query):
    # Check if we have a cached response
    if query in response_cache:
        return response_cache[query]

    # If not, generate a new response
    response = client.chat.completions.create(
        model="gpt-5.5", messages=[{"role": "user", "content": query}]
    )

    # Cache the response for future use
    response_text = response.choices[0].message.content
    response_cache[query] = response_text

    return response_text

Responses API version

Use this version when the selected model supports /v1/responses. messages moves to input, and the final text is read from response.output_text.

import os
from openai import OpenAI

client = OpenAI(
    api_key=os.environ["AVALAI_API_KEY"],
    base_url="https://api.avalai.ir/v1",
)

response_cache = {}


def get_response(query):
    if query in response_cache:
        return response_cache[query]

    response = client.responses.create(model="gpt-5.5", input=query)
    response_cache[query] = response.output_text
    return response.output_text

• messages → input
• system message → instructions or a developer item
• choices[0].message.content → response.output_text
• for tools and multimodal output, inspect response.output by item type.

Example: Optimizing a Customer Service Bot

Let's analyze a sample customer service bot architecture and apply our latency optimization principles.

Initial Architecture

The initial architecture includes:

1. A user sends a message
2. The message is turned into a self-contained query
3. We determine if additional information is needed
4. Retrieval is performed
5. The assistant reasons about the query and search results
6. A response is sent back to the user

Optimizations Applied

1. Combine steps: Merge query contextualization and retrieval check to make fewer requests
2. Use smaller models: Switch to a smaller or fine-tuned model for well-defined tasks
3. Parallelize: Run retrieval checks and reasoning steps simultaneously
4. Shorten field names: Reduce output tokens by using more concise JSON field names

These optimizations can reduce latency while maintaining response quality, but validate them with your own traces. Track first-token latency, final-token latency, generated token count, retry rate, and user-visible completion time before rolling changes into production.

Related Resources

• Prompt Engineering
• Prompt Caching
• Predicted Outputs
• Token Counting
• Streaming Responses
• Model Selection