Cost Optimization

Cost optimization in AvalAI is mostly about reducing tokens, avoiding unnecessary requests, choosing the right model, and routing non-urgent work to cheaper service tiers. OpenAI's cost guide groups these ideas with Batch API and flex processing; in AvalAI, apply the same principles while checking each model, endpoint, and account feature against the AvalAI docs.

Cost Levers

Lever	How it reduces spend	AvalAI docs to check
Fewer requests	Combine steps, cache deterministic results, skip LLM calls for fixed UI logic.	Production best practices
Fewer input tokens	Trim retrieved chunks, compact old context, keep prompts cache-friendly.	Token counting, Context compaction
Fewer output tokens	Set explicit answer budgets and max_output_tokens / max_completion_tokens.	Latency optimization
Smaller model	Route simple tasks to mini/flash/nano models and reserve frontier models for hard work.	Model selection, Pricing
Reasoning budget	Lower reasoning.effort when evals show the same quality with fewer hidden reasoning tokens.	Reasoning, Model selection
Cached tokens	Keep stable instructions and schemas at the start of prompts.	Prompt caching
Async or flex work	Use lower-cost or queued processing for non-urgent tasks.	Service tiers, Batch processing

Make Each Request Cheaper

For /v1/responses, constrain the answer and avoid storing data you do not need later:

Python

import os
from openai import OpenAI

client = OpenAI(
    api_key=os.environ["AVALAI_API_KEY"],
    base_url="https://api.avalai.ir/v1",
)

response = client.responses.create(
    model="gpt-5.4-mini",
    instructions="Answer in at most 5 bullets. Do not include background explanations.",
    input="Summarize the operational risks in this incident note: ...",
    reasoning={"effort": "low"},
    text={"verbosity": "low"},
    max_output_tokens=300,
    store=False,
)

print(response.output_text)

JavaScript

import OpenAI from "openai";

const client = new OpenAI({
  apiKey: process.env.AVALAI_API_KEY,
  baseURL: "https://api.avalai.ir/v1",
});

const response = await client.responses.create({
  model: "gpt-5.4-mini",
  instructions: "Answer in at most 5 bullets. Do not include background explanations.",
  input: "Summarize the operational risks in this incident note: ...",
  reasoning: { effort: "low" },
  text: { verbosity: "low" },
  max_output_tokens: 300,
  store: false,
});

console.log(response.output_text);

For /v1/chat/completions, use max_completion_tokens and keep retrieved context compact.

For GPT-5.5 baselines, medium reasoning is a good quality starting point. Once the workflow passes evals, compare low reasoning and low verbosity on the same dataset. Do not lower reasoning on compliance, safety, financial, code migration, or high-impact decisions unless your evals and human review show the cheaper setting is still reliable.

Do Not Confuse a Token Cap with a Visible-Answer Budget

On reasoning-capable models, max_output_tokens (Responses), max_completion_tokens (Chat Completions), and legacy max_tokens can cover both hidden reasoning and the visible answer. They cap total generation; they do not reserve part of that budget for user-visible text.

This creates a cost trap: a request can consume billable output tokens and still return an empty answer. For example, if a request has a 1,500-token limit and usage.output_tokens_details.reasoning_tokens reports nearly all 1,500 tokens as reasoning, no budget may remain for the final text. A short instruction such as "answer in 200 characters" constrains the desired answer but does not necessarily constrain the model's internal reasoning.

Diagnose budget exhaustion before retrying

Treat the following as token-budget exhaustion rather than an automatic provider outage, content-filter failure, or empty successful result:

• Responses API: status: "incomplete" with incomplete_details.reason: "max_output_tokens"; response.output may contain only a reasoning item and response.output_text may be empty.
• Chat Completions: finish_reason: "length"; visible content can be empty or cut off.
• Usage evidence: output_tokens_details.reasoning_tokens is close to output_tokens, with little or no visible text.

Do not parse, cache, display, or bill an incomplete response as a normal successful answer. Log the status, incomplete or finish reason, configured token limit, visible-text length, reasoning tokens, model, reasoning effort, prompt version, and x-request-id.

Optimize for successful-answer cost

A lower token cap is not cheaper when it causes paid failures and retries. Track both:

cost_per_successful_answer =
  total_cost_of_initial_attempts_and_retries
  / number_of_usable_answers

wasted_reasoning_rate =
  reasoning_tokens_from_incomplete_no-text_responses
  / total_reasoning_tokens

Tune each evaluated prompt class with a measured envelope rather than one global limit:

1. Measure the P50, P95, and P99 reasoning-token usage and visible-answer length for successful requests.
2. Choose the lowest reasoning.effort that passes quality and safety evals.
3. Set the generation limit high enough for expected reasoning plus final-answer headroom, within the model's supported maximum.
4. Alert on incomplete/no-text rates and repeated budget exhaustion by model and prompt version.
5. Re-evaluate after changing the model snapshot, tools, retrieved context, schema, or prompt because all can change reasoning demand.

Use a bounded recovery policy

When exhaustion occurs, retry at most according to an explicit application policy. Depending on task risk and model support, make one controlled adjustment: increase the token limit, lower reasoning.effort to low or none, simplify or split the task, reduce irrelevant context, or route to a better-suited model. Avoid an unchanged retry, which can reproduce the same paid failure, and cap retries so a single user action cannot multiply spend.

For high-impact work, do not automatically reduce reasoning just to save cost; prefer a larger budget, task decomposition, or human review. See Reasoning token budgets and Production best practices.

Route by Task Value

Use a routing policy instead of one default model for everything:

• Tier 1 tasks: classification, extraction, short summaries, and formatting can often use smaller or cheaper models.
• Tier 2 tasks: customer-facing answers and multi-step tool workflows need a stronger default model and strict output budgets.
• Tier 3 tasks: high-value reasoning, code migration, or compliance review can justify larger reasoning models, background processing, and extra verification.

Track accuracy, cost, and latency separately. A cheaper model that requires two retries can cost more than a stronger model that succeeds once.

Add Budget Guardrails

Cost optimization should fail safely, not only show a dashboard after spend happens. Add guardrails at three levels:

• Per request: reject or downshift requests that exceed your token-count budget before calling the model.
• Per workflow: cap retries, tool loops, and parallel fan-out so one user action cannot launch unbounded calls.
• Per account or reseller: use the User API and your own billing records to enforce daily, monthly, or customer-specific budgets.

When a request exceeds budget, choose an explicit fallback: summarize context first, switch to a smaller model, route to flex, move to background processing, or ask the user to confirm a higher-cost action. Do not silently truncate critical compliance, safety, finance, or migration context.

Use Flex for Non-Urgent Work

AvalAI's generally documented public service tiers are default and flex. Use service_tier: "flex" only when the model supports it and the job can tolerate slower or occasionally unavailable capacity.

Python

response = client.responses.create(
    model="gpt-5.4-mini",
    input="Generate 50 synthetic support-ticket examples for evaluation.",
    service_tier="flex",
    store=False,
)

JavaScript

const response = await client.responses.create({
  model: "gpt-5.4-mini",
  input: "Generate 50 synthetic support-ticket examples for evaluation.",
  service_tier: "flex",
  store: false,
});

Bash

curl https://api.avalai.ir/v1/responses \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer $AVALAI_API_KEY" \
  -d '{
    "model": "gpt-5.4-mini",
    "input": "Generate 50 synthetic support-ticket examples for evaluation.",
    "service_tier": "flex",
    "store": false
  }'

OpenAI's Flex guidance trades lower cost for slower responses and possible 429 Resource Unavailable errors. In AvalAI, treat Flex as best-effort capacity unless your account contract says otherwise:

• Increase client timeouts for long Flex jobs instead of assuming the default SDK timeout is enough.
• Retry 429/resource-unavailable responses with exponential backoff for work that can wait.
• Fall back to service_tier: "default" only when completion is more important than the lower-cost route.
• Do not use Flex for interactive checkout, account changes, safety-critical moderation, or any path where capacity delays would break the user experience.

If you are porting an OpenAI example that uses service_tier: "priority", use default on AvalAI unless priority processing is explicitly enabled for your account and route.

Batch and Background Work

For many independent rows, use the Batch processing pattern or your own rate-limit-safe worker until hosted Batch support is available for your route. For one long-running response, use Background processing or an app-managed job table.

Batch is for offline throughput, not faster user-facing latency. OpenAI's reference Batch API uses a separate capacity pool, a 24-hour completion window, .jsonl inputs, unique custom_id values, and result files whose output order may not match input order. When adapting that pattern to AvalAI, verify hosted Batch availability for the exact endpoint and model, preserve custom_id for joins, and treat expired or failed rows as retryable work units in your own job system.

Good async candidates:

• eval runs and prompt comparisons;
• nightly data enrichment;
• long reports that do not block the UI;
• synthetic data generation;
• backfills and migration analysis.

Measure Real Cost

Do not rely only on estimates. Capture:

• model, endpoint, service tier, prompt version, and reasoning effort;
• configured max_output_tokens, max_completion_tokens, or legacy max_tokens;
• response status, incomplete_details.reason, Chat Completions finish_reason, and visible-text length;
• input, output, reasoning, and cached token counts when returned;
• x-request-id from response headers;
• estimated cost for fast UI feedback;
• final billing data from the User API.

Use the same unit-cost formula for every candidate route before switching models. Across providers and models, hidden reasoning tokens are billed at the selected model's output-token rate. First confirm the endpoint's usage semantics: when output_tokens already includes reasoning and output_tokens_details.reasoning_tokens is its breakdown, adding reasoning_tokens again would double-count them.

When total output already includes reasoning:

expected_cost =
  uncached_input_tokens * input_price
  + cached_input_tokens * cached_input_price
  + output_tokens * output_price
  + retry_rate * average_retry_cost

When a route reports visible output and reasoning as separate, non-overlapping counts, use the same output price for both:

expected_cost =
  uncached_input_tokens * input_price
  + cached_input_tokens * cached_input_price
  + visible_output_tokens * output_price
  + reasoning_tokens * output_price
  + retry_rate * average_retry_cost

Keep the prices symbolic in code and load them from your current pricing source. Reconcile estimates with AvalAI billing because providers can expose reasoning usage differently even though reasoning tokens use the output-token rate. The important comparison is not "cheapest model per token"; it is "lowest expected cost per usable answer at the quality, retry rate, and latency your workflow requires."

Use dashboards to find the expensive 10% of workflows first. Optimize prompts and model choice there before chasing small one-off savings.

Related Resources

• Pricing
• Service tiers
• Latency optimization
• Prompt caching
• Token counting
• Background processing
• Batch processing