Evaluating Model Performance

Hosted Evals status

AvalAI does not currently expose a hosted /v1/evals API. Do not build production integrations against https://api.avalai.ir/v1/evals yet. Use the local and CI workflow below with normal AvalAI API calls.

Evaluations (evals) are structured tests for AI behavior. They help you compare models, detect prompt regressions, validate tool use, and decide whether a change is safe to deploy. This guide adapts the official OpenAI evaluation guidance for AvalAI’s OpenAI-compatible endpoints.

OpenAI’s hosted Evals, Datasets, and Graders concepts are useful design patterns, but do not treat the hosted OpenAI Evals object model as an AvalAI API contract. Keep your eval cases, expected outputs, rubrics, and thresholds portable in files so they can run locally today and migrate cleanly if AvalAI later exposes hosted eval infrastructure.

OpenAI’s own hosted Evals platform is also in a deprecation window: existing Evals become read-only for existing users on October 31, 2026, and the platform is scheduled to shut down on November 30, 2026. Avoid designing your AvalAI workflow around one hosted object model. Treat OpenAI’s evaluation best practices—objective, dataset, metrics, run/compare, continuous evaluation—as the durable part, and keep the runner replaceable.

Practical Path Today: Local and CI Evals

Until hosted AvalAI evals are available, keep eval assets in your repository:

• Dataset: representative prompts, expected labels, reference answers, or grading rubrics.
• Data schema: a JSON Schema-like contract for each test row, mirroring OpenAI’s data_source_config concept without depending on a hosted /v1/evals object.
• Runner: Promptfoo, pytest, a small script, or your CI pipeline.
• Models: production and candidate AvalAI model IDs, configured with environment variables.
• Assertions: exact match, JSON schema, semantic similarity, LLM-as-judge, tool-call accuracy, refusal behavior, latency, and cost thresholds.

See Promptfoo Evals with AvalAI for a runnable example adapted from the official OpenAI Cookbook and openai/openai-cookbook, with AvalAI endpoint, API key, and model changes.

Design the Eval

Start with the same five-step loop recommended in OpenAI’s evaluation best practices:

1. Define the objective: specify the behavior that must improve or stay stable.
2. Collect the dataset: include production-like examples, edge cases, multilingual inputs, malformed inputs, and adversarial prompts.
3. Define metrics: choose pass/fail, exact match, rubric scores, tool-call accuracy, retrieval precision/recall, or human review.
4. Run and compare: test the current production model against candidate prompts, model IDs, or routing changes.
5. Continuously evaluate: add failures from logs back into the eval set and run the suite on every release.

Avoid vibe-based evals such as “the answer looks good.” Write measurable criteria before changing prompts or models.

Anti-Patterns to Avoid

OpenAI's evaluation best-practice guidance is especially useful for spotting weak eval design. Keep these failure modes out of AvalAI release gates:

Anti-pattern	Why it fails	Better AvalAI pattern
Generic academic metrics only	BLEU, ROUGE, or perplexity can miss task-specific correctness, safety, and tool behavior	Add task-specific checks such as exact labels, JSON schema, citation accuracy, and tool-argument validity.
Biased or too-clean datasets	A small hand-picked set can overstate quality and miss production drift	Mix production logs, edge cases, multilingual rows, malformed inputs, and adversarial attempts.
Vibe-based approval	"Looks good" does not catch regressions or compare candidates reliably	Define pass/fail criteria before changing prompts, models, tools, or routing.
Automated graders without calibration	LLM judges and heuristic scores can drift away from reviewer intent	Compare grader decisions with human labels, keep disagreement cases, and update rubrics before trusting CI gates.
End-to-end score only	One aggregate score hides where a workflow failed	Evaluate boundaries separately: classification, retrieval, tool calls, final answer, and safety handling.

Prefer comparison-friendly tasks when possible. Pairwise choices, classification, scoring against a rubric, and tool-call accuracy are easier to grade consistently than open-ended "write a good answer" prompts.

Continuous Evaluation Flywheel

Treat evals as a living release system, not a one-time benchmark:

1. Instrument: log request IDs, model IDs, prompt versions, tool calls, retrieved document IDs, token usage, latency, and user-visible failures.
2. Mine: convert production misses, support tickets, red-team findings, and reviewer disagreements into new eval rows.
3. Calibrate: compare automated grader decisions with a small human-labeled batch before trusting a new threshold.
4. Gate: run smoke evals on every pull request and full evals before model, prompt, retrieval, or tool-schema changes.
5. Refresh: retire duplicate rows, keep hard edge cases, and add new failure modes as your product and model mix evolve.

For each row, keep metadata such as case_id, source, risk_level, expected_behavior, owner, and added_after_incident. That context helps future maintainers understand why a row exists and whether a candidate regression is acceptable.

Compare Providers and Deployment Paths

OpenAI's external-model eval guidance separates native models, third-party models, and custom endpoints. Use the same mental model in AvalAI, but run it with local/CI evals instead of a hosted /v1/evals object:

• Keep the dataset and rubric fixed while swapping only model, endpoint route, or provider-specific options.
• Label each run with provider path, endpoint (/v1/responses, /v1/chat/completions, or native route), feature flags, region if applicable, and service tier.
• Treat tool support as a separate dimension. A model can pass text quality evals and still fail if function calls, MCP, web/file search, or streaming differ on the candidate route.
• Include privacy and safety assertions when data crosses provider boundaries: refusal behavior, prompt-injection handling, source leakage, and redacted logging.
• Promote a candidate only after accuracy, latency, cost, quota headroom, and safety gates are all acceptable for that deployment path.

This keeps provider comparison portable across AvalAI model routes and avoids depending on an external hosted eval platform's model picker or account-specific third-party catalog.

Make the Dataset Contract Explicit

OpenAI’s Evals API separates the row schema (data_source_config) from the scoring rules (testing_criteria). Keep the same separation in local AvalAI evals so your tests remain portable:

{ "item": { "ticket_text": "My monitor will not turn on.", "correct_label": "Hardware" } }
{ "item": { "ticket_text": "The VPN client crashes after login.", "correct_label": "Software" } }
{ "item": { "ticket_text": "Can you recommend lunch near the office?", "correct_label": "Other" } }

• ticket_text is the model input you template into Chat Completions or Responses.
• correct_label is the human-reviewed ground truth, equivalent to OpenAI’s {{ item.correct_label }} pattern.
• The generated answer is the local equivalent of {{ sample.output_text }}; keep it in the runner output so failures are debuggable.
• Treat the data file as production code: review labels, include edge cases, and add new production failures before changing prompts.

Add Annotation Fields

OpenAI’s dataset workflow treats generated outputs, ratings, and feedback as first-class evaluation data. Mirror that locally by adding columns that let humans and automated graders learn from the same rows:

Field	Purpose
item.*	Prompt variables and ground-truth fields, such as ticket_text, correct_label, or reference_answer.
sample.output_text	The candidate model output captured from Chat Completions or Responses.
human_rating	A reviewer label such as pass, fail, better_than_baseline, or needs_review.
output_feedback	Short reviewer critique explaining what failed and how the prompt, tool, or retrieval step should improve.
grader_score	Automated score from string checks, schema checks, semantic similarity, or an LLM judge.

For subjective or domain-heavy tasks, ask a subject-matter expert to annotate a small batch before trusting an automated grader. Use disagreement rows to improve the rubric instead of hiding them from the dataset.

Example: Classification Eval

This example tests a support-ticket classifier. The same test can be run against Chat Completions today and against Responses for models that support /v1/responses.

# evals/support-ticket-classifier.yaml
description: Classify IT support tickets
prompts:
  - |-
    Classify the support ticket as Hardware, Software, or Other.
    Return only the label.

    Ticket: {{ticket_text}}
providers:
  - id: openai:chat:gpt-5.5
    label: production
    config:
      apiHost: https://api.avalai.ir/v1
      apiKey: ${AVALAI_API_KEY}
  - id: openai:chat:gpt-5.4
    label: candidate
    config:
      apiHost: https://api.avalai.ir/v1
      apiKey: ${AVALAI_API_KEY}
tests:
  - vars:
      ticket_text: "My monitor will not turn on."
      correct_label: Hardware
    assert:
      - type: equals
        value: Hardware
  - vars:
      ticket_text: "The VPN client crashes after login."
      correct_label: Software
    assert:
      - type: equals
        value: Software

AVALAI_API_KEY=... promptfoo eval -c evals/support-ticket-classifier.yaml

This runs the same dataset against both providers. Promote the candidate only when the pass rate, latency, and cost are acceptable and no high-risk rows regress. For larger suites, save Promptfoo JSON/HTML output as CI artifacts and compare provider labels (production vs. candidate) across releases.

Chat Completions implementation

import os
from openai import OpenAI

client = OpenAI(
    api_key=os.environ["AVALAI_API_KEY"],
    base_url="https://api.avalai.ir/v1",
)


def classify_ticket(ticket: str) -> str:
    response = client.chat.completions.create(
        model=os.getenv("AVALAI_EVAL_MODEL", "gpt-5.5"),
        messages=[
            {
                "role": "system",
                "content": (
                    "Classify the support ticket as Hardware, Software, or Other. "
                    "Return only the label."
                ),
            },
            {"role": "user", "content": ticket},
        ],
        temperature=0,
    )
    return response.choices[0].message.content.strip()

Responses API version

Use this version when the selected model supports /v1/responses. messages moves to input, instructions move to instructions, and the final text is read from response.output_text.

import os
from openai import OpenAI

client = OpenAI(
    api_key=os.environ["AVALAI_API_KEY"],
    base_url="https://api.avalai.ir/v1",
)


def classify_ticket(ticket: str) -> str:
    response = client.responses.create(
        model=os.getenv("AVALAI_EVAL_MODEL", "gpt-5.5"),
        instructions=(
            "Classify the support ticket as Hardware, Software, or Other. "
            "Return only the label."
        ),
        input=ticket,
        temperature=0,
    )
    return response.output_text.strip()

• messages → input
• system/developer instructions → instructions
• choices[0].message.content → response.output_text
• keep the same dataset and assertions so Chat and Responses results are comparable.

What to Evaluate

Use architecture-specific tests rather than one generic score:

Architecture	Eval focus	Example question
Single-turn prompts	instruction following, classification, formatting	Does the output contain only an allowed label?
RAG workflows	context recall, citation accuracy, hallucination rate	Did the answer use retrieved context and avoid unsupported claims?
Tool workflows	tool selection, argument extraction, error handling	Did the model call the right function with the right JSON arguments?
Agents	handoff accuracy, stopping conditions, user safety	Did the agent route to the right specialist and avoid loops?
Multimodal apps	modality coverage, OCR/vision accuracy, refusal behavior	Did image reasoning preserve critical details?

For multi-step workflows, evaluate each boundary separately before relying on an end-to-end score. A support workflow might need one eval for intent classification, one for order-ID extraction, one for tool argument correctness, and one for the final customer-facing answer. This makes failures easier to debug than a single “overall quality” score.

Edge-Case Coverage

Add rows that represent how users and tools actually fail in production:

• multilingual, mixed-language, typo-heavy, and very short inputs;
• malformed JSON, XML, Markdown, CSV, or copy-pasted logs;
• conflicting user requests that try to override developer instructions;
• long conversations where the relevant fact appears in the middle or near the start;
• tool results with ambiguous field names, empty results, stale data, or recoverable errors;
• repeated tool calls, wrong argument extraction, circular handoffs, and refusal paths.

When an incident happens in production, add the failing case to the eval dataset before changing the prompt, model, tool schema, or retrieval logic.

LLM-as-Judge

For subjective outputs, use a judge model only after you define a rubric and calibrate against human labels.

• Prefer pairwise comparison or pass/fail over vague 1–10 scores.
• Control for verbosity bias by requiring similar response lengths.
• Rotate response order in pairwise tests to reduce position bias.
• Validate judge agreement with human reviewers before trusting it in CI.
• Use a strong model such as gpt-5.5 for initial judging, then test cheaper models if the rubric is stable.

Choose the Right Grader

OpenAI’s grader guidance maps well to local AvalAI evals even without hosted /v1/evals. Pick the smallest grader that proves the behavior:

• Use string checks for exact labels, IDs, enum values, and required phrases.
• Use JSON schema checks for structured outputs before scoring semantic quality.
• Use semantic similarity when small wording differences are acceptable.
• Use an LLM judge only for rubric-based quality, safety, helpfulness, or partial credit.
• For tool workflows, grade both the selected tool name and the JSON arguments; use semantic grading for arguments such as addresses, dates, or normalized units that may have equivalent forms.

For a dedicated local implementation pattern, see Local Graders for AvalAI Evals.

Combine Evaluators Deliberately

OpenAI's evaluation guidance separates metric-based checks, human review, and model graders because each catches different failure modes. In AvalAI CI, combine them by risk:

Eval layer	Use it for	Release gate
Deterministic checks	labels, JSON validity, required citations, tool names, argument shape	must pass on every PR
Semantic or retrieval scores	answer similarity, context recall, context precision, citation grounding	threshold plus sampled human review
LLM judge	helpfulness, policy nuance, partial credit, comparative quality	calibrated against human labels before blocking CI
Human review	high-impact domains, new rubrics, grader disagreement, production incidents	required for risky launches

Keep evaluator outputs separate in the run artifact. A single blended score is useful for dashboards, but separate columns make it obvious whether a regression came from retrieval, tool selection, formatting, safety, or final answer quality.

Evaluate Agents by Trajectory

For agentic workflows, do not grade only the final answer. Capture the whole trajectory so regressions are visible. See Agent Workflow Evaluations for a dedicated trace and CI pattern:

• Plan: Did the agent identify the correct goal, constraints, and stop condition?
• Tool sequence: Did it call the right tools in the right order, with safe arguments?
• State handling: Did it preserve previous_response_id, retrieved context, tool outputs, and user constraints across turns?
• Recovery: Did it handle tool errors, empty retrieval results, refusals, and timeouts without looping?
• Final answer: Did it explain the result, cite evidence when needed, and avoid unsupported claims?

Log typed Responses output items, tool-call arguments, tool outputs, request IDs, and pass/fail reasons. Add production failures to the dataset before changing prompts so the eval catches the original bug.

CI and Release Checklist

• Run fast smoke evals on every pull request.
• Run the full suite before changing model IDs, prompts, tools, or retrieval logic.
• Track model, endpoint, prompt version, request ID, latency, input/output tokens, cached tokens, and pass/fail reason.
• Add new production failures to the eval dataset before fixing the prompt.
• Gate risky releases on human review for safety, compliance, or financial decisions.

When Hosted Evals Become Available

When AvalAI launches hosted eval endpoints, keep your local/CI suite as the source of truth and use hosted evals for centralized runs, history, and reporting. Hosted evals should complement—not replace—version-controlled datasets and CI checks. Validate the actual AvalAI schema at launch instead of assuming parity with any OpenAI hosted eval API shape.

Related Guides

• Promptfoo Evals with AvalAI
• Local Graders for AvalAI Evals
• Agent Workflow Evaluations
• Red Teaming AI Applications
• Production Best Practices
• Structured Outputs
• Function Calling
• Safety Best Practices