Local Graders for AvalAI Evals

Graders are automated checks that compare a model output with a reference answer, rubric, schema, or expected tool call. OpenAI’s hosted grader concepts are useful, but AvalAI does not currently expose a hosted /v1/evals or grader API. Treat this guide as a local and CI pattern for normal AvalAI API calls.

Warning

Feature Not Implemented!

This functionality is currently under development and not yet available in AvalAI. We’ll announce its release through our official channels. Stay tuned for updates!

OpenAI is also deprecating the hosted Evals and Graders platform it documents, so keep your datasets, grader code, rubrics, and thresholds portable in your repository. For OpenAI-hosted evals, existing evals become read-only on October 31, 2026 and the platform is scheduled to shut down on November 30, 2026; treat those dates as OpenAI platform context, not AvalAI API availability.

Endpoint migration note

OpenAI grader examples may call hosted endpoints such as /v1/fine_tuning/alpha/graders/validate or /v1/fine_tuning/alpha/graders/run. Do not rewrite those examples to https://api.avalai.ir/v1/... unless AvalAI explicitly announces compatible hosted grader routes. For AvalAI today, validate graders by running local fixtures in CI against normal model outputs.

Core Shape

A grader should take:

• item: the human-reviewed test row, such as a prompt, reference answer, expected JSON, or expected tool call.
• sample: the model output you generated through /v1/responses, /v1/chat/completions, or a provider-native route.
• score: a number from 0 to 1, plus a short reason that helps debug failures.

Use the same names OpenAI uses—item.reference_answer, sample.output_text, sample.output_json, and sample.output_tools—even in local files. That makes migration easier if hosted AvalAI evals become available later.

Migration Map from OpenAI Hosted Graders

When adapting OpenAI grader material to AvalAI, keep the concept but replace the hosted execution surface:

OpenAI hosted concept	AvalAI-safe implementation today
grader JSON object	Version-controlled Python/JavaScript function or Promptfoo assertion.
{{ item.reference_answer }}	Dataset field from JSONL, CSV, YAML, or your test fixture.
{{ sample.output_text }}	Text normalized from /v1/responses output_text or Chat Completions content.
validate endpoint	Unit test that runs the grader against known pass/fail fixtures.
run endpoint	CI job that generates samples with AvalAI and writes score artifacts.
Hosted report URL	Promptfoo report, pytest JSON/JUnit output, or your observability dashboard.

This keeps the scoring logic portable and avoids coupling releases to a deprecated or unavailable hosted API.

Portable Sample Contract

OpenAI grader templates separate dataset fields from generated outputs. Mirror that contract in local files so your graders stay portable:

• item.*: fields from the JSONL row, such as item.ticket, item.correct_label, item.reference_answer, or item.expected_tool.
• sample.output_text: normalized text from response.output_text or choices[0].message.content.
• sample.output_json: parsed structured output when you require JSON or schema-constrained responses.
• sample.output_tools: tool calls from Responses output items or Chat Completions message.tool_calls.
• sample.choices: optional raw Chat Completions choices for debugging migrations.
• sample.output_audio: optional metadata or transcripts for audio evaluations.

Keep the normalized sample.* object small and stable. Store raw provider responses separately when you need audit logs, but grade against the portable fields.

Template Variables and Grader Types

OpenAI grader templates use double braces such as {{ item.reference_answer }} and {{ sample.output_text }}. Keep the same two namespaces locally:

• item.* comes from the dataset row or human-labeled reference.
• sample.* comes from the generated output you are grading.

Map the official grader taxonomy to local checks:

• string_check: implement exact or substring checks. Useful operations are eq, neq, like, and ilike.
• text_similarity: implement fuzzy, BLEU/GLEU, ROUGE, cosine, or embedding-based similarity for open-ended references.
• score_model: call a fixed AvalAI judge model with a rubric and return a numeric score in a defined range.
• python: run deterministic local code for business rules, numeric checks, dates, schema-normalized fields, or custom scoring.
• multi: combine independent sub-scores with a clear formula such as (tool_name + arguments) / 2.

Choose the Smallest Grader

Grader	Use for	Avoid when
String check	exact labels, IDs, enum values, required phrases	wording can vary without changing correctness
JSON schema	structured extraction and tool arguments	semantic quality matters after schema validity
Text similarity	summaries, paraphrases, partial lexical overlap	exact values, IDs, or money amounts are required
LLM judge	subjective quality, helpfulness, safety, style, partial credit	a deterministic check can prove the behavior
Custom Python	business rules, numeric ranges, normalized dates, multi-field scoring	the grader needs network access or secret data
Multi-grader	outputs that need several independent checks	one failure should immediately fail the whole row

Start deterministic. Add an LLM judge only after string, schema, and business-rule graders cannot express the quality you need.

Generate Samples with AvalAI

Keep your existing Chat Completions tests, then add a Responses variant for models that support /v1/responses.

import os
from openai import OpenAI

client = OpenAI(
    api_key=os.environ["AVALAI_API_KEY"],
    base_url="https://api.avalai.ir/v1",
)


def classify_ticket_chat(ticket: str) -> str:
    response = client.chat.completions.create(
        model=os.getenv("AVALAI_EVAL_MODEL", "gpt-5.5"),
        messages=[
            {
                "role": "developer",
                "content": "Classify the ticket as Hardware, Software, or Other. Return only the label.",
            },
            {"role": "user", "content": ticket},
        ],
        temperature=0,
    )
    return response.choices[0].message.content.strip()

Responses API version

import os
from openai import OpenAI

client = OpenAI(
    api_key=os.environ["AVALAI_API_KEY"],
    base_url="https://api.avalai.ir/v1",
)


def classify_ticket_responses(ticket: str) -> str:
    response = client.responses.create(
        model=os.getenv("AVALAI_EVAL_MODEL", "gpt-5.5"),
        instructions="Classify the ticket as Hardware, Software, or Other. Return only the label.",
        input=ticket,
        temperature=0,
    )
    return response.output_text.strip()

• messages → input
• developer or system policy → instructions
• choices[0].message.content → response.output_text
• for tools, inspect response.output items instead of assuming a single text output.

Example Local Grader

Store datasets as JSONL and grade generated samples in CI:

{"item":{"ticket":"My monitor will not turn on.","correct_label":"Hardware"}}
{"item":{"ticket":"The VPN client crashes after login.","correct_label":"Software"}}
{"item":{"ticket":"Can you recommend lunch near the office?","correct_label":"Other"}}

def normalize_label(value: str) -> str:
    return value.strip().lower().replace(".", "")


def grade_label(sample: dict, item: dict) -> dict:
    expected = normalize_label(item["correct_label"])
    actual = normalize_label(sample["output_text"])
    passed = actual == expected
    return {
        "score": 1.0 if passed else 0.0,
        "reason": (
            "exact label match" if passed else f"expected {expected}, got {actual}"
        ),
    }

For tool calls, grade both the tool name and arguments. Use exact checks for the tool name and schema or semantic checks for arguments that can have equivalent forms, such as dates, addresses, currencies, or normalized units.

Tool-Call and Multi-Grader Patterns

Tool-call evals usually need more than one score. Check that the model chose the right tool, then separately check whether the arguments are correct.

import json


def grade_tool_call(sample: dict, item: dict) -> dict:
    calls = sample.get("output_tools") or []
    if not calls:
        return {"score": 0.0, "reason": "no tool call"}

    call = calls[0].get("function", {})
    expected = item["expected_tool"]

    name_score = 1.0 if call.get("name") == expected["name"] else 0.0

    try:
        actual_args = json.loads(call.get("arguments") or "{}")
    except json.JSONDecodeError:
        actual_args = {}

    argument_score = 1.0 if actual_args == expected["arguments"] else 0.0
    score = 0.5 * name_score + 0.5 * argument_score
    return {
        "score": score,
        "reason": f"name={name_score}, arguments={argument_score}",
    }

Exact JSON comparison is useful for IDs and enums, but can under-reward equivalent values like 1 vs 1.0, CA vs California, or alternate date formats. For flexible arguments, normalize first or use a semantic grader that checks the parsed fields.

Local Python Grader Rules

Treat grader code as production test code:

• keep grade(sample, item) deterministic, version-controlled, and reviewed with the prompt or model change;
• do not allow network access, API keys, or secret reads inside CI graders;
• bound runtime and memory so one bad sample cannot hang the suite;
• return a valid float score or a {score, reason} object, depending on your local runner;
• fail safely: exceptions, missing fields, NaN, or invalid scores should become 0.0 with a debug reason.

OpenAI's hosted Python graders document useful sandbox defaults: no network access, bounded runtime, bounded memory/disk, and a small uploaded source size. Mirror those constraints in CI even though you are not using OpenAI's hosted grader runtime; they prevent graders from turning into hidden production jobs.

LLM Judge Guidelines

Use an AvalAI model as a judge when the output quality is subjective:

• calibrate the judge against human-labeled examples before using it in CI;
• prefer pass/fail or pairwise comparison over vague 1–10 scores;
• rotate answer order in pairwise tests to reduce position bias;
• control response length so the judge does not prefer verbose answers;
• freeze the judge model, prompt, temperature, and rubric for each release;
• keep disagreement cases and edge cases in the dataset.

Guard against reward hacking: if a candidate improves the grader score but looks worse to human reviewers, fix the grader before shipping the prompt, model, or tool change.

Reward-Hacking Checks

OpenAI’s grader guidance calls out “grader hacking” as a failure mode: the candidate system can learn to satisfy the scoring rule without improving the real task. Add a small adversarial pack beside each production grader:

• Shortcut answers: outputs that repeat rubric keywords but do not solve the task.
• Prompt-injection answers: outputs that ask the judge to ignore the rubric or award full credit.
• Overlong answers: verbose outputs that look helpful but hide missing facts or unsafe tool calls.
• Schema-only passes: JSON that validates but contains wrong IDs, dates, money amounts, or citations.
• Human disagreement rows: examples where reviewers reject an answer even though the automated score was high.

If these cases score well, do not lower the threshold to compensate. Tighten the grader, add deterministic checks before the LLM judge, or require human review for that release gate.

Calibrate Before CI Blocks

OpenAI's grader guidance recommends testing the grader itself with known answer rankings before trusting it. Keep a tiny calibration pack beside every LLM judge or semantic grader:

{"id":"perfect","reference_answer":"Reset the API key from the dashboard.","candidate":"Reset the API key from the dashboard.","expected_order":1}
{"id":"partial","reference_answer":"Reset the API key from the dashboard.","candidate":"Open the dashboard and rotate credentials.","expected_order":2}
{"id":"wrong","reference_answer":"Reset the API key from the dashboard.","candidate":"Contact billing support for an invoice.","expected_order":3}

Before a grader can block CI, verify that it ranks perfect > partial > wrong, rejects a prompt-injection attempt inside the candidate answer, and explains the failure reason in a field your CI artifact preserves. Re-run this calibration when you change the judge model, rubric, temperature, prompt, or answer length limits.

Related Guides

• Evaluations
• Agent Workflow Evals
• Prompt Engineering
• Structured Outputs
• Function Calling
• OpenAI Graders
• OpenAI Evaluation Best Practices