Promptfoo Evals with AvalAI

Use Promptfoo when you want portable regression tests for prompts, model upgrades, routing logic, or agent workflows. This example keeps the eval beside your application code and calls AvalAI through the OpenAI-compatible SDK.

This guide is adapted from the official OpenAI Cookbook and the OpenAI Cookbook GitHub repository, especially the Promptfoo migration and SchemaFlow eval examples. AvalAI-specific changes are the API key name, base URL, and model IDs.

When to Use This Pattern

Use a local Promptfoo eval when:

• you are comparing two AvalAI models before changing production traffic
• you are tightening a prompt and want to catch regressions
• you need CI to fail when output quality drops
• the native AvalAI Evals API is not available for your account yet

For hosted AvalAI eval features, see Evaluations. For local and CI workflows, the pattern below works with normal API calls.

Translate OpenAI Eval Concepts to Promptfoo

OpenAI evals separate the dataset schema from the criteria that judge model output. Keep the same separation in Promptfoo:

OpenAI eval concept	Promptfoo equivalent	AvalAI note
data_source_config	tests[].vars plus a documented fixture shape	Keep human labels such as expected_label beside the user input.
testing_criteria	assert blocks	Start with deterministic checks before adding LLM-as-judge assertions.
{{ item.correct_label }}	{{expected_label}} or an assertion value	Treat this as ground truth reviewed by a human.
{{ sample.output_text }}	provider return value {"output": ...}	For /v1/responses, return response.output_text.

Write the dataset contract before writing prompts. For this example, each test case has one ticket string and one expected category.

Install

npm install -g promptfoo
python3 -m pip install openai

export AVALAI_API_KEY="your-avalai-api-key"
export AVALAI_BASE_URL="https://api.avalai.ir/v1"

Create a Python Provider

Create evals/support-ticket/avalai_eval_provider.py:

import os
from openai import OpenAI


MODEL = os.getenv("AVALAI_EVAL_MODEL", "gpt-5.5")

client = OpenAI(
    api_key=os.environ["AVALAI_API_KEY"],
    base_url=os.getenv("AVALAI_BASE_URL", "https://api.avalai.ir/v1"),
)


def call_api(prompt, options, context):
    vars_ = (context or {}).get("vars", {})
    ticket = vars_.get("ticket", prompt)

    response = client.chat.completions.create(
        model=MODEL,
        temperature=0,
        messages=[
            {
                "role": "system",
                "content": (
                    "Classify the support ticket as exactly one of: "
                    "Hardware, Software, Billing, Account, Other. "
                    "Return only the label."
                ),
            },
            {"role": "user", "content": ticket},
        ],
    )

    return {"output": response.choices[0].message.content.strip()}

Promptfoo calls call_api() for each test case. The custom provider keeps AvalAI credentials on the machine running the eval and avoids exposing keys in YAML.

Responses API provider variant

For new eval providers, use /v1/responses when the selected model supports it. Put the stable classifier instruction in instructions, pass the ticket as input, and return response.output_text.

def call_api(prompt, options, context):
    vars_ = (context or {}).get("vars", {})
    ticket = vars_.get("ticket", prompt)

    response = client.responses.create(
        model=MODEL,
        temperature=0,
        instructions=(
            "Classify the support ticket as exactly one of: "
            "Hardware, Software, Billing, Account, Other. "
            "Return only the label."
        ),
        input=ticket,
        store=False,
    )

    return {"output": response.output_text.strip()}

• messages → instructions plus input
• choices[0].message.content → response.output_text
• set store=False for local CI evals unless you need to retrieve the response later

Add Test Cases

Create evals/support-ticket/promptfooconfig.yaml:

# yaml-language-server: $schema=https://promptfoo.dev/config-schema.json
description: "AvalAI support-ticket classification regression eval"

prompts:
  - "{{ticket}}"

providers:
  - id: "file://avalai_eval_provider.py"
    label: "avalai-gpt-5.5"
    config:
      pythonExecutable: "python3"

tests:
  - description: "monitor power issue"
    vars:
      ticket: "My monitor will not turn on after I changed desks."
    assert:
      - type: equals
        value: "Hardware"

  - description: "invoice question"
    vars:
      ticket: "Why was my card charged twice this month?"
    assert:
      - type: equals
        value: "Billing"

  - description: "password reset"
    vars:
      ticket: "I cannot sign in and need to reset my password."
    assert:
      - type: equals
        value: "Account"

Add Judge Criteria Carefully

Use exact assertions when the output has a small finite set, like the support-ticket labels above. Add an LLM-as-judge only when the behavior is semantic, multi-step, or too broad for equals and regex. Keep judge criteria short, observable, and tied to the dataset fields.

- description: "ambiguous issue should be escalated"
    vars:
      ticket: "The dashboard looks wrong and my bill changed after an upgrade."
    assert:
      - type: llm-rubric
        value: >-
          The answer must choose either Billing or Other, explain no extra
          facts, and must not invent account details that are not in the ticket.

For production evals, calibrate judge-based checks with a small human-reviewed set before making CI fail on them.

Run Locally

cd evals/support-ticket
promptfoo validate config -c promptfooconfig.yaml
promptfoo eval -c promptfooconfig.yaml --no-cache
promptfoo view

Use --no-cache while developing the eval. Remove it when you want repeated local runs to be faster.

Compare Models

To compare models, run the same config with a different AVALAI_EVAL_MODEL value and compare the saved Promptfoo results.

AVALAI_EVAL_MODEL="gpt-5.4" promptfoo eval -c promptfooconfig.yaml --no-cache
AVALAI_EVAL_MODEL="gpt-5.5" promptfoo eval -c promptfooconfig.yaml --no-cache

Keep the eval dataset fixed while comparing models. Change one thing at a time: model, prompt, tool schema, or retrieval context.

Add to CI

name: prompt-evals

on:
  pull_request:
  workflow_dispatch:

jobs:
  evals:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with:
          node-version: 22
      - uses: actions/setup-python@v5
        with:
          python-version: "3.11"
      - run: npm install -g promptfoo
      - run: python3 -m pip install openai
      - run: promptfoo eval -c evals/support-ticket/promptfooconfig.yaml --no-cache
        env:
          AVALAI_API_KEY: ${{ secrets.AVALAI_API_KEY }}
          AVALAI_BASE_URL: https://api.avalai.ir/v1

Make the Eval Useful

• Keep examples close to real user inputs, including typos and short messages.
• Add negative cases that should refuse, escalate, or return Other.
• Prefer deterministic assertions such as equals, contains, or regex before adding LLM-as-judge checks.
• Define the expected output contract first, then choose assertions that prove that contract.
• Keep human-labeled ground truth stable while comparing prompts or models.
• Store past results when changing production prompts so reviewers can see what improved and what regressed.
• Use small evals in every pull request and larger eval suites before releases.

Source References

• OpenAI Cookbook
• openai/openai-cookbook on GitHub
• OpenAI Cookbook Promptfoo migration guide
• OpenAI Cookbook SchemaFlow example