Offline Regression Checks for Evidence-Based Workflows

Compare a baseline and a proposed change on the same support, product-feedback, and study cases before considering efficiency. This companion to evidence-based workflows separates mechanical validation from a recorded human judgment: correct JSON and genuine quotes can still support a misleading summary.

This is a runnable offline scorer demonstration, using author-written drafts and illustrative human verdicts. Its results do not measure any real model. The label candidate_claimed_cheaper_faster is synthetic; it supplies no price or latency evidence.

1. Save the two scripts together

Use Python 3.10 or newer. Save the complete workflow.py from the previous example and the script below as eval_workflow.py in the same private directory. Only Python's standard library is needed. Importing workflow.py reuses its validators; its guarded live entry point is not called. This evaluator makes no API calls, reads no credentials, and writes no reports automatically.

Each case has independently written expected evidence references and a review question. Do not construct those expectations from the candidate's answer. A human checks whether the summary follows from the source, preserves uncertainty and minority views, and avoids promises or fabricated conclusions. A recorded verdict is pass, fail, or needs_review, with a reason and a digest linking it to the source records and draft. The digest detects changes; it does not authenticate a reviewer or establish that a verdict is correct.

2. Copy the complete evaluator

import hashlib
import json
from copy import deepcopy

from workflow import DEMOS, read_completion, validate_result

CASES = {
    "support": {
        "expected": {
            ("T1", "I was charged twice for order A42."),
            ("T2", "CSV export fails when I select the last 30 days."),
        },
        "rubric": "Identify both issues without approving a refund or promising a fix.",
    },
    "feedback": {
        "expected": {
            ("F1", "We need CSV export to prepare our weekly report."),
            ("F2", "CSV export would remove our manual reporting step."),
            ("F3", "Dark mode would be useful at night."),
        },
        "rubric": "Preserve the minority theme; do not infer willingness to pay.",
    },
    "study": {
        "expected": {
            ("S1", "retrieving information from memory without looking at the notes"),
            ("S2", "Spaced practice distributes study sessions over time."),
        },
        "rubric": "Explain both concepts without inventing an optimal interval.",
    },
}


def review_digest(case_id, draft):
    material = {"records": DEMOS[case_id][0], "draft": draft}
    encoded = json.dumps(material, sort_keys=True, ensure_ascii=False).encode("utf-8")
    return hashlib.sha256(encoded).hexdigest()


def illustrative_runs():
    baseline, candidate = [], []
    reasons = {
        "support": "Both issues are identified and the refund remains undecided.",
        "feedback": "The CSV theme and minority dark-mode request are preserved.",
        "study": "Both concepts are supported and the study interval remains unknown.",
    }
    for case_id in CASES:
        draft = deepcopy(DEMOS[case_id][1])
        response = {"choices": [{
            "finish_reason": "stop",
            "message": {"content": json.dumps(draft), "refusal": None},
        }]}
        baseline.append({
            "case_id": case_id,
            "response": response,
            "human_review": {
                "verdict": "pass", "reason": reasons[case_id],
                "sha256": review_digest(case_id, draft),
            },
        })
        changed = deepcopy(draft)
        verdict, reason = "pass", reasons[case_id]
        if case_id == "support":
            changed["findings"][0]["summary"] = "A refund for order A42 is approved."
            verdict = "fail"
            reason = "The genuine quote reports a charge; it does not approve a refund."
        elif case_id == "study":
            verdict = "needs_review"
            reason = "Illustrative reviewer has not completed the semantic check."
        changed_response = deepcopy(response)
        changed_response["choices"][0]["message"]["content"] = json.dumps(changed)
        candidate.append({
            "case_id": case_id,
            "response": changed_response,
            "human_review": {
                "verdict": verdict, "reason": reason,
                "sha256": review_digest(case_id, changed),
            },
        })
    return baseline, candidate


def evaluate_run(rows, label):
    identifiers = [row.get("case_id") if isinstance(row, dict) else None
                   for row in rows] if isinstance(rows, list) else []
    coverage_ok = (len(identifiers) == len(CASES)
                   and all(isinstance(item, str) for item in identifiers)
                   and set(identifiers) == set(CASES))
    results = []
    for case_id, case in CASES.items():
        matches = [row for row in rows if isinstance(row, dict)
                   and row.get("case_id") == case_id] if isinstance(rows, list) else []
        mechanical, human = "fail", "needs_review"
        mechanical_reason = "Case is missing or duplicated."
        human_reason = "A current human verdict is required."
        if len(matches) == 1:
            row = matches[0]
            try:
                response = row.get("response")
                if (not isinstance(response, dict) or not isinstance(response.get("choices"), list)
                        or not response["choices"] or not isinstance(response["choices"][0], dict)
                        or not isinstance(response["choices"][0].get("message"), dict)):
                    raise ValueError("Malformed completion envelope.")
                draft = read_completion(response)
                validate_result(draft, DEMOS[case_id][0])
                observed = {(item["id"], item["quote"])
                            for finding in draft["findings"] for item in finding["evidence"]}
                if not case["expected"].issubset(observed):
                    raise ValueError("Required evidence reference is missing.")
                mechanical, mechanical_reason = "pass", "Schema, quotes and expected references match."
                review = row.get("human_review")
                if (isinstance(review, dict)
                        and review.get("verdict") in ("pass", "fail", "needs_review")
                        and isinstance(review.get("reason"), str) and review["reason"].strip()
                        and review.get("sha256") == review_digest(case_id, draft)):
                    human, human_reason = review["verdict"], review["reason"]
            except ValueError as error:
                mechanical_reason = str(error)
        results.append({
            "case_id": case_id, "mechanical": mechanical, "human": human,
            "mechanical_reason": mechanical_reason, "human_reason": human_reason,
            "rubric": case["rubric"],
        })
    return {"label": label, "coverage": "pass" if coverage_ok else "fail", "cases": results}


def compare_runs(baseline, candidate):
    runs = [evaluate_run(baseline, "baseline_fixture"),
            evaluate_run(candidate, "candidate_claimed_cheaper_faster")]
    cases = [case for run in runs for case in run["cases"]]
    failed = (any(run["coverage"] == "fail" for run in runs)
              or any(case["mechanical"] == "fail" or case["human"] == "fail" for case in cases))
    pending = any(case["human"] == "needs_review" for case in cases)
    return {
        "mode": "offline_fixture",
        "status": "blocked" if failed else "needs_human_review",
        "fixture_quality_gate": "fail" if failed else "pending" if pending else "pass",
        "runs": runs,
    }


def main():
    baseline, candidate = illustrative_runs()
    print(json.dumps(compare_runs(baseline, candidate), ensure_ascii=False, indent=2))


if __name__ == "__main__":
    main()

3. Run and read the comparison

python3 -B eval_workflow.py

The full JSON reports every case and the review reasons. The expected result is:

Synthetic run	Case	Mechanical check	Recorded human verdict
baseline_fixture	support	pass	pass
baseline_fixture	feedback	pass	pass
baseline_fixture	study	pass	pass
candidate_claimed_cheaper_faster	support	pass	fail
candidate_claimed_cheaper_faster	feedback	pass	pass
candidate_claimed_cheaper_faster	study	pass	needs_review

The comparison has "status": "blocked" and "fixture_quality_gate": "fail". The candidate's support summary approves a refund that the source never approved, despite citing an exact quote. The study review remains unfinished. An aggregate score would hide these case-level differences.

Unknown evidence IDs, invented quotes, omitted expected references, refusals, truncation, and malformed responses fail the mechanical check. Both runs must cover every case exactly once; missing, duplicate, or unknown case IDs block the comparison. Missing, invalid, or stale human review stays needs_review. Even if every fixture passes, the comparison status is needs_human_review: any candidate change still needs review. Neither status authorizes deployment, payment, customer replies, or another action.

Try changing one summary in a returned run without updating its review record: the digest no longer matches, and the human check becomes pending. Change an evidence ID to T99: the mechanical check fails. To try a successful fixture gate, call compare_runs(baseline, deepcopy(baseline)) using the functions above; it still requires candidate review. These edits test the scorer, not model capability.

4. Design a real evaluation separately

Use development cases to refine prompts, validators, and reviewer instructions. Keep held-out cases separate from those iterations; evaluate a candidate on them after its design is fixed. These three public, fully visible cases are development fixtures, not a held-out set. Build representative authorized cases for both languages, including ambiguity, missing information, minority feedback, unsafe instructions, and failures. Have reviewers write expected references and semantic criteria from source records before seeing candidate output.

For later live validation, preserve the same case IDs and source records across baseline and candidate, and record the exact model ID, prompt/version, output, refusal or failure, and human verdict. Repeat cases with the runner's response-replay cache disabled so each trial sends a fresh request. Provider prompt caching is separate; record its conditions when comparing latency and usage. One replayed response or one favorable run cannot establish an improvement. Use Promptfoo with AvalAI for a separate live harness after reviewing its setup and costs.

Assess quality first. Only then compare observed latency distributions and actual AvalAI usage accounting, including failed attempts and any manually approved retries. Record the sample size, measurement conditions, and unresolved review cases. Upstream notebook prices, token counts alone, synthetic labels, and this script's runtime do not establish AvalAI dollar costs or savings. Review current pricing and rate limits before planning live runs. Keep reports private because they contain source text; separate client or classroom data before evaluation.

Sources and validation boundary

Sources reviewed on 2026-10-12. This adaptation uses quality-before-efficiency comparisons and shared case/trace evidence from the official OpenAI Cookbook and openai/openai-cookbook repository:

• Evaluation best practices.
• Optimizing agents for cost and quality.
• Harness-aware plugin evaluations.

The transferable scoring pattern is provider-neutral. Hosted Agents, Batch, MCP integrations, and upstream efficiency claims are not implied AvalAI capabilities. The published code is checked offline against fixed fixtures and rejection cases; no AvalAI key, live model response, or performance measurement validates this example.