Optimizing LLM Accuracy

Maximize correctness and consistent behavior when working with Large Language Models (LLMs).

Introduction

Optimizing LLMs is challenging for several key reasons:

• Knowing how to start optimizing accuracy
• When to use what optimization method
• What level of accuracy is good enough for production

This guide provides a mental model for optimizing LLMs for accuracy and behavior, exploring methods like prompt engineering, retrieval-augmented generation (RAG), evals, model selection, and fine-tuning planning.

For AvalAI apps, treat accuracy work as a release loop rather than a one-time prompt rewrite:

1. Define the user-visible failure and the business cost of that failure.
2. Build a small eval set with representative prompts, expected behavior, and unacceptable answers.
3. Diagnose whether failures come from missing context, inconsistent behavior, or unsafe tool/retrieval flows.
4. Change one lever at a time, then rerun the same evals before shipping.

AvalAI Optimization Flywheel

OpenAI’s model optimization guidance frames quality work as a continuous loop of evals, prompt engineering, and fine-tuning. On AvalAI, use the same loop, but keep hosted-training and hosted-eval assumptions route-dependent:

1. Baseline with evals: run a local or CI eval before changing the prompt, model, retrieval setup, or tool schema.
2. Improve the request: tighten developer instructions, add only the context the model needs, and add few-shot examples when the desired behavior is hard to describe.
3. Compare model and endpoint choices: test /v1/responses, /v1/chat/completions, and provider-native routes only where the selected model supports them.
4. Prepare training data only after eval evidence: collect high-quality input/output examples and hold-out rows, but do not assume hosted fine-tuning is available until AvalAI publishes supported routes.
5. Repeat on production failures: every high-value failure should become an eval row before the next prompt or routing change.

This turns optimization from “try a better prompt” into an evidence loop: dataset → baseline → one change → comparison → rollout decision.

LLM Optimization Context

Rather than viewing optimization as a linear process, it's more useful to think of it as a matrix with two key dimensions:

• Context optimization: Addresses when the model lacks knowledge, has outdated information, or needs proprietary data. This maximizes response accuracy.
• LLM optimization: Addresses inconsistent results, formatting issues, incorrect tone/style, or inconsistent reasoning. This maximizes consistency of behavior.

In practice, optimization becomes an iterative process of evaluation, hypothesis, application, and reassessment.

Use this decision table before adding complexity:

Symptom	First lever	AvalAI implementation
The model lacks private, recent, or domain-specific facts	Context optimization	Add RAG, file inputs, web/search tools, or explicit reference text.
The model has the facts but answers in the wrong format or style	LLM behavior optimization	Improve instructions, examples, schemas, and reasoning.effort / text.verbosity when using /v1/responses.
The same mistake repeats across many similar inputs	Stronger examples now; fine-tuning planning later	Collect production-like examples, add stronger few-shot/schema guidance, and keep a held-out eval set for future hosted training.
Long context seems to hide important facts	Retrieval and context layout	Test different context sizes and chunk positions instead of assuming a larger context window fixes accuracy.

Prompt Engineering

Prompt engineering is typically the best starting point. For use cases like summarization, translation, and code generation, it may be the only method needed to reach production-level accuracy.

Starting with prompt engineering forces you to define what accuracy means for your specific use case. By providing an input and evaluating whether the output meets your expectations, you'll gain insights into what further optimizations may be needed.

Optimization Strategies

• Write clear instructions: Be specific about the desired output format, tone, and constraints.
• Split complex tasks into simpler subtasks: Break down complex reasoning into step-by-step processes.
• Give LLMs time to "think": Encourage the model to work through problems methodically.
• Test changes systematically: Make controlled changes and measure their impact.
• Provide reference text: Include relevant information in the prompt when needed.
• Use external tools: Leverage tools for calculations, data retrieval, or verification.

Example: Language Correction Task

Adding few-shot examples to a basic prompt for Icelandic sentence correction improved BLEU scores from 62 to 70, demonstrating the value of showing the model examples of the desired behavior.

Evaluation

A good evaluation set with questions and ground truth answers is essential before moving to more advanced optimization methods. When you have 20+ examples and understand why failures occur, you have a solid baseline for further optimization.

Consider automating evaluation with:

• Metrics like ROUGE or BERTScore for quick comparisons
• Using a strong judge model with a scoring rubric, calibrated against human-reviewed examples

Judge Calibration Pattern

LLM-as-judge is useful for subjective quality, safety, helpfulness, and partial-credit tasks, but it can introduce position bias, verbosity bias, and rubric drift. Use it only after simpler checks fail to capture quality:

1. Start with string, enum, JSON schema, or tool-argument checks whenever the expected answer is deterministic.
2. Write a rubric with pass/fail or pairwise criteria before asking a judge model to score outputs.
3. Calibrate on a human-labeled set and record disagreement examples.
4. Rotate answer order in pairwise comparisons and constrain response length so the judge does not prefer longer answers by default.
5. Freeze the judge model, rubric, temperature, and prompt version for each eval run; if any of them changes, rerun the production baseline.

For production AvalAI workflows, keep evals close to how the app really runs:

• Include the same retrieved context format, tool outputs, schemas, and safety constraints that production uses.
• Track pass/fail reasons, not just an aggregate score, so each failure maps to a lever.
• Add real production failures to the eval set before changing the prompt.
• Run a quick smoke eval on every model, prompt, retrieval, or routing change, then a fuller suite before launch.
• Record the model ID, endpoint (/v1/responses, /v1/chat/completions, or provider-native route), temperature, retrieval settings, and routing rule for each eval run.

Understanding the Tools

When prompt engineering isn't enough, diagnose whether you're facing an in-context or learned memory problem:

• In-context memory problems: The model lacks necessary information to answer correctly. Solve with RAG by adding relevant context.
• Learned memory problems: The model needs consistent behavior patterns. In AvalAI today, solve first with clearer instructions, examples, structured outputs, tool constraints, and model routing; keep high-quality examples for future hosted fine-tuning when AvalAI announces supported routes.

These approaches are additive, not exclusive - they can be combined for optimal performance.

Retrieval-Augmented Generation (RAG)

RAG retrieves relevant content to augment your LLM's prompt before generating an answer, giving the model access to domain-specific context.

RAG applications can break down in two areas:

1. Retrieval issues: Supplying wrong or irrelevant context
2. LLM issues: The model misuses the correct context

Optimizing RAG requires tuning both the retrieval system and the LLM's instructions.

Fine-Tuning Planning

Fine-tuning continues training an LLM on a smaller, domain-specific dataset to:

• Improve model accuracy on specific tasks
• Improve efficiency (same accuracy with fewer tokens or smaller models)

AvalAI hosted fine-tuning is not currently advertised by data/models.json. Treat this section as preparation guidance: build datasets, evals, and rollout criteria now, but do not present fine-tuning as a runnable AvalAI optimization path until AvalAI announces supported base models and routes.

Best practices for fine-tuning include:

• Start with prompt engineering to establish a baseline
• Focus on quality over quantity (start with 50+ high-quality examples)
• Ensure training examples are representative of real-world inputs
• Use “prompt baking”: log pilot prompts and outputs, prune them into realistic training examples, and remove secrets or low-quality cases.
• Keep a hold-out set for evaluation so a better training score does not hide overfitting.

If hosted fine-tuning becomes available and your production flow uses RAG, include the RAG context in fine-tuning examples. Otherwise the model learns a simpler task than the one it will see in production.

Combined Approaches

These techniques stack on top of each other. Benefits of combining approaches include:

• Using concise prompts, schemas, and examples to minimize repeated instruction tokens
• Teaching complex behavior through prompt examples now, and through fine-tuning only when a supported AvalAI route exists
• Using RAG, file context, search, and tools to inject private, recent, or task-specific information

OpenAI's Icelandic correction case study is useful as a mental model, not an AvalAI runnable recipe: few-shot examples improved behavior, fine-tuning improved consistency in that historical OpenAI setup, and adding RAG decreased the score because the task needed learned behavior more than extra context. For AvalAI, translate that lesson into an eval-first choice: use RAG only when failures are caused by missing context, and use stronger prompts, schemas, routing, or future fine-tuning when failures are caused by inconsistent behavior.

How Much Accuracy is "Good Enough"?

When deciding if your LLM solution is ready for production, consider both business and technical factors:

Business Considerations

• Identify primary success and failure cases with associated costs
• Calculate break-even accuracy based on these costs
• Measure empirical stats (e.g., CSAT scores, decision accuracy, time to resolution)

Technical Approaches

• Design the system to handle failures gracefully
• Consider trade-offs between accuracy, user experience, and operational costs
• Ask for clarification or hand off to a human when confidence is low or the cost of a wrong answer is high.
• Prefer assistant mode over full automation for high-impact actions such as payments, medical/legal/security decisions, or irreversible writes.

Related Resources

• Prompt Engineering
• Model Selection
• Fine-tuning
• Evals
• OpenAI Model Optimization
• OpenAI Evaluation Best Practices