Retrieval

Warning

Hosted Retrieval, provider-managed vector_stores, and hosted file_search are under development in AvalAI. For production today, build retrieval in your application with /v1/embeddings, your own vector/search store, and /v1/responses or /v1/chat/completions.

Retrieval is the process of finding the most relevant private or product-specific knowledge before asking a model to answer. OpenAI's hosted Retrieval and File Search docs are useful design references, but AvalAI projects should currently implement the retrieval step themselves.

AvalAI Availability Matrix

Capability	AvalAI status	Recommended path today
/v1/embeddings	Available	Embed chunks and user queries.
/v1/responses	Available for supported models	Synthesize grounded answers from retrieved context.
/v1/chat/completions	Available	Keep existing chat workflows and pass retrieved context in messages.
Hosted vector_stores	Under development	Store chunks and embeddings in your own database, vector DB, or search index.
Hosted file_search tool	Under development	Use app-side retrieval, then migrate to file_search when hosted support is announced.

Core Retrieval Pattern

1. Prepare documents: extract text, split by topic, and keep stable source IDs.
2. Embed chunks: call /v1/embeddings for each chunk and store the vectors.
3. Retrieve: embed the user query, apply tenant/permission filters, then search for similar chunks.
4. Rerank and trim: keep only high-confidence chunks that fit the model context budget.
5. Synthesize: send the selected context to a model and require citations to source IDs.

Python

retrieved_chunks = [
    {
        "source_id": "refund-policy.md#section=returns",
        "text": "Customers can request a refund within 30 days of purchase.",
        "score": 0.86,
    },
    {
        "source_id": "refund-policy.md#section=exceptions",
        "text": "Digital goods are refundable only when access has not started.",
        "score": 0.79,
    },
]


def format_sources(chunks):
    return "\n\n".join(
        f"[{chunk['source_id']} score={chunk['score']}]\n{chunk['text']}"
        for chunk in chunks
    )


sources = format_sources(retrieved_chunks)

JavaScript

const retrievedChunks = [
  {
    source_id: "refund-policy.md#section=returns",
    text: "Customers can request a refund within 30 days of purchase.",
    score: 0.86,
  },
  {
    source_id: "refund-policy.md#section=exceptions",
    text: "Digital goods are refundable only when access has not started.",
    score: 0.79,
  },
];

function formatSources(chunks) {
  return chunks
    .map((chunk) => `[${chunk.source_id} score=${chunk.score}]\n${chunk.text}`)
    .join("\n\n");
}

const sources = formatSources(retrievedChunks);

Synthesize With Chat Completions

Keep this path for models and integrations that already use /v1/chat/completions.

Python

import os
from openai import OpenAI

client = OpenAI(
    api_key=os.environ["AVALAI_API_KEY"],
    base_url="https://api.avalai.ir/v1",
)

question = "Can a customer refund a digital purchase?"

completion = client.chat.completions.create(
    model="claude-sonnet-5",
    messages=[
        {
            "role": "developer",
            "content": (
                "Answer only from the provided sources. Cite source IDs in brackets. "
                "If the sources are insufficient, say what is missing."
            ),
        },
        {
            "role": "user",
            "content": f"Sources:\n{sources}\n\nQuestion: {question}",
        },
    ],
)

print(completion.choices[0].message.content)

JavaScript

import OpenAI from "openai";

const client = new OpenAI({
  apiKey: process.env.AVALAI_API_KEY,
  baseURL: "https://api.avalai.ir/v1",
});

const question = "Can a customer refund a digital purchase?";

const completion = await client.chat.completions.create({
  model: "claude-sonnet-5",
  messages: [
    {
      role: "developer",
      content:
        "Answer only from the provided sources. Cite source IDs in brackets. If the sources are insufficient, say what is missing.",
    },
    {
      role: "user",
      content: `Sources:\n${sources}\n\nQuestion: ${question}`,
    },
  ],
});

console.log(completion.choices[0].message.content);

Bash

curl https://api.avalai.ir/v1/chat/completions \
  -H "Authorization: Bearer $AVALAI_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "claude-sonnet-5",
    "messages": [
      {
        "role": "developer",
        "content": "Answer only from the provided sources. Cite source IDs in brackets."
      },
      {
        "role": "user",
        "content": "Sources: [refund-policy.md#section=returns] Customers can request a refund within 30 days. Question: Can a customer refund a digital purchase?"
      }
    ]
  }'

Responses API version

Use this version when the selected model supports /v1/responses. The retrieved sources move into input, and final text is read from response.output_text.

Python

response = client.responses.create(
    model="gpt-5.6-luna",
    instructions=(
        "Answer only from the provided sources. Cite source IDs in brackets. "
        "If the sources are insufficient, say what is missing."
    ),
    input=f"Sources:\n{sources}\n\nQuestion: {question}",
)

print(response.output_text)

JavaScript

const response = await client.responses.create({
  model: "gpt-5.6-luna",
  instructions:
    "Answer only from the provided sources. Cite source IDs in brackets. If the sources are insufficient, say what is missing.",
  input: `Sources:\n${sources}\n\nQuestion: ${question}`,
});

console.log(response.output_text);

Bash

curl https://api.avalai.ir/v1/responses \
  -H "Authorization: Bearer $AVALAI_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "gpt-5.6-luna",
    "instructions": "Answer only from the provided sources. Cite source IDs in brackets.",
    "input": "Sources: [refund-policy.md#section=returns] Customers can request a refund within 30 days. Question: Can a customer refund a digital purchase?"
  }'

Migration checklist:

• messages → input
• developer/system guidance → instructions or a developer input item
• choices[0].message.content → response.output_text
• tool calls, citations, and structured items → inspect response.output by item type

Query Rewriting

Some user questions are conversational while vector search works better with concise search phrases. In a manual AvalAI RAG pipeline, rewrite vague questions before retrieval, but log both the original and rewritten query.

Original question	Retrieval query
"Can I get my money back if I never used the product?"	refund policy unused digital product
"What do we do when a customer is in Europe?"	EU customer data handling policy
"Which plan lets me invite the whole team?"	team invitation plan limits

For future hosted vector store search, OpenAI's reference shape includes rewrite_query=true. Treat the returned or logged search_query as retrieval telemetry, not as a replacement for the original user message: save both so you can explain why a document matched.

Filtering And Ranking

Apply deterministic filters before similarity search. Do not rely on the model to enforce access control.

{
  "tenant": "acme",
  "language": "en",
  "document_type": [
    "policy",
    "faq"
  ],
  "published_after": "2026-01-01"
}

After retrieval, tune ranking behavior:

• top_k / max_num_results: lower values reduce latency and context cost; higher values improve recall.
• Score threshold: drop low-confidence chunks and let the model say when context is insufficient.
• ranker: start with auto for hosted search so the provider can choose the current ranker; pin a ranker only for repeatable evals.
• hybrid_search.embedding_weight: increase it when semantic similarity should dominate.
• hybrid_search.text_weight: increase it when exact product names, IDs, or policy terms matter.
• Reranking: run a cheaper first-stage search, then rerank the top candidates before synthesis.

Hosted Retrieval Reference

When AvalAI announces hosted vector stores, the OpenAI-compatible shape is expected to look like this:

Python

import os
from pathlib import Path
from openai import OpenAI

client = OpenAI(
    api_key=os.environ["AVALAI_API_KEY"],
    base_url="https://api.avalai.ir/v1",
)

file_obj = client.files.create(
    file=Path("refund-policy.md").open("rb"),
    purpose="assistants",
)

vector_store = client.vector_stores.create(
    name="support-knowledge",
    expires_after={"anchor": "last_active_at", "days": 14},
)

client.vector_stores.files.create_and_poll(
    vector_store_id=vector_store.id,
    file_id=file_obj.id,
    attributes={"tenant": "acme", "document_type": "policy"},
    chunking_strategy={
        "type": "static",
        "max_chunk_size_tokens": 1000,
        "chunk_overlap_tokens": 200,
    },
)

results = client.vector_stores.search(
    vector_store_id=vector_store.id,
    query="refund policy unused digital product",
    max_num_results=5,
    rewrite_query=True,
    ranking_options={
        "ranker": "auto",
        "score_threshold": 0.25,
        "hybrid_search": {"embedding_weight": 0.7, "text_weight": 0.3},
    },
)

JavaScript

import fs from "node:fs";
import OpenAI from "openai";

const client = new OpenAI({
  apiKey: process.env.AVALAI_API_KEY,
  baseURL: "https://api.avalai.ir/v1",
});

const file = await client.files.create({
  file: fs.createReadStream("refund-policy.md"),
  purpose: "assistants",
});

const vectorStore = await client.vectorStores.create({
  name: "support-knowledge",
  expires_after: { anchor: "last_active_at", days: 14 },
});

await client.vectorStores.files.createAndPoll(vectorStore.id, {
  file_id: file.id,
  attributes: { tenant: "acme", document_type: "policy" },
  chunking_strategy: {
    type: "static",
    max_chunk_size_tokens: 1000,
    chunk_overlap_tokens: 200,
  },
});

const results = await client.vectorStores.search(vectorStore.id, {
  query: "refund policy unused digital product",
  max_num_results: 5,
  rewrite_query: true,
  ranking_options: {
    ranker: "auto",
    score_threshold: 0.25,
    hybrid_search: { embedding_weight: 0.7, text_weight: 0.3 },
  },
});

For batch ingestion, use file_batches.create_and_poll when several files should become searchable together. A batch can share file_ids plus common settings, or use a files array for per-file attributes and chunking_strategy; do not send both shapes in the same request.

Design Notes From OpenAI Retrieval Docs

• Semantic search can find relevant chunks even when the query and result share few keywords.
• Hosted vector stores parse, chunk, embed, and index attached files.
• Direct semantic search returns up to 10 results by default and can be configured up to 50 with max_num_results.
• File attributes support metadata filtering; OpenAI's reference limit is 16 keys with 256 characters per key.
• Direct hosted search can rewrite the query and expose a search_query for debugging; log that alongside filters, ranker, and scores.
• ranking_options includes ranker, score_threshold, and hybrid_search weights. At least one hybrid weight should be greater than zero when hybrid search is configured.
• OpenAI's reference chunking defaults to 800 tokens with 400-token overlap. Custom chunk sizes must be 100-4096 tokens, and overlap should not exceed half the chunk size.
• OpenAI's reference file limits for vector stores are 512 MB and 5,000,000 tokens per file. Treat these as planning references, not AvalAI commitments, until hosted support is announced.
• Batch ingestion is useful for many files; OpenAI's reference supports up to 500 files in a vector store batch.
• expires_after deletes associated vector_store.file objects when a hosted vector store expires; use it for temporary uploads and keep your own retention policy for manual RAG stores.

Safety And Evaluation

• Keep tenant, user, and document-level permissions in your retrieval layer.
• Store source IDs with every chunk and make citations mandatory in model instructions.
• Validate retrieved links and tool arguments to reduce prompt-injection and data-exfiltration risk.
• Log query rewrites, filters, scores, selected chunks, model answers, and user feedback.
• Track retrieval quality separately from answer quality: measure source recall, source precision, first-correct-rank, MRR, and MAP against a small set of real user questions.
• Treat file deletion and vector-store expiration as eventually consistent for application safety; keep authorization checks in your retrieval layer even after cleanup requests.
• Build eval sets with question, expected answer, expected source IDs, and unacceptable hallucinations.

Related Resources

• Embeddings API Reference
• Responses API Reference
• Chat API Reference
• Files API Reference
• File Search Tool
• Citation Formatting
• Manual RAG with Embeddings
• RAG Best Practices