API Deployment Checklist

Use this checklist before shipping an AvalAI-powered app to production. It adapts OpenAI’s official API deployment checklist and production best practices to AvalAI’s OpenAI-compatible base URL, rate-limit tiers, provider routing, and bilingual documentation model.

1. Choose the API Surface

• Prefer /v1/responses for new text, reasoning, structured-output, and tool workflows when your chosen model supports it.
• Keep /v1/chat/completions for existing integrations or models that only expose chat compatibility.
• Document the migration path: messages → input, system/developer prompts → instructions, max_completion_tokens → max_output_tokens, and choices[0].message.content → response.output_text.
• Pin model IDs in config: set AVALAI_MODEL, AVALAI_BASE_URL=https://api.avalai.ir/v1, and a prompt/version label per release.

Warning

OpenAI-hosted Responses features such as hosted tools, background jobs, compaction, encrypted reasoning handoff, or WebSocket mode may be route- and model-dependent in AvalAI. Keep the checklist item, but gate rollout on the actual provider/model behavior you observe in staging.

Before launch, keep a small capability matrix for each model route you ship: /v1/responses, streaming, reasoning.effort, text.verbosity, previous_response_id, prompt_cache_key, tool calling, background jobs, and WebSocket continuation. Every “yes” needs a staging request ID and every “no” needs a documented fallback.

2. Tune Quality, Cost, and Latency

Lever	Use it for	AvalAI guidance
reasoning.effort	reasoning depth	Use the lowest effort that passes evals; reserve high effort for complex decisions.
text.verbosity	response length	Keep production answers concise unless the UX requires detailed explanations.
max_output_tokens / max_completion_tokens	shared output and reasoning budget	Cap every request, but leave visible-answer headroom: reasoning can consume the entire limit and produce an incomplete response with no text. Alert on max_output_tokens / length exhaustion and tune the limit together with reasoning.effort.
prompt_cache_key	repeated context	Keep stable policy/product context early and dynamic user context late; avoid one overly hot cache key for unrelated tenants or workflows.
Assistant phase	long-running agents	Preserve phase: "commentary" versus phase: "final_answer" when replaying assistant history.
tool_search / deferred tools	large tool catalogs	Prefer smaller namespaces and lazy tool loading; keep namespace descriptions short and put detailed usage rules in the deferred tool definitions.
Compaction	long conversations	Use hosted compaction when supported, or summarize state in your app without losing decisions, IDs, and open tasks.
reasoning.encrypted_content	stateless reasoning continuity	Round-trip provider-returned reasoning items exactly when supported; do not parse or rewrite them.
Streaming	perceived latency	Stream visible answers and show status for tool-heavy flows.
Background processing	resumability	Use app-managed jobs for long work unless hosted background Responses support is enabled for your route; hosted background mode generally requires stored response state.
WebSocket mode	multi-turn tool-heavy flows	Use only after staging tests prove the route supports it; one connection should own one in-flight workflow, with separate connections for parallel work.

For support-gated or route-dependent features, write the release contract before shipping:

• Tool search: group tools by intent, keep namespaces small, and use short namespace descriptions. If hosted tool_search or defer_loading is not enabled for your route, filter tool definitions in your application before the AvalAI request.
• Compaction: if hosted compaction is available, pass compacted output forward unchanged; treat it as machine state, not an editable human summary. If it is not available, preserve decisions, IDs, tool outcomes, and open tasks in an app-managed summary.
• Prompt caching: keep stable policy, schema, and tool definitions at the beginning of the prompt, use an opaque prompt_cache_key per workload or tenant, and avoid raw user identifiers or one overloaded global cache key.
• Background jobs: verify whether background: true requires stored response state on the exact route. If your privacy posture requires zero-retention behavior, prefer app-managed queues and workers.
• WebSocket mode: reserve it for long-running, tool-heavy workflows after staging proves reconnect, cancellation, timeout, and HTTP fallback behavior.

3. Secure Credentials and User Identity

• Store AVALAI_API_KEY in a secret manager or environment variable; never ship it in browser/mobile code.
• Use separate keys or projects for development, staging, and production.
• Rotate exposed keys immediately and document who owns rotation.
• Send a hashed safety_identifier for end-user products so abuse reviews can isolate users without storing raw personal data.
• Avoid sending passwords, API keys, private keys, or unnecessary customer data to models even when guardrails are enabled.

4. Lock Down Access and Operations

OpenAI's RBAC, Admin API, workload identity, and IP egress docs are useful operating patterns, but AvalAI account features can differ. Treat them as a checklist and verify each item against the AvalAI dashboard, reseller contract, or account support before launch.

• Least privilege: separate human admin access from runtime API keys. Grant production keys only the routes and models the app actually needs when project/model allowlists are available.
• Project boundaries: keep experiments, staging, production, and reseller/customer environments in separate projects or accounts so files, spend, rate limits, and logs do not mix.
• Service credentials: use workload or service-account credentials for servers and CI where available. If your environment still uses long-lived AVALAI_API_KEY values, rotate them on a schedule and after every incident.
• Model governance: maintain an allowlist of approved model IDs per environment and block experimental models from regulated workloads until privacy, cost, latency, and eval checks pass.
• Spend and rate-limit operations: define owners for budget alerts, quota changes, and rate-limit escalation. Tie every requested limit increase to observed x-request-id evidence and expected traffic.
• Auditability: log who changed keys, routing, model allowlists, prompt versions, data-retention settings, and reseller budgets. Export or archive audit events if your account plan exposes them.
• Network controls: IP allowlists identify traffic from an operated network, not an authenticated user. Keep request authentication, mTLS/OAuth for tools, and signed webhooks in place even when allowlists are enabled.

CI/CD and Workload Identity

OpenAI's workload identity federation docs show a strong production pattern: trusted CI/CD or runtime workloads exchange externally issued identity tokens for short-lived access tokens instead of storing long-lived API keys. AvalAI may not expose the same token-exchange endpoint or service-account mapping model on every account, so adapt the principle rather than copying OpenAI-specific API calls.

Use this pattern for AvalAI deployments:

• Prefer keyless cloud access where possible: let GitHub Actions, Kubernetes, AWS, Azure, GCP, or SPIFFE authenticate to your own secret manager with OIDC/workload identity, then fetch AVALAI_API_KEY just-in-time for the job.
• Match exact claims: restrict issuer, audience, repository, branch/ref, environment, workflow, namespace, or service account. Avoid organization-wide mappings and never grant production credentials to untrusted fork pull requests.
• Separate identities: use different credentials for CI, staging, production runtime, reseller billing jobs, and data backfills. A build pipeline should not share the same key as a live user-facing app.
• Keep tokens short-lived: if your platform mints temporary secrets, exchange them only when a job starts, mask them in logs, and revoke or rotate the underlying AvalAI key after incidents.
• Audit mappings: review repository transfers, branch protection, environment approvals, JWKS/key rotation, and disabled workflows; stale identity mappings can become production access paths.

If AvalAI later adds native workload identity or service-account mappings, keep the same security bar: exact claim matching, least privilege, separate environments, short-lived tokens, and alerting on failed or unexpected exchanges.

5. Add Safety Gates

• Moderate risky inputs and outputs with /v1/moderations.
• Validate structured outputs with schemas before using them in downstream systems.
• Validate tool arguments before execution and tool results before returning them to the model.
• Require human approval for refunds, deletes, writes, shell commands, financial actions, medical/legal/security decisions, and other side effects.
• Run red-team tests for prompt injection, unsafe tool use, data leakage, and policy-boundary pressure.

6. Prepare Evals and Rollout

• Keep a golden dataset in your repo with representative prompts, edge cases, expected behavior, and pass/fail rubrics.
• Run fast smoke evals on every pull request that changes prompts, model IDs, retrieval, tools, moderation, or routing.
• Run a full suite before launch using Promptfoo Evals with AvalAI or your CI runner.
• Canary the change with a small traffic slice, compare production metrics against eval results, and keep rollback config ready.
• Add every production failure to the eval set before fixing the prompt.
• Validate long-context workflows at several context sizes so “lost in the middle” regressions are caught before launch.

7. Handle Scale and Rate Limits

• Read the Rate Limits page for the current generated limits by tier and model.
• Implement exponential backoff with jitter for 429 and transient 5xx errors.
• Watch x-ratelimit-*, x-request-id, latency, token usage, and final status in logs.
• Batch embeddings and other batch-safe work where it reduces request count without increasing output tokens.
• Use queues for bursty workloads and app-managed background workers for long-running flows.

8. Observe Production

Log enough context to reproduce failures without exposing secrets:

• route, model, provider, service tier, prompt version, and deployment version
• x-request-id, response ID, latency, retries, streaming mode, and final status
• input tokens, output tokens, cached tokens, and estimated cost
• tool-call names, validated arguments, result status, and approval decisions
• moderation result, safety_identifier hash, and red-team/eval failure reason

9. Go/No-Go Review

Ship only when all answers are “yes”:

• Does the app use the intended API surface and model config?
• Are API keys, user IDs, and logs privacy-safe?
• Are access controls, model allowlists, spend owners, and audit logs ready?
• Do evals, red-team tests, and schema checks pass?
• Are rate-limit, timeout, retry, and rollback paths tested?
• Are support teams able to trace a user report from x-request-id to prompt version and model route?
• Is the user experience clear about AI limits, review paths, and unsafe-content decisions?

Related Guides

• Production Best Practices
• Responses vs. Chat Completions
• Data Controls
• Rate Limits
• Safety Best Practices
• Red Teaming AI Applications
• Cost Optimization