# ML Junction Documentation

_Source: https://mljunction.com/docs · Retrieved 2026-07-20_

# Get Started

## Overview

ML Junction is a capability-aware LLM gateway. Your application sends one stable request format, and the gateway selects an eligible provider route, enforces capability and privacy requirements, applies retry and fallback policy, records usage, and returns routing and billing evidence.

Use the canonical Responses API when you want the full routing, structured-output, multimodal, context-management, privacy, and receipt surface. Use the OpenAI-compatible Chat Completions and Embeddings endpoints when migrating existing SDK integrations.

| Endpoint | Authentication | Purpose |
| --- | --- | --- |
| /v1/responses | API key | Canonical text, multimodal, tool, reasoning, and structured-output requests |
| /openai/v1/chat/completions | API key | OpenAI SDK-compatible Chat Completions |
| /anthropic/v1/messages | API key | Anthropic SDK-compatible Messages |
| /v1/embeddings | API key | OpenAI-compatible embedding generation |
| /v1/routing/preview | API key | Inspect the route plan without calling a model |
| /v1/models | Public | Model catalog and seven-day popularity statistics |
| /v1/models/{model}/routes | Public | Provider routes, pricing, capabilities, precision, and telemetry |
| /v1/models/{model}/usage | Public | Model request and token graph |
| /v1/providers | Public | Provider directory |
| /v1/status/{scope} | Public | Provider, model, or route health snapshot |
| /v1/status/{scope}/{target}/history | Public | Health and latency history |
| /v1/status/errors | Public | Categorized error graph |
| /v1/activity/requests/{request_id} | User session | Request plan, attempts, receipt, warnings, and errors |

> **Note:** Unknown measured values are omitted instead of returned as zero. Unknown catalog facts with a defined unknown state, such as route precision, use the explicit value "unknown".

## Quickstart

### 1. Configure credentials

Create an API key in the dashboard and store it only on your server. Never embed a secret API key in browser JavaScript, mobile bundles, public repositories, logs, or analytics events.

```bash .env
ASTRO_API_KEY=your_secret_api_key
ASTRO_BASE_URL=https://api.mljunction.com
```

### 2. Send a canonical request

```bash curl
curl "$ASTRO_BASE_URL/v1/responses" \
  -H "Authorization: Bearer $ASTRO_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "gpt-5.5",
    "messages": [
      {
        "role": "user",
        "content": "Explain capability-safe fallback in three paragraphs."
      }
    ],
    "routing": {
      "strategy": "balanced"
    }
  }'
```

### 3. Read the response and receipt

```json response
{
  "id": "resp_...",
  "model": "gpt-5.5",
  "output": [
    {
      "type": "text",
      "text": "Capability-safe fallback means..."
    }
  ],
  "finish_reason": "stop",
  "usage": {
    "input_tokens": 18,
    "cached_input_tokens": 0,
    "output_tokens": 142,
    "reasoning_tokens": 0,
    "total_tokens": 160
  },
  "routing": {
    "selected_route": "openai-gpt-5.5-global-standard",
    "provider": "openai",
    "provider_model": "gpt-5.5",
    "billing_mode": "platform",
    "credential_source": "platform",
    "fallback_used": false,
    "attempts": 1,
    "sticky_hit": false,
    "decision_reason": "primary_route"
  },
  "receipt": {
    "route_id": "openai-gpt-5.5-global-standard",
    "provider": "openai",
    "provider_model": "gpt-5.5",
    "region": "global",
    "fallback_layer": "primary",
    "estimated_charge_usd": 0.0004,
    "actual_charge_usd": 0.00038,
    "pricing_source": "https://..."
  },
  "warnings": []
}
```

> **Note:** The exact model IDs available to your deployment come from GET /v1/models. Do not hard-code IDs copied from a provider website without checking the catalog.

## Authentication

Inference endpoints accept a gateway API key in the Authorization header. Dashboard and organization endpoints use the signed user session supplied by Clerk. Public catalog and status endpoints require no authentication.

```http authorization header
Authorization: Bearer YOUR_API_KEY
```

| Credential | Use | Storage guidance |
| --- | --- | --- |
| Gateway API key | Server-to-server inference calls | Secret manager or encrypted server environment |
| Clerk user token | Dashboard authentication | Obtain from the active Clerk session; organizations and RBAC are owned by ML Junction |
| Provider BYOK secret | Calls billed by your provider account | Submit once through the protected BYOK API; never read back |
| No credential | Catalog, provider directory, usage popularity, and public status | Safe for browser fetching |

- Use separate keys for development, staging, and production.
- Rotate a key immediately if it appears in a repository, screenshot, browser bundle, or support transcript.
- Apply model, RPM, TPM, concurrency, and maximum-output policies to each key.
- Treat a 401 response as invalid or missing authentication and a 403 response as an authorization or policy failure.

## OpenAI SDK Compatibility

Point the official OpenAI SDK at /openai/v1. Chat Completions supports sync/async text, SSE streaming, tools and streamed arguments, strict structured output, reasoning effort, usage, request IDs, and ML Junction extensions.

```javascript Node.js
import OpenAI from 'openai';

const client = new OpenAI({
  apiKey: process.env.ASTRO_API_KEY,
  baseURL: 'https://api.mljunction.com/openai/v1',
});

const completion = await client.chat.completions.create({
  model: 'gpt-5.5',
  messages: [
    { role: 'system', content: 'Be precise and concise.' },
    { role: 'user', content: 'Explain token caching.' },
  ],
  temperature: 0.3,
  max_completion_tokens: 1200,
});

console.log(completion.choices[0].message.content);
console.log(completion.usage);
```

```python Python
from openai import OpenAI

client = OpenAI(
    api_key=os.environ["ASTRO_API_KEY"],
    base_url="https://api.mljunction.com/openai/v1",
)

completion = client.chat.completions.create(
    model="gpt-5.5",
    messages=[
        {"role": "system", "content": "Be precise and concise."},
        {"role": "user", "content": "Explain token caching."},
    ],
    temperature=0.3,
    max_completion_tokens=1200,
)

print(completion.choices[0].message.content)
print(completion.usage)
```

> **Note:** Pass ML Junction-only controls through extra_body={"mlj": {...}} in Python or the mlj request property in JavaScript. The response includes mlj.routing, mlj.receipt, mlj.warnings, mlj.reasoning, and mlj.observability without changing standard SDK fields.

## Anthropic SDK Compatibility

Point the official Anthropic SDK at /anthropic. ML Junction implements POST /anthropic/v1/messages with Anthropic-shaped success, streaming, tool, usage, request-ID, and error behavior.

```python Anthropic Python SDK
import os
from anthropic import Anthropic

client = Anthropic(
    api_key=os.environ["MLJUNCTION_API_KEY"],
    base_url="https://api.mljunction.com/anthropic",
)

message = client.messages.create(
    model="claude-opus-4-6",
    max_tokens=1200,
    system="Be precise.",
    messages=[{"role": "user", "content": "Explain route receipts."}],
    extra_body={
        "mlj": {
            "routing": {
                "provider": "anthropic",
                "require_zdr": True,
                "max_platform_charge_usd": 0.05,
            },
            "session_name": "docs-example",
            "task_name": "explain-receipts",
        }
    },
)

print(message.content[0].text)
print(message.mlj["receipt"])
```

```javascript Anthropic TypeScript SDK
import Anthropic from '@anthropic-ai/sdk';

const client = new Anthropic({
  apiKey: process.env.MLJUNCTION_API_KEY,
  baseURL: 'https://api.mljunction.com/anthropic',
});

const stream = await client.messages.stream({
  model: 'claude-opus-4-6',
  max_tokens: 1200,
  messages: [{ role: 'user', content: 'Stream a deployment checklist.' }],
  mlj: { routing: { strategy: 'quality', provider: 'anthropic' } },
});

for await (const event of stream) console.log(event.type);
```

| Messages feature | Status | Notes |
| --- | --- | --- |
| Text and system prompts | Supported | String and text-block system prompts |
| Streaming | Supported | Official Anthropic event ordering |
| Tool use / tool results | Supported | Mapped to the native function-tool pipeline |
| Images | Supported | URL and base64 sources |
| output_config.format | Supported | Strict schema output; structured streams are atomic |
| thinking / redacted_thinking | Preserved | Opaque blocks remain provider-bound |
| Message Batches, Files, token counting | Not emulated | Use native ML Junction batch/files surfaces where available |

> **Note:** Signed or redacted Anthropic thinking is detected before execution. Non-Anthropic routes and routes without reasoning-continuity support are excluded; incompatible explicit routing returns HTTP 400. Token counts alone are portable telemetry and do not pin a provider.

## LangChain Integration

langchain-mljunction is an independent native integration maintained by ML Junction. It subclasses LangChain BaseChatModel and Embeddings directly and calls /v1/responses and /v1/embeddings; it does not wrap ChatOpenAI.

```bash install
pip install langchain-mljunction
export MLJUNCTION_API_KEY="..."
export MLJUNCTION_BASE_URL="https://api.mljunction.com"
```

```python native LangChain model
from langchain_mljunction import ChatMLJunction

llm = ChatMLJunction(
    model="gpt-5.5",
    temperature=0.2,
    reasoning={"enabled": True, "effort": "medium"},
    requirements={"reasoning": "preferred", "tools": "preferred"},
    routing={
        "strategy": "balanced",
        "service_tier": "auto",
        "require_zdr": True,
        "max_platform_charge_usd": 0.05,
        "fallbacks": {"provider": True, "model": True, "max_total_attempts": 3},
    },
    context={"mode": "auto_fit", "preserve_tool_chains": True},
    session_name="support-session",
    task_name="triage",
    app_name="support-agent",
)

response = llm.invoke("Triage this incident")
print(response.content)
print(response.response_metadata["routing"])
print(response.response_metadata["receipt"])
```

| Constructor control | Native request field |
| --- | --- |
| temperature, top_p, seed, frequency_penalty, presence_penalty | sampling.* |
| max_tokens and output | output.max_tokens and output format/validation |
| reasoning | reasoning enabled/effort/intensity/summary |
| requirements | Required/preferred tools, schema, modalities, reasoning, context |
| routing | All strategy, provider, tier, privacy, BYOK, cap, fallback, sticky controls |
| context | Fitting mode, budget/reserve, preservation controls |
| session/task/app fields | Durable observability identity |
| metadata, idempotency_key, compatibility | Native metadata and execution controls |

Use bind_tools for LangChain tools, with_structured_output for Pydantic or JSON Schema results, stream/astream for SSE, and MLJunctionEmbeddings for governed embeddings. Structured-output streams intentionally return one complete chunk so parsers never receive partial JSON; ordinary text and bound-tool streams remain incremental. include_raw=True returns LangChain's standard raw/parsed/parsing_error dictionary. Standard usage lives in usage_metadata; request ID, route, receipt, warnings, identity, and reasoning state live in response_metadata.

> **Note:** Because this is native, ML Junction features are not compressed into OpenAI fields. The trade-off is that applications intentionally coupled to langchain-mljunction cannot swap to ChatOpenAI without removing ML Junction-specific governance controls.

## SDK Compatibility Limits

| Surface | Implemented | Deliberate limits |
| --- | --- | --- |
| OpenAI Chat Completions | Text, async, streaming, tools, strict schema, usage, reasoning telemetry | n > 1 and logprobs are rejected; provider-hosted tools are not emulated |
| Anthropic Messages | Text, async, streaming, tools, images, schema, thinking preservation | Batches/files/token-count endpoints are not emulated |
| Native API | Complete ML Junction feature surface | Not wire-compatible with vendor SDK response objects |
| LangChain | Native chat, tools, schema, multimodal inputs, streams, embeddings, receipts; standard integration suite verified | Structured-output streams are emitted atomically; ML Junction-specific configuration remains intentional |

- Unknown compatibility request fields are rejected rather than ignored.
- Vendor SDK compatibility describes protocol shape, not a promise that every vendor endpoint exists.
- A compatibility request may route to another provider unless routing.provider, only_providers, or a provider-bound opaque state restricts it.
- Structured streams are buffered until validation succeeds, so first-token latency is intentionally higher than text streaming.
- ML Junction extension fields require extra_body/custom request support in strict vendor SDK versions.

# Requests

## Canonical Request

The canonical request describes intent rather than provider-specific syntax. The gateway translates supported fields for each route and can reject or avoid routes that cannot satisfy strict requirements.

```json complete request
{
  "model": "gpt-5.5",
  "messages": [
    {
      "role": "system",
      "content": "You are a careful technical assistant."
    },
    {
      "role": "user",
      "content": "Compare optimistic and pessimistic locking."
    }
  ],
  "stream": false,
  "sampling": {
    "temperature": 0.3,
    "top_p": 0.9,
    "seed": 42
  },
  "reasoning": {
    "enabled": true,
    "effort": "medium",
    "summary": "concise"
  },
  "output": {
    "max_tokens": 3000,
    "format": {
      "type": "text"
    },
    "streaming_mode": "text",
    "validation": {
      "enabled": true,
      "heal_syntax": false,
      "model_repair": false,
      "max_repair_attempts": 1
    }
  },
  "requirements": {
    "tools": "off",
    "json_schema": "off",
    "vision": "off",
    "audio": "off",
    "pdf": "off",
    "reasoning": "preferred",
    "streaming": false,
    "min_context_tokens": 32000
  },
  "routing": {
    "strategy": "balanced",
    "mode": "platform_only",
    "strict_params": false,
    "only_providers": [],
    "ignored_providers": [],
    "require_zdr": false,
    "allow_provider_training": false,
    "fallbacks": {
      "provider": true,
      "model": true,
      "max_total_attempts": 5,
      "max_retries_per_route": 1,
      "same_or_lower_price": false,
      "preserve_context_window": true,
      "preserve_privacy": true
    },
    "sticky": {
      "enabled": true,
      "scope": "conversation",
      "ttl_seconds": 3600
    }
  },
  "context": {
    "mode": "auto_fit",
    "input_budget_ratio": 0.7,
    "min_output_reserve_tokens": 4096,
    "preserve_system": true,
    "preserve_current_user": true,
    "preserve_tool_chains": true
  },
  "metadata": {
    "feature": "locking-comparison"
  },
  "idempotency_key": "locking-comparison-2026-06-14"
}
```

### Top-level fields

| Field | Required | Meaning |
| --- | --- | --- |
| model | Yes | Public model ID from GET /v1/models |
| messages | Yes | At least one message and at least one user-role message |
| stream | No | Return Server-Sent Events when true |
| sampling | No | Temperature, top-p, and deterministic seed preferences |
| reasoning | No | Reasoning enablement, effort/intensity, and summary preference |
| output | No | Token limit, output format, streaming behavior, and validation |
| tools | No | Up to 128 function tool definitions |
| tool_choice | No | Automatic, required, named, or provider-compatible tool selection |
| requirements | No | Capabilities that eligible routes must or should support |
| routing | No | Strategy, provider controls, privacy, billing mode, fallback, and affinity |
| context | No | Automatic context-fitting and preservation policy |
| metadata | No | Application metadata stored with the request |
| idempotency_key | No | Deduplicates a completed non-streaming request for one hour |

## Messages & Multimodal Input

Message roles are system, developer, user, assistant, and tool. Content can be a plain string or an array of typed parts. Multimodal content automatically upgrades the corresponding route requirement from off to required.

| Part type | Fields | Route requirement |
| --- | --- | --- |
| text | text | Text input |
| image_url | image_url | Vision required |
| input_audio | data and media_type | Audio input required |
| file | file_id or file_url, filename | PDF/file capability required |

```json image request
{
  "model": "gpt-5.5",
  "messages": [
    {
      "role": "user",
      "content": [
        {
          "type": "text",
          "text": "Describe the architecture shown in this diagram."
        },
        {
          "type": "image_url",
          "image_url": "https://example.com/architecture.png"
        }
      ]
    }
  ],
  "requirements": {
    "vision": "required"
  }
}
```

```json PDF request
{
  "model": "gemini-3.5-flash",
  "messages": [
    {
      "role": "user",
      "content": [
        {
          "type": "text",
          "text": "Summarize the risks and action items."
        },
        {
          "type": "file",
          "file_url": "https://example.com/security-review.pdf",
          "filename": "security-review.pdf"
        }
      ]
    }
  ],
  "requirements": {
    "pdf": "required"
  }
}
```

> **Note:** A route must advertise the requested modality. Check capabilities.modalities on GET /v1/models/{model}/routes before presenting upload controls.

## Sampling & Reasoning

| Field | Accepted values | Notes |
| --- | --- | --- |
| sampling.temperature | 0 through 2 | Higher values increase randomness; route support varies |
| sampling.top_p | Greater than 0 through 1 | Nucleus sampling; avoid changing both temperature and top_p without evaluation |
| sampling.seed | Integer | Requests determinism where supported; not a universal guarantee |
| reasoning.enabled | true or false | Requests a reasoning-capable route when paired with a requirement |
| reasoning.effort | none, minimal, low, medium, high, xhigh, max, auto | Portable coarse alias; intensity is more precise |
| reasoning.intensity | 0-100 | Mapped through the selected model route’s DB-backed native levels |
| reasoning.summary | none, auto, concise, detailed | Requests a summary where the provider supports one |
| reasoning.context | auto, current_turn, all_turns | Selects the requested reasoning context scope where supported |
| reasoning.continuation_token | Opaque gwrt_v3 string | Optional route-affinity ticket for provider-native continuation |
| reasoning.input_type | user_message, tool_results, dynamic_context_update | Describes what the current continuation input represents |
| reasoning.pending_turn_action | continue, cancel, fork | Resolves a pending tool loop |

Supported parameters are route-specific. GET /v1/models/{model}/routes returns capabilities.supported_params for the exact provider route. With routing.strict_params set to true, unsupported requested parameters disqualify a route instead of being silently dropped.

> **Note:** Reasoning models may intentionally ignore temperature or expose a different token-control parameter. Use the capability list and strict_params for deterministic production behavior. See Reasoning Continuation for complete history replay, expiration, provider outages, and tool loops.

## Reasoning Continuation

Reasoning continuation lets a later request reuse provider-native reasoning state and prefer the exact route that produced it. It is optional. You can always send ordinary complete message history without a continuation token; the model will continue from the visible conversation through normal routing.

### The simple mental model

A continued conversation has three separate pieces. Your application owns the visible messages and provider reasoning details. ML Junction owns only a small encrypted routing ticket. The message history explains what the conversation means; reasoning details carry provider-native state; the routing ticket identifies where that state can be replayed safely.

| Piece | Owner | What it contains | What to do with it |
| --- | --- | --- | --- |
| messages | Your application | Complete visible system, user, assistant, and tool history | Store in order and send the complete history on the next turn |
| messages[].reasoning_details | Your application | Provider-native reasoning items; some parts may be readable and some opaque | Keep each array on the assistant message that produced it and round-trip it unchanged |
| reasoning.continuation_token | ML Junction | Encrypted issue time and routing affinity for the provider, provider model, route, eligible provider account, API family, and compatibility key | Treat as opaque; store per conversation branch and send the newest successful token |

> **Note:** The continuation token never contains an API key, provider credential, key ID, organization or tenant ID, billing account, conversation history, prompt, answer, reasoning text, signature, or provider-encrypted reasoning content. Provider API keys remain server-configured and are never returned to your application.

### Continuation is optional

If you omit reasoning.continuation_token, the request is a normal stateless conversation turn. Complete visible history still gives the selected model the conversation context. What you give up is exact route/account affinity, provider-native reasoning replay, and any cache benefit that depends on returning to the same provider surface.

| What you send | Allowed? | Result |
| --- | --- | --- |
| Complete history only | Yes | Normal routing and fallback; the model continues from visible messages |
| Complete history + token + matching reasoning_details | Yes; recommended for native continuation | Exact origin is preferred first; native state can be replayed and provider cache affinity may improve |
| Complete history + token, but no reasoning_details | Yes | Route affinity may still be attempted; native state is not reconstructed and PRIOR_REASONING_WILL_NOT_BE_RENDERED is returned |
| Complete history + reasoning_details, but no token | Yes | Visible history continues normally; provider-native details are not trusted for replay and PRIOR_REASONING_NOT_REUSABLE is returned |
| Only a token, without the earlier conversation | Not sufficient for conversation context | The token has no hidden message history; send complete history if the new turn depends on it |

### 1. Start a reasoning turn

```bash first request
curl "$ASTRO_BASE_URL/v1/responses" \
  -H "Authorization: Bearer $ASTRO_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "gpt-5.5",
    "session_id": "sess_checkout_123",
    "messages": [
      {
        "role": "user",
        "content": "A customer paid $120 after a 20% discount. What was the original price?"
      }
    ],
    "reasoning": {
      "enabled": true,
      "effort": "medium",
      "summary": "concise"
    },
    "requirements": {
      "reasoning": "required"
    },
    "output": {
      "max_tokens": 1200
    }
  }'
```

### 2. Save the response state

```json abbreviated first response
{
  "id": "resp_01J...",
  "session_id": "sess_checkout_123",
  "model": "gpt-5.5",
  "output": [
    {
      "type": "text",
      "text": "The original price was $150."
    }
  ],
  "reasoning": {
    "summary": "I treated $120 as 80% of the original price.",
    "details": [
      {
        "type": "reasoning",
        "summary": [
          {
            "type": "summary_text",
            "text": "I divided the paid amount by 0.8."
          }
        ],
        "encrypted_content": "provider-opaque-state"
      }
    ],
    "continuation_token": "gwrt_v3.opaque-routing-ticket",
    "phase": "completed",
    "reusable": true,
    "usage": {
      "reasoning_tokens": 84
    }
  },
  "usage": {
    "input_tokens": 31,
    "output_tokens": 112,
    "reasoning_tokens": 84,
    "total_tokens": 143,
    "provider_reported": true,
    "confidence": "actual"
  },
  "warnings": [],
  "routing": {
    "selected_route": "openai-gpt-5.5-global-standard",
    "provider": "openai",
    "provider_model": "gpt-5.5",
    "fallback_used": false,
    "attempts": 1
  },
  "receipt": {
    "route_id": "openai-gpt-5.5-global-standard",
    "provider": "openai",
    "provider_model": "gpt-5.5",
    "actual_charge_usd": 0.0012,
    "cache": {
      "status": "miss",
      "cached_input_tokens": 0,
      "reason": "provider_reported"
    }
  }
}
```

- Append the visible assistant answer to your conversation history.
- Copy reasoning.details to reasoning_details on that exact assistant message.
- Store reasoning.continuation_token as the latest token for this conversation branch.
- Store reasoning.phase so your application knows whether tool results are pending.
- Replace stored continuation state only after a successful terminal response.

### Readable summaries versus opaque transport state

| Field | May be shown to an end user? | Handling |
| --- | --- | --- |
| reasoning.summary | Yes, when your product policy permits | Render as an optional provider-generated summary, not guaranteed chain of thought |
| Readable summary/thinking text inside reasoning.details | Yes, when clearly identified and permitted | Render only known readable text fields |
| usage.reasoning_tokens | Yes | Show as numeric usage telemetry; it is not readable reasoning and is already part of output usage |
| reasoning.continuation_token | No | Opaque transport value; never render, decode, log, place in a URL, or send to analytics |
| encrypted_content | No | Preserve unchanged for replay; never display as reasoning text |
| signature or provider_signature | No | Preserve unchanged; never expose as user-readable content |
| provider-account reference | No | Internal encrypted routing metadata; it is not a credential and is never exposed as a public field |

### 3. Continue with complete history

```json second request
{
  "model": "gpt-5.5",
  "session_id": "sess_checkout_123",
  "messages": [
    {
      "role": "user",
      "content": "A customer paid $120 after a 20% discount. What was the original price?"
    },
    {
      "role": "assistant",
      "content": "The original price was $150.",
      "reasoning_details": [
        {
          "type": "reasoning",
          "summary": [
            {
              "type": "summary_text",
              "text": "I divided the paid amount by 0.8."
            }
          ],
          "encrypted_content": "provider-opaque-state"
        }
      ]
    },
    {
      "role": "user",
      "content": "What if the discount was 25% instead?"
    }
  ],
  "reasoning": {
    "enabled": true,
    "effort": "medium",
    "summary": "concise",
    "continuation_token": "gwrt_v3.opaque-routing-ticket",
    "input_type": "user_message"
  },
  "requirements": {
    "reasoning": "required"
  },
  "output": {
    "max_tokens": 1200
  }
}
```

> **Note:** Complete history and the token do not override each other. History controls the meaning of the conversation. The token controls safe routing affinity. reasoning_details supplies the provider-native state. If a user edits, removes, or reorders their own history/state, the provider may reject native continuation; ML Junction does not secretly reconstruct client-owned data.

### Soft expiration after 24 hours by default

gwrt_v3 contains an encrypted issue time. The default continuation lifetime is 24 hours, although a deployment may configure a different value. There is no public expires_at field and clients should not decode or calculate token age. Simply send the newest token; the gateway handles expiration.

```json successful response after an expired token
{
  "output": [
    {
      "type": "text",
      "text": "Continuing from the visible conversation..."
    }
  ],
  "warnings": [
    {
      "code": "REASONING_CONTINUATION_EXPIRED",
      "field": "reasoning.continuation_token",
      "action": "ignored",
      "reason": "routing_affinity_token_expired",
      "requested": null,
      "effective": null
    }
  ],
  "reasoning": {
    "continuation_token": "gwrt_v3.fresh-routing-ticket",
    "phase": "completed",
    "reusable": true
  }
}
```

- Expiration is not an error and does not delete visible conversation history.
- The expired routing affinity and provider-native details are not replayed.
- The request goes through normal capability, health, privacy, cost, retry, and fallback routing.
- A successful response supplies fresh continuation state for the route that actually completed the request.
- If you sent only a token and omitted required history, the gateway cannot recover conversation context because tokens never contain messages.

### Provider, route, or account removed between turns

ML Junction prefers the exact origin route and eligible provider account first. It does not make that preference a single point of failure. If the route was disabled or deleted, the provider account was removed, the route is in health cooldown, or its credential is no longer eligible, ML Junction abandons only native replay and continues through the normal routing plan using visible history.

```json graceful route recovery warning
{
  "warnings": [
    {
      "code": "PRIOR_REASONING_NOT_REUSABLE",
      "field": "reasoning.continuation_token",
      "action": "ignored",
      "reason": "origin_route_provider_or_account_unavailable"
    }
  ],
  "routing": {
    "selected_route": "vertex-gemini-fallback",
    "provider": "vertex_ai",
    "provider_model": "gemini-3.5-pro",
    "fallback_used": false,
    "attempts": 1,
    "decision_reason": "primary_route"
  },
  "reasoning": {
    "continuation_token": "gwrt_v3.new-origin-ticket"
  }
}
```

> **Note:** Graceful routing still respects your request. An explicitly pinned provider/route, provider allow-list, required capability, ZDR rule, context requirement, charge cap, or disabled fallback can legitimately leave no eligible alternative. "Graceful fallback" never means bypassing your privacy, capability, billing, or authorization constraints.

### Origin provider fails during the continued request

The origin candidate receives provider-native state. If it fails and another eligible candidate is attempted, ML Junction removes native replay before calling that candidate and sends only portable visible history. The response warning reason is origin_route_failed_falling_back_to_visible_history. This prevents OpenAI-, Anthropic-, Gemini-, DeepSeek-, or xAI-specific state from crossing into an incompatible provider.

- Same-route retry behavior follows routing.fallbacks.max_retries_per_route.
- Cross-route and cross-provider behavior follows routing.fallbacks and max_total_attempts.
- The fallback candidate must satisfy the same model capability, privacy, context, account-policy, and cost rules.
- Continuation-bearing streams are buffered so the gateway can validate terminal provider state and recover before exposing a partial answer.
- The successful fallback becomes the new continuation origin and returns a new gwrt_v3 token.

### Tampered tokens are different from expired tokens

| Situation | HTTP result | What your application should do |
| --- | --- | --- |
| Expired authentic token | Request can succeed with a warning | Keep visible history, accept the fresh token from the successful response |
| Removed/unavailable origin | Request can succeed through normal routing with a warning | Use the selected route evidence and replace continuation state on success |
| Provider outage before completion | Normal retries/fallbacks can succeed | Do not issue a duplicate application retry when the gateway already succeeded |
| Modified, malformed, wrong-version, or non-canonical token | HTTP 400 INVALID_REASONING_CONTINUATION | Discard that token and start a fresh history-only turn; never retry the same token |
| No eligible fallback exists | Normal capability/configuration/provider error | Relax only the constraint your product is allowed to relax, or retry later when retryable |

```json tampered token error
{
  "error": {
    "type": "invalid_request_error",
    "message": "The reasoning continuation routing token is invalid or has been modified.",
    "code": "INVALID_REASONING_CONTINUATION",
    "request_id": "req_01J...",
    "support_code": "support_01J...",
    "retryable": false,
    "details": {}
  }
}
```

### Conversation edits, regeneration, and branching

- Normal next turn: send the complete branch history, reasoning details on their original assistant turns, and the latest token.
- Edit an earlier user message: create a new branch and normally clear continuation state because the old provider state represents a different prefix.
- Regenerate an assistant turn: treat the regenerated response and its returned token/details as the new branch head.
- Fork: copy the visible history you want into a new branch and use pending_turn_action=fork when resolving a pending tool loop; native state may be abandoned with a warning.
- Cancel a pending tool loop: keep the visible transcript, send pending_turn_action=cancel, and expect native continuation to be abandoned.
- Never share one conversation token across unrelated users, sessions, tabs, or branches.

### JavaScript state and request construction

```javascript conversation continuation state
const conversation = {
  sessionId: 'sess_checkout_123',
  messages: [],
  continuation: {
    token: null,
    phase: 'completed',
    reusable: false,
  },
};

function textFromOutput(output = []) {
  return output
    .filter((item) => item.type === 'text')
    .map((item) => item.text ?? '')
    .join('');
}

function acceptSuccessfulResponse(state, response) {
  state.messages.push({
    role: 'assistant',
    content: textFromOutput(response.output),
    tool_calls: response.output
      .filter((item) => item.type === 'tool_call')
      .map((item) => item.tool_call),
    reasoning_details: response.reasoning?.details ?? [],
  });

  state.continuation = {
    token: response.reasoning?.continuation_token ?? null,
    phase: response.reasoning?.phase ?? 'completed',
    reusable: response.reasoning?.reusable ?? false,
  };
}

function buildNextTurn(state, userText) {
  const messages = [
    ...state.messages,
    { role: 'user', content: userText },
  ];

  return {
    model: 'gpt-5.5',
    session_id: state.sessionId,
    messages,
    reasoning: {
      enabled: true,
      effort: 'medium',
      summary: 'concise',
      ...(state.continuation.token
        ? { continuation_token: state.continuation.token }
        : {}),
      input_type: 'user_message',
    },
  };
}
```

> **Note:** Persist continuation state only in storage appropriate for your threat model. Do not put it in a query string, page title, crash report, analytics payload, support screenshot, or client-visible debug panel. It is not an API key, but it is confidential routing metadata and should be handled as an opaque session value.

### OpenAI and Anthropic compatibility surfaces

| Surface | Send continuation controls | Receive/preserve provider state |
| --- | --- | --- |
| Native /v1/responses | reasoning.continuation_token, input_type, pending_turn_action | reasoning.details; attach it to the assistant as reasoning_details |
| OpenAI /openai/v1/chat/completions | mlj.reasoning inside the compatibility extension | Assistant reasoning_details plus mlj.reasoning metadata |
| Anthropic /anthropic/v1/messages | mlj.reasoning inside the compatibility extension | Native thinking/redacted_thinking blocks plus mlj.reasoning metadata |

> **Note:** Use the native Responses API when continuation, route receipts, structured warnings, and portable fallback behavior are first-class application features. Compatibility surfaces preserve their vendor protocol shape and carry ML Junction additions under mlj.

## Structured Output

Use output.format.type=json_object for valid JSON without a schema, or json_schema for a validated object. A JSON Schema request automatically sets requirements.json_schema to required. Syntax healing is off by default so strict output never mutates silently.

```json strict JSON Schema
{
  "model": "gpt-5.5",
  "messages": [
    {
      "role": "user",
      "content": "Extract the incident severity, summary, and action items."
    }
  ],
  "output": {
    "max_tokens": 2000,
    "format": {
      "type": "json_schema",
      "name": "incident",
      "strict": true,
      "schema": {
        "type": "object",
        "additionalProperties": false,
        "properties": {
          "severity": {
            "type": "string",
            "enum": ["low", "medium", "high", "critical"]
          },
          "summary": {
            "type": "string"
          },
          "action_items": {
            "type": "array",
            "items": { "type": "string" }
          }
        },
        "required": ["severity", "summary", "action_items"]
      }
    },
    "validation": {
      "enabled": true,
      "heal_syntax": true,
      "model_repair": false,
      "max_repair_attempts": 1
    }
  },
  "requirements": {
    "json_schema": "required"
  }
}
```

| Validation field | Default | Behavior |
| --- | --- | --- |
| enabled | true | Validate the model output |
| heal_syntax | false | Opt in to repairing safe JSON syntax defects before schema validation |
| model_repair | false | Allow an additional model-assisted repair pass |
| max_repair_attempts | 1 | Zero through two repair attempts |

> **Note:** When stream=true and the output format is not text, the gateway switches to buffered streaming so invalid partial JSON is not exposed as a successful response.

## Tool Calling

Tool definitions use the standard function-tool shape. Supplying tools automatically makes tool support a required route capability unless you explicitly selected another requirement before validation.

```json function tool
{
  "model": "gpt-5.5",
  "messages": [
    {
      "role": "user",
      "content": "What is the weather in Kolkata?"
    }
  ],
  "tools": [
    {
      "type": "function",
      "function": {
        "name": "get_weather",
        "description": "Return current weather for a city.",
        "parameters": {
          "type": "object",
          "additionalProperties": false,
          "properties": {
            "city": { "type": "string" },
            "units": {
              "type": "string",
              "enum": ["metric", "imperial"]
            }
          },
          "required": ["city"]
        }
      }
    }
  ],
  "tool_choice": "auto",
  "requirements": {
    "tools": "required"
  }
}
```

- Preserve tool_call_id when sending a tool result back to the model.
- Keep assistant tool calls and their tool responses together during context truncation.
- Use capabilities.parallel_tools when your agent requires multiple calls in one model turn.
- Treat tool arguments as untrusted input and validate them against your own schema before execution.

### Continue a reasoning tool loop

When the response reasoning.phase is awaiting_tool_results, keep the original assistant tool calls, their reasoning_details, and the continuation token. Execute every required tool, append one tool message per call ID, and send input_type=tool_results. Do not replace the full history with only the newest tool result unless your integration deliberately uses the specialized tool-results-only contract.

```json tool-result continuation request
{
  "model": "gpt-5.5",
  "messages": [
    {
      "role": "user",
      "content": "Look up order 123 and tell me its status."
    },
    {
      "role": "assistant",
      "content": "",
      "tool_calls": [
        {
          "id": "call_order_123",
          "type": "function",
          "function": {
            "name": "lookup_order",
            "arguments": "{\"order_id\":\"123\"}"
          }
        }
      ],
      "reasoning_details": [
        {
          "type": "reasoning",
          "encrypted_content": "provider-opaque-state"
        }
      ]
    },
    {
      "role": "tool",
      "tool_call_id": "call_order_123",
      "content": "{\"status\":\"shipped\",\"eta\":\"2026-07-18\"}"
    }
  ],
  "reasoning": {
    "enabled": true,
    "continuation_token": "gwrt_v3.opaque-routing-ticket",
    "input_type": "tool_results",
    "pending_turn_action": "continue"
  },
  "tools": [
    {
      "type": "function",
      "function": {
        "name": "lookup_order",
        "description": "Look up an order by ID.",
        "parameters": {
          "type": "object",
          "additionalProperties": false,
          "properties": {
            "order_id": { "type": "string" }
          },
          "required": ["order_id"]
        }
      }
    }
  ]
}
```

| Tool-loop problem | Gateway behavior | User action |
| --- | --- | --- |
| Required result is missing | HTTP 409 PENDING_TOOL_LOOP with expected, supplied, and missing IDs | Wait for or supply every result, or explicitly Cancel/Fork |
| Unknown or duplicate result | Unsafe result is dropped and tool_history.dropped is returned | Fix the application mapping; inspect requested/effective IDs |
| Normal user text interrupts a pending loop | PENDING_TOOL_LOOP prevents accidental continuation | Choose Continue, Cancel, or Fork intentionally |
| Origin provider becomes unavailable | Native state is abandoned; eligible fallback receives portable visible history/tool messages | Accept the warning and fresh continuation state on success |

## Streaming

Set stream=true on POST /v1/responses to receive Server-Sent Events. The response uses Content-Type: text/event-stream, Cache-Control: no-cache, and disables reverse-proxy buffering.

```bash streaming curl
curl -N "$ASTRO_BASE_URL/v1/responses" \
  -H "Authorization: Bearer $ASTRO_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "gpt-5.5",
    "stream": true,
    "messages": [
      { "role": "user", "content": "Write a short deployment checklist." }
    ],
    "output": {
      "streaming_mode": "text"
    }
  }'
```

| Mode | Use |
| --- | --- |
| text | Normal incremental text delivery |
| buffered | Hold structured output until validation completes |
| raw_compat | Compatibility-oriented raw stream behavior |

- Do not retry a stream after user-visible content has been emitted unless your UI can explicitly represent a restarted answer.
- Use an AbortController or equivalent cancellation primitive when the user stops generation.
- Disable intermediary buffering in Nginx, CDNs, and serverless proxies.
- Expect usage and final receipt information near stream completion rather than on every token event.
- Requests carrying reasoning continuation state are buffered so terminal provider state can be verified and a pre-commit provider failure can fall back safely.

# Routing

## Routing

Routing begins by eliminating inactive, disabled, unhealthy, capability-incompatible, privacy-incompatible, and policy-incompatible routes. The strategy then orders the remaining candidates. Fallback policy controls what may happen after a failed attempt.

| Strategy | Behavior |
| --- | --- |
| balanced | Balances route quality, health, cost, and latency |
| quality | Prioritizes stronger and more reliable routes |
| cost | Prioritizes the lowest active verified price |
| latency | Prioritizes recent route latency |
| pinned | Requires routing.route or routing.provider |

### Routing controls

| Field | Meaning |
| --- | --- |
| route | Pin or prefer an exact route ID |
| provider | Pin or prefer a provider |
| only_providers | Allow-list of providers |
| ignored_providers | Deny-list of providers |
| region | Requested provider region |
| strict_params | Disqualify routes that cannot honor requested parameters |
| require_zdr | Require a route where zero-data-retention can be enforced |
| allow_provider_training | Permit routes whose policy allows provider training |
| max_platform_charge_usd | Maximum platform-funded charge |
| max_byok_fee_usd | Maximum platform routing fee for BYOK |

```json pinned production route
{
  "model": "gpt-5.5",
  "messages": [
    { "role": "user", "content": "Generate the release decision." }
  ],
  "routing": {
    "strategy": "pinned",
    "production_mode": true,
    "route": "openai-gpt-5.5-global-standard",
    "mode": "platform_only",
    "max_platform_charge_usd": 0.25,
    "strict_params": true,
    "fallbacks": {
      "provider": true,
      "model": false,
      "max_total_attempts": 2,
      "max_retries_per_route": 1,
      "preserve_context_window": true,
      "preserve_privacy": true
    }
  }
}
```

> **Note:** production_mode requires an explicit provider, route, or provider allow-list. It also requires relevant charge caps, enables strict parameter handling, and preserves context and privacy across fallback.

## Routing Presets

| Preset | What it changes | Recommended use |
| --- | --- | --- |
| stable_production | Quality strategy; preserves context and privacy | Production agents and repeatable workflows |
| lowest_cost | Cost strategy; same-or-lower-price fallback | Batch extraction and cost-sensitive workloads |
| lowest_latency | Latency strategy | Interactive chat and latency-sensitive APIs |
| strongest_privacy | Requires ZDR and privacy-preserving fallback | Sensitive workloads |
| long_context | Preserves context-window capacity | Large documents and long conversations |

```json privacy preset
{
  "model": "gpt-5.5",
  "messages": [
    { "role": "user", "content": "Review this confidential contract." }
  ],
  "routing": {
    "preset": "strongest_privacy",
    "allow_provider_training": false
  }
}
```

A preset supplies coherent defaults but does not replace explicit requirements. For example, strongest_privacy controls route privacy; it does not automatically require PDF support for a file upload.

## Service Tiers

Set one canonical service_tier on routing and the gateway translates it to each provider’s native shape. Tiers are a queue-position lever, not a quality lever - the model is identical. If a route can’t serve the requested tier, the request runs on standard and a warning is returned instead of silently downgrading.

| Tier | Meaning | Use for |
| --- | --- | --- |
| auto | Let the provider pick its best available lane | Default; balanced |
| standard | Normal shared serving | Most production traffic |
| priority | Higher scheduling priority, lower tail latency, higher price | Paid users, live UX, checkout |
| flex | Cheaper, slower, lower priority | Evals, enrichment, background agents |

```json priority tier
{
  "model": "gpt-5.5",
  "messages": [{ "role": "user", "content": "The user is waiting." }],
  "routing": { "service_tier": "priority" }
}
```

Pricing is published per tier - flex is cheaper, priority costs more - and the model detail page shows a tier switcher so you can compare before you commit. Tiers do not apply to batch jobs.

## Batch API

Batch runs your requests on the provider’s own native batch lane - typically ~50% cheaper, returned within the completion window. It is async file-in / file-out, not a streaming request, and it is only available for providers that support it natively (you get a clear error otherwise). Service tiers do not apply inside batch.

```bash submit a batch
# 1. Upload a JSONL file of requests
curl $BASE_URL/v1/files -H "Authorization: Bearer $KEY" \
  -F purpose=batch -F model=gpt-5.5 -F file=@requests.jsonl

# 2. Create the batch from the returned file id
curl $BASE_URL/v1/batches -H "Authorization: Bearer $KEY" \
  -H "Content-Type: application/json" \
  -d '{"input_file_id":"<id>","model":"gpt-5.5","endpoint":"/v1/chat/completions"}'

# 3. Poll, then download output (and errors) JSONL
curl $BASE_URL/v1/batches/<batch_id> -H "Authorization: Bearer $KEY"
curl $BASE_URL/v1/batches/<batch_id>/content -H "Authorization: Bearer $KEY"
```

Jobs respect the same BYOK / platform credential resolution as synchronous calls, and are billed at the provider’s batch rate on completion. Track them live on the Batch page in your dashboard.

## Fallbacks & Sticky Routing

### Fallback policy

| Field | Range/default | Meaning |
| --- | --- | --- |
| provider | true | Allow fallback to another provider route |
| model | true | Allow configured model-family fallback |
| max_total_attempts | 1 through 10; default 5 | Hard limit across retries and fallbacks |
| max_retries_per_route | 0 through 3; default 1 | Retries allowed on the same route |
| same_or_lower_price | false | Reject a more expensive fallback |
| preserve_context_window | true | Reject routes that would reduce required context capacity |
| preserve_privacy | true | Reject routes that weaken the selected privacy posture |
| max_price_increase_percent | 0 through 1000 | Bound the fallback price increase |

### Sticky routing

Sticky routing keeps related requests on the same healthy route when possible. This improves behavioral consistency and can improve provider-side prompt-cache reuse. Health and policy checks still override affinity.

| Field | Values | Meaning |
| --- | --- | --- |
| enabled | true or false | Enable affinity |
| scope | conversation, api_key, custom | How requests share an affinity assignment |
| affinity_key | String up to 200 characters | Required for a custom affinity domain |
| ttl_seconds | 60 through 86400 | Affinity lifetime; default 3600 |

> **Note:** Sticky routing is a preference, not an availability guarantee. A disabled, unhealthy, circuit-open, or capability-incompatible route must be bypassed.

### Continuation affinity and fallback

A valid reasoning continuation token is stronger and more exact than ordinary sticky routing: the origin route, provider model, and eligible provider account are tried first so provider-native state is only replayed where it belongs. It still does not disable the platform fallback plan. If the origin is missing, unhealthy, uncredentialed, or fails, native replay is removed and the remaining eligible candidates continue from visible history.

- Expired continuation affinity degrades immediately to normal routing with REASONING_CONTINUATION_EXPIRED.
- Missing route/provider/account degrades with PRIOR_REASONING_NOT_REUSABLE.
- A provider failure can advance to the next candidate after configured same-route retries.
- Explicit provider/route constraints, allow-lists, disabled fallbacks, capability requirements, privacy rules, and charge caps are still enforced.
- The provider that successfully completes the fallback response becomes the origin of the newly returned continuation token.

## Context Management

Context fitting reserves output space and trims input only when necessary. It is atomic around semantic units: system instructions, the current user turn, and assistant/tool chains can be preserved together.

| Field | Default | Meaning |
| --- | --- | --- |
| mode | auto_fit | auto_fit trims safely; strict rejects an oversized input |
| input_budget_ratio | 0.70 | Fraction of available context targeted for input |
| min_output_reserve_tokens | 4096 | Minimum context reserved for generation |
| preserve_system | true | Retain system and developer instructions |
| preserve_current_user | true | Retain the latest user request |
| preserve_tool_chains | true | Keep tool calls with their corresponding results |

```json strict long-context request
{
  "model": "gemini-3.5-flash",
  "messages": [
    { "role": "user", "content": "Analyze the supplied long document." }
  ],
  "requirements": {
    "min_context_tokens": 500000
  },
  "routing": {
    "preset": "long_context"
  },
  "context": {
    "mode": "strict",
    "min_output_reserve_tokens": 12000
  }
}
```

## Routing Preview

POST /v1/routing/preview accepts the same canonical request and returns the planned candidates and routing decisions without executing an upstream model call. Use it for deployment validation, policy debugging, and explaining why a route is or is not eligible.

```bash preview
curl "$ASTRO_BASE_URL/v1/routing/preview" \
  -H "Authorization: Bearer $ASTRO_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "gpt-5.5",
    "messages": [
      { "role": "user", "content": "Preview only." }
    ],
    "requirements": {
      "vision": "required",
      "min_context_tokens": 200000
    },
    "routing": {
      "only_providers": ["openai", "vertex_ai"],
      "require_zdr": true,
      "strict_params": true
    }
  }'
```

- Run previews in CI for production route policies.
- Check that at least one eligible route remains after privacy and capability filters.
- Inspect estimated pricing before raising a charge cap.
- Use exact route IDs from the model-routes endpoint rather than constructing them yourself.

# Catalog & Pricing

## Model Catalog & Leaderboard

GET /v1/models is public and cache-friendly. It powers model discovery and the seven-day popularity leaderboard.

```bash list models
curl "$ASTRO_BASE_URL/v1/models"
```

```json model item
{
  "id": "gpt-5.5",
  "object": "model",
  "owned_by": "mljunction",
  "description": "Default openai route",
  "access": "public_catalog",
  "lifecycle": {
    "status": "active",
    "going_away_at": null,
    "replacement_model": null
  },
  "providers": ["openai"],
  "stats": {
    "rank": 3,
    "requests_7d": 1820000,
    "tokens_7d": 940000000,
    "trend_pct_7d": 12.4
  }
}
```

| Field | Definition |
| --- | --- |
| stats.rank | Descending requests_7d, then tokens_7d, then model ID |
| stats.requests_7d | Requests started in the trailing seven days |
| stats.tokens_7d | Persisted total tokens in the trailing seven days |
| stats.trend_pct_7d | Request-count change versus the preceding seven days |
| lifecycle.status | active, preview, deprecated, or retired |
| lifecycle.going_away_at | Retirement/deprecation timestamp when known |
| lifecycle.replacement_model | Suggested replacement public model |

> **Note:** The stats object is omitted when there is no current or prior seven-day activity. trend_pct_7d is omitted when the previous period is zero. Never treat either omission as a numeric zero.

## Model Routes

GET /v1/models/{public_model}/routes returns every active serving route for a public model. The backend orders routes by 30-day uptime descending, then input price ascending, then route priority.

```bash routes
curl "$ASTRO_BASE_URL/v1/models/gpt-5.5/routes"
```

```json route item
{
  "route_id": "openai-gpt-5.5-global-standard",
  "provider": "openai",
  "provider_model": "gpt-5.5",
  "company": "openai",
  "region": "global",
  "endpoint_type": "serverless",
  "priority": 100,
  "quantization": "unknown",
  "context_tier": "extended",
  "context_window_tokens": 1050000,
  "max_output_tokens": 128000,
  "capabilities": {
    "streaming": true,
    "tools": true,
    "parallel_tools": true,
    "json_object": true,
    "json_schema": true,
    "vision": true,
    "audio": false,
    "pdf": true,
    "reasoning": true,
    "prompt_caching": true,
    "modalities": {
      "input": ["text", "image", "pdf"],
      "output": ["text"]
    },
    "supported_params": ["temperature", "top_p", "seed"]
  },
  "stats": {
    "throughput_tps": 86.4,
    "latency_p50_ms": 280,
    "latency_p95_ms": 540,
    "latency_p99_ms": 910,
    "uptime_30d": 99.94
  },
  "privacy": {
    "supports_zdr": true,
    "zdr_enabled": false,
    "training_allowed": false,
    "retention": "provider_api_policy",
    "privacy_rank": 3
  },
  "pricing": {
    "currency": "USD",
    "input_per_mtok": 1.25,
    "cached_input_per_mtok": 0.125,
    "output_per_mtok": 10,
    "effective_from": "2026-06-13T00:00:00Z",
    "effective_to": null,
    "source": "https://..."
  }
}
```

> **Note:** An unknown model returns 404. Individual telemetry keys or the complete stats object may be absent until the route has enough observations.

## Precision & Context Tiers

Quantization describes the serving route, not the abstract model family. Different providers may serve the same model at different floating-point or integer precision. Do not infer precision from the model name.

| Value | Meaning | Practical interpretation |
| --- | --- | --- |
| fp16 | IEEE 16-bit floating point | Common full/near-full serving precision |
| bf16 | Brain floating point 16 | Wide exponent range with reduced mantissa precision |
| fp8 | 8-bit floating point | Lower memory and higher throughput; quality depends on implementation |
| int8 | 8-bit integer quantization | Compressed weights/operations with possible quality trade-offs |
| int4 | 4-bit integer quantization | Aggressive compression; evaluate task quality carefully |
| unknown | Provider did not publish precision | Undisclosed, not equivalent to full precision |

context_tier is standard or extended. Catalog routes with more than 200,000 input tokens are classified as extended unless an explicitly configured route supplies another tier. Always use context_window_tokens for the exact numeric limit.

- Benchmark route precision on your real workload before using it as a quality filter.
- Use route_id in evaluation logs so precision, provider, region, and model revision stay attributable.
- Do not present unknown precision as fp16, bf16, or "full precision".
- Use min_context_tokens and preserve_context_window when fallback must not reduce usable context.

## Capabilities & Parameters

Capabilities belong to a specific route. A public model may have one provider route with vision and another without it. Build controls and routing requirements from route-level data.

| Capability | Meaning |
| --- | --- |
| streaming | Incremental response delivery |
| tools | Function/tool calling |
| parallel_tools | Multiple tool calls in one assistant turn |
| json_object | Provider-native JSON object mode |
| json_schema | Provider-native strict schema mode |
| vision | Image input |
| audio | Audio input |
| pdf | PDF/file input |
| reasoning | Reasoning/thinking support |
| prompt_caching | Prompt/context caching support |
| modalities.input | Exact input badges: text, image, audio, pdf |
| modalities.output | Current output modality list |
| supported_params | Known accepted canonical/OpenAI parameter names |

> **Note:** Never assume a parameter is supported because another route for the same model supports it. Set routing.strict_params=true when silently dropping a field would be unsafe.

## Pricing & Cost Calculation

Route prices are returned in USD per one million tokens. pricing=null means no verified active price is available; it does not mean the route is free.

| Field | Unit | Meaning |
| --- | --- | --- |
| input_per_mtok | USD / 1,000,000 tokens | Uncached input tokens |
| cached_input_per_mtok | USD / 1,000,000 tokens | Input tokens served from provider cache |
| output_per_mtok | USD / 1,000,000 tokens | Generated output tokens |
| effective_from | UTC timestamp | Beginning of the active price period |
| effective_to | UTC timestamp or null | End of the price period |
| source | URL | Pricing evidence/source |

```text cost formula
uncached_input_tokens = input_tokens - cached_input_tokens

input_cost =
  uncached_input_tokens / 1,000,000 * input_per_mtok

cached_input_cost =
  cached_input_tokens / 1,000,000 * cached_input_per_mtok

output_cost =
  output_tokens / 1,000,000 * output_per_mtok

estimated_total =
  input_cost + cached_input_cost + output_cost
```

```javascript browser cost estimate
function estimateCost({ inputTokens, cachedInputTokens, outputTokens, pricing }) {
  if (!pricing) return null;

  const million = 1_000_000;
  const cached = Math.min(cachedInputTokens ?? 0, inputTokens);
  const uncached = Math.max(0, inputTokens - cached);

  return (
    (uncached / million) * Number(pricing.input_per_mtok ?? 0) +
    (cached / million) * Number(pricing.cached_input_per_mtok ?? pricing.input_per_mtok ?? 0) +
    (outputTokens / million) * Number(pricing.output_per_mtok ?? 0)
  );
}
```

> **Note:** The receipt actual_charge_usd is authoritative for the completed request. Client-side estimates cannot predict provider-specific reasoning, cache, retry, or fallback usage exactly.

# Telemetry

## Usage Graphs

GET /v1/models/{public_model}/usage returns public aggregate request and token history for a model.

| Parameter | Accepted values | Default |
| --- | --- | --- |
| window_days | 1 through 365 | 30 |
| bucket | hour or day | day |

```bash usage graph
curl "$ASTRO_BASE_URL/v1/models/gpt-5.5/usage?window_days=30&bucket=day"
```

```json usage series
{
  "object": "list",
  "model": "gpt-5.5",
  "bucket": "day",
  "data": [
    {
      "bucket_start": "2026-06-14T00:00:00Z",
      "requests": 260000,
      "tokens": 134000000
    }
  ]
}
```

- Only observed buckets are returned.
- Insert missing buckets as zero client-side only when a continuous chart requires them.
- A known model with no activity returns an empty data array.
- An unknown model returns 404.
- Convert UTC bucket timestamps to the viewer locale only at presentation time.

## Route Telemetry

| Metric | Definition |
| --- | --- |
| throughput_tps | Median output tokens per second over successful metered attempts in the trailing 30 days |
| latency_p50_ms | Median successful request latency in the trailing 30 days |
| latency_p95_ms | 95th percentile successful request latency |
| latency_p99_ms | 99th percentile successful request latency |
| uptime_30d | Successes divided by successes plus availability-impacting failures |

Quality failures such as structured-output validation do not reduce uptime. They remain visible as quality_failures and in error series. This prevents a provider that is reachable but produced invalid output from being mislabeled as offline.

- Sort by uptime descending for the default health-first route table.
- Place routes without uptime after measured routes.
- Use input price as the second sort key and priority as the final tie-breaker.
- Hide a missing metric instead of rendering zero throughput, zero latency, or zero uptime.
- Label percentiles explicitly; p50, p95, and p99 answer different user questions.

## Status & Health

Status scopes are providers, models, and routes. Snapshots summarize a recent window; history endpoints return chart buckets.

```bash snapshot
curl "$ASTRO_BASE_URL/v1/status/routes?window_hours=24"
```

| Snapshot field | Meaning |
| --- | --- |
| status | unknown, operational, degraded, or outage |
| events | All health events in the selected window |
| observations | Successes plus availability-impacting failures |
| successes | Successful provider observations |
| availability_failures | Failures that count against uptime |
| quality_failures | Reachable-provider failures that do not count as downtime |
| uptime_percentage | Availability percentage or null when unobserved |
| average_latency_ms | Mean successful latency |
| p95_latency_ms | 95th percentile successful latency |
| last_observed_at | Most recent health event |
| last_failure_at | Most recent failure event |

```bash history
curl "$ASTRO_BASE_URL/v1/status/routes/openai-gpt-5.5-global-standard/history?window_hours=168&bucket=hour"
```

Snapshot window_hours accepts 1 through 720. History window_hours accepts 1 through 2160. History bucket accepts hour or day.

## Error Graphs

```bash error series
curl "$ASTRO_BASE_URL/v1/status/errors?scope=routes&window_hours=168&bucket=hour"
```

```json error point
{
  "bucket_start": "2026-06-14T14:00:00Z",
  "target": "openai-gpt-5.5-global-standard",
  "category": "rate_limit",
  "code": "provider_rate_limited",
  "availability_impact": true,
  "count": 12
}
```

Use availability_impact to separate outages, connection failures, and rate limits from output-quality, configuration, and tool/structure failures. A green availability graph can coexist with visible quality failures.

> **Note:** Do not combine every error into one downtime percentage. Availability and quality are separate operational signals.

## Receipts & Activity

A completed canonical response includes routing metadata, usage, warnings, and a request receipt. Authenticated users can fetch the durable activity record at GET /v1/activity/requests/{request_id}.

| Evidence | What it answers |
| --- | --- |
| selected_route | Which exact route answered? |
| provider / provider_model | Which provider and upstream model were used? |
| billing_mode / credential_source | Was the provider paid by platform or BYOK? |
| fallback_used / attempts | Did retries or fallback occur? |
| decision_reason | Why was this route selected? |
| estimated_charge_usd | What was predicted before execution? |
| actual_charge_usd | What was charged after actual usage? |
| pricing_source | Where did the price evidence come from? |
| cache receipt | Was provider prompt caching reported? |
| privacy receipt | What privacy policy was enforced? |
| warnings | What safe transformations or compatibility compromises occurred? |

> **Note:** Use request IDs in support tooling. Public errors contain a support code, while authenticated activity exposes the user-safe execution history without leaking provider secrets.

### Sessions, tasks, and content capture

Set session_id/session_name and task_id/task_name on canonical requests to build an app → session → task → request hierarchy. Every response returns both its request id and session_id. Reuse session_id across turns to group the complete execution history.

| Endpoint | Purpose |
| --- | --- |
| GET /v1/activity/requests | Search/filter requests by IDs, names, prompt text, model, app, status, USD cost, and latency |
| GET /v1/activity/requests/{id}?include_content=true | Detailed route, retry, trimming, timing, guardrail, prompt, tool, and completion trace |
| GET /v1/activity/sessions | List grouped sessions |
| GET /v1/activity/sessions/{id} | All requests and events in one session |
| GET\|PUT /v1/organizations/current/content-logging | Organization capture and retention policy |

> **Note:** Exact prompt/completion capture is disabled by default. Owner/admin permission is required to enable or view it. API-key policies can inherit, enable, or disable request and completion capture. Retained content is automatically erased after 1-365 days.

# Platform

## Provider Directory

```bash providers
curl "$ASTRO_BASE_URL/v1/providers"
```

```json provider item
{
  "id": "openai",
  "object": "provider",
  "model_company": "openai",
  "endpoint_type": "serverless"
}
```

provider is the serving vendor or platform. model_company is the company that owns the model family. These can differ for routes served through cloud platforms.

## Privacy Controls

| Route privacy field | Meaning |
| --- | --- |
| supports_zdr | The route can support a zero-data-retention configuration |
| zdr_enabled | ZDR is currently enabled for the route |
| training_allowed | Whether provider training is allowed under the route policy |
| retention | Human-readable retention policy classification |
| privacy_rank | Internal relative privacy strength used in routing |

```json strict privacy request
{
  "model": "gpt-5.5",
  "messages": [
    { "role": "user", "content": "Analyze confidential customer data." }
  ],
  "routing": {
    "require_zdr": true,
    "allow_provider_training": false,
    "fallbacks": {
      "preserve_privacy": true
    }
  }
}
```

- Check the route privacy block before displaying a ZDR badge.
- supports_zdr does not mean ZDR is enabled; inspect zdr_enabled or enforce require_zdr.
- Keep preserve_privacy enabled when fallback must not weaken the original policy.
- The final request receipt records the selected provider, region, ZDR enforcement, training policy, and retention classification.

## Bring Your Own Key

BYOK credentials are encrypted and organization-scoped. The upstream provider bills your provider account. ML Junction can charge a separate routing fee, bounded by max_byok_fee_usd.

| Mode | Behavior |
| --- | --- |
| platform_only | Use platform-funded provider credentials only |
| byok_strict | Use customer BYOK credentials only; no platform-funded fallback |
| byok_first_paid_fallback | Try BYOK first, then platform-funded fallback when allowed |
| platform_first_byok_fallback | Try platform funding first, then BYOK fallback |

```json BYOK strict request
{
  "model": "gpt-5.5",
  "messages": [
    { "role": "user", "content": "Run this only with my provider account." }
  ],
  "routing": {
    "mode": "byok_strict",
    "production_mode": true,
    "provider": "openai",
    "max_byok_fee_usd": 0.01
  }
}
```

> **Note:** BYOK still requires enough platform balance to settle the routing fee. The receipt distinguishes billing_mode and credential_source so users can verify who paid the provider.

## Embeddings

POST /v1/embeddings follows the OpenAI embedding shape and uses the same gateway authentication, governance, routing, usage recording, and billing infrastructure.

```bash embedding request
curl "$ASTRO_BASE_URL/v1/embeddings" \
  -H "Authorization: Bearer $ASTRO_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "openai-embedding",
    "input": [
      "Capability-aware routing",
      "Provider-independent embeddings"
    ]
  }'
```

- Select an embedding model from GET /v1/models rather than using a chat model.
- Batch inputs when your latency and provider limits allow it.
- Store the exact model and route identity with generated vectors.
- Do not mix vectors from incompatible embedding models in one index without an explicit migration strategy.

# Operate

## Errors

Gateway errors use a stable envelope with a sanitized public message. Internal provider details and stack traces are retained in protected logs rather than returned to clients.

```json error envelope
{
  "error": {
    "type": "provider_error",
    "message": "The provider could not complete the request.",
    "code": "provider_execution_failed",
    "request_id": "req_...",
    "support_code": "support_...",
    "retryable": true,
    "details": {}
  }
}
```

| Category | Meaning | Client action |
| --- | --- | --- |
| validation_error | Request shape or field constraint failed | Fix the request; do not retry unchanged |
| authentication_error | Missing or invalid credential | Refresh or replace the credential |
| authorization_error | Permission or key policy denied the action | Change policy or request |
| rate_limit | Gateway or provider rate limit | Honor retry guidance and add backoff |
| timeout | Provider exceeded the time budget | Retry only when the operation is safe |
| provider_error | Upstream provider failure | Use retryable and fallback evidence |
| structured_output | Output failed required validation | Review schema and repair settings |
| tool_call | Tool-call generation or validation failed | Validate tool definitions and arguments |
| configuration_error | No route or provider configuration can satisfy the request | Inspect catalog and routing preview |
| conflict_error | An idempotent request is already running | Wait for the original request |

### Continuation recovery codes

| Code | Warning or error | Client behavior |
| --- | --- | --- |
| REASONING_CONTINUATION_EXPIRED | Warning; request may succeed | Continue from visible history and store the fresh token returned by the successful response |
| PRIOR_REASONING_NOT_REUSABLE | Warning; request may succeed | Show that prior native reasoning was not reused; visible history remains authoritative |
| PRIOR_REASONING_WILL_NOT_BE_RENDERED | Warning; request may succeed | Do not claim native reasoning was replayed; keep the successful fresh state if returned |
| tool_history.dropped | Warning; request may succeed | Inspect reason/requested/effective and correct duplicate, orphaned, or unknown tool results |
| PENDING_TOOL_LOOP | HTTP 409 error | Use details.missing_tool_call_ids and details.actions; do not retry unchanged |
| INVALID_REASONING_CONTINUATION | HTTP 400 error | Discard the malformed/tampered token and retry as a fresh history-only turn |
| PAYLOAD_TOO_LARGE | HTTP 413 error | Reduce continuation/native detail size or conversation payload; do not retry unchanged |

```javascript retry policy
async function callWithBackoff(call, maxAttempts = 3) {
  let lastError;

  for (let attempt = 1; attempt <= maxAttempts; attempt += 1) {
    try {
      return await call();
    } catch (error) {
      lastError = error;
      const retryable = error?.body?.error?.retryable === true;
      if (!retryable || attempt === maxAttempts) throw error;

      const delayMs = Math.min(8000, 500 * 2 ** (attempt - 1));
      await new Promise((resolve) => setTimeout(resolve, delayMs));
    }
  }

  throw lastError;
}
```

## Key Controls: Limits, IP & CORS

Every API key carries its own policy, editable from the key’s Controls tab. Spend caps are enforced per rolling window; when any window is exceeded the key is blocked until that window resets.

Above per-key limits, your account has a tier-based, account-wide ceiling on requests/min, tokens/min and concurrency that is shared across every key in the org - so adding more keys never raises your total throughput. A request must pass both the key limit and the account ceiling. Your tier rises automatically as cumulative purchased credits cross each threshold; the current tier, live usage, and the credit needed for the next tier are shown on the Billing page. A per-key limit set above the account ceiling is allowed but still capped at the ceiling. Hitting either limit returns a 429 with a rate_limit error.

| Control | What it does |
| --- | --- |
| Account tier ceiling | Account-wide requests/min, tokens/min and concurrency shared across all keys; scales with purchased credits |
| Spend budgets | Cap USD per hour, day, week, month, or a custom window |
| Rate limits | Per-key requests/min, tokens/min, concurrency, and requests/min per client IP (capped by the account tier) |
| Model / provider / tier allowlists | Restrict a key to specific models, providers, or service tiers |
| IP allow / deny | Permit or block by IP or CIDR range |
| CORS | Per-key allowed origins, methods, headers, and credentials for browser apps |
| App attribution | Send an X-App header to tag and segment usage by your own application |

Client IPs are resolved with a configured trusted-proxy hop count, so X-Forwarded-For values added before your Cloudflare/nginx edge cannot be spoofed. Every request records its IP, app id, service tier and user agent - visible in Activity and broken down in Usage analytics.

```bash attribute a call to your app
curl $BASE_URL/v1/responses -H "Authorization: Bearer $KEY" \
  -H "X-App: checkout-web" \
  -H "Content-Type: application/json" \
  -d '{"model":"gpt-5.5","messages":[{"role":"user","content":"hi"}]}'
```

## Idempotency & Safe Retries

Set idempotency_key for important non-streaming operations. The key is scoped to the gateway API key. A completed result is cached for one hour, and a concurrent duplicate returns a conflict instead of executing twice.

- Generate one stable idempotency key per logical operation, not per HTTP retry.
- Do not reuse a key for different prompts or users.
- Retry validation and authorization failures only after changing the request or policy.
- Retry network failures, rate limits, and retryable provider errors with exponential backoff and jitter.
- Streaming retries require application-level handling because partial content may already be visible.

## Caching

Public model and route endpoints return Cache-Control: public, max-age=60, stale-while-revalidate=300. Their metrics are rolling aggregates, so clients should revalidate rather than treating them as immutable.

| Data | Recommended client behavior |
| --- | --- |
| Model/provider catalog | Cache for about 60 seconds and revalidate in the background |
| Route pricing/capabilities | Use the public cache header; refresh before critical route decisions |
| Popularity and telemetry | Treat as rolling snapshots, not billing records |
| Status graphs | Poll according to UI freshness needs; avoid sub-second polling |
| Inference responses | Cache only when application semantics, privacy, and user isolation permit it |
| Streaming responses | Never intermediary-cache |

> **Note:** Provider prompt caching and HTTP response caching are different systems. Prompt-cache evidence appears in the request receipt and token usage. A continuation token can improve the chance of returning to the same provider surface, but it does not reserve a provider cache entry or keep that cache alive. Expired or unavailable continuation state falls back to visible history and normal routing.

## Frontend Integration Rules

- Hide an unavailable metric key instead of substituting zero.
- Render pricing=null as "price unavailable", never "$0".
- Render quantization="unknown" as undisclosed precision.
- Use route_id as the table key and status-history target.
- Use stats.rank for leaderboard order; do not invent a rank for models without stats.
- Insert missing graph buckets only for visual continuity.
- Convert UTC timestamps to the user locale in the presentation layer.
- Build modality and parameter controls from route capabilities.
- Show API error.message first, then support_code for support workflows.
- Treat warnings as structured objects; show continuation expiry/fallback warnings without exposing the token or opaque reasoning fields.
- Store reasoning_details on the assistant turn that produced them and keep continuation tokens scoped to one conversation branch.
- Use skeletons during catalog and graph loading; keep prior data visible during background refresh.

```javascript catalog fetch
const response = await fetch('/v1/models', {
  headers: { Accept: 'application/json' },
});

if (!response.ok) {
  const body = await response.json().catch(() => null);
  throw new Error(body?.error?.message ?? `Model request failed (${response.status})`);
}

const { data: models } = await response.json();
const rankedModels = models
  .filter((model) => model.stats?.rank != null)
  .sort((a, b) => a.stats.rank - b.stats.rank);
```

```javascript route display guards
function toRouteView(route) {
  return {
    id: route.route_id,
    uptime: route.stats?.uptime_30d ?? null,
    p95Latency: route.stats?.latency_p95_ms ?? null,
    throughput: route.stats?.throughput_tps ?? null,
    inputPrice: route.pricing?.input_per_mtok ?? null,
    precision:
      route.quantization === 'unknown'
        ? 'Undisclosed'
        : route.quantization.toUpperCase(),
    modalities: route.capabilities?.modalities?.input ?? ['text'],
    supportedParams: route.capabilities?.supported_params ?? [],
  };
}
```

## Production Checklist

- Use a production API key with explicit model, RPM, TPM, concurrency, and output-token limits.
- Use routing.preview to verify at least one route satisfies every required capability and privacy rule.
- Enable production_mode for deterministic provider/route controls and charge caps.
- Set strict_params when silently ignored parameters could change behavior.
- Set max_platform_charge_usd and max_byok_fee_usd according to your risk budget.
- Preserve privacy and context capacity across fallback.
- Use idempotency keys for side-effecting or expensive non-streaming workflows.
- Validate structured output and tool arguments in your application as well as at the gateway.
- Log request ID, route ID, provider, model, usage, actual charge, latency, and finish reason.
- Monitor uptime separately from quality failures.
- Alert on p95/p99 latency, availability failures, rate limits, and unexpected fallback frequency.
- Evaluate every route and quantization used in production on representative tasks.
- Keep provider and gateway credentials in a secret manager and rotate them regularly.
- Test cancellation, timeout, rate-limit, malformed-output, and provider-outage paths before launch.
- Test continuation with a valid token, no token, expired token, removed route/account, provider outage, model switch, missing reasoning details, tool-result mismatch, and tampered token.

## Reference Limits

| Field or endpoint | Limit |
| --- | --- |
| model | 1 through 150 characters |
| messages | At least one; must include a user message |
| output.max_tokens | 1 through 131,072 |
| tools | Up to 128 |
| idempotency_key | Up to 200 characters |
| routing.route | Up to 200 characters |
| only_providers / ignored_providers | Up to 20 entries each |
| fallback max_total_attempts | 1 through 10 |
| fallback max_retries_per_route | 0 through 3 |
| sticky ttl_seconds | 60 through 86,400 |
| reasoning.continuation_token | 20 through 131,072 characters at request validation; encrypted payload is also constrained by the server continuation byte limit |
| reasoning continuation soft TTL | Deployment setting; default 86,400 seconds, allowed 60 through 31,536,000 seconds |
| usage window_days | 1 through 365 |
| status snapshot window_hours | 1 through 720 |
| status history/error window_hours | 1 through 2,160 |
| graph bucket | hour or day |

## Complete Native Option Reference

This table mirrors the current /v1/responses request contract. Unknown fields are rejected. Defaults shown here are gateway defaults before governance policy narrows them.

| Path | Type / values | Default or constraint |
| --- | --- | --- |
| model | string | Required; 1-150 characters |
| messages | Message[] | Required; at least one user message |
| messages[].role | system \| developer \| user \| assistant \| tool | Required |
| messages[].content | string \| ContentPart[] \| null | Text or typed multimodal blocks |
| messages[].tool_calls | object[] | Assistant tool calls; preserve IDs and ordering |
| messages[].tool_call_id | string \| null | Required on tool-result messages |
| messages[].reasoning_details | object[] | Public provider-native state attached to the assistant turn that produced it |
| ContentPart.type | text \| image_url \| input_audio \| file \| thinking \| redacted_thinking | Opaque thinking is provider-bound |
| stream | boolean | false |
| sampling.temperature | 0-2 \| null | null |
| sampling.top_p | >0-1 \| null | null |
| sampling.seed | integer \| null | null |
| sampling.frequency_penalty | -2-2 \| null | null |
| sampling.presence_penalty | -2-2 \| null | null |
| sampling.stop | string \| string[] \| null | null |
| reasoning.enabled | boolean | false |
| reasoning.effort | none \| minimal \| low \| medium \| high \| xhigh \| max \| auto | auto |
| reasoning.intensity | integer 0-100 \| null | null |
| reasoning.summary | none \| auto \| concise \| detailed | none |
| reasoning.context | auto \| current_turn \| all_turns | auto |
| reasoning.continuation_token | opaque string \| null | null; gwrt_v3 tokens are optional and must not be decoded by clients |
| reasoning.input_type | user_message \| tool_results \| dynamic_context_update | user_message |
| reasoning.pending_turn_action | continue \| cancel \| fork \| null | null |
| output.max_tokens | 1-131072 \| null | Platform default when null |
| output.format.type | text \| json_object \| json_schema | text |
| output.format.schema | JSON Schema \| null | Required for json_schema |
| output.format.name | string | response |
| output.format.strict | boolean | true |
| output.streaming_mode | text \| buffered \| raw_compat | text; structured output becomes buffered |
| output.validation.enabled | boolean | true |
| output.validation.heal_syntax | boolean | false |
| output.validation.model_repair | boolean | false |
| output.validation.max_repair_attempts | 0-2 | 1 |
| tools | FunctionTool[] | Maximum 128 |
| tool_choice | string \| object \| null | Provider-compatible selection |
| requirements.tools/json_schema/vision/audio/pdf | required \| preferred \| off | off |
| requirements.reasoning | required \| preferred \| off | preferred |
| requirements.streaming | boolean | Derived from stream |
| requirements.min_context_tokens | positive integer \| null | null |
| routing.strategy | quality \| cost \| latency \| balanced \| pinned | balanced |
| routing.preset | stable_production \| lowest_cost \| lowest_latency \| strongest_privacy \| long_context \| null | null |
| routing.production_mode | boolean | false; requires explicit route/provider and charge caps |
| routing.mode | platform_only \| byok_strict \| byok_first_paid_fallback \| platform_first_byok_fallback | platform_only |
| routing.strict_params | boolean | false |
| routing.service_tier | auto \| standard \| priority \| flex | auto |
| routing.route / provider / region | string \| null | null |
| routing.only_providers / ignored_providers | string[] | Maximum 20 each; cannot overlap |
| routing.require_zdr | boolean | false |
| routing.allow_provider_training | boolean | false |
| routing.max_platform_charge_usd | positive number \| null | null |
| routing.max_byok_fee_usd | positive number \| null | null |
| routing.fallbacks.provider / model | boolean | true / true |
| routing.fallbacks.max_total_attempts | 1-10 | 5 |
| routing.fallbacks.max_retries_per_route | 0-3 | 1 |
| routing.fallbacks.same_or_lower_price | boolean | false |
| routing.fallbacks.preserve_context_window / preserve_privacy | boolean | true / true |
| routing.fallbacks.max_price_increase_percent | 0-1000 \| null | null |
| routing.sticky.enabled | boolean | true |
| routing.sticky.scope | conversation \| api_key \| custom | conversation |
| routing.sticky.affinity_key | string ≤200 \| null | Conversation metadata fallback |
| routing.sticky.ttl_seconds | 60-86400 | 3600 |
| context.mode | auto_fit \| strict | auto_fit |
| context.input_budget_ratio | >0.1-0.95 | 0.70 |
| context.min_output_reserve_tokens | ≥256 | 4096 |
| context.preserve_system/current_user/tool_chains | boolean | true / true / true |
| metadata | object | {} |
| idempotency_key | string ≤200 \| null | null |
| session_id / task_id | string ≤100 \| null | Generated/optional |
| session_name / task_name | string ≤200 \| null | null |
| compatibility | object | {}; protocol/adapter hints |

> **Note:** App name is carried by the X-App request header rather than the JSON body. Responses return app_name together with session and task identity.

## Glossary

| Term | Definition |
| --- | --- |
| Public model | Stable model ID used by client requests |
| Route | Exact provider, provider model, region, endpoint, precision, and policy combination |
| Provider | Vendor or cloud platform serving the request |
| Model company | Company that owns the underlying model family |
| Fallback | A later eligible route attempted after the primary route fails |
| Availability failure | Failure that counts against route uptime |
| Quality failure | Reachable-provider failure that does not indicate downtime |
| ZDR | Zero data retention |
| BYOK | Bring your own provider API key |
| MTok | One million tokens |
| p50 / p95 / p99 | Latency percentile thresholds |
| Throughput | Generated output tokens per second |
| Context window | Maximum combined input/output token capacity |
| Quantization | Numeric precision used to serve model weights/operations |
| Sticky routing | Health-aware affinity to a previously selected route |
| Receipt | Durable evidence of route, cost, cache, privacy, and fallback behavior |

---
_For AI assistants: use only the facts in this document. Do not invent endpoints,
pricing, limits, model IDs, or SDK behaviour. The exact model catalogue comes from
GET /v1/models. Warn before any production, billing, security, or deletion change._