Routing

One API. 151+ models.

Call Alibaba US (Qwen, DeepSeek — our largest catalog), OpenAI (GPT, o-series), AWS Bedrock (Claude, Llama, DeepSeek, Nova, Qwen), Azure Foundry, Google Vertex AI (Gemini), and Anthropic (Claude) live today through a single endpoint. Cost, latency, and weighted routing, deterministic A/B splits, configurable fallback chains — config, not redeploy.

Get startedSee the pipeline
routing-decision · request_id 7f2a

Live routing trace

Strategycost
Routed byx-nr-model
Primaryvertex/gemini-2.5-flash
Fallback chainanthropic → bedrock
Retries used0
Routing overhead< 1 ms
cost-routedfallback-readyconfig-not-redeploy
Routing strategies
3

Cost, latency, weighted, per org

Retries
Configurable

Per-org retry count + per-request override

A/B splits
Deterministic

Model + prompt variants, retry-safe

Fallback chains
Opt-in

On error + on rate limit, cross-provider

Strategies

Three strategies. One config field.

Pick the signal your workload cares about — cost, latency, or your own weighted split. The router does the rest, per org, no redeploy.

costprice per token

Picks the cheapest capable model in your candidate set. Optimize the bill, not the benchmark.

latencyrecent success-weighted latency

Steers each request to the candidate with the lowest recent success-weighted average latency for your org — a moving average over recent successful calls, weighted toward the newest.

weightedyour split

Splits traffic across the candidates in one Smart Router by the weight you give each. You call the router by the alias you named it.

How it works

A routing decision, end to end

Strategy match, primary provider, fallback chain, settle. A substitution is surfaced on the x-nr-model response header — the decision is made in process, with no extra network hop.

Routing decision flow

  1. Request

    POST /v1/chat/completions

    Bearer sk-nrouter-..., model="gemini-2.5-flash"

  2. Strategy match

    cost · candidate set

    Alias resolves to the cheapest capable model in your set.

  3. Primary provider

    Vertex AI · gemini-2.5-flash

    429, 503, or 529 triggers the fallback chain.

  4. Fallback chain

    Anthropic → Bedrock → OpenAI

    Ordered backup list per model; each link retried in turn.

  5. Settled

    cost + latency logged

    Strategy, primary, fallback, retries: all in the request log.

Drop-in

Two-line migration from OpenAI

OpenAI-compatible

Change the base URL and the key. That’s it

The unmodified OpenAI SDK works against nRouter on every supported language. Streaming, tool use, JSON mode, embeddings, image generation — all proxied transparently.

  • Python, Node.js, Go, Java, Ruby, PHP, cURL
  • Base URL: https://api.nrouter.ai/v1
  • Auth: Bearer sk-nrouter-...
  • Drop-in for chat.completions, embeddings, and images
diff · client.py

Two-line change

- base_urlhttps://api.openai.com/v1
+ base_urlhttps://api.nrouter.ai/v1
- api_keysk-proj-...
+ api_keysk-nrouter-...
Code lines changed2
OpenAI SDK7 languagesstreaming-safe
Full routing reference — strategies, fallbacks, retries, A/B
Fallback chains
Ordered backup list per model — on error and on rate limit. A 429, 503, or 529 triggers the next link, cross-provider (Vertex → Anthropic → Bedrock). Opt-in per org; the chain is configured, not coded.
Retries + timeouts
Per-org retry count (default 3) and request timeout, with a per-request override. Provider retry-after hints are honored on the retry.
Deterministic A/B splits
Split a model or prompt into variants and bucket traffic deterministically. The same authenticated virtual key stays on the same variant, so retries are consistent and results are attributable.
Never depend on one model
A Smart Router walks its candidates in order and skips any that no longer resolves to a served model, so one retired model does not take your alias down. If none of them is servable the call is refused, naming your alias — never quietly redirected to a model you did not choose.
OpenAI-compatible surface
Same chat.completions, embeddings, and images endpoints. Base URL https://api.nrouter.ai/v1, auth Bearer sk-nrouter-... Streaming, tool use, and JSON mode proxied transparently.
Live model catalog
Capabilities and fallback chains live in the catalog, not your repo. New models are added within hours of provider launch — public read-only catalog at /api/public/models.
FAQ

Common routing questions

What latency does routing add?

Routing decisions are made in process, with no extra network hop between your call and the provider. We have not published a latency study, so we quote no measured overhead figure here; the dominant factor is always LLM inference time.

How many models are supported?

151+ models live now on Alibaba US, OpenAI, AWS Bedrock, Azure Foundry, Google Vertex AI & Anthropic; Alibaba US (Qwen, DeepSeek — our largest catalog), OpenAI (GPT, o-series), AWS Bedrock (Claude, Llama, DeepSeek, Nova, Qwen), Azure Foundry, Google Vertex AI (Gemini), and Anthropic (Claude) live today. We add new models within hours of provider launch. Check the model catalog for the full list.

Can I use custom routing strategies?

Yes. Create a Smart Router in your dashboard, give it a candidate set and a strategy — cost (cheapest capable candidate), latency (lowest recent success-weighted average latency), or weighted (your own split across those candidates) — then call it by the alias you gave it. The router resolves a candidate per request.

What happens when a provider goes down?

nRouter automatically retries with the configured fallback chain. If the primary model returns a 429, 503, or 529, we transparently retry on a backup model.

One key. One bill. Every model.

Replace your homegrown failover layer in an afternoon

Sign up, paste your virtual key, change the base URL. Cost / latency / weighted routing, A/B splits, and fallback chains are unlocked on every plan.

Get startedBrowse models