Routers

A router is a virtual model. Instead of naming one model in a chat, completion or Responses request, you name the router (its id or its name), and the AI Gateway sends each request to one of the router's member models by the router's strategy. Your application never changes when you add a model, shift traffic or let the router learn which model works best.

Why Routers?

• Failover: when the chosen model fails, the next one answers
• Right-sizing: send simple or short requests to smaller models
• Learning: bandit strategies learn from rewards which model earns the best results, and shift traffic to it
• Prompt-aware routing: route by what a prompt is about, how complex it is, or its similarity to example prompts
• Safe rollouts: canary a new model on a set share of traffic

Creating a Router

1. Go to AI Gateway > Routers and click Create Router
2. Enter a Name and pick a Strategy
3. Add members from the models you can use. For each member set:
   • Priority: lower is preferred. Priority decides failover order and which member is "most capable"
   • Weight (weighted strategy): the member's relative share of traffic
   • Handles (content-type strategy): the kinds of prompts the member is good at
4. Fill in the strategy settings (below)
5. Choose what happens when the chosen member fails: try the next member by priority, or return its error
6. Optionally turn on the health check

The editor tells you what is missing before you can save (for example, a complexity router needs the member that gets simple prompts).

Using a Router

from strongly import Strongly

client = Strongly()

response = client.ai.inference.chat_completion(
    model="support-chat",   # a router's name or id
    messages=[{"role": "user", "content": "Where is my order?"}],
)
print(response.content)
print(response.router_decision.model_id, response.router_decision.reason)

Every routed response names the router decision behind it:

• router_decision in the response body: id, router_id, router_name, strategy, model_id (the chosen member) and reason
• the X-Router-Decision-Id response header
• on a streamed response, router_decision on the first chunk

Routers also appear in the platform's model pickers, so workflow AI nodes, agents and apps can use them like any model.

Who Can Use a Router

A router is reached by its owner, the users it is shared with and admins; to anyone else it does not exist (a call naming it is answered as an unknown model). A router calls its members as its owner: each member must be a model its owner may use, whoever calls the router.

Strategies

Strategy	What it does	Settings
Priority	The highest-priority healthy member; the next one when it fails	none
Weighted	Splits traffic by member weights	Sticky: the same caller always gets the same member
Round Robin	Each healthy member in turn	none
Latency Optimized	The member with the lowest recent latency; members not yet measured are tried first	none
Thompson Sampling	Learns from rewards: samples each member's reward belief and picks the highest	Reward
UCB1	Learns from rewards: average reward plus an exploration bonus; every member is tried before any is preferred	Reward, exploration bonus (c)
Epsilon Greedy	Learns from rewards: the best-earning member, except a set share of requests explores at random	Reward, epsilon
LinUCB	Learns from rewards which member suits which kind of request	Reward, exploration (alpha), request features
Complexity	Simple prompts to the member you choose, complex prompts to the most capable	simple member, optional complex member
Content Type	By what the prompt is about: code, data, creative, operations or general	a member per type (optional), default member
Token Budget	Short expected answers to the member you choose, long ones to the most capable	long-answer threshold (tokens), short-answer member
Semantic	By embedding similarity of the prompt to each route's example prompts	embedding model, similarity threshold, default member, routes
RouteLLM	The LMSYS RouteLLM matrix-factorization router: strong or weak member by the strong model's predicted win rate	text-embedding-3-small model, threshold
Canary	A set percentage of traffic to a canary member, the rest to the control	control, canary, canary %
Custom	Your own Python picks the member	code

A router whose settings its strategy cannot act on (for example a canary naming a member that was removed) refuses requests with a 503 that says what to fix; it never routes some other way.

Images and Documents

A request that carries images or PDF documents goes only to a member that accepts them: a model takes images when it declares image input (capabilityConfig.supportsVision) and PDFs when it declares PDF input (capabilityConfig.supportsPdf). Every strategy chooses among those members only, and failover skips the others:

• A strategy that picks among members (priority, weighted, round robin, latency, the learning strategies, custom) picks among the members that accept them.
• A setting that names a member that does not accept them (the canary or control, a semantic route or default, the complexity, content-type or token-budget member) gives way to the highest-priority member that does; the decision's reason says so.
• RouteLLM with only one member that accepts them answers with that member.
• With no member that accepts them, the request is refused with a 400 that names what each member lacks.

Learning Strategies (Bandits)

Thompson Sampling, UCB1, Epsilon Greedy and LinUCB learn from rewards. Every reward is a number from 0 to 1 for one routed response, and the router's next decisions read every reward it has received.

The reward

Pick what a reward is when you create the router:

Reward	Meaning	Scored by
Success	1 when the member answers, 0 when it fails	the gateway, on every response
Latency	1 - latency / max latency (0 at or beyond the max you set)	the gateway, on every response
Feedback	your own score for each response	you, with feedback

Whatever the reward, a member that was chosen and failed earns 0 (even when the next member answered the request). Responses served from a cache and requests refused before any member was called earn nothing.

Settings

• Epsilon (Epsilon Greedy): the share of requests, from 0 to 1, sent to a random member to keep exploring. 0.1 by default.
• Exploration bonus (UCB1): c in average reward + c x sqrt(ln(total rewards) / member rewards). 1.41 by default; 0 only exploits.
• Exploration (alpha) (LinUCB): how strongly uncertain members are tried. 1.0 by default.
• Max latency: where the latency reward reaches 0.

LinUCB request features

LinUCB learns a model per member over features read from each request, so it can learn, for example, that one model does better on code questions and another on long conversations. Every value is between 0 and 1, and each member also learns its own baseline:

Feature	Value
Conversation length	length of all message text, log scale
Last user message length	log scale
User turns	number of user messages, up to 20
Has a system prompt	1 or 0
Offers tools	1 or 0
Asks for JSON output	1 or 0
Contains images	1 or 0
Classified complex	1 when the complexity classifier calls the prompt complex
About code / data / creative writing / operations	1 for the prompt's content type

Changing a LinUCB router's features starts its learning afresh (a model over other features does not carry over). Changing any other setting, the strategy or the members keeps what the router has learned.

Feedback

A router whose reward is Feedback learns from your scores. Take the decision id from the routed response and send a reward from 0 (bad) to 1 (good):

client.agent_router.feedback(
    router.id,
    decision_id=response.router_decision.id,
    reward=0.9,
    label="resolved the customer's question",   # optional
)

Or over REST:

POST /api/v1/ai-gateway/routers/{routerId}/feedback
{ "decision_id": "<router_decision.id>", "reward": 0.9, "label": "optional note" }

• One reward per decision: a second score for the same decision is refused (409).
• A router scored by the gateway (success, latency) refuses feedback (409): its rewards are already computed.
• A decision whose chosen member failed already has its reward (0).
• You can also score recent responses from the router's Stats on the Routers page.

Stats and Decisions

Open a router's Stats (chart icon on its card) to see, for any window from the last hour to the last 30 days:

• per member: decisions, share of the router's traffic, answered / failed / failed over and average latency
• for learning routers: the average reward in the window, and over the router's life its pulls (rewards received), average reward and last reward
• for LinUCB: the reward weight each member learned for each request feature
• the router's decisions, newest first: the member chosen, the strategy's reason, who answered, latency and the reward (with a thumbs up / down to score a pending response of a feedback router)

The same numbers come from client.agent_router.stats() / decisions() and GET /api/v1/ai-gateway/routers/{id}/stats / decisions.

Strategy Details

Complexity

Each prompt is classified simple or complex without a model call, from the last user message: short messages, slash commands and status-style questions are simple; multi-step requests (build, design, debug, write), long messages with several sentences and deep conversations are complex. Simple prompts go to the member you pick; complex ones to the member you pick (or the most capable: lowest priority number).

Content Type

Each prompt is classified as code, data, creative, ops or general by its keywords. The router sends it to the member you mapped to that type, else to a member whose Handles include it, else to the default member.

Token Budget

The router estimates how long the answer will be (short questions about 100 tokens, "how / why / compare" about 500, "explain / write / build" about 2000). Answers up to the threshold (500 by default) go to the short-answer member you pick; longer ones to the most capable.

Semantic

See Semantic Routing.

RouteLLM

RouteLLM (LMSYS) predicts from the prompt how likely a strong model's answer is to beat a weak model's. The router needs exactly two members: the lower priority number is the strong member. It embeds the prompt with a text-embedding-3-small model you choose (the space the RouteLLM checkpoint was trained in; add it as a third-party model first) and sends the request to the strong member when the predicted win rate reaches the threshold.

The threshold decides the split: RouteLLM's published calibration of 0.11593 sends about half of typical traffic to the strong member; raise it to send less (and save more), lower it to send more.

Canary

Set the control member, the canary member and the canary percentage. Raise the percentage as the canary proves itself (10%, 25%, 50%, 100%).

Custom

Write Python that sets result to a member's model id. It can read variants (each member's id, weight, priority and capabilities), request_context (user_id, messages, last_user_message) and history (total_requests, variant_counts: decisions per member), and use math, random, json, re and datetime. Code that fails, runs longer than 5 seconds or names a non-member refuses the request with the reason.

Health Checks

With the health check on, a member that fails several requests in a row (3 by default) is skipped until it recovers; if every member is unhealthy, the router refuses requests with the reason for each.

REST API

See Routers API for every endpoint:

GET    /api/v1/ai-gateway/routers                         List routers
POST   /api/v1/ai-gateway/routers                         Create a router
GET    /api/v1/ai-gateway/routers/{id}                    Get a router
PATCH    /api/v1/ai-gateway/routers/{id}                    Update a router
DELETE /api/v1/ai-gateway/routers/{id}                    Delete a router
POST   /api/v1/ai-gateway/routers/{id}/members            Add a member model
DELETE /api/v1/ai-gateway/routers/{id}/members/{modelId}  Remove a member model
GET    /api/v1/ai-gateway/routers/{id}/stats              What the router did and learned
GET    /api/v1/ai-gateway/routers/{id}/decisions          The router's decisions, newest first
POST   /api/v1/ai-gateway/routers/{id}/feedback           Score a routed response