Semantic Routing

A semantic router sends each prompt to the member whose example prompts it most resembles. It uses one of your own embedding models, so you control where prompts go and what it costs.

How It Works

1. Define routes, each with a member and example prompts:
   • "simple_qa" to GPT-4o-mini: "what is X?", "define Y"
   • "code" to a code model: "write a function", "debug this"
   • "analysis" to GPT-4o: "analyze this data", "compare A and B"
2. Each route's centroid is the average embedding of its examples, computed once and reused until you change the examples or the embedding model.
3. At request time the last user message is embedded and compared (cosine similarity) with every route's centroid.
4. The most similar route wins when its similarity reaches the threshold; otherwise the request goes to the router's default member. A request with no user text also goes to the default member.

The decision's reason names the route and the similarity (for example Semantic route 'code' (similarity=0.912)), in the response's router_decision and in the router's Stats.

Setting It Up

1. Add an embedding model (third-party or self-hosted) if you do not have one
2. Create a router with the Semantic strategy and add the members
3. Choose the embedding model, the similarity threshold (0.75 by default) and the default member
4. Add routes: a name, the member, and a few example prompts per line

Via the SDK:

router = client.agent_router.create(
    name="smart-router",
    strategy="semantic",
    members=[
        {"model_id": "<gpt-4o-mini id>", "priority": 2},
        {"model_id": "<code model id>", "priority": 1},
    ],
    semantic_config={
        "embedding_model_id": "<embedding model id>",
        "confidence_threshold": 0.75,
        "default_model_id": "<gpt-4o-mini id>",
        "routes": [
            {"name": "code", "model_id": "<code model id>",
             "examples": ["write a function", "debug this code", "fix the bug"]},
            {"name": "simple_qa", "model_id": "<gpt-4o-mini id>",
             "examples": ["what is X?", "define Y", "how does Z work?"]},
        ],
    },
)

Choosing a Threshold

• Higher (0.85 and up): only close matches take a route; more requests go to the default member
• Lower (0.6 to 0.7): routes catch looser matches

Similarity values depend on the embedding model, so check a few decisions in the router's Stats and adjust.

Performance

Each request makes one embedding call for its prompt (repeated prompts reuse the embedding). Route centroids are embedded once per gateway and reused.