Routing Pipeline

Routed model calls pass through separate stages for request understanding, policy, and model execution. The selected recipe defines those stages; the harness owns the surrounding task loop and tool execution.

Signals: extract facts

A signal gives a name to something detected from the request, conversation, identity, or content. Some signals are deterministic, such as a keyword, metadata predicate, or context-length band. Others use embeddings or classifier models for semantic intent, domain, complexity, PII, or jailbreak detection.

Signals describe facts; they do not choose the backend. The same signal can be reused by several decisions inside one recipe.

See the signal overview for the maintained heuristic and learned families.

Projections: coordinate evidence

Projections turn several signal outputs into a reusable result:

• a partition chooses a coherent winner among overlapping matches;
• a score combines weighted inputs; and
• a mapping converts a score into named routing bands.

For example, domain evidence can form one exclusive domain partition while complexity, context, and verification evidence form a difficulty score. Several decisions can then reference those results without repeating the combination logic.

See Projections.

Decisions: apply policy

A decision combines signals and projection outputs with boolean rules. Its priority and the recipe's strategy determine which matching route wins. The matched decision supplies candidate models, an optional algorithm, and route-local plugins.

Keep hard requirements visible here. Authorization, local-only processing, modality, context capacity, and tool compatibility should make a route eligible or ineligible before an optimizer compares cost or latency.

See Decisions.

Algorithms: select or coordinate models

After a decision matches, its algorithm handles the candidate set.

• Selection algorithms choose one candidate using a fixed order, semantic fit, latency, feedback, or another bounded policy.
• Looper algorithms coordinate several calls through a cascade, panel, multi-round process, or workflow.

When a decision has one candidate, the simplest static behavior is often the right choice. Orchestration should be reserved for tasks where its additional latency and compute have measured value.

See Algorithms.

Plugins: apply route-specific behavior

Plugins add behavior associated with the selected route. Depending on the plugin, that behavior may run before the provider request, during execution, or while processing the response. Examples include request parameter changes, context compression, retrieval, memory, replay, response caching, and response controls.

Detection and enforcement are different. For example, a PII signal reports a match; the decision and plugin policy determine whether to block, reroute, transform, or simply observe it.

See Plugins.

Model pools: execute the request

Providers bind logical model names to physical inference endpoints. A pool can contain local vLLM or Ollama services, Kubernetes-hosted models, or remote OpenAI-compatible providers. Semantic Router chooses the model path; the model server or backend scheduler executes it and owns replica placement.

The decision models used by signals and selection algorithms have a separate model-runtime replica pool. Its worker dispatch does not choose the Chat backend for the request.

Capability and runtime metadata are useful only within policy boundaries. A fast backend is not eligible if it cannot handle the request's modality, context, tools, or locality requirement.

Recipes keep policies isolated

An entrypoint selects one recipe. That recipe owns its signals, projections, decisions, algorithms, plugins, cache namespace, and routing state. Providers, stores, and router-owned model assets can be shared without allowing one recipe's policy to leak into another.

Concrete backend model names are direct pass-through requests. They bypass recipe signals, decisions, route-local plugins, cache, learning, and session routing. If a virtual entrypoint's recipe has no matching decision, the Router uses the configured default provider model.

See Virtual Models for the complete configuration contract.

Example: a coding call with tools

A harness sends a conversation, tool definitions, and a virtual model name. The entrypoint selects a recipe; signals identify the coding request and its conversation facts. Decisions establish the route and candidate set, then capability and context checks narrow that set before the algorithm selects a model. Route plugins can apply tool policy or context compression when configured. The response returns to the harness, which owns any tool execution and the next task step.

For multi-turn routing, configure session identity and protection deliberately. An active tool loop or nonportable provider state can restrict model switches; a candidate excluded by policy cannot be restored merely to preserve continuity. See the agent harness guide for these integration boundaries.

Workload, Router, and pool

A useful routing decision joins three views:

View	Examples
Workload	intent, language, complexity, context, tools, modality, identity
Router policy	eligibility, priority, objective, plugins, algorithm
Model pool	capabilities, locality, health, latency, load, relative cost

The workload says what is needed. Policy says what is allowed and preferred. The pool says what is currently available. Keeping these views separate makes the system easier to change and evaluate.