8 posts tagged with "semantic-router"

Open source grows when people can see where their work belongs and who they can build with. vLLM Semantic Router now has seven Workgroups, each responsible for one durable technical direction.

If you are new to the project, start here: vLLM Semantic Router sits between an AI application and its model or agent backends. It understands the request, chooses how it should be handled, executes that route, and measures the result. The Workgroups divide that system into clear places to contribute.

One System, Seven Clear Owners​

A routed request crosses several responsibilities:

1. Developer Experience & Ecosystem provides the CLI, Dashboard, APIs, recipes, and learning path.
2. Enterprise & Environment applies access, usage, lifecycle, and deployment policy.
3. Router Models & Inference Runtime produces the signals used to route.
4. MoM & Routing chooses models and multi-model strategies.
5. Agentic & Context manages context and chooses or composes agents.
6. Data Plane & Networking executes the chosen path.
7. Evaluation & Quality measures results and catches regressions.

They form one system, separated by responsibility rather than isolated code ownership. Every Epic below has one owning Workgroup. Dependencies on other groups are recorded as shared interfaces in the linked charter.

Choose the direction that matches the problem you want to solve. GitHub labels on each linked Epic are the source of truth for acceptance and delivery status.

MoM & Routing​

Mission: Make a pool of models behave like one measurable and improving Mixture-of-Models.

The problem​

A user should be able to call one stable model name without choosing a backend for every request. A Mixture-of-Models (MoM) needs a qualified model pool and a versioned recipe that can improve without making behavior unpredictable.

What this Workgroup owns​

• Model pools, model roles, portable recipes, and their versioned lifecycle.
• Model selection and collaboration through fallback, cascade, judging, synthesis, and bounded workflows.
• Offline-to-online improvement of recipes and pool members against explicit quality, cost, latency, safety, domain, or modality objectives.
• Modality-aware pools, approved reasoning reuse, and safe reuse of compatible computation across models.

What it does not own​

This Workgroup does not train the lightweight models that produce routing signals, build the live network path, decide how conversation history is compressed, or operate a hosted service.

Owned Epics​

• Define portable MoM model pools, routing recipes, evaluation, and model cards
• Optimize routing recipes through an offline-to-online lifecycle
• Optimize model-pool members against Mixture-of-Models objectives
• Advance bounded multi-model collaboration algorithms
• Enable safe cross-model KV-cache reuse
• Enable vLLM-Omni backends and modality-aware MoM pools
• Reuse validated reasoning experiences for small-model inference

Router Models & Inference Runtime​

Mission: Build better Router Models and one extensible runtime for executing them across the ecosystem.

The problem​

Routing depends on signals such as intent, complexity, safety, preference, and expected quality. The models that produce those signals must improve over time, and new models should not spread engine-specific code throughout the Router.

What this Workgroup owns​

• Improve, calibrate, and release the Router Models built into the project.
• Develop routing-native model families beyond BERT-only designs.
• Build reproducible self-improvement, distillation, and fine-tuning pipelines.
• Provide one versioned execution contract across supported engines and hardware, with clear activation, diagnostics, and rollback.

What it does not own​

This Workgroup produces routing intelligence; it does not choose the end-user MoM pool, own generic gateway forwarding, manage users and quotas, or rebuild the tensor engines and GPU schedulers it integrates with.

Owned Epics​

• Build an extensible inference runtime for Router Models
• Improve current Router Models and develop routing-native model families
• Build a reproducible SLM self-improvement and Router Model fine-tuning pipeline
• Harden multimodal and image-routing signal robustness

Data Plane & Networking​

Mission: Execute every live routing decision through a fast, reliable, and portable request path.

The problem​

A routing decision has little value if the request path is slow, fragile, or different in every deployment. Standalone serving and gateway integration need the same behavior and failure semantics.

What this Workgroup owns​

• Standalone OpenAI-compatible serving and Envoy or gateway integrations.
• Request, response, streaming, dispatch, retry, fallback, error, and telemetry behavior.
• Engine-neutral backend connectivity and inference-aware endpoint selection.
• Safe semantic caching, performance optimization, and failure recovery.

What it does not own​

This Workgroup executes access and routing policy but does not define who may access a model, which quota applies, which hardware is officially supported, how a Router Model is trained, or which MoM recipe is best.

Owned Epics​

• Support standalone and gateway-integrated data plane modes
• Connect semantic routing to inference-aware backend selection
• Make semantic caching safe, measurable, and lifecycle-aware
• Optimize data-plane performance, streaming, and failure recovery

Enterprise & Environment​

Mission: Make vLLM Semantic Router production-grade across supported organizations, environments, and hardware.

The problem​

Production users need clear answers to practical questions: Who can call each model? How much can they use? What changed? Is the system healthy? Can a model, recipe, or Router upgrade be rolled out and reversed safely? Which deployment path is maintained, and which components does it own? The answers must remain consistent across deployment environments.

What this Workgroup owns​

• Users, organizations, serving identities, model access, API keys, quotas, and usage accounting.
• Tenant isolation, audit, reliability, scalability, monitoring, and diagnostics.
• Model, recipe, configuration, and vLLM-SR activation, rollout, and rollback.
• Stable deployment and lifecycle APIs, maintained reference stacks, and a tested support matrix across deployment environments and hardware.

What it does not own​

This Workgroup does not promise a public hosted-service SLA, expose private infrastructure or credentials, define model quality, own evaluation standards, or implement networking protocols. It supplies reusable open-source production capabilities rather than publishing private product plans.

Owned Epics​

• Build multi-tenant inference access, quotas, and usage controls
• Build versioned configuration activation and rollback
• Define deployment architecture and reference stacks across environments and hardware
• Establish production observability and supported-environment qualification

Agentic & Context​

Mission: Manage context and safely select, hand off, and compose agent backends for long-running workloads.

The problem​

Long-running work accumulates messages, memory, tool output, cost, and risk. Important instructions can be lost, while the best model or agent may change as the task evolves. The Router must handle those changes without becoming a general-purpose agent framework.

What this Workgroup owns​

• Context compression, pruning, memory selection, prompt restructuring, and protection of critical instructions.
• Session budgets, state boundaries, retention, recovery, and graceful degradation.
• Agent selection, fallback, handoff, and bounded multi-agent composition.
• Safe model or workflow switching as a session evolves, with measurable task, cost, latency, and safety outcomes.

What it does not own​

This Workgroup does not build an unrestricted agent orchestrator, own all MoM selection algorithms, transport KV caches between models, automate CLI installation, or allow silent lossy transformation and unbounded online training.

Owned Epics​

• Optimize context for long-session and agentic workloads
• Enable agent-based routing and multi-agent composition
• Develop safe model and workflow switching for long-running agents

Developer Experience & Ecosystem​

Mission: Make vLLM Semantic Router easy to discover, install, configure, extend, and operate.

The problem​

Technical depth has little impact when a new user cannot reach a first request or understand what happened. The project also needs clear extension paths so contributors, model builders, infrastructure projects, and educators can build on it without reverse-engineering the repository.

What this Workgroup owns​

• Installation, configuration, deployment, tuning, diagnosis, and operation through the CLI, Dashboard, and APIs.
• An agent-facing skill for deployment, recipe generation, evaluation, tuning, and reviewed operations.
• Documentation, reference recipes, tutorials, model cards, blogs, videos, and use-case sharing.
• Clear extension and contribution paths for models, runtimes, gateways, and deployment systems.

What it does not own​

This Workgroup does not control repository permissions or promotion, run marketing events or AMD-internal programs, or redefine algorithms, production policy, and quality standards owned by other Workgroups.

Owned Epics​

• Build agent-assisted vLLM-SR deployment, recipe, evaluation, and tuning workflows
• Reduce time to the first successful routed request
• Publish maintained reference recipes and end-to-end tutorials

Evaluation & Quality​

Mission: Make every supported capability measurable and every change verifiable.

The problem​

Claims about a Router Model, MoM recipe, agent-selection policy, runtime optimization, or deployment are difficult to trust when each uses a different dataset and reporting method. The project needs shared evaluation contracts and regression gates. Each technical Workgroup remains accountable for what it builds; this Workgroup makes the results comparable.

What this Workgroup owns​

• Common benchmark, data provenance, metric, comparison, reproducibility, and publication contracts.
• First-class evaluation of each MoM against standalone models, with a common core and objective-specific extensions.
• Shared evaluation for routing, agents, context, serving, platform, and developer workflows.
• CI, E2E, compatibility, security, performance, and operational regression gates.

What it does not own​

This Workgroup does not choose another direction's quality target, accept an issue, make the final release decision, replace Maintainer review, or own the model research itself. It defines shared measurement and gates.

Owned Epics​

• Build reproducible decision-level routing evaluation
• Evaluate each Mixture-of-Models as a first-class model
• Establish cross-platform performance regression coverage
• Expand end-to-end quality coverage across Router, Dashboard, and deployments

How Workgroups Operate​

A Workgroup is a technical home, not a permission level.

Each Workgroup owns one durable technical direction and its bounded Epics. It connects contributors, maintains the charter, and helps prepare work for acceptance. The Open Source Team retains final acceptance, merge, role, and release authority.

Lead. Every active Workgroup has at least one Lead and may have several. A Lead is a Committer or Maintainer, or a Contributor with a merged commit and a Committer or Maintainer Sponsor. Leads maintain the charter and Epic map and coordinate triage.

Member. A Member has at least one merged repository commit and continues to build in the Direction. Anyone may collaborate before meeting the roster requirement, and people may join more than one Workgroup.

To join, open the charter and comment with your requested role, a merged contribution, your focus, availability, and Sponsor when required. Confirmed Leads and Members appear on the public Workgroups map; these roles provide visibility and responsibility, not additional repository permissions.

Proposing a New Workgroup​

Create a Workgroup only for a durable technical problem that spans releases, contains several Epics, and cannot fit an existing charter. A feature, algorithm, integration, event, or milestone belongs inside an existing group.

A proposal must define the problem, scope, non-scope, shared interfaces, initial Epic map, why existing groups cannot own it, and at least one eligible Lead. After Maintainer acceptance, it receives a charter, owner label, public roster, and Lead or Member self-nomination.

Come Build With Us​

Start with the Workgroups map, choose a charter, and introduce yourself. Join an Epic, shape a contributor-ready issue, or propose a focused piece of work.

Find your focus and passion. Build with the community. Grow together.

Semantic Router can now verify grounded responses with a generative span detector served by vLLM. The new endpoint detector backend runs LettuceDetect v2 against every fact-checkable answer: unsupported spans are located to the character, typed against a hallucination taxonomy, and explained — in one call, before the response reaches the user.

The models come out of a joint paper between KR Labs and the Semantic Router team, Beyond Document Grounding: Span-Level Hallucination Detection over Code, Tool Output, and Documents (arXiv:2607.00895). This post walks through the paper — the benchmark, the taxonomy, the models, and what they score — and then through the integration that puts the detector into the serving stack.

Most AI applications are built around a single model endpoint. But as models, devices, and deployment constraints diversify, no single model is the best fit for every request or environment. The practical question is how multiple specialized models can be coordinated, evaluated, and served through one interface. We call this systems approach Mixture-of-Models.

In less than a year since its public launch, vLLM Semantic Router has reached 5,000 stars, 150+ contributors, and more than 300,000 cumulative downloads across our Hugging Face model family. Across three major releases—Iris, Athena, and Themis—the system boundary moved from choosing a model, to governing multi-model inference, to preserving state and coordination across sessions. Those releases built the foundation for the MoM architecture envisioned from day 0.

This post describes the next step for vLLM Semantic Router: moving from routing among models to building dependable model systems from them. Under one versioned contract, independent models, policies, preferences, and execution paths become a system that can be trained, evaluated, exported, imported, deployed, and invoked through one interface. Our goal is to make vLLM Semantic Router a training, evaluation, and inference engine for Mixture-of-Models.

Figure 1: A Mixture-of-Models turns a heterogeneous model portfolio into one model experience.

How vLLM-SR Got Here​

The first vLLM Semantic Router post asked a practical question: why give simple and difficult requests the same reasoning budget? A lightweight classifier used fixed domain labels to choose between fast and reasoning paths, helping vLLM spend inference compute more selectively.

Production traffic quickly exposed the limit of that design. Domain alone could not represent privacy, safety, context, language, modality, tools, preferences, latency, and authorization. A static label also could not account for an endpoint that was cheap but overloaded, capable but remote, or unsafe to switch into midway through an agent session.

We rebuilt the classifier layer around modular model support, shared LoRA computation, Rust/Candle inference, and Go integration. We then replaced fixed classification with a Signal–Decision architecture that separated observed evidence from policy and execution. This became the spine of the next three releases.

Figure 2: Each stage changed the unit of control: model, decision, system, session, and finally the complete model lifecycle.

Iris made routing composable. Domain, keyword, embedding, factuality, feedback, and preference signals fed explicit decisions, while safety, PII protection, caching, hallucination detection, and tool selection became route-scoped behavior. Iris also introduced the MoM model family and described vLLM-SR as “System Level Intelligence for Mixture-of-Models.”

Athena added first-class model selection, memory and RAG, a multilingual and multimodal model stack, ROCm acceleration, and an operating dashboard. The project was becoming the control system around multi-model inference, not just a classifier in front of vLLM.

Themis turned that broader system into an operable contract:

Signals become projections. Projections feed decisions. Decisions choose algorithms. Algorithms select models.

Themis added session-aware agentic routing, replayable traces, stronger protocol support, an operator console, and runtime paths across AMD ROCm, NVIDIA CUDA, Intel OpenVINO, and CPU environments. It also made a route explainable: operators can see the evidence, policy, algorithm, and physical model behind each decision.

From Signal–Decision to Workload–Router–Pool​

The releases built the runtime. Two project papers explained the architecture behind it.

The white paper, Signal Driven Decision Routing for Mixture-of-Modality Models, formalized the separation between neural evidence and symbolic policy. Fast heuristics and learned classifiers turn prompts, context, identity, safety, and modality into a structured signal vector; a Boolean engine then composes those signals into auditable policy. A typed neural-symbolic DSL parses and validates that policy before compiling it into deployable configuration. When the paper was published, the system covered thirteen signal types and thirteen model-selection algorithms, with per-decision plugins for caching, RAG, memory, safety, provider handling, and response validation.

The vision paper, The Workload–Router–Pool Architecture for LLM Inference Optimization, widened the frame. It argues that three variables have to be designed together:

• Workload: chat or agent, single-turn or multi-turn, warm or cold, prefill-heavy or decode-heavy
• Router: static semantic policy, online feedback or bandit adaptation, RL-based selection, and quality-aware cascades
• Pool: homogeneous or heterogeneous accelerators, prefill/decode topology, model placement, and KV-cache management

Those variables cannot be optimized independently. Workload shape changes which routing policy works; routing policy changes the required pool size and topology; pool state changes which route is efficient. Safety and privacy cut across all three dimensions, while cost, quality, latency, and energy define the optimization frontier. The paper maps the project's research into a 3 × 3 WRP matrix and identifies twenty-one open directions where those dimensions still need to meet.

Figure 3: The white paper defines the programmable routing engine; the vision paper connects it to workload and physical pool design.

Together, the papers made routing programmable and tied it to workload and hardware—the two foundations MoM brings under one model contract.

Meanwhile, the runtime was already moving beyond single-model selection. Fusion, ReMoM, Confidence, Ratings, and bounded Workflows let one request invoke a controlled collaboration among models. As the Micro-Agent work showed, a client can call one model name while the serving layer selects a recipe, fans out to workers, verifies or synthesizes their results, and returns one ordinary response.

Routing remains fundamental. It is how a Mixture-of-Models allocates work, applies policy, and coordinates its parts. But routing is the mechanism. The model system is the product.

Why the Model Boundary Has to Move​

Today's AI stack is fragmented along four axes:

• Models are fragmented. Closed frontier models, open general models, domain experts, compact local models, verifiers, and multimodal models will coexist. None wins simultaneously on quality, cost, latency, trust, privacy, and domain fit.

• Compute is fragmented. GPUs, CPUs, specialized accelerators, edge devices, cloud capacity, and private clusters differ in memory, kernels, availability, price, and energy use. Model choice and placement are becoming the same decision.

• Location is fragmented. Inference spans cloud, data center, and edge. Privacy or residency may rule out a stronger remote model, while a local workload may still need an on-demand cloud expert.

• Preference is fragmented. There is no universal “best.” Products and users make different tradeoffs among accuracy, latency, price, privacy, safety, style, and multimodality. Those choices should shape execution directly.

Today, each application has to reconcile these fragments on its own.

Figure 4: Before MoM, fragmented intelligence becomes application-side routing glue.

Mixture-of-Models moves that responsibility behind one model boundary.

At that boundary, intelligent allocation becomes part of the model. The engine determines which models are eligible, where execution can run, whether models should collaborate, and how to satisfy hard constraints.

Energy makes allocation inseparable from efficiency. Hardware and inference engines improve the supply side by producing more tokens per watt per dollar. The allocation layer controls demand: which work deserves those tokens, and which model or collaboration can provide them within the required quality, latency, and energy budget.

The application selects one versioned model identity and receives one attributable response. Its physical realization can still span open and closed models, cloud and edge, and different accelerator generations. The fragmentation remains, but it becomes internal to the model system instead of leaking into every application.

Figure 5: With MoM, the same fragmented resources become the internal realization of one model.

What We Mean by Mixture-of-Models​

A Mixture-of-Models is a versioned composite model whose engine realizes each request through a preference-conditioned, resource-bounded path across independent models and operators. It is presented to the user through one model interface and returns one attributable result.

A multi-upstream gateway can forward traffic without owning system quality. An MoM owns an objective, an evaluation contract, a reproducible composition, and the runtime that executes it.

MoM also differs from Mixture-of-Experts. MoE routes tokens among internal experts during one forward pass; MoM coordinates independent models that may differ in architecture, owner, license, modality, protocol, context window, and hardware. An MoE checkpoint can itself be one MoM component.

Figure 6: Selection is one MoM topology. Cascades, parallel fusion, and bounded workflows share the same model boundary.

A portable MoM therefore needs more than weights and configuration: it needs a component manifest, capability metadata, routing and collaboration recipes, policies, preferences, evaluation suites, runtime constraints, provenance, and version history.

Open checkpoints can travel with the artifact; closed models remain authenticated external references with explicit capability and policy contracts. Exporting an MoM does not make a proprietary checkpoint portable. It makes the model system reproducible.

Turn Preferences into Models​

Preferences become concrete when they are published as model identities. One MoM family can offer several operating points:

The built-in identities currently shipped by the CLI are listed below. Run vllm-sr model list for the authoritative catalog state.

Each name is a versioned model contract, not a router preset. The application chooses the behavior it needs; vLLM-SR selects and coordinates the models that deliver it while preserving hard privacy, residency, authorization, and safety constraints.

To an application, the full system remains an ordinary model call:

{
  "model": "vllm-sr/mom-v1-ultra",
  "messages": [
    {"role": "user", "content": "Review this design and identify its weakest assumption."}
  ]
}

That identity may select one model, escalate through a cascade, compare parallel answers, require grounding, or run a bounded workflow—without changing the external interface, version, or response contract.

Figure 7: Preferences are published as bounded, versioned model contracts—not hidden application-side routing presets.

Four planes separate ownership:

Figure 8: A complete MoM spans four planes: artifact, learning, execution, and physical realization.

A deployment must map logical requirements onto the models and machines available in its environment. The proposal uses four objects:

1. The bundle fixes the interface, graph, policies, behavior variant, bounds, and immutable semantic assets.
2. The binding maps logical components to eligible deployments without changing the model's decision semantics.
3. The resolution lock freezes the constituent revisions, runtimes, images, accelerators, and provider observations.
4. The run record attributes every decision, call, constraint check, cost, and outcome to the bundle, binding, and lock that produced it.

Figure 9: One stable model identity, from portable contract to attributable run.

This separation keeps portability honest. The same mom-v1-ultra can bind to ROCm, CUDA, a private CPU or NPU node, or a hybrid deployment without promising identical outputs from opaque providers. Instead, it preserves control semantics, exposes substitutions, and gives serving and evaluation the same resolved system.

vLLM-SR as the MoM Engine​

Training, evaluation, and inference must share one contract; otherwise research, benchmarks, and production drift into different systems.

Training allocation, not only weights​

MoM training covers router-owned embeddings, signal encoders, preference and safety models, and selectors. It also learns allocation and collaboration: which path fits a workload and budget, when a cascade should stop, how a panel should judge or synthesize, and when an agent session should switch models. Because constituents may be independent or closed, progress does not require gradients through all of them; policies, thresholds, pools, prompts, contracts, and topology can be optimized from traces and outcomes.

The target is a frontier across quality, latency, cost, safety, privacy, reliability, locality, and energy. Replay and outcomes feed production experience back into offline training without letting the hot path silently rewrite policy.

Evaluating the MoM as one model​

Evaluation must score the model identity end to end; backend benchmarks are inputs, not the result. A versioned scorecard should measure routing regret, collaboration gain, recovery, session continuity, tail latency, cost, safety, privacy, and energy. It should stress provider failures, device loss, model disagreement, workload drift, and preference changes. Each declared operating point also needs its own test: blend for balanced trade-offs and recovery, lite against its quality floor, flash on its latency–quality frontier, ultra within its budget, and vault for privacy and locality enforcement.

The scientific test is stricter than asking whether more calls improve a benchmark. Under matched active compute, can a conditional system exploit complementary strengths and failure modes better than the best fixed model? Without that control, MoM can hide brute-force scaling behind a clever graph. Evaluations must report calls, tokens, cost, latency, and energy alongside quality—and publish when composition does not help.

Figure 10: Composition gain is meaningful only under matched active compute, with quality reported alongside calls, tokens, cost, latency, and energy.

Executing intelligence at inference time​

At inference time, the engine decides whether one model is enough. It may choose a local specialist, preserve a warm session, escalate through a confidence cascade, require retrieval or verification, run a Fusion panel, or execute a bounded workflow. The runtime owns the budget, topology, fallback, trace, and response contract; the application makes a normal model call.

Figure 11: MoM is a closed lifecycle: train the allocation policy, evaluate the full system, execute it, and turn outcomes into the next validated version.

One Model That Can Move​

Our target is a complete MoM that can be built, exported, imported, versioned, evaluated, deployed, and invoked as a unified model. A logical specification compiles into an immutable bundle, binds to an environment, resolves the concrete deployment, and retains the same identity for serving and evaluation.

The artifact should run across developer machines, private clusters, cloud fleets, and edge environments while its physical realization changes. A specialist may resolve to an admissible local checkpoint or managed endpoint; an accelerator runtime may be replaced. If privacy makes a remote expert unavailable, the engine follows a declared fallback or abstention path. A binding cannot silently rewrite the graph, relax a guard, or turn a panel into a cascade—those changes require a new model version.

“Run on any hardware” is an architectural requirement, not a claim that every component is portable today. The project already supports paths across ROCm, CUDA, OpenVINO, and CPU. Next, hardware capability and placement become part of the MoM contract, allowing the engine to map the model system onto what is available.

The standard for the user experience is simple:

One model identity. Many models. Any hardware.

Figure 12: One logical model identity can be realized across developer, data-center, cloud, and edge hardware.

If the application needs to know which provider owns every submodel, which device runs it, or which fallback graph to execute, the abstraction has leaked.

What Changes Now​

The next stage focuses on four connected areas:

1. Define a portable MoM specification. Package components, objectives, policy, preferences, evaluation, constraints, and execution semantics as one versioned artifact.
2. Close the training–evaluation–inference loop. Improve models and recipes from evaluation and replay, then ship them through reviewable, rollback-safe releases.
3. Build a heterogeneous runtime. Map one MoM across cloud, data center, and edge using hardware, locality, energy, and data boundaries as inputs.
4. Keep the model interface boring. Make an MoM as easy to import, deploy, and invoke as a single model.

Figure 13: Four connected workstreams turn Mixture-of-Models from an execution pattern into the next model architecture.

This is a research program for how independent models should specialize, compete, verify, and collaborate; how to measure the resulting system; and how one model contract can survive across devices and environments. Our mission is:

Advancing the science of intelligence across models, devices, and environments.

We will study when composition produces capabilities beyond a single checkpoint, treat placement and energy as part of intelligence, and carry the same model contract from edge to cloud and from research to production.

Build It With Us​

Building Mixture-of-Models requires more than routing. The work spans model training, evaluation, serving systems, hardware, and production operations.

Iris, Athena, and Themis improved because contributors brought real workloads, added backends, trained models, published benchmarks, found failure cases, and argued for better interfaces. MoM needs the same range of work: learned allocation, preference optimization, model cooperation, energy-aware inference, portable artifacts, open evaluation, and heterogeneous runtimes.

If you work on these problems, we want to learn from your workloads and measurements. Build an operating point, add a runtime, test a collaboration recipe, or publish a case where composition fails. MoM will be stronger if its assumptions are tested in the open.

Acknowledgments​

vLLM-SR has grown through work across engineering, research, and the wider ecosystem. We thank Xunzhuo Liu, Huamin Chen, Bowei He, Yankai Chen, Fuyuan Lyu, and Steve Liu for helping shape its technical and research direction. We also thank Andy Luo and Haichen Zhang for their work on ROCm enablement, router-model training, and open MoM experimentation.

The work has also been carried by FAUST, David Shrader, Yang Wu, Ramakrishnan Sathyavageeswaran, Kuntai Wu, Aayush Saini, siloteemu, Chen Wang, Yue Zhu, Senan Zedan, Yossi Ovadia, Samzong Lu, Liav Weiss, Asaad Balum, Yehudit, Noa Limoy, Marina Koushnir, Jared Wen, Abdallah Samara, Hen Schwartz, Srinivas A, Yang Zhu, Jintao Zhang, yuluo-yx, cryo, Bishen Yu, Zhijie Wang, Hao Wu, and Qiping Pan. Their code, reviews, testing, documentation, and stewardship carried the project from one release to the next.

At this milestone, the project stands at 1,734 commits and 150+ contributors. We thank collaborators at MBZUAI, McGill University, Mila, and Rice University, and the broader vLLM, AMD, Intel, Meta, Red Hat, Microsoft, Google, IBM, NVIDIA, Hugging Face, NASA, Nutanix, DaoCloud, and open-source communities. This milestone belongs to everyone who helped turn an early router into a real system.

Figure 14: Building the MoM engine is an open systems problem that needs the full model and infrastructure community.

Join us on GitHub, explore the documentation, try the MoM model family, and meet the community in the #semantic-router channel on vLLM Slack.

vLLM Semantic Router began by helping infrastructure choose the right model for each request.

Now we are extending that foundation beyond a single model: toward systems that can coordinate, evaluate, and operate multiple models across devices and environments.

We invite the community to help build and test that approach in the open.

The Feature Everyone Wants and Almost Nobody Has​

Cursor's Auto mode is deceptively simple: the developer types, and the IDE chooses whether a prompt deserves a frontier model or something faster and cheaper. It is easy to stop noticing — until moving to an open tool where every request starts with a model dropdown.

Agent systems that span multiple models — a local endpoint for coding, a frontier cloud model for deep reasoning, and a fast general-purpose model for everyday tasks — all face the same routing question: how should each request be directed to the right backend?

Many deployments start with a lightweight Python proxy or keyword matcher in front of the gateway. That approach works at small scale, but misroutes grow quickly as traffic, languages, and task types diversify. This post shows how vLLM Semantic Router running as an Envoy ExtProc sidecar inside AgentGateway replaces that pattern with semantic, config-driven routing.

Most agent systems start with a simple idea: call model: auto and let the inference layer pick the right model. That is useful, but it is not enough for long-running agents.

A coding agent can begin with architecture work, call tools, receive short tool outputs, continue with "fix that", then ask a privacy-sensitive question in the same user session. The latest message may look simple, but the route cannot be chosen from the latest message alone. The router also has to know whether this is a safe moment to switch models.

This guide shows how to deploy that pattern on AMD ROCm with vLLM Semantic Router. You will start one ROCm vLLM backend, serve the agentic routing recipe, open the dashboard, validate the OpenAI-compatible API, and use Inferoa to experience route decisions and Router Learning behavior from an agent client.

Agentic routing is not only choosing a model. It is choosing when to keep one.

Running vLLM Semantic Router on AMD Developer Cloud is not just about bringing up one more inference endpoint. It is about turning it into a routed multi-tier system that can classify requests, choose a semantic lane, and make replay and Insights immediately useful.

This post walks through the practical path: start the ROCm backend on an AMD Developer Cloud instance, install vLLM-SR, import the reference profile, and validate the deployment end to end.

v0.3, codename Themis, is our production-readiness release for Semantic Router. The theme is simple: Stability at Scale. After Athena expanded the system brain, Themis is the release where we make that intelligence dependable across real environments, clearer to operate, and safer to ship into production.

This roadmap is not just about adding more capability. It is about making the full system coherent: one stable contract across Docker and Kubernetes, one cleaner deployment path, one real version story for images and packages, stronger performance validation on both NVIDIA and AMD, and a research track that directly improves the product instead of sitting outside it.