Writing

Field notes · October 20, 2026

Unified Model Routing via LoRA Switching

A developer extended IBM's Granite Switch architecture to create a single-model gateway that routes chat, embeddings, calibrated decisions, and guardrails through distinct low-rank adapters. This log examines the mechanism of adapter-based

Unified Model Routing via LoRA Switching
I built a gateway that serves Chat, Embeddings, Calibrated Decisions, and Guardrails from a single model

A recent post on the r/LocalLLaMA forum details an independent engineering effort to generalize a model-routing strategy originally published by IBM Research. The author reports building a software gateway named Rubick, which serves four distinct functions—chat, embeddings, calibrated decisions, and guardrails—from a single underlying large language model endpoint. The project was directly inspired by the release of IBM’s Granite Switch architecture, documented in a blog post on Hugging Face. The author explicitly notes that no artificial intelligence tools were used in the writing of the forum post. Rubick is designed to point to a Llama-served endpoint, applying different low-rank adaptation (LoRA) modules depending on the incoming request type. By doing so, the system attempts to replicate the efficiency of IBM’s approach, wherein a single base model can be dynamically reconfigured to perform specialized tasks without requiring multiple separate model deployments.

I built a gateway that serves Chat, Embeddings, Calibrated Decisions, and Guardrails from a single model

The theoretical foundation of this work rests on parameter-efficient fine-tuning, specifically the mechanics of low-rank adaptations. In standard LoRA implementations, the weights of a pre-trained base model are kept frozen, and small, trainable rank-decomposition matrices are injected into the transformer layers. These matrices capture task-specific knowledge while altering only a fraction of the total parameters. IBM’s Granite Switch extends this concept by training multiple distinct LoRA adapters on the same base model—one optimized for conversational generation, another for producing vector embeddings, another for structured decision-making, and another for safety filtering. A routing mechanism then selects the appropriate adapter at inference time based on the nature of the user query or system requirement. The Rubick project attempts to operationalize this exact mechanism using a Llama architecture as the base, demonstrating that the Granite Switch strategy is not strictly bound to IBM’s proprietary models but represents a generalizable architectural pattern.

The mechanism offers significant computational advantages. Traditionally, serving chat, embeddings, and guardrails simultaneously requires hosting multiple distinct models, each consuming dedicated memory and compute resources. By unifying these roles behind a single base model with swappable adapters, the memory footprint is drastically reduced. Only the small adapter weights need to be swapped or loaded dynamically, rather than instantiating entirely new neural networks. For local deployment scenarios—such as running models on institutional hardware without relying on external API providers—this consolidation is highly relevant.

However, the evidence supporting the efficacy of this specific implementation remains limited. The forum post describes the architecture and its inspiration but does not provide empirical benchmarks comparing Rubick’s performance against dedicated, standalone models for each task. It is theoretically established that while LoRA adapters are highly efficient, they may not always match the performance ceiling of a fully fine-tuned model, particularly for tasks as mathematically distinct as generating conversational text versus producing high-fidelity semantic embeddings. Embeddings require mapping text into a continuous vector space where geometric distance reflects semantic similarity, whereas chat requires autoregressive token prediction optimized for coherence and instruction following. Forcing both capabilities through adapters on the same frozen base model introduces potential interference or suboptimal representations if the base model’s latent space is not sufficiently aligned with both objectives. Furthermore, the routing logic itself introduces a point of failure; if the gateway misclassifies a request, the wrong adapter will process it, leading to degraded outputs.

When considering the implications of unified adapter routing for teaching and learning with AI, the architectural shift from multiple models to a single routed model addresses several practical barriers in educational technology. Schools and universities frequently face constraints regarding hardware procurement, server maintenance, and data privacy. Deploying a suite of separate AI tools for tutoring, content moderation, and document retrieval often exceeds local IT capacities. A system like Rubick, built on the Granite Switch paradigm, suggests that an institution could host a single open-weight Llama model locally and serve all necessary educational functions by swapping adapters.

In practice, this means a student-facing application could route a learner’s question through the chat adapter to generate a pedagogical response, while simultaneously passing the interaction through the guardrail adapter to ensure the output complies with institutional safety policies. The calibrated decision adapter could be employed by instructors to evaluate student submissions against specific rubrics, providing structured scoring rather than generative text. Because all these processes occur on the same base model hosted locally, student data does not need to traverse external networks, addressing stringent privacy requirements common in K-12 and higher education contexts.

I built a gateway that serves Chat, Embeddings, Calibrated Decisions, and Guardrails from a single model

Additionally, the separation of concerns via adapters allows educational technologists to update or refine one capability without disrupting others. If an institution identifies that its safety filters are overly restrictive for advanced research students, it can swap or fine-tune only the guardrail adapter without retraining the entire conversational model. This modularity supports iterative improvement in educational AI deployments. However, educators and developers must remain cautious about the limits of the current evidence. Without rigorous testing in classroom settings, it is unclear whether the latency introduced by dynamic adapter switching might disrupt real-time interactive learning, or whether the embedding adapter produces vectors precise enough for reliable retrieval-augmented generation in academic databases. The architectural promise of unified routing is substantial, but its translation into effective educational infrastructure requires empirical validation beyond initial engineering proofs of concept.

Source: I built a gateway that serves Chat, Embeddings, Calibrated Decisions, and Guardrails from a single model

Read the original