Field notes · November 9, 2026
Speculative Decoding for Local Mixture-of-Experts Models
RustingStrata achieves 160 tokens per second on a 12 GB RTX 3060 by combining multi-token prediction speculative decoding with a quantized Qwen mixture-of-experts model. This result demonstrates that consumer-grade hardware can support high

A recent development report posted to the r/LocalLLaMA community and documented in the RustingStudio GitHub repository details a significant performance milestone for local artificial intelligence inference. The author states that the RustingStrata inference engine now achieves a decode speed of 160 tokens per second on an NVIDIA RTX 3060 graphics processing unit equipped with 12 gigabytes of video random access memory (VRAM). The model utilized in this benchmark is Qwen3.6-35B-A3B, quantized to the Q2_K_XL format. According to the author, the resulting model file occupies 11.7 gibibytes, meaning it does not entirely fit within the 12-gigabyte VRAM constraint of the GPU. To manage this limitation, the remaining expert parameters are offloaded to the system's central processing unit (CPU) memory. The author further notes that prior to the addition of multi-token prediction (MTP) speculative decoding, RustingStrata already decoded Qwen models approximately 1.7 times faster than llama.cpp on the same hardware configuration. The integration of MTP speculative decoding is credited as the primary mechanism enabling the reported 160 tokens per second throughput.

The theoretical significance of this result lies in the intersection of three distinct computational strategies: mixture-of-experts (MoE) architectures, aggressive low-bit quantization, and speculative decoding via multi-token prediction. An MoE model such as Qwen3.6-35B-A3B possesses a total parameter count of 35 billion, but activates only 3 billion parameters during any single forward pass. This sparse activation is what makes it possible to run a model of this nominal size on consumer hardware, as the computational burden per token is dictated by the active parameters rather than the total parameter count. However, the full set of weights must still reside in accessible memory. By applying Q2_K_XL quantization, the developer compresses the model weights sufficiently to reduce the total footprint to 11.7 gibibytes. Because this marginally exceeds the 12-gigabyte VRAM capacity, the system employs partial CPU offloading, keeping inactive experts in system RAM and transferring them as needed. While CPU offloading typically introduces severe latency penalties due to the bandwidth disparity between PCIe buses and internal GPU memory interconnects, the sparse nature of the MoE architecture limits how frequently these transfers must occur.
The most consequential technical element reported is the application of MTP speculative decoding. Standard autoregressive decoding generates one token at a time, requiring a full memory read of the active weights for each token produced. Speculative decoding alters this paradigm by using a smaller, faster draft mechanism to propose multiple tokens simultaneously, which the larger target model then verifies in a single parallel forward pass. Multi-token prediction extends this concept by training or configuring the model to inherently predict several future tokens at once, thereby increasing the acceptance rate of the speculative drafts. When the draft tokens are accepted, the effective throughput increases dramatically without altering the mathematical output distribution of the base model. Achieving 160 tokens per second under these constrained memory conditions suggests that the verification overhead introduced by the larger model is effectively masked by the high acceptance rate of the MTP drafts, even when some expert weights must be fetched from the CPU.
There are notable limits to the evidence presented in this report. The data originates from a single developer's self-reported benchmark on a specific hardware configuration. There is no independent replication provided, nor is there detailed profiling data regarding the exact acceptance rate of the speculative tokens, the latency of the first token (time-to-first-token), or the thermal throttling behavior of the RTX 3060 under sustained load. Furthermore, the Q2_K_XL quantization level represents an extreme compression of the model's weights. In information-theoretic terms, reducing weight precision to approximately two bits per parameter inevitably degrades the model's representational capacity. While perplexity benchmarks might remain acceptable for certain tasks, the degradation in nuanced reasoning, factual recall, or linguistic subtlety is not quantified in the report. Therefore, the 160 tokens per second figure represents raw generation speed, not necessarily usable educational quality.

For the domain of teaching and learning with artificial intelligence, the implications of this result are substantial if the link between high-speed local inference and pedagogical utility holds. Historically, deploying large language models in educational settings has required reliance on cloud-based application programming interfaces, which introduces recurring financial costs, network latency, and significant student data privacy concerns. Alternatively, running models locally on standard classroom or home computers has been restricted to much smaller, less capable dense models that often lack the reasoning depth required for complex tutoring interactions. The demonstration that a 35-billion-parameter MoE model can generate text at 160 tokens per second on a mid-range consumer GPU fundamentally shifts this calculus. High-throughput local inference enables real-time, interactive AI tutoring systems that operate entirely offline, ensuring that student data never leaves the local machine. Moreover, the sheer speed of 160 tokens per second far exceeds human reading rates, meaning the bottleneck in human-computer interaction shifts entirely from the machine's generation time to the learner's comprehension time. This allows for instantaneous feedback loops, dynamic content generation, and conversational AI agents that respond without perceptible delay, provided the quantization-induced quality loss does not compromise the accuracy required for educational instruction. Future research must evaluate whether the specific artifacts introduced by two-bit quantization in MoE architectures materially affect the reliability of AI-generated educational content.
Source: RustingStrata now hits 160 tok/s on a 12 GB RTX 3060