Writing

Field notes · October 30, 2026

Token Divergence Across Coding Agents on a Shared Model

A controlled benchmark of ten coding agents using a single local Qwen 3.8 27B model reveals that token generation and cache reuse vary significantly more than task pass rates, highlighting architectural differences in agent design.

Token Divergence Across Coding Agents on a Shared Model
10 coding agents, one Qwen 3.8 27B, same llama-server: tokens and cache reuse differ more than pass rates

A recent empirical evaluation published by a community researcher on the r/LocalLLaMA forum investigated the behavioral divergence of ten distinct AI coding agents when constrained to interact with a single, shared language model server. The experiment was designed as the second part of a broader testing series, following an initial evaluation using a smaller four-billion-parameter model. For this iteration, the author utilized a MacBook Pro equipped with an M5 Max processor and sixty-four gigabytes of memory, running the Qwen 3.8 27B model quantized to four bits via llama.cpp, with extended thinking capabilities disabled. Crucially, every agent was pointed at the exact same model endpoint. The experimental protocol involved five small coding tasks, each containing a deliberate logical trap. To ensure strict isolation, each agent was confined to a sandbox environment where it could only access its own directory and communicate exclusively with the model server. Auto-approval for tool use was enabled, telemetry was disabled, and a hard time limit of five minutes per task was enforced. The source notes that while the final pass rates across the ten agents were relatively comparable, the underlying telemetry—specifically the volume of tokens generated and the patterns of prompt cache reuse—differed substantially from one agent architecture to another.

10 coding agents, one Qwen 3.8 27B, same llama-server: tokens and cache reuse differ more than pass rates

The primary theoretical interest of this result lies not in the absolute performance of any single agent, but in the mechanistic explanation for why identical models yield vastly different token economies depending on the agentic wrapper orchestrating them. When multiple software agents interface with the same large language model, the model itself remains a static function approximator. The variance in token generation and cache utilization is therefore entirely attributable to the agent's system prompt architecture, its internal state management, and its strategy for context window manipulation. Some agents are evidently designed to submit highly verbose, repetitive context blocks with each turn, failing to leverage the key-value (KV) cache optimizations available in modern inference servers like llama.cpp. Others likely employ more sophisticated context-pruning strategies, summarizing previous turns or selectively retrieving only the most relevant code snippets before generating a new prompt. This divergence indicates that the efficiency of an AI coding assistant is heavily determined by its orchestration layer rather than the raw capability of its underlying foundation model.

There are notable limits to the evidence presented in this benchmark that must be acknowledged before extrapolating the findings. First, the sample size of five small coding tasks is insufficient to establish statistical significance regarding general coding proficiency. The inclusion of deliberate traps tests specific failure modes rather than holistic software engineering capability. Second, the hardware and configuration constraints—a locally hosted, four-bit quantized twenty-seven-billion-parameter model with thinking disabled—represent a specific point in the vast configuration space of modern AI deployment. The behaviors observed might shift dramatically if the agents were connected to a larger, unquantized model or if chain-of-thought reasoning were permitted. Third, the five-minute time limit introduces a temporal pressure that may disproportionately penalize agents whose architectures require longer context-gathering phases before executing a solution. Finally, because the specific pass rate data table is truncated in the provided notes, the precise correlation between token inefficiency and task failure cannot be definitively calculated, though the author asserts the variance in tokens exceeded the variance in outcomes.

Despite these limitations, the architectural divergence observed in this benchmark carries significant implications for teaching and learning with AI, particularly within computer science education. As educational institutions increasingly integrate AI coding assistants into their curricula, there is a prevailing tendency to evaluate these tools solely on their ability to produce correct code. This benchmark demonstrates that such an evaluation is incomplete. If two agents achieve the same pass rate but one consumes orders of magnitude more tokens and fails to utilize caching efficiently, they represent fundamentally different pedagogical tools. In a classroom setting where students might run local models on constrained hardware, or where institutional API budgets are finite, the token economy of an agent directly impacts accessibility and sustainability.

Furthermore, the variance in how agents construct and manage context provides a rich subject for instruction in AI literacy. Computer science educators can use these findings to teach students that interacting with an AI model is not merely about asking the right question, but about understanding how the software mediating that question structures information. Students can be taught to analyze the system prompts and context-management strategies of different agents, transforming the AI from a black-box oracle into a transparent software artifact that can be critiqued and optimized. Understanding why one agent repeatedly sends redundant context while another efficiently reuses cached prompts requires knowledge of data structures, network latency, and inference optimization—core computer science concepts that are often abstracted away by commercial AI products.

10 coding agents, one Qwen 3.8 27B, same llama-server: tokens and cache reuse differ more than pass rates

Ultimately, this community-driven benchmark underscores a critical reality for educational technology: the agent framework is as important as the model itself. When designing AI-assisted learning environments, educators and developers must look beyond pass rates and consider the computational mechanics of the tools they deploy. An agent that solves a problem efficiently, managing its context window with precision, models better software engineering practices than one that brute-forces a solution through excessive token generation. By exposing students to these underlying mechanics, educators can foster a deeper, more critical engagement with artificial intelligence, preparing them not just to use these tools, but to understand and build them responsibly.

Source: 10 coding agents, one Qwen 3.8 27B, same llama-server: tokens and cache reuse differ more than pass rates

Read the original