Field notes · November 8, 2026
Edge Inference Speed for Nemotron Bolt on Jetson
A community report documents running NVIDIA's Nemotron 3.5 Bolt E4B model at 27.7 tokens per second on a Jetson Orin Nano Super, demonstrating near 1.8x speed improvements over baseline implementations.

A recent post on the r/LocalLLaMA forum, authored by representatives of embedl.com, reports that NVIDIA’s Nemotron 3.5 Bolt E4B model has been executed at a throughput of 27.7 tokens per second on a Jetson Orin Nano Super device. The authors state that this performance represents an approximate 1.8-fold increase over the official release benchmark of 15 tokens per second achieved using Llama CPP. The post further notes that the model is capable of generating code to program games while running entirely on the edge hardware. The authors provide a link to their website where the model is hosted on their own Jetson devices, allowing users to interact with the system directly. No peer-reviewed methodology or detailed architectural modifications are provided in the post; the claims rest on the empirical demonstration linked via the embedl.com platform.

The theoretical significance of this result lies in the intersection of model quantization, hardware-specific inference optimization, and the viability of local artificial intelligence deployment in educational contexts. The Nemotron 3.5 Bolt E4B designation indicates the use of a specific quantization format—likely a 4-bit block floating-point representation—which drastically reduces the memory footprint and computational overhead required to run the model. Large language models typically demand substantial graphical processing unit (GPU) memory and bandwidth. By compressing the model weights into a lower-precision format, the arithmetic intensity shifts from being memory-bandwidth-bound to compute-bound, which aligns more favorably with the architecture of edge accelerators like the Jetson Orin Nano.
The reported 1.8x speedup over the Llama CPP baseline suggests that the optimization extends beyond simple quantization. General-purpose inference engines like Llama CPP are designed for broad hardware compatibility across various central processing units (CPUs) and GPUs. However, they may not fully exploit the specialized tensor cores or unified memory architecture of a system-on-module like the Jetson Orin Nano Super. Achieving 27.7 tokens per second likely involves custom kernel optimizations, precise memory management strategies tailored to the device's shared memory pool, or the utilization of NVIDIA’s TensorRT-LLM framework, which compiles models specifically for NVIDIA silicon. This mechanism highlights a fundamental principle in applied machine learning: raw model capability is only one variable in deployment; the software-hardware co-design dictates actual user experience.
There are notable limits to the evidence presented in the source material. First, the throughput metric of 27.7 tokens per second is reported without specifying the context length, prompt size, or batch size used during the benchmark. Inference speed in transformer architectures is highly sensitive to sequence length; generation speed typically degrades as the context window fills due to the quadratic complexity of the attention mechanism in standard implementations, though linear attention variants mitigate this. Second, the qualitative claim that the model "can program games" lacks rigorous evaluation metrics regarding code correctness, compilation success rates, or logical coherence. A high token generation rate does not inherently guarantee output quality, particularly when operating in heavily quantized precision formats where numerical degradation can subtly impair reasoning capabilities. Third, the comparison is made against a specific baseline (Llama CPP), which, while popular, may not represent the absolute optimized ceiling for that hardware prior to the authors' intervention.
Despite these evidentiary constraints, the implications for teaching and learning with artificial intelligence are substantial if the technical claims hold under broader scrutiny. Historically, deploying interactive AI agents in classroom settings has relied on cloud-based application programming interfaces (APIs). This dependency introduces several friction points for educators: recurring financial costs, latency issues dependent on network infrastructure, data privacy concerns regarding student interactions, and the requirement for persistent internet connectivity. The demonstration of a capable coding model running at nearly 28 tokens per second on a low-cost edge device directly challenges these constraints.
In computer science education, for instance, the ability to run a local model that assists students in programming games transforms the pedagogical dynamic. Students can receive immediate, iterative feedback on their code without transmitting proprietary or personal data to external servers. The latency reduction inherent in edge computing—where inference occurs millimeters from the user rather than across a network—creates a more fluid interaction loop. When a student queries an AI assistant for debugging help, the response time must be sufficiently rapid to maintain engagement and flow. A throughput of 27.7 tokens per second means a typical short response is generated in under two seconds, crossing the threshold from noticeable delay to conversational immediacy.

Furthermore, this result underscores the importance of teaching students and educators about the infrastructure underlying AI tools. Understanding that a model's utility is governed not just by its training data but by its quantization level and the hardware executing it fosters deeper computational literacy. If schools can deploy affordable Jetson-class devices equipped with optimized local models, the barrier to entry for AI-assisted learning drops significantly. This shifts the paradigm from renting intelligence via cloud subscriptions to owning and operating localized, privacy-preserving educational tools. The challenge remains in developing accessible deployment pipelines so that educators without deep systems engineering expertise can leverage these optimized edge configurations effectively.
Source: Nemotron 3.5 Bolt E4B on Jetson Orin Nano Super at 27.7 tok/s!