Breaking the Single-Token Bottleneck with Speculative Decoding on Modalix MLSoC
The semiconductor industry has spent years conditioning engineers to evaluate AI silicon based on peak TOPS. But in Physical AI, where on-device intelligence must operate within strict power and memory constraints, peak TOPS is a dangerously misleading measure of real-world performance.
When deploying Large Language Models (LLMs) in autonomous robots, automotive assistants, or industrial systems, increasing compute doesn’t necessarily produce faster responses. Traditional LLM decoding is often constrained by memory bandwidth rather than arithmetic throughput. Even powerful accelerators can spend much of their time waiting for model weights to move through memory.
The bottleneck isn’t always a lack of compute. It’s how efficiently you use it.
The Autoregressive Trap
Standard autoregressive decoding operates under a rigid constraint: one forward pass through the target model generates one new token. Producing a complete response requires repeating this process, token after token, with each pass accessing model weights and moving data through the memory subsystem.
At a batch size of one, this process is typically memory-bound. Adding more arithmetic units does little to improve performance when execution engines are already waiting on data movement. More compute won’t move bits faster because it does not eliminate the fundamental inefficiency of generating tokens sequentially.
For Physical AI applications, the consequences extend beyond disappointing tokens-per-second (TPS) benchmarks. An automotive assistant may process a driver’s request quickly, yet still spend precious time generating the structured command needed to act on it. The time required to produce a complete response matters just as much as the rate at which individual tokens are generated.
Why Wait for Tokens?
Speculative decoding attacks the bottleneck differently. Instead of forcing the full target model to generate every token individually, a lightweight drafting mechanism predicts several candidate tokens ahead of time. The target model then verifies those candidates together in a single forward pass.
By accepting multiple valid tokens per verification step, speculative decoding increases the useful work accomplished each time the target model accesses its weights. This approach makes better use of computational capacity that might otherwise remain underutilized during memory-bound decoding, without sacrificing the target model’s output behavior.
What Modalix Demonstrates
SiMa.ai has implemented three speculative decoding approaches—EAGLE3, DFlash, and Gemma 4 Multi-Token Prediction (MTP)—within its LLiMa inference framework, running on the Modalix MLSoC. The results challenge the assumption that faster LLM inference demands more power.
Using DFlash N16 with Qwen3.5 9B, token generation increased from 11.98 to 41.75 TPS, a 3.49× speedup, while mean SoC power remained essentially unchanged at approximately 7.5W. This improvement also reduced compute energy per token by approximately 71%, proving Modalix MLSoC scales inference performance without breaking the power envelope.
But raw TPS is only part of the story.
In a separate automotive assistant benchmark, Gemma 4 E4B with MTP reduced the time required to generate a complete structured tool call from 1.170 seconds to just 0.569 seconds—a 2.06× improvement, with 51.4% lower SoC compute energy per request.
Unlike raw throughput measurements, this benchmark captures the time required to produce a complete, actionable command. It demonstrates how faster decoding can translate into meaningful improvements in application responsiveness. For conversational assistants, users spend less time waiting—and experience a smoother, more responsive user experience.

Figure 1 – Performance improvements with no increase in power
Reason Faster, Without Brute Force
Unlike cloud data centers, Physical AI systems operate within strict power, thermal, and BOM constraints that make brute-force compute scaling unsustainable. Speculative decoding offers another path: extracting more useful performance from existing compute resources by addressing the inefficiencies of sequential token generation.
In Physical AI, winning the performance race isn’t simply about adding more TOPS. It’s about extracting more intelligence from every watt and every model execution.
Dive Deeper
How do EAGLE3, DFlash, and Gemma 4 MTP achieve these improvements, and what are their implications for on-device performance and energy efficiency?
Explore the architectures, benchmark results, and engineering insights in our white paper, Accelerating LLM Inference for Physical AI with Speculative Decoding.