Edge AI agents face rigorous MLPerf testing across multi-turn trajectories, where NVIDIA Jetson Thor hardware cut execution runtime 6.4x against reference software configurations.
MLPerf Inference v6.1 from MLCommons added an Edge Agentic Inference workload alongside an End-to-End Retrieval-Augmented Generation benchmark. The workload evaluates autonomous software-engineering tasks rather than separate single-turn prompts. Models must execute consecutive turns, manage tools, and parse feedback while operating within tight local power and memory limits.
Miro Hodak, MLPerf Inference working group co-chair, said: “We are working hard to ensure that the MLPerf Inference benchmark continues to reflect the scenarios that the AI community values most.
“We added the end-to-end RAG test because it’s clear that query-answering has evolved beyond simply an LLM trained on a corpus; stakeholders need to understand the real-world performance of the types of multi-step, multi-component pipelines that are being built today.
“Likewise, we added the Edge Agentic Inference test because complex inference systems with agentic properties are increasingly hosted on edge computing devices, creating a new set of performance challenges our customers face today.
“We are committed to providing timely and relevant performance data that reflects real-world production systems and the performance optimisations that are being deployed today.”
NVIDIA Jetson Thor accelerates multi-turn edge agentic inference
The Edge Agentic benchmark measures an OpenAI-compatible endpoint across performance and accuracy phases.
During performance testing, systems replay twenty recorded software-engineering trajectories covering 1,007 turns. Context histories grow across turns to roughly 23.5K tokens, placing severe pressure on local DRAM bandwidth. Accuracy runs deploy Berkeley Function Calling Leaderboard v4 prompts in single-turn mode to verify tool-call validity without active reasoning steps.
NVIDIA submitted results for its Jetson AGX Thor Developer Kit under the MAXN power profile with 128 GB of unified memory.
Running Qwen3.6-27B on ‘TensorRT Edge-LLM’ in SingleStream mode, the board achieved 52.33 tokens per second. Median time to first token measured 247.12 ms, while median time per output token registered at 14.68 ms. The configuration finished the entire 1,007-turn suite in 24 minutes and 36 seconds.
The board’s overall accuracy score reached 87.94 percent on function-calling tests. A baseline llama.cpp implementation with Q4_K_M quantisation on identical Jetson hardware completed the run in 2 hours and 37 minutes.
Engineering teams secured this speed advantage through memory compression and execution caching. The submission formatted weights and activations in NVFP4 on the Blackwell graphics processor, keeping the key-value cache in FP8 precision. This 4-bit layout conserved unified memory for long context sequences.
TensorRT Edge-LLM retained cached attention pages alongside recurrent states across turns. Because the system prefilled only new suffixes, warm cache served approximately 96 percent of prompt tokens. That mechanism avoided recalculating 13.1 million tokens out of the 13.6 million prompt tokens in the workload.
A tree-based multi-token prediction algorithm drafted eight steps forward across a 16-node verification tree. This mechanism lifted decoding throughput by roughly 40 percent compared to three-step linear drafting.
Speculative decoding and API harnesses expand testing
The v6.1 release introduced edge silicon options alongside datacentre hardware. Submissions featured AMD Ryzen AI Max+ 395 processors, Intel Arc Pro B70 graphics cards, and AMD Instinct MI350P enterprise accelerators. Preview submissions evaluated NVIDIA Rubin and NVIDIA Vera Rubin NVL72 architectures.
Datacentre submissions demonstrated rapid gains: per-accelerator throughput on DeepSeek R1 rose 5.7x over results from twelve months ago, while Visual Language Model performance climbed 2.99x from six months prior.
Workloads scaled to 512 accelerators in the largest single cluster. Submitters demonstrated two novel heterogeneous environments: one crossed the Pacific Ocean with distributed nodes, and another combined high-speed networking with accelerators from two distinct hardware vendors.
For interactive services, MLCommons added speculative decoding into the GPT-OSS task to verify multiple candidate tokens within a single forward pass.
Frank Han, MLPerf Inference working group co-chair, said: “With the critical data from the MLPerf Inference benchmark, the AI community is once again proving that what can be measured can be improved.
“As workloads and the underlying systems increase in capability, scale, and complexity across a broader variety of form factors and physical configurations, we are clearly seeing a renewed interest in ensuring that inference can take full advantage of hardware, software, and architectural advances – both in the datacentre and at the edge.
“With performance data from the Inference v6.1 benchmark, customers can better understand the cost-benefit tradeoffs and make informed decisions on how to procure and deploy their AI systems.”
Participation climbed to thirty organisations, including first-time submitters Atlas Inference, Crusoe, Orrick Industries LLC, ScitiX, VibeHPC, and independent engineer Naeem Khoshnevis. More than 50 percent of entrants adopted an API-centric harness that uses client-server requests to reflect datacentre environments.
MLCommons confirmed that this interface serves as the foundation for its upcoming MLPerf Endpoints suite, which will succeed the standalone Inference benchmark.
See also: Ambarella and ZEDEDA bring cloud orchestration to edge AI silicon
Want to learn more about the IoT from industry leaders? Check out IoT Tech Expo taking place in Amsterdam, California, and London. The comprehensive event is part of TechEx and is co-located with other leading technology events including AI & Big Data Expo and the Cyber Security Expo. Click here for more information.
IoT News is powered by TechForge Media. Explore other upcoming enterprise technology events and webinars here.
