An agent loop teaches itself a strange NPU: 9.293x geomean on three Gemma-4-12B decode kernels in 12 days
Synopsis
Frontier large language models write correct CUDA kernels for mature platforms, yet their accuracy collapses on accelerators whose architecture and programming language they have rarely seen. A multi-agent loop on the FuriosaAI RNGD NPU first builds its own target knowledge, a checked summary of the DSL documentation, chip facts measured by microbenchmarks and compiler-derived legality rules, then searches whole-kernel designs and small changes in per-kernel Design Trees. After 12 days the three Gemma-4-12B decode kernels reach a 9.293 geomean speedup over the official baseline, each within 1.25 of the cycles needed just to move its weights.
Interpretation
A loop of LLM agents optimized kernels on an accelerator with almost no ready-made knowledge: the target is the FuriosaAI RNGD NPU, programmed in the sparsely documented Rust DSL furiosa-opt, and the loop optimized three Gemma-4-12B decode kernels over 12 days. Earlier LLM kernel optimizers start from human-written target knowledge in their prompts or learn across many tasks; a new target has no such knowledge, and one model offers only a few tasks. The three final kernels speed up 3.75, 11.17 and 19.16 times over the official baseline, a 9.293 geomean that placed 7th of 79 teams on the public board; each kernel takes 1.11 to 1.25 its peak, the fewest cycles needed just to move its weights.
The loop treats target knowledge as something to build first: agents write a cited summary of the furiosa-opt book and check it with a script and two agent passes, measure chip behavior with microbenchmarks, and turn compile failures and device findings into legality rules. The toolchain and grader changed mid-round to furiosa-opt 0.8.1, so the loop paused, moved its kernels to the new release and repeated onboarding. Of the summary's 716 citations, 683 fully supported their claims and none contradicted the book; after the redo the cheat sheet held 141 legality rules, 78 re-verified, 16 rewritten, 2 retired and 45 carried over unverified because the new toolchain neither confirmed nor contradicted them.
Search runs on per-kernel Design Trees whose nodes are whole-kernel designs or their variants, and each kernel keeps one head, the best version that accumulates changes passing the acceptance test, instead of a beam of alternatives. Early in the run the loop used beam search and accepted a change only with a gain of at least 3%, but most single changes gained less, 1.5% to 3% on ffn, so few passed. Cheap checks screen new designs before any device run and variants are raced on the device; of 337 rendered variants, 4 won their races and one of them passed the acceptance test.
Because the device is shared and readings are noisy, the loop judges candidates with a repeated acceptance test: candidate and head are submitted in pairs, one second apart and in alternating order, and a paired statistic decides acceptance. Pairing was meant to cancel shared noise, but the two readings of a pair proved uncorrelated on Arena, so the test worked as a repeated comparison in balanced order. On the seeded grader the head's readings within an acceptance test varied with a median coefficient of variation of 0.70% on qkv, 0.64% on attn_out and 0.24% on ffn; a single test of 5 to 11 pairs passes a candidate with no real gain about 7% to 9% of the time.
Perspective
The results cover one chip, one competition and one run, with three decode kernels and two model families; the final kernels were tested only on the organizers' generated inputs, not with the real model's weights. The authors propose next adding a slower outcome to the acceptance test and designing a stable, isolated measurement environment and a grader that draws varied inputs.
The authors state that no ablations were run, so which part of the loop produced the gain is unknown, and the loop never ran without its knowledge base, so how much onboarding added is unknown. Whether the measurement noise came from the shared Arena servers or from the chip and its software could not be separated; the acceptance test passes a candidate with no real gain more often than its nominal 5%, and in practice it had no slower outcome. The peak is a planning reference, not a proven bound, and the public-board rank is not an official ranking.