Edge AI agents now choose their own next step: up to 26% higher accuracy and 28% lower resource use than fixed workflows
Synopsis
Edge AI development has relied on developers hand-building fixed agent workflows that struggle with heterogeneous data modalities and diverse task pipelines. SE2 distills development into four atomic operations, retrieve, reason, code and test, and lets a coordinator agent pick the next one from the current context, while mobile runtime feedback and reinforcement fine-tuning let the agents keep evolving. On the IoTBench benchmark and three multimodal edge AI applications, its synthesized programs reach up to 26% higher accuracy and 28% lower resource consumption than eight state-of-the-art baselines.
Interpretation
Agentic edge AI development shifts from workflows written in advance by developers to agents deciding their own next step from the current context. Earlier systems such as AutoIOT, AutoEmbed and IoTPilot ran predefined pipelines of knowledge retrieval, task decomposition, code generation and debugging, with the LLM acting only as a code generator following fixed instructions. On human activity recognition with the XRF55 dataset, repeated 100 times with random modality removal, only 32% of programs synthesized by these fixed-workflow methods executed successfully and only 8% matched a handcrafted program's performance.
The system decomposes development into four atomic operations, retrieve, reason, code and test, and a coordinator agent keeps a global view while dispatching four task-specific agents. A single agent handling every operation accumulates long contexts and can repeat the same action without terminating; the coordinator-actor split reduces long-context reasoning complexity. In 100 trials synthesizing a multimodal human activity recognition program on a Jetson Nano, the single-agent setup succeeded only 8 times, comparable to unmodified existing agents.
Runtime semantics from the target device are collected by synthesized probe programs and returned to the agents as actionable feedback for adjusting code and decisions. Even with detailed device specifications, LLMs lack runtime feedback from real mobile devices and often overlook CPU architecture, environment compatibility and memory allocation constraints. In the human activity recognition ablation, removing full feedback cut success rate by 66% and doubled the time to produce executable code; replacing probes with the general monitor psutil raised synthesis time about 1.5 times.
Coordinator and atomic agents are continuously updated by ensemble mutual-actor-critic reinforcement fine-tuning, while a self-reflective grader scores the programmer on code quality, performance and algorithm specialization. Relying on vanilla LLMs for decisions cannot keep pace with emerging edge AI knowledge and tasks, while maintaining a retrieval database demands substantial domain expertise. On the augmented IoTBench set of 200 coding tasks, split 20% for tuning and 80% for testing, SE2 averaged 11% higher code similarity, 17% higher pass@1 and 1.6 more code-quality points than baselines.
Perspective
The demonstrated scope is the augmented IoTBench benchmark and three multimodal edge AI applications, human activity recognition, vision-language understanding and autonomous-driving object detection, deployed on a Raspberry Pi, a Jetson Nano and a 16GB-VRAM edge laptop; the user study involved 10 expert and 10 non-expert users.
The single-agent feasibility test succeeded only 8 times out of 100, so adaptive workflows alone do not guarantee success; the authors also note that LLMs can hallucinate during multi-step reasoning and rely on Reflexion-style reflection to mitigate it, and acknowledge that a larger, more diverse participant cohort would more fully assess overall and long-term performance.