Remote sensing models stop being hand-built: given a harness and a sealed scorer, an agent beats published results on four benchmarks
Synopsis
Remote sensing models have long been built by hand, with experts designing a training recipe for each new task and tuning it against a benchmark. RSI4EO casts a task as two parts: a harness that fixes the data, pretrained weights and compute an agent may use, and a sealed verifier that returns only a score. A general-purpose coding agent then writes the training code, trains models and improves them recursively. On four benchmarks with published results, spanning optical, SAR and multispectral time-series imagery, the agent-built models surpass the state of the art, and with about 1% of the training labels they keep most of that accuracy.
Interpretation
Building a remote sensing model shifts from humans designing the training recipe to a general-purpose coding agent working inside given resources and a given success criterion. Previously each new sensor, region or task started another round in which experts chose a backbone, designed a training recipe and tuned it against a benchmark; remote sensing agents had only orchestrated fixed human-built models and tools, and a recent agentic research loop improved fully supervised baselines by one to three points. On OpenEarthMap, BRIGHT, HRSID and PASTIS, the four benchmarks with published results, the agent-built models exceed the best published result by 4.1 mIoU, 2.7 mIoU, 4.8 mask AP and 2.6 mIoU respectively, each within 12 hours on four GB200 GPUs.
The task is split into a harness and a verifier, and the agent receives success only as the score the verifier returns, using it to improve recursively the models it trained itself. The harness fixes the labeled data, the unlabeled image pool, pretrained encoders, the software stack and the compute budget, while the verifier runs the submitted pipeline in a separate container and returns only a score, keeping validation images and labels out of the agent's reach. In the OpenEarthMap run with only six labeled tiles, the agent used the verifier to choose an encoder, labeled all 4,194 unlabeled tiles with its best model, then ran eight teacher-student generations, raising the verifier score from 56.7 to 61.9 mIoU, and finally wrote scripts that read the verifier log and promote the best pipeline to the next teacher.
Perspective
The demonstrated scope is full-label and few-label tasks on five remote sensing benchmarks, four of which have published results to compare against, while Toulouse is compared only with the starter pipeline shipped in the harness; each task runs once per agent, and the authors describe this as an ongoing project that is scaling up the number of tasks and agents.
The authors state this is an ongoing project that is scaling up the experiments and will update the results; on Toulouse the gap between verifier score and hidden-test score is much larger than on other benchmarks, on OpenEarthMap the validation-to-test gap concentrates in the bareland and water classes, and on BRIGHT most of the gap comes from the destroyed class, which the authors attribute to differences between the validation and test splits.