Meta slimmed its data-pipeline agent router: new column setup fell from days to under an hour
Synopsis
Meta replaced the orchestration layer of its data-pipeline agents with a thin router that only sequences work and runs reliability checks, leaving domain reasoning to 14 specialized subagents with isolated context. A single agent loaded with all domain knowledge worked in demos but began hallucinating in production, mixing naming conventions and code-generation patterns across subsystems. Configuration prep for a new data column dropped from three to four days by hand, or about a day with the earlier config-driven system, to under an hour, with about two-thirds of changes landing without significant human edits.
Interpretation
The orchestration layer of the data-pipeline agents is reduced to a router that does no domain work, classifying intent, gathering required fields, running reliability checks and delegating to specialist subagents. The first approach was one agent loaded with all domain knowledge, covering configuration schemas, code-generation rules, pipeline routing logic and population patterns for more than ten data sources; it worked in demos and began hallucinating in production. An alternative single-agent route reached 70-80% success with manual prompt engineering per task and fell to 10-20% without that handholding, pointing to an architectural rather than prompt-level bottleneck.
Every request follows a fixed seven-step protocol in which only the sixth step, delegation to a subagent, involves large language model reasoning; the other steps are deterministic. The conventional approach makes the orchestrator the smartest component, giving it all the context and letting it reason about what to do and how. The orchestrator is about 1,300 lines and routes more than 20 workflows to 14 specialized agents; each subagent loads only its own domain knowledge, typically a few thousand tokens, and the population agent for one data source never sees another's patterns.
New use cases are added by resequencing existing subagents through workflow templates rather than building new agents. Supporting pipeline migration, a completely new use case, required no new agent code, only a new workflow template that sequenced existing agents differently. More than 25 knowledge files load on demand at subagent execution time, and an update takes effect in the next session without orchestrator changes.
Perspective
The architecture was validated on Meta's recommendation-system data column onboarding, a process spanning three code repositories, three programming languages and up to six pipeline stages that must stay in sync. It targets production agent teams that must orchestrate multi-step workflows across heterogeneous subsystems, with a general orchestrator and specialist subagents. Next steps are scaling the architecture to new pipeline domains as data columns grow, and exploring automated feedback loops that identify failure patterns and propose targeted improvements to subagent knowledge, with human review gating any changes.
About two-thirds of changes landing without significant human edits comes at the cost of trading generality for deep domain knowledge, a deliberate architectural choice. Scaling to new pipeline domains is predicted from the compositional design rather than measured after expansion. The automated feedback loop is still exploratory, and human review gates any changes.