Natural language now builds Dify workflows: ChatDify's planner-executor-reviewer agents reach 0.96 success on 100 production-style tasks
Synopsis
Building an LLM workflow on a platform like Dify used to mean manually picking nodes, filling in parameters and wiring connections, which is slow and error-prone. ChatDify hands the job to three collaborating agents: a planner that decomposes the request into a directed acyclic graph, an executor that synthesizes node parameters under schema constraints and compiles them into Dify's DSL, and a reviewer that runs static checks plus sandboxed execution with LLM judging in a closed refinement loop. On 100 production-inspired Dify tasks with Qwen-3.7, GLM-5.1 and DeepSeek-v4 as backends, success rates were 0.96, 0.91 and 0.75, above the zero-shot baseline with dynamic repair.
Interpretation
A user describes the goal in natural language and ChatDify produces a complete workflow that runs directly on Dify, including node selection, parameter values and the connections between nodes. Previously users had to select, connect and configure each node by hand in the Dify interface, which demanded both an understanding of the task logic and familiarity with the platform's specific features, and the cognitive load grew with workflow complexity. On 100 production-inspired Dify tasks spanning eight categories, ChatDify reached success rates of 0.96, 0.91 and 0.75 with the Qwen-3.7, GLM-5.1 and DeepSeek-v4 backends, versus 0.67, 0.45 and 0.36 for the zero-shot baseline with dynamic repair.
The three agents split the three levels a workflow must satisfy at once: structure, configuration and semantics. The planner breaks the request into functional blocks, retrieves similar workflow sub-graph templates from the Dify knowledge base, and instantiates them into concrete nodes and dependencies to form a directed acyclic graph plan. The executor synthesizes parameters conditioned on upstream node outputs and compiles them into Dify DSL under node-schema constraints; the reviewer first runs static checks on topology, variable types and schema compliance, then executes the workflow end to end in a sandbox and uses an LLM judge to check semantic fit, returning to a repair loop when checks fail.
Reliability comes mainly from the full planner-executor-reviewer pipeline, not from runtime repair alone. Removing dynamic runtime probing dropped success rates from 0.96 to 0.73, from 0.91 to 0.71 and from 0.75 to 0.44 across the three backends, while static verification added a further 3 to 8 percentage points. A refinement-epoch analysis shows most of the gain arrives within two epochs, for example Qwen-3.7 rising from 0.73 to 0.95, with the third epoch adding less at higher token cost.
Perspective
Evaluation is confined to the Dify v1.11.2 platform and 100 production-inspired Dify tasks covering eight categories, with Qwen-3.7 max, GLM-5.1 and DeepSeek-v4 pro as backends. The authors note that transferring to other workflow platforms requires adapting the knowledge base, node schemas and sandbox executor, though these components are modular in the planner-executor-reviewer design.
Success rate measures execution success together with LLM-judged semantic success and does not fully reflect human judgment of task completion; each check relies on a single synthesized test case, and the judge shares its backbone LLM with the generator, so success rate may overestimate true performance. The authors' manual spot checks found most workflows behaved as expected, but a stricter assessment would need multiple test cases and independent annotators. Dynamic probing and LLM judging also add token cost, with input tokens accounting for 80 to 90 percent of reported totals.