Evals stop at failure reports: GEPA writes them into the prompt, lifting Qwen3-32b Text2SQL test accuracy by 10%
Synopsis
Eval Protocol plugs GEPA prompt optimization into an existing evaluation interface, so the same datasets, metrics and task constraints both measure an agent and automatically rewrite its system prompt. In a Text2SQL case, Qwen3-32b gained 10% test-set accuracy and more than 5% validation-set accuracy without touching model weights, with about 45 minutes of prompt search; the same eval then served as the objective for RFT.
Interpretation
An eval suite no longer only reports where an agent fails; it converts failure signals into enforceable prompt edits and feeds them back into the agent. Previously, failure cases were handled by hand-adding instructions and examples and re-running evals while watching the number wobble. In the Text2SQL case, the unoptimized system prompt produced a correct SQL query 0 times out of 8 tries, and the optimized prompt produced a correct query 3 out of 8 tries.
The optimizer reads passing and failing examples from the eval, proposes structured prompt edits, keeps only changes that reliably move the metric up, and outputs a system prompt that can be diffed and inspected. GEPA performs this work, handling prompt manipulation and candidate evaluation through DSPy, and is called in a few lines on top of an already defined evaluation test. Prompt optimization ran on DeepSeek V3 with the same model, tasks and eval definitions, about 45 minutes of prompt search, and no change to model weights.
After prompt optimization, the same eval can serve as the objective for reinforcement fine-tuning, carrying gains from the prompt stage into training. Prompt engineering usually flattens out, after which further gains are sought by switching to a larger model, fine-tuning or running RLHF. An RFT run on Qwen3-32b with the optimized system prompt over 4 epochs reported absolute score gains of about 20%.
Perspective
The demonstrated scope is one Text2SQL agent case: a synthetic OpenFlights-style database, 243 natural-language questions, pass/fail by comparing result sets, and a validation/test split.
The reported gains come from a single Text2SQL case and specific model settings, with about 45 minutes of prompt search and 4 RFT epochs, which bound how far the results extend.