SpBench hands cross-engine lakehouse benchmarking to AI agents: 400-plus queries run on a cadence, and authoring drops from weeks to one loop
Synopsis
Cross-engine lakehouse benchmarking has long depended on cross-functional collaboration and domain experts, and the same logical query is hard to compare fairly across execution engines. SpBench compiles platform-agnostic benchmark definitions into engine-specific implementations through typed constructs, records every run as queryable telemetry, and lets AI agents distill, build and analyze benchmarks through three composable skills. The system is in production use for benchmarking Snowpark, Snowpark Connect and other environments, with more than 400 queries running on a regular cadence to catch regressions, and the authoring cycle has shrunk from weeks or months to a single AI development loop.
Interpretation
Cross-engine benchmarks move from human convention to equivalence guaranteed by construction: one platform-agnostic benchmark definition compiles into implementations for each engine, so the comparison covers the same logical query rather than separately hand-written versions. Workload specifications previously standardized what to run but said nothing about how to compose engine-specific implementations of the same logical query, how to match compute tiers of comparable cost, or how to verify that two implementations produced equivalent results; ad-hoc engineering filled those gaps. A fair comparison must hold data, query semantics and compute tier equal, and the same BenchmarkDefinition produces all artifacts, so query equivalence is enforced by construction rather than by convention.
Every run is written as a star-schema telemetry record, so agents can query results with deterministic SQL instead of parsing free-form output. Naive agent-in-a-shell approaches, where a model issues raw shell commands without typed schemas or structured intermediaries, often fail to produce comparable cross-engine implementations and struggle to fit multi-engine understanding into a single context. A run emits one fact_runs row, one fact_queries row per query with timing and structured error fields, and content-addressed dimension records for deployment, compute tier and environment, exposed as external tables; a per-step timing table localizes whether latency sits in data loading, compilation or execution.
The agent layer pairs three composable skills with two governed loops, splitting benchmark authoring into static code generation, code review, SF1 validation and full-scale execution. A monolithic agent that plans, writes, debugs and analyzes in one continuous reasoning chain breaks down on the Build skill under context overload as iterations accumulate. The Build skill was developed with TeraSort as a golden end-to-end test case, targeting reliable execution across every supported engine, execution mode and compute configuration, until a single invocation reached the validated state without manual intervention.
Perspective
The demonstrated scope is a single-organization production deployment inside Snowflake over a fixed engine set, covering Snowpark, Snowpark Connect for Apache Spark and Apache Spark on other lakehouse platforms, with benchmarks drawn from standardized suites such as TPC-DI and TPCx-BB and from custom benchmarks distilled from production workload patterns.
These outcomes reflect a single-organization deployment across a fixed engine set, and broadening the evaluation across organizations and engines is ongoing work; the stated next step is a model in which a customer-reported signal can be resolved by anyone, guided by AI.