Knowledge-work benchmarks no longer stop at text: MKA tests agents on 200 visually dense tasks, with 96% expert-autograder agreement
Synopsis
Most knowledge-work benchmarks can be solved with text alone, without ever looking at an image or page layout, yet everyday professional work such as architectural blueprints, engineering schematics and polished reports is deeply visual. Google and Artificial Analysis launched MKA, a multimodal knowledge agent benchmark whose 200 tasks were written by domain experts across 10 verticals and 83 occupations, each paired with roughly 70 yes/no rubric checks weighted 75% on factual accuracy and 25% on visual design. When domain experts graded model outputs themselves, they agreed with the autorater 96% of the time; the 200-task set stays private, and 10 public tasks are released on Hugging Face so anyone can inspect the format.
Interpretation
The evaluation target for knowledge-work agents moves from text-only question answering to multi-step projects that require reading visually complex files, and MKA breaks that work into item-by-item rubric checks. Most earlier knowledge-work benchmarks could be solved with text alone, without ever needing to look at an image or page layout. MKA's 200 tasks were created by domain experts across 10 verticals and 83 occupations, and each task reflects a multi-step project that takes a seasoned professional three to eight hours to complete.
Grading is not a single overall score but an agentic autorater that opens files, renders pages and inspects spreadsheet formulas, checking each deliverable against the expert rubric. The rubric is expert-written, with roughly 70 yes/no checks per task weighted 75% on factual accuracy and 25% on visual design. When domain experts graded model outputs themselves, they agreed with the autorater 96% of the time.
Every model runs in the same open-source agent harness, Stirrup, with the same tools, sandbox and turn budget, so differences in results come from the models themselves. Artificial Analysis runs the 200-task evaluation set privately so the tasks cannot leak into training data or be found by the model via search. A separate set of 10 public tasks is released on Hugging Face so anyone can inspect the format and test their own agents.
Perspective
The evaluation covers 200 tasks across 10 verticals and 83 occupations, each reflecting three to eight hours of work for a seasoned professional; the 200-task set stays private, and only the 10 tasks on Hugging Face are public.
The public material states only that expert and autorater agreement is 96% and gives no per-model scores on the 200 tasks; the technical report says frontier models differ widely in efficiency, using between 30 and 130 turns on average per task, and shows models trying and failing to compensate when vision is removed.