Artificial Analysis

Most knowledge-work benchmarks can be solved with text alone, without ever looking at an image or page layout, yet everyday professional work such as architectural blueprints, engineering schematics and polished reports is deeply visual. Google and Artificial Analysis launched MKA, a multimodal knowledge agent benchmark whose 200 tasks were written by domain experts across 10 verticals and 83 occupations, each paired with roughly 70 yes/no rubric checks weighted 75% on factual accuracy and 25% on visual design. When domain experts graded model outputs themselves, they agreed with the autorater 96% of the time; the 200-task set stays private, and 10 public tasks are released on Hugging Face so anyone can inspect the format.

Most knowledge-work benchmarks can be solved with text alone, without ever looking at an image or page layout, yet everyday professional work such as architectural blueprints, engineering schematics and polished reports is deeply visual. Google and Artificial Analysis launched MKA, a multimodal knowledge agent benchmark whose 200 tasks were written by domain experts across 10 verticals and 83 occupations, each paired with roughly 70 yes/no rubric checks weighted 75% on factual accuracy and 25% on visual design. When domain experts graded model outputs themselves, they agreed with the autorater 96% of the time; the 200-task set stays private, and 10 public tasks are released on Hugging Face so anyone can inspect the format.

Cognition built FrontierCode 1.1 with maintainers of 36 open source repositories, asking agents to change real repositories and judging whether a maintainer would merge the change. Grading goes beyond passing tests to cover excessive changes, edits outside the relevant scope, and unhelpful tests. Artificial Analysis runs the 100-task Main set in one shared Stirrup harness with three attempts per task; Claude Opus 5.5 leads at 52% average score and 58% pass rate, yet leading models still fail more than 40% of attempts.

Cognition built FrontierCode 1.1 with maintainers of 36 open source repositories, asking agents to change real repositories and judging whether a maintainer would merge the change. Grading goes beyond passing tests to cover excessive changes, edits outside the relevant scope, and unhelpful tests. Artificial Analysis runs the 100-task Main set in one shared Stirrup harness with three attempts per task; Claude Opus 5.5 leads at 52% average score and 58% pass rate, yet leading models still fail more than 40% of attempts.