An independent LLM judge audits PET depolymerisation records: 67.5% of 3,942 pass as written, with fixes concentrated in catalyst and solvent mass
Synopsis
Experimental data on homogeneous catalytic PET depolymerisation is scattered across prose, tables and supporting information, and manual consolidation does not scale. This pipeline has an extractor tag each record with the unique identifiers of its source text chunks, then an independent LLM judge reads the full paper, checks each field and proposes cited corrections. On 295 hand-curated ionic-liquid PET glycolysis experiments the judge agreed with human evaluation on 85.7% of reviewed records; across 3,942 records from 561 papers it accepted 67.5% as written, with corrections concentrated in quantitative fields such as catalyst and solvent mass.
Interpretation
Experimental records can now be pulled from chemistry literature in bulk with traceable provenance and independently re-checked by a second model, without a hand-curated answer key to compare against. Automated extraction of structured data from chemistry literature has generally been validated against existing reference datasets or judged at the record or criterion level, which does not establish that each field is supported by the source and belongs to the same experiment. On 295 hand-curated ionic-liquid-catalysed PET glycolysis experiments from 24 papers, the judge matched human evaluation on 96 of 112 sampled records, or 85.7%.
Extraction and review are split between two models: one turns papers into structured records, the other only looks for errors. The extractor is required to fill in the unique identifiers of the source chunks behind each record, and the judge receives both the full paper and that record. The judge checks field by field whether a value is supported by the source and belongs to the same experimental run, writing a critique that cites chunk identifiers before issuing an accept-or-reject verdict and corrected values.
Across the full corpus the judge marks most records usable and concentrates the cases needing human attention in a few quantitative fields. The 561 papers yielded 3,942 homogeneous catalytic records; the judge accepted 2,661 as written (67.5%), proposed cited corrections for 1,202 (30.5%) and rejected 79 outright (2.0%). Corrections cluster in catalyst mass (508 fixes) and solvent mass (327 fixes), because authors often state absolute masses once in a methods paragraph and then vary them implicitly as ratios in a results table.
Perspective
The demonstrated scope is homogeneous catalytic PET depolymerisation literature: the benchmark is 295 ionic-liquid-catalysed glycolysis experiments from 24 papers, and the at-scale result covers 3,942 records from 561 papers; the extraction schema applies equally to methanolysis and hydrolysis, but those two routes were not benchmarked against a hand-curated reference key.
The audit checks whether extracted records are accurate, not whether they are complete: it cannot catch experiments the extractor missed entirely, and it raises some false alarms, so its corrections are released as an inspectable changelog. Coverage was also limited by access, with full text obtained for 1,027 of 1,575 relevant papers and the 548 missing articles concentrated among specific publishers, while 3.5% of chunk citations failed to link to real text blocks.