Public articles linked to the same research event.
arXiv Using 52,500 judgments from seven open-weight models of 1B to 32B parameters, collected under three prompt conditions with ten samples per item on a public benchmark with human labels, the authors test the latent class assumption that repeated ratings are conditionally independent, find intraclass correlations of 0.50 to 0.93 on the balanced split so that ten ratings carry the information of 1.07 to 1.81 independent ratings, and propose a beta-binomial latent class model with a single correlation parameter, establishing its identifiability and that of extensions to covariates, treatment effects and partially labeled data, raising coverage of nominal 95% intervals for prevalence and the false acceptance rate to 0.90 on the balanced split.
Using 52,500 judgments from seven open-weight models of 1B to 32B parameters, collected under three prompt conditions with ten samples per item on a public benchmark with human labels, the authors test the latent class assumption that repeated ratings are conditionally independent, find intraclass correlations of 0.50 to 0.93 on the balanced split so that ten ratings carry the information of 1.07 to 1.81 independent ratings, and propose a beta-binomial latent class model with a single correlation parameter, establishing its identifiability and that of extensions to covariates, treatment effects and partially labeled data, raising coverage of nominal 95% intervals for prevalence and the false acceptance rate to 0.90 on the balanced split.
Using 52,500 judgments from seven open-weight models of 1B to 32B parameters, collected under three prompt conditions with ten samples per item on a public benchmark with human labels, the authors test the latent class assumption that repeated ratings are conditionally independent, find intraclass correlations of 0.50 to 0.93 on the balanced split so that ten ratings carry the information of 1.07 to 1.81 independent ratings, and propose a beta-binomial latent class model with a single correlation parameter, establishing its identifiability and that of extensions to covariates, treatment effects and partially labeled data, raising coverage of nominal 95% intervals for prevalence and the false acceptance rate to 0.90 on the balanced split.
Using 52,500 judgments from seven open-weight models of 1B to 32B parameters, collected under three prompt conditions with ten samples per item on a public benchmark with human labels, the authors test the latent class assumption that repeated ratings are conditionally independent, find intraclass correlations of 0.50 to 0.93 on the balanced split so that ten ratings carry the information of 1.07 to 1.81 independent ratings, and propose a beta-binomial latent class model with a single correlation parameter, establishing its identifiability and that of extensions to covariates, treatment effects and partially labeled data, raising coverage of nominal 95% intervals for prevalence and the false acceptance rate to 0.90 on the balanced split.