Nearly half of a widely used diabetes symptom dataset repeats itself: grouping by profile drops ensemble accuracy from 98% to 90–93%

Row-level random cross-validation lets copies of one symptom profile appear in both training and test folds, so measured accuracy partly reflects recognition of training records rather than generalization to new profiles. Previously reported accuracies on this dataset range from about 97% to 99%, including one study reporting 98.1% with 10-fold cross-validation and 100% without it. Under random partitioning an average of 68.2% of test records had an identical profile in the training folds; under profile grouping this proportion was zero, and MCC fell by 0.105 to 0.157 across ten ensembles, by 0.084 for logistic regression and by 0.064 for the nearest-neighbour reference.

Keeping every record of a profile in the same fold brings ensemble accuracy down to 90.2–93.2%, against the near-ceiling values reported in the literature. Under random partitioning the nine tree-based ensembles reached 97.2–98.2% accuracy and weighted F1 of 0.972–0.982, consistent with the 96.8–99.0% reported previously. Both schemes used identical pipelines, so differences reflect partitioning alone; optimism is reported as the mean over ten repetitions with 2.5th–97.5th percentiles, for example 0.157 (0.097 to 0.220) in MCC for the stacking ensemble.

Once the partitioning changes, model rankings are no longer stable and most pairwise differences between ensembles cannot be distinguished statistically. Under random partitioning the stacking ensemble tied for the highest MCC at 0.961; under profile grouping it fell to 0.804 and ranked eighth. Kendall's τ between the two rankings was 0.71 for MCC and 0.82 for ROC-AUC; after Holm adjustment only AdaBoost had significantly lower MCC than extra trees, and no ROC-AUC difference was significant.

Near-duplicate symptom patterns that differ only in age add a further, model-dependent optimism, and the nearest-neighbour rule loses the most. Under profile grouping the nearest-neighbour reference reached MCC 0.870, close to extra trees at 0.857. Grouping by symptom pattern instead lowered its MCC to 0.799 and its ROC-AUC to 0.902, the lowest of the twelve classifiers, while extra trees retained the highest MCC (0.826) and ROC-AUC (0.972).

The findings concern one dataset: 520 records, 16 predictors and a binary status label from Sylhet Diabetes Hospital, Bangladesh. Twelve classifiers were evaluated with hyperparameters fixed before analysis, using ten repetitions of 5-fold cross-validation and profile-level cluster bootstrap intervals. The reported values are internal estimates for a dataset-provided label and do not establish diagnostic or screening utility.

The origin of the duplicate profiles is unknown: if some represent different individuals with identical answers, similar profiles may occur among future patients and performance for new patients would lie between the grouped and random estimates. Symptom-pattern grouping may separate distinct individuals whose answers coincide, so it is treated as a lower bound. The outcome definition, diabetes type and timing of symptom recording are undocumented, the data come from a single hospital, and the 61.5% positive proportion far exceeds screening prevalence, so absolute probabilities would need recalibration before use elsewhere.