Seven KNN hyperparameter tuning strategies compared across 15 datasets: PSO ranks best overall but significantly beats only random search

Under a unified nested validation protocol, the seven tuning strategies showed statistically significant differences in composite ranks across 15 datasets (Friedman test χ²_F = 22.859, p = 0.000845), with PSO ranking first overall (3.580), followed closely by grid search (3.680) and Bayesian optimization (3.760), while random search ranked worst (5.033). Prior studies often evaluated only one or two tuning strategies or focused solely on predictive performance; this study compares sampling-based, model-based, and population-based optimizers under the same evaluation budget (B = 40) and the same nested cross-validation framework, reporting both predictive quality and computational cost. Based on five-metric rankings across 15 public datasets (UCI, OpenML, Kaggle), using the Friedman non-parametric test and Holm-corrected Wilcoxon signed-rank tests with a rigorous statistical procedure; however, each stochastic method was run only once per dataset with a single random seed (RngSeed = 1).

PSO's best overall rank was primarily driven by its runtime advantage (runtime rank 1.267), while its accuracy rank was relatively weaker (4.500); Bayesian optimization achieved the best average accuracy rank (3.367), and grid search performed particularly well in macro AUC (3.200) and cross-validation loss (2.633). These results reveal trade-offs between predictive performance and computational cost across tuning strategies rather than uniform dominance by any single method, providing a multi-metric basis for selecting tuning strategies by scenario. Composite ranks were computed as equally weighted averages of five metrics with clear directions (accuracy and macro AUC higher is better; CV loss, CE loss, and runtime lower is better), with ties handled by average ranks.

Holm-corrected paired Wilcoxon tests showed that PSO held a statistically significant advantage only over random search (p_Holm = 0.000366, median Δ = −1.2), while differences with grey wolf optimizer, surrogate optimization, genetic algorithm, Bayesian optimization, and grid search were not significant (p_Holm ≥ 0.316). This finding refines the overall significance of the Friedman test into specific pairwise comparisons, showing that most pairwise differences among the seven methods are statistically indistinguishable, preventing over-interpretation of composite rankings. Based on paired rank differences across 15 datasets using the Wilcoxon signed-rank test with Holm's step-down procedure to control the family-wise error rate, with a complete multiple-comparison correction workflow.

The study notes that all methods achieved perfect performance on the heart disease dataset (accuracy 1, macro AUC 1, CE loss 0), but this dataset version contains 723 duplicate records among 1025 instances, which may cause an instance-based classifier such as KNN to produce overestimated performance estimates. The authors proactively flag this potential data leakage risk, cautioning readers to interpret the perfect results on this dataset cautiously, demonstrating attention to benchmark data quality issues. Based on inspection and reporting of duplicate records in the dataset; duplicates were retained rather than removed to preserve the original benchmark datasets.

The results apply to tuning the classical KNN classifier over a mixed search space comprising integer neighborhood size k ∈ [1, 51], categorical distance metrics {euclidean, cityblock, chebychev, minkowski}, and continuous Minkowski exponent p ∈ [1, 5], with an evaluation budget fixed at B = 40 objective evaluations (except grid search, which performed 182 evaluations). Experiments ran in MATLAB R2022b using an outer 80/20 stratified holdout and inner 5-fold stratified cross-validation, with standardization performed strictly within training folds to avoid data leakage. The framework provides a reproducible benchmark for selecting tuning strategies under limited computational resources, with code publicly available on GitHub. Conclusions are relevant to small-to-medium tabular classification datasets, and the authors recommend selecting tuning strategies based on dataset dimensionality, class structure, and objective irregularity.

Each stochastic method in this study was run only once per dataset with a single random seed (RngSeed = 1), and the authors explicitly list multiple-seed repeated experiments as future work, so the stability of the current ranking results remains to be verified. Composite rankings use equally weighted averages of five metrics, and the authors note that sensitivity analysis of alternative weighting schemes is needed. Grid search performed 182 evaluations while other methods used only 40, so the two are not fully evaluation-matched, and the impact of this difference on rankings warrants attention. Additionally, the heart disease dataset contains 723 duplicate records leading to perfect performance for all methods, so results on this dataset should not serve as a basis for method discrimination. The authors also propose future work on multi-objective tuning, hybrid strategies combining surrogate modeling with population-based search, and newer metaheuristics, which are not yet covered in this study.