Predictive Resource Estimation For Efficient HPC Cluster Scheduling Using Machine Learning And Slurm Simulation

Authors: Hariprasad Thorve, Pratik Ghadage, Muskan Shaikh, Prafull Barathe, Dr. R.V. Babar, Prof.A.P. Joshi

Abstract: High-performance computing (HPC) environments depend on effective resource allocation to maintain throughput, fairness, and utilization. In practice, many jobs request more memory and execution time than they consume, causing frag-mentation and delays. This paper presents a framework for predictive resource estimation using supervised machine learning, monitoring, and Slurm simulation. Historical job attributes are used to estimate actual memory demand. Linear Regression, Decision Tree, and XGBoost were evaluated. XGBoost achieved the best performance with lowest error and highest score. The study integrates prediction, observability, and simulation to improve scheduling efficiency in institutional HPC clusters.