Monitoring Cloud Performance Metrics Utilizing AI to Estimate the Efficiency of Cloud Operations

Cloud self-learning can be conducted through machine learning and deep learning based on AI and leads to predicting cloud performance to increase scalability in real-time applications. The focus of this paper is the Deep Neural Network (DNN) and LightGBM-based artificial intelligence approach to cloud resource optimization and monitoring. A data preparation and feature engineering strategy combined with an exploratory data analysis (EDA) approach generate useful information based on both execution time data as well as memory consumption, CPU utilization, and network traffic among other cloud performance metrics. The predictive models are more accurate and more stable with analysis of MAE, MSE, RMSE and R2 score compared to decision tree regression (DTR), and convolutional neural networks (CNN) analysis. The suggested models showed reduced MAE (0.7209 with LightGBM and 0.7218 with DNN) along with reduced RMSE (0.8307 with all of them) but with high computational efficiency. LightGBM offers faster training with excellent scaling capabilities on large workloads but has similar error distribution characteristics with DNN, and thus suitable to real-time cloud monitoring applications. These findings indicate the potential of AI-based models to improve the management of the cloud infrastructure, make use of the resources more efficiently, and reduce the cost of operation.