Regime-Aware LightGBM for Stock Market Forecasting: A Validated Walk-Forward Framework with Statistical Rigor and Explainable AI Analysis

Can machine learning generate statistically validated alpha in equity markets while adapting to changing market conditions? This study addresses this question by proposing a regime-aware LightGBM framework conditioned on market regimes detected via a rolling Hidden Markov Model, eliminating look-ahead bias. Backtested on 51 NASDAQ-100 constituents (2015–2026), the strategy achieved a portfolio Sharpe ratio of 1.18 (95% CI: [0.53, 1.84]) and outperformed four baseline models. The key findings include the following: (i) cross-asset features (Bitcoin as a leading indicator) contribute the most predictive value; (ii) macroeconomic indicators outweigh traditional technical indicators for high-beta stocks; (iii) the model autonomously adapts its decision logic across regimes, shifting from mean reversion in bear markets to risk appetite monitoring in bull markets. While block bootstrap tests confirm statistical significance (p<0.001), the Deflated Sharpe Ratio (0.69) does not reach formal significance after multiple testing correction—an honest finding we report transparently.