Hybrid spectrum sensing framework for optical OTSM waveform in 5G visible light communication systems

Abstract This paper presents a robust hybrid spectrum sensing framework for optical orthogonal time–space modulation (OTSM)-based 5G visible light communication (VLC) systems aimed at enhancing detection accuracy, spectral efficiency, and communication reliability under practical optical channel conditions. The proposed method integrates time-domain energy detection with delay–Doppler domain feature extraction and applies adaptive SNR-dependent decision fusion to improve sensing performance across varying noise levels. A comprehensive mathematical system model is developed considering IM/DD constraints, LED nonlinearity, multipath VLC propagation, and additive optical noise. The framework is implemented in MATLAB and evaluated using key performance metrics including probability of detection (Pd), probability of false alarm (Pfa), receiver operating characteristic (ROC), adaptive threshold behavior, bit error rate (BER), and power spectral density (PSD). Simulation results demonstrate that the proposed hybrid algorithm achieves significant SNR gains at a target detection probability and substantially reduces false alarm rates compared to ED, MF, CSD, ML-based sensing, and conventional optical OTSM approaches. At a BER of 10 −3 , considerable SNR savings are achieved, confirming enhanced reliability. Improved spectral confinement and reduced out-of-band leakage further validate efficient bandwidth utilization, making the proposed framework suitable for next-generation VLC networks.