YOLOBT: a novel ERP bad trial detection network dynamically adjusting based on global signal quality

Event-related potentials (ERPs) are time-locked voltage changes in averaged EEG signals reflecting neural responses to specific events. ERPs are extracted from EEG by repeating the same stimulus across multiple trials and averaging the recordings. In ERP studies, artifact-contaminated trials (commonly termed "bad trials") refer to data segments deemed unsuitable for analysis due to excessive noise or artifacts. The criteria for determining such trials depend on overall data quality: researchers increase artifact tolerance when a subject's data quality is poor to retain statistical power, while applying stricter standards when quality is high to ensure analytical purity and accuracy. Current automated bad trial detection methods rely on static thresholds and fail to replicate the adaptive strategies employed by experts. To address this limitation, we propose YOLOBT, a YOLO-based deep learning framework that mimics expert judgment by integrating global signal quality assessment with dynamic threshold adjustment. By treating EEG signals as visualized waveform images, our approach naturally aligns with expert visual inspection methods while enabling context-aware artifact detection. Our technical contributions include: (1) a Cross-Layer Attention Bottleneck (CLAB) enhancing artifact feature extraction through cross-layer attention mechanisms; (2) a Hierarchical Feature Guidance Module (HFGM) leveraging high-level semantic features to guide low-level feature refinement; and (3) a Global Information Classification Module (GICM) enabling dynamic threshold adjustment based on comprehensive signal quality assessment. Experiments on our manually annotated dataset showed YOLOBT achieved 88.76% precision, 86.89% recall, 92.76% mAP, and 87.82% F1 score, outperforming classical models. Heatmap visualization confirmed the model adaptively adjusts artifact detection strategies based on signal quality, similar to expert judgment processes.