Koopman-Driven Predictive Modeling of Vehicle-Driver Interaction

This work uses Koopman operator theory to present a data-driven, control-oriented framework for vehicle and driver action prediction. Sensor data and visual context can now be included in vehicle control systems thanks to recent automotive sensing, computation, and machine learning developments. Building on these advances, we suggest a Koopman-based method whereby the nonlinear interactions between the driver, vehicle, and surroundings are raised into a higher-dimensional space from where they can be modelled with linear operators. Using a ResNet-18 based feature extractor, our approach combines visual cues extracted from RGBD images with time-delayed vehicle state data, capturing dynamic and contextual information about driver behaviour. Using extended dynamic mode decomposition, separate Koopman operators are learnt for vehicle states and driver actions, enabling multi-step prediction and the use of linear control strategies. Validation with unseen data shows that adding visual context to vehicle states greatly increases driver response prediction accuracy over models that depend just on time-series data. By allowing automation agents to predict driver actions, the proposed framework helps build shared control systems, improving cooperation and safety in intelligent vehicle applications.