Hierarchical Localization and Tracking for In-Vehicle Mixed Reality

Effective localization and tracking are cornerstones of immersive Mixed Reality (MR). However, existing systems often fail in environments with complex motion, such as the interior of a moving vehicle. This setting presents unique challenges, including conflicting visual information and corrupted inertial sensor data. We propose a novel hierarchical system for in-vehicle MR that separates the user’s pose relative to the vehicle from the vehicle’s global pose. By using an edge device with a high-precision Global Navigation Satellite System (GNSS) receiver and an Inertial Measurement Unit (IMU) to estimate the vehicle’s state, and fusing gyroscope-based tracking with marker-based localization for the user, our system demonstrates a promising approach. Experiments show our method significantly outperforms a commercial solution, reducing the average localization error from 164.68m to 8.45m.