Non-vision-based localization and feedback control of untethered magnetic microrobots with hall-effect magnetic sensors under actuation field interference

Ocular trauma, including corneal opacity and intraocular bleeding, are often caused by blast, laser, or shrapnel injuries. Such opacities obstruct optical tracking and thereby impede intraoperative imaging and the visual control of untethered magnetic microrobots. Magnetic sensors have been used as an alternative approach for non-vision-based tracking to cope with visual confinement. However, accurate measurement is challenged by interference from an external magnetic field used for actuation. To resolve this issue, a non-vision-based localization approach using an alternating off–on strategy is introduced: (1) during localization (off), the field is temporarily deactivated to eliminate interference, allowing magnetic sensors to measure the microrobot’s intrinsic magnetic field; (2) during actuation (on), Helmholtz coils generate a controlled magnetic field. In the localization phase, a deep neural network estimates positions under low signal-to-noise conditions (15–17 dB), and a Kalman filter incorporating the dynamic model refines the estimates. Experimental validation in a 20 × 20 × 20 mm³ workspace within 1000 cSt silicone oil demonstrates a non-vision-based localization and feedback control, achieving a mean position error of 1.5 mm and confirming potential suitability for intraocular surgical applications, where visual feedback is limited. ( a ) Schematic representation of the external magnetic field excitation and magnetic field acquisition for microrobot localization using an off-on strategy. Phase I (off period) corresponds to a brief holding period during which magnetic sensors capture the microrobot's magnetic field for localization, while Phase II (on period) involves the application of an external magnetic field gradient for microrobot actuation. ( b ) Diagram of the non-vision-based localization framework for single and swarm microrobots. A single microrobot is localized using a deep neural network (DNN) combined with a Kalman filter (KF) that incorporates the microrobot’s dynamic model, while swarm microrobots are localized using an Extended Kalman Filter (EKF) that integrates their respective dynamic models. • Introduces a non-vision-based localization framework for untethered magnetic microrobots using Hall-effect magnetic sensors and an off–on actuation strategy that temporally separates sensing from magnetic actuation to mitigate interference. • Combines a deep neural network (DNN) with a Kalman filter to estimate microrobot position from magnetic flux measurements under low signal-to-noise conditions (15–17 dB) , enabling reliable localization without visual feedback. • Demonstrates accurate 3D localization and closed-loop control in a 20 × 20 × 20 mm³ workspace by integrating DNN+KF localization with Model Predictive Control (MPC), enabling autonomous trajectory tracking along geometric paths (S, C, star, hourglass) with average tracking errors below ~1.5 mm without visual sensing . • Demonstrates dual-agent localization capability using an Extended Kalman Filter (EKF) , enabling simultaneous tracking of multiple microrobots using magnetic sensor measurements. • Provides proof-of-concept for non-vision-based localization , demonstrating feasibility of magnetic-sensor-based tracking for microrobot navigation in environments with limited visual access.