Federated and Locally Differentially Private Riemannian Masked Autoencoders for Multimodal Emotion and Activity Recognition

<div> This work addresses the challenge that current federated and locally-differentially private learning methods either leak sensitive gradient information or drown useful highdimensional signals in noise. Existing DP-FL approaches applied to Euclidean MAEs require prohibitive noise levels, while standard masked autoencoders ignore the Riemannian geometry of covariance-based biosignals. To close these gaps, we introduce the FM 3 AE-RLDP framework, a federated, multimodal masked autoencoder operating on Riemannian latent spaces under clientside local differential privacy. Multivariate sensor patches (EEG, ECG, IMU, depth) are encoded as SPD matrices, masked via geometry-aware scheduling, and reconstructed by a cross-modal Graph Transformer in the tangent space. Client updates are perturbed with an adaptive log-Euclidean Gaussian mechanism under personalized budgets and accelerated with momentum before aggregation via the Karcher mean. Adaptive PAC-Bayes bounds for joint reconstruction and classification on manifolds guide optimal mask ratios and noise schedules. We evaluate it across six public emotion and four HAR corpora, demonstrating superior utility-privacy-activity trade-offs. </div>