DP-Prox: A Robust and Differentially Private Framework for Federated Instruction Tuning of Small LLMs on 8 GB Edge Devices

This paper introduces DP-Prox, an end-to-end framework for robust and differentially private federated instruction tuning of small Large Language Models (LLMs) on 8 GB edge devices. DP-Prox is instantiated using the 3.8 billionparameter Phi-3-mini model with 4-bit Quantized Low-Rank Adaptation (QLoRA), and combines (i) FedProx-style proximal regularization on low-rank adapters to stabilize optimization under highly Non-IID smart-home data, and (ii) Differentially Private SGD (DP-SGD) applied directly to adapter gradients to provide user-level (ϵ, δ)-differential privacy against an honestbut-curious server. In a realistic IoT instruction-following benchmark with Dirichlet-skewed clients (α = 0.3), DP-Prox improves F1-score from 0.80 (Edge-FIT with FedAvg) to 0.82 while reducing convergence rounds from T = 50 to T ≈ 40. Under a strict privacy budget (ϵ = 5, δ = 10 −5), DP-Prox maintains an F1-score of 0.77 with only +15%-20% local overhead, and preserves the communication efficiency of QLoRA-based federated training. Qualitative analyses further show that DP-Prox yields more consistent instruction-following behavior across heterogeneous homes, especially on multi-device routines and privacy-sensitive queries. Together, these results position DP-Prox as a practical and regulation-ready blueprint for deploying small, edge-resident LLMs in smart-home IoT environments.