Understanding the Interplay Between User Experience and Performance in Multimodal Virtual Reality Interaction Across Input Modalities and Perspectives

Multimodal interaction plays an important role in enabling efficient and engaging interaction in virtual reality (VR) systems. Contemporary VR applications increasingly combine multiple input modalities, such as controller-based interaction, hand tracking, and voice commands, often supported by AI-based assistance mechanisms. This paper presents a controlled experimental investigation of multimodal interaction in VR, focusing on the relationship between objective performance measures and subjective user experience. A structured VR task environment was implemented in which input modality (controller, hand tracking, and voice input) and spatial perspective (bird's-eye view and life-sized view) were systematically varied. The results show clear differences between interaction configurations. Controller-based interaction provides the highest performance across all objective metrics, while hand tracking and voice input result in lower efficiency and increased workload. The findings indicate that performance and user experience do not improve uniformly across configurations, but are shaped by a complex interplay of input modality and spatial perspective.