Unified Timbre Transfer: A Compact Model for Real-Time Multi-Instrument Sound Morphing

Recent advances in transformer-and diffusion-based deep-generative models have significantly impacted the field of music and audio synthesis. However, controllable and real-time interactive models, such as those used for timbre transfer in music production, remain largely dominated by auto-encoders and generative adversarial networks. In pursuit of efficient and flexible timbre morphing and multi-instrument timbre transfer, we propose a simplified modeling approach, utilizing an upsampled two-dimensional timbre space in conjunction with engineered and instrument-dependent excitation signals. Different from many other works, our model enables any-to-many timbre transfer with added control over timbre, pitch, and loudness. We additionally allow for seamless interpolation between instruments, eliminating the need for separate model training. Our evaluation shows performance comparable to specialized models, making it highly relevant for the broader creative audio community.