scMoE: Integrating Heterogeneous Single-Cell and Molecular Foundation Models Using a Mixture-of-Experts Framework

Foundation models, which exploit self-supervised learning on vast amounts of data, have demonstrated superior performance in various biomedical domains. However, the differences in existing foundation models, such as pre-training data sources, model architectures, and pre-training tasks, lead to noticeable variations in their performance across diverse biomedical tasks. Therefore, it remains unclear how to select the most appropriate foundation model for a new task. Here, we propose an alternative solution scMoE to integrate multiple foundation models within the same domain through a mixture-of-expert framework. The key idea of scMoE is to selectively activate certain foundation models according to the specific task and input using a multi-head sparse router. scMoE can be further extended to integrate foundation models across modalities, such as integrating multiple single-cell foundation models and multiple molecular foundation models. Our experiments across three biomedical tasks, including cancer drug response prediction, drug combination perturbation prediction, and synergistic drug combination prediction, demonstrate that scMoE significantly outperforms individual foundation models, highlighting its effectiveness and robustness in complex biomedical applications. Overall, scMoE provides a novel integrative framework for utilizing heterogeneous foundation models, paving the path for applying multiple diverse foundation models to new tasks.