SPoT: Better Frozen Model Adaptation through Soft Prompt Transfer

There has been growing interest in parameterefficient methods to apply pre-trained language models to downstream tasks. Building on the PROMPTTUNING approach of Lester et al. ( SPOT first learns a prompt on one or more source tasks and then uses it to initialize the prompt for a target task. We show that SPOT significantly boosts the performance of PROMPT-TUNING across many tasks. More remarkably, across all model sizes, SPOT matches or outperforms standard MODELTUNING (which finetunes all model parameters) on the SUPER-GLUE benchmark, while using up to 27,000 fewer task-specific parameters. To understand where SPOT is most effective, we conduct a large-scale study on task transferability with 26 NLP tasks in 160 combinations, and demonstrate that many tasks can benefit each other via prompt transfer. Finally, we propose an efficient retrieval approach that interprets task prompts as task embeddings to identify similar tasks and predict the most transferable source tasks for a novel target task.