MM-instruct: Generated visual instructions for large multimodal model alignment

This paper presents MM-Instruct, an automated pipeline for generating diverse and high-quality visual instruction data to better align large multimodal models (LMMs) with real-world use cases. While previous works have focused on question-answering data, their generated instruction datasets pose challenges for broader application scenarios. Additionally, manually collecting diverse instruction data at scale from users is prohibitively costly. To mitigate these issues, MM-Instruct leverages ChatGPT to automatically generate diverse instructions from a limited set of seed instructions through augmentation and summarization. It then uses an open-sourced large language model (LLM) to construct instruction-following answers and builds a large-scale visual instruction dataset. Evaluating the LLaVA-Instruct models trained with the generated data shows significant improvements in instruction-following capabilities compared to LLaVA-1.5 models. MM-Instruct releases its synthetic dataset containing diverse instructions and high-quality instruction-answer pairs to support training LMMs for real-world applications. • Introduces MM-Instruct: An automated pipeline generating diverse visual instruction data to improve LMM performance on real-world tasks beyond simple QA. • Demonstrates Improved Instruction Following: Models trained with MM-Instruct data significantly outperform prior models (e.g., LLaVA-1.5) in executing diverse visual instructions. • Releases Large-Scale Dataset: Contributes a new dataset with 293 diverse instructions and 234 k high-quality visual instruction pairs to advance LMM instruction-following research.