DTIA: Disruptive Text-Image Alignment for Countering Text-to-Image Diffusion Model Personalization

Abstract Personalized content synthesis technologies based on diffusion models have achieved significant breakthroughs, allowing for the creation of specific images from just a few reference photos. However, when these technologies are used to create fake news or unsettling content targeting individuals, they pose a substantial risk to society. To address this issue, current methods generate adversarial samples by adversarially maximizing the training loss, thus disrupting any personalized generation model trained with these samples. However, these methods are not efficient and do not fully consider the intrinsic mechanisms of successful personalization attacks. In this paper, we introduce an innovative Disruptive Text-Image Alignment (DTIA) framework. Based on the analysis that existing methods succeed in their attacks due to an overfitting of text to noise, which results in inaccurate face information matching, we have designed a Text-Image Mis-Match Attack framework. This framework aims to disrupt the model’s learning of associations between input faces and specific texts, thereby reducing unnecessary computational load. We also studied how the choice of timestep in diffusion models affects adversarial attacks and proposed a step schedule strategy to enhance algorithm efficiency. Extensive experiments on facial benchmarks have demonstrated that our DTIA framework not only disrupts personalized generation models but also significantly improves model efficiency.