Explainable AI in Conceptual Design: An LLM-Powered Framework to Enhance Originality and Credibility

Abstract Recent advances in Large Language Models (LLMs) offer new opportunities for conceptual design, but their opaque reasoning often limits their adoption as active design agents. Explainable Artificial Intelligence (XAI), defined as techniques that make AI reasoning understandable to humans, provides a path to integrate these models into engineering design workflows with greater transparency and trust. In this paper, we explore the role of explainability mechanisms in conceptual design by introducing an XAI-based design framework structured around divergence and convergence cycles. To this end, LLMs generate and evaluate design solutions, while explainability mechanisms allow designers to visualize concept relationships and clarify model recommendations. The framework's effectiveness is tested through a controlled user study with 88 design engineering participants across three groups: GPT-4 without explanations (n=39), XAI-enhanced framework (n=39), and raters (n=10). Participants from the first two groups addressed one of three realistic design challenges and produced conceptual solutions, while evaluators from the third group rated the answers on key design metrics: usefulness, feasibility, originality, and credibility. Data was analyzed using linear mixed-effects models to ensure robustness, whereas pairwise comparisons and simulation-based power analyses were used to control Type I and Type II errors respectively. Results show that XAI significantly improves the originality and credibility of AI-assisted design outputs, while no differences were found in the usefulness and feasibility metrics. These results provide rigorous evidence that explainability enhances human–AI collaboration for conceptual design.