MultEval: Collaboratively Creating Criteria for LLM-as-a-Judge Systems

LLM-as-a-judge approaches have emerged as a scalable solution for evaluating model behaviors, yet they rely on evaluation criteria often created by a single individual, embedding that person’s assumptions, priorities, and interpretive lens. In practice, defining such criteria is a collaborative and contested process involving multiple evaluators with different values, interpretations, and priorities—an aspect largely unsupported by existing tools. We present MultEval, a system designed to support collaborative criteria creation for LLM-as-a-judge workflows. MultEval enables multiple evaluators to jointly create, compare, and refine evaluation criteria by surfacing disagreements early, diagnosing sources of conflict using consensus-building theory, and encouraging structured exploration of alternative criteria formulations. Our system demonstrates how intelligent user interfaces can better support multi-stakeholder alignment in human–AI evaluation.