EvalAgent: Interactive Comparative Evaluation of Computer-Using GUI Agents

As generalist Computer-Using Agents (CUAs) gain popularity, evaluating them solely based on task completion is insufficient to understand their reasoning processes, assess their robustness, or ensure that their actions align with user intentions. Developers and end-users lack systematic tools to compare the behaviors and reasoning processes of different agents or configurations, and to assess their alignment with user preferences and values. To address this gap, we present EvalAgent, an interactive system designed for criteria-based evaluation of LLM-powered web agents using an LLM-as-a-Judge paradigm. EvalAgent enables users to (1) configure agent models and personas as experimental conditions; (2) visualize execution trajectories as directed acyclic graphs (DAGs); and (3) employ Context-Aware Judge—a multi-scale auditing mechanism that dynamically adjusts the granularity of LLM-as-a-Judge evaluations based on user-defined criteria. Beyond binary pass/fail verdicts, EvalAgent highlights relevant segments within execution traces to surface how and why decisions were made. This workflow supports fine-grained, full-trace behavioral auditing, helping bridge the gap between high-level policy constraints, nuanced user preferences and values, and low-level agent execution.