MingLLM Meta-Judge: A Machine Learning Arbitration Framework for Multi-Model LLM Answer Evaluation

MingLLM Meta-Judge is a machine learning arbitration framework designed to evaluate and rank responses from multiple large language models (LLMs). Instead of relying on a single model, the Meta-Judge compares candidate answers from heterogeneous LLM fleets such as Grok, ChatGPT or Gemini, and Anthropic Claude, and then selects the most accurate, safe, and complete response. This work presents LLM answer arbitration as a listwise learning to rank problem. The system uses a transformer-based cross-encoder that jointly reads the query, optional context, and each candidate answer. The judge outputs multi-dimensional quality scores for correctness, completeness, safety, and clarity, and then produces a calibrated ranking over the candidate responses. The system includes: A multi-stage data pipeline with human preference annotations Dimension-wise scoring for interpretability Automatic signals for correctness and safety A composite training objective with listwise ranking, regression calibration, and a safety-aware penalty Theoretical analysis that provides improved ranking error bounds and probabilistic safety guarantees Practical deployment considerations such as latency budgets, uncertainty thresholds, and fallback behavior Across internal benchmarks that include code generation, factual question answering, and open-ended chat, the Meta-Judge achieves: 85.4 percent agreement with human preferred answers More than 20 point improvements in correctness compared to heuristic routing 98.9 percent safe response selection, outperforming both single LLM strategies and prompt-based judges This framework provides a model-agnostic and safety-aware arbitration layer suitable for organizations that operate diverse LLM fleets in real-world environments.