Automated QA framework for PetaScale data challenges

Over the lifetime of the STAR Experiment, a large investment of workforce time has gone into a variety of QA efforts, including continuous processing of a portion of the data for automated calibration and iterative convergence and quality assurance purposes. A rotating workforce coupled with ever-increasing volumes of information to examine led to sometimes inconsistent or incomplete reporting of issues, eventually leading to additional work. The traditional approach of manually screening a data sample was no longer adequate and doomed to eventual failure with planned future growth in data extents. To prevent this collapse we have developed a new system employing user-defined reference histograms, permitting automated comparisons and flagging of issues. Based on the ROOT framework at its core, the front end is a web based service allowing shift personnel to visualize the results, and to set test parameters and thresholds defining success or failure. The versatile and flexible approach allows for a slew of histograms to be configured and grouped into categories (results and thresholds may depend on experimental triggers and data types) ensuring framework evolution with the years of running to come. Historical information is also saved to track changes and allow for rapid convergence of future tuning. Database storage and processing of data are handled outside the web server for security and fault tolerance.