Curation at the point of measurement and traceability of measurement workflows

In this paper we introduce a method that can digitally capture machine actionable metadata, tag them to the associated measurement data, and upload to a curated database. Our method is packaged as a tool to enable scientists to capture and store curated data at the point of measurement. By ‘data’ we include the primary measurement and any associated information such as calibration data, processing/analysis scripts, multi-modal data, etc. Combining the associated data together enhances re-usability through metadata and confidence though calibration data. We extend this process by adding new data at each stage of the data capture and analysis workflow to develop a completely traceable data processing pipeline. We achieve this by cumulatively updating the ‘data’ at each stage and by using versioning in our database for complete generality. Here each version is a self-contained curated container of all relevant data and codes providing a reproducible ‘snapshot’ in a traceable analytical pipeline. Within each ‘snapshot’ we store the outputs from the relevant data analysis (figures, models, hypothesis tests, etc), the raw data, and each step (codes, converters, etc) between them, resulting in a fully transparent and reproducible workflow. The ‘snapshots’ are updated along the analytical pipeline and we demonstrate this with several steps including: at the point of measurement; conversion to an open format; pre-processing (feature selection, noise reduction, etc); and data analysis. We demonstrate our method with a large cohort of mass spectrometry imaging experiments as an exemplar case study.