Validation study of end-of-life risk prediction model in active chemotherapy patients.

e13543 Background: In February of 2020, Integra Connect (IC) released a machine learning (ML) risk prediction model called Integra Predict (IP) to predict End-of-Life (EOL) in active chemotherapy patients within the next six months. Pilot Study: Regional Cancer Care Associates (RCCA) physicians rated of likelihood of EOL in the next six months to compare concordance with the IP model predictions. Part 2 (April 2022): Patient date of death updated at the end of the six-month period will undergo post-hoc comparisons. Physician Prediction vs Actual Outcome. Integra Prediction vs Actual Outcome. Methods: A stratified random sample of 265 active chemotherapy patients from 33 RCCA providers was securely emailed via pdf documents or excel sheets to RCCA for each provider. Worksheets contained provider name, patient name, age, cancer type, and the risk of dying in the next six months (Low, Medium, High). The sample included patients from the top 10 cancer types: breast, carcinoma in situ of breast, chronic leukemia, gastro/esophageal, lung, lymphoma, multiple myeloma, pancreatic, prostate, and small intestine/colorectal cancers. The sample was comprised of patients with active chemotherapy in the care of RCCA between July 2021 through September 2021. RCCA emailed the study documentation to each provider and asked providers securely emailed risk assessments to IC. Provider responses for EOL in the next six months were scored categorically - low risk (1), medium risk (2), and high risk (3). Concordance between provider responses and IP risk predictions was determined. E.g., a provider rating of low compared to a risk model rating of high would result in a difference of 1-3 = -2. Results: Forty-two percent (14/33) providers responded to our request to rate patient specific EOL which corresponds to 40% (106/265) patients sampled. Current results have a large standard deviation (0.235) resulting in a large confidence interval (.175, .64) or (18% to 64% concordance). Ninety-five percent of the time concordance will fall in this range. Table: As displayed in Table, provider estimates matched the IP risk score estimates 41.3% of the time. The average concordance between provider risk estimate and IP risk estimate was -0.7, which is almost one risk level lower, range (-1.4, -0.3). Conclusions: Study hypothesized that risk estimates would not be 50% concordant at an alpha of 0.05. 41% are concordant with a large standard deviation of 0.235, which resulted in overlapping confidence intervals. There is no significant difference in the two risk estimates.