The Impact of Healthcare Professionals’ Characteristics on the Evaluation of Clinical Decision Support Systems: Insights from a Cross-Country Usability and Technology Acceptance Study of the iCARE Tool

The development of clinical decision support systems (CDSS) is a complex process requiring early healthcare professional (HCP) involvement. However, engaging HCP is resource-intensive, as reliable assessments often require large participant groups and multiple evaluation rounds. Additional challenge is that HCPs are not a homogenous group, and their characteristics may influence the results that should be considered when designing evaluation studies of CDSS. This study investigated the relationship between HCP characteristics and the ratings of usability and technology acceptance when evaluating CDSS with an AI module. We hypothesize that HCPs who are open to adopting new technology, comfortable with technology, have a higher level of education and are younger will provide more satisfied ratings. The study was conducted with 139 HCPs from seven countries. The evaluated CDSS (iCARE tool) was designed to assist in making treatment decisions for older, multi-morbid patients in home and nursing home settings. HCPs were tasked with deciding whether to continue or discontinue pharmacological and non-pharmacological treatments with the aid of an AI module in the CDSS. Finally, the HCPs completed post-questionnaires evaluating usability and technology acceptance of the CDSS. Data were analyzed using two approaches: statistical modeling (hypothesis-driven) and clustering (data-driven). The statistical models tested the pre-defined hypotheses. Clustering identified participant profiles linking HCP characteristics with the ratings of usability and technology acceptance without any pre-defined hypotheses or assumptions. HCPs who were more open to adopting new technologies or who felt more comfortable with technology evaluated the perceived usefulness of the CDSS higher. Cluster analysis found that prior experience with predictive technologies like the one evaluated in this study was a moderating factor to the satisfaction of the CDSS under evaluation. These findings can inform the design of CDSS evaluation studies, particularly in selecting participant groups to ensure meaningful and representative assessments.