Search for a command to run...
<b><i>Evaluation Summary Brief</i></b><br /> <br /> <b>1.Background to the NIHR GHR Portfolio</b><br /> There has been a growing effort across the global health research community to build applied research and innovation capacity in low- and middle-income countries (LMICs), nurture and develop research talent, and maximise research impact to address health challenges specifically faced by LMICs. To complement efforts from existing Overseas Development Aid (ODA) research programmes following the 2015 UK Aid Strategy, the Department for Health and Social Care (DHSC) and the National Institute for Health and Care Research (NIHR) established the Global Health Research (GHR) Portfolio in 2016 to support applied health research and training in LMICs, to address unmet needs and strengthen research capabilities.<br /> <br /> By the end of the first phase (2016/17-20/21), the GHR Portfolio had evolved to include 30 distinct programmes which funded awards for individual researchers or consortia of researchers and institutions in LMICs and the UK. Some programmes are directly managed by NIHR and some are managed by Partners. This included a range of initiatives focused on career development, training and research opportunities which complement the capacity-strengthening objectives that are embedded as a key principle across all programmes. Further, NIHR’s strong emphasis on community engagement and inclusion (CEI) and equitable partnerships as core principles of funding has been instrumental in encouraging greater engagement with these approaches among award holders who applied for research funding. <br /> <br /> The GHR Portfolio's Theory of Change (ToC) (see Figure 1) outlines the NIHR’s ambition to improve global health outcomes. The ToC is presented in the form of a flow diagram that links inputs, activities, outputs, outcomes and impacts. It reflects the reality that it may take 3 to 10 years for research outputs to influence policy, practice, and behaviour changes and 10-25 years for these changes to lead to strengthened health systems and increased capacity for health promotion and disease prevention.<br /> <br /> <b>2. The Evaluation</b><br /> The DHSC commissioned Ecorys in December 2021 to undertake an evaluation of NIHR GHR’s first phase. The evaluation was delivered in four stages: <i>inception, interim evaluation, final evaluation</i> and a <i>dissemination phase</i>. It used a theory-based approach to assess the contributions of the GHR Portfolio to the intended outputs, outcomes, and likely impact of investments, including whether work was on track to deliver the expected results given the long-term nature of the anticipated research impact. The evaluation process involved testing the GHR Portfolio's ToC, including its assumptions and causal links, to determine the validity of the underlying theory using a contribution analysis<i><i><b>[1]</b></i></i> (CA) approach.<br /> <br /> The key methods comprised a document and data review of GHR Portfolio, programme and award documentation; interviews with stakeholders at GHR Portfolio, programme and award levels; and an online survey disseminated to all funded awards. Social Network Analysis (SNA) was conducted to assess networks and interactions between funded institutions, as well as Bibliometric Analysis (BA) to examine research outputs and collaboration. A purposive sampling approach to programmes and awards for in-depth review provided rich learning and insights at both portfolio and award levels.<br /> <br /> <b>3. Findings</b><br /> The evaluation questions were organised around the Organisation for Economic Co-operation and Development's Development Assistance Committee (OECD-DAC) evaluation criteria<i>: Relevance, Coherence, Effectiveness, Efficiency, Adaptability and Learning, Impact and Sustainability</i>. The evaluation also included questions related to <i>Adaptability and Learning, CEI, and Value for Money (VfM).</i> Overall findings from the evaluation are presented below.<br /> <br /> 3.1 <b>Overall findings by evaluation questions</b><br /> <br /> <b>Relevance: To what extent is the GHR Portfolio addressing priority areas of health research in LMICs where there are unmet needs identified by government and/or civil society in the relevant countries?</b><br /> The GHR Portfolio has been successful in ensuring that its investments address priority areas in health research within LMICs. The portfolio rapidly evolved from an initial focus on non-communicable diseases to wider research in infectious disease, including in support of COVID-19 response. This has been facilitated through valuable partnerships that supported investment in a range of themes and geographies. Health inequalities were given priority, and efforts were made to understand the underlying causes of these disparities. Unmet needs were identified through consultations with UK and international partners experienced in global health research and through calls for researchers from UK and LMIC Higher Education Institutions or Research Institutes, who understood neglected thematic areas. In some programmes, a researcher-led approach facilitated this.<br /> Researchers and LMIC country stakeholders were consulted during the design and development of the portfolio, and NIHR plan to enhance engagement to further promote sustainability and impact, particularly in addressing gender, equity, and ethical considerations. The involvement of government, policy makers and civil society actors representing marginalised populations increased over time, particularly where researchers have obtained subsequent awards to continue and expand their research. NIHR’s ongoing efforts to fund more LMIC-led research partnerships demonstrate further potential for supporting local priorities and embedding research in local systems.<br /> <br /> <b>Coherence: To what extent is the GHR Portfolio a coherent funding mechanism to meet its stated outcomes?</b><br /> Given the scale of unmet health needs in LMICs, internal and external coherence is crucial for optimising the GHR Portfolio’s impact on improving health in LMICs. DHSC has successfully designed the GHR Portfolio so that it complements existing government research funding mechanisms, as well as other UK and global health funders. DHSC and NIHR have successfully coordinated and collaborated externally with other UK, partner country and global health research initiatives. At portfolio level, proactive and ongoing engagement with other UK funders has helped strengthen and facilitate strategic collaboration and operational coherence across key ODA partners.<br /> NIHR has increased thematic and operational complementarity between NIHR-led and Partner-led programmes over time, with many programmes aligned with DHSC’s results framework to harmonise reporting. Efforts to increase internal harmonisation between programmes and approaches have strengthened over time.<br /> The GHR network has successfully and actively established global health collaborations with a wide range of institutions within the global health research community, including many LMIC institutions and other key stakeholders in global, regional, national and local contexts. Within this, there are large communities of institutions within the overall GHR network, presenting strong opportunities for collaboration and the exchange / dissemination of knowledge in NIHR-led programmes, in particular. The stronger, smaller, and concentrated number of community groups among NIHR-led awards indicates that institutions funded directly by NIHR may be more closely related in research interests, expertise, or geographic proximity, and have better opportunities for cross-collaboration and knowledge exchange within the NIHR portfolio.<br /> <br /> <b>Effectiveness: How effective has the GHR Portfolio been in achieving its intended results?</b><br /> The GHR Portfolio has funded programmes and awards that have made significant progress in producing high-quality, policy-relevant research outputs, expanding research capacity, and establishing equitable partnerships, and promoting CEI. During the first phase, it delivered 3,494 publications with a notable citation count (86,161).<b> </b>There is strong evidence of the GHR Portfolio’s success in individual research capacity strengthening, including supporting the career progression of researchers. Strengthening the capacity and career development of female researchers is progressing, as is the development of institutional capacity among LMIC research partner institutions. However, evidence of system-level capacity changes is more limited at this stage.<br /> The GHR Portfolio has made good progress in promoting equitable partnerships. While the majority of institutions in the GHR Portfolio are from LMICs, UK institutions tend to dominate in terms of their importance, influence, ability to connect with other influential actors, and play a significant role in connecting other institutions to NIHR or partner funding. due to challenges that remain in relation to power dynamics within partnerships and the wider health research funding landscape, addressing barriers to equitable participation of LMIC researchers is a complex task beyond the scope of individual awards.<br /> CEI is a core commitment for NIHR, with effective examples at the award level. There is scope and interest within NIHR to increase understanding of how communities experience CEI initiatives with a deeper focus on gender and marginalised groups. Effective initiatives involve funding researchers embedded in communities and health systems, with successful models incorporating centres of excellence to build local and regional capacity through collaboration. <br /> <br /> <b>Efficiency: Has the GHR Portfolio and its delivery partners been able to convert inputs into outputs in a timely and effective way?</b><br /> The GHR Portfolio is developing a VfM framework. The evaluation focused on qualitative evidence to assess the extent to which operational structures and processes support timely and effective delivery. The findings indicate that overall, structures and processes facilitated by DHSC and partners, have successfully delivered outputs despite challenges posed by the COVID-19 pandemic.<br /> Leveraging partners’ expertise in LMICs, within awards, and involving key stakeholders (including policymakers and communities) early in the research process enhanced operational effectiveness and efficiency. Delivery partners’ relationships, expertise, and systems in global health research funding overall benefitted the GHR Portfolio. Further efficiency and continuous improvement of approaches would be facilitated by enhancing DHSC staff’s LMIC in-country experience and increasing resources for public engagement, knowledge exchange, and dissemination. Approaches to developing and monitoring CEI and equitable partnerships efforts at the award level could also be enhanced, as well as increased mechanisms for feedback loops for LMIC award holders on management practices.<br /> <br /> <b>Adaptability and learning: How well is the GHR Portfolio adapting and embedding learning?</b><br /> NIHR’s approach to iterative learning has been beneficial in adapting the GHR Portfolio to the need for rapid allocation of funding during Phase 1, as well as to the unique challenges posed by the COVID-19 pandemic and operational delays. It allowed for flexibility, as evidenced by the number of awards granted increased time (no cost extensions) for delivery. Thematic learning on specific health topics and undertaking research in different contexts/ settings from across programmes and awards was more limited, and there is variation in learning practices within programmes.<br /> Monitoring, evaluation and learning (MEL) at the Portfolio level has developed over time, with an increase in coverage of Annual Reviews, Programme Completion Reviews and After Action Reviews along with improved systems for data collation and sharing during Phase 1. Further resources dedicated to MEL would support a more comprehensive, strategic approach to measuring outcome level results.<br /> <br /> <b>Impact: Is there any early evidence that funded research and capacity-strengthening activities are on track to/have the potential to contribute towards 3-10 year anticipated impacts?</b><br /> The impacts of the GHR Portfolio outlined in the ToC are expected to be observed from 10-25 years from the start of the funding in 2016/17. At this stage, the Portfolio has contributed to medium-term outputs and outcomes in line with the desired pathways of change. Most assumptions about how activities will support longer term global health outcomes are holding, with the enabling factors of follow-on funding and LMIC award leadership requiring monitoring and potentially mitigation.<br /> The GHR Portfolio's research and capacity-strengthening activities to date demonstrate the potential to influence health policy and practice and strengthen health systems in LMICs, especially where research agendas are sustained. There are early signs that where awards are at a more mature stage, they have begun to successfully raise awareness of research topics and influence access to research findings among policy makers, practitioners, and the public in LMICs. Success is attributed to building networks and structures for meaningful engagement with stakeholders. The GHR Portfolio has taken steps to address important considerations for creating an environment for influencing policy and practice, such as CEI, equitable partnerships, and coordination with other stakeholders. A Portfolio-wide approach and systematic approach to developing these activities and monitoring performance in these areas would support longer-term impacts. <br /> <br /> <b>Sustainability: To what extent will the net benefits of the GHR Portfolio continue, or likely continue, beyond the funded period?</b><br /> There are examples at both the programme and award levels that showcase research impact and gains in individual capacity strengthening, and in turn their contributions to wider health systems that have the potential to be sustained beyond the funding period. Linkages and partnerships supported through award funding contribute to sustainability by enabling further collaboration and funding opportunities. However, issues with availability of subsequent implementation funding for continuing and expanding the research funded by NIHR GHR undermine the potential for long-term gains, as first awards often only address initial research needs. Due to the early stage of the GHR Portfolio and the long timescale required for sustainability effects to materialise, data and insights on sustainable net benefits are also limited although there are positive signs of progress towards this being achieved.<br /> <br /> 3.2 <b>Social Network Analysis (SNA) </b><br /> The evaluation used SNA to supplement the evaluation questions on aspects related to the effectiveness of networks funded by the GHR Portfolio, whether there is coherence with other health research funders, and the approaches that the networks use to disseminate evidence and improve their accessibility. The SNA provides supporting evidence most notably against <i>Coherence</i> (the extent to which effective partnerships are formed or expanded through research funded by the GHR Portfolio) and <i>Effectiveness</i> (the extent to which opportunities for knowledge exchange are likely to be created). Findings from the SNA are presented below.<br /> <br /> The GHR network, representing the reach of NIHR funding as per available data, shows that NIHR and its partners have reached 1,158 institutions from 108 countries, with 72% of institutions in the GHR Portfolio network from LMICs compared to 28% of non-LMIC institutions<i><i><b>[2]</b></i></i>. When considering the primary network funded directly by NIHR and its partners through awards, it has reached 770 institutions. The most commonly represented countries of institutions in the GHR Portfolio network include the United Kingdom (11%), followed by Kenya (6%), India (6%), Uganda (5%), Nigeria (4%) and the United States (4%). While the majority of institutions in the GHR Portfolio network are from LMICs (72%), evidence suggests that UK institutions tend to dominate in their importance, influence, ability to connect with other influential actors, and play a significant role in connecting other institutions to NIHR or partner funding.<i><i><b>[3]</b></i></i> Figure 2 provides a visual representation of connections between institutions within the portfolio network.<br /> <br /> 3.3 <b>Bibliometric Analysis </b><br /> The evaluation conducted a BA as part of the final evaluation phase to explore the reach and impact of including publications and other research The BA supported a key of the of the the and policy of research outputs through performance and citation as well as insights into equitable partnerships through The evaluation identified 3,494 publications relevant to the GHR Portfolio which formed the for the from the BA are presented below.<br /> <br /> The evaluation that are the most with and being in which publications are are supported by funders. The NIHR and DHSC are funders of these often in A significant of the research is citation to be on whether they are more or access (see The majority of are with in and middle-income This the United United and among There are only a and middle-income countries in the 30 countries, including and /> <br /> The publications a wide range of thematic with notable focus on health and delivery of There is significant with Development especially Health and There was a significant in especially during the COVID-19 which have with some being Research a number of with a of per In terms of Global in the UK has a in first and /> <br /> <br /> and /> <br /> The NIHR GHR Portfolio has successfully delivered research activities and established as a and significant in the GHR in the UK and the international The GHR Portfolio activities have to priority needs of health research in LMICs, the research capacity of and institutions in the UK and LMICs, equitable partnerships, and the of community engagement as an part of global health research. progress with the GHR Portfolio’s longer-term expected contributions to changes in policy and practice, strengthened health systems and improved health outcomes. from this evaluation are designed to of the The were with the DHSC and NIHR to support and of /> The and are as GHR Portfolio’s programmes are to priority and health research areas in LMICs. with a range of partners, the portfolio was able to and and support a wide range of themes and geographic areas. portfolio is and a more focused approach with greater collaboration within and across programmes in the GHR Portfolio and beyond would greater potential to and support the strategic NIHR continue to the GHR and there is a need for a more focused such as potentially on countries, and LMIC where progress has been This be in with key funding partners with emphasis on the most beneficial funding partnerships and ensuring ongoing /> <br /> The GHR Portfolio has in a of research publications and many outputs at policy and practice on the in health The and of engagement with LMIC researchers and other LMIC stakeholders in this process has been increasing and could be further improved and towards the for wider policy and changes in health /> <br /> policy mechanisms in the research design phase to the policy environment is ensuring the stakeholders are involved in and outputs are likely to be and to scale NIHR also more support for policy-relevant outputs and to support policy potentially with a external /> <br /> Some award holders are with of and It is crucial for this practice to be embedded to that supported research is aligned with including with the of There is a need for improved on ethical as well as NIHR’s on promoting health /> <br /> research in develop specific around the ethical considerations of research in and The support learning on understanding and to health research to and ensuring that research and benefits communities in /> <br /> The GHR Portfolio is research capacity and progress towards equitable research partnerships. contribution also evidence of the GHR Portfolio’s contribution to longer term outcomes including improved policy and in GHR Portfolio level MEL capacity and systems would support the ability to track overall portfolio /> <br /> Strengthening GHR Portfolio level and strengthen its MEL and framework to with the a ToC and systems and processes for and using the results of all GHR Portfolio NIHR support awards to better understand and from the extent to which their equitable partnerships, CEI and approaches are supporting This will expertise and /> /> There is significant learning from the first phase, which has supported including during are opportunities for the commitment to learning to be to support systematic learning for and from award holders and programme across the GHR /> <br /> in strategic learning and knowledge NIHR more in opportunities for strategic and knowledge exchange to enhance research impact and capacity strengthening, and further the CEI and equitable partnerships /> <br /> The NIHR is promoting LMIC leadership of global health research through on equitable partnerships, CEI and is in more and strategic approaches that have other funding partners, supported approaches at the award level, and learning from LMIC However, of the research and and and power dynamics is or /> <br /> its to awards to the need for on research and and inequalities into the implementation and monitoring of their including their policy equitable CEI and /> <br /> deeper understanding of CEI and support awards to CEI more across the research including across their monitoring and and sustainability approaches to better understand pathways of and good and could also explore for support to awards, and funding research focused on /> <br /> approach to the GHR Portfolio has collaboration and progress on equitable further to address barriers to equitable participation for LMIC partners would strengthen the /> <br /> LMIC research NIHR further support to LMIC research partners to operational challenges and build management capacity by and and considering funding for wider research implementation /> <br /> research for potential VfM the framework for VfM is being /> <br /> a VfM NIHR continue to develop an VfM framework and for all programmes and awards to track and assess the by This could involve adapting existing VfM used by other UK funders to of ODA /> <br /> NIHR that the GHR has and capacity processes and to the with a capacity review to better understand and operational /> <br /> <br /> <br /> <br /> The allowed for a systematic of the GHR Portfolio outputs and their potential contribution to the GHR Portfolio’s expected outcomes, as in the This qualitative data from programme and award level document and the survey as evidence for A was developed to ToC outputs against its outcomes and including an of the of evidence and of /> <br /> This both primary and networks identified using available /> <br /> including and were to this /> <br /> access are in which the all and related available for on the a or is by the other access is a where the to to individual