Evaluation of the CIHR Antimicrobial Resistance Research Initiative (AMRI)
Final Report

August 2024

At the Canadian Institutes of Health Research (CIHR), we know that research has the power to change lives. As Canada's health research investment agency, we collaborate with partners and researchers to support the discoveries and innovations that improve our health and strengthen our health care system.

Canadian Institutes of Health Research
160 Elgin Street, 9th Floor
Address Locator 4809A
Ottawa, Ontario K1A 0W9

Acknowledgements

Special thanks to all participants in this evaluation - survey respondents and key informant interview participants. Also, thank you to those who supported the evaluation: Simon Provençal, Paul Khayat, and Guillaume Roberge (Science-Metrix Inc.), staff from the Initiative Management and Institute Support (IMIS), Funding Analytics, Financial Planning and Advisory Services, members of the Antimicrobial Resistance Working Group and Evaluation Advisory Committee, the Institute Teams and Scientific Co-Leads from the Institutes of Infection and Immunity (III), Dr. Charu Kaushic, and Population and Public Health (IPPH) former Scientific Director Dr. Steven Hoffman.

The Antimicrobial Resistance Research Initiative Evaluation Team

Alice Ndayishimiye, Ellie Radke, Alison Croke, Jonathan Gilbert, Rachelle Desrochers, Jean-Christian Maillet, Michael Goodyer

For more information and to obtain copies, please contact: Evaluation@cihr-irsc.gc.ca.

Table of Contents

• Executive Summary
  • Program Overview
  • Evaluation Objective, Scope and Methodology
  • Key Findings
    • Relevance
    • Design and Delivery
    • Performance
  • Recommendations
• Overview of the AMRI
• About the Evaluation
  • Purpose and Scope
  • Evaluation Methodology
    • Evaluation Questions
    • Evaluation Approach
    • Limitations of this Evaluation
• Evaluation Findings
  • Relevance
  • Design and Delivery
  • Performance
• Conclusions and Recommendations
• Appendices
  • Appendix A: Figures
  • Appendix B: Methodology

List of Acronyms

Executive Summary

Program Overview

The Antimicrobial Resistance Research Initiative (AMRI) was launched in 2016 as a result of the 2015 federal budget announcement of $1.8 million in annual on-going funding for the Canadian Institutes of Health Research (CIHR) to support Antimicrobial Resistance (AMR) Research. The overarching goal of the AMRI is to support the federal government priorities detailed within the Antimicrobial Resistance and Use in Canada: A Federal Framework for Action and Federal Action Plan on AMR and Use in Canada, by promoting innovation through funding collaborative research efforts both domestically and internationally. AMRI’s vision is for Canada to increase stewardship, prevention and reduction of AMR and improve treatment of patients infected with antimicrobial resistant organisms through evidence-informed policies and health care practice across all levels in the health care system. The initiative focuses on four broad streams of funding opportunities: Point of Care Diagnostics; Joint Programming Initiative on Antimicrobial Resistance (JPIAMR); Network for Global Governance Research on Infectious Disease; and Priority Announcements.

Evaluation Objective, Scope and Methodology

The objective of this evaluation is to provide CIHR senior management with valid, insightful, and actionable findings regarding the following:

• Needs addressed by AMRI and the initiative’s alignment with CIHR and Government of Canada priorities;
• Effectiveness of the design and delivery of the initiative in supporting the achievement of intended outputs and outcomes; and
• Achievement of the initiative’s expected outputs, and immediate, intermediate and ultimate outcomes.

The AMRI evaluation covers the period from 2016-17 to 2020-21. This is the first evaluation of the initiative since inception in 2016. The evaluation was committed to as part of CIHR’s 2018-19 Evaluation Plan and designed to meet CIHR’s evaluation requirements to the Treasury Board of Canada Secretariat (TBS) under the 2015 program authorities, the Policy on Results and the Financial Administration Act.

Key Findings

Relevance

AMR remains a public health threat not only in Canada, but globally. If resistance were to reach 40% by 2050, the cumulative cost to the Canadian healthcare system is estimated to reach $120 billion. In addition to this projected burden, the capacity to conduct research in Canada is decreasing. The AMRI is inadequately funded to meet its intended objectives for AMR research in Canada. Despite insufficient funding, the evaluation found that the AMRI continues to address a demonstrated need to support AMR research by investing in priority areas. The evaluation also found that the AMRI could better address some areas, including the adoption of a One Health approach that considers human, animal, and environmental transmission, as well as Indigenous engagement and the needs of equity seeking groups across funding opportunities.

The AMRI is aligned with the federal government roles and responsibilities to protect the health of Canadians against disease threats of national concern, such as the threat of AMR. It is also aligned with CIHR’s mandate to support initiatives that will lead to the improved health of Canadians as well as a strengthened healthcare system.

Additionally, the AMRI’s objective is well aligned with the federal government’s AMR priority areas outlined in federal AMR documents and the Minister of Health’s 2019 and 2021 mandate letters to address the threat of AMR. Furthermore, the Initiative aligns with CIHR’s strategic priorities, particularly with Priority A: Advancing Research Excellence in All its Diversity, Priority B: Strengthen Canadian Health Research Capacity, and Priority E: Integrate Evidence in Health Decisions.

Design and Delivery

The design of the AMRI supports the achievement of intended objectives by aligning research calls with key Government of Canada AMR priority areas: surveillance, stewardship, and research and innovation. However, it is worth noting that there were concerns that JPIAMR funding calls favoured researchers with existing international collaborations. It was also suggested that the AMRI could more effectively meet its objective as a research network.

The delivery of the AMRI is effective, with clear roles and authorities and two Scientific Directors providing balanced scientific leadership. However, due to inadequate funding, challenges were encountered in meeting the Scientific Directors’ respective mandates. Also, given the nature and demand of AMR research at the national and international level, the initiative was reported to have a high administrative burden.

AMRI’s governance was found to be appropriate and effective. Given the initiative’s limited resources, it was suggested that an external advisory committee be established, as indicated in the program authority. This committee could provide guidance on how to maximize the funding.

Additionally, the evaluation found the cost-efficiency of the AMRI at 13.9% to be higher than CIHR at 5.3% over the same period which indicates inefficiencies. This is attributed to the relatively low grants and award expenditures combined with consistent direct salary expenditures and low non-salary administrative costs.

In terms of the COVID-19 pandemic’s impact on the design and delivery of the AMRI, it was found to be limited.

Performance

The AMRI is achieving its expected outputs of funding research grants in priority areas and shaping AMR priorities. The AMRI has funded grants addressing federal priorities across CIHR’s research themes but most notably, biomedical research and social, cultural, environmental and population health. However, the evaluation found that few researchers adopt a multisectoral One Health approach to AMRI funded research. Also, there is a lack of Indigenous representation and lack of diversity among AMRI researchers, with most self identifying as men and not belonging to any visible minority group.

The AMRI leadership has helped shape research priorities within Canada through development of Government of Canada AMR policy documents. Scientific leadership has also contributed to shaping research priorities internationally through partnership with JPIAMR and representing Canada on the Transatlantic Taskforce on Antimicrobial Resistance (TATFAR), the Global AMR Research and Development Hub and the Global Research Collaborations for Infectious Disease Preparedness (GloPID-R). Although, it was reportedly challenging to influence JPIAMR’s research agenda given the need for consensus among the other countries.

The AMRI is making progress on its expected immediate outcomes. It is contributing to increased capacity for AMR-related research among both Canadian and international trainees. The AMRI has also led to the establishment of Canadian and international research partnerships and collaborations that have contributed to the achievement of the AMRI project outcomes. Specifically, by enhancing the capacity for Canadian researchers to establish national and international interdisciplinary research collaborations. Researchers surveyed indicated that as a result of the AMRI, they were able to form collaborations with researchers within and outside of their discipline, and both in Canada and internationally (75%, n = 9). When examining AMRI funded research publications, bibliometric analysis indicates that 93% of publications are published with Canadian authors with different institutional affiliations and 44% are published with authors affiliated to an international institution.

Additionally, the Initiative is advancing knowledge through peer review publications. Between 2017 and 2021, there were 55 publications produced acknowledging AMRI funding. These publications were proportionally well-represented in open access journals and published in high impact journals at a rate higher than the global average. The AMRI has supported knowledge translation activities, most notably by being involved in the creation of JPIAMR-Virtual Research Institute (VRI) which provided a platform to facilitate knowledge exchange. Researchers reported producing various knowledge translation products such as publications, presentations, books, policy briefs, and social media products.

Despite its early stage of implementation, the AMRI is already making progress towards achieving its intermediate outcomes. There is emerging evidence indicating that AMRI funded research is having impacts beyond academia. Based on the altmetric analysis, AMRI publications (n = 55) are being shared across several media outlets at a rate higher than the global average. Two AMRI publications funded through Point of Care Diagnostics and Priority Announcement funding streams were cited by three patents. Another publication funded through JPIAMR was cited in a policy document commissioned by the Centers for Disease Control and Prevention and the Wellcome Trust.

Research funded through the Point of Care Diagnostics stream led to two new patents and the development of a spin-off company. International collaborations through JPIAMR funded research have also led to a spin-off company that has conducted research for Pfizer and collected venture financing to advance research efforts. It has also received further funding from the Gates foundation. Additionally, JPIAMR funded research has led to the identification of lead molecules that are under further development as preclinical candidates.

Most AMRI researchers reported being negatively impacted by COVID-19, citing the following impacts: delayed research progress, reduced lab access, networking challenges and a shift in research funding priorities to COVID-19 research. The pandemic has also created an opportunity to build on lessons learned in responding to AMR.

Recommendations

The evaluation makes five recommendations to improve the performance of the AMRI to achieve its expected results.

Recommendation 1:

CIHR should continue to invest in priority-driven AMR research through the AMRI and assess the level of funding to achieve AMRI expected outcomes and support the federal government AMR research priorities.

Recommendation 2:

CIHR should be guided by a One Health approach for the AMRI that recognizes the interconnectedness of humans, animals and the environment within health research to address the threat of AMR.

Recommendation 3:

CIHR should engage with other federal research funding agencies to better support a One Health approach to AMR research.

Recommendation 4:

CIHR needs to embed equity, diversity and inclusion considerations as well as engagement with Indigenous communities into all aspects of the AMRI and, by extension, AMR research in general.

Recommendation 5:

CIHR should consider a mechanism for external independent advice on AMR Research Priorities as outlined in the program authorities.

Overview of the AMRI

Program Description

The Antimicrobial Resistance Research Initiative (AMRI) is a strategic intersectoral research initiative funded by the Canadian Institutes of Health Research (CIHR) to promote innovation and collaborative research on antimicrobial resistance (AMR) both domestically and internationally. While AMR has been a key research priority for the CIHR since 2000, the AMRI was formally created following Canada’s 2015 Budget announcement of $1.8 million in annual grant funds. The AMRI was implemented in 2016 and led by the CIHR Institute of Infection and Immunity (III) and the CIHR Institute of Population and Public Health (IPPH)^{Footnote 1}.

The vision of AMRI is for Canada to increase stewardship, prevention and reduction of AMR and improve treatment of patients infected with antimicrobial resistant organisms through evidence-informed policies and health care practice across all levels in the health care system. Broadly, the AMRI aims to improve treatment, care, and quality of life for patients infected with AMR organisms and reduce the inappropriate consumption of antibiotics.

AMR research funded by CIHR is intended to align with the priorities (Surveillance, Stewardship, and Innovation) described in the Government of Canada’s Antimicrobial Resistance and Use in Canada: A Federal Framework for Action (2014)^{Footnote 2} and the Federal Action Plan on AMR and Use in Canada (2015)^{Footnote 3}. These two documents were released to map out a coordinated collaborative federal approach to responding to the threat of AMR. In addition to these documents in 2017, the Government of Canada released the Tackling Antimicrobial Resistance and Antimicrobial Use: A Pan-Canadian Framework for Action to coordinate efforts across all levels of government and sectors and to fulfill Canada’s 2015 World Health Assembly commitment, focusing on four key pillars: surveillance, infection prevention and control, stewardship, and research and innovation^{Footnote 4}.

Recently, the Government of Canada released the Pan-Canadian Action Plan on AMR (Action Plan) in 2023, which outlines a 5-year (2023-2027) blueprint for strengthening Canada’s collective AMR preparedness and response across the One Health spectrum, focusing on the four pillars listed above and Leadership as an additional pillar^{Footnote 5}.

Funding Opportunities

The AMRI supports research across national and international funding opportunities through four broad streams: Point of Care Diagnostics, the Joint Programming Initiative on Antimicrobial Resistance (JPIAMR), the Network for Global Governance Research on Infectious Disease, and Priority Announcements.

Point of Care Diagnostics

The Point of Care Diagnostic funding supports research teams in the development, evaluation, or implementation of point-of-care diagnostic tools for viral/bacterial discrimination or identification of specific priority pathogens. In phase 1, CIHR provided $1.4 million over two years starting in 2017 to five research teams. In phase 2, CIHR provided up to $2 million over three years starting in 2019 to three teams to facilitate the uptake of AMR projects to commercialization, direct application, or equitable implementation in a healthcare setting.

The Joint Programming Initiative on Antimicrobial Resistance

CIHR has been partnering with the JPIAMR to encourage international partnerships for Canadian AMR researchers. In fact, CIHR is a member and major funder of the JPIAMR; a collaboration of 29 nations aimed at coordinating research in AMR to achieve long-term reductions in resistance levels and better public health outcomes. Since the beginning of AMRI, CIHR has been involved in the following JPIAMR calls:

• Joint Programming Initiative on Antimicrobial Resistance Prevention and Intervention Strategies to Control AMR Infections;
• Building the Foundation of the JPIAMR Virtual Research Institute;
• Joint Programming Initiative on Antimicrobial Resistance Diagnostics and Surveillance of Antimicrobial Resistance: Development of tools, technologies and methods for global use;
• Joint Transnational Call for Network Plus within the joint programming initiative on antimicrobial resistance, JPIAMR Network Plus 2020;
• Joint Transnational Call for Antimicrobial Transmission Interventions on One Health interventions to prevent or reduce the development and transmission of AMR; and,
• Joint Transnational Call for Antimicrobial Transmission Interventions on Disrupting drug Resistance Using Innovative Design.

The Network for Global Governance Research on Infectious Disease

The Catalyst Grant on Global Governance AMR and Related Infectious Disease Threats preceded the inception of the Network for Global Governance Research on Infectious Diseases. The objective of the opportunity was to help build the foundation for a future possible international network of research centres focused in this area of research. CIHR funded two grants for one year for a total of $200,000. The Network for Global Governance Research on Infectious Diseases is expected to facilitate the establishment of a Canadian research network focused on a comprehensive, interdisciplinary, and intersectoral approach to global governance of infectious diseases and AMR and to enable better preparedness for infectious diseases through the application of knowledge, sharing of lessons learned, and the creation of improved governance arrangements. CIHR funded one grant for a total of $1.8 million over four years.

Priority Announcements

Priority Announcements were additional sources of funding for highly ranked applications that were relevant to AMR. CIHR funded three grants for one year for a total of $300,000.

Activities

AMR is a shared and complex responsibility in Canada that requires full collaboration across federal departments and agencies, as well as the engagement of all provinces as territories, to address effectively. To this end, the Public Health Agency of Canada (PHAC) has been coordinating the Government of Canada activities on AMR such as developing policy documents as well as establishing federal/provincial/territorial (F/P/T) and interdepartmental committees. Through III, IPPH and Initiative Management and Institute Support Branch (IMIS), CIHR contributes and supports PHAC in national activities and represents Canada in many international activities related to AMR. Through these activities, CIHR is able to shape AMR priorities that reflect its mandate and commitments, as well as help ensure funded research is aligned with Government of Canada and global priorities. The following outlines the key national and international documents and initiatives in the area of AMR:

• National:
  • Antimicrobial Resistance and Use in Canada: A Federal Framework for Action (2014): The Framework maps out a coordinated, collaborative federal approach to responding to the threat of AMR.
  • Federal Action Plan on Antimicrobial Resistance and Use in Canada (2015): The Action Plan builds on the strategic areas of focus and priority action items outlined in the Framework by identifying concrete steps that will be undertaken by the PHAC, Health Canada, the Canadian Food Inspection Agency, CIHR, Agriculture and Agri-Food Canada, the National Research Council, and Industry Canada.
  • Tackling Antimicrobial Resistance and Antimicrobial Use: A Pan-Canadian Framework for Action (2017): The Framework’s goal is to strengthen Canada’s ability to combat the risks of AMR in a coordinated, multisectoral and effective manner.
  • Pan-Canadian Action Plan on Antimicrobial Resistance (2023): The Action Plan was developed in collaboration with F/P/T partners, and responds to calls to action from industry, academia, and other partners across One Health sectors to support implementation of the Pan-Canadian Framework on AMR. The CIHR-III Scientific Director was the co-chair of the Research and Innovation group involved in the development of the Pan-Canadian AMR Action Plan. IMIS provided secretariat support for the Research and Innovation group and CIHR’s VP Research was a member of the Federal AMR Steering Committee.
• International:
  • JPIAMR-VRI is led by CIHR, through III. The JPIAMR-VRI is a virtual platform that was built to connect research networks, research performing institutes, centres and infrastructures beyond sectorial and geographic boundaries in a larger global network under JPIAMR topics, using a One Health approach. By connecting the global scientific community along the six pillars of the joint Strategic Research and Innovation Agenda, the JPIAMR-VRI aims to provide an unprecedented level of knowledge exchange, facilitate the analysis of knowledge gaps, increase capacity, improve coordination, implement breakthrough collaborative research and increase the visibility of the research performed.
  • Global AMR Research and Development (R&D) Hub was launched by the G20 Health Working Group under Germany’s leadership in the field of AMR research and product development. This hub is aimed at maximizing the impact of existing and new initiatives in antimicrobial basic and clinical research as well as product development.
  • TATFAR (Transatlantic Taskforce on Antimicrobial Resistance) was created in 2009 with the collaboration across government agencies from Canada, EU, Norway, and U.S to address the urgent threat of AMR in key focus areas such as improve antibiotic use in humans and animals, prevent infections and their spread, and strengthen the drug pipeline.
  • GLoPID-R (Global Research Collaboration for Infectious Disease Preparedness) is a network of research funding organizations in infectious disease preparedness research. The goal of GloPID-R is to facilitate an effective research response to new or re- emerging infectious diseases with epidemic and pandemic potential, in order to save lives and economies worldwide. Dr. Kaushic is the Chair of GLoPID-R.

Resources

Between 2016-17 and 2020-21, the AMRI represents a total investment of $8.9 million dollars for CIHR, with an average annual investment of $2.1 million starting in 2017-18. As shown in Figure 1: CIHR Investments in AMR Research, of Appendix A, investments in AMRI represent an average of 7.6% of CIHR’s total investments in AMR between 2017-18 and 2020-21^{Footnote 6}. CIHR’s investment in the AMRI stems from four different sources of funding: an annual ring-fenced budget, supplemental investments from the Institute-Specific Initiative budgets of III and IPPH as well as from “other funding”, such as from the Environments and Health Signature Initiative (EHSI) (Figure 2: CIHR Investments in AMRI by Sources of Funding).

About the Evaluation

Purpose and Scope

The purpose of this evaluation was to provide senior management with independent, objective, and actionable findings regarding the:

• Needs addressed by the AMRI and its alignment with the mandates of CIHR and Government of Canada priorities;
• Effectiveness of the design and delivery of the program in supporting the achievement of intended outputs and outcomes; and
• Achievement of the program’s expected outputs, and immediate and intermediate outcomes.

By addressing these issues, the evaluation will help inform the co-lead Institutes and CIHR program management decision-making and planning regarding the AMRI. The evaluation is required to meet the evaluation commitments outlined in the 2015 program authorities and the evaluation requirements outlined in the Policy on Results and subsection 42.1 of the Financial Administration Act.

The evaluation of the AMRI was conducted by the CIHR Evaluation Unit and covers the period from 2016-17 to 2020-21. It is worth noting that where possible, the evaluation assessed linkages between the AMRI and CIHR’s broader investments in AMR research. In addition, the evaluation looked at the activities related to the Federal Framework and its Action Plan and the activities related to the launch of the research initiative, both of which may have occurred prior to 2016-17, as well as the release of the Action Plan in 2023. The publication of the Action Plan occurred late in the reporting phase of the evaluation, after the completion of data collection and analysis; therefore, the key findings in this report reflect views gathered prior to the Action Plan’s release. Further, with its release in 2023, the Action Plan did not guide the activities of the AMRI during the period under review; however, it has been considered in the context of the evaluation in relation to the on-going relevance of the AMRI.

As per Treasury Board requirements, a Performance Measurement Strategy for the AMRI was developed at the time of initiative launch. However, as the initiative’s implementation evolved within CIHR, the logic model was refined to reflect the AMRI’s focus on strategic funding opportunities with an overarching objective of establishing national and international partnerships. Despite the fact that this revised logic model was not fully approved by CIHR’s relevant governance committees (e.g., the Subcommittee on Implementation and Oversight), the evaluation questions and indicators focused predominantly on the revised logic model as it better reflects the implementation of the initiative to date (see Figure 3: AMRI Logic Model, Appendix A). Given the relatively early stages of implementation of the AMRI vis-à-vis the achievement of its intermediate and long-term outcomes, the evaluation focused on pathways to impact while mapping the AMRI activities and outcomes against the Government of Canada’s four pillars for action: surveillance, stewardship, infection prevention and control; and, research and innovation, as outlined in Tackling Antimicrobial Resistance and Antimicrobial Use: A Pan- Canadian Framework for Action. The Action Plan introduced leadership as a fifth pillar for action in 2023; however, this was beyond the scope and period under review of the current evaluation which necessarily focused on the four pillars established in the Framework.

Evaluation Methodology

Evaluation Questions

The evaluation was guided by three main evaluation issues to assess the relevance, the design and delivery as well as the performance of the AMRI. Given the unprecedented impact of the COVID-19 pandemic on the AMRI research landscape, including the possible shift in research activities by some researchers, the evaluation included questions regarding the impact of COVID-19 – current and prospective –on the initiative.

The relevance of the AMRI was assessed by examining the needs addressed by the program, its alignment with the federal government and the priorities of CIHR as well as alignment with the federal government and CIHR roles and responsibilities. The design and delivery issue was assessed by analyzing the extent to which design features of the AMRI facilitate the achievement of objectives, the effectiveness of the governance structure and oversight as well as the cost-efficiency of the initiative. The performance of the AMRI focused on the achievement of outputs and immediate outcomes and, where sufficient time has elapsed and/or data allowed, focused on intermediate outcomes.

The evaluation issues and questions are outlined below:

Relevance

• 1.1 To what extent does AMRI continue to address a demonstrated need?
• 1.2 To what extent is AMRI aligned with the federal government and CIHR roles and responsibilities?
• 1.3 To what extent is AMRI aligned with federal government and CIHR priorities?

Design and Delivery

• 2.1 To what extent do the design features of the AMRI facilitate the achievement of its objectives?
• 2.2 To what extent has AMRI been effectively and efficiently delivered?
• 2.3 To what extent has the AMRI design and delivery been impacted by the COVID- 19 pandemic?

Performance

• 3.1 To what extent has the AMRI achieved expected outputs?
• 3.2 To what extent has the AMRI made progress toward the achievement of expected immediate outcomes?
• 3.3 To what extent has the AMRI made progress toward the achievement of expected intermediate outcomes?
• 3.4 To what extent has the COVID-19 pandemic impacted intended outcomes of AMRI?

Evaluation Approach

The evaluation employed both quantitative and qualitative data collection methods and analyses. Consistent with Treasury Board of Canada Secretariat (TBS) guidance and recognized best practice in evaluation^{Footnote 7}, multiple lines of evidence were used to triangulate evaluation findings. This included document review, literature review, an analysis of administrative and financial data, bibliometric and altmetric analysis of AMRI funded research publications, a survey of grant recipients (n = 12) and applicants (n = 29), and key informant interviews (KII) conducted with CIHR management (n = 4), AMRI funded researchers (n = 6) and partners (n = 6).

Gender-based analysis plus (GBA+) and equity, diversity, and inclusion (EDI) considerations were built into the evaluation framework via specific evaluation indicators.

Given the limited number of lines of evidence with small population and sample sizes, the following qualifiers have been used to indicate the frequency of responses for key informant interviews and surveys. It is important to note that these qualifiers have been used in order to summarize statements about qualitative data; they should not necessarily serve as a measure of the importance of the respective finding.

Additional details about the methodology are provided in Appendix B.

Limitations of this Evaluation

It is common for evaluations to face limitations that can influence the validity and reliability of findings. The main limitations associated with this evaluation are:

• The AMRI’s population size of researchers and applicants was small which, in turn, resulted in small sample sizes for the researcher survey. Therefore, it was not possible to conduct statistical tests and only descriptive statistics were calculated. It is also possible that the data collected through KII may not have reached saturation due to the small number of interviewees in subgroups.
• Data obtained from the researcher survey and KII is self-reported by the respondents. Self-report data comes with its own biases and limitations (e.g., response bias, possible misinterpretation of questions), which can limit the accuracy, validity, and reliability of findings.
• The analysis of AMRI performance data was limited by the extent of end of grant reporting available for the period under review. For example, given the maturity of the initiative combined with the impact of the COVID-19 pandemic, a number of funded grants were not completed and had not yet submitted end of grant reports at the time of data collection.

The limitations and mitigation strategies are discussed in more detail in Appendix B.

Evaluation Findings

Relevance

Key Findings:

The AMRI continues to address a demonstrated need to support AMR research, however it is not sufficiently funded to meet its intended objectives for AMR research in Canada.

There is a clear and continued need to support AMR research in Canada. The World Health Organization has declared AMR as one of the top ten global public health threats facing humanity. If it is left unaddressed, it will have a devastating impact^{Footnote 8}.

In Canada, AMR was associated with an estimated 14,000 deaths in 2018. Of these deaths, 5,400 could be considered directly attributable to AMR itself. If no pre-emptive action is taken, the cumulative number of deaths due to AMR is estimated to reach 390,000 by 2050. In addition to the loss of life, there will be significant economic costs. By 2050, Canada’s cumulative Gross Domestic Product is predicted to decline by $388 billion and the cumulative cost to the healthcare system is estimated to reach $120 billion if resistance to first-line antimicrobials (those drugs that are the preferred first choice for use in treating infections) increases from the current rate of 26 percent to 40 percent^{Footnote 9}.

In face of the projected increase in the burden of AMR on individuals and society, Canada’s capacity to conduct AMR research is decreasing. The bibliometric analysis of Canadian AMR research conducted as part of this evaluation found that Canada produced 5,554 publications between 2011 and 2021, ranking 12th in global output; however, Canada’s AMR publications as a percentage of the world’s AMR publications declined steadily from 4.2% to 2.8% over the same period (Figure 4: Publication Output and World Share of the Top Publishing Countries in AMR Research (2011–2021)). Similarly, Canada’s specialization index in AMR research was at the world average of 1.00 between 2011 and 2013, then steadily declined to reach a low of 0.76 in 2021 (Figure 5: Canada’s Global Share of Publications and Specialization Index). It is important to note that although Canada is lagging in terms of output and specialization, Canadian AMR publications are cited at a rate higher than the world average (ranking 6th globally) and are published in journals with a high impact factor (ranking 5th globally).

The threat and burden posed by AMR as well as Canada’s declining AMR research capacity indicate a clear need to invest in AMR research. To this end, CIHR has invested $139.6 million in AMR between 2016-17 and 2020-21, with $101.2 million directed to investigator-initiated research; $30.8 million directed to research in priority areas, including AMRI; and $7.6 million directed to training and career support (Figure 6: CIHR Investments in AMR Research by Program Year).

Despite these investments, a review of the literature revealed that Canada is lagging behind in research and innovation investments compared to other high-income countries^{Footnote 10Footnote 11}. Similarly, all key informants (9/9) asked whether CIHR’s funding of the AMRI was adequate agreed that AMRI’s funding of $1.8 million annually is not sufficient for the Initiative to achieve its intended objectives and contribute to meeting the continued need for AMR research given the profound global threat posed by AMR in Canada and internationally.

Despite reporting that AMRI is not sufficiently funded, most key informants across respondent categories (11/14) agreed that the AMRI meets some of the needs for AMR research in Canada by strategically investing in key federal government priority areas of surveillance, stewardship, diagnostics, transmission, and governance. These findings are further supported by the researcher survey, which found that all recipients (100%, n = 12) and almost all applicants (88%, n = 34) agreed that AMRI funding responds to an important need for AMR research that would otherwise not occur. In fact, more than two-thirds of AMRI applicants (68%, n = 25) indicated that their proposed AMR research project did not proceed in the absence of funding, and just over half of AMRI recipients (58%, n = 7) reported that their project would not have proceeded had they not received AMRI funding. All key informants interviewed (12/12) indicated that they were not aware of any duplication of the AMRI with other national funding initiatives that support AMR.

While the AMRI is addressing some needs, there was evidence that some other needs were not well addressed. For example, many AMRI researcher and partner key informants (6/12) suggested that more AMR research could adopt a One Health approach to AMR, recognizing the interconnectedness of humans, animals, and the environment. Additionally, most researcher, partner and CIHR management key informants (12/16) highlighted that the AMRI was not explicitly addressing the needs of equity seeking groups and Indigenous community members across funding opportunities, despite knowledge that AMR disproportionately impacts marginalized groups (e.g., residents of low- and middle-income countries, Indigenous communities). For example, some researcher key informants (3/6) felt that current AMRI funding opportunities do not adequately support research on the impacts of AMR on Indigenous communities, with one (1/6) emphasizing that available funding opportunities are not long enough to allow researchers to establish meaningful relationships with Indigenous community members, despite Indigenous health have been identified as a priority in the 2014 Federal Framework for Action and Indigenous engagement being cited as a priority in the 2023 Pan- Canadian Action Plan.

The AMRI is aligned with the federal government and CIHR’s roles and responsibilities.

The AMRI is aligned with the federal government and CIHR’s roles and responsibilities. The Government of Canada is committed to taking action to prevent, limit, and control the emergence and spread of AMR. It is recognized that addressing this threat is a shared responsibility that will require collaboration across federal departments and agencies, hence, the release of the Federal Framework for Action in 2014^{Footnote 2}. This framework outlines a cohesive and collaborative approach across federal departments with mandates to address and mitigate AMR. To build on this, the Federal Action Plan on AMR and use in Canada^{Footnote 3} was released in 2015 with the aim of identifying specific steps that will be undertaken by federal departments and agencies to achieve the priority actions established in the Federal Framework for Action. In addition to the federal framework and action plan, the Pan-Canadian Framework was released in 2017^{Footnote 4}. The purpose of this document was to expand the roles and responsibilities beyond federal departments to include all stakeholders (federal/provincial/territorial governments and public and private sector partners, including professional associations, industry, academia and the public) who are needed in a One Health approach to tackle AMR. This framework marks Canada’s fulfilment of its commitment to the WHO Global Action Plan on AMR to develop a National Action Plan.

The Pan-Canadian Action Plan on AMR, released in 2023, is intended to act as a five-year (2023-2027) blueprint to coordinate an accelerated pan-Canadian response to address AMR through the implementation of ten priority actions with a focus on some of the key principles identified by respondents, including One Health, equity, collaboration, and momentum^{Footnote 5}. The Pan-Canadian Action Plan on AMR differs from its predecessors by including a fifth pillar – Leadership, a stronger focus on Indigenous engagement, and further reinforces the importance of taking a One Health approach to tackle AMR.

Despite the release of these federal AMR framework documents, some CIHR management and partner key informants (6/16) indicated that a more active coordinated approach from all levels of government and across sectors is required. Specifically, these key informants (6/16) reported that enhanced coordination would better support knowledge mobilization and identification of research gaps by knowledge users, allowing AMR research to be more targeted at addressing the needs of end-users. Some interviewees (6/16) also felt that it would prevent duplication of efforts and promote stronger collaboration. A few also suggested (3/16) that this coordination needs to take a One Health approach that recognizes the interconnectedness of humans, animals and the environment and aims to ensure a coordinated, collaborative, multi-disciplinary approach to address health risks that originate at the human-animal- ecosystems interface.

The importance of ensuring coordinated effort regarding AMR research also aligns with the Report of the House of Commons Standing Committee on Health on the status of AMR in 2018 that recommended the Government of Canada explore the possibility of funding a network of centres of excellence to address AMR^{Footnote 12}. In addition, as part of the federal government’s role in addressing the threat of AMR, the Standing Committee recommended that the Government of Canada provide stable and adequate funding to support research and innovation in AMR. This recommendation was in response to concerns by witnesses that Canada’s low investment in research and innovation has led to fewer trainees wanting to pursue a career in the field of AMR.

Further, CIHR’s role in providing support for AMR research through the AMRI is directly aligned with the CIHR Act (S.C. 2000, c6)^{Footnote 13}. The Act acknowledges the importance of supporting initiatives that will lead to the improved health of Canadians as well as strengthen the healthcare system, and among other objectives, aims at “addressing emerging health opportunities, threats and challenges and accelerating the discovery of cures and treatments and improvements to health care, prevention and wellness strategies.” AMRI’s vision closely aligns with and supports the Act’s objective regarding the importance of “pursuing opportunities and providing support for the participation of Canadian scientists in international collaboration and partnerships in health research.”

As lead on the Government of Canada’s research and innovation pillar, partner key informants suggested that CIHR would be well placed to play the role of a convener, by bringing together other relevant federal government agencies to advance AMR research.

The AMRI’s objective is well aligned with federal government AMR priority areas and CIHR strategic priorities.

The AMRI aligns with federal government priorities, as outlined in several key AMR policy documents, most recently the 2023 Pan-Canadian Action Plan on AMR. The Action Plan provides a 5-year (2023 to 2027) blueprint for strengthening Canada’s collective AMR preparedness and response with a focus on the One Health spectrum. The AMRI also aligns with the 2017 Pan-Canadian Framework which has an overarching goal of strengthening Canada’s ability to combat the risks of AMR in a coordinated and effective manner as well as to support Canada’s international commitments. It establishes the pillars of surveillance, stewardship, infection prevention and control, and research and innovation). Similarly, the initiative is well aligned with the 2014 Federal Framework for Action on AMR and the 2015 Federal Action Plan that outline actions to be taken by the federal government to address AMR.

The Government of Canada's commitment to address the threat posed by AMR is further evidenced by the priorities established in the Minister of Health’s 2019 and 2021 mandate letters. The 2021 mandate letter committed to “work with partners to take increased and expedited action to monitor, prevent and mitigate the serious and growing threat of antimicrobial resistance and preserve the effectiveness of the antimicrobials Canadians rely upon every day” as crucial in order to deliver on the Government of Canada’s priorities^{Footnote 14Footnote 15}.

The AMRI closely aligns with CIHR’s priorities outlined in both current and recent Strategic Plans in place during the period of the evaluation. Based on the recent Strategic Plan, A Vision for a Healthier Future (2021-2031)^{Footnote 16}. Specifically, the AMRI aligns with Priority A (Advance Research Excellence in All Its Diversity) through the support of strong research teams (A2), promotion of open science (A3) as well as enhancement of national and international collaborations (A4). Given the global threat posed by AMR, the AMRI is aligned with Priority B (Strengthen Canadian Health Research Capacity) by supporting enhanced training and career support (B3) and the establishment of international collaborations that enable rapid and evidence-based responses to emerging health threats (B4). Additionally, these international collaborations align with AMR research with Priority D (Pursue Health Equity through Research) by driving progress on global health research (D3) through AMRI’s international component, that supports participation of researchers from low and middle-income countries. Finally, the overarching objective of CIHR’s AMRI is to integrate innovative practices and novel therapeutics into the health care system directly aligns with Priority E (Integrate Evidence in Health Decisions), specifically by advancing knowledge (E1), developing evidence informed policies (E2) and ultimately strengthening Canada’s health system (E3).

The AMRI is also aligned with CIHR’s previous Strategic Plan for 2014-15 to 2018-19, Health Research Roadmap II: Capturing Innovation to Produce Better Health and Health Care for Canadians^{Footnote 17}, which emphasized the importance of enhancing patient experiences and outcomes through health innovations (Research Priority A). Similarly, AMRI has a focus on promoting innovative diagnostic tools that can be integrated into healthcare settings in order for patients to receive the right treatment and therefore improve health outcomes.

Design and Delivery

Key Findings:

The design of the AMRI supports the initiative in achieving its objectives.

The AMRI was launched in 2016-17 and was implemented as a series of funding opportunities across four major streams: JPIAMR, Point of Care Diagnostics, Network for Global Governance Research on Infectious Diseases and Priority Announcements. A review of the funding opportunity objectives and program documents revealed that all funding opportunities across all four funding streams align with key Government of Canada AMR priority areas: surveillance, stewardship, and research and innovation. Through clear alignment, AMRI’s design supports its overall objective to increase stewardship, prevention and reduction of AMR and improve health through evidence-informed polices and health care practices.

Specifically, the objectives and priorities (i.e., therapeutics, diagnostics, surveillance, transmission, environment, and interventions) of JPIAMR funding calls align both with AMRI’s objectives and federal government priorities. Most notably, the JPIAMR aims to support international collaborative action which is well aligned with the federal government’s innovation stream with the objective to promote innovation through funding collaborative research and development efforts on AMR both domestically and internationally. Similarly, Point of Care Diagnostics funding calls aim to support health services professionals in making decisions and assessments regarding choice of treatment in a timely and appropriate manner, which aligns with AMRI’s objective to improve health outcomes for patients through evidence-informed healthcare practices. The Network for Global Governance Research on Infectious Diseases funding calls align with AMRI’s objectives by leveraging research capacity in social sciences and policy research to foster national and international collaborations to enable better preparedness of emerging health threats, like AMR, through the promotion of the use of research results by relevant stakeholders. Finally, through the strategic use of Priority Announcements, the AMRI initiative provides bridge funding to high rated applications in CIHR’s project grant competitions that align to the AMRI’s objectives.

Overall, all key informants (16/16) were supportive of the AMRI’s design features, stating that funding across the initiative’s four major funding streams was aligned with the initiative’s objectives. Most key informants (8/13) stated that AMRI’s continued investment in the federal government priority areas of surveillance, stewardship, research and innovation will facilitate the achievement of AMRI’s objectives.

However, the views of CIHR management and partner key informants were mixed when it came to the value of AMRI’s specific design elements. Most key informants (7/13) specifically highlighted that investments in JPIAMR allow CIHR to maximize the AMRI’s limited budget and support synergy between Canadian and international researchers by leveraging the complementary research strengths of both parties. However, a couple interviewees (2/16) noted that JPIAMR funding calls favour researchers with existing international collaborations, thus deterring some AMR researchers from applying. Additionally, most researcher and CIHR management key informants (6/10) observed that the AMRI could more efficiently achieve its objectives if it were a coordinated research network, supporting greater collaboration, coordination across disciplines and accelerating knowledge mobilization.

The delivery of AMRI has been effective, with clear roles and authorities and a balanced approach in scientific leadership.

A review of program documents provided clarity on the roles and authorities guiding the delivery of the AMRI. The operational and administrative leadership is provided by CIHR’s IMIS Branch, with its Director General acting as the Program Official. The scientific directors of CIHR-III and CIHR-IPPH provide scientific leadership to the AMRI, which include ensuring that each of the four funding streams fund research that facilitates the achievement of the initiative’s objectives as well as the development of partnerships both nationally and internationally.

Overall, key informants (4/4) believed that the AMRI was being delivered efficiently. Most interviewees, who were asked (3/4), reported that the shared leadership between CIHR-III and CIHR-IPPH allowed for priority-setting expertise across the two institutes’ mandate areas, which was perceived as advantageous given the multifaceted nature of AMR. However, many respondents (4/5) noted challenges in the delivery of the AMRI, owing specifically to the relatively limited resources of the initiatives. For example, some key informants (2/5) cited concerns regarding long-term sustainability of the initiative given the limited available resources for grants. Many (3/5) also noted that the initiative had a high administrative burden to meet the demands of AMR research at the national and international level, and at times it was challenging for the scientific co-leads to balance their respective Institute mandates with the Initiative’s cross-sectoral objectives, given the AMRI’s limited resources.

The AMRI governance is effective but could benefit from external independent advice.

The main governance body for the AMRI initiative is the CIHR AMR Working Group. The working group is chaired by the IMIS Project Lead and is composed of CIHR staff from across the agency’s III and IPPH branches. Broadly, the working group’s mandate is to provide operational guidance and project management support for the implementation of AMRI, including guidance on partnership development, stakeholder engagement as well as program and funding opportunity design elements.

A review of the program authority highlighted the requirement for an external advisory committee. At the time of the AMRI’s inception, it was deemed that an external advisory body was not necessary but that this decision would be revisited as the initiative progressed. It was determined that advice could be sought through various existing bodies such as the JPIAMR management committee, the Federal, Provincial and Territorial AMR Research and Innovation Committee and the PHAC AMR Taskforce. It is important to note that despite this approach, a couple key informants (2/16) noted that AMRI could have benefited from advice from an external advisory committee, particularly on issues related to how best to maximize a limited initiative budget to help achieve a balanced approach in activities and research investments across the multidisciplinary and multisectoral facets of AMR research.

It is not clear whether the AMRI is being delivered in a cost-efficient manner.

The percentage of direct program administrative costs to total program expenditures is a measure of how efficiently the AMRI is being administered. For the period under review, the overall percentage of total direct administrative costs to total expenditures for the AMRI was 13.9%, which is higher than CIHR’s percentage of 5.3% for the same period under review (Figure 7: Cost-Efficiency Table). The percentage of direct administrative costs to total program expenditures was 45.1% in 2016-17, which corresponds to the first year of the initiative with the grants and award expenditures at their lowest level and schedule to be ramped up in subsequent years. Indeed, as grants and awards expenditures began ramping up in 2017-18, the percentage of direct operating costs to total program expenditures stabilized to between 13.5% and 11% for the remainder of the period under review.

AMRI’s relatively high percentage of direct administrative costs to total program expenditures is driven by its direct operating expenditures, which includes direct salary costs. The direct operating expenditures range between $244k and $318k between 2016-17 and 2020-21, which are, on average, 43.4% higher than the $200k that was stipulated by the program authorities. Although CIHR operating expenditures for the AMRI program are higher than initially planned, key informants who could speak on the matter (3/4) indicated that despite AMRI’s small grants and awards budget, its administrative requirements such as national and international partnership management, are comparable to other CIHR initiatives with greater grants and awards expenditures.

The COVID-19 pandemic has had minimal impact on the design and delivery of the AMRI.

Most CIHR management key informants (3/4) indicated that the COVID-19 pandemic did not have a major negative impact on the design and delivery of the AMRI, citing effective planning and the small size of the initiative as major mitigating factors. However, there were a couple key informants (2/4) who cited challenges with the coordination of JPIAMR funded projects, stating that stronger communication with international partners would ensure all parties were on the same page regarding the impact of COVID-19 on funded projects (e.g., extensions, expectations around end-of-grant reporting). Some interviewees (5/16) also indicated that the COVID-19 pandemic contributed to a shift in priorities which ultimately delayed the release of the Pan-Canadian Action Plan, arguing that this delay could indirectly affect long-term planning of the AMRI.

Performance

Key Findings:

The AMRI is achieving its expected outputs of shaping AMR priorities and funding research grants in priority areas, however important gaps remain.

The AMRI’s main expected outputs are to provide leadership and advice on the identification priority research areas for AMR and fund research and researchers in these areas. There is evidence that the initiative is achieving these outputs, however, a review of administrative data and key stakeholders indicated gaps in the nature of research funded as well the diversity of researchers supported.

AMRI leadership is establishing and shaping research priorities nationally and internationally.

AMRI leadership is establishing and shaping AMR research priorities by engaging in national and international activities. At the national level, AMRI leadership was instrumental in the development of the 2014 Federal Framework for Action, the 2015 Federal Action Plan on AMR, and the 2017 Pan-Canadian Framework for Action by providing recommendations for action under the innovation pillar. AMRI leadership also co-chaired the Research and Innovation Task Group involved in the development of the Pan-Canadian Action Plan on AMR, released in 2023. Through partnership with JPIAMR, the scientific leadership of the AMRI contributed to shaping research priorities internationally by being part of the management board, which is the main decision-making body of JPIAMR. They are also members of the steering committee that provide steering direction of the JPIAMR and strategic input to undertake the JPIAMR mission. Although many key informants (8/16) believed this partnership to be beneficial, some (4/16) highlighted the challenges of influencing the research agenda given the need for consensus among other countries involved.

Additionally, the AMRI scientific leadership represents Canada by chairing GloPID-R, vice- chairing alongside Germany on the board of the Global AMR R&D Hub, as well as providing technical expertise to TATFAR.

The AMRI is funding research grants in key priority areas.

For the period under evaluation, CIHR invested a total of $8.8M through 30 grants across its four major funding streams. Specifically, JPIAMR represents 50% ($4.4M, n = 15 grants) of AMRI’s total investment, followed by Point of Care Diagnostics at 34% ($3.0M, n = 8 grants), Network in Global Governance of Infectious Diseases at 13% ($1.1M, n = 4 grants) and Priority Announcements at 3% ($311K n = 3 grants) (Figure 8: AMRI Investment by Funding Streams). Collectively, funded projects across all of AMRI’s four streams are represented in three PHAC priority areas with 90% ($7.9M) of AMRI’s investments in the area of research and innovation, 46% ($4.0M) in the area of stewardship and 12% ($1.1M) in the area of surveillance (Figure 9: Proportion (%) of CIHR Investment in PHAC Priority Areas).

There is an opportunity to enhance engagement with expertise in the behavioural and social sciences and support adoption of a One Health approach to AMR research.

AMRI’s investments are represented across all of CIHR’s research themes. The majority of the investments fall under the biomedical research theme, and the social, cultural, environmental, population health research theme, accounting for 38% ($3.4M) and 35% ($3.1M), respectively. It is worth noting that the proportion of AMRI’s investment in the social, cultural, environmental, population health theme is greater than the proportion of CIHR’s total investment in AMR for the same theme (35% in AMRI compared to 9% in AMR), thus indicating that the AMRI is contributing to addressing a gap in research investments within this theme (Figure 10: Proportion (%) of AMR and AMRI Investments by CIHR Primary Themes).

In addition to investing across all CIHR’s themes of health research, AMRI researchers worked in interdisciplinary teams that included expertise across CIHR themes. The majority of surveyed researchers (83.3%, n = 12) reported conducting their AMR research project with an interdisciplinary team comprised of members from two or more disciplines. Similarly, most of the researchers interviewed (4/5) indicated that they worked in interdisciplinary teams with expertise from areas such as biomedical, social sciences, clinical and knowledge users. Despite evidence of engagement with interdisciplinary teams, many key informants (8/16) felt that most AMRI research still has a biomedical focus and that there are opportunities to enhance engagement with expertise in the behavioural and social sciences.

Although CIHR themes of health research are well represented in AMRI funded research, there was limited evidence of adoption of a One Health approach within funded research. A One Health approach recognizes the interconnectedness of humans, animals and the environment and aims to ensure a coordinated, collaborative, multi-disciplinary approach to address health risks that originate at the human-animal-ecosystems interface. Key informants (6/16) highlighted the importance of funding AMR research with a One Health lens, with some (5/16) arguing that there is not enough attention paid to funding One Health research. Based on the researcher survey, on a 5-point scale, from ‘not at all’ to ‘to a very great extent’, more recipients (M = 3.7 out of 5, SD = 1.5, n = 12) than applicants (M = 2.7, SD = 1.8, n = 12) indicated that their AMR project adopted a One Health approach. While 41.7% of recipients (n = 12) reported that their AMR project adopted a One Health approach ‘to a very great extent”, the same proportion of applicants (41.7%, n = 12) reported not adopting a One Health approach at all. While some of the researchers reported adopting a One Health approach in the survey, only one of the researchers interviewed (1/6) indicated that they had adopted a One Health approach within their AMRI-funded research project. It should be noted that this research project was funded through an opportunity explicitly designed to fund One Health AMR research. Given the importance of using a One Health approach to effectively tackle the threat of AMR, there is an opportunity for the AMRI to support adoption of One Health approach within research.

The AMRI is not adequately supporting the recruitment of diverse AMR researchers.

Although CIHR is committed to using GBA+ and adopting the Tri-Agency EDI Action Plan by supporting equitable access to funding opportunities for all members of the research community, this is not well reflected in the diversity of AMRI researchers with some groups disproportionately underrepresented. Surveyed AMRI recipients were given the option to self- report their demographic information. Of the AMRI recipients surveyed, 65% (n = 11) self-identified as men and 29% (n = 5) as women. Only one recipient (8%) identified as a visible minority, with none identifying as Indigenous, a potential concern for the on-going relevance of the Initiative given that Indigenous leadership has been prioritized in the 2023 Pan-Canadian Action Plan. Almost three-quarters (71%, n = 12) indicated that their first official language was English, while almost one-quarter’s (24%, n = 4) was French. This lack of diversity could be related to some of the AMRI funding opportunities predating CIHR’s formalized EDI commitment. Survey findings were supported by many key informants (8/16) who stated that while the AMRI has made advances in engaging diverse research, there is an opportunity to make EDI+ considerations more explicit in funding opportunities in order to better meet the needs of disproportionately underrepresented groups. Overall, the AMRI has the opportunity to take steps to increase diversity of funded researchers going forward.

The AMRI is making progress on its immediate outcomes.

The AMRI is making progress on its expected immediate outcomes, which include generating and disseminating new knowledge, developing capacity in AMR research and establishing research partnerships and collaborations.

The AMRI is advancing knowledge that is well disseminated in open access and impactful journals.

The AMRI is advancing knowledge through peer reviewed publications. According to the bibliometric analysis, within the period of 2017 and 2021, there were 55 publications that acknowledged AMRI funding. These publications were proportionally well-represented in open access (normalized share of 1.27 compared to 1.03 globally^{Footnote 18}) and published in high impact journals at a rate 30% higher than the global average (ARC is 1.30 compared to 1.01 globally^{Footnote 19}).

In addition to the bibliometrics analysis, almost all AMRI recipients surveyed (92%, n = 11) reported having at least one publication attributed to their AMRI research either published or in press. The average number of publications produced, or in press per researcher, was 3.69 (SD = 4.4, n = 11). Of those who reported publications, the most common type were peer-reviewed journal articles (M = 5.0, SD = 4.2, n = 11).

The AMRI leadership and researchers are supporting knowledge translation.

AMRI leadership has also supported various knowledge translation activities, such as hosting a Multi-Sectoral Antimicrobial Stewardship Expert forum in 2016 in collaboration with Merck Canada, supporting Canadian participation in an AMR Hackathon event in Sweden, and, in the past few years, having been involved in the World Antimicrobial Awareness Week. Additionally, AMRI’s leadership team was involved in the creation of JPIAMR-VRI which provided a platform to facilitate knowledge exchange and capacity development across the globe.

Researchers have also reported producing knowledge translation products as a result of their AMRI research. All researchers surveyed (n = 12) reported either producing or being involved in producing presentations. The average number of presentations produced by recipients was 3.6 (SD = 3.1, n = 12). Half of surveyed recipients reported producing workshops, held either in person or virtually (6/12, M = 2.8, SD = 1.8), while nearly half produced webinars (5/12, M = 1.8, SD = 1.9). Other products produced by AMRI recipients include an average of 1.5 books or book chapters (SD = 1.2, n = 6), 8.6 reports (SD = 12.5, n = 5), 1.3 pieces of grey literature (SD = 1.8, n = 6) and 4.7 social media products (SD = 1.0, n = 6) such as Facebook pages or websites.

These survey findings are supported by findings in key informant interviews and program documents. All the researchers interviewed (6/6) reported having generated or being in the process of generating and disseminating new knowledge through various products and activities such as publications, community reports, policy briefs, and presentations. Based on program documents, the Global One Health Network produced a Health Equity Toolkit for the Governance of Infectious Diseases with the aim of informing health practitioners, researchers, and decision makers at all governmental levels about the AMR concerns disproportionately affecting equity-seeking groups. This network has also developed three policy briefs on One Health, started a YouTube Channel with a series of videos explaining the One Health concept as well as maintained a Twitter page with 210 followers as of 2021.^{Footnote 20}

The AMRI is helping increase capacity for AMR related research.

The AMRI has made strides in developing AMR research capacity by supporting Canadian and International trainees. The AMRI has provided three PhD scholarship and other scholarship opportunities for master’s and undergraduate students through the Global One Health Network. All AMRI researchers surveyed reported involving t rainees in their AMRI funded research project (100%, n = 12). Of the trainees involved, 38 were undergraduate students, 13 were master’s students, 21 were PhD students, 8 post-health professional degree fellows, and 17 post-doctoral fellows (Figure 11: Number of Trainees Directly Involved in the AMRI Research Project). Additionally, researchers surveyed reported offering several types of training opportunities, including research skill development (M = 4.6 out of 5, SD = 0.7, n = 12), professional skill development (M = 4.3, SD = 1.0, n = 12), interdisciplinary research opportunities (M = 4.3, SD = 1.2, n = 12), leadership skills (M = 4.2, SD = 0.7, n = 12) and technical skill development (M = 4.1, SD = 1.0, n = 12) were reportedly offered to a great- to-very great extent, while intersectoral research opportunities (M = 3.8, SD = 1.6, n = 12) and international research opportunities were offered to moderate-to-great extent. The AMRI should maintain efforts aimed at increasing capacity for AMR related research.

The AMRI has led to the establishment of Canadian and international partnerships in various sectors.

The AMRI has led to the establishment of Canadian and international research partnerships. Based on program data, AMRI partners at the time of application could be categorized into three broad categories: international; public sector, particularly federal and provincial government; as well as private sector, such as biomedical companies. These partners have contributed a total of $12M, of which international partners contributed the greatest amount at $11M, followed by public at $758K, private at $256K and $150K from an uncategorized partner (Figure 12: AMRI Partner Contribution by Category).

In addition to these application partners, it appears AMRI researchers formed other partnerships through the course of their research. According to the researcher survey, two-thirds of recipients (67%, n = 8) included partners in their AMR research project. These partners were also from private sector (M = 2.0 out of 5, SD = 1.1, n = 12), health charities (M = 1.6 out of 5, SD = 0.7, n = 11), and public sectors specifically, federal (M = 2.5 out of 5, SD = 1.0, n = 11), provincial (M = 1.9 out of 5, SD = 1.0, n = 11) and municipal government representatives (M = 1. 3 out of 5, SD = 0.7, n = 11) who were less frequent. On a 5-point scale from ‘not at all’ to ‘to a very great extent’, researchers reported that these partners contributed to the achievement of their project’s outcomes to a moderate-to-very great extent (8/12, M = 4.3, SD = 0.7).

Similarly, many researchers who were interviewed (3/6) indicated that they were able to partner with federal and provincial governments as knowledge users to facilitate the translation of knowledge into action. Partnerships with industries were also formed for commercialization opportunities (2/6). Other partnerships formed (3/6) were with laboratories for testing and academic institutions for resources and expertise. Finally, partnerships with hospitals and community groups allowed for data collection. All key informants (4/4) also noted that the project would not have been possible without these partnerships.

The AMRI is enhancing the capacity for Canadian researchers to establish national and international as well as interdisciplinary research collaborations.

Many key informants (8/16) across respondent groups reported that through JPIAMR, researchers have enhanced their capacity to conduct international and interdisciplinary research, including with researchers in low- and middle-income countries (3/16). Many key informants (7/16) highlighted that these collaborations have allowed for access to resources, innovative ideas, and expertise. Some researchers interviewed (2/6) also indicated that collaborators were involved in all phases of the research, which facilitated the conduct and completion of research projects.

In addition to this, there is evidence that AMRI researchers are establishing collaborations within Canada and internationally, mainly within the same discipline, however, some are with other disciplines. Based on the survey, researchers reported collaborating with researchers within their discipline (100%, n = 12) more than with those outside their discipline (92%, n = 11) (Figure 13: Proportions (%) of AMRI Researchers with Collaborations). All recipients (100%, n = 12) reported collaborating with researchers within their discipline, both in Canada (M = 3.7 out of 5, SD = 0.8, n = 11) and outside Canada (M = 3.3 out of 5, SD = 1.1, n = 12). Of these collaborations, many were newly established (Figure 14: Proportion (%) of AMRI Researchers with New Collaborations). More than two-thirds (67%, n = 8) of collaborations with researchers in the same discipline, both inside and outside of Canada, and three-quarters (75%, n = 9) of collaborations with researchers in the same discipline, both inside and outside of Canada, were established during the recipient’s AMRI research project. On a five-point scale from ‘significant negative impact’ to ‘significant positive impact’, the majority of recipients indicated that their AMRI research project had a positive impact on these collaborations (M = 4.4 out of 5, SD = 0.7, n = 12).

When examining AMRI funded research publications, bibliometric analysis indicates that 93% of publications are published with Canadian authors with different institutional affiliations and 44% are published with authors affiliated to an international institution, suggesting that collaborations are contributing to the development of research products.

The AMRI is already supporting AMR research that informs decision-making beyond academia.

Though the AMRI is still in relatively early stages of scientific productivity, emerging evidence suggests that AMRI funded research is having impacts beyond academia. Altmetric analysis found that AMRI funded research has been cited in media outlets (i.e., Twitter, Facebook, Wikipedia, News, and Blogs) at a rate that is at least 1.12 times the world average. It should be noted that comparisons of AMRI funded research could not be computed with the world average for citations in patents, clinical trials, clinical guidelines and policy documents as part of the altmetric analysis, as the number of publications was less than 30 for each of these categories.

However, the analysis did reveal that three AMRI -funded publications were cited by three patents and one policy document, including:

• a publication funded through the Point of Care Diagnostic stream was cited by a patent related to detecting and assessing the risk of Clostridoides Difficile Infections and another patent on Adenovirus vectors;
• a publication funded through the Priority Announcement stream was cited by a patent on bacterial conjugative systems; and,
• a publication funded through the JPIAMR funding stream was cited by a policy document commissioned by the CDC and the Wellcome Trust on Essential Initiatives to Mitigate AMR in the environment.

A review of the program documents indicated that Point of Care Diagnostics research on the rapid diagnosis of urinary tract infections has led to two new patents and the development of a spin-off company that partnered with Dynalife Medical Lab to evaluate the Hopper platform for urinary tract infection diagnostics in Edmonton. The Hopper™ technology developed under this partnership aims to provide clinicians with timely information for properly treating urinary tract infections. Knowing exactly which species of microbe is present and which medications will be most effective in treating it allows doctors to precisely tailor the dose and type of medication they prescribe. This will minimize the incorrect treatment of these infections, reduce inappropriate antibiotic use, and ultimately contribute to improved patient outcomes.

Program documents further indicated that investments in JPIAMR calls involving Canadian researchers have also led to a spin-off company called Adapsyn Bioscience. This spin-off company has conducted research for Pfizer and collected venture financing to further the efforts began with the grant. This has led to the creation of over 50 research jobs in Hamilton, Ontario. The spin-off company has also received funding from the Gates Foundation to use the technology created from the grant to define new medicines for malaria and drug resistant tuberculosis treatment.

JPIAMR funded research has also led to the identification of lead molecules that are under further development as preclinical candidates. These lead molecules have the potential to treat wounds, bloodstream and medical device associated infections caused by pseudomonas aeruginosa.

Additionally, JPIAMR funded research has led to ongoing activities such as project members participating in policy initiatives nationally and internationally, forming new collaborations with the Tuberculosis Alliance, as well as securing further funding from a recent JPIAMR call and the Combating Antibiotic-Resistant Bacteria Biopharmaceutical Accelerator strategy.

While the COVID-19 pandemic has had a negative impact on AMRI research activities, it created an opportunity to build on lessons learned for AMR response.

Most surveyed AMRI researchers (83%, n = 10) reported having been negatively impacted by COVID-19, citing the following impacts: delayed research progress (n = 4), networking challenges (n = 3), shift in research funding priorities to COVID-19 research (n = 2), challenges recruiting staff and trainees (n = 1), and reduced lab access (n = 1). More than two-thirds of researchers (67%, n = 8) reported that their AMRI research project has recovered from these impacts, while one-quarter (25%, n = 3) reported that their AMRI research project has not recovered. More than half of researchers (58%, n = 7) do not anticipate any future impacts on their AMR research as a result of COVID-19. Some key informants (6/16) with international collaborators indicated that their research progress was slowed down by challenges collaborating with partners in different countries, who experienced pandemic measures and policies different from those in Canada.

Going forward, most key informants (10/16) across respondent groups felt that the long-term impact of COVID-19 could have implications on AMR research broadly. Notably, they (10/16) expressed that the pandemic has highlighted the importance of paying attention to emerging health threats and having a proactive coordinated effort in tackling health issues; and has helped build capacity in infectious disease research that can be leveraged for AMR.

Conclusions and Recommendations

Relevance

The evaluation concludes that there is an ongoing need to support AMR research as AMR remains a public health threat for Canada as well as globally. AMRI’s current funding is not sufficient to meet its intended objectives for AMR research in Canada. Despite this inadequate funding, the evaluation found that the AMRI is addressing some needs by investing in priority areas although opportunities specifically focused on equity-seeking groups and Indigenous leadership remain lacking.

The AMRI is aligned with the federal government and CIHR’s roles and responsibilities of addressing AMR as a global health threat. Although, findings indicate that a more active coordinated approach from all levels of government with a One Health approach is required.

Additionally, the AMRI’s objective aligns with the priorities set out by the federal government in the Minister of Health’s 2019 and 2021 mandate letters and other federal AMR documents. It also aligns with CIHR’s strategic priorities, particularly with the ones indicated in the current Strategic Plan.

Design and Delivery

The design of the AMRI supports the initiative in achieving its objectives by funding key Government of Canada AMR priority areas. The four major funding streams were also well aligned with the initiative’s objective. However, there were concerns that JPIAMR funding calls favoured researchers with existing international collaborations, thus deterring some researchers from applying. It was also suggested that the AMRI could more effectively meet its objective as a research network. In terms of the delivery of the AMRI, it has been effective by having clear roles and authorities as well as having two Scientific Directors that provide a balanced approach in scientific leadership. Although, this can be challenging for the scientific co-leads to balance their respective Institute mandates within the initiative, given the AMRI’s limited resources. It was also noted that the initiative had a high administrative burden to meet the demands of AMR research at the national and international level.

The AMRI governance was also found to be appropriate and effective. However, it was suggested that the AMRI could benefit from having an external advisory body as originally indicated in the program authorities. With regard to the cost-efficiency of the AMRI, the total ratio of AMRI direct administrative costs to total expenditures was 13.9%, which is higher than the CIHR ratio of 5.3% for the same period under review.

Overall, the COVID-19 pandemic was found to have limited impact on the design and delivery of the AMRI. However, there were concerns that delays in the release of the Pan-Canadian Action Plan (released in June 22 2023) due to the pandemic will indirectly affect the long-term planning of the initiative.

Performance

The AMRI is achieving its expected outputs of funding research grants in priority areas and shaping AMR priorities. The AMRI has funded grants addressing federal priorities across CIHR’s research themes but most notably, biomedical research and social, cultural, environmental and population health. However, the evaluation determined that there is a need for enhanced engagement with expertise in the behavioural and social sciences and adoption of a One Health approach to AMR research. Findings also suggested that the AMRI is not adequately supporting the recruitment of diverse research partners, with some groups disproportionately underrepresented such as Indigenous communities whose leadership is a priority in the 2023 Pan-Canadian Action Plan.

The AMRI leadership has helped shape research priorities within Canada through development of Government of Canada AMR policy documents. Scientific leadership has also contributed to shaping research priorities internationally through partnership with JPIAMR and representing Canada on TATFAR, the Global AMR R&D Hub and GloPID-R. Although, it was reportedly challenging to influence JPIAMR’s research agenda given the need for consensus among the other countries.

The AMRI is making progress towards achieving its expected immediate outcomes. The AMRI is advancing and disseminating knowledge that is proportionally well-represented in open access journals and published in high impact journals at a rate higher than the global average. The evaluation also found that the AMRI is supporting knowledge translation, through involvement of the AMRI leadership in the creation of JPIAMR-VRI that provides a platform to facilitate knowledge exchange and capacity development across the globe. Moreover, AMRI researchers also reported producing knowledge translation products as a result of their AMRI funded research.

The AMRI has made strides in developing AMR research capacity by supporting Canadian and International trainees and by increasing capacity for collaborations on AMR research. AMRI is has led to the establishment of Canadian and international research partnerships and surveyed AMRI researchers reported collaborating with researchers both within and outside their discipline. Researchers indicated that collaborators were involved in all phases of the research project, which facilitated the conduct and completion of research projects.

Although it is too early to fully determine the extent to which intermediate and ultimate outcomes are being achieved, there is evidence that the AMRI is already making progress towards its expected intermediate outcome that AMRI-funded research will have impacts beyond academia. Altmetric analysis indicates that AMRI funded publications were cited by three patents and in one policy document. Research funded through the Point of Care Diagnostics funding stream has led to two new patents and the development of a spin-off company. International collaborations through JPIAMR have also led to a spin-off company that has conducted research for Pfizer and received venture financing and further funding from the Gates Foundation. Additionally, JPIAMR funded research has led to the identification of lead molecules that are under further development as preclinical candidates.

Not surprisingly, the evaluation found that most AMRI researchers were negatively impacted by COVID-19, citing the following impacts: delayed research progress, reduced lab access, networking challenges and a shift in research funding priorities to COVID-19 research. However, the pandemic also created an opportunity to build on the lessons learned in responding to the threat of AMR.

Recommendations

The evaluation makes five recommendations to improve the performance of the AMRI to better achieve its expected results.

Recommendation 1:

CIHR should continue to invest in priority-driven AMR research through the AMRI and assess the level of funding to achieve AMRI expected outcomes and support the federal government AMR research priorities.

Recommendation 2:

CIHR should be guided by a One Health approach for the AMRI that recognizes the interconnectedness of humans, animals and the environment within health research to address the threat of AMR.

Recommendation 3:

CIHR should engage with other federal research funding agencies to better support a One Health approach to AMR research.

Recommendation 4:

CIHR needs to embed equity, diversity and inclusion considerations as well as engagement with Indigenous communities into all aspects of the AMRI and, by extension, AMR research in general.

Recommendation 5:

CIHR should consider a mechanism for external independent advice on AMR Research Priorities as outlined in the program authorities.

Appendices

Appendix A: Figures

Figure 1: CIHR Investments in AMR Research^{Footnote *}

Source: CIHR Funding Analytics

Figure 2: CIHR Investments in AMRI by Sources of Funding

Source: CIHR Finance

Figure 3: AMRI Logic Model

Figure 4: Publication Output and World Share of the Top Publishing Countries in AMR Research (2011–2021)

Source: Science-Metrix, Publication data retrieved from Scopus, April 2022

Figure 5: Canada’s Global Share of Publications and Specialization Index

Source: Science-Metrix, Publication data retrieved from Scopus, April 2022

Figure 6: CIHR Investments in AMR Research by Program Year

Source: Science-Metrix, Publication data retrieved from Scopus, April 2022

Figure 7: Cost-Efficiency Table

Source: Award expenditure and administrative costs data obtained from CIHR Finance

Figure 8: AMRI Investment by Funding Streams

Source: CIHR Electronic Information System (EIS)

Figure 9: Proportion (%) of CIHR Investment in PHAC Priority Areas

Source: CIHR Electronic Information System (EIS)

Figure 10: Proportion (%) of AMR and AMRI Investments by CIHR Primary Themes

Source: CIHR Electronic Information System (EIS)

Figure 11: Number of Trainees Directly Involved in the AMRI Research Project

Source: AMRI Researcher Survey Results, 2021

Figure 12: AMRI Partner Contribution by Category

Source: CIHR Electronic Information System (EIS)

Figure 13: Proportions (%) of AMRI Researchers with Collaborations

Source: AMRI Researcher Survey Results, 2021

Figure 14: Proportion (%) of AMRI Researchers with New Collaborations

Source: AMRI Researcher Survey Results, 2021

Appendix B: Methodology

This section presents additional details about the methodology used to conduct the evaluation of the AMRI.

Document and Literature Review

The document review consisted of an examination of government documents, AMRI general documentation (e.g., initiative documents, briefing notes, federal government policy and strategic framework documents, Treasury Board submissions), AMRI core element performance reports and annual reports, AMRI Working Group materials and partners’ documentation. The literature review focused on reviewing academic and professional literature at the national and international level.

Both the document and literature review assisted the evaluation team in assessing the ongoing relevance of the AMRI. Specifically, the nature and extent of the ongoing need to prioritize research on AMR and the extent to which the AMRI is relevant to addressing this need. These analyses focused on exploring the current state of AMR research in Canada as well as the burden of disease.

Administrative and Financial Data Analysis

The administrative data review included the review of data from the CIHR’s Electronic Information System [EIS] and Research Reporting System [RRS]. This component of the data review contributed to addressing design and delivery and performance by assessing progress toward expected outcomes over time.

An analysis of financial data was conducted to examine the initiative’s cost-efficiency. The efficiency analysis assesses the evolution of the ratio of administrative costs to total expenditures during the evaluation period. Total expenditures are determined by combining the total award expenditures and total administrative costs. At CIHR, administrative costs represent about 5% of the total budget, which was used in the analysis as the reference for management efficiency ratio. If the ratio of administrative costs to total expenditures is below 5%, the program will be performing efficiently and not otherwise.

Survey of AMRI Researchers

The AMRI researcher survey targeted all researchers who have either received (i.e., recipients) or applied (i.e., applicants) for an AMRI funding opportunity between January 2016 and December 2019. The inclusion of applicants allowed for counterfactual comparison, explored whether alternative funding was obtained and the extent to which the achievement of objectives, capacity development and research outcomes were impacted by not obtaining AMRI funding. The researcher sample frame was constructed from CIHR grant data. Duplicate researchers were removed for the purposes of surveying, to ensure that each researcher only received the survey once. The final sampling frame comprised of 104 researchers: 24 recipients and 80 applicants. The resulting survey data utilized for the purpose of this evaluation consisted of 41 respondents (response rate: 39.4%), 29 applicants (response rate: 36.3%) and 12 recipients (response rate: 50%).

The AMRI researcher survey contributed to addressing evaluation questions related to both the design and delivery of the initiative as well as progress in contributing to the initiative’s expected outputs, and immediate and intermediate outcomes. Researchers who received funding answered questions pertaining to the nature of their research, capacity development, research outputs (e.g., publications, workshops) and collaborations formed as a result of involvement in the initiative. While the survey of applicants explored the extent to which they were able to proceed with their research, whether they obtained any alternate sources of funding, capacity development, research outputs (e.g., publications, workshops) and collaborations formed.

The surveys was conducted online using professional Canada-based survey software (i.e., Voxco) and in the respondents’ official language of choice.

Key Informant Interviews

In total, 16 key informants were interviewed from three respondent groups to provide in-depth perspectives on relevance, design and delivery and performance questions as well as the impact of COVID-19 pandemic on AMR research. It should be noted that the key informants under the researcher respondent group were selected based on the fact that they had not filled out the survey to ensure their feedback was provided.

In total three interview guides were created to ensure questions were tailored and relevant for the key respondent groups. The guides were semi-structured to ensure that input was gathered for the relevant evaluation indicators, while allowing respondents the flexibility to respond to the questions in their own words.

Interviews were 45 to 60 minutes long and conducted via a secured private MS Teams channel. With the permission of the interviewee, the interviews were recorded, and a transcript was generated by MS Teams.

All the interviews were conducted by two members of the AMRI evaluation, one was responsible for asking questions and the other for note taking and recording.

Once the transcripts were generated by MS Teams and notes from unrecorded interviews were cleaned. If requested, respondents were offered the opportunity to validate their interview transcripts/notes.

The following table outlines the number and type of respondents who participated in key informant interviews.

End of Grant Report Data

AMRI grant recipients are expected to complete the end of grant report within 18 months of grant expiry. The end of grant report assesses recipients’ outputs and outcomes during the tenure of the grant in terms of numbers of collaborations, trainees involved in the project, and knowledge products (e.g., journal articles, patents).

Available end of grant report included data from 11 grant recipients covering the competition period 2014-15 to 2020-21. The analysis compiled data from Point of Care Diagnostics end of grant reports (2016-18, n = 4); JPIAMR (2014-2017, n=5); and Global Governance (2017, n=2).

Bibliometric and Altmetric Analysis

The bibliometric and altmetric analysis was conducted by an external contractor, Science-Metrix Inc. The objective was to obtain bibliometric and alternative metrics indicators on the publication and knowledge product output of CIHR’s AMRI in order to compare its knowledge production with the international community of researchers publishing in the field of AMR research.

Science-Metrix used Scopus as the main bibliometric database for this study. Data from Scopus were complemented by other sources of data, such as Unpaywall to measure the availability of publications in open access, PlumX to measure citations in social media and popular websites, Overton to measure citations in policy documents, PatentSight to measure citations in patent applications, ClinicalTrials.gov to measure citations in clinical trials, and PubMed to measure citations in practice guidelines.

The analysis involved a list of 55 AMRI publications, covering the period of 2017-2018 to 2020- 2021. This list was used to match to records in Scopus and all 55 publications were successfully matched. Science Metrix also created a data set of publications in AMR (covering the period of 2011-12 to 2020-2021) by first defining a set of core keyphrases or specialized journals, whose specificity to AMR is very high. These keyphrases and journals were then used to capture relevant publications from Scopus to obtain seed data set. This set was then expanded by selecting specific keyphrases, queried in the title, abstract and keywords of all publications indexed in Scopus. The data set was built in consultation with CIHR Project Authority (staff from III, IPPH, IMIS and the Evaluation Unit) and the list of keyphrases was shared for their review and approval. The resulting data sets included 160,770AMR papers. In addition to the AMR publications, a portion of AMR data set of publications was also categorized by some policy objectives of interest to CIHR Project Authority. These included issues on diagnostics, policy and governance, stewardship, surveillance, transmission, and the One Health approach pertaining to AMR research. Among the overall output in AMR research, roughly 10,000 publications addressed each of the surveillance and transmission issues, close to 7,000 addressed each of the diagnostics and stewardship issues, 3,110 publications addressed the policy and governance issue, and 534 covered the One Health approach.

Bibliometric indicators used in the analysis:

• number of publications
• share of publications acknowledging support from CIHR
• share of domestic (Canadian) and international co-publications
• specialization index
• normalized share of publications available in open access
• citation impact indicators
• number of citations
• relative citation score
• average of relative citations
• share of highly cited publications
• average of relative citeScores

Altmetric indicators used in the analysis:

• normalized share of publications cited in patents
• normalized share of publications cited in policy documents
• normalized share of publications cited in clinical trials
• normalized share of publications cited in practice guideline publications
• normalized share of publications referenced or mentioned in news items
• normalized share of publications referenced or mentioned in blogs
• normalized share of publications referenced or mentioned on Wikipedia
• normalized share of publications referenced or mentioned on Twitter
• normalized share of publications referenced or mentioned on Facebook
• normalized share of publications referenced or mentioned on YouTube
• normalized share of publications referenced or mentioned on Reddit

Limitations and mitigations

The following describes the key limitations associated with the evaluation, which were mitigated through the tri-angulation of results across data sources as well as consideration of the relative strengths and weaknesses of each data source. The mitigation strategies employed throughout the evaluation helped to ensure that the evaluation results can be used with confidence to inform program decision making.

Date modified:

2025-01-14