Mid-term evaluation of the Global Strategy to Eliminate Yellow Fever Epidemics (EYE) 2017 - 2026

Executive summary

Background

In 2016, a widespread yellow fever outbreak in Angola, also affecting the capital city, caused unprecedented spread, affecting neighbouring countries with an urban outbreak in Kinshasa (in the Democratic Republic of the Congo) and viraemic travellers to Asia. The following year, yellow fever spread to coastal areas in Brazil including large urban centres that had not seen yellow fever outbreaks in several decades. In response to these outbreaks and the threat of international spread, the WHO,
Gavi and UNICEF developed a comprehensive multi-partner global strategy to Eliminate Yellow fever Epidemics (EYE) 2017-2026. The EYE strategy has three overall strategic objectives, to:

• protect at risk populations;

• prevent international spread; and

• contain outbreaks rapidly.

Forty countries considered high risk for yellow fever outbreaks are targeted under the EYE strategy.
This includes 27 countries in Africa and 13 countries in the Americas.

Purpose, objectives and methodology

The mid-term evaluation was included as a milestone in the EYE strategy, and in the WHO evaluation workplan 2022–2023 which was approved by the WHO Executive Board at its 150th session in January 2022. Undertaken in collaboration with GAVI and UNICEF by the WHO Evaluation Office working with the Regional Offices for Africa and the Americas, the evaluation was commissioned to a competitively selected independent company, Euro Health Group, in May 2022. The purpose of the mid-term evaluation was to assess the relevance, coherence, effectiveness, efficiency and sustainability of the strategy implementation to date and to review inclusion of gender, equity and human rights considerations. This included programme delivery aspects as well as strategy management and governance aspects.

The main objectives of the evaluation were to:

• document key achievements, best practices, challenges, gaps, and areas for improvement in the design and implementation of the strategy;

• identify the key contextual factors and changes that are affecting yellow fever spread and transmission risk profile, and influencing programme implementation; and

• make recommendations as appropriate on the way forward to improve performance and implementation, and to ensure sustainability in the future beyond 2026.

The overall approach to the evaluation was theory-based and included developing a theory of change for the EYE strategy. The theory-based evaluation was combined with a process evaluation, to look in detail at the implementation of the strategy to date.

This report presents findings for five high level evaluation questions (and 15 sub-evaluation questions) and related conclusions and recommendations based on a review of comprehensive datasets and more than 250 documents, 61 key informant interviews carried out at a global, regional and country level; and through a survey of 118 country level key stakeholders across 40 yellow fever high risk countries. In addition, several focus group discussions were undertaken as part of the evaluation (a theory of change workshop, a “strengths, weaknesses, opportunities and threats” analysis, and smaller group discussions with key stakeholders). To further document best practices and lessons learned for strategic actions under the EYE strategy, two country case studies were conducted with missions to Brazil and Ghana.

Analytical approaches comprised: triangulation of data (both across and within categories of data sources); thematic analysis (thematic coding and analysis of secondary documents, key informant interviews and focus group discussion notes); statistical analysis of key EYE M&E indicators and results of the online survey and contribution analysis.

The evaluation methodology was broadly implemented as proposed in the evaluation inception report, with no significant departures from the terms of reference. The timing of the data collection period - over the main holiday season (June-August 2022) - caused challenges related to availability of key informants and online survey respondents. Another limitation was related to the EYE M&E framework, data quality concerns and data gaps with few mid-term targets, missing baseline values, and unavailability of data from the Americas at the EYE secretariat level on several EYE strategic indicators.

Baseline data were reconstructed by the evaluation team using 2017 data from validated EYE data sources where available, and the evaluation relied on projections up to 2026 to establish mid-term targets. Examples of unavailable data and/or data quality concerns have been highlighted throughout the report where applicable. For the results/effectiveness evaluation question, the mid-term evaluation mainly considered progress on the 16 indicators prioritized by the EYE partnership as strategic indicators for the strategy. The evaluation did not consider performance across yellow fever medium- or low-risk countries because the EYE strategy implementation specifically targets the yellow fever high-risk countries.