FCDO annual evaluation report 2022 to 2023
Published 23 January 2024
© Crown copyright 2024
This publication is licensed under the terms of the Open Government Licence v3.0 except where otherwise stated. To view this licence, visit nationalarchives.gov.uk/doc/open-government-licence/version/3 or write to the Information Policy Team, The National Archives, Kew, London TW9 4DU, or email: psi@nationalarchives.gov.uk.
Where we have identified any third party copyright information you will need to obtain permission from the copyright holders concerned.
This publication is available at https://www.gov.uk/government/publications/fcdo-annual-evaluation-report-2022-to-2023/fcdo-annual-evaluation-report-2022-to-2023
1. Introduction
Evaluation is one of the tools in the evidence toolbox that we can use to understand how well things work and why. This supports the extent to which the Foreign, Commonwealth and Development Office (FCDO) is able to achieve its objectives, improve performance, and identify good practice. It helps to target resources to deliver the most impact and maximise the value of taxpayers’ money.
This Annual Report presents an overview of evaluation in the FCDO from April 2022 to March 2023. Over this period the FCDO evaluation policy was implemented, which sets out principles and standards for evaluation, and the first FCDO evaluation strategy which sets out key outcomes to advance and strengthen the practice, quality and use of evaluation.
This report outlines the work of the FCDO Evaluation Unit (section 2), including work on central thematic evaluations, support to the Centre of Excellence for Development Impact and Learning (CEDIL) and a brief overview of the new Strategic Impact Evaluation and Learning (SIEL) programme. Section 3 provides an overview of evaluation across the business during this period and ongoing efforts to ensure quality and rigour.
2. Central evaluation service
The Evaluation Unit supports evaluation across the FCDO. It is part of the Economics and Evaluation Directorate in FCDO, led by the Chief Economist. As of March 2023, the unit comprised 3 Evaluation Advisors, one Data and Insights lead, 3 Programme Managers and a Fast Stream Economist with 3 additional advisors and an evaluation uptake lead scheduled to join the team. In addition, the Evaluation Unit works closely with the Head of Profession for Evaluation and Social Research, who leads and champions the evaluation community and evaluation within FCDO.
Its focus is to deliver the evaluation strategy, in collaboration with teams across the FCDO, so that:
- strategic evaluation evidence is produced and used in strategy, policy and programming
- evaluation evidence is systematic and objective
- learning from evaluations is shared and used in decision-making
- FCDO has an evaluative culture, the right evaluation expertise and capability
2.1 Thematic evaluations
Strategy outcome 1 focuses on producing relevant, timely, high-quality evaluation evidence that is used in areas of strategic importance for FCDO, HM Government and international partners. This is achieved in part through an enhanced focus on thematic and portfolio evaluations. Drawing on evidence from across several projects, programmes or activities, these evaluations aim to answer strategic and cross-cutting questions which will inform policymaking.
In 2022 to 2023, the following thematic evaluation was published: Nutrition and food security data in complex operating environments (PDF, 17 MB) and the first internal bidding window was held to identify opportunities for thematic and portfolio evaluations across the organisation. This resulted in 5 new evaluations covering nutrition, gender and social inclusion, humanitarian action, and weather and climate, all due for completion during financial year 2023 to 2024. Regular bidding windows are planned to continue to support evaluation at the portfolio and thematic level.
2.2 Central programmes
This reporting period included the final year of the Centre of Excellence for Development Impact and Learning (CEDIL) programme[footnote 1] that developed new rigorous methodologies for hard-to-measure problems. Over the 6 operational years of the programme, it has delivered:
- 50 working papers including evaluations, synthesis and lessons learned papers
- 33 papers for academic journals
- 27 user-oriented summary products including briefing notes, portals/maps and methodological guidance
- 68 dissemination and training events held with FCDO or other users
- 42 blog and vlog posts published on the CEDIL website
- 2,006 CEDIL Twitter account followers
- 6,218 attendees at CEDIL Directorate events, including post-event online viewing
All materials are publicly available on the CEDIL website.
The Evaluation Unit is currently developing a new impact evaluation programme, Strategic Impact Evaluation and Learning (SIEL), due to launch in financial year 2023 to 2024. SIEL will deliver strategic, high priority impact evaluations of FCDO activities, follow up studies to assess the long-term impact of previous interventions, and capacity building activities across FCDO.
Following the success of the Evaluation Quality Assurance and Learning Service (EQUALS), the implementation of EQUALS 2 began in April 2022. Further detail relating to the EQUALS service is covered below.
3. Evaluation across the organisation
3.1 FCDO evaluation in numbers
Eighteen evaluation products were published in 2022 to 2023[footnote 2], of which nearly all were FCDO led. Evaluation products include inception reports through to final reports and case studies.
Of these 18 products, 3 were in the Health sector[footnote 3]. The remaining 15 evaluation products fall within the sectors illustrated by Figure 1 below.
Figure 1: Number of evaluation products by sector April 2022 to March 2023
Figure 1 description
Figure 1 shows the number of FCDO evaluation products published by sector in 2022 to 2023.
| Sector | Number of evaluations published |
|---|---|
| Health | 3 |
| Relief coordination, protection and support services | 2 |
| Research | 2 |
| Humanitarian research | 2 |
| Emergency food aid | 1 |
| Security system management and reform | 1 |
| Family planning | 1 |
| Disaster prevention and preparedness | 1 |
| Statistical capacity building | 1 |
| Climate change and environment | 1 |
| Economic and development policy and planning | 1 |
| Primary education | 1 |
| Economic research | 1 |
Figure 2 shows the geographic spread of the evaluation products, from Guyana in the West to Cambodia in the East.
Evaluative activity tends to be concentrated in the Africa region with a few in Asia. It is worth noting that evaluative activity over the last year has broadened across Africa compared to the previous year (Figure 3).
Figure 2: Geographic distribution of evaluations 2022 to 2023[footnote 4]
Copyright Australian Bureau of Statistics, GeoNames, Microsoft, OpenStreetMap, Tom Tom, Wikipedia
Figure 2 description
Figure 2 is a map showing the geographic spread of evaluation products published in the year 2022 to 2023. Evaluation products were published about:
- Angola
- Antigua and Barbuda
- Bangladesh
- Belize
- Burkina Faso
- Burundi
- Cambodia
- Central African Republic
- Chad
- Democratic Republic of the Congo
- Dominica
- Eritrea
- Ethiopia
- Ghana
- Grenada
- Guinea
- Guyana
- India
- Côte d’Ivoire
- Jamaica
- Kenya
- Madagascar
- Malawi
- Mali
- Mauritania
- Montserrat
- Mozambique
- Nepal
- Niger
- Nigeria
- Pakistan
- Rwanda
- Saint Lucia
- Saint Vincent and the Grenadines
- Senegal
- Sierra Leone
- Somalia
- South Africa
- South Sudan
- Syria
- Tanzania
- The Gambia
- Togo
- Uganda
- United Kingdom
- Willemstad*
- Zambia
- Zimbabwe
*Willemstad is the capital and largest city of Curaçao, an island in the southern Caribbean Sea that forms a constituent country of the Kingdom of the Netherlands.
Figure 3: Geographic distribution of evaluations 2021 to 2022
Copyright Australian Bureau of Statistics, GeoNames, Microsoft, OpenStreetMap, Tom Tom, Wikipedia
Figure 3 description
Figure 3 is a map showing the geographic spread of evaluation products published in the year 2021 to 2022. Evaluation products were published about:
- Armenia
- Bangladesh
- Benin
- Cameroon
- Democratic Republic of the Congo
- Ethiopia
- Gabon
- Ghana
- Indonesia
- Côte d’Ivoire
- Kenya
- Liberia
- Madagascar
- Malawi
- Mozambique
- Pakistan
- Rwanda
- Sierra Leone
- Somalia
- Sudan
- Tanzania
- Zimbabwe
3.2 Procuring evaluations
The Global Monitoring and Evaluation Framework Agreement (GEMFA) was successfully launched in February 2023. This framework enables FCDO and other government departments access to qualified pre-selected suppliers ensuring the provision of efficient and effective expert services for the design and implementation Monitoring and/or Monitoring Evaluation and Learning (MEL) of programmes. The framework agreement has 23 unique lead suppliers, each able to access a consortium of specialist suppliers, and provides global coverage (including the UK). Evaluations and monitoring can be undertaken for Official Development Assistance (ODA) and non-ODA programmes across 4 different lots covering a wide range of contexts within 7 thematic pillars.
3.3 Ensuring quality
FCDO is committed to ensuring rigour and quality in all our evaluations. As set out in the FCDO evaluation policy, all evaluations are required to go through our Evaluation Quality Assurance and Learning Service (EQUALS) or another equally rigorous independent Quality Assurance (QA) process.
The EQUALS 2 programme offers call-down evaluation technical assistance, as well as independent quality assurance of all evaluation products. Each product being quality assured is assessed against a standardised template[footnote 5] with criteria covering a range of areas such as independence, stakeholder engagement, ethics and safeguarding, as well as methodological rigour.
During the review period April 2022 to March 2023, a total of 41 requests to quality assure evaluation products[footnote 6] were raised from FCDO, including 4 re-reviews. Figure 4 outlines the type of products that were sent to EQUALS for review. Products are rated excellent, good, fair and unsatisfactory. Products are improved based on qualitative feedback before publication, with those rated unsatisfactory having to undergo re-review following improvements. Of the products EQUALS quality assured, 90% met the minimum quality criteria on their first review.
Figure 4: EQUALS Quality Assurance request type as a percentage, April 2022 to March 2023
Figure 4 description
Figure 4 shows the type of products sent to EQUALS for review, expressed as a percentage of the total:
| Request type | % |
|---|---|
| Baseline | 8% |
| Evaluation report | 53% |
| Inception | 18% |
| Terms of reference | 23% |
The service is also open to other ODA spending government departments. In this review period, the service has been utilised by the Department for Business, Energy & Industrial Strategy (BEIS)[footnote 7] and Department Environment, Food & Rural Affairs (Defra).
3.4 Evaluation impact: case studies
Annex B provides case study examples of how evaluations have influenced decision making within and beyond the FCDO, ranging from informing programme design to strategic planning to refining programme implementation. These case studies are:
- Independent Evaluation of the African Risk Capacity
- Delivering sustainable and equitable increases in family planning in Kenya
- UK PACT 2022 Thematic Evaluation: Complementarities and Synergies
- Danwadaag Programme, Somalia humanitarian and resilience programme (2017 to 2022)
- Building Resilient Communities In Somalia (BRCiS) Programme, Somalia humanitarian and resilience programme (2017 to 2022)
- Evaluation of Nigeria WFP Country Strategic Plan (2019 to 2022)
4. Looking forward: building an evaluation system for FCDO
Over the coming year, FCDO will continue focusing on the implementation of the 3 year evaluation strategy which works towards establishing a strong evaluation system within the FCDO. Its overarching goal is to advance and strengthen the practice, quality and use of evaluation so that FCDO’s strategy, policy and programming are more coherent, relevant, efficient, effective, and have greater impact.
In order to achieve this ambition, we are working towards 4 key outcomes, outlined below, with a series of activities and milestones to ensure we monitor our progress. These workplans and milestones have been reviewed and updated as appropriate, with progress reported to our internal Investment and Delivery Committee and will continue to be reviewed annually.
4.1 Strategic evaluation evidence is produced and used in strategy, policy and programming
FCDO seeks to ensure relevant, timely, high-quality evaluation evidence is produced and used in areas of strategic importance for FCDO, His Majesty’s Government and international partners. This includes delivering demand-responsive thematic and portfolio evaluations, supporting the evidence needs of the organisation delivering our international priorities. We will also establish our new SIEL programme to provide financial and technical support for experimental impact evaluations of high-priority FCDO activities and develop monitoring and evaluation tools to strengthen our approach to measuring the impact of foreign policy and diplomatic priorities.
Finally, we will work to improve our management information systems that will allow us greater oversight of the evaluation landscape and enhance our ability to target resources into high priority areas.
4.2 Evaluation evidence is systematic and objective
Confidence in the findings generated from FCDO evaluations is vital. Activities under this outcome include the continued implementation and embedding of the FCDO evaluation policy, which ensures a common understanding of evaluation principles and the minimum standards to support these.
Evaluation Quality Assurance and Learning Service 2 (EQUALS 2) provides a professional and responsive quality assurance service that supports the full breadth of FCDO development and diplomatic activities. Evaluation Unit will continue to promote its use across the FCDO to support robust monitoring and evaluation activities across the departmental portfolio.
The Evaluation Unit has updated guidance documents and templates used for evaluative activity and will continue to develop and share these. These will be added to a new repository of best practice examples to share across the organisation.
4.3 Learning from evaluations is shared and used in decision making
Evaluation Unit has created a publicly available evaluation database of evaluations published by FCDO to enable access to evaluation evidence. We will build on this over the coming year to improve accessibility and support greater use of evaluation findings.
A number of internal learning documents (‘Evaluation Insights’) have been completed which draw lessons from across FCDO evaluations. These are intended to maximise the learning from evaluative activity and feed into decisions about future investments and internal capacity building needs. One such example includes a review of conflict prevention approaches, drawing on what has and hasn’t worked in the past to consider how best to evaluate future programming. A subcomponent of this review was looking at evidence gaps and identifying opportunities for future research areas. The learning from this review was presented within the FCDO.
Ongoing Evaluation Insights focus on a review of where monitoring and evaluation have been particularly impactful and where we can draw lessons from evaluations that helped to identify less effective interventions, allowing for course correction. These aim to capture lessons about types of or features of interventions that can strengthen the design or implementation for future activities.
4.4 FCDO has an evaluative culture and the right evaluation expertise and capability
This objective seeks to develop an FCDO that is sufficiently resourced with skilled advisers possessing up-to-date knowledge of evaluation, and that minimum standards of evaluation literacy are mainstreamed across the organisation. FCDO will continue to develop capability, through skills training and by drawing on our membership of the cross-government network of Government Social Research professionals. Identifying capacity building priorities will be bolstered by the use of improved management information dashboards generated through the EQUALS 2 programme.
5. Annex A: evaluation reports
5.1 List of evaluation reports published 2022 to 2023
Note: this is a list of evaluation reports and case studies published in the timeframe of April 2022 to March 2023.
5.2 List of evaluation reports completed in 2022 to 2023
Note: this is a list of programmes that had completed evaluation products timeframe of April 2022 to March 2023. There may be delays in the publication of some evaluations.
6 Annex B: case studies
6.1 Independent evaluation of the African risk capacity
Evaluation type: Formative Evaluation
Publication date: October 2022
- Full report: Independent Evaluation of the African Risk Capacity (PDF, 4.4 MB)
- Management response (PDF, 490 KB)
The programme
African Risk Capacity (ARC) is a specialised agency of the African Union (AU) which supports African governments to improve their resilience to climate risks. ARC provides technical assistance to countries to help strengthen their disaster risk management systems and, through its regional risk insurance pool, provides access to rapid, pre-positioned finance following a disaster. Since 2014, ARC has transferred over $1 billion of climate risk to international markets, insuring 30 million people per year.
The evaluation
The ten-year independent evaluation has included formative evaluations, a pilot impact evaluation in Senegal, and additional forthcoming impact evaluations[footnote 8]. This second formative evaluation, published in October 2022, is theory-based and uses mixed methods to test ARC’s theory of change and assess whether ARC is effectively contributing to disaster response and disaster risk management capacity in member states, growing demand for services and products, and developing long term sustainability.
The impact: how recommendations were used
UK funding for the evaluation has led to reforms and improvements to ARC which will serve to improve their effectiveness and sustainability. The management response to the second formative evaluation included an action plan to address key recommendations, the implementation of which is overseen by the ARC Group Board.
The evaluation was developed by an Evaluation Steering Group which was chaired by the African Development Bank and included a range of stakeholders in ARC, thus securing broad support from the donor group for the recommendations and corresponding action plan. Furthermore, other donors chose to fund the implementing partner, Oxford Policy Management (OPM), to produce additional complementary evaluations.
The ARC Conference of Parties, comprised of the 35 ARC member states and ARC Group Board, have used the evaluation to identify necessary reforms and improvement, to hold ARC management accountable for delivery, and ultimately improve the performance of ARC.
For example, the evaluation identified two value for money criteria which were judged to be ‘poor’ – the timeliness of payouts and the targeting of payouts – both of which are the responsibility of the Peer Review Mechanism, a committee of the ARC Group Board responsible for reviewing and approving contingency plans. The UK, through their seat on the Peer Review Mechanism committee, is working with other committee members and ARC staff to address these areas for improvement to improve the overall value for money proposition of ARC and improve the effectiveness of payouts.
The second formative evaluation has also contributed to building an evidence base for disaster risk finance generally and risk pools specifically. The UK has championed risk pools as an integral disaster risk finance tool, which can offer a cost-effective means of climate resilience. Some of the recommendations of the second formative evaluation will be applicable to other global risk pools supported by the UK and other donors in the Caribbean, Pacific, and Southeast Asia.
6.2 Delivering sustainable and equitable increases in family planning in Kenya (DESIP)
Evaluation type: Third Party Monitoring, Final Report
Publication date: September 2022
Full report: Delivering sustainable and equitable increases in family planning in Kenya (PDF, 962 KB)
The programme
The In-Their-Hands (ITH) project in Kenya has been implemented by Triggerise since April 2017. From 1st September 2020 to 28th February 2022 a scale-up of the project in 16 counties[footnote 9] was supported through a development impact bond (DIB) issued to the Children’s Investment Fund Foundation (CIFF) and FCDO.
The ITH project generates demand for contraceptives and sexual and reproductive health (SRH) services among adolescent girls who are provided with information from a variety of sources including peers, mobilisers, and online campaigns. The girls are encouraged to sign up to a digital platform (branded as ‘Tiko’), where they can access health information and information about service providers. Girls in need of health services, including contraception and HIV testing, are directed to a network of ITH partners including pharmacies and franchised private or faith-based health facilities, and are encouraged to rate the services they receive.
Adolescent girls who access and rate services or who refer peers to the platform collect digital reward points (called ‘Tiko Miles’) which they can cash in for a variety of products in local businesses using a smartphone application, SMS messaging or participant cards.
The evaluation
A pre-post survey design was adopted and identical ‘DIB project areas’ were selected for both surveys. ‘DIB project areas’ were defined as all households within a radius of 5 kilometres, around 46 sampled health facilities and pharmacies. The selected DIB project areas covered nearly half of all adolescent girls reached by the project up to April 2021. The sample size for each survey was approximately 5,000.
Overall, the study found a 2.3 percent increase in modern contraceptive prevalence rate (mCPR) over a period of 18 months. Both surveys revealed low levels of awareness among girls in the project areas about the Tiko platform and the low use of the cards provided by the project for obtaining service discounts (may also be as a result of changing the name from t-safe to Tiko). This was, to some extent, also related to the fact that male condoms, which were not covered by the project, were the most frequently used contraceptive method, and that public health facilities, which are not included among Triggerise project partners, were the most frequently used.
The impact: how recommendations were used
Recommendations from the programme evaluation have been used to influence the Delivering Sustainable and Equitable Increases in Family Planning (DESIP) programme which ends in January 2024. Findings show the types of youth friendly reproductive health services and innovative products that appeal to adolescents and how to increase service uptake. The programme has been able to provide disaggregated data to inform activities in a country where many counties are faced with the plight of high teen pregnancies and high maternal deaths.
Findings from the endline study resulted in a high-level dissemination meeting co-hosted by the UK government and the Kenyan Ministry of Health which have informed the ongoing development of the current Adolescent Sexual and Reproductive Health (ASRH) policy in Kenya and the scale up of the ASRH DIB in 10 counties by the UN-SDG platform in Kenya.
6.3 UK PACT (Partnering for Accelerated Climate Transitions) 2022 Thematic Evaluation: Complementarities and Synergies
Evaluation type: Thematic
Publication date: Completed September 2022
Further reading: UK PACT resources
The programme
UK PACT (Partnering for Accelerated Climate Transitions) is a £260 million technical assistance programme that supports ODA-eligible countries with high mitigation potential to reduce their emissions and accelerate just transitions to clean growth. It is jointly funded by the FCDO and the Department for Energy Security and Net Zero (DESNZ) and managed by the FCDO. It started in 2018 and has funding until March 2026.
In 2022, UK PACT consisted of 3 components:
- country programmes: grant funding for capacity-building projects in partner countries
- Green Recovery Challenge Fund: supporting innovative capacity building projects in a wider range of ODA-eligible countries to promote emissions reductions and low carbon solutions
- skill-shares and secondments: providing a direct mechanism to support partner countries[footnote 10]
The evaluation
In 2022, an evaluation of UK PACT was carried out on the theme of Complementarities and Synergies. The mixed-methods theory-based evaluation tested the programme assumption that ‘the delivery model creates opportunities for synergies, knowledge sharing and learning between projects, enabling the programme to deliver results greater than the sum of its parts.’ The evaluation sought to identify the strengths and weaknesses of current practices in order to improve future delivery.
The overall evaluation finding was that there was limited available evidence to validate the programme assumption being tested. UK PACT had scaled up significantly in the 3 years being evaluated, in the challenging context of COVID, and the programme structure was complex. Nevertheless, the evaluators found examples where synergies had been found and exploited during programme delivery, showing that the assumption remains valid, and that the potential for future synergies is high.
The evaluation made a number of recommendations: formalise how different components should work together; ensure the MEL system is fit for purpose for a global portfolio, focus on outcomes; empower and resource the teams that drive UK PACT’s coherence and synergies; clarify UK PACT’s transformative objectives and allocate adequate resources; and strengthen the selection of partners, incentivise collaboration, and foster ongoing learning.
The impact: how recommendations were used
The evaluation findings echoed themes from the “delivery model assessment” of UK PACT that was carried out in 2021 and early 2022. This meant many recommendations had already been actioned, including integration of the components managed by different delivery partners into 1 contract with the aim to bring more of a country-by-country focus to the programme, with better defined country strategies, integrated project and skill-share delivery, empowered country teams and outcome-focused monitoring. The evaluation provided more evidence to support this, reassuring senior leaders that this was the right approach.
The programme team are taking forward the practical recommendations from the evaluation. Many have already been completed (for example, the development of a global strategy for the programme articulating how the different elements aim to work together to create transformational change, changing the governance system so there is more focus on country-level decisions making, and simplification of the current MEL system). Although it is too early to assess the impact of these changes, they have been well-received by the programme’s delivery team.
6.4 Danwadaag Programme, Somalia humanitarian and resilience programme (2017 to 2022)
Evaluation type: Final Programme Evaluation
Publication date: March 2023
The programme
Danwadaag is a durable solutions consortium led by the UN International Organization for Migration (IOM), with local and international NGO partners. Danwadaag aimed to enhance progress towards durable solutions and (re)integration for displacement affected communities (DACs) in urban centres by working directly with the DAC in its target locations and with government authorities to reduce needs and vulnerabilities while strengthening self-reliance and resilience. This was achieved through an area-based approach that strengthens land tenure security, increases access to sustainable basic services and self-reliance. The target groups include the host communities and local authorities as well as returnees and internally displaced people (IDPs). It had a specific focus on vulnerable groups including women and youth.
The evaluation
The programme evaluation was designed to provide accountability and draw out learnings and recommendations on reintegration in Somalia and contribute to a wider community of practice.
The evaluation applied a range of statistical models to evaluate different aspects of local reintegration (perceived integration, and inequalities) across DACs (that is, host communities, IDPs and returnees). This was complemented by qualitative methods to provide an understanding of the differences between the DAC cohorts. This mixed methodology provided insights into change over time, combined with analysis that accommodated the dynamic nature of the programme, examining the influence of specific interventions and the operating context, including shocks.
Primary findings and conclusions
- the proportion of IDP and returnee households in Danwadaag programme areas feeling well-integrated increased from one third (35%) to two thirds (65%) over the course of the programme
- overall, Danwadaag programme communities moved towards less inequality between host community members and IDPs
- the latent measure of local reintegration (Reintegration Index) showed minimal differences between host communities, IDPs and returnees by the end of the programme. This reflects positive progress towards (re)integration with a large change from both baseline and midline when the Reintegration Index for host communities was much higher than for IDPs, and moderately higher than returnees
The impact: how recommendations were us and lessons learnt
Learnings were shared with the consortium to inform future programming supported by other donors.
Recommendations have also informed the development of other FCDO funded programmes such as Somalia Durable Programmes and Humanitarian Assistance and Resilience Building in Somalia (HARBS). For example, the evaluation could not identify ‘graduation’ of IDPs to host community status, or vice-versa, in part due to the complexity in defining and confirming the displacement status of households. To correct this, it was recommended that pathways to local reintegration, including defined stages of progress (stage-gating), be commissioned to inform integration and graduation opportunities for IDPs across the humanitarian-development-peace-nexus (HDPN). This work was commissioned in 2022 and is being used to inform FCDO Durable Solutions work in Somalia and to bridge investments with economic development colleagues at post.
A recommendation to simplify the Local (Re)Integration Assessment tool (LORA) and do more research on the measurement of reintegration, including the systematisation of learning around land tenure security, will be embedded within the HARBS Durable Solutions programming and portfolio-wide monitoring, evaluation and learning strategies. Specifically, ensuring the continued uptake and adaptation of LORA and dissemination of learnings and opportunities for replication within key Durable Solutions forums.
6.5 Building Resilient Communities In Somalia (BRCiS) Programme, Somalia humanitarian and resilience programme (2017 to 2022)
Evaluation type: Final Programme Evaluation
Publication date: March 2023
The programme
The BRCiS programme supports Somali communities in developing their capacity to resist and absorb shocks without undermining their ability to move out of poverty. The programme balances its response to short-term humanitarian needs with this longer-term aim of building resilience. This includes combining lifesaving basic service delivery - such as the provision of clean water, health and nutrition service and cash to purchase food and necessities - with market-based approaches to increase the supply and demand of nutritious foods and public private partnerships to empower and protect communities from future shocks.
This report is specific to the work taken forward under the FCDO Somalia Humanitarian and Resilience Programme (SHARP, 2018 to 2022). BRCiS is an NGO consortium that implemented resilience-building activities in 34 districts in Somaliland, Puntland and South-Central Somalia.
The evaluation
The BRCiS programme evaluation was to provide accountability, draw lessons and recommendations about resilience in Somalia and contribute to a wider community of practice.
It applied a mixed method approach and was designed to combine two widely accepted methods for measuring the complex and dynamic concept of ‘resilience’: Resilience Index Measurement and Analysis (FAO-RIMA2) and the Technical Assistance to NGOs International (TANGO) – Resilience Measure. This was complemented with analyses of BRCiS participant and area communities[footnote 11] and household-level survey data to measure changes in resilience over time, compare levels of resilience between communities, and investigate the influence of BRCiS activities in times of shocks. Due to the lack of intervention/non-intervention comparisons in the design, the analyses could only hypothesise the extent to which any changes in resilience were caused by BRCiS programme activities. These results were therefore triangulated with additional quantitative analyses, qualitative evidence, and stakeholder input to strengthen the evidence of contribution of BRCiS to changes in resilience.
The primary findings and conclusions were:
- there is evidence of an increase in resilience between baseline and midline using some measures: Looking at both FAO-RIMA2 Resilience Capacity Index (RCI) and TANGO RCI, BRCiS households showed an increase in resilience between baseline and midline. Greater increases were seen in participant communities than area communities[footnote 12], although, unexpectedly, participant communities started with a higher RCI. Starting shortly before the midline and continuing through the programme endline, a severe drought occurred in Somalia, which affected 4 key proxy indicators of resilience, which all show significant declines in levels since the start of the programme. Although a reduction in resilience was observed at endline due to the drought, there is some evidence that BRCiS activities contributed to mitigating this reduction
- experience of certain shocks significantly influenced the reduced Coping Strategy Index (rCSI). Households experiencing either drought or unemployment shock reported greater reductions in rCSI than those who did not report these shocks. Over 80% of households reported experiencing at least one of these shocks, indicating that this influence on changes in rCSI generally applied across the entire programme
The impact: how recommendations were used/ lessons learnt
Lessons and recommendations from the evaluation have been presented to the BRCiS Consortium for integration within future programming and to FCDO for inclusion in future resilience work in Somalia. Selected examples of these are detailed below:
- the recommendation to review the community selection approach and selection criteria to consider broader-area based targeting and more detailed household-level targeting for specific activities (e.g. safety-net, livelihoods etc.), will be used to target relevant livelihoods interventions and to design appropriate participatory tools to ensure engagement of and outreach to the most vulnerable and socially excluded
- linked to the above, a recommendation on the standardisation of area-based community level resilience committees, a pioneered approach which improved the quality and relevance of programming and created greater levels of social capital, will be integrated in future programming and streamlined with GESI approaches and vulnerability-based targeting to enhance inclusion
- the recommendation to focus resilience programme evaluations to investigate more closely the impacts of specific activities or combinations of these activities on communities to measure resilience over time will be actioned by the FCDO MEL provider who will be responsible for recommending approaches which inform and test opportunities for the packaging and layering resilience investments. The MEL provider is further responsible for undertaking continued research and analysis into the measurement of resilience, incorporating both food security aspects (FCS, rCSI) and social cohesion and social capital aspects as recommended by the evaluation
It is also expected that the findings, lessons and recommendations from the evaluation will inform the design of BRCiS monitoring and evaluation activities and research regarding resilience in Somalia and could be applied to other complex operating environments.
6.6 Evaluation of Nigeria WFP Country Strategic Plan 2019 to 2022
Evaluation type: Evaluation of delivery of WFP’s 3-year Nigeria Country Strategy
Publication date: January 2023
Full report: Evaluation of Nigeria WFP Country Strategic Plan 2019 to 2022 – World Food Programme
The programme
FCDO Nigeria’s Humanitarian Assistance and Resilience Programme (HARP) is a flexible £150 million, 3-year programme that will provide life-saving humanitarian assistance to people affected by conflict in North-East Nigeria. HARP will be a critical part of His Majesty’s Government’s comprehensive response to the Lake Chad Basin crisis. HARP will reduce mortality and increase resilience for people with the most severe humanitarian needs by:
- addressing food insecurity and malnutrition
- protecting vulnerable people from harm
HARP will flex to respond to humanitarian needs elsewhere in the country and leverage UK technical and diplomatic leadership on the Lake Chad Basin crisis.
One of the programme’s aims is to improve food security and nutrition by:
- providing assistance (cash transfers wherever possible) to increase vulnerable people’s access to food
- increasing its focus on resilience-building in conducive areas
- being scalable, should the situation deteriorate the UK reallocate resources to response
- focusing on real-time data and participatory evidence generation
Building on lessons learned from North East Nigeria Transition to Development programme (NENTAD), we will fund the provision of cash transfers (as opposed to vouchers or emergency food rations) wherever possible, as this helps build people’s resilience to crisis. This work will build on the UK-promoted interagency Humanitarian Situation Monitoring for inaccessible areas and will strengthen real-time monitoring throughout North-East Nigeria.
The evaluation
The evaluation was commissioned by the World Food Programme (WFP)’s Independent Office of Evaluation to provide evaluative evidence for accountability and learning to inform the design of the next WFP Country Strategic Plan (CSP) in Nigeria. It covers WFP activities implemented between 2019 and 2021 and assesses continuity from the previous programme cycle, the extent to which the CSP introduced strategic shifts and implications for such shifts for performance and results.
The evaluation concluded that:
- WFP can report on major achievements as part of the first Nigeria CSP. Against the background of a deteriorating humanitarian situation, WFP managed to refocus and scale-up its efforts to respond to growing emergency needs, but this came at the expense of implementing the anticipated strategic shift towards development support. WFP was at times challenged in fully adhering to humanitarian principles and there is still room for strengthening WFP’s approach protection, accountability to affected populations and gender equality
- performance against targets was solid with the available resources in mind and WFP was capable to play its part in preventing a humanitarian catastrophe. However, significant numbers of people in need remained without assistance, due to funding shortfalls, unclear division of responsibilities between agencies, and beneficiary management challenges
- nascent initiatives in resilience building, capacity strengthening and policy coherence have put WFP on the map locally and have cemented WFP’s good reputation with national and local authorities and partners but are yet to mature in sustainability
The impact: how recommendations were used
The evaluation directly fed into WFP’s new Country Strategic Plan 2022 to 2025 for Nigeria: Nigeria country strategic plan (2023 to 2027): World Food Programme. HARP is funding Strategic Objective 1 and 5 of this plan.
-
Based on data from Devtracker publications. ↩
-
This figure relates to the primary sectoral focal area as defined by the OECD DAC, though some evaluations cover more than 1 sector. ↩
-
Several evaluations were non-specific in terms of country focus or categorised as ‘Global’. These, and those classified as regional evaluations, are not represented in Figure 2. Benefiting locations for each evaluation can be found on the relevant programme devtracker page. ↩
-
Templates were reviewed and updated by the EQUALS project director and FCDO Evaluation Unit in November 2022 ↩
-
Includes Terms of Reference, baseline, inception reports, mid-term, and end-term evaluation reports ↩
-
In February 2023, BEIS was replaced by the Department for Business and Trade (DBT), the Department for Energy Security and Net Zero (DESNZ) and the Department for Science, Innovation and Technology (DSIT). ↩
-
All documents publicly available on the OPM website. ↩
-
Bungoma, Busia, Homa Bay, Kajiado, Kakamega, Kilifi, Kisii, Kisumu, Migori, Mombasa, Nairobi, Nakuru, Nyamira, Siaya, Uasin Gishu, Vihiga. Programming in Uasin Gishu was halted in June 2021 and programming was started in Machakos. ↩
-
More information can be found at www.ukpact.co.uk ↩
-
BRCiS selected programme communities using detailed criteria to define ‘area communities’ that would receive low-intensity programming and, within those, ‘participant communities’: more vulnerable communities that would receive high -intensity programming. The intention was to enable large populations to reach and maintain moderate levels of resilience, and then target the most vulnerable to ensure they can reach these same levels. ↩
-
BRCiS cumulative reach since September 2018 – March 2022 targeted 423 communities across 34 districts in Somalia. Participant communities were selected as a cluster of geographically close communities during the inception phase; they are at the core of the programming. Area communities are located in the direct vicinity of the participant communities; while they are not benefitting from the full scope of project interventions, they are part of the catchment area for some activities and particularly benefit from basic services and early action. ↩