Policy paper

FCDO evaluation policy

Updated 15 September 2023

1. Introduction

Purpose of the evaluation policy

This Foreign, Commonwealth and Development Office (FCDO) evaluation policy ensures a common understanding of evaluation principles and the minimum standards that support the principles. It provides- Applies to any evaluation undertaken by external evaluators, including partner-led.a mechanism to promote, monitor and enforce minimum standards on ethics and quality in evaluation. It demonstrates explicitly that FCDO is upholding our cross-government and international commitments to high quality, objective and transparent evaluation. Relevant, high-quality evaluation enables us to have a greater impact from our policies and interventions, and ultimately on affected populations.

The FCDO evaluation policy complements the Programme Operating Framework (PrOF). It covers standards required in evaluations, including evaluations which sit outside of programme spend. Evaluation is not (and should not be) mandatory on every programme, policy or intervention, and evaluations may cover areas which are not linked to programme spend. However, where an evaluation is conducted by, commissioned or supported by FCDO, it is expected that it will meet the principles and standards outlined in this document.

This document is complemented by the FCDO Evaluation Strategy, which outlines how we will drive the quality and use of evaluation, and how we will guide what is evaluated in FCDO. It was developed to align to UK government and international standards and informed by a review of policies and strategies of peer organisations.

The Evaluation Unit (EvU) is responsible for overseeing and supporting adherence to the policy. The EvU will also review the Evaluation Policy annually, to respond to ongoing organisational changes and progress delivering against the FCDO Evaluation Strategy.

UK government commitment to evaluation

The UK government has made a strong commitment to evaluation.

The Evaluation Task Force (ETF), part of the Cabinet Office, was established in 2021 to drive continuous improvements in the way government programmes are evaluated in order to inform operational and spending decisions. The ETF holds departments to account on both priorities for evaluation and ensuring the transparency of the government’s evaluation work. Through the Spending Review settlement, they are also ensuring departments act upon the findings of the 2021 National Audit Office review on evaluating government spending,^{[footnote 1]} which found inconsistency in how evaluations were conducted across departments, and a lack of coherent evaluation strategies for individual departments.

The Government Social Research publication protocol,^{[footnote 2]} updated in December 2021, outlines five principles to ensure government social research products (including robust evaluations) are made publicly available, are well communicated and are trustworthy.

The FCDO has specific scrutiny and accountability obligations as an Official Development Assistance (ODA) spending department. The 2015 International Development Act ^{[footnote 3]}. includes the following clause: the Secretary of State must make arrangements for the independent evaluation of the extent to which ODA provided by the United Kingdom represents value. In addition, OECD Peer Reviews assess donor organisations’ approach to results, evaluation and learning, and outlines that “aid agencies should have an evaluation policy with clearly established guidelines and methods and with a clear definition of its role and responsibilities and its place in institutional aid structure ^{[footnote 4]}. Finally, the International Commission for Aid Impact (ICAI) scrutinises UK aid spending, including our approaches to monitoring, evaluation and learning.

Defining evaluation

The FCDO Evaluation Policy applies to all areas of FCDO’s activities, such as evaluations of Official Development Assistance spend, international programme spend and policies or workstreams not linked to spend. It applies to internal and external evaluation,^{[footnote 5]} as well as that led by partners. The degree of compliance to the standards will depend on the subject and evaluation type and approach: see details in section 3.

Based on the government’s Magenta book (HM Treasury guidance on what to consider when designing an evaluation) definition ^{[footnote 6]} and the OECD Development Assistance Committee (DAC) Network on Development Evaluation,^{[footnote 7]} we define evaluation as:

A systematic and objective assessment of the design, implementation and outcomes of an intervention, programme or policy, or a portfolio of interventions, programmes or policies in a particular area or theme. It can involve understanding the process behind implementation, the effects or outcomes, for whom and why. It identifies what can be improved, and where appropriate, can estimate the overall impacts and cost-effectiveness.

Evaluation can be conducted by internal or external experts, depending on the most appropriate tools for the information need. As per the above definition, an evaluation is so classified in FCDO if it is:

1. Systematic: uses recognised robust and replicable methodology which is appropriate to the evaluation question.
2. Objective: includes a level of independence from the strategy, intervention or policy in question.

The principles and standards in this document apply to any FCDO evaluative activity which meets these criteria.

When to evaluate

Evaluation is a significant part of the FCDO’s approach for using evidence more effectively to achieve maximum impact from its policies and interventions. It complements FCDO’s research investments and its work to strengthen the way we measure and understand results across the organisation, such as through evidence-based theories of change. It supports the requirement of decisions and programmes to be based on evidence^{[footnote 8]}. Evaluations can be applied to programmes, policies, portfolios, themes or strategic areas, to improve our efficiency, effectiveness, impact and value for money. Evaluation includes a wide range of tools to meet different information needs and purposes.

Evaluation may be undertaken for learning and/or accountability purposes, and often these two purposes overlap. Evaluation for learning can include: managing risk and uncertainty; improving current actions or interventions by providing the evidence to make better decisions; gaining a general understanding of what works, for whom and when, as evidence to inform future interventions and generate examples for future policy-making. Evaluation for accountability can include: generating evidence of how an intervention has worked, for example its efficiency, effectiveness or impact; or, for openness, scrutiny and challenge. Ultimately, evaluation seeks to understand and improve our work and our impact on the ground, and therefore increase the value of each pound we spend. Further guidance and criteria on when to evaluate are available in the Evaluation PrOF guide.

Evaluation and its tools should be incorporated into every stage of the policy and programme cycle. Evaluations are part of the evidence toolbox, and should be used proportionately alongside other tools for evidence generation, learning or accountability. For example, monitoring, learning reviews, audit reviews and annual reviews / programme completion reviews, which may be used for compliance, performance management, results tracking and light-touch learning. Evaluation differs from these tools and processes because of the expected quality standards, methodology and levels of objectivity, and the scope of the subject which is being assessed.

2. Principles

All evaluations conducted by FCDO and its partners should strive toward the following guiding principles:^{[footnote 9]}

Useful

• fills identified evidence gaps, responds to FCDO and Director General level plans and priorities
• provides clear opportunities to influence change, and findings are timed to inform internal or external decision making
• asks useful, well-defined questions that are feasible to answer
• considers use of evaluation findings throughout design, implementation and beyond finalisation of the evaluation, including recommendations which are actionable and have practical implications for adaptations
• engages internal and external stakeholders, provides opportunities for mutual learning, representation of different groups and recognition of context, and to inform decision making for other stakeholders, e.g. partners on delivery / funding / diplomacy, global communities
• works with evaluation respondents, ensuring participation in processes as appropriate and strong mechanisms for feedback

Credible

• appropriate level of objectivity, through commissioning independent evaluators, and/or through involving independent figures in steering and peer reviewing both the design and outputs of the evaluation
• seeks to represent the diversity of people that FCDO’s work is designed to benefit and involve them in evaluation work where feasible
• processes and products are transparent to the extent possible, and teams are accountable for the findings and related follow-up actions
• aligns with partnership principles as relevant, for example commitments in the Paris Declaration, Accra and Busan Agreements and Sustainable Development Goal 17’s aim to strengthen the means of implementation and revitalise the global partnership for sustainable development

Robust

• applies approach and methodology that are feasible and appropriate, reliable and replicable, and stand up to independent scrutiny from design stage
• engages right expertise in design, implementation, analysis and quality assurance
• aligns with best practice in evaluation quality standards, such as the Magenta book and relevant DAC criteria, quality standards and principles
• design and implementation take into account contextual factors, including conflict and Gender Equality and Social Inclusion (GESI) dynamics
• suitable management and governance structures in place

Proportionate

• level of investment in evaluation is aligned to level of scrutiny required, type of learning needed and availability of resources
• evaluation approaches meet the learning need of the evaluation, in terms of scope, budget, timeline, participation, context and questions to be answered
• experimental and quasi-experimental evaluations are used only where it is feasible to apply this type of method, on a high priority intervention or topic, and where there is a clear evidence gap and need for new rigorous studies
• demonstrates value for money

Safe and ethical

• considers balance of benefits and risks and takes action to mitigate risk
• aligns to FCDO safeguarding rules
• contributes to strengthening availability of disaggregated data and evidence, in line with the Inclusive Data Charter
• ensures respect and dignity of people affected by the subject of the evaluation, their equitable participation in the evaluation and the dissemination of findings to them. Follows best practice in ethical research, as outlined in FCDO Ethical Research, Evaluation and Monitoring guidance ^{[footnote 10]}
• management of data adheres to GDPR rules or equivalent

3. Standards

All FCDO evaluations and FCDO-funded evaluations that are led, managed and commissioned by partners should strive to comply with the principles outlined above. The following minimum standards of compliance for evaluation are in place to help achieve the principles outlined in section 2, upholding a high quality in FCDO evaluations and maximising learning. The standards are focussed around (1) publication and (2) independent quality assurance.

At a minimum, and recognising the cases for application and exceptions outlined in the table, the standards must be adhered to when commissioning an external evaluation or providing funds towards a partner-managed and commissioned external evaluation. However, all FCDO evaluations (including internal) should aim to uphold these standards, in a proportionate and feasible manner, and should justify where they have not been applied.

It is not mandatory for monitoring activities to meet these standards although it is recommended as good practice where possible eg seeking quality assurance for Terms of Reference for Third Party Monitoring (TPM) contracts or publishing TPM outputs where the focus is largely on impact.

Process	Purpose	Application/ exceptions	Responsibility
1. Publication a) of evaluation reports - Programme team (publication of reports, developing management response) b) of management response & follow up c) of data in alignment with open data standards, wherever possible	* Provide accountability to general public for taxpayer spend * Maximise learning for FCDO and others in the sector; ensure learning incorporated into planning & policy development including after evaluation has ended. * Uphold reputation as a learning institution * Ensure senior management ownership of recommendations and associated actions * Maximise VfM of evaluation	* Applies to any evaluation undertaken by external evaluators, including partner-led * Applies to all end-line reports and is encouraged for mid-line reports. * Exceptions apply for sensitive and poor-quality evaluations – see Annex 1 * Management response, which should include levels of acceptance of recommendations, and related actions and responsibilities. This should be published with the report unless the report is exempt from publication – see Annex 1	* Programme team (publication of reports, developing management response) * Senior leadership (approval of management response, follow-up of actions) * Head of Evaluation Unit and relevant Director General (approval for exemptions – see Annex 1).
2. Independent quality assurance a) Terms of Reference b) Inception c) Final reports	* Check for feasibility and rigour in design * Check for quality in implementation and analysis of evaluation, and presentation of results * Ensure the intended audience is clearly stated in the Terms of Reference and there is a strategy for supporting uptake * Ensure compliance with safeguarding and ethics * Ensure evaluations are commissioned with clearly defined methods, use and quality standards and strong governance arrangements * Ensure evaluations do not get published where they do not meet quality standards or contain misleading content	* Applies to any externally sourced evaluations, including partner-led * Terms of References should be quality assured before procurement * Contracts should specify that evaluation products must pass FCDO quality assurance process prior to payment * All reports must be quality assured before publication * Teams should use the EQUALS2[footnote 11] service for all quality assurance; partners can use alternative independent quality assurance system, if considered of sufficient independence and quality	* Programme team responsible for submitting for quality assurance and signing off final evaluation TORs and reports * EvU manage independent quality assurance providers (EQUALS2) and check for compliance, unless using alternative partner-led system * If using a partner-led system, programme team responsible for confirming it meets standards; EvU available to support this as required

4. Support and services to support principles and standards

The Evaluation Unit

The Evaluation Unit, part of the Economics and Evaluation Directorate, has responsibility for promoting a strategic approach, setting evaluation standards and overseeing FCDO’s performance in meeting them, providing targeted technical support for evaluations across the organisation, learning about and promoting innovative methods and approaches, developing evaluation competencies and performing quality assurance. The team includes Evaluation Advisers, Data and Insights specialists and Programme Managers. The Evaluation Unit conducts central evaluations and manages evaluation programmes, including the Evaluation Quality Assurance and Learning Service (EQUALS) 2.

Embedded Monitoring and Evaluation Advisers

Expert Advisers specialising in evaluation, monitoring and results are embedded at post or in central teams, many of whom are formally accredited in a rigorous process aligned with the cross-HMG Government Social Research profession. Evaluation Advisers have a unique blend of strong analytical skills and deep experience in embedding evidence and learning into policy and programming decisions, thereby ensuring maximum benefit and value for money from HMG investments.

Evaluation Quality Assurance and Learning Service (EQUALS) 2

EQUALS2 conducts independent quality assurance of FCDO’s evaluation and monitoring products, at no additional cost to teams. It also provides evaluation and monitoring technical advice, e.g. evaluability assessments; participation in evaluation governance structures; theory of change development. It comprises a panel of over 100 evaluation experts across the range of thematic areas and evaluation types. Evaluation products are quality assured against a template developed with FCDO, based on the principles outlined above; with the reviewer providing feedback on how the product can be improved. For further information on how to access EQUALS, please contact the Evaluation Helpdesk.

Evaluation Procurement Frameworks

The evaluation procurement frameworks provide FCDO and other government departments access to a panel of pre-qualified suppliers to procure evaluation and monitoring services, reducing the lead times required to mobilise FCDO-funded evaluation and monitoring work. Commissioners can have confidence in the quality of the suppliers as they were subject to robust assessment from members of the Evaluation Unit in order to win a place on the Framework. The Global Evaluation and Monitoring Framework agreement launched in February 2022 and is available to use.

Helpdesk and reporting concerns

The Evaluation Unit provides a helpdesk service (evaluationhelp@fcdo.gov.uk), in which Evaluation Advisers are on hand to provide FCDO staff with support to questions, issues or problems on evaluation design, procurement and implementation.

Where suppliers, partners, beneficiaries, FCDO staff or others have concerns that independence or standards on ethics or safeguarding may have been compromised, or concerns about misuse of funds, this should be reported immediately to FCDO’s Reporting Concerns inbox or through the confidential reporting hot line (reportingconcerns@fcdo.gov.uk / +44 (0)1355 843 747).

Guidance

Overarching guidance can be found through the Magenta book and annexes. FCDO staff seeking guidance or support on particular evaluation methods or approaches, or templates for terms of references, evaluation reports or management responses, should contact the Evaluation Unit.

Annex 1: Publication exemptions process

FCDO’s policy is to make information transparent as standard, excluding it from publication only where there is a reason based on sensitivity of the content or concerns about methodology. This is aligned to the GSR Publication Protocol,^{[footnote 12]} and for programme funds, FCDO PrOF Rule 7. In line with this, the Evaluation Policy states that all FCDO evaluations should be made publicly available and actively disseminated. Programme teams should ensure that these documents are published as a rule and that they meet the standards outlined in the document publication guidance. Heads of Department have overall accountability for the quality of publications.

However, if evaluations contain sensitive information, it may be inappropriate to publish the full content of an evaluation. Sensitive information includes situations where publishing information could put at risk the safety and security of staff, beneficiaries or evaluation respondents, or where information is security related, political or diplomatic, industry contentious, controversial or high interest, personal or private, or commercially sensitive. The final decision to withhold information from publication on grounds of sensitivity must be signed off by a Director General.

If an evaluation is deemed irredeemably flawed or inaccurate, it may need to be ‘downgraded’ and/or have a warning to readers about the quality, but it may still be published. This includes cases where methodology is poor and results are unlikely to be accurate, presenting risks of misleading readers and damaging the credibility of our evaluation efforts. Assessment of the rigour must be conducted independently (e.g. through EQUALS2). Final decision to withhold information from publication, or downgrade an evaluation to a review, on grounds of methodology must be signed off by the Head of the Evaluation Unit.

The decision for all publication exemptions will be documented and saved in the permanent record. These protocols apply even where contracts have ended or been terminated early.

Sensitive information

Sensitive information must be handled appropriately and only excluded from publication as a last resort. Wherever possible, evaluation findings should be published, with any sensitive information handled through anonymising personal or commercial information, or by including a “Confidential Annex” for sensitive information separate to the main, published evaluation report. Such an annex is not published and is given a higher security classification, providing teams can demonstrate that the reasons for such an annex clearly reflect current exclusion criteria. Sensitive material includes:

1. Safety and security: are there security, safety, safeguarding reasons why certain organisations, individuals or activity should not be named?
2. Political: are there current political issues either internationally or domestically which make the documents particularly sensitive? This, in itself, may be insufficient to prevent publication, but may present a need for more detailed publication plans.
3. Personal information: do the documents compromise the privacy of an individual, or could they contravene confidentiality?
4. Commercially sensitive information: does the information risk doing harm to FCDO or our partners’ or suppliers’ commercial interests?

Assessment of suitability to publish, or remove certain information prior to publication, must be informed by the Government Security Classifications.

Teams seeking to exclude all or part of an evaluation report from publication must inform the Head of the Evaluation Unit for advice on next steps. Final approval to withhold publication must be obtained from the relevant Director General. Decisions will be documented for transparency.

Methodological rigour and quality

Methodological rigour and quality must be independently assessed.

There should be a pause on publication of evaluations which have scored a red (unsatisfactory) rating through the Evaluation Quality Assurance and Learning Service (EQUALS) 2; they need to be re-submitted to EQUALS2 and published once they have scored an amber (fair) rating or above ^{[footnote 13]}.

If, after at least 2 consecutive EQUALS reviews, the evaluation achieves an overall rating of amber (fair), but retains red (unsatisfactory) ratings on the following key criteria relating to methodological rigour, the team should seek advice from the Head of Evaluation Unit on appropriate next steps:

• evaluation methodology and design: the evaluation methods are described and justified. These methods are appropriate for addressing the evaluation questions
• evaluation methodology and design: any methodological limitations are acknowledged and their impact on the evaluation discussed. The limitations are acceptable and/or they are adequately addressed
• evaluation methodology and design: the sampling strategy is described and is appropriate. Primary and secondary data sources are appropriate, adequate and reliable. Sample sizes are adequate
• implementation: the evaluation process was transparent enough to ensure its legitimacy
• analysis: evidence can be traced through the analysis and into findings and recommendations. There is sufficient cross-referencing
• findings: the evidence is clear and sufficiently triangulated

Independence: Overall EQUALS rating of 4 independence indicators (Differences of opinion (within the evaluation team, or amongst stakeholders consulted) are fully acknowledged in the report; Any conflicts of interest are openly discussed; The report indicates whether the evaluation team was able to work freely and without interference; and Information sources and their contributions were independent of other parties with an interest in the evaluation.)

There may be situations where a team does not wish to publish an evaluation on quality grounds even where it has received a satisfactory rating through EQUALS, or where it does want to publish an evaluation which has received unsatisfactory (‘red’) ratings. In these cases, relevant teams, in consultation with an evaluation adviser, will need to contact the Head of Evaluation Unit requesting publication or non-publication of a particular evaluation.

The Head of the Evaluation Unit (consulting Government Social Research (GSR) Head of Profession where appropriate to ensure GSR compliance) will review the evaluation report. The report may need to be ‘downgraded’ to a review and/or have a warning to readers about the quality, but wherever possible, will still be published. More complex cases will be escalated to the Director of the Economics and Evaluation Directorate, in consultation with the Chief Scientific Adviser and Chief Analyst as required. Decisions will be documented for transparency.

1. https://www.nao.org.uk/report/evaluating-government-spending/ ↩

2. https://assets.publishing.service.gov.uk/government/uploads/system/uploads/attachment_data/file/1041569/2021-GSR_Publication_protocol_v3_Final.pdf ↩

3. https://www.legislation.gov.uk/ukpga/2015/12/pdfs/ukpga_20150012_en.pdf ↩

4. Evaluating Development Co-Operation: Summary of Key Norms and Standards 2010 ↩

5. Internal evaluation is that which is conducted by personnel within FCDO (a level of independence and objectivity should still be maintained). External evaluation is where a supplier is contracted to conduct the evaluation. ↩

6. HM Treasury (2020), Magenta Book: Central Government guidance on evaluation https://www.gov.uk/government/publications/the-magenta-book ↩

7. OECD Glossary of Key Terms in Evaluation and Results Based Management (2002) https://www.oecd.org/dac/evaluation/2754804.pdf ↩

8. FCDO PrOF include a Principle to be Evidence-based; the Civil Service Code includes a value of objectivity, that is ‘basing your advice and decisions on rigorous analysis of the evidence’. ↩

9. These are aligned to the HMG Magenta book general principles but include additional detail to respond to the specific requirements of FCDO’s work. ↩

10. FCDO Ethical Guidance for Research, Evaluation and Monitoring Activities - GOV.UK (www.gov.uk) ↩

11. EQUALS (Evaluation Quality Assurance and Learning Service) 2 is a programme managed by the Evaluation Unit which provides independent quality assurance on evaluation products. ↩

12. https://assets.publishing.service.gov.uk/government/uploads/system/uploads/attachment_data/file/1041569/2021-GSR_Publication_protocol_v3_Final.pdf ↩

13. The EQUALS quality assurance process assesses 69 indicators to judge the quality of the evaluation report. ↩