This report presents the findings of a study undertaken by Itad for DFID
to critically review the methods used to evaluate Making Markets Work
for The Poor (M4P) programmes and thereby help guide the design and
implementation of future evaluations.
The review compiled information about the scope and purpose of
evaluations of M4P programmes and analysed the trends, strengths, and
weaknesses of these. The reviewers analysed 32 M4P programme reviews and
evaluations, monitoring and evaluation (M&E) frameworks and
programme reports, including 14 evaluation reports, and consulted with
14 individuals and organisations in the field.
M4P programmes are defined as playing a facilitative, adaptive role in
order to contribute to systemic, large scale and sustainable market
changes that positively affect the poor. The nature of the approach and
the complexity of the markets within which it operates present a number
of challenges for evaluation. Evaluation approaches that address these
challenges are needed to avoid inaccurate estimations of impact.
The M4P evaluations reviewed were generally weak in terms of:
- consideration of systemic, sustainable changes in market systems
- data quality (small sample sizes with little consideration of sampling
frames, statistical significance or bias)
- triangulation practices (particularly with regard to qualitative data
- the use of theories of change (those used were often linear, not
externally vetted, with assumptions not adequately tested)
- consistency in units for facilitating accurate aggregation
- consideration of unintended negative effects.
A number of recommendations are made with regard to the evaluation method (how), evaluation timing (when), evaluation responsibilities (who) and evaluation level (what).
Ruffer, T.; Wach, E. Review of Making Markets Work for The Poor (M4P): Evaluation Methods and Approaches. DFID Working Paper 41. DFID, London, UK (2013) iii + 57 pp.
Review of Making Markets Work for The Poor (M4P): Evaluation Methods and Approaches. DFID Working Paper 41.