Research and development stakeholders working on sustainable
intensification of agro-ecosystems are striving to become more effective
in achieving development outcomes. Key questions facing them are:
- How to evaluate alternative research and development strategies in
terms of their potential impact on productivity, environmental
services and welfare goals, including trade-offs among these goals?
- How to cost-effectively measure and monitor actual effectiveness of
interventions and general progress towards achieving sustainable
The purpose of this review was to identify lessons and opportunities for
the derivation and use of data from monitoring initiatives in the
sustainable intensification of agriculture. The ultimate goal is to
provide decision-makers with tools that they can use to explore
trade-offs between food security, environmental and socio-economic
goals. The analysis is intended to inform the development of any future
DFID research investments and engagement with stakeholders in this area.
The first step was to identify key initiatives in data monitoring
systems relating to agriculture, paying particular attention to those
that also acknowledge the impact on ecosystem health, and/or poverty and
well-being. A total of 103 monitoring initiatives were identified. The
second step was to review the identified initiatives with respect to
their degree of achievement in meeting a set of 34 criteria that had
been established from a general literature review. All initiatives were
evaluated with respect to their conceptual framework and a subset of 24
initiatives was screened against the full set of criteria. Pertinent
additional findings from the literature on monitoring of data in other
fields were also considered. Based on this information a gap-analysis of
the systems, indicators and metrics was conducted identifying strengths
and weaknesses in methodology and use. Insights, lessons and
recommendations were then drawn. Preliminary findings were shared with a
group of experts and stakeholders identified in consultation with DFID
and their feedback incorporated.
Common weaknesses were identified in a number of areas. Many initiatives
lack a clear conceptual framework that could demonstrate an
understanding of the system under study. In particular theories of
change on how the monitoring results would affect behaviours and
explicit linkage to specific decisions are lacking in most initiatives.
There are problems in most initiatives in defining the target inference
space (geography, population) and how that is being sampled. Very few
initiatives have come to terms with how to integrate biophysical and
socio-economic indicators and sampling frames. A common problem is the
lack of well-defined sample. Use of statistically sound study designs to
allow attribution of outcomes to interventions is still rare. A lack of
consideration of uncertainties both conceptually and in communicating
results is pervasive. Similarly few initiatives have tackled trade-offs
among objectives. Data sharing agreements were found to be wanting.
Evaluation of monitoring initiatives themselves is lacking and there is
virtually no information on cost-effectiveness of the measurements or
the impact of the initiatives. Few initiatives have been sustained over
the long term pointing to inadequate consideration of institutional
An over-riding lesson is the surprising lack of evidence for the impact
of monitoring initiatives on decision-making and management. Useful
insights were drawn from public health surveillance, systems thinking in
industry and public services, and decision sciences. A set of
recommendations for good practice in monitoring initiatives is given and
opportunities for new thinking in monitoring design are identified,
including a decision analytic conceptual framework.
As well as the review, some additional files are also appended:
Appendix 2 - Two Excel spreadsheets giving the scoring of the
individual monitoring initiatives against the criteria.
Appendix 5 - Applied Information Economics. Two pdfs - The Need for an
Intervention Decision Model, and Applied Information Economics Example.
Shepherd, K.D.; Farrow, A.; Ringler, C.; Gassner, A.; Jarvis, D. Review of the Evidence on Indicators, Metrics and Monitoring Systems. World Agroforestry Centre (ICRAF), Nairobi, Kenya (2013) 94 pp.