Last week was a tough one for our small group of IFPRI researchers in Accra. On Tuesday, we delivered a message to one of our partner organizations there that its project to promote integrated soil fertility management (ISFM) to Ghanaian smallholders did not yield any measurable impacts during the course of our two-year evaluation. Farmers were no more likely to know about, understand or apply ISFM as a result of the project, plain and simple.

But it’s never so simple, is it? Maybe the project did have sizable impacts. Our implementing partners had to make a few course corrections during the project to reach the number of farmers they had originally targeted. Unfortunately, that correction disrupted our evaluation design. In effect, we were measuring outcomes in a sample of farmers that was no longer representative of the population served by the project. So we just couldn’t detect any impacts even if they did occur.

That’s life in the impact evaluation world. Many of our colleagues already know this, but it’s worth repeating—again and again—to be better aware of the potential for such situations and to learn how to avoid them in the future.

That is what we did after delivering our disappointing message. Then we discussed our experience further with colleagues from ISSER, IPA, UDS, CABI, IFDC, AGRA and IITA at a learning event in Accra on April 25. We used this occasion to review our approaches to project implementation and impact evaluation, focusing our attention on how agricultural extension services can be leveraged to promote natural resource management (NRM) practices and principles to Ghanaian smallholders.

NRM is complicated: It encompasses a range of knowledge-intensive and site-specific management principles combined with a broad array of modern inputs and technologies. When applied successfully, good NRM principles and practices can improve and sustain the productive use of land, soil, water and biodiversity for present and future generations.

Programming and evaluating in the NRM space can be extremely complex. Here are a few thoughts, gleaned from our discussions, on how to deal with that complexity.

• Choose your unit of analysis carefully. Many of our colleagues working on NRM think intently about designing projects to achieve landscape-level impacts. But evaluators tend to focus on individuals and households as the unit of analysis because that allows for simplification from high, often abstract, levels of aggregation to something more tractable to infer causal relationships. But does that mean we are missing some of the big picture, landscape-level issues?
• Be ready with flexible designs. Rigorous evaluation designs are good and fine, but we always need to prepare for the unexpected. Changes in the content and timing of project rollout, treatment groups not being treated, control groups being treated, mass attrition between baseline and endline — these are but a few of the problems evaluators faced. Maybe we need to think small: Small field experiments at the outset of the project, such as survey-based and lab-in-field experiments, to understand farmers’ behaviors and preferences, augmented by a lot of well-thought out qualitative inquiries. These small experiments can produce useful insights at the outset of a project and are particularly useful in case the evaluation itself goes awry.
• Recognize the dictatorship of near-termism. Project implementers often want proof that their NRM project had an impact on household welfare, especially income, consumption or nutrition. But those outcomes often take time to show up in the data. And it’s surprisingly difficult to measure productivity changes in a single year or season, not only because of measurement error (e.g., this recent paper from my colleagues) but also because some technologies take a long time to generate benefits.
• Explore choices and behaviors, not technologies, per se. Efficacy trials — research station trials and farmer-managed trials — are useful in testing whether a technology works. In these trials, we hold farmer behavior constant and vary the inputs, practices and technologies. When we move into effectiveness trials, we are ultimately more interested in whether efforts to promote technologies are successful and how the input/practice/technology affects farmers’ choices and behaviors. Do they invest more in complementary inputs, do they expand area under cultivation, do they work harder? In a sense, we want to elicit reactions to a treatment and don’t really care about disentangling those reactions from treatment itself. (see this paper and a related blog for interesting perspectives on the topic.)

A negative evaluation is often an implementer’s worst-case scenario. It may give a government or donor a reason to pull project funding. But a good evaluator can help cushion the blow by highlighting lessons learned, encouraging innovation and opening opportunities for further evaluation. The reality is that many implementers are too afraid to experiment or too afraid to subject their experiments to rigorous evaluations. But learning from failures is as important as learning from success, as the growing popularity of fail festivals suggests.

By David J. Spielman, Kwaw Andam and Simrin Makhija

David J. Spielman is a Senior Research Fellow with IFPRI's Environment and Production Technology Division (EPTD). Kwaw Andam is a Research Fellow with IFPRI's Ghana Strategy Support Program. Simrin Makhija is an EPTD Research Analyst. The post first appeared on IFPRI’s website here.

The learning event in Accra was organized by IFPRI and its partners in Ghana. The event was generously supported by the International Initiative for Impact Evaluation (3ie); the U.S. Agency for International Development (USAID) Feed the Future Developing Local Extension Capacity (DLEC) project, led by Digital Green; the CGIAR Research Program on Policies, Institutions and Markets (PIM); and the Alliance for a Green Revolution in Africa (AGRA).