Revolutionizing Impact Monitoring Through Earth Observations Data and Deep learning: CIAT’s System for Evaluation of Impact Using Remote Sensing (SEIRS)

May 10, 2019

This post was written by Louis Reymondin and Andres Cubides with the Center for International Tropical Agriculture, Cali, Colombia, and Peter Richards and Baboyma Kagniniwa with USAID.

Estimating the impact of agricultural projects and policies is a vexing problem for many development professionals. On the one hand development professionals often declare ‘success’ if yields increase; on the other hand, they often blame the weather if yields decline. The International Center for Tropical Agriculture (CIAT) and USAID are piloting an alternative approach to impact monitoring. Our pilot accounts for weather conditions. At the same time, we are using only methods which are highly scalable, implementable at a relatively low cost, and which rely on freely available data. Fortunately, recent innovations in economics (including the growing use of so-called synthetic counterfactual variables), remote sensing, and new tools for deep learning, are making this all possible.

Deep Learning, Earth Observations Data, and Estimating Impact

A little over a decade ago, CIAT researchers were tasked with developing a method to identify deforestation, in near real time, using only freely available earth observation (EO) data and flexible enough to work anywhere and anytime in the world. CIAT began this project with the hypothesis that they could predict forest vegetation for any location, at any point in time, from any location’s own historical trends, and the weather conditions under which they occurred. When the observed values (negatively) diverged from the predicted outcomes, we assumed that it was because the forest cover had been removed. The process worked so well that it eventually developed into Terra-i, a first of its kind system for near-real time forest loss detection that is now widely used for environmental monitoring.

Today, we continue to build on the foundational hypotheses of Terra-i, and the concept that the best way to identify changes in dynamic environments is to continuously generate counterfactuals for comparison. Under the USAID-supported System for Estimation of Impact through Remote Sensing (SEIRS) activity, we are applying this same concept to identify and estimate the impact of development programs.

SEIRS rests on the concept that good agricultural yields are generally correlated with more vegetation during the growing season. If development programs are leading to better yields and more vegetation, then the effects, in theory, could be observed from space. To observe these effects, however, we needed to generate a counterfactual outcome which could account for the weather conditions of a growing season. Yet we also needed a way to do this that would be transferable over a wide range of geographies and farming systems. This isn’t an easy problem. Developing country farming systems are highly diverse in both their physical and social environments, and exhibit very different growing outcomes and growing conditions. Fortunately, recent advancements in computing are making problems like these increasingly surmountable.

Observed vs. Synthetic Vegetation data on four farm plots in northern Senegal

Parcel level vegetation trends in the observed (in blue) and concurrent, CNN-generated synthetic trends (red) for several parcels targeted by a USAID-funded development activity. The CNN ‘learns’ from the patterns observable within individual pixels before 2012, then predicts outcomes, at a pixel level, based on historical observations (in this case pre-2012) and weather conditions. The differences between the observed values and the synthetic trends can indicate changes in management. In the top two rows farmers shifted to a dry season planting patten; in the third (from the top) row, a new area was brought into production, in this case after improvements under an MCC compact; however, since initial usage, production has been intermittent; the bottom row is a low intensity cropping system which has been planted erratically in recent years.

One class of deep learning models, convolutional neural networks (CNN), are particularly suited for learning complex and often highly diverse patterns and relationships. Under SEIRS, using only historical, pre-development intervention EO data, we are able to train a CNN to learn vegetation patterns and their relationship to growing season conditions. We could then program the CNN to predict an entire stream of synthetic vegetation data (23 observations per year), for hundreds of thousands of agricultural fields, with each individual prediction adjusted to current and preceding growing conditions. This ultimately gives us what amounts to a synthetic dataset, a CNN-generated stream of EO data that mirrors, in structure, resolution and frequency, the actual, observed libraries of satellite data. Simply put, this synthetic dataset tells us what we should be seeing, given the local weather conditions, if we were to assume that landowners management structures were to have remained stable.

The differences between the observed data and the CNN-generated synthetic data offers an unprecedented perspective on how agricultural systems are changing. When development professionals can tie pixels to their programs, these differences can also offer insights into where, or to what extent, specific programs or policies might be having an effect. It could help extension providers spot innovations, help farmers accelerate adaptation to changing climates, and shed light on where agricultural practices demonstrate resilience.

After tying thousands of location points targeted by U.S. Government-funded development programs, we were able to estimate program impacts in key areas of interest across Sub-Saharan Africa. In one flagship program in northern Senegal, we estimated that target farms were associated with, on average, a six percent increase in productivity each year since the program was rolled out in 2016. Nearby, we were able to not only estimate the effects of a Millenium Challenge Corporation Compact, but estimate the effects associated with specific target areas. In Nepal, we used SEIRS to identify and estimate declining agricultural investments in the country’s hill region, and to generate measures of resilience and yield losses in the wake of the 2016 earthquake. In Niger, we are now deploying this same approach to identify the impact of resilience programming. New and interesting applications for SEIRS continue to be discovered.

Differences between observed and synthetic yields in Senegal, Niger and Nepal. In Senegal, many of the ‘best’ performing areas were located in the country’s Senegal River Valley. The Senegal River Valley has been home to a series of development programs in recent years, including the USAID-funded Naatal Mbay activity.

The EO Advantage for Impact Monitoring

EO data and new analytical methods have the potential to change how we monitor agricultural trends or assess the impacts of development programs. SEIRS was designed to take full advantage of the strengths of both EO data and the emerging class of deep learning algorithms which are likely to revolutionize their usage in the coming years. The advantage of SEIRS is that it can be applied anywhere, including in areas deemed too difficult to reach, or in areas which might otherwise be too dangerous for traditional impact assessment tools. For development professionals wishing to identify the impact of their programs, the only thing they will need is the location and timing of their activities.