On the wrong path: CGIAR Strategy and Results Framework

Conclusions

The framework of the SRF (not the practical proposals in the second half) is naïve, over-ambitious, and not grounded in the large theoretical and empirical literature of how advances in science, innovation and adoption processes work (for a review of this literature and further references, see Ruttan, 2001). Within the document there are internal contradictions; and doubts about its implementability that have not been followed to their logical conclusions. It will be very difficult to adapt the CRPs to the SRF, or to use the SRF to review a CRP. Trying to impose it on CRP preparation and implementation will lead to un-necessary overhead costs for research planning and coordination. I found the evaluation guidelines of the ISCP more useful to the task I was given of evaluating CRP2. By and large found the CRP2 document a realistic and appropriate approach to the enormous research agenda of policies, markets and institutions, even though in the end it fails on a number of the ISCP criteria and of course cannot even approach to reach the unnecessary and unattainable standards of research program design of the SRF.

Analysis of the SRF framework
As I understand it, the SRF Framework (up to and including section 3.2 on organizing core competencies) advocates for three major approaches:
1. It uses the fairly new paradigm of AR4D to link the all CGIAR research (other than cross cutting functions and capacities) to four system level outcomes (SLOs) namely reduction of rural poverty, increase in food security, improving nutrition and health, and more sustainable management of natural resources. It wants to hold not just the centers, but the entire CGIAR responsible for achieving these outcomes (para 82). Positive and negative impact pathways are to be specified explicitly in advance, and quantified as far as possible.
2. It wants to organized most of CGIAR research activities (except cross cutting functions and capacities) in an integrated way across disciplines, centers, and partners to achieve the SLOs (rather than clearly specified intermediary outputs and impacts).
3. And it wants to involve all stakeholders in the design of most research programs.
In subsequent sections the SRF comes down to more practical proposals. These are quite sensible and I have few problems with them. Upon reflection, most, if not all of the practical proposals of chapter 4 could have been made, and indeed have been made, with little or no reference to point 1 of the SRF above. It is as if two entirely different teams had drafted the framework and the practical proposals.
As a new paradigm, AR4D only is only about 5 years old. Advocacy has been loud and widespread. Unlike for scientific paradigm shifts, opposition has been primarily in the form of behind the scenes grumbling of those on which it has been imposed. It has not been subject to scientific testing and scrutiny for its effectiveness. Indeed, the limited literature that I have seen (e.g. Conway and Waage, 2010) provides at best some cases of impact, but not any examples of success in achieving SLOs by organizing the research ex ante according to AR4D, with clear specification of impact pathways and ex-ante quantification of impacts, and in an integrated fashion involving all stakeholders in research priority setting and design. Of course this is impossible, because AR4D has been around for such a short period. More disturbingly it is being imposed on the entire system as the ruling paradigm without any such tests. The SRF does not even propose that such tests be conducted, and subject to both ex ante and ex post impact evaluation. Thus AR4D looks more like a fashion, than a serious attempt to shift a scientific paradigm.

Finally, the SRF makes the following breathtaking suggestion: “This marks a fundamental shift from individual Centers being responsible for impact—arising essentially from the research outputs developed around particular mandates—to the CG system itself being responsible for impact.” (para 82). What will happen to the CGIAR if its donors judge it to have fallen short on impact? Replace the whole system by something else, rather than just one center, or one CRP, or one line of research within it?

Of course this does not mean that I am not advocating attention to impact on the SLOs. During all my research life I have strived for impact, especially on poverty reduction and food security, and to a lesser extent on sustainable management of natural resources. I have paid attention to the pathways in which such impacts would be achieved. And I have had some successes. But I would never have thought it possible to specify ex-ante the entire set of likely impacts and the pathways by which impact would be achieved, or tried to quantify them ex ante.
The pioneers of the Green Revolution may have worked on rust resistance of wheat, or on short-stemmed and fertilizer responsive rice, but they did not do so out of love for rust research or short stemmed rice, or for their scientific glory, but because they expected an impact on food security at the national and global level. They did not explicitly spell out the pathways of the impact, let alone all of the impacts, nor try to quantify them ex-ante. Hundreds of high level research years have done so ex-post, and there are still disputes about the impacts. Not in their wildest dreams would the pioneers have been able to imagine that their research would have the profound poverty reducing direct and indirect impacts at a global scale, or anticipated all the pathways through which the various impacts were achieved.
But let us take a more recent example: Nobody who worked on BT cotton was ever asked to spell out how exactly BT cotton would impact on poverty and sustainable natural resources management (via the reduction in pesticide needs); via which pathways; and where. Yet it has perhaps been the most important impact of research on rural poverty that we have seen in South Asia since the Green Revolution. Nor did those who worked on and with BT cotton ever try to integrate all the research capacities ex ante or involve all the stakeholders in research priority setting and research design. Integrating other actors happened ex post, after the key scientific breakthroughs and the strategic research leading to incorporation of the gene in breeders’ lines that then could be taken up by private sector operator who nobody coordinated to do so.
The SRF advocates for AR4D, integration, and involvement of stakeholders based not on an empirically validated theory of scientific and applied technology innovation. Indeed it does not even spell out the assumptions it makes about these or review and refer to the relevant literature. If the SRO had been informed by the vast literature on how scientific breakthroughs are achieved, how the research frontier shifts, how this generates basic and strategic innovations, how these are then translated into technologies through a multiplicity of actors, and finally how they are adopted and produce impact, it would most certainly not have come up with the proposed grandiose ex-ante framework. It would instead have emphasized that scientific advances are highly contingent and often come from unexpected corners and institutions, that translating them into technologies, policies or institutions takes advantage of unexpected opportunities and serendipity, that the location, clients, and pace of adoption can hardly ever be accurately predicted, and that impacts depend on context and so many other factors (such as whether the economy is open or closed), so that predicting and quantifying it would require high level analytical skills that are extremely scarce (and better used on doing research than just planning it in excessive detail).
The SRF does not seem to be focused on scientific excellence, a word that is mentioned only twice. In just two paragraphs of over 200 (paras 84 and 158), the SRF introduces some discussion of science, its contingent and unpredictable nature, and shows an inkling of self doubt that this might stand in the way of the model they are advocating. But that self doubt is never followed through or entered into a risk analysis, which is totally absent from the SRF. I started to wonder whether the underlying hostility to science that can be found often in agricultural technology debates has unwittingly crept into the CGIAR, or at least into its rhetoric. Nor does the SRF focus on finding and exploiting golden eggs such as BT cotton that keep on producing impacts over and over in an ever expanding set of waves. All that is said is that we want international public goods.
In its discussion of the specific SLOs the document is quite candid about how hard it is to establish impact pathways on outcomes and quantify them. It also recognizes that the different SLOs have different impact metrics, some not quantitative at all, that cannot possibly be brought to a common measurement scale, or expressed in dollars for benefit-cost analysis. In the case of the impacts on nutrition and health and on sustainable natural resource management the text expresses the gravest doubts that impact pathways can ever be properly specified and measured (paras 101-115).
For nutrition and health it states correctly: “There is a need for more evidence on where and how local improvements in agriculture lead to reduced undernutrition.” At the recent New Delhi agriculture and nutrition conference, (other than in terms of reducing adverse health impacts of agriculture and breeding for micro-nutrient content), there were hardly any examples of such linkages. Para 106 all but admits that it will be nearly impossible to measure the impact of agriculture on nutrition. If it cannot be measured, how will anybody be held accountable?
Other than by reducing adverse health impacts and breeding for micronutrient content, I would not have a clue how to define an important and scalable nutrition or health impact of agriculture that could be achieved via CGIAR research. And to my knowledge, neither did anyone else at the agriculture and nutrition jamboree. Yet, despite its cogent analysis and all its self doubts, the SRF assumes that there are many and important such research activities, and endorses the elevation of nutrition and health as SLO for the agricultural research of the CGIAR on par with poverty reduction and food security.

For Natural Resources Management the situation is equally incongruous (paras 107-115): Para 107 to 109 are a litany of failure to come to grips in the CGIAR with how to work in this area. While the CGIAR tried to focus on sustainable production systems, it was pulled by funding opportunities to take on different problems: “It was quickly displaced by a focus on landscapes as the appropriate research and management unit, partly driven by the increasing interest in provision of and payment for ecosystem services, especially the development of carbon markets. However, it remains to be seen whether managing carbon, hydrology, nutrient flows, and biodiversity within critical landscapes is possible and generates potential benefits. As with INRM, there have not been mechanisms within the CG to systematically test the approach.” What this says is that nobody agrees on what the problem structure should be or even if they did, nobody has any idea whether operating with the particular problem structure could lead to well defined systems outcomes for the CGIAR. Thus we have an entire SLO no agreed and well defined problem structure.

The rationale for coordination in NRM is completely undermined in para 127 (in the practical part of the document) which states: “This last point raises the succeeding question of whether there would be gains through more effective priority setting in the NRM area. Priority setting is essentially an aggregated model of ex-ante impact assessment and has rarely, if ever, been applied in the NRM area across sectors. Methodology development in the area of ex-post impact assessment in the NRM area is noted for both its complexity and the need to adapt the methods to the particular problem or resource. A comparative methodology across different resource sectors does not exist, much less to disaggregate and prioritize investments by different research components or problem areas. Development of such a priority setting capacity would require significant investment and would only be justified if budgets for
the NRM research portfolio were especially constrained and some type of priority setting framework were deemed to aid in defining a fair allocation of resources.”

The devastating self critique in these paragraphs has no influence whatsoever on the framework for decision making and implementation proposed in the SRF

Instead of following up on its own analysis, the SRO focuses on integration and coordination of activities, by suggestion that such integration needs to and can be planned ex-ante by central research managers rather than emerging ex-post as a consequence of incentives and opportunities that arise over time. The last time a major institution banked on ex-ante and planned integration for poverty reduction, it was the World Bank’s approach to integrated rural development (IRD) in the form of area development programs. It turned out that rural development could not be coordinated across sectors, actors and institutions, but only by the affected populations themselves at the local level and in communities (which is why today the World Bank is trying to scale up local and community-driven development and promote the associated policy, fiscal and institutional reforms). IRD failed spectacularly and was completely discredited in the early 1990s. It lived only for 20 years, and was already shown to have failed by Uma Lele and others within 10 years of its launching. Billions and billions of dollars went into hundreds of such programs without hardly any sustainable impact on poverty or institutions anywhere.
Forgotten in the SRF are the incentives and funding problems that have long been diagnosed to be at the root of many of the systems problems: The decline of core funding, and the excessive power of donors to shape the research program of the system and of individual centers and programs via special projects. The word “incentives”, both positive and negative is never even mentioned, or used to analyze why things have gone wrong and at what level. Indeed notably absent from the SRF is a reform of incentives within the CGIAR. There is consequently no analysis of what it would take to upgrade the eroded scientific capacities in many of the centers, and whether or not the SRF framework would make the recruitment and retention of top researchers and scientists easier. I suspect it will turn off many of the most talented and creative who will look for opportunities in institutions that take more pragmatic and flexible approaches to the management of research.
Fortunately the challenge programs, and the CRPs, and the attendant reorganization of the CGIAR are intended to address the funding problems. But they too fall short of an analysis of incentives, and how to align them to achieve the desired multiple and diverse systems outcomes. The SRF goes much beyond what is needed in terms of restoring core funding and systems autonomy. It instead imposes an unproven organization and planning model that smacks of central planning and undermines decentralized initiatives of scientists and individual programs in the system, and perhaps even science itself.
A final weakness of the SRF is that it is devoid of any analysis of risk. The question what would happen if the CGIAR, or any of its CRPs failed short of expectations, and how to guard against such risks is not even raised. None of the other risk facing the CGIAR, such as donors not buying in to the SRF even though they demanded and shaped it, are analyzed.
But why has the SRF and apparently a broad range of the research community and civil society organizations embraced this framework so wholeheartedly, as evidenced for example by the momentum of GCARD, or the Africa Challenge Program spearheaded by FARA? Clearly several things have gone wrong in the CGIAR in terms of impacts on the SLOs, coordination, and participation of stakeholders. (i) Much costly research is not sufficiently focused on achieving outputs which are scalable and have a good chance of contribution to the SLOs at the desired scale. (ii) The number of instances is very large where the lack of farmer involvement in the design of technologies or involvement of stakeholders in policy making has been lacking and therefore the wrong outputs have been produced that were not adopted, or that were unable to be scaled up. (iii) In addition, many actors have long ago raised expectations of the CGIAR in areas such as NRM, gender, and more recently nutrition that have simply not (or not yet) been fulfilled. There are probably many other sources of frustration, but these three are sufficient to understand the resulting despair. Deeply frustrated stakeholders have therefore been looking for solutions for at least two decades. Now many of them seem to have found the magic bullet to solve the underlying problems. Magic bullets have characterized the field of agricultural and rural development again and again. Unfortunately, as I have been emphasizing in my Elmhirst lecture (2006) none of them has fulfilled their promises, and none will ever be found.
Since I am concerned about SLOs, participation, and better work in the area of NRM, what proposals do I have? Clearly the CGIAR and its research managers should insist that the research activities and programs have a reasonable chance of contribution at scale to the SLOs. That does imply some indication of the plausible route and scale at which they might do so. But nothing like the rigorous and time and skills intensive (and therefore very costly) analysis suggested by the SRF. I like the much broader and more balanced research evaluation criteria that the ISPC proposes. At the applied end, the CGIAR should also insist that scientists test their ideas in the field with farmers prior to initiating research and at the earliest opportunity for doing so once research has started, for example via the now increasingly popular participatory breeding and selection methods. At the strategic innovation end could be more rigorous specification of likely golden eggs that will produce growing benefits over time in a wide range of places, such as the bio-control of cassava mealy bug (that produced enough benefits to cover the entire costs of CGIAR research in Africa since its inception), NERICA Rice, BT cotton, resistance to stressors via BT or conventional techniques, the Village Dynamics studies in South Asia, or the panel data sets that are now being financed by the Bill and Melinda Gates Foundation. The agricultural science program of the GATES Foundation does look for such golden eggs, although they do not call them that.
Once achieved, any golden egg will pull in the required collaborators and partners to exploit it, without having to specify and plan for all of that in advance. Everybody will have huge incentives to do so. Proposals for golden eggs are not likely to come from civil society, donors or even the farmers, but instead from the informed judgment of the scientific community, and of brilliant individual researchers. Research to achieve golden eggs might take place entirely outside the CGIAR centers but be co-financed by the common fund, with CGIAR centers and partners only coming in when research moves to more applied levels. The panel data sets financed by the Gates foundation are neither carried out or financed by the CGIAR , but present tremendous research opportunities, especially for CRP 2. In terms of the CRPs, the guidelines of the ISPC contain many excellent criteria that can be applied with a grain of wisdom in their design and evaluation.
Part 2 of the SRF is another example of a more pragmatic approach to research management in the CGIAR than via the grandiose single management and implementation framework advocated in the first half of the document. As discussed before, I do not see how the framework was necessary to come up with these proposals. Instead they are based on informed and pragmatic judgment and knowledge what has worked in the past and what not, and a good understanding of what might work in the future.
To conclude: The framework of the SRF (not the practical proposals in the second half) is naïve, over-ambitious, and not grounded in the large theoretical and empirical literature of how advances in science, innovation and adoption processes work. Within the document there are internal contradictions; and doubts about its implementability that have not been followed to their logical conclusions. It will be very difficult to adapt the CRPs to the SRF. Trying to do so will lead to un-necessary overhead costs for research planning and coordination.
Refereces
Conway, Gordon and Jeff Waage, 2010, Science and Innovation for Development, UK Collaborative on Development Sciences
Ruttan, Vernon W, 2001, Technology, Growth, and Development: An Induced Innovation Perspective, New York, Oxford University Press

Binswanger, H.P., and K. Deininger, “Explaining Agricultural and Agrarian Policies in Developing Countries.” Journal of Economic Literature, 1997, Vol.35: pp.1958–2005.

Hans P. Binswanger, “Empowering Rural People for their own Development.” in Keijiro Otsuka and Kaliappa Kalirajan (eds.), Contributions of Agricultural
Economics to Critical Policy Issues. Malden, MA: Blackwell, 2008.
Hans P. Binswanger, Jacomina de Regt, and Stephen Spector, Local and Community-Driven Development, Moving to Scale in Theory and Practice, New Frontiers in Social Policy, Washington DC, World Bank, 2009.