No one-size fits all
Basically, there is no ‘monopoly’ from any one discipline to determine what constitutes the quality of a piece of research (although economists will push on this one, having a majority at the conference!). The methods used depend on who is using the information and what their motivations are. For example, is a conservation project or an economic instrument like Payments for Ecosystem Services working from the perspective of the people affected, rather than those who implement it? Maybe it needs to demonstrate the evidence of its impact to an external, wider audience at the macro level and highlight or challenge the role of the State. Perhaps there is a need to understand what determines the costs of agriculture adaptation to climate change, and see who wins and who loses? Perhaps the interest is to further the theoretical understanding of an instrument like Reducing Emissions from deforestation and degradation (REDD) through modelling?
The quality of the research is decided upon factors such as:
- demonstrating systematic and transparent research methods,
- establishing the research problem, objectives and methods,
- data collection, and how it is analysed,
- and what conclusions are reached.
This applies to both qualitative and quantitative research. This is an iterative (not a linear) process, involving readjustments, and in many cases lateral thinking.
There is a spectrum of research methods or approaches available when studying ecosystem services, such as qualitative, participatory, quantitative, and multicriteria methods (which considers multiple conflicting criteria in a decision-making process). All methods have strengths and weaknesses, trade-offs in terms of internal validity (the ‘cause and effect’) and external validity (whether these results can be extrapolated).
There is a strong view that there should be no hierarchy of methods. Different methods should be considered depending on the situation, and creativity is often required when working in difficult situations. According to Andreas Kontoleon from Cambridge University, the main aim of quantitative research impact is to try to identify a causal link between policy (treatment) and impact or outcome. Quantitative impact evaluation techniques are also used to “neutralise” confounding factors and identify causality; in other words to account for the lack of counterfactual, and to discern fact from advocacy. Hunches, beliefs and myths can affect policy – valuation tries to bring in facts to test these perceptions.
For Hanne Svarstad, from the Norwegian Institute for Nature Research, qualitative research is not less rigorous, but follows different methods, and is extremely useful (if not unique) for:
- uncovering the legitimacy of a conservation measure,
- showing how people perceive distributive and procedural justice and how this affects their own sense of justice.
A consultation of methods for the UK Social Policy Association shows how ‘traditional’ high-quality methods like experimental designs are not necessarily at the top of the list, and ‘softer’ approaches, like in-depth interviews are preferred by a significant number of respondents. In-between approaches, like the UNEP valuation toolkit, try to ‘standardise’ research methods and make research quality more available across users.
Whatever the method, there is a push to be transparent about the limitations of the research, an understanding that it is values and not prices we are after, and a better grasp at how to bring these insights into what traditional accounting of costs and benefits.
Improving credibility and acceptability of valuation
Development and conservation science needs clear, explicit and transparent criteria or benchmarks for determining the quality of research. Studies should at least aim to be:
- believable and consistent with other values (credibility)
- relevant and/or transferable to other context and timescales
- have correspondence between data and conceptualization? (internal validity)
Other criteria to consider include:
- Is it possible to generalize from the research? (external validity)
- Do they have units (monetary, energy, time, etc) that can be easily understood, and if possible, incorporated into wider decision-making processes?
- Are the results available to peer-review and public scrutiny?
The Plenary session brought a lively discussion on how to make valuation worth its penny.
As the threats to our natural resources increases, the need for research that helps understand the impacts of action, and inaction, is stronger. The Stern review, for example, has helped to bring the climate change debate straight into the public eye. According to Prof Nick Hanley, valuation methods can help build understanding on the costs and benefits, the winners and losers, and the trade-offs of different policies relative to a base scenario.
Demonstrating what the values of managing ecosystems are, and to whom (for example through habitat restoration), is a step towards trying to capture it (i.e. through market-based mechanisms, like Payments for Ecosystem Services), says Peter Carte from the European Investment Bank. There is no denying that those impacts with clear units (not necessarily monetary!) are better understood by policy makers than those without, says Jaroslav Mysiak from the Foundazione Eni Enrico Mattei (FEEM). This is why traditional accounting through GDP (gross domestic product), which does not take into account the impacts on ecosystems, provides an erroneous sense of gain, says Paulo Nunes from the Mediterranean Science Commission.
The challenge for research is to make sure that we push valuation into different metrics and methods, going beyond regression analysis and monetary values. Natural ecosystems have many values that go beyond the utilitarian perspective and must be acknowledged. But regardless of what we believe about nature’s values, we must be able to know the costs of using it.
As Pushpam Kumar from UNEP said, policy makers are lost when confronted with Total Economic Values. For example, numbers like those from the well-known 1990s ecological economics study by Costanza, estimating the world's ecosystem services and natural capital at US$33 trillion per year are just too abstract (and who wouldn’t feel lost?). Meaningful values of ecosystems are not global. They are specific to context and time, grounded on costs and linked to demand. But ultimately, any valuation is not free of judgement, and will reflect the preconceptions of the theories they adhere to. Understanding what lies in the assumptions is the first step to believing (or not) in the conclusions.
Dr Ina Porras, IIED
This article first appeared on IIED's blog pages.
GEC selected for the Global Gateway Dialogue Platform
In a world riddled with complex challenges, international cooperation with all stakeholders is essential to drive transformation.
The IEA's Net Zero roadmap: a narrow path to 1.5 degrees
Our statement on the outcomes of the International Climate and Energy Summit, Madrid, October 2nd