Quantitative models are an important part of environmental and economic research and policymaking. For instance, IIASA models such as GLOBIOM and GAINS have long assisted the European Commission in impact assessment and policy analysis2; and the energy policies in the US have long been guided by a national energy systems model (NEMS)3.
Despite such successful modelling applications, model criticisms often make the headlines. Either in scientific literature or in popular media, some critiques highlight that models are used as if they are precise predictors and that they don’t deal with uncertainties adequately4,5,6, whereas others accuse models of not accurately replicating reality7. Still more criticize models for extrapolating historical data as if it is a good estimate of the future8, and for their limited scopes that omit relevant and important processes9,10.
Validation is the modeling step employed to deal with such criticism and to ensure that a model is credible. However, validation means different things in different modelling fields, to different practitioners and to different decision makers. Some consider validity as an accurate representation of reality, based either on the processes included in the model scope or on the match between the model output and empirical data. According to others, an accurate representation is impossible; therefore, a model’s validity depends on how useful it is to understand the complexity and to test different assumptions.
Given this variety of views, we conducted a text-mining analysis on a large body of academic literature to understand the prevalent views and approaches in the model validation practice. We then complemented this analysis with an online survey among modeling practitioners. The purpose of the survey was to investigate the practitioners’ perspectives, and how it depends on background factors.
According to our results, published recently in Eker et al. (2018)1, data and prediction are the most prevalent themes in the model validation literature in all main areas of sustainability science such as energy, hydrology and ecosystems. As Figure 1 below shows, the largest fraction of practitioners (41%) think that a match between the past data and model output is a strong indicator of a model’s predictive power (Question 3). Around one third of the respondents disagree that a model is valid if it replicates the past since multiple models can achieve this, while another one third agree (Question 4). A large majority (69%) disagrees with Question 5, that models cannot provide accurate projects, implying that they support using models for prediction purposes. Overall, there is no strong consensus among the practitioners about the role of historical data in model validation. Still, objections to relying on data-oriented validation have not been widely reflected in practice.
Figure 1: Survey responses to the key issues in model validation. Source: Eker et al. (2018)
According to most practitioners who participated in the survey, decision-makers find a model credible if it replicates the historical data (Question 6), and if the assumptions and uncertainties are communicated clearly (Question 8). Therefore, practitioners think that decision makers demand that models match historical data. They also acknowledge the calls for a clear communication of uncertainties and assumptions, which is increasingly considered as best-practice in modeling.
One intriguing finding is that the acknowledgement of uncertainties and assumptions depends on experience level. The practitioners with a very low experience level (0-2 years) or with very long experience (more than 10 years) tend to agree more with the importance of clarifying uncertainties and assumptions. Could it be because a longer engagement in modeling and a longer interaction with decision makers help to acknowledge the necessity of communicating uncertainties and assumptions? Would inexperienced modelers favor uncertainty communication due to their fresh training on the best-practice and their understanding of the methods to deal with uncertainty? Would the employment conditions of modelers play a role in this finding?
As a modeler by myself, I am surprised by the variety of views on validation and their differences from my prior view. With such findings and questions raised, I think this paper can provide model developers and users with reflections on and insights into their practice. It can also facilitate communication in the interface between modelling and decision-making, so that the two parties can elaborate on what makes their models valid and how it can contribute to decision-making.
Model validation is a heated topic that would inevitably stay discordant. Still, one consensus to reach is that a model is a representation of reality, not the reality itself, just like the disclaimer of René Magritte that his perfectly curved and brightly polished pipe is not a pipe.
Eker S, Rovenskaya E, Obersteiner M, Langan S. Practice and perspectives in the validation of resource management models. Nature Communications 2018, 9(1): 5359. DOI: 10.1038/s41467-018-07811-9 [pure.iiasa.ac.at/id/eprint/15646/]
Nuccitelli D. Climate scientists just debunked deniers’ favorite argument. The Guardian. 2017. https://www.theguardian.com/environment/climate-consensus-97-per-cent/2017/jun/28/climate-scientists-just-debunked-deniers-favorite-argument
As a true Eurasianist and a member of the European Society for Eurasian Cooperation I was really interested in attending the conference.
It was opened by the governor of the Austrian Central Bank, Ewald Nowotny, who said that cooperation between Asia and Europe is vital, especially with China’s growing economic and political influence. Nowotny expressed regret that some countries see this as a challenge rather than an opportunity. Europe, however, remains the best place to be because of its economic strength.
Marc Uzan, the executive director of the Reinventing Bretton Woods Committee, noted that we live in a new age of connectivity. The economic ties between the EU and Asia are quite strong but there is still space for stronger connectivity in the form of physical and non-physical infrastructure, market integration, and maintaining stability in Central Asia. Uzan highlighted the role of the European Investment Bank in various connecting projects.
During the panel session on “Integration in Europe: European Union and Eurasia”, Elena Rovenskaya, the program director for Advanced Systems Analysis at IIASA, presented the institute as a neutral platform for depoliticized dialogue. IIASA has been running a project on the “Challenges and Opportunities of Economic Integration within a Wider European and Eurasian Space” since 2014, analyzing transport corridors, foreign direct investment, and convergence of technical product standards between EU and the Eurasian Economic Union.
This report was especially exciting for me because I had a great opportunity of participating in the International Youth Forum “Future of Eurasian and European Integration: Foresight-2040”, hosted by IIASA in December 2017, and found it interesting to see how research into Eurasian integration at IIASA has advanced since then. The concept of dividing the integration in two subgroups (bottom-up and top-down) suggested by Rovenskaya also seemed new to me.
‘Bottom-up’ integration requires coordination between participating countries and involves development of transport and infrastructure – known as the Belt and Road Initiative – including development of the Kosice-Vienna broad gauge railway extension, and the Arctic railway in Finland. The top-down scenario would be based on cooperation between regional organizations and programs such as the EU, the EAEU and the Eastern Partnership. The challenge lies in harmonizing different integration processes.
I find it unfortunate that despite the positive impact of theoretical EU-EAEU economic integration and cooperation showed by IIASA’s research, the economic relations between the EU and the EAEU are currently defined by foreign policies and not by economic reasoning.
In his address, William Tompson, the head of Eurasia Department at the Global Relations Secretariat of the OECD, highlighted that the benefits of enhanced connectivity were not automatic and that complex packages, going beyond trade and infrastructure, would be needed. I consider that Tompson raised an important point that we should not exaggerate the benefits – landlocked locations and distance to global markets can be mitigated but not eliminated. Coordination among countries to remove infrastructure and non-infrastructure bottlenecks will necessary.
Tompson’s empirics convinced me that there is a call for change. Kazakhstan pays US$250/t of freight to reach the countries with 20% of the global GDP, compared to just US$50 for Germany and the US. This is due to factors like distance, speed, and border crossings.
I was impressed by Tompson’s international freight model. It shows that logistics performance is generally poor, and competition could be enhanced. The link between policy objectives and investment choices is often unclear. Tompson also criticized the ministries of transport, which he called “ministries of road-building”, for not knowing that transport was far more than that.
The head of unit in the European Commission, Petros Sourmelis, presented the EU’s perspective. According to him, the EU is open to deeper cooperation and trade relationships with its Eastern partners, however, there are many barriers, including the EAEU’s incomplete internal market.
I consider the proposal made by Sourmelis that “one needs to start somewhere” and his hope for more engagement quite promising, but engagement at the political level is some way off. However, the EU has seen constructive steps from Russia and is open to talks to build trust.
Member of the Board of the Eurasian Economic Commission Tatyana Valovaya closed the high-level panel session. I think it was a good lead-up to start with a historical analogy of the ancient Silk Road. According to her, the global trade geography in the 21st century is shifting once again to Asia and China was likely to become a leading power within the next 20 years. I was encouraged by the idea that regional economic unions will likely lead to better global governance and building interregional partnerships between Europe, Asia and Eurasia will be vital to achieve it.
Valovaya reminded delegates that in 2003 a lot of political and technical work had been achieved towards EU-Russia cooperation, which had then been stopped for political reasons. In 2015, the EAEU began wider cooperation with China as part of the Belt and Road Initiative, and in May 2018 a non-preferential agreement was signed to harmonize technical standards and custom regulations, to decrease non-tariff barriers as much as possible and to support cooperation projects in the digital economy.
I share the view of Valovaya that the EAEU should not only consider China as a key partner. Valovaya gave the US as a good example, which has multiple economic partnership agreements. She admitted that the EAEU had some “growth pains” but stressed it is normal for such a project and efforts are focused on solving the problems.
As for me, I believe it is necessary to understand the fundamental differences for the further connectivity. Valovaya emphasized that the EAEU was not aiming to introduce a common currency or to create a political union like the EU. EU-EAEU cooperation will strengthen both unions. More technical cooperation will be needed. And, of course, the leaders of the EU should be participating in the dialogue to better understand the EAEU and its work towards more connectivity in Eurasia.
Note: This article gives the views of the author, and not the position of the Nexus blog, nor of the International Institute for Applied Systems Analysis.
By Alma Mendoza, Colosio Fellow with the IIASA Ecosystems Services and Management Program
Changes in land use cover can have a crucial impact on the environment in terms of biodiversity and the benefits that ecosystems provide to people. Assessing, quantifying, and identifying where these changes are the most drastic is especially important in countries that have high biodiversity along with high rates of natural vegetation loss. Socioeconomic pressures often drive land use change and the impacts are expected to increase due to population growth and climate change.
To better understand the possible impacts of land use change in Mexico over the short, medium, and long term, my colleagues and I used the Shared Socioeconomic Pathways–a set of pathways that span a wide range of feasible future developments in areas such as agriculture, population, and the economy–together with a set of climatic scenarios known as the Representative Concentration Pathways. We focused on Mexico, because the country is large enough to encompass different ecosystems, socioeconomic characteristics, and climates. In addition, Mexico is characterized by high deforestation rates, huge biodiversity, and a large number of communities with contrasting land management practices. Incorporating all these features, allowed us to take the complexity of socioecological systems into account.
We designed a model to test how socioeconomic and biophysical drivers, like slope or altitude, may unfold under different scenarios and affect land use. Our model includes 13 categories of which eight represent the most important ecosystems in Mexico (temperate forests, cloud forests, mangroves, scrublands, tropical evergreen and -dry forests, natural grasslands, and other vegetation such as desert ecosystems or natural palms), four represent anthropogenic uses (pasture, rainfed and irrigated agriculture, and human settlements), and one constitutes barren lands. We set two plausible scenarios: “Business as usual” and an optimistic scenario called the “green scenario”. We projected the “business as usual” scenario using medium rates of vegetation loss based on historical trends and combined it with a medium population and economic growth with medium increases in climatic conditions. For the “green scenario”, we projected the lowest rates of native vegetation loss and the highest rates of native vegetation recovery with a low population and medium economic growth in a future with low climatic changes.
Our results show that natural vegetation will undergo significant reductions in Mexico and that different types of vegetation will be affected differently. Tropical dry and evergreen forests, followed by ‘other’ vegetation and cloud forests are the most vulnerable ecosystems in the country. For example, according to the “business as usual” scenario, tropical dry forests might decrease in extent by 47% by the end of the century. This is extremely important considering that the most recent rates, for the period 2007 to 2011, were even higher than the medium rates we used in this scenario. In contrast, the “green scenario” allowed us to see that, with feasible changes of rate, this ecosystem could increase their distribution. However, even 80 years of regeneration would not be enough to reach the extent these forests had in 1985, when they accounted for around 12% of land cover in Mexico. Moreover, the expansion of anthropogenic land cover (such as agriculture, pastures, and human settlements) might reach 37% of land cover in the country by 2050 and 43% by 2100 under the same scenario. In terms of CO2 emissions due to land use cover change we found that Mexico was responsible for 1-2% of global emissions that are the result of land use cover change, but by 2100 it could account for as much as 5%.
Our findings show that conservation policies have not been effective enough to avoid land use cover change, especially in tropical evergreen forests and drier ecosystems such as tropical dry forests, natural grasslands, and other vegetation. Cloud forests have also been badly affected. As a biologically and culturally rich country, Mexico is responsible for maintaining its diversity by implementing a sustainable and intelligent management of its territory.
Our study identified hotspots of land use change that can help to prioritize areas for improving environmental performance. Our project is currently linking the hotspots of change with the most threatened and endemic species of Mexican terrestrial vertebrates (mammals, amphibians, reptiles, and birds) to provide useful results that can help prioritize ecosystems, species, or municipalities in Mexico.
Mendoza Ponce A, Corona-Núñez R, Kraxner F, Leduc S, & Patrizio P (2018). Identifying effects of land use cover changes and climate change on terrestrial ecosystems and carbon stocks in Mexico. Global Environmental Change 53: 12-23. [pure.iiasa.ac.at/15462]
Note: This article gives the views of the author, and not the position of the Nexus blog, nor of the International Institute for Applied Systems Analysis.
By Melina Filzinger, IIASA Science Communication Fellow
Imagine you are heading home from work and are stuck in evening rush hour traffic. You see an opportunity to save time by cutting another driver off, but this will lead to a delay for other cars, possibly causing a traffic jam. Would you do it? Situations like these, where you can benefit from acting selfishly while causing the community as a whole to be worse off, are known as social dilemmas, and are at the heart of many areas of research in economics.
The social dilemma becomes particularly important when considering so-called common pool resources such as water reservoirs that are depleted when people use them. For instance, picture several farmers using water from the same river to irrigate their farmland. The river might carry enough water for all of them, but if there is no incentive for the upstream farmers to take the needs of the farmers living further downstream into consideration, they might use more than their share of the water, not leaving enough for the rest of the group. Situations like this are particularly relevant in developing countries, where small-scale farmers that manage the irrigation of their farmland themselves play a significant role in ensuring food security.
Growing up in southwestern Cambodia, YSSP participant Tum Nhim saw how the surrounding farmers shared water among themselves, and how important water was to their livelihoods. Not having enough water often meant that there were no crops for a whole year, and many farmers were forced to take on loans in order to feed their families. “Now that climate change is starting to affect Cambodia, and water scarcity is becoming an even bigger problem, it is more important than ever to investigate fair and efficient ways of sharing water,” explains Nhim.
As a water engineer, Nhim used to design and build water infrastructure. He however soon learned that not considering how human decision making affects the water supply will cause situations where the infrastructure provides enough water, but some farmers are still left high and dry. “I think that human behavior is the most important factor to consider when managing common pool resources,” he says.
To find possible solutions for distributing water in a way that yields an optimal outcome for the community, Nhim and his colleagues from the IIASA Advanced Systems Analysis Program use a bottom-up approach–they model the behavior of a number of individual farmers that interact according to certain rules. The researchers can then look at the collective outcome of these interactions after a certain time and ask questions like, “Will the farmers cooperate?” or, “Will some farmers be left without water?” In their model the researchers take into account both the water itself, a common pool resource, and the water infrastructure, which is not depleted by use.
Several mechanisms can be used to ensure the fair distribution of water. Some of them are formal; like laws and regulations, but it is often difficult to keep people from extracting water, because using a given water resource might be a long-standing cultural tradition or legal right. There are however also more informal mechanisms that can help. For example, individuals often prefer to be good citizens in order to ensure that they have a high social standing in their community that will bring them benefits.
This reputational mechanism is especially relevant in small communities with everyday contact between members. If someone takes too much water, or doesn’t invest in the common water infrastructure, they will gain a bad reputation, which will in turn limit their ability to get support from their neighbors later on.
The main question Nhim is investigating in his YSSP project is if this mechanism can spread across several villages that share a common water resource and irrigation infrastructure, and lead to an outcome where everyone cooperates. If this turns out to be true, the reputational mechanism could be a very inexpensive and natural solution for managing common goods across several communities.
Note: This article gives the views of the author, and not the position of the Nexus blog, nor of the International Institute for Applied Systems Analysis.
By Sandra Ortellado, IIASA Science Communication Fellow 2018
In 2007, Sepo Hachigonta was a first-year PhD student studying crop and climate modeling and member of the YSSP cohort. Today, he is the director in the strategic partnership directorate at the National Research Foundation (NRF) in South Africa and one of the editors of the recently launched book Systems Analysis for Complex Global Challenges, which summarizes systems analysis research and its policy implications for issues in South Africa.
From left: Gansen Pillay, Deputy Chief Executive Officer: Research and Innovation Support and Advancement, NRF, Sepo Hachigonta, Editor, Priscilla Mensah, Editor, David Katerere, Editor, Andreas Roodt Editor
But the YSSP program is what first planted the seed for systems analysis thinking, he says, with lots of potential for growth.
Through his YSSP experience, Hachigonta saw that his research could impact the policy system within his home country of South Africa and the nearby region, and he forged lasting bonds with his peers. Together, they were able to think broadly about both academic and cultural issues, giving them the tools to challenge uncertainty and lead systems analysis research across the globe.
Afterwards, Hachigonta spent four years as part of a team leading the NRF, the South African IIASA national member organization (NMO), as well as the Southern African Young Scientists Summer Program (SA-YSSP), which later matured into the South African Systems Analysis Centre. The impressive accomplishments that resulted from these programs deserved to be recognized and highlighted, says Hachigonta, so he and his colleagues collected several years’ worth of research and learning into the book, a collaboration between both IIASA and South African experts.
“After we looked back at the investment we put in the YSSP, we had lots of programs that were happening in South Africa, and lots of publications and collaboration that we wanted to reignite,” said Hachigonta. “We want to look at the issues that we tackled with system analysis as well as the impact of our collaborations with IIASA.”
Now, many years into the relationship between IIASA and South Africa, that partnership has grown.
Between 2012 and 2015, the number of joint programs and collaborations between IIASA and South Africa increased substantially, and the SA-YSSP taught systems analysis skills to over 80 doctoral students from 30 countries, including 35 young scholars from South Africa.
In fact, several of the co-authors are former SA-YSSP alumni and supervisors turned experts in their fields.
“We wanted to use the book as a barometer to show that thanks to NMO public entity funding, students have matured and developed into experts and are able to use what they learned towards the betterment of the people,” says Hachigonta. The book is localized towards issues in South Africa, so it will bring home ideas about how to apply systems analysis thinking to problems like HIV and economic inequality, he adds.
“It’s not just a modeling component in the book, it still speaks to issues that are faced by society.”
Complex social dilemmas like these require clear and thoughtful communication for broader audiences, so the abstracts of the book are organized in sections to discuss how each chapter aligns systems analysis with policymaking and social improvement. That way, the reader can look at the abstract to make sense of the chapter without going into the modeling details.
“Systems analysis is like a black box, we do it every day but don’t learn what exactly it is. But in different countries and different sectors, people are always using systems analysis methodologies,” said Hachigonta, “so we’re hoping this book will enlighten the research community as well as other stakeholders on what systems analysis is and how it can be used to understand some of the challenges that we have.”
“Enlightenment” is a poetic way to frame their goal: recalling the age of human reason that popularized science and paved the way for political revolutions, Hachigonta knows the value of passing down years of intellectual heritage from one cohort of researchers to the next.
“You are watching this seed that was planted grow over time, which keeps you motivated,” says Hachigonta.
“Looking back, I am where I am now because of my involvement with IIASA 11 years ago, which has been shaping my life and the leadership role I’ve been playing within South Africa ever since.”
By Alan Nicol, Strategic Program Leader at the International Water Management Institute (IWMI)
I was at the local corner store in Uganda last week and noticed the profusion of rice being sold, the origin of which was from either India or Pakistan. It is highly likely that this rice being consumed in Eastern Africa, was produced in the Indus Basin, using Indus waters, and was then processed and shipped to Africa. That is not exceptional in its own right and is, arguably, a sign of a healthy global trading system.
Nevertheless, the rice in question is likely from a system under increasing stress, one that is often simply viewed as a hydrological (i.e., basin) unit. What my trip to the corner store shows is that perhaps more than ever before a system such as the Indus is no longer confined–it extends well beyond its physical (hydrological) borders.
Not only does this rice represent embedded ‘virtual’ water (the water used to grow and refine the produce), but it also represents policy decisions, embedded labor value, and the gamut of economic agreements between distribution companies and import entities, as well as the political relationship between East Africa and South Asia. On top of that are the global prices for commodities and international market forces.
In that sense, the Indus River Basin is the epitome of a complex system in which simple, linear causality may not be a useful way for decision makers to determine what to do and how to invest in managing the system into the future. Integral to this biophysical system, are social, economic, and political systems in which elements of climate, population growth and movement, and political uncertainty make decisions hard to get right.
Like other systems, it is constantly changing and endlessly complex, representing a great deal of interconnectivity. This poses questions about stability, sustainability, and hard choices and trade-offs that need to be made, not least in terms of the social and economic cost-benefit of huge rice production and export.
So how do we go about planning in a system that is in such constant flux?
Coping with system complexity in the Indus is the overarching theme of the third Indus Basin Knowledge Forum (IBKF) being co-hosted this week by the International Centre for Integrated Mountain Development (ICIMOD), the International Institute for Applied Systems Analysis (IIASA), the International Water Management Institute (IWMI), and the World Bank. Titled Managing Systems Under Stress: Science for Solutions in the Indus Basin, the Forum brings together researchers and other knowledge producers to interface with knowledge users like policymakers to work together to develop the future direction for the basin, while improving the science-decision-making relationship. Participants from four riparian countries–Afghanistan, China, India, and Pakistan–as well as from international organizations that conduct interdisciplinary research on factors that impact the basin, will work through a ‘marketplace’ for ideas, funding sources, and potential applications. The aim is to narrow down a set of practical and useful activities with defined outcomes that can be tracked and traced in coming years under the auspices of future fora.
The meeting builds on the work already done and, crucially, on relations already established in this complex geopolitical space, including under the Indus Forum and the Upper Indus Basin Network. By sharing knowledge, asking tough questions, and identifying opportunities for working together, the IBKF hopes to pin down concrete commitments from both funders and policymakers, but also from researchers, to ensure high quality outputs that are of real, practical relevance to this system under stress–from within and externally.
Feeding into the IBKF3, and directly preceding the forum, the Integrated Solutions for Water, Energy, and Land Project (ISWEL) will bring together policymakers and other stakeholders from the basin to explore a policy tool that looks at how best to model basin futures. This approach will help the group conceive possible futures and model the pathways leading to the best possible outcomes for the most people. This ‘policy exercise approach’ will involve six steps to identify and evaluate possible future pathways:
Understanding power relations, underlying interests, and their role in nexus policy development
Developing and selecting nexus solutions
Identifying synergies, and
Building pathways with key milestones for future investments and implementation of solutions.
The summary of this scenario development workshop and a vision for the Indus Basin will be shared as part of the IBKF3 at the end of the event, and will help the participants collectively consider what actions can be taken to ensure a prosperous, sustainable, and equitable future for those living in the basin.
The rice that helps feed parts of East Africa plays a key global role–the challenge will be ensuring that this important trading relationship is not jeopardized by a system that moves from pressure points to eventual collapse. Open science-policy and decision-making collaboration are key to making sure that this does not happen.