Quantitative models are an important part of environmental and economic research and policymaking. For instance, IIASA models such as GLOBIOM and GAINS have long assisted the European Commission in impact assessment and policy analysis2; and the energy policies in the US have long been guided by a national energy systems model (NEMS)3.
Despite such successful modelling applications, model criticisms often make the headlines. Either in scientific literature or in popular media, some critiques highlight that models are used as if they are precise predictors and that they don’t deal with uncertainties adequately4,5,6, whereas others accuse models of not accurately replicating reality7. Still more criticize models for extrapolating historical data as if it is a good estimate of the future8, and for their limited scopes that omit relevant and important processes9,10.
Validation is the modeling step employed to deal with such criticism and to ensure that a model is credible. However, validation means different things in different modelling fields, to different practitioners and to different decision makers. Some consider validity as an accurate representation of reality, based either on the processes included in the model scope or on the match between the model output and empirical data. According to others, an accurate representation is impossible; therefore, a model’s validity depends on how useful it is to understand the complexity and to test different assumptions.
Given this variety of views, we conducted a text-mining analysis on a large body of academic literature to understand the prevalent views and approaches in the model validation practice. We then complemented this analysis with an online survey among modeling practitioners. The purpose of the survey was to investigate the practitioners’ perspectives, and how it depends on background factors.
According to our results, published recently in Eker et al. (2018)1, data and prediction are the most prevalent themes in the model validation literature in all main areas of sustainability science such as energy, hydrology and ecosystems. As Figure 1 below shows, the largest fraction of practitioners (41%) think that a match between the past data and model output is a strong indicator of a model’s predictive power (Question 3). Around one third of the respondents disagree that a model is valid if it replicates the past since multiple models can achieve this, while another one third agree (Question 4). A large majority (69%) disagrees with Question 5, that models cannot provide accurate projects, implying that they support using models for prediction purposes. Overall, there is no strong consensus among the practitioners about the role of historical data in model validation. Still, objections to relying on data-oriented validation have not been widely reflected in practice.
Figure 1: Survey responses to the key issues in model validation. Source: Eker et al. (2018)
According to most practitioners who participated in the survey, decision-makers find a model credible if it replicates the historical data (Question 6), and if the assumptions and uncertainties are communicated clearly (Question 8). Therefore, practitioners think that decision makers demand that models match historical data. They also acknowledge the calls for a clear communication of uncertainties and assumptions, which is increasingly considered as best-practice in modeling.
One intriguing finding is that the acknowledgement of uncertainties and assumptions depends on experience level. The practitioners with a very low experience level (0-2 years) or with very long experience (more than 10 years) tend to agree more with the importance of clarifying uncertainties and assumptions. Could it be because a longer engagement in modeling and a longer interaction with decision makers help to acknowledge the necessity of communicating uncertainties and assumptions? Would inexperienced modelers favor uncertainty communication due to their fresh training on the best-practice and their understanding of the methods to deal with uncertainty? Would the employment conditions of modelers play a role in this finding?
As a modeler by myself, I am surprised by the variety of views on validation and their differences from my prior view. With such findings and questions raised, I think this paper can provide model developers and users with reflections on and insights into their practice. It can also facilitate communication in the interface between modelling and decision-making, so that the two parties can elaborate on what makes their models valid and how it can contribute to decision-making.
Model validation is a heated topic that would inevitably stay discordant. Still, one consensus to reach is that a model is a representation of reality, not the reality itself, just like the disclaimer of René Magritte that his perfectly curved and brightly polished pipe is not a pipe.
Eker S, Rovenskaya E, Obersteiner M, Langan S. Practice and perspectives in the validation of resource management models. Nature Communications 2018, 9(1): 5359. DOI: 10.1038/s41467-018-07811-9 [pure.iiasa.ac.at/id/eprint/15646/]
Nuccitelli D. Climate scientists just debunked deniers’ favorite argument. The Guardian. 2017. https://www.theguardian.com/environment/climate-consensus-97-per-cent/2017/jun/28/climate-scientists-just-debunked-deniers-favorite-argument
By Sandra Ortellado, IIASA Science Communication Fellow 2018
In 2007, Sepo Hachigonta was a first-year PhD student studying crop and climate modeling and member of the YSSP cohort. Today, he is the director in the strategic partnership directorate at the National Research Foundation (NRF) in South Africa and one of the editors of the recently launched book Systems Analysis for Complex Global Challenges, which summarizes systems analysis research and its policy implications for issues in South Africa.
From left: Gansen Pillay, Deputy Chief Executive Officer: Research and Innovation Support and Advancement, NRF, Sepo Hachigonta, Editor, Priscilla Mensah, Editor, David Katerere, Editor, Andreas Roodt Editor
But the YSSP program is what first planted the seed for systems analysis thinking, he says, with lots of potential for growth.
Through his YSSP experience, Hachigonta saw that his research could impact the policy system within his home country of South Africa and the nearby region, and he forged lasting bonds with his peers. Together, they were able to think broadly about both academic and cultural issues, giving them the tools to challenge uncertainty and lead systems analysis research across the globe.
Afterwards, Hachigonta spent four years as part of a team leading the NRF, the South African IIASA national member organization (NMO), as well as the Southern African Young Scientists Summer Program (SA-YSSP), which later matured into the South African Systems Analysis Centre. The impressive accomplishments that resulted from these programs deserved to be recognized and highlighted, says Hachigonta, so he and his colleagues collected several years’ worth of research and learning into the book, a collaboration between both IIASA and South African experts.
“After we looked back at the investment we put in the YSSP, we had lots of programs that were happening in South Africa, and lots of publications and collaboration that we wanted to reignite,” said Hachigonta. “We want to look at the issues that we tackled with system analysis as well as the impact of our collaborations with IIASA.”
Now, many years into the relationship between IIASA and South Africa, that partnership has grown.
Between 2012 and 2015, the number of joint programs and collaborations between IIASA and South Africa increased substantially, and the SA-YSSP taught systems analysis skills to over 80 doctoral students from 30 countries, including 35 young scholars from South Africa.
In fact, several of the co-authors are former SA-YSSP alumni and supervisors turned experts in their fields.
“We wanted to use the book as a barometer to show that thanks to NMO public entity funding, students have matured and developed into experts and are able to use what they learned towards the betterment of the people,” says Hachigonta. The book is localized towards issues in South Africa, so it will bring home ideas about how to apply systems analysis thinking to problems like HIV and economic inequality, he adds.
“It’s not just a modeling component in the book, it still speaks to issues that are faced by society.”
Complex social dilemmas like these require clear and thoughtful communication for broader audiences, so the abstracts of the book are organized in sections to discuss how each chapter aligns systems analysis with policymaking and social improvement. That way, the reader can look at the abstract to make sense of the chapter without going into the modeling details.
“Systems analysis is like a black box, we do it every day but don’t learn what exactly it is. But in different countries and different sectors, people are always using systems analysis methodologies,” said Hachigonta, “so we’re hoping this book will enlighten the research community as well as other stakeholders on what systems analysis is and how it can be used to understand some of the challenges that we have.”
“Enlightenment” is a poetic way to frame their goal: recalling the age of human reason that popularized science and paved the way for political revolutions, Hachigonta knows the value of passing down years of intellectual heritage from one cohort of researchers to the next.
“You are watching this seed that was planted grow over time, which keeps you motivated,” says Hachigonta.
“Looking back, I am where I am now because of my involvement with IIASA 11 years ago, which has been shaping my life and the leadership role I’ve been playing within South Africa ever since.”
By Melina Filzinger, IIASA Science Communication Fellow
In his lecture at IIASA, Maurizio Bona, senior advisor for relations with parliaments and science for policy, and senior advisor on knowledge transfer at the European Organization for Nuclear Research (CERN) discussed the question “Science and diplomacy–two different worlds?”, focusing on the dual role of CERN as both a research laboratory and an intergovernmental institution.
According to Bona, international research centers like CERN and IIASA foster international and intercultural communication by bringing together people with different backgrounds and ideas to work on a common goal. In this context, these organizations act as communication channels where science is used as a universal language.
CERN was established in 1953 to carry out research on particle physics, but also to reunite a Europe that was divided after World War II, and to re-open the dialogue between European countries and beyond. While CERN was not involved in politics directly, an important point of the lecture was that science can provide a neutral field for dialogue and connect people that would not meet otherwise. In this way, international research institutes can contribute to science diplomacy in a very indirect and informal way.
IIASA was founded in 1972 to find solutions to global problems, and with a similar goal of using scientific cooperation to build bridges across the Cold War divide. Despite the vastly different research done at CERN and IIASA, both organizations have roots in science diplomacy that stem from the fact that today’s problems, regardless of whether they are fundamental or applied in nature, are often too complex to be solved by one country or discipline alone.
Even though CERN is a European organization, it attracts researchers from all over the world, like IIASA. In January 2018, 41% of scientific users (researchers using CERN facilities that are not paid by CERN) were from non-member countries and contributed their expertise as well as research equipment. In order to ensure that scientific advancement and not national interests are the basis of the research objectives at CERN, it is based on a simple but strong Convention that excludes military applications and ensures transparency. Additionally, CERN stays away from political affiliations.
Based on the success of the CERN model, the first particle accelerator in the Middle East, Synchrotron-Light for Experimental Science and Applications in the Middle East (SESAME), was established in 2004. Its organizational structure is based largely on that of CERN, and it was thought out explicitly as a way to bring together conflicting Middle Eastern countries, while at the same time advancing science. SESAME’s member states are Cyprus, Egypt, Iran, Israel, Jordan, Pakistan, Palestine, and Turkey and the facility has been open to scientific users from the Middle East and beyond since 2017.
Beyond fostering international and intercultural communication by bringing together people with different backgrounds and ideas to work on a common goal, international research institutes can also influence policy more directly. For example, CERN has been an observing member at the UN general assembly since 2012 and has had an influence on shaping the UN 2030 Agenda for Sustainable Development, advocating for the importance of education and fundamental research.
IIASA goes one step further, explicitly aiming to shape policies and help politicians make informed, evidence-based decisions. IIASA research has for example shaped European air pollution policy and has led to real improvements in the sustainable management of scarce resources in a number of countries. The institute’s independence and political neutrality are key for its credibility as an adviser to policy makers. IIASA is nongovernmental and is instead sponsored by its 23 national member organizations. Today IIASA member countries make up 71% of the global economy and 63% of the global population, making IIASA particularly well-suited to address global challenges.
Maurizio Bona closed his lecture with the following quote by Daniel Barenboim, a world-famous pianist and conductor:
“Let me tell you something: This is not going to bring peace. What it can bring is understanding, the patience, the courage, and the curiosity to listen to the narrative of the other.”
Daniel Barenboim, Ramallah concert, August 2005.
This quote was originally meant to be understood in the context of international collaboration in music, but is also applicable to science, and in fact to any endeavor that brings together people from different backgrounds to work towards a common goal.
The lecture left the audience with some open questions, like how to measure the impact of science on society, or how involved science should be in diplomacy. Some of these questions were picked up on in a lively discussion after the talk. For now, I think it is fair to conclude that the histories of both CERN and IIASA show that international research institutes can have a positive impact on society while remaining politically neutral and unbiased in their scientific goals.
Note: This article gives the views of the authors, and not the position of the Nexus blog, nor of the International Institute for Applied Systems Analysis.
On 14 and 15 May, Vienna hosted two important events within the frame of the world energy and climate change agendas: the Vienna Energy Forum and the R20 Austrian World Summit. Since I had the pleasure and privilege to attend both, I would like to share some insights and relevant messages I took home with me.
To begin with, ‘renewable energy’ was the buzzword of the moment. Renewable energy is not only the future, it is the present. Recently, 20-year solar PV contracts were signed for US$0.02/kWh. However, renewable energy is not only about mitigating the effects of climate change, but also about turning the planet into a world we (humans from all regions, regardless of the local conditions) want to live in. It is not only about producing energy, about reaching a number of KWh equivalent to the expected demand–renewables are about providing a service to communities, meeting their needs, and improving their ways of life. It does not consist only of taking a solar LED lamp to a remote rural house in India or Africa. It is about first understanding the problem and then seeking the right solution. Such a light will be of no use if a mother has to spend the whole day walking 10 km to find water at the closest spring or well, and come back by sunset to work on her loom, only to find that the lamp has run out of battery. Why? Because her son had to take it to school to light his way back home.
This is where the concept of ‘nexus’ entered the room, and I have to say that more than once it was brought up by IIASA Deputy Director General Nebojsa Nakicenovic. A nexus approach means adopting an integrated approach and understanding both the problems and the solutions, the cross and rebound effects, and the synergies; and it is on the latter that we should focus our efforts to maximize the effect with minimal effort. Looking at the nexus involves addressing the interdependencies between the water, energy, and food sectors, but also expanding the reach to other critical dimensions such as health, poverty, education, and gender. Overall, this means pursuing the Sustainable Development Goals (SDGs).
Another key word that was repeatedly mentioned was finance. The question was how to raise and mobilize funds for the implementation of the required solutions and initiatives. The answer: blended funding and private funding mobilization. This means combining different funding sources, including crowd funding and citizen-social funding initiatives, and engaging the private sector by reducing the risk for investors. A wonderful example was presented by the city of Vienna, where a solar power plant was completely funded (and thus owned) by Viennese citizens through the purchase of shares.
This connects with the last message: the importance of a bottom-up approach and the critical role of those at the local level. Speakers and panelists gave several examples of successful initiatives in Mali, India, Vienna, and California. Most of the debates focused on how to search for solutions and facilitate access to funding and implementation in the Global South. However, two things became clear. Firstly, massive political and investment efforts are required in emerging countries to set up the infrastructural and social environment (including capacity building) to achieve the SDGs. Secondly, the effort and cost of dismantling a well-rooted technological and infrastructural system once put in place, such as fossil fuel-based power networks in the case of developed countries, are also huge. Hence, the importance of emerging economies going directly for sustainable solutions, which will pay off in the future in all possible aspects. HRH Princess Abze Djigma from Burkina Faso emphasized that this is already happening in Africa. Progress is being made at a critical rate, triggered by local initiatives that will displace the age of huge, donor-funded, top-down projects, to give way to bottom-up, collaborative co-funding and co-development.
Overall, if I had to pick just one message among the information overload I faced over these two days, it would be the statement by a young fellow in the audience from African Champions: “Africa is not underdeveloped, it is waiting and watching not to repeat the mistakes made by the rest of the world.” We should keep this message in mind.
By Luke Kirwan,Open Access Manager at the IIASA Library
World Intellectual Property Day is celebrated annually on 26 April to bring a greater awareness of the role that intellectual property rights, such as copyright, patents, and industrial designs play in encouraging innovation and creativity. Unlike traditional property, intellectual property is intangible. It is far harder to protect one’s intellectual property from infringement or copying than it is to protect physical property. Intellectual property rights are important as, when well implemented, they provide the creator sufficient protection to benefit from their creation, but aren’t so stringent that they prevent widespread use.
Intellectual property refers to an individual’s original, intellectual creations, whether that is scientific, artistic, technical, or otherwise. As with other types of property, your intellectual property is covered by certain rights and protections automatically granted to the creator. These convey upon the owner rights over the control and utilization of their intellectual output. Depending on the situation, your intellectual property rights will also be covered by one or more types of protection, varying from patents to trademarks. These types of protection are intended to prevent unauthorized use or piracy of intellectual property, and to confer upon the creator time-limited, exclusive rights to their intellectual output.
Creative commons licenses
When you write an article, that type of intellectual output is automatically covered by copyright. This is regulated through the Berne Convention. This convention confers a number of rights to the author, including the right to translate, make adaptations, and make reproductions of a work. Depending on the specific jurisdiction in which a work is created, copyright protection lasts for the lifetime of the creator plus a specific period (circa 50 to 70 years). In terms of producing a scientific article, one of the most important rights conferred upon an author by copyright protections is the right to sell or transfer these rights to another individual. Usually, when an author publishes an article with a journal, they sign a contract ceding their copyright to the publisher. Depending on the individual publisher, the author may retain some rights, such as the ability to distribute an earlier version of their paper and the right to proper attribution. However, the journal now has control of the dissemination, distribution, translation, and reproduction rights, among others.
Creative Commons licenses are designed to assist you in keeping your research openly accessible and distributable. For a creative commons license, the author retains all of the copyright, but has licensed their work for use and reuse under different circumstances, depending on the license. When publishing a paper under a creative commons license, rather than transferring the copyright to the publisher, the author instead licenses certain rights to the publisher to allow them to distribute the work. Creative commons licenses run from CC-0, which leaves a work completely free to reuse, redistribute, alter, and utilize in any manner, to CC-BY-NC-ND, which makes a work accessible, but restricts redistribution and commercial use. Similarly some license types employ an additional stipulation known as copyleft. In terms of a creative commons license this is known as share-alike. Essentially copyleft licensing allows people to freely distribute copies and modified versions as long as they adhere to the original licensing.
If you wish to make a paper open access, a journal will usually charge an Article Processing Charge (APC). However, the IIASA library maintains agreements with several publishers that allow a work to be made open access without charge. In instances where no waiver is in place, we also have an open access fund from which IIASA researchers can apply to have part of the APC charges paid for.
Note: This article gives the views of the authors, and not the position of the Nexus blog, nor of the International Institute for Applied Systems Analysis.