Quantitative models are an important part of environmental and economic research and policymaking. For instance, IIASA models such as GLOBIOM and GAINS have long assisted the European Commission in impact assessment and policy analysis2; and the energy policies in the US have long been guided by a national energy systems model (NEMS)3.
Despite such successful modelling applications, model criticisms often make the headlines. Either in scientific literature or in popular media, some critiques highlight that models are used as if they are precise predictors and that they don’t deal with uncertainties adequately4,5,6, whereas others accuse models of not accurately replicating reality7. Still more criticize models for extrapolating historical data as if it is a good estimate of the future8, and for their limited scopes that omit relevant and important processes9,10.
Validation is the modeling step employed to deal with such criticism and to ensure that a model is credible. However, validation means different things in different modelling fields, to different practitioners and to different decision makers. Some consider validity as an accurate representation of reality, based either on the processes included in the model scope or on the match between the model output and empirical data. According to others, an accurate representation is impossible; therefore, a model’s validity depends on how useful it is to understand the complexity and to test different assumptions.
Given this variety of views, we conducted a text-mining analysis on a large body of academic literature to understand the prevalent views and approaches in the model validation practice. We then complemented this analysis with an online survey among modeling practitioners. The purpose of the survey was to investigate the practitioners’ perspectives, and how it depends on background factors.
According to our results, published recently in Eker et al. (2018)1, data and prediction are the most prevalent themes in the model validation literature in all main areas of sustainability science such as energy, hydrology and ecosystems. As Figure 1 below shows, the largest fraction of practitioners (41%) think that a match between the past data and model output is a strong indicator of a model’s predictive power (Question 3). Around one third of the respondents disagree that a model is valid if it replicates the past since multiple models can achieve this, while another one third agree (Question 4). A large majority (69%) disagrees with Question 5, that models cannot provide accurate projects, implying that they support using models for prediction purposes. Overall, there is no strong consensus among the practitioners about the role of historical data in model validation. Still, objections to relying on data-oriented validation have not been widely reflected in practice.
Figure 1: Survey responses to the key issues in model validation. Source: Eker et al. (2018)
According to most practitioners who participated in the survey, decision-makers find a model credible if it replicates the historical data (Question 6), and if the assumptions and uncertainties are communicated clearly (Question 8). Therefore, practitioners think that decision makers demand that models match historical data. They also acknowledge the calls for a clear communication of uncertainties and assumptions, which is increasingly considered as best-practice in modeling.
One intriguing finding is that the acknowledgement of uncertainties and assumptions depends on experience level. The practitioners with a very low experience level (0-2 years) or with very long experience (more than 10 years) tend to agree more with the importance of clarifying uncertainties and assumptions. Could it be because a longer engagement in modeling and a longer interaction with decision makers help to acknowledge the necessity of communicating uncertainties and assumptions? Would inexperienced modelers favor uncertainty communication due to their fresh training on the best-practice and their understanding of the methods to deal with uncertainty? Would the employment conditions of modelers play a role in this finding?
As a modeler by myself, I am surprised by the variety of views on validation and their differences from my prior view. With such findings and questions raised, I think this paper can provide model developers and users with reflections on and insights into their practice. It can also facilitate communication in the interface between modelling and decision-making, so that the two parties can elaborate on what makes their models valid and how it can contribute to decision-making.
Model validation is a heated topic that would inevitably stay discordant. Still, one consensus to reach is that a model is a representation of reality, not the reality itself, just like the disclaimer of René Magritte that his perfectly curved and brightly polished pipe is not a pipe.
Eker S, Rovenskaya E, Obersteiner M, Langan S. Practice and perspectives in the validation of resource management models. Nature Communications 2018, 9(1): 5359. DOI: 10.1038/s41467-018-07811-9 [pure.iiasa.ac.at/id/eprint/15646/]
Nuccitelli D. Climate scientists just debunked deniers’ favorite argument. The Guardian. 2017. https://www.theguardian.com/environment/climate-consensus-97-per-cent/2017/jun/28/climate-scientists-just-debunked-deniers-favorite-argument
By Marcus Thomson, researcher, IIASA Ecosystems Services and Management Program
While living in Cairo in 2010, I witnessed first-hand the human toll of political and environmental disasters that washed over Africa at the end of the last century. Unprecedented numbers of migrants were pressing into North Africa, many pushed out of their homelands by conflict and state-failure, pulled towards safer, richer, less fragile places like Europe. Throughout Sub-Saharan Africa, climate change was driving up competition for scarce land and water, and raising pressure on farmers to maintain the quantity and quality of their crops.
It is a similar story throughout the developing world, where many farmers do without the use of expensive chemical fertilizer and pesticides, complex irrigation, or boutique seed varieties. They rely instead on traditional land management practices that developed over long periods with consistent, predictable conditions. It is difficult to predict how dryland farmers will respond to climate change; so it is challenging to plan for various social, economic, and political problems expected to develop under, or be exacerbated by, climate change. Will it spur innovation or, as has been argued for the Syrian civil war, set up conflict? A major stumbling block is that the dynamics of human social behavior are so difficult to model.
Instead of attempting to predict farmers’ responses to climate change by modelling human behavior, we can look to the responses to environmental changes of farmers from the past as analogues for many subsistence farmers of the future. Methods to fill in historical gaps, and reconstruct the prehistoric record, are valuable because they expand the set of observed cases of societal-scale responses to environmental change. For instance, some 2000 years ago, an expansive maize-growing cultural complex, the Ancestral Puebloans (APs), was well established in the arid American Southwest. By AD 1000, members of this AP complex produced unique and innovative material culture including the famed “Great Houses”, the largest built structures in the United States until the 19th century. However, between AD 1150 and 1350, there was a profound demographic transformation throughout the Southwest linked to climate change. We now know that many APs migrated elsewhere. As a PhD student at the University of California, Los Angeles, I wondered whether a shift to cooler, more variable conditions of the “Little Ice Age” (LIA, roughly AD 1300 to 1850) was linked to the production of their staple crop, maize.
I came to IIASA as a YSSP in 2016 to collaborate with crop modelers on this question, and our work has just been published in the journal Quaternary International. I brought with me high-resolution data from a state-of-the-art climate model to drive the crop simulations, and AP site information collected by archaeologists. Because AP maize was quite different from modern corn, I worked with IIASA soil scientist Juraj Balkovič to modify the crop simulator with parameters derived from heirloom varieties still grown by indigenous peoples in the Southwest. I and IIASA economic geographer Tamás Krisztin developed a statistical technique to analyze the dynamical relationship between AP site occupation and simulated yield outcomes.
We found that for the most climate-stressed high-elevation sites, abandonments were most associated with increased year-to-year yield variability; and for the least stressed low-elevation and well-watered sites, abandonment was more likely due to endogenous stressors, such as soil degradation and population pressure. Crucially, we found that across all regions, populations peaked during periods of the most stable year-to-year crop yields, even though these were also relatively warm and dry periods. In short, we found that AP maize farmers adapted well to gradually rising temperatures and drought, during the MCA, but failed to adapt to increased climate variability after ~AD 1150, during the LIA. Because increased variability is one of the near certainties for dryland farming zones under global warming, the AP experience offers a cautionary example of the limits of low-technology adaptation to climate change, a business-as-usual direction for many sub-Saharan dryland farmers.
This is a lesson from the past that policymakers might take note of.
 Kelley, C. P., Mohtadi, S., Cane, M. A., Seager, R., & Kushnir, Y. (2015). Climate change in the Fertile Crescent and implications of the recent Syrian drought. Proceedings of the National Academy of Sciences, 201421533.
 Thomson, M. J., Balkovič, J., Krisztin, T., MacDonald, G. M. (2018). Simulated crop yield for Zea mays for Fremont Ancestral Puebloan sites in Utah between 850-1499 CE based on temperature dailies from a statistically downscaled climate model. Quaternary International. https://doi.org/10.1016/j.quaint.2018.09.031
Rafael Morais is a recent participant in the IIASA-CAPES Doctorate Sandwich Program, he spent nine months at IIASA working in the Energy program.
In 2016, the Brazilian Federal Agency for Support and Evaluation of Graduate Education (CAPES) partnered with IIASA on a new initiative offering support to doctoral and postdoctoral researchers interested in collaborating with established IIASA researchers. As part of this initiative, IIASA and CAPES annually offer up to three fellowships for Brazilian PhD students to spend three to twelve months at IIASA as part of the joint IIASA-CAPES Doctorate Sandwich Program, as well as up to four postdoc fellowships that enable Brazilian researchers to work at IIASA for up to 24 months.
Rafael Morais, a PhD candidate at the Energy Planning Program of the Federal University of Rio de Janeiro, was part of the first group of Brazilian PhD students funded by CAPES to participate in this program. He spent nine months with the Energy Program at IIASA in 2017. We recently caught up with him and asked him about his research and what the fellowship has meant to him:
What is your PhD research about?
My research involves modeling the contribution of renewable energy sources in electric systems. My doctorate thesis includes a case study on Brazil, where we have large potential for wind and solar power generation in various regions. My main objective is to investigate how total costs develop considering the number of wind and solar plants in the Brazilian electricity system.
Why did you choose IIASA for your doctorate program (over other places)?
I chose IIASA because it is a very reputable think tank for energy and model development. People are very capable and well prepared. They have been working on energy systems modeling for many years, and their experience motivated my decision to come to IIASA. I talked with some people that were at IIASA before me and they were all very grateful for the experience. Another important factor was that it is an international institute, where one can have contact with people from many different countries, and the main language is English.
How did your participation in the program benefit you?
I had the opportunity to get into contact with diverse approaches to my research questions, thus enriching my thesis. Unlike my home institution, IIASA does not have only energy experts, but also computer scientists, mathematicians, and physics experts, all working in the same group, and all contributing to a great modeling team. Being here was an excellent opportunity to collaborate with them. As my first experience abroad, it was also a chance for me to grow and develop other skills, both on a professional and a personal level.
Would you recommend that people apply for the IIASA-CAPES doctorate program?
Yes, I would definitely recommend it! IIASA is a very nice place to work. People really care about a harmonious work environment, and IIASA staff are always available to help you with any issue. Apart from that, the people that I worked with during my time here are very knowledgeable and kind. In short, it was a great experience being at IIASA for nine months during my PhD.
Applications for the 2019 IIASA-CAPES Doctorate Sandwich Program and Postdoctoral Fellowship Program opened on 1 September 2018 and will run until 15 October 2018. Candidates have to apply to both CAPES (on the CAPES website) and IIASA. Successful applicants will be informed of the selection results by mid-December 2018. Selected candidates are expected to take up their position at IIASA between March and October 2019.
Note: This article gives the views of the author, and not the position of the Nexus blog, nor of the International Institute for Applied Systems Analysis.
Last year, I had the fantastic opportunity to spend three months at IIASA as part of the Young Scientists Summer Program (YSSP), to collaborate with the Ecosystems Services and Management (ESM) research program. During this very enriching experience, both intellectually, socially, and culturally, I worked with Petr Havlik, David Leclère, and Christian Folberth on modeling global rangelands and pasturelands under farming and climate scenarios. I also progressed on the development of a global animal stocking rate optimizer. The overall objective of this YSSP project, and more broadly of my PhD, is to assess the role of grazing systems in a sustainable food system.
However, my trip to IIASA was not my only adventure last year. Just before moving to Vienna, I received the great news that I was selected along with 77 other women to take part in a women in science and leadership program called Homeward Bound.
What would our world look like if women and men were equally represented, respected, and valued at the leadership table? How might we manage our resources and our communities differently? How might we coordinate our response to global problems like food security and climate change?
Homeward Bound is a worldwide and world-class initiative that seeks to support and encourage women with scientific backgrounds into leadership roles, believing that diversity in leadership is key to addressing these complex and far-reaching issues. The program’s bold mission is to create a 1000-strong collective of women in science around the world over the next 10 years, with the enhanced leadership, strategic, and visibility capacity to influence policy and decision making for the benefit of the planet.
This year-long program culminated in an intensive three-week training course in Antarctica, a journey from which I have just come back. The voyage to Antarctica was incredible. We learnt intensively during this 24/7 floating conference in the midst of majestic icebergs, very cute penguins, graceful whales, and extraordinary women from various cultures and backgrounds, from PhD students to Nobel Laureates. I have returned full of hope for the planet, deeply inspired, and emotionally energized. It was a truly unforgettable experience, one that will keep me reflecting for a lifetime.
Our days in Antarctica typically followed a similar routine – half of the day was dedicated to a landing (we visited Argentinian, Chinese, US, and UK research stations) and the other half to classes and workshops. We discussed systemic gender issues and learnt about leadership styles, peer-coaching, the art of providing feedback, science communication, core personal values, or what matter to us. The list goes on! We were also encouraged to practice reflective journaling. Regularly recording activities, situations, and thoughts on paper is actually a very powerful technique for self-discovery and personal and professional growth as it helps us think in a critical and analytical way about our behaviors, values, and emotions. We also spent quite some time developing our personal and professional strategies: What is our purpose as individuals? What are our core values, aspirations, and short- and long-term goals? From that, we developed a roadmap that could be executed as soon as we stepped off the ship. While I haven’t solved all my life’s mysteries, this activity gave me strong foundations to keep growing and actively shape my own life, rather than letting society do it for me.
In the evenings, we watched our film faculty sharing their tips with us on television, including primatologist Jane Goodall, world leading marine biologist Sylvia Earle, and former Executive Secretary of the UN Framework Convention on Climate Change (UNFCC), Christiana Figueres. We also had a collective art project called “Confluence: A Journey Homeward Bound”, which was underpinned by our inner journey of reflection, growth, and transformation and our outer physical journey to Antarctica.
Both my stay at IIASA and my journey to Antarctica taught me a lot about the value of getting out of my comfort zone, exploring different leadership styles, and collaborating. I have also witnessed how visibility (visibility to ourselves, to understand who we are, and visibility to others, to let the world know we exist) helps to open up opportunities. The good news is that the beliefs we have about ourselves are just that – beliefs – and these beliefs can be changed.
My visibility to others has also increased notably in relation to my involvement in Homeward Bound and my recent award of the Queensland Women in STEM prize. This Australian annual prize, awarded by the Minister for Environment and Science, Leeanne Enoch and Acting Chief Scientist Dr Christine Williams, aims to celebrate the achievements of women who are making a difference in the fields of science, technology, engineering, and mathematics. As a result, I have been contacted by fascinating people from various fields of work, from researchers and teachers to entrepreneurs, start-ups, and industries. All these connections have broadened my approach to food security and global change and helped me shape my research vision, purpose, and values.
When we were in Antarctica, our story reached 750 million people. Why? Because, and may we never forget, the world believes in us – ‘us’ in its broadest sense: humans, scientists, women, etc. – in our skill, compassion, and capability. While we are facing alarming global social, economic, and environmental challenges, I believe that the many collaborations that embrace diversity of knowledge, skills, processes, and leadership styles that are currently emerging all around the world, will help us get closer to our development goals.
Brian, now 71, is one of the most influential early thinkers of the SFI, a place that without exaggeration could be called the cradle of complexity science.
Brian became famous with his theory of increasing returns. An idea that has been developed in Vienna, by the way, where Brian was part of a theoretical group at the IIASA in the early days of his career: from 1978 to 1982.
“I was very lucky,” he recalls. “I was allowed to work on what I wanted, so I worked on increasing returns.”
The paper he wrote at that time introduced the concept of positive feedbacks into economy.
The concept of “increasing returns”
Increasing returns are the tendency for that which is ahead to get further ahead, for that which loses advantage to lose further advantage. They are mechanisms of positive feedback that operate—within markets, businesses, and industries—to reinforce that which gains success or aggravate that which suffers loss. Increasing returns generate not equilibrium but instability: If a product or a company or a technology—one of many competing in a market—gets ahead by chance or clever strategy, increasing returns can magnify this advantage, and the product or company or technology can go on to lock in the market.”
(W Brian Arthur, Harvard Business Review 1996)
This was a slap in the face of orthodox theories which saw–and some still see–economy in a state of equilibrium. “Kind of like a spiders web,” Brian explains me in our short conversation last Friday, “each part of the economy holding the others in an equalization of forces.”
The answer to heresy in science is that it does not get published. Brian’s article was turned down for six years. Today it counts more than 10.000 citations.
At the latest it was the development and triumphant advance of Silicon Valley’s tech firms that proved the concept true. “In fact, that’s now the way how Silicon Valley runs,” Brian says.
The youngest man on a Stanford chair
William Brian Arthur is Irish. He was born and raised in Belfast and first studied in England. But soon he moved to the US. After the PhD and his five years in Vienna he returned to California where he became the youngest chair holder in Stanford with 37 years.
Five years later he changed again – to Santa Fe, to an institute that had been set up around 1983 but had been quite quiet so far.
Q: From one of the most prestigious universities in the world to an unknown little place in the desert. Why did you do that?
A: In 1987 Kenneth Arrow, an economics Nobel Prize winner and mentor of mine, said to me at Stanford: We’re holding a small conference in September in a place in the Rockies, in Santa Fe, would you go?
When a Nobel Prize winner asks you such a question, you say yes of course. So I went to Santa Fe.
We were about ten scientists and ten economists at that conference, all chosen by Nobel Prize winners. We talked about the economy as an evolving complex system.
Veni, vidi, vici
Brian came – and stayed: The unorthodox ideas discussed at the meeting and the “wild” and free atmosphere of thinking at “the Institute”, as he calls the Santa Fe Institute (SFI), thrilled him right away.
In 1988 Brian dared to leave Stanford and started to set up the first research program at Santa Fe. Subject was the economy treated as a complex system.
Q: What was so special about SF?
A: The idea of complexity was quite new at that time. But people began to see certain patterns in all sorts of fields, whether it was chemistry or the economy or parts of physics, that interacting elements would together create these patterns…To investigate this in universities with their particular disciplines, with their fixed theories, fixed orthodoxies–where it is all fixed how to do things–turned out to be difficult.
Take the economy for example. Until then people thought it was in an equilibrium. And there we came and proved, no, economics is no equilibrium! The Stanford department would immediately say: You can’t do that! Don’t do that! Or they would consider you to be very eccentric…
So a bunch of senior fellows at Los Alamos in the 1980s thought it would be a good idea if there was an independent institute to research these common questions that came to be called complexity.
At Santa Fe you could talk about any science and any basic assumptions you wanted without anybody saying you couldn’t or shouldn’t do that.
Our group as the first there set a lot of this wild style of research. There were lots of discussions, lots of open questions, without particular disciplines… In the beginning there were no students, there was no teaching. It was all very free.
This wild style became more or less the pattern that has been followed ever since. I think the Hub is following this model too.
The magic formula for excellence
Q: Was this just a lucky concurrence: the right people and atmosphere at the right time? Or is there a pattern behind it that possibly could be repeated?
A: I am sure: If you want to do interdisciplinary science – which complexity is: It is a different way of looking at things! – you need an atmosphere where people aren’t reinforced into all the assumptions of the different disciplines.
This freedom is crucial to excellent science altogether. It worked out not only for Santa Fe. Take the Rand Corporation for instance, that invented a lot of things including the architecture of the internet, or the Bell Labs in the Fifties that invented the transistor. The Cavendish Lab in Cambridge is another one, with the DNA or nuclear astronomy…
The magic formula seems to be this:
First get some first rate people. It must be absolutely top-notch people, maybe ten or twenty of them.
Make sure they interact a lot.
Allow them to do what they want – be confident that they will do something important.
And then when you protect them and see that they are well funded, you are off and running.
Probably in seven cases out of ten that will not produce much. But quite a few times you will get something spectacular – game changing things like quantum theory or the internet.
Don’t choose programs, choose people
Q: This does not seem to be the way officials are funding science…
A: Yes, in many places you have officials telling people what they need to research. Or where people insist on performance and indices… especially in Europe, I have the impression, you have a tradition of funding science by insisting on all these things like indices and performance and publications or citation numbers. But that’s not a very good formula.
Excellence is not measurable by performance indicators. In fact that’s the opposite of doing science.
I notice at places where everybody emphasize all this they are not on the forefront. Maybe it works for standard science; and to get out the really bad science. But it doesn’t work if you want to push boundaries.
Many officials don’t understand that.
In Singapore the authorities once asked me: How did you decide on the research projects in Santa Fe? I said, I didn’t decide on the research projects. They repeated their question. I said again, I did not decide on the research projects. I only decided on people. I got absolutely first rate people, we discussed vaguely the direction we wanted things to be in, and they decided on their research projects.
That answer did not compute with them. They are the civil service, they are extraordinarily bright, they’ve got a lot of money. So they think they should decide what needs to be researched.
I should have told them – I regret I didn’t: This is fine if you want to find solutions for certain things, like getting the traffic running or fixing the health care system. Surely with taxpayer’s money you have to figure such things out. But you will never get great science with that. All you get is mediocrity.
Of course now they asked, how do we decide which people should be funded? And I said: “You don’t! Just allow top people to bring in top people. Give them funding and the task of being daring.”
Any other way of managing top science doesn’t seem to work.
I think the Hub could be such a place – all the ingredients are here. Just make sure to attract some more absolutely first rate people. If they are well funded the Hub will put itself on the map very quickly.