What did we learn from COVID-19 models?

By Sibel Eker, researcher in the IIASA Energy Program

IIASA researcher Sibel Eker explores the usefulness and reliability of COVID-19 models for informing decision making about the extent of the epidemic and the healthcare problem.

© zack Ng 99 | Dreamstime.com

In the early days of the COVID-19 pandemic, when facts were uncertain, decisions were urgent, and stakes were very high, both the public and policymakers turned not to oracles, but to mathematical modelers to ask how many people could be infected and how the pandemic would evolve. The response was a plethora of hypothetical models shared on online platforms and numerous better calibrated scientific models published in online repositories. A few such models were announced to support governments’ decision-making processes in countries like Austria, the UK, and the US.

With this announcement, a heated debate began about the accuracy of model projections and their reliability. In the UK, for instance, the model developed by the MRC Centre for Global Infectious Disease Analysis at Imperial College London projected around 500,000 and 20,000 deaths without and with strict measures, respectively. These different policy scenarios were misinterpreted by the media as a drastic variation in the model assumptions, and hence a lack of reliability. In the US, projections of the model developed by the University of Washington’s Institute for Health Metrics and Evaluation (IHME) changed as new data were fed into the model, sparking further debate about the accuracy thereof.

This discussion about the accuracy and reliability of COVID-19 models led me to rethink model validity and validation. In a previous study, my colleagues and I showed that, based on a vast scientific literature on model validation and practitioners’ views, validity often equates with how good a model represents the reality, which is often measured by how accurately the model replicates the observed data. However, representativeness does not always imply the usefulness of a model. A commentary following that study emphasized the tradeoff between representativeness and the propagation error caused by it, thereby cautioning against an exaggerated focus on extending model boundaries and creating a modeling hubris.

Following these previous studies, in my latest commentary in Humanities and Social Sciences Communications, I briefly reviewed the COVID-19 models used in public policymaking in Austria, the UK, and the US in terms of how they capture the complexity of reality, how they report their validation, and how they communicate their assumptions and uncertainties. I concluded that the three models are undeniably useful for informing the public and policy debate about the extent of the epidemic and the healthcare problem. They serve the purpose of synthesizing the best available knowledge and data, and they provide a testbed for altering our assumptions and creating a variety of “what-if” scenarios. However, they cannot be seen as accurate prediction tools, not only because no model is able to do this, but also because these models lacked thorough formal validation according to their reports in late March. While it may be true that media misinterpretation triggered the debate about accuracy, there are expressions of overconfidence in the reporting of these models, even though the communication of uncertainties and assumptions are not fully clear.

© Jaka Vukotič | Dreamstime.com

© Jaka Vukotič | Dreamstime.com

The uncertainty and urgency associated with pandemic decision-making is familiar to many policymaking situations from climate change mitigation to sustainable resource management. Therefore, the lessons learned from the use of COVID models can resonate in other disciplines. Post-crisis research can analyze the usefulness of these models in the discourse and decision making so that we can better prepare for the next outbreak and we can better utilize policy models in any situation. Until then, we should take the prediction claims of any model with caution, focus on the scenario analysis capability of models, and remind ourselves one more time that a model is a representation of reality, not the reality itself, like René Magritte notes that his perfectly curved and brightly polished pipe is not a pipe.

References

Eker S (2020). Validity and usefulness of COVID-19 models. Humanities and Social Sciences Communications 7 (1) [pure.iiasa.ac.at/16614]

Note: This article gives the views of the author, and not the position of the Nexus blog, nor of the International Institute for Applied Systems Analysis.

COVID19 – Let’s Not Forget the Humans

By Nicole Arbour, external relations manager in the IIASA Communications and External Relations department

As Canadian expats in Austria, one of the things that has particularly struck my family and I is the orderliness with which the country is dealing with the pandemic. As quarantine policies were put into place, we saw panic toilet paper hoarding in other countries, but here in Austria people were (amazingly) compliant and seemed to obey instructions and timelines provided by the authorities. We never worried about our basic needs.  Grocery stores were always well stocked, public transit was always there and on time – and masks were readily available when required as physical barrier to protect others.

© Luca Santilli | Dreamstime.com

Expert opinions, governments, and publics are making it clear that there is no one-size-fits-all solution to this pandemic.  What works in Austria might not be what worked for South Korea; and likely not the same as what works in other parts of Europe.  Consider the Canadian landscape.  There is huge variation in sociopolitical and cultural dynamics between and within provinces and territories.  What works for some parts of Canada (virtual home schooling, grocery shopping) is impossible for others (Canada’s North).  Cultural norms (multigenerational living, child/elder care) vary across the vast landscape. The “At Home on the Land” initiative – aimed at the particular needs of Indigenous communities is an example of a culturally-grounded way to address the pandemic. Finding solutions isn’t always as intuitive as we might like.

Humans tend to look for the easiest way out – we want simple solutions to complex problems.  We don’t seem to want to think about the problems, we want them magically disappear. And thinking “outside of the box” isn’t always appreciated.  Hand washing, clean water and the advent of antibiotics have made enormous leaps in our ability to tackle public health outbreaks – significant results. Where the bubonic plague is estimated to have killed 30%-60% of Europe’s population in the Middle Ages, modern outbreaks are now quickly identified and contained (were you even aware of the 2017 outbreak in Madagascar?). Understanding transmission routes has significantly impacted public health outcomes.  The identification of tainted water as a vector for cholera transmission by John Snow led to the advent of modern epidemiology. But, as we find solutions to larger challenges, those that remain are more complex with increasing numbers of variables making solutions harder to come by.

There is some global agreement: lots of testing, quick results/containment, use of masks/physical barriers for community protection, social distancing, data collection. However, certain measures work better in some jurisdictions than others. What policies and practices are working and why are they working in these contexts?  What is applicable in different contexts?

Our current global situation, has reminded me of a presentation I saw on the 2014 Ebola outbreak (Professor Melissa Leach, IDS), and how important it is to remember the human factor in crises. She discussed how the key elements that made the Ebola pandemic so persistent – despite the best efforts of global public health engagement – was a/the failure to understand how historic context, trust, cultural dynamics played into the spread of the virus. Those providing interventions did not appreciate how historic context (i.e. post-colonialism, slavery, medical testing scandals) and mistrust in the intentions of Western interventions factored into the willingness of the local population to accept the solutions provided. Awareness of social structures, influencers and leaders, and co-creation were also important to developing solutions that would be adopted by affected communities.

Evidence is more than the numbers of tests, infections and deaths. It is understanding the social context of communities, society writ large, and how they interact within and between.  It’s about understanding historical context and how it feeds into local culture, social interactions and trust relationships.  It’s about community dynamics, power struggles and the struggle for some to meet basic survival needs.  It’s about timing of decision-making, political landscapes and different ways of leading.  As with many of our global challenges, it’s a complex and multifaceted systems problem – in which the human factor is a huge driver.

As we strive for solutions to this global crisis – bring on innovation, research and science funding. We will need these – but please, also bring along those who study the complexity that is humanity: epidemiologists, anthropologists, economists, ethicists, political scientists, sociologists, futurists, etc. In an era where evidence is being questioned, fake news is rampant and anti-science sentiments are strong, it is crucial that we remember that one piece to engaging with this and the world’s other wicked problems is our relationships with our communities – the ones we are trying to protect. Public trust, built on understanding of the importance of human dynamics is key to broad acceptance and uptake. Solutions need to be palatable to society, or they won’t be adopted.

As we focus on the virus, let’s not forget the humans.

Note: This article gives the views of the author, and not the position of the Nexus blog, nor of the International Institute for Applied Systems Analysis. Link to original post: https://sciencepolicy.ca/news/save-covid19-lets-not-forget-humans

Climate change or behavior change?

By Leila Niamir, post-doctoral researcher at the Mercator Research Institute on Global Commons and Climate Change (MCC), Germany and IIASA YSSP alumna.

© Cienpies Design Illustrations | Dreamstime

 

Weather patterns and events are changing and becoming more extreme, sea levels are rising, and greenhouse gas emissions are now at their highest levels in history[1]. Climate change is affecting every individual in every city on every continent. It imposes adverse impact on people, communities, and countries, disrupting regional and national economies.

Climate change mitigation refers to efforts to reduce or prevent emissions of greenhouse gases to limit the magnitude of long-term climate change. Human consumption, in combination with a growing population, contributes to climate change by increasing the rate of greenhouse gas emissions. Over the last decade, instigated by the Paris Agreement, the efforts to limit global warming have been expanding. Significant attention is being devoted to new energy technologies on both the production and consumption sides, however, changes in individual behavior and management practices as part of the mitigation strategy are often neglected[2]. This might derive from the complex nature of human which makes explaining and affecting human behavior a difficult task. As a result, quantitative tools to assess household emissions, considering the diversity of behaviors and a variety of psychological and social factors influencing them beyond purely economic considerations, are scarce. Policymakers would benefit from reliable decision supporting tools that explore the interaction of economic decision-making and behavioral heterogeneity in households behavioral and lifestyle changes, when testing climate mitigation policies (e.g. carbon pricing, subsidies)[3].

To address this issue, during my PhD research I studied the potential of behavioral changes among heterogeneous households regarding energy use and their role in mitigating climate change. By designing and conducting comprehensive household surveys, it was explored how individuals choose to change their energy behaviour and what factors trigger or inhibit these choices[4]. Decision support tools are designed to study large-scale regional effects of individual actions, and to explore how they may change over time and space. The model explicitly treats behavioral triggers and barriers at the individual level, assuming that energy use decision making is a multi-stage process. This theoretically and empirically grounded simulation model offers policymakers ways to explore various policy portfolios by running diverse micro and macro scenarios.

This model was further developed during my collaboration with the IIASA the Young Scientists Summer Program (YSSP), to estimate macro impacts of individuals’ energy behavioral changes on carbon emissions[5]. Within this research, we illustrate that individual energy behavior, especially when amplified through social context, shapes energy demand and, consequently, carbon emissions. Our results show that residential energy demand is strongly linked to personal and social norms. When assessing the cumulative impacts of these behavioral processes, we quantify individual and combined effects of social dynamics and of carbon pricing on individual energy efficiency and on the aggregated regional energy demand and emissions.

In summary, mitigating climate change requires massive worldwide efforts and strong involvement of regions, cities, businesses and individuals, in addition to the commitments at the national levels. We should always keep in mind that every single behavior matters. In the transition to a sustainable and resilient society, we –as individuals- are more than just consumers.

Note: This article gives the views of the author, and not the position of the Nexus blog, nor of the International Institute for Applied Systems Analysis.

[1] Climate Action– United Nations Sustainable Development Goals https://www.un.org/sustainabledevelopment/climate-change/
[2] Creutzig, F., et al. (2018). Towards demand-side solutions for mitigating climate change. Nature Climate Change 8, 268-271; Grubler, A., et al. (2018). A low energy demand scenario for meeting the 1.5 degrees C target and sustainable development goals without negative emission technologies. Nature Energy 3, 515-527; Creutzig, F., et al. (2016). Beyond Technology: Demand-Side Solutions for Climate Change Mitigation. Annual Review of Environment and Resources, Vol 41 41, 173-198
[3] Niamir, L. (2019). Behavioural Climate Change Mitigation: from individual energy choices to demand-side potential (University of Twente); Creutzig, F., et al. (2018). Towards demand-side solutions for mitigating climate change. Nature Climate Change 8, 268-271; Niamir, L., et al. (2018). Transition to low-carbon economy: Assessing cumulative impacts of individual behavioural changes. Energy Policy, 118; Stern N. Economics: Current climate models are grossly misleading. Nature 530(7591):407–9.
[4] Niamir, L. et al. (2020). Demand-side solutions for climate mitigation: Bottom-up drivers of household energy behaviour change in the Netherlands and Spain. Energy Research & Social Science, 62, 101356.
[5] The results of this collaboration was presented at Impacts World 2017 and won the best prize, and also published at Climatic Change Journal.

Vienna as a global hub for science diplomacy

By Jan Marco Müller, IIASA Acting Chief Operations Officer

Jan Marco Müller shares his insights into the recent high-level forum in Vienna that brought together science advisors to ministers of foreign affairs from across the world and other experts in the practice, theory, and discussion of science diplomacy.

Established following an initiative by the United States and the Soviet Union during the Cold War, IIASA can be considered a child of diplomacy for science. At the same time, the institute has always been one of the world’s premier vehicles of science for diplomacy, by using science to build bridges between nations including those with special relations. However, there is another dimension of science diplomacy which has gained traction in recent years: the support scientists can provide to diplomats and policymakers in the foreign policy domain – known as science in diplomacy.

© IIASA

As global challenges become more complex and interdependent and technological progress advances at an ever-increasing speed, the scientific-technical dimension of foreign policies has gained increasing attention. This is illustrated by four examples:

  • Climate change impacts everybody on the planet, regardless of national borders.
  • Many digital technologies escape national jurisdictions and so create tensions between nations: e.g. cryptocurrencies, deep fakes, and internet trolls.
  • Trade agreements are often hampered by disagreements on technical standards, which themselves are influenced by societal values: people may still remember the discussions around chlorinated chicken in the US-EU trade negotiations a few years ago.
  • National interests are increasingly entering international spaces, which in the past have been governed by science, such as the Arctic/Antarctic, the deep sea and outer space.

Ministries of foreign affairs and diplomatic services around the world are all confronted with similar issues and critically depend on advice provided by scientists.

With this in mind, on 25-26 November 2019 IIASA, together with the International Network for Government Science Advice (INGSA), the Austrian Federal Ministry of Europe, Integration and Foreign Affairs, the Diplomatische Akademie Wien (Vienna School of International Studies), and the Natural History Museum Vienna, held the global meeting of the Foreign Ministries Science & Technology Advice Network (FMSTAN).

FMSTAN gathers science advisors to ministers of foreign affairs from around the planet, providing a platform for the exchange of information and best practices. IIASA hosted the first meeting of this network in October 2016, which has since grown significantly, with some 50 countries now participating in its biannual meetings.

The global meeting in November was organized back to back with the meetings of two other important networks in the science diplomacy arena: the Science Policy in Diplomacy and External Relations Network (SPIDER) – which is the science diplomacy branch of the International Network for Government Science Advice – and the Big Research Infrastructures for Diplomacy and Global Engagement through Science (BRIDGES) Network. BRIDGES was established a year ago following an initiative by my colleague Maurizio Bona at CERN and myself, with the aim of uniting the science diplomacy officers of all major international research infrastructures. In addition, a 3-day training course organized by the EU-funded project Using science for/in diplomacy for addressing global challenges (S4D4C) was arranged in parallel to achieve maximum synergies.

© IIASA (L-R): Jan Marco Müller and Jeffrey Sachs, Special Advisor to the UN Secretary-General on SDGs; Director of the Center for Sustainable Development at Columbia University, NY.

The meetings were attended by around 100 science diplomats including the President-elect of the International Science Council Sir Peter Gluckman, the UN Advisor on the Sustainable Development Goals Jeffrey Sachs, the former Rector of the University for Peace of the UN Martin Lees, the S&T Advisor to the US Secretary of State Matt Chessen, the S&T Advisor to the Japanese Foreign Minister Teruo Kishi, the Science Diplomacy Advisor to the Mexican Foreign Minister José Ramón López Portillo, and the Chief Science Advisor in the Dutch Ministry of Foreign Affairs Dirk-Jan Koch, to name just a few.

Six major topics were discussed:

  • The role of science diplomacy in achieving the Sustainable Development Goals
  • The importance of science in international security policies
  • The challenges for science diplomacy in the current geopolitical environment
  • Technology Diplomacy
  • The role of science in diplomatic curricula (and vice versa)
  • Future challenges for science diplomacy and the role of systemic thinking in policymaking

The Vienna meeting offered a unique platform for all those who “speak science” in the diplomatic arena to exchange ideas and experiences, while fostering a common global agenda. For additional insights I recommend reading the piece “Science Diplomacy: A Pragmatic Perspective from the Inside” which aims at making the term science diplomacy more operational – all the four authors participated to the Vienna meeting.

Overall the event demonstrated once again the convening power of IIASA and the leadership of the institute in confirming Vienna as one of the global hubs for science diplomacy.

© Mahmood | BMEIA. FMSTAN/SPIDER members and participants of the S4D4C training course.

Note: This article gives the views of the author, and not the position of the Nexus blog, nor of the International Institute for Applied Systems Analysis.

Ethical research is a quest for truth

Bettina Greenwell, communications officer at IIASA, talks to Dorsamy (Gansen) Pillay, Deputy CEO: Research and Innovation Support and Advancement (RISA), National Research Foundation (NRF), and IIASA council member for South Africa, about the NRF’s statement on ethical research and scholarly publishing practices. The statement was jointly issued in August 2019 with South African partners within the National System of Innovation (NSI) in South Africa.

Dorsamy (Gansen) Pillay, Deputy CEO: Research and Innovation Support and Advancement (RISA), National Research Foundation (NRF), and IIASA council member for South Africa

What is ethics in research and why is it so important?

Research is a quest for truth. The research must be well conceptualized with a clear research question(s) which can lead to new knowledge. Good ethics and integrity dictate that the truth must be presented in its absolute form, and the findings need to be appropriately interpreted and should be reproducible. 

South Africa was awarded the right to host the 7th World Conference on Research Integrity (WCRI) in 2021 in Cape Town – the first time it will be hosted on an African continent. Why is this statement important for the science community in South Africa?

Firstly, it is a privilege to host this conference and South Africa feels very honored. The statement on ethical research and scholarly publishing practices is an important contribution to this conference. We have noticed that South African academics and researchers, especially new and emerging researchers, are under a lot of pressure to publish their work for a variety of reasons. In some instances, ethical principles have been violated. This included the dissemination of research through predatory journals.  However, this was not unique to South Africa only as other countries also faced similar challenges.  The NRF as a science granting foundation felt compelled to respond to this challenge. The NRF sees itself as a custodian and guardian of research ethics and integrity. Through our peer-review processes, we ensure that research proposals for funding have been robustly interrogated, and the highest ethical principles upheld. As a consequence the NRF developed and issued a joint statement on ethical research and scholarly publications in collaboration with the Department of Science and Innovation (DSI), the Department of Higher Education and Training (DHET), the Council on Higher Education (CHE), the Academy of science of South Africa (ASSAf), Universities South African (USAf) and the NRF. In addition, the NRF has previously issued a statement on predatory publishing.

What do you think will be the key scientific challenges to face South Africa in the next few years? And how do you envision IIASA helping South Africa to tackle these?

There are several challenges, and some of the most pressing ones are poverty and inequality, population migration and unemployment. Given the systems analysis approach, we feel we can draw on IIASA’s expertise to address these challenges. IIASA has used South Africa as a laboratory for its population studies research over several years. It is now time to ensure that this research is translated into policies so that it may impact positively on society.

Housing is also a problem in South Africa. There is a lack of decent, affordable housing for people. The new IIASA strategic plan focuses on smart cities – this could play a role in addressing these housing challenges.

IIASA’s expertise is a systems analysis approach which can be applied to complex issues. The important part of the work is when scientific results are turned into policy – that’s when there is an actual, tangible societal benefit.

South Africa has been an IIASA member since 2007. What have been the highlights of the South Africa-IIASA membership until now? 

We see the South Africa IIASA membership as a partnership, and many benefits have accrued through this partnership over the past decade. An example is the Southern African Young Scientists Summer Program (SA-YSSP), which was inspired by the success of the IIASA YSSP. This program ran from 2012 to 2015, and trained the next generation of young scientists.

Another example is the Southern African Systems Analysis Centre (SASAC) initiative, which focused on expanding systems analysis expertise in Southern Africa. Both initiatives were endorsed by the South African Department of Science and Innovation.

About NRF and Dorsamy (Gansen) Pillay

As an entity of the Department of Science and Technology (DST), the NRF promotes and supports research through funding, human resource development and the provision of National Research Facilities in all fields of natural and social sciences, humanities and technology. Dr Dorsamy (Gansen) Pillay is currently the Deputy Chief Executive Officer (DCEO): Research and Innovation Support and Advancement (RISA) of the National Research Foundation (NRF) of South Africa. His thirty-year career in research, teaching, management and leadership includes academic, management and leadership positions at the former University of Durban-Westville and at the Durban University of Technology. His research has focused on both prokaryotic and eukaryotic microorganisms, from human diseases to bacterial plant diseases with particular emphasis on elucidating the molecular architecture of the causal microorganisms with a view to understanding genetic diversity, extra-chromosomal elements and developing rapid disease diagnoses. He is currently Vice Chair of the IIASA Council.

Notes:
Please click on the link to read the statement on ethical research and scholarly publishing.
More information on IIASA and South Africa.
This article gives the views of the authors, and not the position of the Nexus blog, nor of the International Institute for Applied Systems Analysis.

5 years of Vietnam membership at IIASA

Tran Thi Vo-Quyen, IIASA guest research scholar from the Vietnam Academy of Science and Technology (VAST), talks to Professor Dr. Ninh Khac Ban, Director General of the International Cooperation Department at VAST and IIASA council member for Vietnam, about achievements and challenges that Vietnam has faced in the last 5 years, and how IIASA research will help Vietnam and VAST in the future.

Professor Dr. Ninh Khac Ban, Director General of the International Cooperation Department at VAST and IIASA council member for Vietnam

What have been the highlights of Vietnam-IIASA membership until now?

In 2017, IIASA and VAST researchers started working on a joint project to support air pollution management in the Hanoi region which ultimately led to the successful development of the IIASA Greenhouse Gas – Air  Pollution Interactions and Synergies (GAINS) model for the Hanoi region. The success of the project will contribute to a system for forecasting the changing trend of air pollution and will help local policy makers develop cost effective policy and management plans for improving air quality, in particular, in Hanoi and more widely in Vietnam.

IIASA capacity building programs have also been successful for Vietnam, with a participant of the 2017 Young Scientists Summer Program (YSSP) becoming a key coordinator of the GAINS project. VAST has also benefited from two members of its International Cooperation Department visiting the IIASA External Relations Department for a period of 3 months in 2018 and 2019, to learn about how IIASA deals with its National Member Organizations (NMOs) and to assist IIASA in developing its activities with Vietnam.

What do you think will be the key scientific challenges to face Vietnam in the next few years? And how do you envision IIASA helping Vietnam to tackle these? 

In the global context Vietnam is facing many challenges relating to climate change, energy issues and environmental pollution, which will continue in the coming years. IIASA can help key members of Vietnam’s scientific community to build specific scenarios, access in-depth knowledge and obtain global data that will help them advise Vietnamese government officials on how best they can overcome the negative impact of these issues.

As Director General of the International Cooperation Department, can you explain your role in VAST and as representative to IIASA in a little more detail?

In leading the International Cooperation Department at VAST, I coordinate all collaborative science and technology activities between VAST and more than 50 international partner institutions that collaborate with VAST.

As the IIASA council representative for Vietnam, I participate in the biannual meeting for the IIASA council, I was also a member of the recent task force developed to implement the recommendations of a recent independent review of the institute. I was involved in consulting on the future strategies, organizational structure, NMO value proposition and need to improve the management system of IIASA.

In Vietnam, I advised on the establishment of a Vietnam network for joining IIASA and I implement IIASA-Vietnam activities, coordinating with other IIASA NMOs to ensure Vietnam is well represented in their countries.

You mentioned the development of the Vietnam-IIASA GAINS Model. Can you explain why this was so important to Vietnam and how it is helping to improve air quality and shape Vietnamese policy around air pollution? 

Air pollution levels in Vietnam in the last years has had an adverse effect on public health and has caused significant environmental degradation, including greenhouse gas (GHG) emissions, undermining the potential for sustainable socioeconomic development of the country and impacting the poor. It was important for Vietnam to use IIASA researchers’ expertise and models to help them improve the current situation, and to help Vietnam in developing the scientific infrastructure for a long-lasting science-policy interface for air quality management.

The project is helping Vietnamese researchers in a number of ways, including helping us to develop a multi-disciplinary research community in Vietnam on integrated air quality management, and in providing local decision makers with the capacity to develop cost-effective management plans for the Hanoi metropolitan area and surrounding regions and, in the longer-term, the whole of Vietnam.

About VAST and Ninh Khac Ban

VAST was established in 1975 by the Vietnamese government to carry out basic research in natural sciences and to provide objective grounds for science and technology management, for shaping policies, strategies and plans for socio-economic development in Vietnam. Ninh Khac Ban obtained his PhD in Biology from VAST’s Institute of Ecology and Biological Resources in 2001. He has managed several large research projects as a principal advisor, including several multinational joint research projects. His successful academic career has led to the publication of more than 34 international articles with a high ranking, and more than 60 national articles, and 2 registered patents. He has supervised 5 master’s and 9 PhD level students successfully to graduation and has contributed to pedagogical texts for postgraduate training in his field of expertise. 

Notes:
More information on IIASA and Vietnam collaborations. This article gives the views of the authors, and not the position of the Nexus blog, nor of the International Institute for Applied Systems Analysis.