ANALYSIS
Cameron Evers
Ikon Images / Alamy Stock Photo
July 14, 2016
If the CIA had a crystal ball, then they would probably not be routinely blindsided by world events. Lacking such a device, the agency has endured notable analytical failures. During the early 1990s, sudden collapses of Somalia, Zaire, Rwanda, and the Soviet Union seemingly appeared without warning.
Strategic surprises have always been a problem for intelligence agencies. The material impossibility of having eyes everywhere requires making judgments without seeing a complete picture, let alone the future. Assessing the likeliness of future rare political events has had dubious reliability.
Thus, in 1994, the CIA's Directorate of Intelligence commissioned the Political Instability Task Force (PITF), formerly known as the State Failure Task Force, a clairvoyant-esque squad of social-scientist brainiacs charged with churning global political data into global instability forecasts.
The creation of the PITF began at end of the Cold War. The PITF's mission is straightforward — make intelligence analysis as holistic as possible, and locate where the next crisis might be, and why.
"The collapse of the Soviet Union completely caught the government off guard. Their models didn't capture that at all. [Their models] didn't even accept it," Monty Marshall, a senior consultant for the PITF and director of the Center for Systemic Peace told War Is Boring.
"The intelligence community was looking for alternative explanations," he added. "The old way of thinking, wasn't catching the new dynamics, trends, that don't fit into the way they understand things."
To meet this task, the team recruited from American academia and included leading political scientists, sociologists and methodologists. In the beginning, they focused on variables as broad as environmental degradation and social conflict. The focus later shifted to cover four main topics — revolutionary and ethnic civil war onset, adverse regime change, state collapse, and genocide.
PITF calculates each event's chance of occurring with probabilistic forecasts from six months to two years out, in 167 countries, which the team monitors on a daily basis. Within every country, the PITF's global model accounts for baseline political dynamics, and disruptions in patterns within these dynamics.
The results of the forecasts hold impressive heuristic accuracy. "[With] what this approach can do — probabilistic models — they're stuck at about 80 percent accuracy. That's good. That's why we're still around," Marshall said.
In addition to accurate forecasting, the PITF's reports inform the intelligence community and U.S. policymakers. According to Marshall, the PITF's reports are used mainly for the National Intelligence Council's annual intelligence estimates.
Interestingly, the relationship between military coups and civil wars are closer than previously thought. According to the PITF's data, government officials will often resort to regime change as a tactic to prevent civil war from occurring.
Thai troops on the streets during the 2010 political crisis | (Null0/Flickr photo/Courtesy of War is Boring)
"A relatively strong government will try [a military coup] to avoid a conflict dynamic that would otherwise lead to civil war," Marshall said. "Sometimes they are successful at averting civil war and sometimes they are not."
"We discovered that the lead indicator was an obscure variable in the data, which we call factionalism," he added. "That is the most powerful driver in the global model, and the most powerful driver at predicting regime change."
Considering this finding regarding "factionalism" — or oppositional groups that are close to a nation's leader — could affect analysis of autocratic regimes around the world, including dangerous ones to international security such as Syria, North Korea, and Iran.
ADVERTISEMENT
"Regime change" as a topic of forecasting is not limited to the PITF's current global model. Jay Ulfelder, former director of the PITF, currently runs a semi-yearly “Coup Forecasts” analysis with his own predictive model, quantifying the likeliness of coup attempts around the world.
"Over the past few years, most coup attempts have happened in countries in the top 30 on my risk assessments, and often in ones pretty close to the top of those lists," Ulfeder told War Is Boringvia email. "Burkina Faso was fifth with a predicted probability of about 15 percent, and Burundi was 26th with a predicted probability of about 5 percent."
Burkina Faso and Burundi experienced coup attempts in 2015. If his last forecasts, quantified in 2014, still hold probabilistic weight today, then political analysts should keep an eye on Guinea, Madagascar, Mali, Equatorial-Guinea, and Niger — the most likely places for military coups, according to the data.
Ulfelder's focus on military coups are due to the relative accuracy in predicting them, as opposed to other types of upheaval.
"Topics like social unrest and the onset of insurgencies have turned out to be harder to forecast well," he added. "So progress has been uneven."
Despite the useful applications of the PITF, the relationship between academically inclined forecasters and government consumers has not been without problems.
The academics at PITF seek to improve their social science craft and the expand the furthering of knowledge, while their spy handlers seek quick answers — which the project is not fully suited to provide.
Predicting world politics "is a superficial understanding of it, but that's what the government wants," Marshall said. "They're still working hard, and they can't get it. We were looking at our understanding of how things work, whereas the intelligence community was looking for something to you know, give them the answers."
"A cheat sheet, mainly."
This misunderstanding of what predictive modeling is supposed frustrates Ulfelder as well. Trends point where to look, not what will happen.
"Most of the phenomena studied by social scientists are inherently hard to predict well," Ulfelder said. "Making a probabilistic forecast is more about trying to quantify our uncertainty than it is about presuming we can be 'right' all the time."
Furthermore, the focus of the intelligence community on data-driven forecasts has resulted in more quantitative methodologists — rather than more social scientists — being added to the PITF team. This shift, according to Marshall, has become problematic.
"The analysts can't inform the policy community if they get all of their information from the machine. They have to understand the politics that go into the machine."
Marshall maintains there should be an emphasis on the human elements of forecasting. "There are almost infinite combinations of variables to use in the model. The results are mainly noise, and to identity the signal is the job of the analyst — to separate that from the noise."
In this vein, the CIA pursues other, more mechanized forecast projects seeking to further automate global trend-casting. The Lockheed Martin managed Integrated Crisis Early Warning System (ICEWS) derives its abilities from "Big Data," tapping into the new availability of open source materials provided principally by the Internet.
"A relatively strong government will try [a military coup] to avoid a conflict dynamic that would otherwise lead to civil war," Marshall said. "Sometimes they are successful at averting civil war and sometimes they are not."
"We discovered that the lead indicator was an obscure variable in the data, which we call factionalism," he added. "That is the most powerful driver in the global model, and the most powerful driver at predicting regime change."
Considering this finding regarding "factionalism" — or oppositional groups that are close to a nation's leader — could affect analysis of autocratic regimes around the world, including dangerous ones to international security such as Syria, North Korea, and Iran.
ADVERTISEMENT
"Regime change" as a topic of forecasting is not limited to the PITF's current global model. Jay Ulfelder, former director of the PITF, currently runs a semi-yearly “Coup Forecasts” analysis with his own predictive model, quantifying the likeliness of coup attempts around the world.
"Over the past few years, most coup attempts have happened in countries in the top 30 on my risk assessments, and often in ones pretty close to the top of those lists," Ulfeder told War Is Boringvia email. "Burkina Faso was fifth with a predicted probability of about 15 percent, and Burundi was 26th with a predicted probability of about 5 percent."
Burkina Faso and Burundi experienced coup attempts in 2015. If his last forecasts, quantified in 2014, still hold probabilistic weight today, then political analysts should keep an eye on Guinea, Madagascar, Mali, Equatorial-Guinea, and Niger — the most likely places for military coups, according to the data.
Ulfelder's focus on military coups are due to the relative accuracy in predicting them, as opposed to other types of upheaval.
"Topics like social unrest and the onset of insurgencies have turned out to be harder to forecast well," he added. "So progress has been uneven."
Despite the useful applications of the PITF, the relationship between academically inclined forecasters and government consumers has not been without problems.
The academics at PITF seek to improve their social science craft and the expand the furthering of knowledge, while their spy handlers seek quick answers — which the project is not fully suited to provide.
Predicting world politics "is a superficial understanding of it, but that's what the government wants," Marshall said. "They're still working hard, and they can't get it. We were looking at our understanding of how things work, whereas the intelligence community was looking for something to you know, give them the answers."
"A cheat sheet, mainly."
This misunderstanding of what predictive modeling is supposed frustrates Ulfelder as well. Trends point where to look, not what will happen.
"Most of the phenomena studied by social scientists are inherently hard to predict well," Ulfelder said. "Making a probabilistic forecast is more about trying to quantify our uncertainty than it is about presuming we can be 'right' all the time."
Furthermore, the focus of the intelligence community on data-driven forecasts has resulted in more quantitative methodologists — rather than more social scientists — being added to the PITF team. This shift, according to Marshall, has become problematic.
"The analysts can't inform the policy community if they get all of their information from the machine. They have to understand the politics that go into the machine."
Marshall maintains there should be an emphasis on the human elements of forecasting. "There are almost infinite combinations of variables to use in the model. The results are mainly noise, and to identity the signal is the job of the analyst — to separate that from the noise."
In this vein, the CIA pursues other, more mechanized forecast projects seeking to further automate global trend-casting. The Lockheed Martin managed Integrated Crisis Early Warning System (ICEWS) derives its abilities from "Big Data," tapping into the new availability of open source materials provided principally by the Internet.
A Ghanaian peacekeeper in Liberia | (U.N. photo/Courtesy of War is Boring)
But even Big Data methods are far from infallible.
"It's like watching a river evolve, only with geological time sped up to real time," Ulfelder noted. "So how do you design a process to turn all that information into data, and then learn things from those data, that isn't already broken as soon as you finish designing it?"
Beyond searching for work-arounds of the social science aspects of PITF, the CIA is interested in removing humans from the equation altogether. "They are furiously working away to find a mechanical way to replace me," Marshall said. "I can't blame them."
"What makes the intelligence handlers most uncomfortable is that I supply almost all the data that runs their models."
While deliberation over what directions the PITF should take will continue, both political scientists are confident in the nature of their work, and the need for more like it. The scholars even have philanthropic forecasting projects that use predictive modeling for humanitarian work.
Ulfelder is a consultant at the Center for the Prevention of Genocide, where he designs, builds, and advises on the operation and development of a public early warning system to prevent mass atrocities around the world.
But even Big Data methods are far from infallible.
"It's like watching a river evolve, only with geological time sped up to real time," Ulfelder noted. "So how do you design a process to turn all that information into data, and then learn things from those data, that isn't already broken as soon as you finish designing it?"
Beyond searching for work-arounds of the social science aspects of PITF, the CIA is interested in removing humans from the equation altogether. "They are furiously working away to find a mechanical way to replace me," Marshall said. "I can't blame them."
"What makes the intelligence handlers most uncomfortable is that I supply almost all the data that runs their models."
While deliberation over what directions the PITF should take will continue, both political scientists are confident in the nature of their work, and the need for more like it. The scholars even have philanthropic forecasting projects that use predictive modeling for humanitarian work.
Ulfelder is a consultant at the Center for the Prevention of Genocide, where he designs, builds, and advises on the operation and development of a public early warning system to prevent mass atrocities around the world.
In a related function, Marshall heads the non-profit Center for Systemic Peace. The center's Global Reports outline the possibilities of reducing political violence through a systemic understanding of the world. For both this project, and the work of the PITF, Marshall believes in the practical application of both forecasting and understanding global trends.
"The overall important finding is that there is a scientific basis to human behavior and how to view human behavior, make it predictable, manageable even understandable," he said. "We can manage human behavior and we can avoid violence as an outcome."
Indeed, if the PITF's global model continues to forecast with 80 percent accuracy, then preventing negative political outcomes will remain a possibility that many — inside and outside of government — would like to pursue.
No comments:
Post a Comment