Predicting The Future Through Online Data Mining

The information to predict whether tomorrow it will rain is limited by the number of weather sites available, but sites that might refer to upcoming anti-American demonstrations in the Middle East are infinitely more numerous [AFP]

Recorded Future predicts When and Where A Demonstration Will Occur After Mining From The Web All The Related Activities.

By Santiago Zabala,

Martin Heidegger's greatest concerns when it came to technological innovations was the formation of existential conditions where, as he said, the "lack of emergency is the only emergency". In this condition human beings would be completely "uprooted" from the earth, that is, "framed" ("Ge-stell") by a technological power they are no longer able to control.

As it turns out, the software company Recorded Future (which has recently been praised by Wired, the MIT Technology Review and other media outlets, after the CIA and Google invested millions in their services) seems to be offering its clients something similar: a world where emergencies, that is, future events, can be calculated in advance. But how does this start-up actually function, and why are the German philosopher’s concerns relevant to its services? 

Recorded Future is based in Gothenburg and has offices in London, Boston, Arlington and New York. A team of 20 computer scientists, statisticians and experts in linguistics "calculate" the future. While Yahoo, Google and Bing use links to connect and rank different web pages, Recorded Future goes further by scouring (in real time) thousands of available information sources such as blogs, websites and Twitter comments in order to find "invisible links", that is, relationships among actions, people and institutions that refer to related events in the future. 

"Recorded Future is based in Gothenburg and has offices in London, Boston, Arlington and New York. A team of 20 computer scientists, statisticians and experts in linguistics 'calculate' the future."

After mining from the web all the related people ("Bashar al-Assad"), places ("Syria") and activities ("military interventions") that refer to a possible demonstration, Recorded Future uses algorithms to predict when and where a demonstration will occur. 

An example of a predicted demonstration is available in a video on the company website which illustrates how its powerful engines monitor these protests not only in the Middle East, but also in South America and North Africa. The fact that Google and the CIA have already invested millions in this company is an indication that it will be used to conserve certain interest against others as the example above indicates.

The different fee levels for the customers of Recorded Future are probably related to the quality and quantity of information they wish to purchase, making this, and similar companies, at the service of the wealthiest and most powerful. 

From a philosophical point of view, the most interesting feature of all this is not that these demonstrations can be predicted, but rather how technology has finally uprooted and dislodged man from the world, that is, has given human existence to a power beyond human control. The secured, comfortable and calculated environment that allowed the creation of society has now become so functional and rationalised that we cannot help but become victims by existing in it.

This existential dilemma does not arise from the fact that it’s finally possible to organise all the things the web already knows about the future, which could certainly become useful to prevent diseases or famine, but rather that a private company now means to know everything, that is, all human projects. 

We have entered an age where only those framed within the approved interests of Recorded Future clients will be able to live freely, that is, without being predicted. But how free is an existence that is completely revealed to the modern "lack of emergencies"?

As Heidegger explained, emergencies do not arise when something doesn't function correctly, but rather when "everything functions … and propels everything more and more toward further functioning". It's within this logic that as soon as something critical to the interests of those who can afford it fails to function, Recorded Future will alert its customers, who will then take the appropriate measures to conserve the previous condition.

In sum, Heidegger's concerns over a world lacking "emergencies" more than 50 years ago was meant to point out how technologies such as that employed by Recorded Future (and similar companies) aim to avoid the future, that is, to change the world.

Via: "Al-Jazeera"

CIA: Our Mideast Forecasts Sucks

By Spencer Ackerman
February 10, 2011 | 12:00 pm
Courtesy Of "Wired's Danger Room"


Twitter and the mainstream press are filled with rumors that Egyptian dictator Hosni Mubarak may be forced to step down as early as Thursday night. What does CIA Director Leon Panetta think? All he could tell a congressional panel on Thursday morning is that he, like them, is relying on the media for his info. According to those press reports, he said, there’s a “strong likelihood that Mubarak may step down this evening.”

Unsurprisingly, that didn’t satisfy the House intelligence committee’s leaders, who want to know how a spy apparatus that costs $80 billion annually didn’t forecast the Egyptian uprising. As spymasters have saidrepeatedly over the past two weeks, intelligence officials have reported for “decades” that unrest brews in the Mideast. “But we are not clairvoyant,” said James Clapper, the director of national intelligence.

Vague much? Clapper’s thick opening statement contained three noncommittal sentences on what’s happening in Egypt. (“Instability, fueled in large part by economic and political grievances, clearly has reached a critical point in recent weeks and will have a long-lasting impact.”)

Clapper and his intelligence colleagues at this morning’s hearing came prepared to get beaten up over not predicting the depth and the speed of the current Mideast uprisings. The director said he had reviewed “literally thousands” of intelligence reports on popular dissatisfaction across the region, and he thought intel analysts had done “yeoman work.”

But that defense came with concessions: Tunisian President Zine El Abidine Ben Ali made a “snap decision” to abdicate. Message: spies can’t predict any of that.

It was up to Panetta to implicitly concede that he was fixing a problem. The CIA needs “better collection when it comes to the triggers that ignite these conditions.”

He’s put together a 35-member task force within the CIA to look at where the Mideast uprisings might spread next. The metrics to be examined: “the large, unmet expectations of the people in those countries,” the loyalties of the military, the youth bulge, and “the whole role of the internet and the ability to put a demonstration together in quick time.”

Rep. Jan Schakowsky (D-Illinois) asked the obvious follow up question: How stable are the governments of Syria, Saudi Arabia and Jordan? In a public congressional session, Panetta declined to make any claim.

But he said “a number of countries in the Arab world” have the same problems Egypt and Tunisia face: “lack of freedoms, lack of political reform, lack of truly free and open elections, economic stagnation…. All of which means we have got to pay a great deal of attention.”

Photo: AlJazeera English/Flickr

Computer Model Predicts Future Conflicts

Calculated Terror

How a computer model predicts the future in some of the world's most volatile hotspots.

BY AARON MANNES, V.S. SUBRAHMANIAN
DECEMBER 15, 2009
Courtesy Of Foreign Policy Magazine

On Oct. 27, a Katyusha rocket was fired from Lebanon and struck down in an open area outside the northern Israeli town of Kiryat Shmone. This was the ninth such rocket strike since the end of the 2006 war between Israel and Hezbollah. No group claimed responsibility for the attack, but smaller Palestinian groups hoping to spark another round of fighting are the most likely suspect. Hezbollah, despite its extreme anti-Israel politics, did not join the fight, even after Israeli counterstrikes.

The "Blue Line" separating Israel from Lebanon is one of the most volatile borders in the world. But predicting when this area, and other tense regions throughout the world, will erupt into violence often appears to be little more than guesswork. How can policymakers overcome their own biases and limited information to anticipate if an incident like the recent rocket strike on Israel will spark a larger conflict, like the 2006 war, or fizzle out?

COMMENTS (2) SHARE:

Digg

Facebook

Reddit

More...

Increasingly, the answer is: Develop a computer model from historical data. The University of Maryland's Laboratory for Computational Cultural Dynamics (LCCD) constructed one such model that predicted this period of quiet along the Israeli-Lebanese border, and also provides insight into Hezbollah's priorities. LCCD developed a framework, known as Stochastic Opponent Modeling Agents (SOMA), that examines historical data and automatically generates rules assessing the probability that a group will take certain actions under certain conditions.

SOMA examines historical data about the group's behavior and tries to find conditions such that, when the condition is true, the group takes a given action with high probability and, when the condition is false, the group takes the action with very low probability. A human analyst could make these connections when there are relatively few variables being tracked. But when there are dozens of variables there are millions of such possible rules -- far more than an analyst can process. This is often the case in the interconnected world of Middle East politics, where events are shaped by the actions of many actors working in a diverse array of countries.

SOMA rules have also been extracted on the behavior of other Middle Eastern groups. Hamas, for example, is twice as likely to commit kidnappings during periods of conflict with other Palestinian organizations (the probability increases from approximately 33 percent to 67 percent). If another round of Fatah-Hamas fighting erupts in the West Bank, this may present a new challenge for Israeli security. While the rules had not been extracted in 2006, it is worth noting that the Israeli soldier Galid Shalit was kidnapped as the conflict between Hamas and Fatah expanded after the 2006 Palestinian elections.

SOMA is not specifically designed to model the behavior or Hezbollah or even of terrorist organizations -- it has also examined the behaviors of various actors in the Afghan drug trade under different circumstances. This model was built on a hypothetical situation, not systematically gathered data, but demonstrates the way in which SOMA can be applied to a broad range of conflicts and scenarios. The analysis showed that frequently used strategies such as burning poppy fields and destroying drug labs in Afghanistan are unlikely to lead to a long-term decline in the Afghan drug trade.

Models require data, and limitations of that data can limit the accuracy of a system such as SOMA. For the analysis of Hezbollah (and several other groups) SOMA used the Minorities at Risk Organization Behavior (MAROB) data set created at the University of Maryland's Center for International Development and Conflict Management. MAROB identifies factors that motivate members of ethnic minorities to form activist organizations and move from conventional politics to terrorism. MAROB has systematically collected information on more than 150 variables from over 100 organizations across the Middle East during the last several decades. Hezbollah is one of the organizations profiled; the data collected covers Hezbollah from its 1982 founding through 2004.

In examining the rules generated by SOMA about Hezbollah's behavior, the most striking finding was the correlation between Hezbollah attacks on Israeli citizens and Lebanese elections. Since the re-establishment of Lebanon's parliamentary democracy in 1992, there was a 62 percent chance Hezbollah would target Israeli civilians (primarily through rocket attacks) in any given year through 2004. In off-election years the likelihood jumped to 78 percent, while in election years the probability was negligible. The one election year in which Hezbollah fired rockets at Israel was 1996. Though Hezbollah won a propaganda victory when Israel's response caused heavy Lebanese civilian casualties, the organization lost parliament seats in the 1996 elections. Hezbollah has since sought to keep its fighting with Israel within certain boundaries, avoided major escalations during election years, and re-emphasized its provision of social services within Lebanon.

The test for any model is whether or not its predictions hold. During Israel's Operation Cast Lead against Hamas in Gaza, there was concern that Hezbollah would initiate a second front to aid its ally. But Hezbollah offered only rhetorical support to Hamas. During the Gaza operation, a few rockets were fired from Lebanon into Israel and Hezbollah quickly and credibly denied responsibility. With an election later in the year, Hezbollah determined that it could not risk renewed violence with Israel -- particularly in the wake of the 2006 war, which many Lebanese felt was brought on by Hezbollah and that left much of south Lebanon in ruin.

Beyond its predictive value, these findings provide insight into Hezbollah's behavior and priorities. The SOMA results highlight how Hezbollah needs to maintain its position within Lebanon's political system, even if that restricts its ability to wage war on Israel. Though the conflict with Israel is the organization's raison d'être, Hezbollah's leadership has taken steps -- such as participating in the Lebanese parliament -- to prevent the party from being politically isolated.

However, this balancing act is becoming more difficult to maintain, as these two aspects of Hezbollah are coming into increasing conflict. The 2006 war inflicted massive costs on the Lebanese people, which has made them less likely to tolerate Hezbollah's foreign adventurism. Earlier in October an explosion at a private home revealed the presence of a Hezbollah arms cache (Hezbollah disputes this). This incident reminded the Lebanese that Hezbollah remains capable of launching another round of fighting with Israel, and raised the specter of the conflict being sparked by accident. The recent Israeli seizure of a cargo ship carrying nearly 400 tons of weapons apparently intended for Hezbollah raised further concerns that Lebanon could again become a battlefield between foreign powers. If one appreciates that Lebanese popular opinion exerts a strong influence on Hezbollah's actions, it should be clearer that another Israel-Hezbollah war remains unlikely.

SOMA's analysis of Hezbollah's behavior serves both analysts and policymakers by making a specific prediction about the group's likely actions, and also by highlighting this important underlying dynamic. As the data collected expands in breadth and depth, it may become possible to make specific predictions about how, when, and under what circumstances regional changes will occur. While the Oct. 27 rocket strike on northern Israel seems to be just the sort of incident which could cause an unpredictable chain reaction in the region, in the future its repercussions may largely be known before the rocket leaves the ground.