Minority Report Has Arrived

Maryland and Pennsylvania Using Computers To Predict Future Crimes

When police in Minority Report predicted who would commit crimes and stopped them before they did it, it was considered so futuristic, the film was set in 2054.

Now, however, law enforcers in two American states are using crime-prediction software to predict which freed prisoners are most likely to commit murder, and supervising them accordingly.

The Minority Report-style software is already being used in Baltimore and Philadelphia to predict future murderers, and will be extended to Washington D.C. soon.

It has been developed by Professor Richard Berk, a criminologist at the University of Pennsylvania, who believes it will reduce the murder rate and those of other crimes.

Prof. Berk says his algorithm could be used to help set bail amounts and also decide sentences in the future.  It could also be modified to predict lesser crimes.

His software, he said, replaces that 'ad-hoc' decision making, and should identify eight future murderers out of 100.

He said: 'People assume that if someone murdered then they will murder in the future, but what really matters is what that person did as a young individual.

Prof. Berk's researchers used the details of more than 60,000 crimes then wrote an algorithm to find the people behind the crimes who were more likely to commit murder when they were out of prison.

Criteria including criminal record, type of crime, location, and age at which the individual committed the crime were analysed, with type of crime and age proving to be the most reliable predictors of future crime.

He said even his students at the University of Pennsylvania compared his work to Minority Report, the 2002 film starring Tom Cruise in which gifted humans called 'PreCogs' can see into the future and predict who will commit crimes.

Via: "The Daily Mail"

Precrime Creeps Closer To Reality

Predictive Smartphone Location Tracking

A British research group has developed software that can predict, within 20 meters, where you will be 24 hours from now.

It’s actually surprisingly easy to predict your general, routine movements — home, car, office, lunch, car, home — but it has always been nigh impossible to predictbreaks in routine, such as a trip to the cinema or a holiday abroad.

The researchers, Mirco Musolesi, Manlio Domenico, and Antonio Lima of the University of Birmingham, cracked this problem by factoring in the location of your friends and your social interactions with those friends (phone calls, meet-ups, etc.) By simply analyzing how many calls you make to a friend, and by correlating your movement patterns, the researchers can predict your movements over the next 24 hours — even if you deviate dramatically from routine.

The repercussions of such an algorithm are immense, with possible applications that range from awesome to terrifying. On the relatively benign side of things, you can imagine a version of Google Now that knows where you will be tomorrow, and offers up suggestions for which clothes to wear, which other friends will be in the area, and where you should eat (plus a coupon, if the restaurant is one of Google’s partner). This application of the algorithm would be opt-in — if you want to enjoy the services that Google can provide by knowing your predicted location, then that’s your choice.

On the nefarious end of the spectrum, though, this algorithm could be the cornerstone of a Precrime Police Division, a la Minority Report. Precrime would track the location of known criminals via their smartphones, and put a tap on their calls to correlate their movements with their friends/known associates. Very quickly, the Precrime Police could create a map of where every criminal will be in the next 24 hours. It would probably be difficult to predict actual crimes, but at least you’d know where to station your cops.

It’s worth noting that some police departments are already doing something similar, but on a much broader scale: They’re collating all of the reports and arrests in their database, and then plotting them on a map to see where crime is most likely to occur on any given day. In regions where police forces are being downsized, technology will become increasingly important as a force amplifier — and eventually, I wouldn’t be surprised if a real, per-criminal precrime system is deployed.

Research paper: Interdependence and Predictability of Human Mobility and Social Interactions

Via: "Extreme Tech"

The "Stellar Wind" Program

By Laura Poitras

filmmaker Laura Poitras profiles William Binney, a 32-year veteran of the National Security Agency who helped design a top-secret program he says is broadly collecting Americans’ personal data.

It took me a few days to work up the nerve to phone William Binney. As someone already a “target” of the United States government, I found it difficult not to worry about the chain of unintended consequences I might unleash by calling Mr. Binney, a 32-year veteran of the National Security Agency turned whistle-blower. He picked up. I nervously explained I was a documentary filmmaker and wanted to speak to him. To my surprise he replied: “I’m tired of my government harassing me and violating the Constitution. Yes, I’ll talk to you.”

Two weeks later, driving past the headquarters of the N.S.A. in Maryland, outside Washington, Mr. Binney described details about Stellar Wind, the N.S.A.’s top-secret domestic spying program begun after 9/11, which was so controversial that it nearly caused top Justice Department officials to resign in protest, in 2004.

“The decision must have been made in September 2001,” Mr. Binney told me and the cinematographer Kirsten Johnson. “That’s when the equipment started coming in.” In this Op-Doc, Mr. Binney explains how the program he created for foreign intelligence gathering was turned inward on this country. He resigned over this in 2001 and began speaking out publicly in the last year. He is among a group of N.S.A. whistle-blowers, including Thomas A. Drake, who have each risked everything — their freedom, livelihoods and personal relationships — to warn Americans about the dangers of N.S.A. domestic spying.

To those who understand state surveillance as an abstraction, I will try to describe a little about how it has affected me. The United States apparently placed me on a “watch-list” in 2006 after I completed a film about the Iraq war. I have been detained at the border more than 40 times. Once, in 2011, when I was stopped at John F. Kennedy International Airport in New York and asserted my First Amendment right not to answer questions about my work, the border agent replied, “If you don’t answer our questions, we’ll find our answers on your electronics.”’ As a filmmaker and journalist entrusted to protect the people who share information with me, it is becoming increasingly difficult for me to work in the United States. Although I take every effort to secure my material, I know the N.S.A. has technical abilities that are nearly impossible to defend against if you are targeted.

The 2008 amendments to the Foreign Intelligence Surveillance Act, which oversees the N.S.A. activities, are up for renewal in December. Two members of the Senate Select Committee on Intelligence, Senators Ron Wyden of Oregon and Mark Udall of Colorado, both Democrats, are trying to revise the amendments to insure greater privacy protections. They have been warning about “secret interpretations” of laws and backdoor “loopholes” that allow the government to collect our private communications. Thirteen senators have signed a letter expressing concern about a “loophole” in the law that permits the collection of United States data. The A.C.L.U. and other groups have also challenged the constitutionality of the law, and the Supreme Court will hear arguments in that case on Oct. 29.

Laura Poitras is a documentary filmmaker who has been nominated for an Academy Award and whose work was exhibited in the 2012 Whitney Biennial. She is working on a trilogy of films about post-9/11 America. This Op-Doc is adapted from a work in progress to be released in 2013.

This video is part of a series by independent filmmakers who have received grants from the BRITDOC Foundation and the Sundance Institute. 

NYPD To Launch All-Seeing "Domain Awareness System"

The New York Police Department is embracing online surveillance in a wide-eyed way. Representatives from Microsoft and the NYPD announced the launch of their new Domain Awareness System (DAS) at a Lower Manhattan press conference today. Using DAS, police are able to monitor thousands of CCTV cameras around the five boroughs, scan license plates, find out the kind of radiation cars are emitting, and extrapolate info on criminal and terrorism suspects from dozens of criminal databases ... all in near-real time.

New York Police Commissioner Ray Kelly first announced that Microsoft had the NYPD's Domain Awareness System under development at the Aspen Security Forum in July. Microsoft has quietly become one of the world's largest providers of integrated intelligence solutions for police departments and security agencies. Although DAS is officially being touted as an anti-terrorism solution, it will also give the NYPD access to technologies that—depending on the individual's perspectives—veer on science fiction or Big Brother to combat street crime. The City of New York and Microsoft will be licensing DAS out to other cities; according to Mayor Michael Bloomberg, New York City's government will take a 30% cut of any profits. "Citizens do not like higher taxes, so we will (find other revenue outlets)," said Bloomberg. Bloomberg continued that "I hope Microsoft sells a lot of copies of this system, because 30% of the profits will go to us."

According to publicly available documents, the system will collect and archive data from thousands of NYPD- and private-operated CCTV cameras in New York City, integrate license plate readers, and instantly compare data from multiple non-NYPD intelligence databases. Facial recognition technology is not utilized and only public areas will be monitored, officials say. Monitoring will take place 24 hours a day, seven days a week at a specialized location in Lower Manhattan. Video will be held for 30 days and then deleted unless the NYPD chooses to archive it. Metadata and license plate info collected by DAS will be retained for five years, and unspecified “environmental data” will be stored indefinitely.

Cameras are primarily deployed in the Financial District, Midtown Manhattan, and at strategic transportation points like bridges and tunnels. In addition, radiation detectors capable of identifying radiation contamination from chemotherapy, x-rays, medication, industrial uses, and terrorism will also be deployed.

Although NYPD documents indicate that the system is specifically designed for anti-terrorism operations, any incidental data it collects “for a legitimate law enforcement or public safety purpose” by DAS can be utilized by the police department. The NYPD will also share data and video with third parties not limited to law enforcement if either a subpoena or memorandum of understanding exists. The DAS system is headquartered in a lower Manhattan office tower in a command-and-control center staffed around the clock by both New York police and "private stakeholders." When this reporter visited, seats were clearly designated with signs for organizations such as the Federal Reserve, the Bank of New York, Goldman Sachs, Pfizer, and CitiGroup.

The system also allows deep, granular analysis of crime patterns in real time. Information about suspects can also be quickly called up. At a press conference, Microsoft's Jennifer Tisch showed how integrated geographic information systems could display layers of real-time crime analysis for both misdemeanors and felonies. In addition, real-time access to multiple databases belonging to the NYC and other organizations can bring up a massive personal history--including both criminal and public domain information--from any suspect in a matter of seconds.

The NYPD has been at the center of recent controversies involving civil rights and surveillance; in July, a 911 call revealing an NYPD anti-terrorism safe house in New Jersey was released. The safe house in the college town of New Brunswick was monitoring Muslim-American college students; the safe house/apartment's landlord feared the NYPD apartment might have been harboring terrorists.

Via: "Fast Company"

Army Wants To Monitor Your Computer Activity

By Joe Gould - Staff writer
Posted : Saturday May 5, 2012 12:22:14 EDT
Courtesy Of "The Army Times"

In the wake of the biggest dump of classified information in the history of the Army, the brass is searching for ways to watch what every soldier is doing on his or her Army computer.

The Army wants to look at keystrokes, downloads and Web searches on computers that soldiers use.

Maj. Gen. Steven Smith, chief of the Army Cyber Directorate, said the software was one of his chief priorities, joking that it would take the place of a lower-tech solution: “A guy with a large bat behind every user as they go to search the Internet.”

“Now we’ve been in the news — I don’t know if you’ve seen it — with a little insider threat issue,” Smith continued.

Smith did not mention Pfc. Bradley Manning by name. However, the effort comes in the wake of the former intelligence analyst’s alleged leak of hundreds of thousands of pages of classified documents to the anti-secrecy organization WikiLeaks in 2009 and 2010. Manning faces a military trial on 22 counts, including aiding the enemy.

According to Smith, the Army will soon shop for software pre-programmed to detect a user’s abnormal behavior and record it, catching malicious insiders in the act. Though it is unclear how broadly the Army plans to adopt the program, the Army has more than 900,000 users on its computers.

Smith explained how it might work.

“So I’m on the South American desk, doing intelligence work and all of a sudden I start going around to China, let’s say,” Smith said. “That might be an anomaly, it might be justified, but I would sure like to know that and let someone make a decision, almost at the speed of thought.”

The scenario echoes the allegations against Manning: As an intelligence analyst charged with researching the Shiite threat to Iraqi elections, Manning raided classified networks for State Department cables, Afghanistan and Iraq war logs and video from a helicopter attack, according to courtroom testimony.

Software of the type Smith describes is at various stages of development in the public and private sectors. Such software could spy on virtually any activity on a desktop depending on its programming, to detect when a soldier searches outside of his or her job description, downloads massive amounts of data from a shared hard drive or moves the data onto a removable drive.

The program could respond by recording the activity, alerting an administrator, shutting down the user’s access, or by feeding the person “dummy data” to watch what they do next, said Charles Beard, a cybersecurity executive with the defense firm SAIC’s intelligence, surveillance and reconnaissance group.

“It’s a giant game of cat and mouse with some of these actors,” Beard said.

What’s exciting, Smith said, is the possibility of detecting problems as they happen, on what cybersecurity experts call “zero day,” as opposed to after the fact.

“We don’t want to be forensics experts. We want to catch it at the perimeter,” Smith said. “We want to catch this before it has a chance to be exploited.”

A GOVERNMENTWIDE EFFORT

The Army’s efforts dovetail with a broader federal government initiative. President Obama signed an executive order last October that established an Insider Threat Task Force to develop a governmentwide program to deter, detect and mitigate insider threats.

Among other responsibilities, it would create policies for safeguarding classified information and networks, and for auditing and monitoring users.

In January, the White House’s Office of Management and Budget issued a memo directing government agencies that deal with classified information to ensure they adhere to security rules enacted after the WikiLeaks debacle.

Beyond technical solutions, the document asks agencies to create their own “insider threat program” to monitor employees for “behavioral changes” suggesting they might leak sensitive information.

The interagency Insider Threat Task Force is aiming to complete work on the new standards by October. These standards may address training and employee awareness protocols, said John Swift III, senior policy adviser to a task force now working on the draft policy.

Deanna Caputo, lead behavioral psychologist for Mitre Corp., said both technical solutions and monitoring of human behaviors are needed for a successful detection and prevention program.

“To think that we can tackle the problem simply by technical solutions is a mistake,” Caputo said.

A “culture of reporting” is essential, she said. “We need to up the ante and expect a little bit more from our people” to report abnormal behaviors among their co-workers. However, “there is a fine line with that [reporting]. People need to trust they are in a safe environment to do their job.”

Carnegie Mellon’s Software Engineering Institute has compiled 700 insider threat case studies, and come up with two broad profiles of insiders who steal intellectual property in business settings.

One is an “entitled independent” disgruntled with his job who typically exfiltrates his work a month before leaving. The other is an “ambitious leader” who steals information on entire systems and product lines, sometimes to take to a foreign country, such as China.

According to Patrick Reidy, who leads the FBI’s insider threat program, such users may be conducting authorized activities for malicious ends, and their actions would not register on intrusion detection or anti-virus systems.

“People look at computers and networks but not people and data,” he said. “The insider threat is all about people.”

Reidy, Swift and Caputo discussed the effort at a defense industry convention in Washington, D.C., on April 4.

THE ‘PRE-CRIME’ DIVISION

Private industry and the Defense Advanced Research Projects Agency are among the entities that have technological solutions in various stages of progress.

Raytheon’s SureView software captures any security breach or policy violation it’s programmed to find and can “replay the event like a DVR,” for a local administrator or others to view, according to the company’s website. The software’s trigger is programmable and can be set to any behavior considered suspicious or not.

Working with Raytheon, a group of cadets from the U.S. Military Academy at West Point last year conducted a simulation of an insider attack at a forward operating base. Cadets looked at how to fine-tune the way SureView detects potential threats and eliminate false positives for innocuous behavior, said West Point computer science professor Col. Greg Conti.

“It was very powerful, very flexible and allowed you to monitor with very fine resolution activities on the desktop, and the real trick becomes how you detect anomalous behavior,” Conti said. “Predictive models are kind of the holy grail. When you see that no one else has done something but bad guys, you can start being predictive.”

At SAIC, which is testing a behavior analytics system, Beard likened behavioral modeling to the Pre-Crime unit from the science fiction movie “Minority Report.” Instead of using psychics to stop crimes before they occur, the software would be programmed to detect behavior that has preceded malicious acts in the past.

In real life, researchers are examining the behavior of malicious insiders to see what actions they took before they acted out. That in turn would be used to teach the software what behavior to flag.

“We may want to administer policies that say, ‘Gee, gosh, why do you really want to download 300 [megabytes] of stuff or a gig of data in a single session?’ ” Beard said. “We look for the antecedents of behavior that would suggest based on past history that bad things are going to take place.”

That could be visiting restricted websites, requesting access to information outside of one’s job description or asking for large amounts of storage media — or likely some combination of the above. Individually, the actions may not seem problematic, but combined and in the context of human intelligence, they could raise alarms.

“We start taking those things and recombining them to say, ‘What is going on in the environment?’ ” Beard said. “Any one of those things independently can be totally innocuous and innocent, but when you put them together — plus their job, plus their access, plus the things they are working on — you may be looking at it as a counterintel kind of thing.”

DRAWBACKS AND CHALLENGES

Cybersecurity expert Michael Tanji, an Army veteran who has spent nearly 20 years in the U.S. intelligence community, said he sees potential drawbacks and unanswered policy questions. He asked how the Army would implement such technology without unintentionally stifling cross-disciplinary collaboration among soldiers.

Knowing they are being monitored, personnel might avoid enterprising or creative behavior for fear it would be flagged by monitoring software, he said.

Tanji also predicted the technology would come at a considerable financial cost, both to warehouse the data collected by the software and to pay the added staff needed to monitor the reports it generates.

“A brigade-sized element that uses computers on a regular basis would probably need a company-sized element just to keep up with the data that comes in,” he said.

Reidy, the FBI official, said such concerns were valid. Because software may report benign behavior as malicious and vice versa, he cautioned against using technical solutions alone to solve insider threats.

“After a major incident, and no offense to any vendors, but the charlatanism always goes up,” he said. “It’s absolutely amazing how many phone calls I get from people who say they have solved the WikiLeaks problem or solved this or that problem. Everybody’s got to eat, but it’s simply not true.”

Finding bad behavior amid the vast sea of keystrokes, downloads and Web browsing on military computers is no easy task, DARPA acknowledges.

A DARPA solicitation for Suspected Malicious Insider Threat Elimination, or SMITE, announces it is attempting to recognize “moving targets” — telltale patterns of behavior amid “enormous amounts of noise (observational data of no immediate relevance).”

The program, based in behavioral science, would have to distinguish anomalous behavior from normal behavior, and deceptive and malicious behavior from anomalous behavior, the solicitation reads.

A solicitation for another program — Anomaly Detection at Multiple Scales, or ADAMS — uses accused Fort Hood shooter Maj. Nidal Hasan to frame the problem. It asks how to sift for anomalies through millions of data points — the emails and text messages on Fort Hood, for instance — using a unique algorithm, to rank threats and learn based on user feedback.

The program is trying to look beyond computers to spot the point when a good soldier turns, whether that means homicidal or suicidal or ready to dump stolen data.

“When we look through the evidence after the fact, we often find a trail — sometimes even an ‘obvious’ one,” the solicitation states. “The question is, can we pick up the trail before the fact, giving us time to intervene and prevent an incident? Why is that so hard?”

Electronic Police States

Posted by kdawson
On Monday May 11, @11:54PM
Courtesy Of Slash Dot

gerddie writes "Cryptohippie has published what may be called a first attempt to describe the 'electronic police state' (PDF). Based on information available from different organizations such as Electronic Privacy Information Center, Reporters Without Borders, and Freedom House, countries were rated on 17 criteria with regard to how close they are already to an electronic police state. The rankings are for 2008. Not too surprisingly, one finds China, North Korea, Belarus, and Russia at the top of the list. But the next slots are occupied by the UK (England and Wales), the US, Singapore, Israel, France, and Germany." This is a good start, but it would be good to see details of their methodology. They do provide the raw data (in XLS format), but no indication of the weightings they apply to the elements of "electronic police state" behavior they are scoring.

Related Stories

Firehose:The Electronic Police State by gerddie (173963)

The FBI's Department Of PreCrime

By Tom Burghardt
Sunday, May 3, 2009
Courtesy of Anti-Fascist Calling

As they walked along the busy, yellow-lit tiers of offices, Anderton said: "You're acquainted with the theory of precrime, of course. I presume we can take that for granted." -- Philip K. Dick, The Minority Report

From COINTELPRO to the illegal targeting of antiwar activists and Muslim-Americans, the FBI is America's premier political police agency. And now, from the folks who brought us Wi-Fi hacking, viral computer spyware and al-Qaeda triple agent Ali Mohamed comes the Bureau's Department of Precrime!

A chilling new report by the Electronic Frontier Foundation (EFF) reveals the breadth and scope of the FBI's Investigative Data Warehouse (IDW), the Bureau's massive data-mining project.

With more than a billion records "many of which contain information on American citizens," EFF is calling on Congress to demand FBI accountability and strict oversight of this Orwellian project. By all accounts IDW is huge and growing at a geometric pace. According to the Bureau's own narrative,

The IDW received its initial authority to operate in September 2005, and successfully completed a Federal Information Security Management Act audit in May 2007. As of September 2008, the IDW had: 7,223 active user accounts; 3,826 FBI personnel trained on the system, and 997,368,450 unique searchable documents. The IDW transitioned to the operations and maintenance phase during FY 2008. (Federal Bureau of Investigation, "Investigative Data Warehouse," no date)

EFF notes that "the Library on Congress by way of comparison, has about 138 million (138,313,427) items in its collection."

Kurt Opsahl, EFF's Senior Staff Attorney and the author of the new report said: "The IDW includes more than four times as many documents as the Library of Congress, and the FBI has asked for millions of dollars to data-mine this warehouse, using unproven science in an attempt to predict future crimes from past behavior. We need to know all of what's in the IDW, and how our privacy will be protected."

In 2008, the National Academy of Science's National Research Council issued a stinging report that questioned the efficacy of data-mining as an investigative tool for combatting terrorism.

That report, "Protecting Individual Privacy in the Struggle Against Terrorists: A Framework for Assessment," concluded that automated programs such as IDW that collect and mine data should be evaluated for their impact on the privacy rights of citizens. An NRC press release stated candidly:

Far more problematic are automated data-mining techniques that search databases for unusual patterns of activity not already known to be associated with terrorists, the report says. Although these methods have been useful in the private sector for spotting consumer fraud, they are less helpful for counterterrorism precisely because so little is known about what patterns indicate terrorist activity; as a result, they are likely to generate huge numbers of false leads. Such techniques might, however, have some value as secondary components of a counterterrorism system to assist human analysts. Actions such as arrest, search, or denial of rights should never be taken solely on the basis of an automated data-mining result, the report adds.

The committee also examined behavioral surveillance techniques, which try to identify terrorists by observing behavior or measuring physiological states. There is no scientific consensus on whether these techniques are ready for use at all in counterterrorism, the report says; at most they should be used for preliminary screening, to identify those who merit follow-up investigation. Further, they have enormous potential for privacy violations because they will inevitably force targeted individuals to explain and justify their mental and emotional states. (National Academy of Science, National Research Council, "All Counterterrorism Programs That Collect and Mine Data Should Be Evaluated for Effectiveness, Privacy Impacts," Press Release, October 7, 2008)

Noting that the Bureau is withholding critical information from public scrutiny, and that mining data gleaned from dozens of disparate sources is at the heart of IDW, EFF reports that the FBI "has identified only 38 of the 53 'data sources' that feed into the IDW," and has refused to hand over remaining documents, the result of a 2006 Freedom of Information Act request.

In a subsequent court action over the Bureau's document stonewall, the civil liberties' group reported that the Department of Justice told the court that "no additional material will be disclosed," despite Obama administration assertions that it has "new policies on open government."

Indeed, a May 12, 2005 email obtained by EFF from "an unidentified employee in the FBI's Office of the General Counsel to FBI General Counsel Valerie Caproni" notes that the author was "nervous about mentioning PIA [Privacy Impact Assessment] in context of national security systems."

The author admitted that "It is true the FBI currently requires PIAs for NS [national security] systems as well as non-NS systems." EFF reports that the author "thought that the policy might change." Accordingly the anonymous writer "recommend[ed] against raising congressional consciousness levels and expectations re NS PIAs." Caproni's response is short: "ok."

However, "congressional consciousness levels" were raised after an August 30, 2006 Washington Post piece exposed the intrusive nature of the IDW system.

The Bureau's response? Several emails revealed the FBI's desire to play down privacy concerns, noting cynically: "I'm with [Redacted] in view that if everyone [Redacted] starts running around with their hair on fire on this, they will just be pouring gas on something that quite possibly would just fade away if we just shrug it off."

Given the corporate media's snail-like attention span when it comes to anything other than puppies trapped in a well or the shenanigans of various "celebrities," it's a sure-fire bet something as mundane as the rights of ordinary citizens "would just fade away."

IDW: A Web-Based Panopticon and Cash Cow for Corporate Spooks

The Electronic Frontier Foundation's report, citing the Bureau's own description, characterizes the Investigative Data Warehouse as "the FBI's single largest repository of operational and intelligence information."

In 2005, FBI Section Chief Michael Morehart said that "IDW is a centralized, web-enabled, closed system repository for intelligence and investigative data." Unidentified FBI agents have described it as "one-stop shopping" for FBI agents and an "uber-Google." According to the Bureau, "[t]he IDW system provides data storage, database management, search, information presentation, and security services."

Documents released to EFF show that the FBI began spending funds on IDW in 2002 and that "system implementation was completed in FY 2005." Version 1.1 was released in July 2004 "with enhanced functionality, including batch processing capabilities."

But as with all things related to "national security," early-on in the game the FBI forged a "public-private partnership" with spooky corporations in the defense and security industry, including Science Applications International Corporation (SAIC), Convera and Chiliad to develop the project.

As the Project on Government Oversight (POGO) notes in their Federal Contractor Misconduct Database, the San Diego-based SAIC has paid out some $14.5 million in fines on $5.3 billion in revenue largely derived from contracts in the defense, intelligence and security fields.

Misconduct ranged from false claims and defective pricing to conflict of interest violations. Last August, SAIC was forced to drop its bid with the Federal Emergency Management Agency (FEMA) for the agency's TOPOFF 5 national disaster drill "after allegations of improprieties in the contracting process" were uncovered, according to Washington Technology.

Indeed, SAIC had been hired by the FBI to build an early version of IDW known as the Virtual Case File (VCF). According to Washington Technology, SAIC was contracted by the Bureau in 2001 to build VCF "but pulled the plug in 2005 after realizing the system would not work."

The 2007 appropriations bill directed the Bureau to "retrieve as much as $104 million from the defaulted VCF contract" and in unusual language for the Senate, "expects FBI to use all means necessary, including legal action, to recover all erroneous charges from the VCF contractor," Washington Technology revealed.

Federal Computer Week reported in 2005 that Aerospace, an independent contractor hired to evaluate the system concluded that SAIC "did a poor coding job" and that it was "virtually impossible to update the system."

Despite these revelations, the San Diego defense and security giant has cornered billions of dollars in new contracts from the Defense, Homeland Security and Justice Departments.

Convera, describing itself as "the leading technology provider of intelligent search," the Vienna, Virginia corporation claims it is "an established leader in the business of search technologies." Apparently, the company is less than sanguine about trumpeting its products for the FBI. A search of their website returned zero hits on the terms "FBI-IDW."

However, Washington Technology revealed in 2004, Convera won a contract worth more than $2 million to "provide an agency-wide search and discovery platform for the FBI."

The contract "covers a perpetual license for the company's RetrievalWare software as the search technology." The 2004 award was "a follow-on from an earlier contract worth approximately $1.5 million ... for search and categorization software for the FBI's Investigative Data Warehouse," the technology insider publication reported.

On the other hand Chiliad avers that they will help "organizations realize the full business value of all of their disparate information resources," and their innovative products "in enterprise search and analysis technology, and virtual information sharing" will "help organizations 'Connect the Dots' and arrive at truly actionable intelligence." In this spirit, Chiliad boasts that the FBI as the lead agency for "domestic counterterrorism" has purchased a "worldwide enterprise license to Chiliad's software."

Founded in 1999, the Washington, D.C.-based firm's customer base include such spooky corporations as defense giant BAE, Booz Allen Hamilton, described by investigative journalist Tim Shorrock in Spies For Hire as a "revolving door" connecting the corporate security world and agencies such as NSA, General Dynamics, ITT, Northrop Grumman, SAIC and many, many more!

According to EFF, the FBI is busily putting these products to the test.

In addition to storing vast quantities of data, the IDW provides a content management and data mining system that is designed to permit a wide range of FBI personnel (investigative, analytical, administrative, and intelligence) to access and analyze aggregated data from over fifty previously separate datasets included in the warehouse. Moving forward, the FBI intends to increase its use of the IDW for "link analysis" (looking for links between suspects and other people--i.e. the Kevin Bacon game) and to start "pattern analysis" (defining a "predictive pattern of behavior" and searching for that pattern in the IDW's datasets before any criminal offence is committed--i.e. pre-crime). (Kurt Opsahl, "Report on the Investigative Data Warehouse," Electronic Frontier Foundation, April 2009)

Accordingly, EFF revealed that then-Assistant Director for the Counterterrorism Division, Willie Hulon said in 2004 that the FBI was "introducing advanced analytical tools" that would "make the most" of IDW data.

Hulon went on to state that when IDW is completed, "Agents, JTTF [Joint Terrorism Task Force] members and analysts," using the new data-mining technology "will be able to search rapidly for pictures of known terrorists and match or compare the pictures with other individuals in minutes rather than days. They will be able to extract subjects' addresses, phone numbers, and other data in seconds, rather than searching for it manually. They will have the ability to identify relationships across cases. They will be able to search up to 100 million pages of international terrorism-related documents in seconds." EFF notes that since 2004, "the number of records has grown nearly ten-fold."

According to an April 1 press release from the American Civil Liberties Union, FBI Joint Terrorism Task Forces and the related national nexus of Fusion Centers, comprised of the FBI, local police, the military (U.S. Northern Command) and private outfits in the corporate security world, relying heavily on data-mining and link analysis "have experienced a mission creep in the last several years, becoming more of a threat than a security device."

Indeed, the ACLU noted that Fusion Centers have routinely targeted activists across the political spectrum, relying on specious data-mining techologies as well as paid provocateurs and informants (HUMINT) that label any and all government critics as "extremists" to be monitored and indexed in national security databases. The civil liberties' group averred: "From directing local police to investigate non-violent political activists and religious groups in Texas to advocating surveillance of third-party presidential candidate supporters in Missouri, there have been repeated and persistent disclosures of troubling memos and reports from local fusions centers."

Since 2004, EFF has identified 38 separate data sources feeding the FBI's Investigative Data Warehouse. In addition to the FBI's Automated Case System (ACS), soon to be replaced by the Sentinel Case Management System after the $170 million "Virtual Case File" fiasco briefly described above, IDW compiles information from the following sources:

Secure Automated Messaging Network (SAMNet). SAMNet consists of all message traffic sent by the CIA, Defense Intelligence Agency, including Intelligence Information Reports (IIRs) and Technical Disseminations (TD) to the FBI. These include Secret classified information but not those designated Top Secret and above, including Sensitive Compartmented Information (SCI), the highest security classification.

Joint Intelligence Committee Inquiry (JICI) Documents of "all FBI documents related to Islamic extremist networks between 1993 and 2002."

Open Source News, collected from the MiTAP system run by San Diego State University. EFF describes MiTAP as a "system that collects raw data from the internet, standardizes the format, extracts named entities, and routes documents into appropriate newsgroups. This dataset is part of the Defense Advanced Research Projects Agency (DARPA) Translingual Information Detection, Extraction and Summarization (TIDES) Open Source Data project."

Violent Gang and Terrorist Organization File (VGTOF), provided by the FBI National Crime Information Center (NCIC). It includes "biographical data and photos" of individuals "who the FBI believes to be associated with violent gangs and terrorism." However, numerous abuses of the VGTOF classification system have been uncovered by the ACLU. According to the ACLU of Colorado, the FBI's JTTF added anarchists and eight separate categories of "extremists" to the VGTOF, including "environmental extremist" and "Black extremist." Indeed, Colorado antiwar activist Bill Sulzman, a campaigner against the weaponization of space, was listed in the VGTOF as a "terrorist," according to an article in the Colorado Springs Independent.

CIA Intelligence Information Reports (IIR) and Technical Disseminations (TD), "designed to provide the FBI with the specific results of classified intelligence collected on internationally-based terrorist suspects and activities, chiefly abroad."

Eleven IntelPlus scanned document libraries "related to FBI's major terrorism-related cases."

Eleven Financial Crimes Enforcement Network (FinCEN) Databases.

Selectee List: Copies of a Transportation Security Administration (TSA) "list of individuals that the TSA believes warrant additional security attention prior to boarding a commercial airliner."

Terrorist Watch List (TWL): according to EFF, the "FBI Terrorist Watch and Warning Unit (TWWU) list of names, aliases, and biographical information regarding individuals submitted to the Terrorist Screening Center (TSC) for inclusion into VGTOF and TIPOFF watch lists. Also called the Terrorist Screening Database (TSDB), the database 'contained a total of 724,442 records as of April 30, 2007'." The TWL has balooned to 1,192,000 names as of May 3, 2009.

According to the ACLU, "members of Congress, nuns, war heroes and other 'suspicious characters' ... have become trapped in the Kafkaesque clutches of this list, with little hope of escape." Barry Steinhardt, director of the ACLU Technology and Liberty Project said last summer: "Putting a million names on a watch list is a guarantee that the list will do more harm than good by interfering with the travel of innocent people and wasting huge amounts of our limited security resources on bureaucratic wheel-spinning. I doubt this thing would even be effective at catching a real terrorist." While true enough as far as it goes, perhaps the list's true intent is not to prevent terrorism but rather to terrorize the American people.

At the heart of these systems is data mining, that is, the deployment of a vast infrastructure capable of receiving, processing, managing and analyzing data flowing into the system from disparate sources. Indeed, documents released to EFF disclosed that the Bureau's 2008 budget justification explained that "[t]he Investigative Data Warehouse (IDW), combined with FTTTF's [Foreign Terrorist Tracking Task Force] existing applications and business processes, will form the backbone of the NSB's [National Security Branch] data exploitation system." The FBI also requested "$11,969,000 ... for the National Security Branch Analysis Center (NSAC)." The FBI claimed:

Once operational, the NSAC will be tasked to satisfy unmet analytical and technical needs of the NSB, particularly in the areas of bulk data analysis, pattern analysis, and trend analysis. … The NSAC will provide subject-based "link analysis" through the utilization of the FBI's collection datasets, combined with public records on predicated subjects. "Link analysis" uses datasets to find links between subjects, suspects, and addresses or other pieces of relevant information, and other persons, places, and things. This technique is currently being used on a limited basis by the FBI; the NSAC will provide improved processes and greater access to this technique to all NSB components. The NSAC will also pursue "pattern analysis" as part of its service to the NSB. "Pattern analysis" queries take a predictive model or pattern of behavior and search for that pattern in datasets. The FBI's efforts to define predictive models and patterns of behavior will improve efforts to identify "sleeper cells."

When this request was submitted to Congress, NSAC said it would "bring together nearly 1.5 billion records created or collected by the FBI and other government agencies," expected to quadruple by 2012. The House Science and Technology Committee was so alarmed that they demanded that the Government Accountability Office investigate the National Security Branch Analysis Center.

ABC News' Brian Ross reported that lawmakers are "questioning whether a proposed FBI anti-terrorist program is worth the price, both in taxpayer dollars and the possible loss of Americans' privacy."

Noting that the the FBI has a history "of improperly--even illegally--gathering personal information on Americans, most recently through the widespread abuse of so-called National Security Letters," ABC reported that congressional investigators are demanding to know "whether there are protections in place to make sure all the data in the program was legally collected."

Given the track record of the Bureau when it comes to targeting political opponents, I wouldn't hold my breath.

Two years later, EFF notes in a letter to Senator Patrick Leahy (D-VT) that the FBI has refused to release documents filed under the Freedom on Information Act and that the Bureau "has published neither a 'system of records notice' (as required by the Privacy Act) nor a 'privacy impact assessment' (as required by the E-Government Act) for the IDW, thus depriving the public of the kind of accountability that usually comes with the creation and maintenance of large database systems containing sensitive personal information."

Citing Leahy's own assertion that the IDW is a "system ripe for abuse," EFF has called on the Judiciary Committee to examine IDW closely and "provide the public with needed assurances concerning its potential impact on the privacy rights of citizens."

Stay tuned...