Big Brother and Big Data

Bloomberg Businessweek reports that US government intelligence agencies are putting big money into social media and big data analytics in order to track people’s online behavior.

Not surprisingly, the federal government is interested in low-cost methods for tracking people’s every move on the Internet. The intelligence agencies have jumped in to fund analytics startups. They are also building their own analytics tools. Two years ago the National Security Agency talked publicly about creating a new type of technology that would grab “essentially every kind of data there is.” In July, the Defense Advanced Research Projects Agency, the military’s R&D arm, issued a Social Media in Strategic Communication proposal that called for $42 million to go toward better social networking analysis. The idea is to track the spread of ideas on networks such as Facebook, find people participating in “persuasion campaigns,” and develop countermeasures. “We must eliminate our current reliance on a combination of luck and unsophisticated manual methods,” the agency said.

In the decade since the 9/11 attacks, various attempts to leverage analytics in the service of national security have been reported. There was a US Senate Judiciary committee hearing in 2007 entitled “Balancing Privacy and Security: The Privacy Implications of Government Data Mining Programs”. Some people remain skeptical about the viability of predictive modeling to predict outcomes such as terrorist incidents given the lack of historical data. The committee noted consensus as to the value of methods such as link analysis and ‘predicate-based analytics’.

Read More...

Indus Insights is a specialized consulting firm that assists organization in leveraging analytics to drive business performance. They use state-of-the-art mathematical and statistical techniques to unlock game-changing insights hidden in data; and then translate these insights into actionable strategies.

Predicting Manhole Explosions

Every so often in New York City, a disk of cast iron weighing up to 300 pounds will burst out of the street and fly as high as several stories before clattering back. Ever since Thomas Edison fired up the city’s commercial electric grid in 1882, New Yorkers have had to contend with the random hazards of smoking, flaming and exploding manholes. Many of the blasts result from decrepit wiring, which can lead to sparks. Throw in a bit of gas and a confined space and, like a combustion engine, the blast can move metal. Until recently, there was no way of knowing where or when the next outburst would occur; repairs commenced only after a manhole had growled.

But in 2004 Con Edison began a proactive inspection program, with the goal of finding the places in New York’s snaking network of electrical cable where trouble is most likely to strike. The company called upon a team of Columbia University researchers to help predict the likelihood of manhole explosions.

The team tackled Manhattan first, where beneath the borough’s streets and avenues lies 21,000 miles of cable. The scientists weeded out irrelevant information such as parking information for ConEd vehicles. They then ranked the manholes of Manhattan by vulnerability to serious events and developed an algorithm that directs a computer to identify subterranean trouble spots.

Serious manhole events are rare — only a few hundred occur each year even though there are 51,000-odd manhole and service boxes in Manhattan. “Finding a pattern when something is very rare is very hard,” says computer scientist Gary Weiss of Fordham University in the New York City. “If you only have a few examples, there are so many patterns that can fit those few examples… you can’t really tell the difference between a pattern that is meaningful and one that is coincidental.”

The algorithm’s job was to “learn” from the past records and find meaningful patterns. Then it could predict the likelihood that a particular manhole with particular characteristics would have a future flare-up. The team discovered that manholes with larger cables — and so a larger amount of insulation subject to decay and thus to sparking — turned out to be more vulnerable to serious events.

Con Edison blind-tested the team’s model by withholding information on a recent set of fires and explosions. The top 2% of manholes ranked as vulnerable by the algorithm included 11% of the manholes that had recently had a fire or explosion.

Tweaking and adding more data has improved the model further. Con Edison is now using it to help prioritize inspection and repairs on the grid. The team has just completed rankings for manholes in Brooklyn and the Bronx.

Read More...

Consolidated Edison, Inc. (NYSE: ED) is one of the largest investor-owned energy companies in the United States, with approximately $14B in annual revenues and $33B in assets. The company provides a wide range of energy-related products and services to its customers through its subsidiaries.

Indus Insights is a specialized consulting firm that assists organization in leveraging analytics to drive business performance. They use state-of-the-art mathematical and statistical techniques to unlock game-changing insights hidden in data; and then translate these insights into actionable strategies.

Police Department uses Analytics Technology to combat crime

Canada’s Edmonton Police Services (EPS) is deploying business analytics technology to prevent crime and increase public safety. Crime data analytics is a powerful tool that enables law enforcement officials to sift through historic incident, offense, arrest and call-for-service records to accurately pinpoint crime rates and patterns. The quick access to relevant information allows commanders and frontline officers to be better equipped and intelligence-led when problem solving and dealing with day-to-day responsibilities.

The initiative that began as an effort to provide accountability of public dollars has now amassed enough data to identify crime trends and locations. The agency’s investment recently helped them deal effectively with a potential increase in arson activity. By quickly comparing new information with data from previous years, the police service was able to ascertain that the trend was likely to increase from March to July. Police efforts were ultimately successful in nearly eliminating arsons in that particular area during this period.

In the future, the force hopes to take advantage of mobile devices to directly disseminate relevant information to officers, regardless of location. The end goal is the ability to place resources in advance, to put police into certain areas of the city because they predict crime will take place in that area and be able to mitigate that crime with the police presence, said John Warden, BI project team lead for EPS.

Today, the EPS is better trained, better educated and more diverse than at any other time in history. Our technological capabilities allow officers and support staff to be better equipped and intelligence-led when problem solving and dealing with day-to-day responsibilities.

Read more...

Edmonton Police Services is a leading police organization that is responsible for over one million residents in Alberta. They are well respected and highly regarded across Canada for their members’ professionalism and integrity.

Indus Insights is a specialized consulting firm that assists organization in leveraging analytics to drive business performance. They use state-of-the-art mathematical and statistical techniques to unlock game-changing insights hidden in data; and then translate these insights into actionable strategies.