Feeds

Stamen punts new approach to data aggregation

Let the data choose the questions

Beginner's guide to SSL certificates

Etech: "Give me everything," says Tom Carden, a programmer and designer at Stamen Design. Stamen takes huge amounts of data and turns them into images you can interact with. Let the data choose the questions, he said, rather than the other way around.

Carden's work is part of a trend at this year's Etech emerging technology conference in San Diego, toward aggregating huge amounts of data into something meaningful.

Since Google has the most data flowing through its coffers, it's logical that Google has two aggregating experiments.

The first is prediction markets. Bo Cowgill of Google's economics group has about 1,500 Googlers signed up to bet on things like the dates of product launches. The biggest factor in getting people to bet in synch with each other is that they sit close to each other. The key finding: people tend, optimistically, to overprice things that are good for Google.

Google research director Peter Norvig has been experimenting with improving translation and spell-checking engines by throwing the web at them. Give a word processor a dictionary and all it knows is what's in that dictionary. Tell it to look around the web for samples, and colleague Merhan Sahami stops being "corrected" to "Tehran Salami".

Doing spelling is a little trickier, because so many people on the web can't spell consistently. For that, Norvig invokes probability theory. Using Bayesian rules for language applications was first proposed in 1990 at IBM, but "they didn't have the data and the computing power".

But Norvig's chief interest is practical. A theory-based approach might tell you to look for your keys under the nearest lamp post because that's where the light is brightest. Like Carden, he wants to let what works define the theory.

Carden's samples included a MySociety project that remaps the London tube according to travel time. A more complex version plots travel time to the BBC's Television Centre and house prices. Slide the bar at the top to choose your house price and travel time and see where to live. (Imagine this technique applied to tune the application of probabilities to Norvig's language efforts.)

Carden wants it all: the location of every taxi in San Francisco, all the crimes in Oakland, all the Diggs in a day. The point he says, is exploration.

"Good visualization asks questions. How many things, how many categories. As soon as you make a visualization to answer those questions, you create new ones to ask and explore. To say visualization has to answer questions would be to miss the trick of looking at the data in the first place. If you think you already know the answer you can make visualization that gives you the answer."

If, instead, you explore the pattern and see what questions it generates, you're doing something more like science.

"People tend to come to us with vast amounts of information already," he said. "Most spreadsheets are brilliant only because everyone sees the whole of the data and people can look for what they want, even if it's not in the chart."

Analytics and business intelligence, by contrast, "often start by hiding too much". ®

Security for virtualized datacentres

More from The Register

next story
'Windows 9' LEAK: Microsoft's playing catchup with Linux
Multiple desktops and live tiles in restored Start button star in new vids
Not appy with your Chromebook? Well now it can run Android apps
Google offers beta of tricky OS-inside-OS tech
New 'Cosmos' browser surfs the net by TXT alone
No data plan? No WiFi? No worries ... except sluggish download speed
iOS 8 release: WebGL now runs everywhere. Hurrah for 3D graphics!
HTML 5's pretty neat ... when your browser supports it
Greater dev access to iOS 8 will put us AT RISK from HACKERS
Knocking holes in Apple's walled garden could backfire, says securo-chap
NHS grows a NoSQL backbone and rips out its Oracle Spine
Open source? In the government? Ha ha! What, wait ...?
Google extends app refund window to two hours
You now have 120 minutes to finish that game instead of 15
Intel: Hey, enterprises, drop everything and DO HADOOP
Big Data analytics projected to run on more servers than any other app
SUSE Linux owner Attachmate gobbled by Micro Focus for $2.3bn
Merger will lead to mainframe and COBOL powerhouse
prev story

Whitepapers

Providing a secure and efficient Helpdesk
A single remote control platform for user support is be key to providing an efficient helpdesk. Retain full control over the way in which screen and keystroke data is transmitted.
WIN a very cool portable ZX Spectrum
Win a one-off portable Spectrum built by legendary hardware hacker Ben Heck
Storage capacity and performance optimization at Mizuno USA
Mizuno USA turn to Tegile storage technology to solve both their SAN and backup issues.
High Performance for All
While HPC is not new, it has traditionally been seen as a specialist area – is it now geared up to meet more mainstream requirements?
Security and trust: The backbone of doing business over the internet
Explores the current state of website security and the contributions Symantec is making to help organizations protect critical data and build trust with customers.