Feeds

Only Kinect: Microsoft boffins build Minority Report-style tools

Big, expensive goggles no longer compulsory for AR

The Power of One Brief: Top reasons to choose HP BladeSystem

In the two years since Microsoft launched Kinect, there’s been huge interest among enthusiasts in adapting the hands-free controller to new settings beyond gaming.

Techies and geeks of open- and closed-source shades have hacked the hardware, used the tools or adapted Kinect to steer computerised, wheeled armchairs.

Among those doing the fiddling is Microsoft Research in Cambridge, the Microsoft division whose work on real-time human pose recognition and depth-sensing algorithms (here and here) helped make Kinect a reality in the first place.

These engineers have been using Kinect in different experiments in an attempt to construct Star Trek-style holodecks and Minority Report-style 3D touch-displays.

They have combined feeds from multiple Kinect cameras positioned around a room, stitching together a 3D map of the room and then combining the map with moving pico projectors. The result has been to combine Kinect’s depth sensing with the pico data so you can see and touch the computer-generated pixels without the need for special glasses, headsets or gloves to operate. Among the achievements so far is a pixel-generated car driving around the floor.

The Kinect sensor has also been combined with mirror and web-camera tracking to produce a holodesk and Kinect sensors used as hand-held units to scan objects in 3D.

Microsoft Research 3D car

Augmented reality car using Kinect, from Microsoft Research

Shahram Izadi, who jointly leads Microsoft Research Cambridge’s interactive 3D technologies group, reckons augmented reality – or what he also calls “looking at pixels in the air without a screen” - will be the “next big thing” thanks to Kinect and other industry efforts. On Izadi’s team is Andrew Fitzgibbon who helped author that original paper on human skeletal 3D recognition for the Kinect.

“We are starting to move towards this notion of augment reality. It’s about augmentation with the instrumentation of the environment, instead of augmentation of the user,” he said on a recent visit to Microsoft in Cambridge.

“We have been working a lot with the Kinect and know the potential,” Izadi told The Reg.

Izadi points to a number of innovations in a relatively short period of time, including Kinect. We've also had multi-touch motion gaming and augmented reality in smartphones from Apple and Google.

“Companies like Microsoft will be looking to exploit it,” Izadi predicted.

Izadi reckons on reason augmented reality is about to step from science fiction to fact is the relatively ubiquity of the Kinect. The Kinect sold so many units during its first 60 says (eight million) that it set a Guinness World record. The Kinect's companion Xbox 360 game unit – it has since been made available for the PC, too – is also now the biggest selling games console in the US.

Another factor on the Kinect’s side is its price point: £129 from the Microsoft online store in the UK. This makes getting your hands on the Kinect doable - which is not the case with most augmented-realty kit. “Kinect is just a starting point,” Izadi says.

“The reason I’m so excited is two years ago there was nothing like this on the market and it cost tends of thousands to buy [the equipment]. Now we are talking about natural language user interface and augmenter reality as if they are almost upon us, and two years ago we were far away form that vision.”

Izadi’s team has been capitalising on Kinect’s depth-sensing technology to meet the vision. The depth-sensing camera can tell the system how far an object is from the sensor. The Kinect is a rich source of 3D data and it is mostly this information that is being interpreted. An image is smaller than one you’ll find in most digital cameras, 640 x 480 pixels, but the Kinect manages to track 300,000 points at a rate of 30 frames per second.

But there are niggles, hence the work’s still at the prototype stage.

One problem is that the various Kinects interfere with each other when set up and running at the same time, a fact that creates visual noise on the display. This is something Microsoft Research realised could be solved for now with the addition of a little vibration to each unit; the vibes are supplied by a small motor fixed to the top of each Kinect in the holodeck idea (here and here).

Also, sensor readings can be incorrect, with 3m being read as 2.9 meters for example, or they can fluctuate. In other scenarios depth data can’t be scanned, for example in the case of the Dell logo on a Dell PC box.

Izadi also reckons the next challenge is not input - although issues still remain over the input received via the Kinect – but rather the lag in output technologies, for the moment filled by pico projectors.

When will holodecks and holodesks become a reality?

Despite his enthusiasm, Izadi points to the fact the work at Microsoft Research tends doesn’t pay off immediately – it’s generally a good decade before something gets turned into a product. Kinect was quicker, of course. It was first discussed in 2007 and officially unveiled by Microsoft at E3 in June 2009 under the codename Project Natal.

In the meantime, the pace is being pushed on tablets and smartphones thanks to companies like Layar and Qualcomm, which lets you point your phone at a real-world object and receive information, with the phone acting as a head-up-display.

“Many companies are now developing augmented reality or immersive technologies,” Izadi says, "and that will accelerate a lot of what we are seeing. But to add the disclaimer: we are a search group and the work we are doing is five to 10 years out on what could feasibly be a product... [however] there is this real acceleration effect in we are seeing." ®

Securing Web Applications Made Simple and Scalable

More from The Register

next story
Apple fanbois SCREAM as update BRICKS their Macbook Airs
Ragegasm spills over as firmware upgrade kills machines
HIDDEN packet sniffer spy tech in MILLIONS of iPhones, iPads – expert
Don't panic though – Apple's backdoor is not wide open to all, guru tells us
Mozilla fixes CRITICAL security holes in Firefox, urges v31 upgrade
Misc memory hazards 'could be exploited' - and guess what, one's a Javascript vuln
NO MORE ALL CAPS and other pleasures of Visual Studio 14
Unpicking a packed preview that breaks down ASP.NET
Captain Kirk sets phaser to SLAUGHTER after trying new Facebook app
William Shatner less-than-impressed by Zuck's celebrity-only app
Cheer up, Nokia fans. It can start making mobes again in 18 months
The real winner of the Nokia sale is *drumroll* ... Nokia
EU dons gloves, pokes Google's deals with Android mobe makers
El Reg cops a squint at investigatory letters
Chrome browser has been DRAINING PC batteries for YEARS
Google is only now fixing ancient, energy-sapping bug
prev story

Whitepapers

Designing a Defense for Mobile Applications
Learn about the various considerations for defending mobile applications - from the application architecture itself to the myriad testing technologies.
How modern custom applications can spur business growth
Learn how to create, deploy and manage custom applications without consuming or expanding the need for scarce, expensive IT resources.
Reducing security risks from open source software
Follow a few strategies and your organization can gain the full benefits of open source and the cloud without compromising the security of your applications.
Boost IT visibility and business value
How building a great service catalog relieves pressure points and demonstrates the value of IT service management.
Consolidation: the foundation for IT and business transformation
In this whitepaper learn how effective consolidation of IT and business resources can enable multiple, meaningful business benefits.