Feeds

Only Kinect: Microsoft boffins build Minority Report-style tools

Big, expensive goggles no longer compulsory for AR

Internet Security Threat Report 2014

In the two years since Microsoft launched Kinect, there’s been huge interest among enthusiasts in adapting the hands-free controller to new settings beyond gaming.

Techies and geeks of open- and closed-source shades have hacked the hardware, used the tools or adapted Kinect to steer computerised, wheeled armchairs.

Among those doing the fiddling is Microsoft Research in Cambridge, the Microsoft division whose work on real-time human pose recognition and depth-sensing algorithms (here and here) helped make Kinect a reality in the first place.

These engineers have been using Kinect in different experiments in an attempt to construct Star Trek-style holodecks and Minority Report-style 3D touch-displays.

They have combined feeds from multiple Kinect cameras positioned around a room, stitching together a 3D map of the room and then combining the map with moving pico projectors. The result has been to combine Kinect’s depth sensing with the pico data so you can see and touch the computer-generated pixels without the need for special glasses, headsets or gloves to operate. Among the achievements so far is a pixel-generated car driving around the floor.

The Kinect sensor has also been combined with mirror and web-camera tracking to produce a holodesk and Kinect sensors used as hand-held units to scan objects in 3D.

Microsoft Research 3D car

Augmented reality car using Kinect, from Microsoft Research

Shahram Izadi, who jointly leads Microsoft Research Cambridge’s interactive 3D technologies group, reckons augmented reality – or what he also calls “looking at pixels in the air without a screen” - will be the “next big thing” thanks to Kinect and other industry efforts. On Izadi’s team is Andrew Fitzgibbon who helped author that original paper on human skeletal 3D recognition for the Kinect.

“We are starting to move towards this notion of augment reality. It’s about augmentation with the instrumentation of the environment, instead of augmentation of the user,” he said on a recent visit to Microsoft in Cambridge.

“We have been working a lot with the Kinect and know the potential,” Izadi told The Reg.

Izadi points to a number of innovations in a relatively short period of time, including Kinect. We've also had multi-touch motion gaming and augmented reality in smartphones from Apple and Google.

“Companies like Microsoft will be looking to exploit it,” Izadi predicted.

Izadi reckons on reason augmented reality is about to step from science fiction to fact is the relatively ubiquity of the Kinect. The Kinect sold so many units during its first 60 says (eight million) that it set a Guinness World record. The Kinect's companion Xbox 360 game unit – it has since been made available for the PC, too – is also now the biggest selling games console in the US.

Another factor on the Kinect’s side is its price point: £129 from the Microsoft online store in the UK. This makes getting your hands on the Kinect doable - which is not the case with most augmented-realty kit. “Kinect is just a starting point,” Izadi says.

“The reason I’m so excited is two years ago there was nothing like this on the market and it cost tends of thousands to buy [the equipment]. Now we are talking about natural language user interface and augmenter reality as if they are almost upon us, and two years ago we were far away form that vision.”

Izadi’s team has been capitalising on Kinect’s depth-sensing technology to meet the vision. The depth-sensing camera can tell the system how far an object is from the sensor. The Kinect is a rich source of 3D data and it is mostly this information that is being interpreted. An image is smaller than one you’ll find in most digital cameras, 640 x 480 pixels, but the Kinect manages to track 300,000 points at a rate of 30 frames per second.

But there are niggles, hence the work’s still at the prototype stage.

One problem is that the various Kinects interfere with each other when set up and running at the same time, a fact that creates visual noise on the display. This is something Microsoft Research realised could be solved for now with the addition of a little vibration to each unit; the vibes are supplied by a small motor fixed to the top of each Kinect in the holodeck idea (here and here).

Also, sensor readings can be incorrect, with 3m being read as 2.9 meters for example, or they can fluctuate. In other scenarios depth data can’t be scanned, for example in the case of the Dell logo on a Dell PC box.

Izadi also reckons the next challenge is not input - although issues still remain over the input received via the Kinect – but rather the lag in output technologies, for the moment filled by pico projectors.

When will holodecks and holodesks become a reality?

Despite his enthusiasm, Izadi points to the fact the work at Microsoft Research tends doesn’t pay off immediately – it’s generally a good decade before something gets turned into a product. Kinect was quicker, of course. It was first discussed in 2007 and officially unveiled by Microsoft at E3 in June 2009 under the codename Project Natal.

In the meantime, the pace is being pushed on tablets and smartphones thanks to companies like Layar and Qualcomm, which lets you point your phone at a real-world object and receive information, with the phone acting as a head-up-display.

“Many companies are now developing augmented reality or immersive technologies,” Izadi says, "and that will accelerate a lot of what we are seeing. But to add the disclaimer: we are a search group and the work we are doing is five to 10 years out on what could feasibly be a product... [however] there is this real acceleration effect in we are seeing." ®

Security for virtualized datacentres

More from The Register

next story
Microsoft WINDOWS 10: Seven ATE Nine. Or Eight did really
Windows NEIN skipped, tech preview due out on Wednesday
Business is back, baby! Hasta la VISTA, Win 8... Oh, yeah, Windows 9
Forget touchscreen millennials, Microsoft goes for mouse crowd
Apple: SO sorry for the iOS 8.0.1 UPDATE BUNGLE HORROR
Apple kills 'upgrade'. Hey, Microsoft. You sure you want to be like these guys?
ARM gives Internet of Things a piece of its mind – the Cortex-M7
32-bit core packs some DSP for VIP IoT CPU LOL
Microsoft on the Threshold of a new name for Windows next week
Rebranded OS reportedly set to be flung open by Redmond
Lotus Notes inventor Ozzie invents app to talk to people on your phone
Imagine that. Startup floats with voice collab app for Win iPhone
'Google is NOT the gatekeeper to the web, as some claim'
Plus: 'Pretty sure iOS 8.0.2 will just turn the iPhone into a fax machine'
prev story

Whitepapers

Forging a new future with identity relationship management
Learn about ForgeRock's next generation IRM platform and how it is designed to empower CEOS's and enterprises to engage with consumers.
Storage capacity and performance optimization at Mizuno USA
Mizuno USA turn to Tegile storage technology to solve both their SAN and backup issues.
The next step in data security
With recent increased privacy concerns and computers becoming more powerful, the chance of hackers being able to crack smaller-sized RSA keys increases.
Security for virtualized datacentres
Legacy security solutions are inefficient due to the architectural differences between physical and virtual environments.
A strategic approach to identity relationship management
ForgeRock commissioned Forrester to evaluate companies’ IAM practices and requirements when it comes to customer-facing scenarios versus employee-facing ones.