Original URL: https://www.theregister.com/2013/07/15/nokia_lumia_1020_technical_walkthrough/
Pure boffinry: We peek inside Nokia's miracle cameraphone
How did they miniaturise a monster?
Posted in Personal Tech, 15th July 2013 11:56 GMT
Special Report These days, “tech” means anything except technology. The word has become so debased that a group of marketing people talking about their ideas for websites are a “tech scene”. And hosting meetings for marketing people to talk about websites in London in 2013 actually makes you a “tech entrepreneur”. The concerns of real engineering innovators take second place to the needs of the Magic Roundabout.
So it’s almost shocking in 2013 to see real engineers pushing the limits and producing something with the capacity to startle.
Last week we saw one of Europe’s biggest tech companies, Nokia, do just that, with its Lumia 1020 phone. The device breaks several barriers for miniature cameras, and it’s the convergence of several technology projects that only a large company with an expensive, long term R&D engineering commitment could undertake. It’s the miniaturisation that’s the deal here. It’s so extreme, it’s quite hard to comprehend. It’s utterly insane.
You’ll know that without the constraints of cost and space, great things are still possible. For example, the 50 megapixel Hasselblad H3D-50 (pdf) takes superb photos – as you can see in this spread.
But it comes with a base price of £20,000 and it weighs 2.3kg – the larger lenses you’ll need will cost you more and weigh more too. Similarly, if space is not a serious constraint, then every camera would use optical zoom.
Nokia’s engineers operate within very demanding cost and space constraints. Nokia is not making a standalone camera, but a tiny unit that must conform to the price and size constraints of the phone market, a market which will not buy bulky or freakish designs in significant numbers.
The Lumia 1020 combines and shrinks two technologies into a smaller package: slimmer than the 920, the bulge protrudes by only around 3mm
Here’s the camera unit itself: it contains 100 parts, is 10mm tall and is apparently larger than the phone itself when you place them side by side. What’s even more astonishing is that there are no compromises.
You could forgive Nokia for introducing a cut-down on the oversampling technology it already had, or the image stabilisation unit it had already made, just to keep the cost or size down. Many predicted this was what Nokia would actually do: a similar but lower fidelity version of what was on the market. But in fact it’s improved on both main technologies in this tiny package.
How did they do that? The 1020 imaging unit appears to be larger than the phone housing it.
A schematic view of the Lumia 1020 imaging unit, which contains over 100 parts. Ball bearings isolate the lenses giving hardware stabilisation. The Lumia 920 used a spring.
With the help of Juha Alakarhu, lead engineer for the project, and his boss imaging chief Eero Salmelin, I found out how they did that:
Four discrete technology projects converge on this product. Three have appeared before, but never together or in such a slimline package. These are the oversampling sensor, the floating OIS technology and the audio recording system. The new tech comes in the form of a new kind of Xenon flash - prior models which incorporated Xenon flashes used tubular capacitors, while this model uses flat capacitors to shave millimetres off the space required to jam in all the components.
Each technology is the culmination of many years of work and partnerships with specialist suppliers, whose engineers in turn had to do something new and unusual with their expertise. The product constraint was a unit 10mm tall. As a consequence every part of the 1020 is new, and nothing could be reused from earlier products.
Two less-highlighted technologies in the 1020 each illustrate the necessity for long-term R&D and the need for partnerships to produce bespoke hardware. Both are a case of Nokia requiring others to push the limits of what they can do. These are the flash unit and the audio recording technology.
Nokia’s Rich Recording audio system (PDF) has a wide dynamic range, capable of recording whispers alongside gunshots, and no distortion in a range of sound pressure ranging from 33 dB (Sound Pressure Level) to 140dB (SPL). The sound of a watch ticking is 30 dB SPL and the human pain threshold around 120 dB SPL.
To miniaturise this required the design and production of custom High Amplitude Audio Capture microphones. Xenon flash, which gives a burst around 100 times as bright as that of an LED, is very rarely seen on smartphones for cost and size reasons.
Nokia insisted on having it, and this in turn required miniaturising conventional tubular Xenon capacitors - inevitably replaced with flat capacitors.
Six lenses, one huge sensor
A model of the six lenses inside the Lumia 1020 imaging unit
The two main imaging technology projects are a miniature version of a floating image unit seen on dedicated equipment, and a huge sensor. Last year’s Lumia’s 920 introduced an optical image stabilisation OIS system, with the unit floating on gyroscopes and sensors compensating for hand movement.
The point of this was to give smoother video recording and better low light photography, since the shutter on a steadier, isolated unit can be held open for longer, absorbing more information. The 920’s OIS was much smaller than those found on dedicated cameras and this one is even smaller. So the second-generation OIS uses ball bearings rather than gyros. The device achieves another demanding requirement – it leaves the shutter open longer than the 920.
Most of the 100+ components in the Lumia 1020 camera unit perform an OIS function. The Nokia engineers said OIS was the hardest to perfect, since the lens has to move in the X-Y dimensions on the ball bearing mounts. At this scale, it also requires new and complex manufacturing techniques – which is why Nokia’s R&D will maintain a lead over rivals for some considerable time.
The huge sensor was a long-term project, begun in 2007, to improve mobile photography. The space constraints preclude physical optical lenses, so capturing accurate detail and smooth zooming must be performed by miniature hardware and offloaded onto powerful CPUs and GPUs in modern smartphones. Which is easier said than done.
Most megapixel specifications on mobile phones are a lie: the final photograph has far fewer bytes than the sensor recorded. That’s because the imaging units take one colour per pixel and then estimate the missing colours using Bayer interpolation, a technique named after the Eastman Kodak scientist who devised and patented it, Bryce Beyer. So an 8MP camera doesn’t produce an 8MP image – each pixel is a single colour and much else is used for interpolation – aka, guessing – then thrown away.
But if you make the sensor larger, then the less guessing is required, and the picture can be more true to life. A large sensor and powerful hardware can also deliver a very smooth and faithful zoom carried out in software, rather than using huge physical lenses.
The “needle in a haystack” zoom is a party trick of the 808, and plays on our experience of zooming in on a digital image only to find blur and artefacts. Zooming in an 808 image reveals only more and more accurate detail.
After five years the huge sensor appeared in last year’s PureView 808, which had a dedicated GPU for performing the intense computation required, while giving lag-free operation, as no standard part permitted the required throughput.
The advances in mobile GPUs are now sufficient for everything to be carried out in software – while the PureView 808 required a custom GPU. Although as I discovered, this required dedicated drivers and Nokia and Qualcomm engineers re-writing the off-the-shelf imaging stack for such hardware.
The result was 5MP of far greater depth, accuracy and fidelity. Compare Nokia’s Lumia 925 – the best modern smartphone camera you can buy today – with the PureView 808.
Example of detail captured by the 808 PureView, right, and the Lumia 925, left
So the phone makes a “master recording” capturing a 35MP or 38MP image, and instantly produces a 5MP version for manipulation and sharing.
The 1020 packs six lenses into the unit, posing another manufacturing challenge: both sides of the tiny lenses have to be aligned very accurately, and they have to be robust in everyday conditions. The manufacturing lines for the 1020 were different to the 808, “and we have the OIS actuators and gyros move in X-Y dimension on the ball bearings,” say Alakarhu and Salmelin.
Asked how it could shrink so much, Nokia listed the following. I presume the order reflects the importance of each change, although Nokia's engineers cautioned that, “like heartburn, there’s not one single element that makes the big difference”.
Nokia’s walkthrough of the new Pro Camera app
Firstly, a backside illuminated sensor was used - a smaller unit than the frontsize illuminated sensor. Secondly much more advanced manufacturing technology was deployed. Thirdly, the engineers decided to spread the components horizontally, rather than stack them vertically. The switch from ball bearings to springs helped, as did the new flat Xenon flash capacitors.
Nokia has done a pretty good job with its Pro Camera of providing a "one click" UI with one that brings out the powerful capabilities of a real camera, with easy access to exposure, light settings
Asked if the rigid hardware control of Windows Phone had been an additional constraint Alakarhu replied: “Windows Phone is not limiting us in any way. It’s the other way around”.
So. Was it worth it?
You can forgive Nokia crowing a little about the imaging technology here, which is the culmination of trying to produce engineering answers to completely unreasonable marketing requirements. Comparing notes after technical sessions, I would say it’s some years since journalists came away from a launch quite so impressed.
There’s a kind of insanity in trying to overcome such engineering challenges, when bending or removing the constraint would have been so much easier. Make the unit bigger. Make it less sensitive. The punter won’t mind.
Yet the result gives Nokia an imaging lead over Samsung and Apple that’s it is likely to maintain for some time. It may lag in market adoption, or fart apps, but Nokia can boast the best photos by far – something that should filter down to less costly devices.
This is the kind of thing for which you need really resourceful and exacting hardware and software engineers – graphic designers don’t cut it – and Nokia's stuff isn’t copied easily. ®