Feeds

Leap second bug cripples Linux servers at airlines, Reddit, LinkedIn

Not a good time to be Australian

Intelligent flash storage arrays

The leap second inserted at the weekend crippled Linux-powered servers running one of the world’s largest airline reservation systems - delaying and cancelling flights.

Machines running the mighty Amadeus Altea system were brought down soon after an extra second was added to Coordinated Universal Time (UTC) at midnight on Saturday, 30 June. The bonus second was inserted at the direction of time boffins to keep UTC synchronised with Earth's slowing rotation.

The Altea system was taken offline for an hour, and staff at Qantas and Virgin Australia had to check in passengers manually, disrupting flight plans.

A spokesperson for Amadeus confirmed to The Reg today that the outage had been caused by a bug in the kernel of the open-source Linux operating system, and the flaw was triggered by the leap-second change on Saturday night. He said the problem has been sidestepped using a workaround within an hour, but Amadeus is investigating how to avoid and detect similar bugs in advance.

Servers run by Mozilla, StumbleUpon, Yelp, FourSquare, Reddit and LinkedIn were also reported to have been hit by the same bug. Mozilla said its implementation of the Java-based Hadoop data processing framework and ElasticSearch weren’t working properly on Saturday evening.

Mozilla’s Eric Ziegenhorn posted at 0517 PT, minutes after the leap second was added:

Servers running Java apps such as Hadoop and ElasticSearch and Java doesn't appear to be working. We believe this is related to the leap second happening tonight because it happened at midnight GMT.

Of all these, however, it’s the Altea outage that was by far the more troubling: Amadeus provides the backend booking and reservation system for a growing number of the world’s airlines. Amadeus claims it is the world’s second largest processor of online bookings while it is reported to handle 25 per cent of the world’s 84,000 daily flights.

Amadeus claims 135 airlines have implemented its Altea reservation system, more than 100 have purchased Altea inventory and 60 use Altea departure control. The Altea system was rewritten from a mainframe app in 2004 and moved to Unix-like systems in a response by Amadeus to keep up with changing demands.

Rolled out in 2005, Altea is a set of software modules for booking and reservations that run on Linux and Unix servers, using Java Enterprise Edition, Spring and Apache. Amadeus built the system to move off ageing big iron.

The leap second was added to compensate for the Earth’s uneven rotation by the International Earth Rotation and Reference and System Service. UTC is the time standard for all clocks, devices and applications, as well as POSIX-compliant operating systems, and a second is periodically inserted. There’s traditionally been three ways of implementing the change, which are described here.

Linux distro biz Red Hat published a patch for its Enterprise Linux here - patches for other flavours of the OS are circulating. It is believed the leap second causes the Linux kernel to livelock when the system attempts to adjust the time and date of the computer, causing processors to spin on their wheels doing nothing and hampering services as a result.

Google has modified its internal NTP time servers to gradually add a couple of milliseconds to its regular clock adjustments during a window before the actual leap second is required. ®

Top 5 reasons to deploy VMware with Tegile

More from The Register

next story
Netscape Navigator - the browser that started it all - turns 20
It was 20 years ago today, Marc Andreeesen taught the band to play
Sway: Microsoft's new Office app doesn't have an Undo function
Content aggregation, meet the workplace ... oh
Sign off my IT project or I’ll PHONE your MUM
Honestly, it’s a piece of piss
Return of the Jedi – Apache reclaims web server crown
.london, .hamburg and .公司 - that's .com in Chinese - storm the web server charts
NetWare sales revive in China thanks to that man Snowden
If it ain't Microsoft, it's in fashion behind the Great Firewall
Chrome 38's new HTML tag support makes fatties FIT and SKINNIER
First browser to protect networks' bandwith using official spec
Admins! Never mind POODLE, there're NEW OpenSSL bugs to splat
Four new patches for open-source crypto libraries
prev story

Whitepapers

Forging a new future with identity relationship management
Learn about ForgeRock's next generation IRM platform and how it is designed to empower CEOS's and enterprises to engage with consumers.
Cloud and hybrid-cloud data protection for VMware
Learn how quick and easy it is to configure backups and perform restores for VMware environments.
Three 1TB solid state scorchers up for grabs
Big SSDs can be expensive but think big and think free because you could be the lucky winner of one of three 1TB Samsung SSD 840 EVO drives that we’re giving away worth over £300 apiece.
Reg Reader Research: SaaS based Email and Office Productivity Tools
Read this Reg reader report which provides advice and guidance for SMBs towards the use of SaaS based email and Office productivity tools.
Security for virtualized datacentres
Legacy security solutions are inefficient due to the architectural differences between physical and virtual environments.