Feeds

Leap second bug cripples Linux servers at airlines, Reddit, LinkedIn

Not a good time to be Australian

High performance access to file storage

The leap second inserted at the weekend crippled Linux-powered servers running one of the world’s largest airline reservation systems - delaying and cancelling flights.

Machines running the mighty Amadeus Altea system were brought down soon after an extra second was added to Coordinated Universal Time (UTC) at midnight on Saturday, 30 June. The bonus second was inserted at the direction of time boffins to keep UTC synchronised with Earth's slowing rotation.

The Altea system was taken offline for an hour, and staff at Qantas and Virgin Australia had to check in passengers manually, disrupting flight plans.

A spokesperson for Amadeus confirmed to The Reg today that the outage had been caused by a bug in the kernel of the open-source Linux operating system, and the flaw was triggered by the leap-second change on Saturday night. He said the problem has been sidestepped using a workaround within an hour, but Amadeus is investigating how to avoid and detect similar bugs in advance.

Servers run by Mozilla, StumbleUpon, Yelp, FourSquare, Reddit and LinkedIn were also reported to have been hit by the same bug. Mozilla said its implementation of the Java-based Hadoop data processing framework and ElasticSearch weren’t working properly on Saturday evening.

Mozilla’s Eric Ziegenhorn posted at 0517 PT, minutes after the leap second was added:

Servers running Java apps such as Hadoop and ElasticSearch and Java doesn't appear to be working. We believe this is related to the leap second happening tonight because it happened at midnight GMT.

Of all these, however, it’s the Altea outage that was by far the more troubling: Amadeus provides the backend booking and reservation system for a growing number of the world’s airlines. Amadeus claims it is the world’s second largest processor of online bookings while it is reported to handle 25 per cent of the world’s 84,000 daily flights.

Amadeus claims 135 airlines have implemented its Altea reservation system, more than 100 have purchased Altea inventory and 60 use Altea departure control. The Altea system was rewritten from a mainframe app in 2004 and moved to Unix-like systems in a response by Amadeus to keep up with changing demands.

Rolled out in 2005, Altea is a set of software modules for booking and reservations that run on Linux and Unix servers, using Java Enterprise Edition, Spring and Apache. Amadeus built the system to move off ageing big iron.

The leap second was added to compensate for the Earth’s uneven rotation by the International Earth Rotation and Reference and System Service. UTC is the time standard for all clocks, devices and applications, as well as POSIX-compliant operating systems, and a second is periodically inserted. There’s traditionally been three ways of implementing the change, which are described here.

Linux distro biz Red Hat published a patch for its Enterprise Linux here - patches for other flavours of the OS are circulating. It is believed the leap second causes the Linux kernel to livelock when the system attempts to adjust the time and date of the computer, causing processors to spin on their wheels doing nothing and hampering services as a result.

Google has modified its internal NTP time servers to gradually add a couple of milliseconds to its regular clock adjustments during a window before the actual leap second is required. ®

High performance access to file storage

More from The Register

next story
Android engineer: We DIDN'T copy Apple OR follow Samsung's orders
Veep testifies for Samsung during Apple patent trial
Windows 8.1, which you probably haven't upgraded to yet, ALREADY OBSOLETE
Pre-Update versions of new Windows version will no longer support patches
Microsoft lobs pre-release Windows Phone 8.1 at devs who dare
App makers can load it before anyone else, but if they do they're stuck with it
This time it's 'Personal': new Office 365 sub covers just two devices
Redmond also brings Office into Google's back yard
Half of Twitter's 'active users' are SILENT STALKERS
Nearly 50% have NEVER tweeted a word
Batten down the hatches, Ubuntu 14.04 LTS due in TWO DAYS
Admins dab straining server brows in advance of Trusty Tahr's long-term support landing
Windows XP still has 27 per cent market share on its deathbed
Windows 7 making some gains on XP Death Day
Internet-of-stuff startup dumps NoSQL for ... SQL?
NoSQL taste great at first but lacks proper nutrients, says startup cloud whiz
US taxman blows Win XP deadline, must now spend millions on custom support
Gov't IT likened to 'a Model T with a lot of things on top of it'
prev story

Whitepapers

Securing web applications made simple and scalable
In this whitepaper learn how automated security testing can provide a simple and scalable way to protect your web applications.
Five 3D headsets to be won!
We were so impressed by the Durovis Dive headset we’ve asked the company to give some away to Reg readers.
HP ArcSight ESM solution helps Finansbank
Based on their experience using HP ArcSight Enterprise Security Manager for IT security operations, Finansbank moved to HP ArcSight ESM for fraud management.
The benefits of software based PBX
Why you should break free from your proprietary PBX and how to leverage your existing server hardware.
Mobile application security study
Download this report to see the alarming realities regarding the sheer number of applications vulnerable to attack, as well as the most common and easily addressable vulnerability errors.