Feeds

Leap second bug cripples Linux servers at airlines, Reddit, LinkedIn

Not a good time to be Australian

Choosing a cloud hosting partner with confidence

The leap second inserted at the weekend crippled Linux-powered servers running one of the world’s largest airline reservation systems - delaying and cancelling flights.

Machines running the mighty Amadeus Altea system were brought down soon after an extra second was added to Coordinated Universal Time (UTC) at midnight on Saturday, 30 June. The bonus second was inserted at the direction of time boffins to keep UTC synchronised with Earth's slowing rotation.

The Altea system was taken offline for an hour, and staff at Qantas and Virgin Australia had to check in passengers manually, disrupting flight plans.

A spokesperson for Amadeus confirmed to The Reg today that the outage had been caused by a bug in the kernel of the open-source Linux operating system, and the flaw was triggered by the leap-second change on Saturday night. He said the problem has been sidestepped using a workaround within an hour, but Amadeus is investigating how to avoid and detect similar bugs in advance.

Servers run by Mozilla, StumbleUpon, Yelp, FourSquare, Reddit and LinkedIn were also reported to have been hit by the same bug. Mozilla said its implementation of the Java-based Hadoop data processing framework and ElasticSearch weren’t working properly on Saturday evening.

Mozilla’s Eric Ziegenhorn posted at 0517 PT, minutes after the leap second was added:

Servers running Java apps such as Hadoop and ElasticSearch and Java doesn't appear to be working. We believe this is related to the leap second happening tonight because it happened at midnight GMT.

Of all these, however, it’s the Altea outage that was by far the more troubling: Amadeus provides the backend booking and reservation system for a growing number of the world’s airlines. Amadeus claims it is the world’s second largest processor of online bookings while it is reported to handle 25 per cent of the world’s 84,000 daily flights.

Amadeus claims 135 airlines have implemented its Altea reservation system, more than 100 have purchased Altea inventory and 60 use Altea departure control. The Altea system was rewritten from a mainframe app in 2004 and moved to Unix-like systems in a response by Amadeus to keep up with changing demands.

Rolled out in 2005, Altea is a set of software modules for booking and reservations that run on Linux and Unix servers, using Java Enterprise Edition, Spring and Apache. Amadeus built the system to move off ageing big iron.

The leap second was added to compensate for the Earth’s uneven rotation by the International Earth Rotation and Reference and System Service. UTC is the time standard for all clocks, devices and applications, as well as POSIX-compliant operating systems, and a second is periodically inserted. There’s traditionally been three ways of implementing the change, which are described here.

Linux distro biz Red Hat published a patch for its Enterprise Linux here - patches for other flavours of the OS are circulating. It is believed the leap second causes the Linux kernel to livelock when the system attempts to adjust the time and date of the computer, causing processors to spin on their wheels doing nothing and hampering services as a result.

Google has modified its internal NTP time servers to gradually add a couple of milliseconds to its regular clock adjustments during a window before the actual leap second is required. ®

Remote control for virtualized desktops

More from The Register

next story
Download alert: Nearly ALL top 100 Android, iOS paid apps hacked
Attack of the Clones? Yeah, but much, much scarier – report
You stupid BRICK! PCs running Avast AV can't handle Windows fixes
Fix issued, fingers pointed, forums in flames
Microsoft: Your Linux Docker containers are now OURS to command
New tool lets admins wrangle Linux apps from Windows
Facebook, working on Facebook at Work, works on Facebook. At Work
You don't want your cat or drunk pics at the office
Soz, web devs: Google snatches its Wallet off the table
Killing off web service in 3 months... but app-happy bonkers are fine
First in line to order a Nexus 6? AT&T has a BRICK for you
Black Screen of Death plagues early Google-mobe batch
prev story

Whitepapers

Why and how to choose the right cloud vendor
The benefits of cloud-based storage in your processes. Eliminate onsite, disk-based backup and archiving in favor of cloud-based data protection.
A strategic approach to identity relationship management
ForgeRock commissioned Forrester to evaluate companies’ IAM practices and requirements when it comes to customer-facing scenarios versus employee-facing ones.
Go beyond APM with real-time IT operations analytics
How IT operations teams can harness the wealth of wire data already flowing through their environment for real-time operational intelligence.
The total economic impact of Druva inSync
Examining the ROI enterprises may realize by implementing inSync, as they look to improve backup and recovery of endpoint data in a cost-effective manner.
Reg Reader Research: SaaS based Email and Office Productivity Tools
Read this Reg reader report which provides advice and guidance for SMBs towards the use of SaaS based email and Office productivity tools.