Feeds

Breaching Fort Apache.org - What went wrong?

Open-sourcers put locks on keys

5 things you didn’t know about cloud backup

Administrators at the Apache Software Foundation have pledged to restrict the use of Secure Shell keys for accessing servers over their network following a security breach on Monday that briefly forced the closure the popular open-source website.

In an detailed postmortem describing how hackers penetrated several heavily fortified machines, site admins identified their use of SSH keys as one of the flaws that made the attack possible. They went on to lay out concrete ways they plan to fix the problems, which also included faulty procedures for backing up data and methods for providing geographically localized servers for downloads.

"At no time were any Apache Software Foundation code repositories, downloads, or users put at risk by this intrusion," they wrote here. "However, we believe that providing a detailed account of what happened will make the internet a better place, by allowing others to learn from our mistakes."

The hack started with the compromise of apachecon.com, a website that's owned by the ApacheCon conference production company. Although logs confirming the exact cause were destroyed, investigators suspect it was the exploit of one or more local root vulnerabilities in the Linux kernel for which Red Hat issued a patch seven days earlier but had not yet been installed. They then used the SSH key for a backup account to access the server that runs people.apache.org.

With an unprivileged user account, the attackers added common gateway interface scripts to the document root folders for several Apache websites. Routine backup processes then copied the scripts to the foundation's production server, where they became visible to the outside world. Those scripts, which allowed the hackers to obtain remote shells, were aided by Apache's use of ExecCGI.

"The use of SSH keys facilitated this attack," the postmortem continued. "In hindsight, our implementation left a lot to be desired - we did not restrict SSH keys appropriately, and we were unaware of their misuse."

The admins have since recreated new SSH keys with minimum lengths of 4096 bits and mandated the use of a separate one for each host doing backups. Additionally, they now require the use of from="" and command="" strings and allow connections only from machines that are backing up data to prevent third party machines from being able to establish connections.

Other changes include the requirement that all users with elevated privileges use a one-time password for everything for sudo on certain machines. The foundation is also considering the disabling of CGI support on most website systems.

Apache deserves credit for the number of security-in-depth measures already in place that prevented the attack from penetrating more sensitive parts of the foundation's system.

For example, core servers on the network employed a variety of hardened operating systems, including CentOS, FreeBSD-7, and Solaris 10, creating a diverse target that made it hard to attackers to escalate privileges. What's more, the use of the ZFS filesystem and redundant services in two locations made it possible for admins to restore service quickly.

Even more commendable is the foundation's willingness to publicly lay out how its mistakes contributed to the breach. It's never a pleasant task to publicly lay out blunders in such painstaking detail - just ask Apple, which steadfastly refuses to discuss security missteps, however obvious they are to the rest of the world. But as the Apache postmortem shows, such disclosures can leave users feeling surprisingly reassured about the state of one's security. ®

Secure remote control for conventional and virtual desktops

More from The Register

next story
Ice cream headache as black hat hacks sack Dairy Queen
I scream, you scream, we all scream 'DATA BREACH'!
Goog says patch⁵⁰ your Chrome
64-bit browser loads cat vids FIFTEEN PERCENT faster!
JLaw, Kate Upton exposed in celeb nude pics hack
100 women victimised as Apple iCloud accounts reportedly popped
NIST to sysadmins: clean up your SSH mess
Too many keys, too badly managed
Scratched PC-dispatch patch patched, hatched in batch rematch
Windows security update fixed after triggering blue screens (and screams) of death
Researchers camouflage haxxor traps with fake application traffic
Honeypots sweetened to resemble actual workloads, complete with 'secure' logins
Attack flogged through shiny-clicky social media buttons
66,000 users popped by malicious Flash fudging add-on
New Snowden leak: How NSA shared 850-billion-plus metadata records
'Federated search' spaffed info all over Five Eyes chums
Three quarters of South Korea popped in online gaming raids
Records used to plunder game items, sold off to low lifes
Oz fed police in PDF redaction SNAFU
Give us your metadata, we'll publish your data
prev story

Whitepapers

Endpoint data privacy in the cloud is easier than you think
Innovations in encryption and storage resolve issues of data privacy and key requirements for companies to look for in a solution.
Implementing global e-invoicing with guaranteed legal certainty
Explaining the role local tax compliance plays in successful supply chain management and e-business and how leading global brands are addressing this.
Advanced data protection for your virtualized environments
Find a natural fit for optimizing protection for the often resource-constrained data protection process found in virtual environments.
Boost IT visibility and business value
How building a great service catalog relieves pressure points and demonstrates the value of IT service management.
Next gen security for virtualised datacentres
Legacy security solutions are inefficient due to the architectural differences between physical and virtual environments.