Check your machines for malware, Linux developers told
Kernel.org reopens under hacking pall
Following a series of embarrassing intrusions that hit the servers used to maintain and distribute the Linux operating system, project elders have advised all developers to check their Linux machines for signs of compromise.
Emails sent Friday by Linux kernel lead developers Greg Kroah-Hartman and H Peter Anvin arrived as volunteers with the open-source project worked to bring LinuxFoundation.org, Linux.com, and Kernel.org back online following attacks that gained root access to the multiple servers that host the sites.
Among other things, project leaders are requiring all developers to regenerate the cryptographic keys used to upload source code to the site, and to ensure their systems are free of rootkits and other types of malware.
"The compromise of kernel.org and related machines has made it clear that some developers, at least, have had their systems penetrated," Kroah-Hartman wrote in one message. "As we seek to secure our infrastructure, it is imperative that nobody falls victim to the belief that it cannot happen to them. We all need to check our systems for intrusion."
He went on to advise developers follow seven steps to see if their systems have been targeted, including running chkrootkit, a rootkit detection application for Linux machines. A separate email sent by Anvin laid out the process for regenerating a new set of RSA keys after the old ones were compromised in the attacks.
This hygiene lesson comes as kernel.org and linuxfoundation.org came back online on Monday after an outage that lasted at least three weeks. The homepage of the related linux.com said the website remained down for maintenance and would be restored soon.
Kernel.org was shuttered following the discovery in late August that the personal machine used by Anvin and kernel.org servers known as Hera and Odin1 were infected by malware that gained root access. The trojan sat undetected for at least 17 days before it was discovered on August 28.
A week later, project leaders took linux.com and linuxfoundation.org offline after detecting those systems had also been compromised.
It's fair to say the mass infection and subsequent clean up of Linux developers' machines and servers don't stand as the project's finest hour. The platform is held up by its most ardent fans as a paragon of security that's largely immune to the types of compromises that routinely hit systems running Microsoft's Windows operating system. At time of writing, more than five weeks after the hacks first came to light, the SSL certificate used to authenticate https://www.kernel.org was configured incorrectly and git.kernel.org remained unavailable.
Project leaders have yet to say how they were penetrated, so it's hard for an outsider to know whether they've plugged the holes that allowed them to be compromised. If they hope to regain the trust of users, they'd do well to provide the kind of detailed postmortem that followed the rooting of Apache.org last year. ®
We have been here before. People just forgot.
First of all, there ain't such thing as a secure OS.
Second, in the days before the authors of Back Orifice showed that a windows rootkit is possible Linux was the primary target. I used to run a mid-size academic network in the mid-90es and there was a point where the average time before we got hit by a _NEW_ rootkit variety was down to 48 hours. Sendmail compromises, compromises in basic daemons like ntalk, compromises in bind, etc - you name it. I lost 7 kg spending sleepless nights in front of the keyboard with tcpdump chasing k1dd10tz (it was in the days before snort), rewriting code and patching systems like mad.
The first automated exploit framework observed in the wild was targeting linux too (I had to deal with the fallout from that one too in my day job).
These petered out towards 1998-2000 and dropped to nearly nothing after all major distributions picked up key components out of OpenBSD.
All of this happened versus the backdrop of the rising wave of Windows rootkits so people simply forgot where we started. It however never went away. It was there, it is there.
The real security problem remains the meaty bit.
KILL ALL HUMANS!!!
I had some botnet problems arise on my Linux box, so I know how these things go. I still have no idea how they managed to execute commands from the web server and cause the www-data user to run an eggdrop IRC bot along with a menagerie of IRC stuff, but it did, nonetheless. Luckily I run the web server as www-data or it would have rooted me, and that's definitely not good. After updating Wordpress, Apache, PHP, and other things, it seems to have gone away. That, and I blocked the IPs that were constantly scanning me and the providers of the malware in the iptables firewall, which really took a chunk out of the botnet's ability to take over my server. Netstat and tcpdump are your best friends when dealing with this type of problem. Most of the addresses were in China, Russia, and other such places.