Original URL: http://www.theregister.co.uk/2010/10/07/sorbs_cockup/
Spam blacklist snafu prompts global gnashing of teeth
Legit IPs blocked in SORBS cockup
Many email users around the world have been unable to send messages because of ongoing technical problems with a popular service designed to prevent spam from reaching its intended destination.
The problems at SORBS — short for the Spam and Open Relay Blocking System — began on Wednesday and continued into much of Thursday, said Michelle Sullivan, who founded the real-time blacklisting service in 2002 and sold it to GFI Software last year. As a result, messages sent from a huge number of legitimate mail servers were labeled as junkmail and returned to sender.
The widespread outage caused migraines for sales and support staff around the world, as this ongoing Twitter feed demonstrates.
The snafu was the result of a transition from one SORBS system to another that corrupted a database containing potentially millions of IP addresses, Sullivan told The Register. SORBS admins have responded by temporarily clearing out the entire table of faulty listings under the theory that it's better to let through spam than to block real email. They are in the process of rebuilding the database and populating it to user servers around the world, a process that could take up to 24 hours.
Sullivan denied reports such as those here and here that the problem caused all inbound email from Google, Yahoo, and Rackspace, to name a few, to be blocked from SORBS subscribers. The portion of the database that was corrupted stored entries for its DUHL, or dynamic user host list, which mainly contains dynamic IP addresses offered by ISPs, Sullivan explained.
The outage was nonetheless large. There were more than 79,000 corrupted entries, each of which could contain millions of individual IP numbers, said Sullivan, who was unable to provide an estimate of the total number of addresses that were incorrectly being filtered.
The DUHL portion of the database contains a huge amount of historical data, mainly in the form of IP addresses that were listed as spam sources in the past but are no longer considered as such. During a data migration, the flags that were used to indicate that a listing was historical were deleted, causing the addresses to be considered current. SORBS is in the process of restoring the flags. The historical entries are used to help programs rate the likelihood that a given IP address is malicious.
To make matters worse, the database problem coincided with a denial-of-service attack that made it hard for people to reach the SORBS website. The DoS assault has been ongoing for several weeks, Sullivan said, but in the last 24 hours, it turned “smart,” meaning it was able to evade the defensive measures admins had put in place to filter out the garbage traffic.
SORBS has been able to repel the latest attacks, although the site still appears to be sluggish at times. The attacks in no way caused the database problems. ®