MessageLabs in partial email outage
Email slowed to snail mail for eight hours
A substantial number of virus filtering firm MessageLabs' customers experienced severe delays in sending and receiving email yesterday because of network routing problems.
Customers, mainly in the London area, connected to two of the managed service providers 20 server racks would have noticed slow email receipt and difficult sending messages from around midday until 7.45pm yesterday, when it was able to resolve the problem.
Alex Shipp, chief antivirus technologist at MessageLabs, explained: "The problem was caused by network misconfiguration at one of our telecommunication providers. This caused SMTP sessions to certain IP ranges to initiate, but to fail after a while, which meant that mail to and from those IP ranges was very slow."
"Lower level routing protocols could not automatically discover the problem (the route worked *most* of the time) and so did not automatically switch routes," he added.
None of the messages were lost but the delays to important messages caused consternation at a number of MessageLabs' customers, who contacted us about the problem.
"This is supposed to be a high-availability, business critical service. There must be a lot of pissed off people about who were expecting important emails to arrive yesterday afternoon," said one network admin who contacted us about the issue.
MessageLabs apologised for any inconvenience caused and said it would introduce diagnostic software that would identify the caused of any future partial failure and re-route traffic. Ironically if a particular connection with one of MessageLabs' telco partners had failed entirely traffic would be re-routed automatically and customers would not have noticed any problem. ®