Original URL: http://www.theregister.co.uk/2009/09/09/sagepay_outage/
Sage payment processor suffers 24-hour outage
Transaction history repeats itself (again)
SagePay - the payments processor serving 25,000 companies in the UK and Ireland - experienced a major outage on Monday evening, and as problems dragged on late into Tuesday, one of its biggest competitors claimed a significant uptick in business.
Caused by a "database-related" issue, the outage began at 7:48pm London time on Monday, and though credit and debit card payments processing resumed two hours and 22 minutes later, users were unable to access historical transaction data for another 22 hours. Service was completely restored at about 8:30pm on Tuesday night.
"Urgh. I am extremely annoyed by this, we had our first failure at 19:06 [Monday] and are now down £000's worth," another user wrote on the uk business forums at 22:16 that day. "Of course it would have to have happened at the busiest time! I'm surprised the issues have lasted this long, not impressed at all."
Historical data was unavailable, so users were unable to use several secondary services, including the ability refund transactions, void them, and authorize them against any archived transactions.
After the primary service was restored, users continued to report that transactions were failing. "Looks like it is still down. Have done a few test orders on ours but it keeps saying the cards have been declined," one user said. "Maybe a lesson to get a secondary payment installed such as paypal now."
According to Simon Black, managing director of SagePay, historical data was unavailable for such a long stretch because the company chose to rebuild its entire database from scratch. "The cause of the outage was database-related," he says. "We made the decision to prioritize restoring all data, if we could possible do that. What that meant is that we had to painstakingly do a rebuild, which in the end took the best part of Tuesday."
All data was indeed restored, Black says. After primary service was moved to a new data center on Monday evening, the database was rebuilt separately, and on Tuesday night, the two were merged back together.
Originally known as Protx, SagePay was renamed in 2006 after it was acquired by Sage, the Intuit-like business and accountancy software firm. And as the company struggled to restore service on Tuesday, conspicuous Sage competitor KashFlow didn't miss the opportunity to point the finger.
KashFlow was originally a Protx customer, but once the service was acquired by Sage, it switched to IridiumCorp for payments processing. But according to KashFlow's chief executive and founder Duane Jackson, the company was still affected by the outage because its VOIP provider, Voipfone, is a SagePay user. "We couldn't use any of our phones because we couldn't add credit," Jackson tells The Reg.
Meanwhile, Iridium tells us that during the outage, nearly 200 businesses contacted the company to switch their payment processing from SagePay to Iridium. Iridium CEO Sean Brietsche says the company picked up about 50,000 transactions a month from 189 merchants.
Iridium offered to provide free service fees equal to the amount of time remaining on a customer's SagePay contract. So, if you had six months left on your service, six months would be free.