Sage Pay prevents punters blowing Xmas wad
Four hour ordeal leaves Santa's sack empty
Regcast training : Hyper-V 3.0, VM high availability and disaster recovery
Sage Pay has apologised for a four hour outage yesterday that left online merchants with no way to process payments for Christmas orders.
Simon Black, managing director of Sage Pay, said: "I deeply regret the impact that we've had on your business, at what is the worst possible time for our systems to fail."
The length of the downtime was blamed on "systems diagnostics being carried out to pinpoint the root cause".
Sage Pay online payment processing was unavailable for most customers between 6pm and 10pm yesterday, Monday 6 December.
The company said the problem was caused by running a background job which caused "an existing, but unknown, hardware flaw".
Simon Black told the Reg: "We were running a data processing report which caused an overload which then led to a hardware failure. Because we were unable to diagnose quickly what had gone wrong we were unable to go straight to a back up system.
"But there was no loss of data and there is no backlog of orders."
Black said customer reaction was mixed - he emailed customers early this morning explaining the problem. He said some replies thanked him for the company's openess but others were understandably angry.
Black said: "I and my team deeply regret this downtime and apologise to our customers. It was an isolated incident and the same issue will not happen again."
Black said the issue is identified as fully resolved and "was not related to our new system infrastructure."
Go here to see the email from Sage Pay's Simon Black. ®
Regcast training : Hyper-V 3.0, VM high availability and disaster recovery
COMMENTS
A failure a day
We use sagpay on our website. I used to screen-scrape Sage's status page for our alerting system until they got wise to that and made it restricted access (they don't like people knowing that their systems are knackered).
After that I started scraping their RSS feed, but that didn't work very well because they used to only update that once they'd (temporarily) fixed their problems, with a 'all OK' message.
However, since their last 'upgrade', they've started being more honest on their RSS feed, and actually reporting when there are problems. The down-side to this is that there have been very few days in the last month when that alert hasn't been fired because of their poor systems.
So I've had to turn of alerting for their system.
I suspect the Windows XP laptop they run their system on is become a bit old an knackered now.
It's been like this
...ever since they switched from Protx to Sagepay, the change of name bought with it failure after failure after failure. Only one you can definately say wasn't their fault, and that was the fire at the Paddington telephone exchange, the rest seems to be deep mismanagement and failure to adaquately test their unwanted "improvements" to the system.
Not surprised
Spent 10 years working for Sage (Well I say working, attending Sage is more truthful), the whole design ethos isn't on quality at all. I remember when the technicians used to come in for Saturday morning overtime to do product bug testing. It wasn't particularly structured. Basically we would just go through the stuff customers do (This worked well as support bods have seen pretty much every stupid thing a customer would try). R&D didn't like this very much as we caught a load of bugs early so they decided to bring in a "professional testing team", made up of bods off the street. Much fanfare from R&D, our new product launched with X% less bugs than normal. There were actually more bugs, just that the new muppets didn't know what to look for.
As for T'interweb stuff Sage really isn't best placed to operate in that marketplace. They don't tend to like spending money on fully resiliant services
Anonymous coward cos they like to sue!
IT infrastructure monitoring strategies
What you need to know about cloud backup
Enabling efficient data center monitoring
Agentless Backup is Not a Myth
Top 10 SIEM implementer’s checklist
