By Anonymous CowardPosted Tuesday 8th April 2008 12:23 GMT
there'll be a IT Director job advertised pretty soon.
Remember HSBC, fire them, don't let them resign and don't give them any golden parachutes. It's long passed time these jokers were punished for thier failures just like the rest of us.
By Seanie RyanPosted Tuesday 8th April 2008 12:44 GMT
while its not acceptable for HSBC to be down like this and not keep clients updated, there is also a finger to be pointed at clients who rely on this exclusively and dont have any alternative.
If something is _SO_ crucial to your business, why no backup option?
I can never understand this.
Anything to do with computers can and will go wrong eventually. Fact of life.
If you could potentially lose thousands at any outage, then its your responsibility to have backup plans. No excuses. Even if its just paypal. anything.
By Anonymous CowardPosted Tuesday 8th April 2008 12:55 GMT
Banks may have fancy web sites, but their IT systems are still ancient.
Their IT departments are run like civil service departments - lots of meetings , lots of changing priorities, and very little done.
The real work gets done by external contractors and a few skilful souls in the bank who have to battle against blind incompetence and 'job for life' managers who prefer trying their luck with the call-centre staff, the silly old fools..
By Matt CurryPosted Tuesday 8th April 2008 13:12 GMT
HSBC e-payments system is horrific to integrate with, to add another service and be able to switch between the 2 adds yet another level of complexity. Sticking a paypal button might seem like a good idea but
a) you'd lose cusotmer confidence more than just saying "we're having a problem"
b) you can't do transactions such as pre-Auth and Post-Auth, which we use so that the card is charged correctly post-delivery.
Currently, we're developing a buffering system that hopefully(!) will allow us to buffer transactions bound for pre-auth when HSBC invariably go down again.
Although, by that time, we're likely to have moved.
By Anonymous CowardPosted Tuesday 8th April 2008 13:23 GMT
I have to agree with Seanie that people should have some form of business continuity plan should a service or services like these fail. Even taking manual payments with a PDQ keeps ya going!
It would not be hard to integrate a second gateway into most if not all shops/stores/whatever and fallover to that should or when HSBC breaks. You get paid into the same account it is not a major issue then.
Look at the mess with Protx a while back when people were screaming the same thing that they were loosing £10,000s as it was offline - surely if your putting that much into a gateway you 'should' have a backup.
And surely as a "business" you don't have one copy of your accounts ... payroll ... customers list ... Do you????
Mind you lost faith with integrating HSBCs payment thing ages ago. Far faster ones exsist nowadays thank god.
By Anonymous HeroPosted Tuesday 8th April 2008 13:40 GMT
I couldn't agree more.
It's called business continuity planning, and most companies never bother to think about it or realise that it's something, just like all the other costs of doing business, you need to pay for. The problem is that doing business via the web has been tarted up as some kind of panacea of fault tolerancy and commercial utopia, when in fact it's exactly not. Businesses are brainwashed into thinking that they should pay little or nothing for a 24x7x365 presence to run their web shop fronts.
It never fails to amaze me the irate calls we get claiming losses of 10's of thousands of pounds of sales if one of our shared (tenner a month) boxes hits some problem for a few minutes. Yet when we offer them a dedicated box with failover which probably costs them less a month than the leasing payments on their Jags and Astons sitting in their company parking spaces they suddenly lose interest.
If your business is that bloody important and you do it on the web make sure you have two of everything (at least) - and that includes an alternate payment gateway (which may cost you a bit more per transaction, but at least you can still take a payment/authorise in real time).
By Anonymous CowardPosted Tuesday 8th April 2008 14:06 GMT
"The real work gets done by external contractors and a few skilful souls in the bank who have to battle against blind incompetence and 'job for life' managers who prefer trying their luck with the call-centre staff, the silly old fools.."
Is this supposed to be tongue in cheek?! Has this idiot ever worked for a Bank? I love the attitude of people who feel this could never happen to them. Has it ever occurred to anyone that it could be an external vendor at fault?! It's all too easy to criticise without the necessary facts to hand...
By Anonymous CowardPosted Tuesday 8th April 2008 14:13 GMT
This is what happens when you try to build banking systems with Visual Basic and graduate programmers. Bring back the AS/400's!
It's not like I can call every customer and take details over the phone #
By Matt CurryPosted Tuesday 8th April 2008 14:21 GMT
-->It never fails to amaze me the irate calls we get claiming losses of 10's of thousands of pounds of sales if one of our shared (tenner a month) boxes hits some problem for a few minutes.
but when your annual HSBC processing bill comes to just under £20k - you can see why you could get a bit irate
integrating with another system to provide failover would cause more problems that it solves.
For example with HSBC you get a response code - 22 I think it is, to say that the customer's supplied postcode doesn't match their card accounts. If you then integrate with RBOS or whoever, who return code NoMATCHPOST instead - you have to have logic for every type of transaction and every possible outcome, twice, all the time relating it to which processor you're using at the time, just becuase the major international bank you've been using can't get it's arse in gear.
If your payments system is suitably abstracted, handling different responses from different gateways should be a simple case a of writing a translation layer.
The additional time and cost spent implementing a second payment gateway far outweighs the risk of not being able to process transactions.
If your business depends heavily on taking online payments you'd be crazy to rely on a single provider, no matter how good their track record. It's the same rule that applies to hosting, backup and pretty much anything else that can fail.
It's easy to close an HSBC account; you go into a branch, and say "I'd like all my money please. Bankers Draft. If you can't do that because of a computer error, cash will be fine. I assume you can still count without using a calculator?"
You then promptly take your money down to another bank, and deposit in a new account. Setting up the services you need may take longer, but there are alternatives in the mean time. Assume HSBC have outsourced the fault finding to India, and they're using a remote desktop session over a 28.8k modem, and a local engineer whos qualifications top out at "have upgraded the ROM in a CPU."
The important thing is your money is not with HSBC, and that's what they don't want to happen.
By Mic HewittPosted Tuesday 8th April 2008 16:32 GMT
We have had a backup system in place on our API based HSBC system that checks to see if the HSBC server is there using CURL. If you are a developer then look into this. You can then allow your system to automatically decide if the payment is taken there and then or it stores it for later, so that it can be manually processed when the system comes back up.
Yes we have lost thousands in the past from HSBC but sh1t happens people and it will continue to happen all over the world with many things. Take some time off tomorrow and plan your contingency, do not waste time on complaining!
By Matt CurryPosted Tuesday 8th April 2008 19:10 GMT
The transaction buffer we've spent today coding is now up and running. HSBC are going to pay!(for the development of the buffer on a time and materials basis)
Latest news from HSBC is that they are turning the service off (when was it on?) until 9am tomorrow.
thanks to the people at nochex.com they have saved my bacon and what a bonus they dont even charge a monthly fee :-) + they give me an account manager that helped me switch
By Anonymous CowardPosted Wednesday 9th April 2008 09:28 GMT
I have been in contact with a few "insiders" who have been able to confirm the cause of the problem.
After the recent loss of CDs they decided to upgrade their systems to use DVDs, they used up their stocks of CDRs and were due to go live with their new improved DVD based communications system for handling transactions - however they realised a bit late that the server only had a CD burner, not a DVD burner and naturally they had no spare CDRs to revert to the old system.
The delay in getting back online isn't being helped by the fact that for some reason their supplier is not taking any orders due to some sort of payment processing outage.
By Seanie RyanPosted Wednesday 9th April 2008 09:52 GMT
You still dont get it do you?
Lets say that next week NoChex decide to do an upgrade and it all goes wrong.
What then? It _could_ happen.
You have just seen how easy it was to switch. Why didnt everyone who is complaining setup a nochex account months ago and have, at the very LEAST, 2 sets of html/php/asp pages, so you can put your nochex in when you want, and your hsbc (or other) when you want. Its that easy guys!!.
I just hope you all kept your HSBC account active and a copy of the integration.
That means you NOW have a backup for the future. Pat yourself on the back.
hope NoChex doesnt buckle under the new load !! ;-p
By Anonymous HeroPosted Thursday 10th April 2008 12:14 GMT
If you're doing 10k of business a day, a 20k bill from HSBC is a drop in the ocean....and it's called 'cost of sale' which so many retailers fail to understand....but that's another rant for another time.
In answer to "integrating with another system to provide failover would cause more problems that it solves.", as Whelk correctly states, you wrap your payment processes up in an API that abstracts away the specifics of the payment service.
By Anonymous CowardPosted Friday 11th April 2008 11:26 GMT
Yes it was slightly tongue in cheek, no I don't know the specific facts of the case, and yes I have worked for a bank - for a very short period of time (NR as it happens - ho ho ho).
But you are quite right - I should know better. On the other hand, I think there was more than a grain of truth in what I said. Apologies to all hard working sould who work in banking IT depts. Blame the managers is always my watchword - they get paid to do the planning after all...
By IshkandarPosted Wednesday 16th April 2008 10:11 GMT
They've just been hit by one of the biggest losses in the sub-prime crisis. Those poor things are still reeling from the loss of their massive bonuses and their annual junket^H^H^H^H^H^H board meeting in their Bahamas Headquarters !!
Please take pity on those poor, deprived persons struggling to make do with their last Rolls Royce and multi-million quid country home !!
Comments on: HSBC e-payments system goes titsup (again)
Sounds like #
By Anonymous Coward Posted Tuesday 8th April 2008 12:23 GMT
2 failures #
By Seanie Ryan Posted Tuesday 8th April 2008 12:44 GMT
Complaint Number #
By Adam Posted Tuesday 8th April 2008 12:46 GMT
Fixing the problem #
By Ash Posted Tuesday 8th April 2008 12:46 GMT
Banking IT #
By Anonymous Coward Posted Tuesday 8th April 2008 12:55 GMT
@Ash #
By Mr B Posted Tuesday 8th April 2008 13:09 GMT
Backup systems #
By Matt Curry Posted Tuesday 8th April 2008 13:12 GMT
The way it is sometimes #
By Anonymous Coward Posted Tuesday 8th April 2008 13:23 GMT
change control? #
By Biton Walstra Posted Tuesday 8th April 2008 13:35 GMT
@Seanie #
By Anonymous Hero Posted Tuesday 8th April 2008 13:40 GMT
My HSBC Fix #
By Anonymous Coward Posted Tuesday 8th April 2008 13:41 GMT
Tongue in cheek? #
By Anonymous Coward Posted Tuesday 8th April 2008 14:06 GMT
Crappy software #
By Anonymous Coward Posted Tuesday 8th April 2008 14:13 GMT
It's not like I can call every customer and take details over the phone #
By Matt Curry Posted Tuesday 8th April 2008 14:21 GMT
@Matt Curry #
By Whelk Posted Tuesday 8th April 2008 14:50 GMT
thanks for the Tip (hsbc fix) #
By Gail Posted Tuesday 8th April 2008 15:09 GMT
@Mr B #
By Ash Posted Tuesday 8th April 2008 15:10 GMT
Merchant..rhyming slang? #
By Sam Posted Tuesday 8th April 2008 15:46 GMT
Service will not be resumed until 9am tomorrow at the earliest #
By Adam Harris Posted Tuesday 8th April 2008 16:18 GMT
All businesses need contingency #
By Mic Hewitt Posted Tuesday 8th April 2008 16:32 GMT
phew #
By Matt Curry Posted Tuesday 8th April 2008 19:10 GMT
Thanks to Nochex #
By Gail Posted Wednesday 9th April 2008 08:58 GMT
Reason Known #
By Anonymous Coward Posted Wednesday 9th April 2008 09:28 GMT
@ all new nochex clients #
By Seanie Ryan Posted Wednesday 9th April 2008 09:52 GMT
HSBC #
By 2FishInATank Posted Wednesday 9th April 2008 13:45 GMT
@Matt #
By Anonymous Hero Posted Thursday 10th April 2008 12:14 GMT
re: Tongue in cheek? #
By Anonymous Coward Posted Friday 11th April 2008 11:26 GMT
Relax, guys !! #
By Ishkandar Posted Wednesday 16th April 2008 10:11 GMT