Original URL: https://www.theregister.com/2013/06/27/tech_threats_to_banking/
How City IT is under attack from politicians, diesel bugs, HR
Oh, and the stock exchange could blow any moment....
Posted in On-Prem, 27th June 2013 11:04 GMT
Comment The stupidest thing I’ve ever said was “if it was a jet, the tower would have collapsed” on September 11th and I feel the same about RBS. As I pass it most days, part of me expects to see crowds outside, perhaps including the police and TV camera crews, because I can’t understand why it still functions.
The Reg has covered in depth the antics of its offshored systems but they’ve left both Reg readers and, for lack of a better word, RBS “management” with the wrong idea about the threats to it.
As a headhunter I hear just too many stories about the people they are hiring. OK, the outfit to which they’d outsourced HR/recruiting didn't help matters much, but the core problem is that bright people are extremely reluctant to work there, seeing the worst of two worlds where you have to work bank hours but with what are reportedly unappetizing pay levels.
There is some raw talent there since newish grads are being put in positions of responsibility that normally would have taken years to achieve, which in some ways is good … so long as you are hiring from them, rather than actually trusting RBS with your money in the wake of the mainframe cockup.
Ironically the recession that the chimps helped create is helping protect RBS for now because there are some very smart people still on board waiting for the right time to jump ship and the slowdown in banking means that has taken longer to appear. So the worst at RBS may be yet to come, when defections hit some critical level and juniors make mistakes when there’s no grownups left to bail them out.
Shafting Stephen Hester may have kept the arts graduates in Parliament and the BBC happy, but the RBS clusterfuck really wasn’t his fault. He arrived afterwards to help sort it out and screwing him out of his bonus then “letting him resign” sent a clear and bad message to everyone. You probably aren’t sympathetic to bankers, but unless the government introduces conscription to fight the tech/finance Vietnam at RBS, then you the taxpayer will own a huge bank that simply doesn’t work.
The rest of the City is also running risks with most of the consequences so far being kept within the banks themselves and since the regulators are far behind the curve, the pressure won’t really increase until something goes bad enough to get on TV.
Investment banking IT is quite different from the retail banks both in the types of technology and the complexity with which it is strung together. Although RBS retail banking has nightmarishly complex data flows, the systems are more homogenous, which means that a smart operator can deal with multiple types of issue because there are few types of basically different systems. An investment bank is not a single business like Sainsburys.
It’s lots of business units varying in size from six to a few hundred people and each has different tech needs, to the extent that there will be multiple networks, every major server O/S and more different applications than staff, with Excel/VBA playing a far larger part than any rational person would choose or even imagine. At RBS retail (or Sainsburys), it would be astonishing for a branch to hire some C++ programmers or to use FPGAs when x86 CPUs were far short of ideal for a purpose that they could not explain to central IT.
Gather round, children: I tell a tale of SmallTalk
There’s also industrial-scale grids, big data mining and some ancient legacy code as well. Not only Cobol, but SmallTalk, which for you younger people (I graduated in 1984) is an interesting bit of recent history. JP Morgan, for example, is critically dependent on it.
Unlike retail banks which haven’t innovated for decades, internet banks are constantly trying to find an angle and IT is whipped up to support it. Back when I ran a group that had to interface to JPM’s IT, I was shocked by the shabby state of its core technology, but they seem to have handled the SmallTalk issue better than most banks. But if you suddenly needed a SmallTalk programmer who understands banking, where would you get one?
My first direct experience of this was when I ran the group at the grand old age of 37. I was told “Dominic, you’re old” when I came into work and asked why my desk was surrounded by children in their 20s. I hobbled over to a desk, as befitted my age, and saw BTrieve error messages scrolling up the command Windows.
“Your indexes are screwed,” I said. The children were impressed. This was why I was the most expensive guy in the room.
“Can you fix it?”
“No.”
I’d deduced that “NDX” meant indexes in the viciously terse Btrieve messages and remembered from my misspent youth that this was the most common problem, and one reason it wasn’t used any more.
Turns out that they’d fired the one guy who still understood it and to make it even more fun the children asked me to look at the source code. On a server. Somewhere. In one of the largest banks on the planet. Somewhere. The last developer of this stupidly useful system had connected to the server dynamically from the command line using “NET USE S:…” No one knew which one. This apparently was my problem and I lost great credibility by not being able to fix a system without source code that I’d barely been able to understand.
That leads us to the main vulnerability at investment banks. Their diversity increases their earnings (or makes their losses more newsworthy) but the number of things that go wrong goes up at least as quickly. But using so many different types of technology stacked on top of each other means that there are people whose main skill is holding them together. This is so much so that when I do careers counselling I often refer to the “Perl Pothole”, where an ace IT Pro finds himself jammed in a dead end, but doing a critical job pumping data from system A to system B.
This is an issue because banks have been managing down headcount, which is very different from managing down cost, so that IT managers have to make hard and risky decisions. It's now much harder to say “we’ve only got one person who understands the shredding of Bloomberg downloads” without getting a response of “so what’s the problem”?
Contractors will fill these holes, but no matter how sharp their skills the combinations and reasons will take time to master. I like to think of myself as someone who can wing any kind of programming, but when writing this I simply couldn’t think of anything to write about SmallTalk interfacing to Perl. Or for that matter Cobol. Can you?
Secret source, security and BYOD
Pic credit: Kodomut, Flickr
Banks have been pretty resistant to the idea of BYOD beyond letting people use their own smartphones and most (but not all) have got it into their heads that unlocked USB ports are accidents pausing to choose which place to happen.
But then again investment banks face different threats to their country cousins in retail. For example, Goldman Sachs is still concerned about how Sergey Aleynikov made off with some of its source code. Aleynikov was convicted in 2010 in federal court but freed after one year in prison by an appeals court. But New York state prosecutors last year re-charged the former Goldman coder with illegally copying computer software. The programmer has pleaded not guilty.
Most data and source code at a firm isn’t even interesting to the people who work there, much less anyone else and the dullest part of my expert witness work is going though people’s email. However drab my life sometimes feels, it is joy compared to many. The problem is that if you make it hard to get at the source, then when something bad happens, the reaction is slow.
Because important code can be deliciously complex, coming to it cold takes up valuable time and increases the chance that your fix will make it worse and I can share that recent incidents, both public and kept private have made most banks and especially hedge funds apply much tighter controls over source code that previously was left on open access servers and shared freely around the firm. You can’t eliminate risk here, just choose one which will hurt least.
Data storage is skyrocketing, not just because of big data analytics but because regulators want a lot more records kept and for longer, which means previous capacity planning is often wrong and occasionally spectacularly so, like when one IT manager found he was 140TB short of what he needed now and Hitachi had to get the gear on site within 24 hours.
Good suppliers can do this, but you’d be a mug to plan it that way and a quiet casualty of the banking recession is that the supporting ecosystem of specialist tech vendors have been cut to the bone or have gone under. This has removed a crutch for many IT managers including myself and leaning on a crutch that isn’t there may look funny, but trust me it isn’t.
Because banks rarely skimp on network hardware when it is installed. IT leaders have found it an easy and painless way to make a clear saving, at least on paper since they can point to the money not spent without denying a new service to anyone important. In many cases this has actually led to an improvement in reliability since upgrades always have a few snags and the decent quality physical equipment is mostly in the nice bit of the reliability curve between bedding in and getting old.
But the banking recession has dragged on a bit and where this coincided with a timely infrastructure upgrade, there is weakness, especially in the midscale firms. Huawei is a subject of much debate, the kit is attractively priced and seems to do the job and although Cisco might sometimes act with, er, "unmitigated gall", its misbehaviour remains within bounds. Meanwhile, Huawei’s alleged shadowy links to the Chinese military and its hackers frightens a lot of IT management. But smaller firms will see the prices and be tempted - and yes I’m aware I may be falling for Cisco PR here.
Disaster recovery is better, almost good, because for once the regulators seem to have done their job on the structurally important firms, requiring them to treat DR as less of a joke and burden than previously, most are now not only properly equipped, but also tested. Almost gone are the times when “business continuity” relied upon hardware demoted from front line use and of questionable reliability and performance. But a critical weakness for them in the past and for smaller firms still is that “testing” was done by the technologists at weekends.
This means that the systems look OK to people whose jobs are to build them but not to use them, with the results that are sometimes very nearly funny. Of course the spreadsheet that no one in front office bothered to tell central IT about doesn’t get replicated. And the devs were lucky to discover in one particular test that none of the traders could log in, because IT had tested it as themselves with supervisor access.
The weakness in all of these cases is that the big boys usually still assume that IT pros are from the planet Krypton and thus cannot be stopped by terrorism - or even just late-running trains on the London Underground - with too little effort being applied to multi-skilling people to cover for those who don’t make it to the DR site.
Threats to you
Compliance now has teeth, or at least the regulators are holding their mouths closer to your groin. Compliance managers tell me that the steer they are getting is that the measure of a good compliance regime is no longer a lack of reported incidents but rather the punishment for those who get caught.
This includes the whole bank. In the good old days, traders worked out the boundaries of what was acceptable by pushing until someone gently pushed them back, this being hardly an IT problem at all. In fact if an IT pro reckoned there was something dodgy going on, he’d be reckless with his career if he made any sort of fuss.
At this point I could mention the ability of the regulators to protect whistleblowers but we both know I’d be taking the piss if I did anything other than sneer at the idea. By all means write an innocuous email that cites “some possible issues we need to address at someone point” to try to cover your back, which is at most better than nothing.
Power
The neglect of the UK’s infrastructure may not be the fault of City IT managers, but it certainly will be their problem. Not only have we gone beyond the point where we can avoid power cuts due to lack of generating capacity but even the cables and transformers that get it there are so badly maintained or overloaded that we are seeing explosions both under pavements and in their housings. One of the larger transformer buildings in the heart of the City almost next to where the Stock Exchange servers live is surprisingly warm to the touch. Yes that’s a lot of power, and no I’m not saying where.
The world’s major governments pander to farmers by using a mix of sticks and carrots to persuade fuel users to burn food for power or transport in the form of biodiesel. Given that most that go hungry aren’t even slightly white and their farmers are usually pink, this ain’t going to change any time soon and of course the earnest arts graduate who heads up “corporate responsibility” at your bank will lever IT into fuelling standby generators with biodiesel.
Without bugging you with too much organic chemistry, the short version is that if you leave oil in a can for a million years, it doesn’t change much. It's obvious really when you think about how much of it has survived. But biodiesel rots. Bugs live in it and unless you are a complete evangelical it is reasonable to suspect that ever more bugs are evolving to feed upon it.
Bugs don’t burn. Yes OK they do, but not well and certainly not in the fine aerosols that you get in diesel engines, which means the engines block up. That’s less of an issue in a car, since if it bungs up, you can get it fixed, a hassle but nothing more. That’s not the case with a data centre power supply, you need it *now* and the batteries won’t keep things going for long enough to get it cleaned out.
Security
I didn’t mention the site above because there is increasing concern about hacktivists, DDoS attacks et al, even though so far this has amounted to little more than defacing the occasional website which is further away from critical systems than the coffee machines that fuel their IT pros.
In summary, I can’t tell you what will blow up next, but it is a system under strain with regulators driven by politicians whose grasp of any IT rarely reaches the giddy heights of owning an iPad - and where some serious money is handled by systems that would be scoffed at by a midsized supermarket chain. ®
Dominic Connor is a City headhunter, former CIO and before that a grunt programmer at various banks. You can connect to him here.