just why the financial industry keeps asking for financial experience in their development recruitment - perhaps because some of us might start asking the wrong questions!

He should have blamed it on the Excel bug

The human psyche is not well developed for "eternal vigilance", and this defect is propagated into systems and processes we build and run.

Derivatives traders and quants seek and exploit "market inefficiencies", so you have to expect them to game their own systems the same way.

I really don't understand why banks run multi-billion dollar businesses on Excel spreadsheets. How the hell do you do unit, integration and regression testing on a spreadsheet? This must represent the utter failure of IT to address a dynamic set of user requirements in a controlled development process.

But from what I've read, IT people in banks are near the bottom of the food chain. I'm always perplexed to hear fellow (non-banking) programmers wistfully speculate about how great it would be to work in banking.

If the geeks in these firms were allowed to throw their toys around as much as the traders, there might be more balanced (and less fuck-up-prone) working environment.

John

>I really don't understand why banks run multi-billion dollar businesses on Excel spreadsheets

They don't.

They run multi-*billion* $ businesses on Excel :)

Excel is a good tactical development environment, can talk to financial data sources, and produces pretty graphs. As I mention there are tools like Xenomorph which help you make spreadsheet apps industrial strength, but most banks don't use them much.

>I'm always perplexed to hear fellow (non-banking) programmers wistfully >speculate about how great it would be to work in banking.

I've worked in non banking. It was shit. (computer manufacturers, circuit board maker, hotel IT, and even publishing)

IT people are the bottom of the food chain pretty much everywhere, indeed with outsourcing many have been moved out of the chain altogether. You've seen IT departments outside banking right ? Have you ever seen an IT dept. that wasn't in the worst bit of the building ? Often underground ?

In banking it's just as bad, but you get paid more, as a pimp in this area, that average a factor of two. If you're outrageously good at C++, let me know...

John, banks run 'multi-billion dollar businesses on Excel spreadshets', precisely because the geeks are *not* allowed to throw their toys around. IT would love to move everyone off cobbled-together spreadsheets and onto a enterprise-level trading & risk system, but never get the budget or resources.

Until it's too late of course...

Paris just because...

Not a solution. There's always a "enterprise-level trading & risk system" around the corner. It's generally due to replace whatever tactical system I'm working on in "six months".

Generally it never does as the hordes of overpaid BAs specced out the system two years previously and treat any attempt to update it to the actual current needs of the business as "scope-creep". Then there's the ovepromoted System Architects producing PowerPoint to impress the higher-ups with plans of how their system can do everything you could want in the bank, when in reality it wont do half of what it need to do for the users.

Until senior IT management give up on their pipe-dreams of eliminating EUC and Tactial/RAD dev and accept that it is better to support and understand it the harass and block, this ludicrous situation will go on.

where's the IT angle ? come on we're speaking of *banks* here, ginormous infra powered by a start up mentality...

i work as a UNIX system engineer (contractor) at a large competitor of SocGen, and what surprise me is that our shitty infrastructure, absent management makes *billions*

i work nearly 60h/w (yeah, some french works, but then i am more confortable with my english coworkers than the french ones) for what ? mending stupid holes, hacking kilometers of *ugly* scripts (seems like every single guy who work/worked here has *his* langage of choice, that *no one else* use. hell, a dev has scripts in *brainfuck*)

SocGen is known for having a not-too-stupid management of its IT assets. i wonder how much we would lose ? (yeah our risk management app, a few thousand blades, ginormous databases, and.... rolling a 10% of its potential because of crap algorithms. and a blade a day to change...)

ah au fait. i am mainly paid of coffee and slaps behind the head.

dominic, whenever you want :)

... for an informative and well-written article. I really appreciate the insight into this situation from someone who has good insight into the industry in question.

Too bloody right it could. I have a lot of experience in banks at a lower pay grade than Dominic but have seen all of the same issues. This could happen at any investment bank in the UK or globally (same banks after all).

Poor controls, tick in the box auditing, password sharing as standard, billions of dollars managed from excel, middle managers with all the nous of piece of blutac, it's all true.

I tell to people that if you knew what I knew you'd keep your money in a sock under your mattress. I almost mean it.

who's to say that Power will not Flower in IT2...4FlowurPower2 and ITs BasiCurrent .... Life's Original Source CodeXXXX for the Generation of Generations with Virgin Imagination to Nurture and Feed/Damage with Greed.

"Often underground ?"

That would be for those IT Boffins with their Trigger fingers on the NEUKlearer Key Pad, Dominic, pushing all the Right buttons to make the SunShine. Best to keep Pimping them their Pleasure, lest they turn out the Lights.

"IT people are the bottom of the food chain pretty much everywhere,.." ... a popular misperception that they wouldn't be bothered to correct, given that that which they do, can so fundamentally [for Good] and so catastrophically [for Bad] effect everything that they can do ..... for you.

Without their Input, where would you be? Where would any of us be? Probably still daubing ochre on cave walls.

"embed usernames and passwords into applications, especially Excel report sheets".

Is this looney, or just plain DUMB. It seems to me that the people ABOVE IT in the banks food chain just want to see nice clean spreadsheets on paper. Never mind what the numbers are, just print them out. Oh, the numbers say "LOSS", change it to say "PROFIT". Print and show to the next person up the chain.

Won't anybody learn? (Somehow I doubt it!)

I've worked in at least 15 different industries, and the Australian banks I worked at were by a considerable measure the most backwards shops I'd been in. I've found a system which controlled two billion dollars of superannuation funds for the bank's own employees which stored data in a btrieve database that was located on a wide-open share. As a consultant I wrote a scathing report recommending that the person responsible be shot and that a full audit be performed. In the name of "client relations" I had to soften the report to "mistakes were made."

In another bank I was responsible for reviewing the architecture of a new ForEx system being developed by a company whose owner was a crony of the ForEx director. Anticipating a scenario like SocGen's, it was all I could do to get this vendor to modify the system so that it would be at least marginally difficult to game. No way could I get the ForEx group to back me in forcing the vendor to make the system geniuinely secure...

Unfortunately, the problem is exacerbated by the IT departments... they're so unresponsive that the business units take carte blanche in building whatever the hell skunkworks/cowboy system they need, including the previously mentioned spreadsheets from hell.

Interesting ,but alas we are only being shown a very tiny fragment of a bigger picture in order to hide other losses incurred by the bank in question in regard to the scandalous overboard exposure to the US Mortgage Sub Prime failures !

One can see the fix is already being written with the latest round of news of the number of charges being laid and how as well !

Onwards to 2012 , we come like lambs to the slaughter !

In the old days the money in the till and the cashbook had to balance.

Then the banks started to get lazy, realising ot cost money to pay people to count pennies......so they stopped....yup even in your local branch the cash in the till and the money in the computer cashbook don't have to balance.....slackness and false economy is endemic in the banking and financial sector from top to bottom.

The money isn't real money, it's little specks of differently charged dust.... a fact that financials seem to have forgotten. Most financials aren't really banks in any original form...they're really huge IT systems cobbled together from lava flows and stovepipes.

Bankers don't realise this.....as evidenced by their attempts to save money by outsourcing. Their entire business is manipulating the little 1's and 0's .....anyone but a suited banker or accountant would think it madness to outsource that to anyone someone nice and cheap.

high level bankers grew up in a world where people worked in banks all their lives and it was a well paid and highly respected profession.....this is no longer the case, but the basic assumption that once you are a bank employee you are "honourable" is the basis for most security cost/benefit decisions at assistant director level and above.

As the Houses of Parliament are finding out blithely assumng everyone in the place is "honourable" and refusing to put in checks and balances on that basis leads eventually to embarassment and ridicule.

I worked in one august financial institution who's IT department did not contain a single IT graduate. Every member of the (several 100 strong)department had joined from school or College/University (usually Arts or languages) and gone through a 6 week IT and COBOL course. Unsurprisingly the rare outsiders they hired found it difficult to remain cool and calm when dealing with long-service co-workers who considered themselves IT gurus, but had never worked outside that building.

Developer security inside every financial I worked in was a joke, and who knows what the hell was happening in the Indian outsourcing shop.

But t was of course cheaper...and everyone else was doing it so it must be OK.......

No so cheap when you lose several billion overnight.

k

His managers knew. He was in the black on these positions at the end of '07. They let him do it because they liked the results.

He also didn't lose as much as they are blaming on him. This is to cover other losses (backdating losses in '08 to '07 numbers is REALLY weird). heystoopid is right. This is a front, and it will unravel over the next month.

Stay tuned...

If these people can collectively lose billions, they can't be very clever.

Paris and I are smarter!

Yes, I could.

I worked for a Major Bank developing a POS system, and they sent us a data file to test our code. A file containing thousands (as in 000's) of live, true-to-life, accounts. The file contained account numbers, balances, date last accessed- everything. The lot of us drooled at the large balances and speculated on whether an account last accessed 10 years previously would be accessed shortly after we withdrew tremendous sums from it.

I have no knowledge of my co-workers who may have departed the company for tropical climes, but out of cowardice I never broke faith with my masters. And I am still employed. Brilliant.

Bankers Infighting ....... Handbags at Dawn.

Do El Regers think the System is in Meltdown and/or being Milked. A Double Whammy of Despair and Disrepair? Money never disappears, it only gets deposited elsewhere and it can just as easily identify you as a Dumb Bandit as a SMART Cookie ........ although it would still be in the System so that would suggest a number of Systems available to Bankers/Launderers?

Whose been passing around that BCCI Dope then. Come on, own up, come clean. The stench is overpowering and a real giveaway.

I've worked in banks that have tried to implement "developer security", and the effects were hilarious. Well at least they were a few months later in the pub.

Outsourcing is no worse or better than other business activities, but the desire to save money means in most cases a decrease in the nominal size of the IT cost centre by imposing costs on other areas. It is insane to save £3 per hour to have a chepaer person when someone whose full up costs is 50 is left without the technology their work is dependant upon.

Also there is an inherent problem that most IT people do try to keep costs down and do a good job. But once you join an outsourcer job pay is based upon keeping the client cost up.

Also staff turnover in outsourcers is amazingly high, so the staff don't identify with their employer, much less the client.

Hope you will forgive me, but it's difficult for a vendor not to avoid this opportunity for a bit of self-promotion. For those readers who are interested in controlling native Excel then please consider the reports below.

IDC subscribers see "Spreadsheets Power Undocumented Applications: ClusterSeven enforces Control by Kathleen Wilhide. Ref: 209683

Gartner subscribers see "Bank Controls Runaway Use of Microsoft Excel, Improves Risk Management" by David Furlonger and Jay Heiser. Ref: G00147957

Ralph Baxter, www.clusterseven.com

I have retired now, but when I owned an IT business 3 years ago, we banked with NAB. Standing around one day in the perpetual queue to pay some money in, I noticed what I think was an IBM 3270 PC (The cabling looked a lot beefier than I remembered from a bog-standard XT).

My first thought was "Stingey bastards - Another record profit year and they still have this in use.". Particularly as this bank seems to have the rudest staff available - I had been banking there for 10 years, the account was always in the black, and the staff would sneer if you tried to withdraw your own money.

After careful reflection, I decided that this was not too bad. Nobody under the age of 35 would have the faintest idea on how to get into their systems.

The post-script was when I closed the account a few months later - The front office had been revamped and flat screen PCs were everywhere. When I enquired about closing the account, I was told that I had to leave a couple of thousand in it for 10 days to allow uncashed cheques that 'might' be in the system to appear. The staff seemed surprised when I told them that a cheque could be presented up to 6 months later, with the expectation of being cashed, so perhaps I should keep the money in for a few months "just in case". We sorted it out in the end when I told the cashier that I would close the account later. Later, was after I had withdrawn all but a hundred dollars in cash. Sigh.

I'm not sure I want to outlaw gambling but I am fed-up with the prices in the shops and my savings being effected by a bunch of ill-educated, responsibility shy, rich kids.

If they want to play, give them there own space but we need to insulate the real world from all of this nonsense.

To paraphrase Yes Minister, there are two kinds of potential borrowers, those who can pay their bills and those who can't.

Now, I feel better!

Nice article and the bit about Excel certainly matches with my experience in banks. Of course you also get the "my report doesn't work due to permissions problems.", to which I reply what permissions do you think you need?" to which I get "it's needed by the board NOW, so give me admin rights". When I protest I just get over-ruled............

Again, I have also read this with interest. Applying controls to spreadsheets isn't hard. Ensuring who has access, what they are able to do, and keeping a record of all of this is, I think, what any business whould do. Controls like these can be horrendously expensive, I know some systems are in the £100k plus bracket, but that is a choice banks or organizations make. For the more sensible, there are systems, such as ComplyXL which work out at around £6k. A guess, as they say, the choice is yours ... http://www.lyquidity.com

The users want something now, if not sooner.

The IT department has been bogged down in red tape that was created to try to prevent IT foul-ups, and at the same time has had it's staffing levels trimmed by accountants.

Result - the users get sick of waiting and hack together something in Excel or Access. These systems are usualy badly designed and a nightmare to maintain.

At some point management discovers that a major function of the bank is being run on one or more of these systems (normaly they find out when something goes wrong), so the systems are taken off of the users and given to IT to make work properly (thus reducing the amount of time available for writing new stuff). At the same time more rules are created for the IT department to prevent systems like this being written.

Repeat.

> but I am fed-up with the prices in the shops and my savings being effected by a >bunch of ill-educated, responsibility shy, rich kids.

Nearly all of my people have a masters on top of a degree in a real subject (physics, maths, engineering even some CS). A large % have PhDs, a small, but not trivial % have two PhDs.

Almost none inherited any wealth, certainly their current position is in no way inherited, which is rather different to politicians (Clintons, Kennedys, Bushes, even Benns)

>If they want to play, give them there own space but we need to insulate the real >world from all of this nonsense.

The allocation of resources is very real, and I trust them more. Middle class white kids like Peter Hain who committed the level of dishonesty he has demonstrated would disappear from a bank within femtoseconds of their discovery. Certainly wouldn't hang on for weeks.

And I emphasise the "white kid" part about Peter Hain, the most common name on my database is Mohammed. The current Labour cabinet has the same number of black or Moslem blokes on it as the executive council of the British National Party.

ie zero.

Yeah he did good stuff when he was young, but like many of his political class does not believe the law applies to him.

Am I alone in not seeing any gloating from those who "won" the $7,200,000,000?

If my take on the market impact is right then it will have mostly ended up in the pockets of the algorithmic traders.

For example, even a light-weight IT security certification like the CISSP advises mandatory vacation for audits. Permission handling obviously must be done with individual passwords, that expire or are actively extended only if approved again. Several other things. Quite frankly, if a student of IT sec at my university makes the mistakes these people likely made (and I think your speculation is entirely plausible), he or she will fail the exam because of gross incompetence.

Seems to me your run-of-the-mill foodstore has better security than these people. It should be criminally neglient to operate a business like this with amateur-level security.

Pretty much every single flaw quoted in the article is present in the power industry. Sharing passwords? Check. Massive purchases and sales of power based on cobbled-together spreadsheets? Check. Critical control systems left wide open so as to not inconvenience (L)users? Check. Auditors who have no clue what they're supposed to be reviewing? Check.

It frelling amazes me every day that the US power grid stays up. It's a good thing that terrorists are dumb as dirt.

Good article, want to see more like it.

I was just commiserating with a buddy who manages part of a financial company with software that is literally older than the decade old institution itself. It's a tough sell to go to pretty much anything that isn't already fully depreciated, because it kills your ROA. Excel spreadsheet reporting is actually a step up for some financial companies...

Gewhir could you tell me how you teach your students to deal with senior staff telling them to break elementary security ?

I assume you teach social engineering, but what about advocacy skills ?

Enjoyed this article, and it's good to see the author responding in comments too. I've been in banking quite a while and I agree, this could happen pretty much anywhere. What is surprising is that SocGen did not act on warnings from Eurex in Oct/Nov. Seems as though their award from Risk magazine went to their heads!

And I'd agree with The Pimp in stating that Investment Banking is full of very clever, very talented and very hard working people. It's just that some people use those talents in rather odd ways. And that the business demands simply don't tally with development reality.

I doubt JK used VBA to hide his positions. More likely he entered false offsetting futures/forwards/swaps trades and/or made risk amendments (in my experience, risk systems allow you to manually "correct" your risk, which is intended for use where a position isn't feeding properly or a trade hasn't settled yet) to make his risk look overall flat. D1 traders' gross asset limits are often v.high (i.e. measured in the billions) - it's the delta (i.e. the sum of the net long/short positions) that's under far tighter limits - hence the name for this type of business, Delta One, which implies no risk.

To give an example, if I'm long £1bn worth of FTSE 100 stocks and short £995m worth of FTSE 100 futures, my gross asset value is £1.995bn but my net position is long £5m.

Rumour has it JK had taken large unauthorised index futures positions. To make himself look flat, risk-wise, all he had to do was make false entries in the opposite direction.

In my experience, Excel/VBA is more used for tactical risk modelling/calculation by traders/quants. The real risk management (i.e. what the controllers look at) are separate systems (either developed in-house or bought from a third-party vendor) which take dumps from the various position-keeping systems at the end of the day to compare against traders'/desks' risk limits and do the number-crunching necessary to calculate the company's overall VaR figure.

It's not at all surprising that a relatively intelligent individual who was both familiar with the risk systems and was determined to circumvent them, was able to do so.

-- The Accidental Trader

Certainly Excel misuse is only part of the problem. Given his skillset it would have been in the mix, but you must be right that he used other tools. Indeed I think the access others PCs "helping" them with Excel may have been far more of a factor.

I hear what you say about entering the offsetting positions, but why weren't any cash flows noticed ? Certainly I stick to the point that either the reporting at SG is totally crap, or that he compromised it (or some combination of those two factors).

Agree'd there are some *really* big players in the power industry that have appaling procedures for certain aspects and houses built on sand. The previous owner was no different and that one nearly went the same way as enron. probably the only reason it didnt was because it was just not found out.

i work at the station level and here we run a very tight ship but at the corporate level security is a joke, particularily within trading as they seem to think they are 'god' and tend to get away with whatever. the main trainin gpc's have postit notes with passwords on them stuck to the sides of monitors. we castrate people for doing that here but down there it's the norm to share and logon as others!