Original URL: http://www.theregister.co.uk/2013/06/12/lehmans_it_implosion/

What did the Lehman Brothers implosion look like to a techie?

Insider tells all about the Gnab Gib at Lehmans

By Damon Hart-Davis

Posted in Business, 12th June 2013 08:03 GMT

I was at Lehman Brothers (AKA "Lehman's", or "The Brothers", spoken in a slightly menacing US-Italian accent) from January 1996 to June 2008, bar a three year gap in the middle for dotcom japes. I was an IT contractor/consultant with several groups in the Fixed Income division, including Credit, which dealt with the infamous CDOs and their ilk.

LB was one of the best and most meritocratic organisations that I have worked with. Its internal IT and pricing systems were also about the best that I have worked with in finance: I wouldn't trust my deposit account to some of the other rubbish that I have worked with in the City.

Lehman's also managed to get by with relatively little "office politics" – at least, to start with. As we started to make our move up from something like number 10 to number 4 in size, and became more global with it, some of that camaraderie soured.

Lehman's had all the expected long hours, drinking, high jinks and shouting that one might imagine, but very little really bad behaviour – that I observed, anyway. There were significant moments of expensive carelessness.

I saw $1m lost on a single trade where the quant - mathematician/trader - insisted on writing their own code, not discovered until I came to optimise their messy C++: keep the scope of your variables as narrow as possible, folks! And there were some sociopathic incidents, and individuals, too.

I also got to see rather more of New York and Tokyo than I might otherwise have, under LB's wing. I had a lot of fun and I think that I made the world a slightly better place each working day.

Lehman's, though a buttoned-up red-toothed sales-driven organisation, could be cerebral and thoughtful and generous. It took some unsung far-sighted actions, such as its research on "The Business of Climate Change" (which you can read here) and charging back energy costs of their honking computer systems to each desk to provide feedback on footprint to people who spent all day tweaking numbers to improve their bottom line.

LB could also be pretty good at paying more than lip-service to diversity, from my observations, and cared about outcomes over appearances – except in what you were allowed to wear on dress-down Fridays. I hope that LB's surviving peers continue some of that good work.

Order of Merit

The interesting thing about a meritocracy, which City institutions seem to be to a reasonable degree, is that your official status doesn't matter if you're right about something important and someone nominally higher up the food chain is getting in the way. Time and again I often had the higher-up removed even though I was a mere no-status contractor.

Lehman Brothers London HQ

Lehman Brothers HQ, London

Lehman's survived a couple of actual disasters (the Russian bond default [1] and 9/11 spring to mind) because, on the whole, things seemed to be reasonably well managed. And we had luck on our side. For example, at 9/11 most of the systems that I helped to write and build (around 300k lines of cross-platform C++ code from the parallelisable build system up through to very sophisticated curve generation) were designed to be able to run anywhere in the world and efficiently too.

For example, we used replicated local libraries supported by a massively distributed NFS filesystem. This meant London could take up much of the slack immediately when NYC was completely incapacitated. Obvious tech now; less so then, I believe. NYC was able to bring up trading within 48 hours even with HQ wrecked.

LB also survived a lot of apparent malicious talking down of our stock. I think we all just got used to ignoring anything the markets said about us as likely contaminated with rumour-mongering by speculators shorting our stock or even competitors out to damage us. Of course, it turned out that we shouldn’t have ignored all of the bad talk.

Comedy of errors

Not all of LB's 9/11 response was glorious. With beams smashed through the windows of its powerless down-town data centre, and not enough tapes nor other materiel to do the job “right”, people were despatched to the wrecked centre with backpacks to physically remove disc drives from the stricken servers to recover data elsewhere, such as LB offices at 101 Hudson over the river in New Jersey.

Apparently the stench of the trading positions wasn't at the front of the disc rescuers’ minds, as the increasingly backed-up and unflushable toilets throughout the building began to add their own “data” stream...

Still, 9/11 cut the Gordian Knot of the glacially slow data centre rewiring project, rather in the same way that the IRA, some years earlier, sharply accelerated the wind-down of Nomura's London-Docklands-based mainframe, as its building's windows were blown out. This led to overheating and a rapid transfer to various Unix-based systems. Disasters can have small silver linings.

Why 2008?

So why did Lehman's not survive September 2008? Partially bad luck, with possibly an element of LB being picked to be the fall guy: it was probably reasonable to let one of the banks fail as a warning against moral hazard, but LB seems to have been run much more conservatively in some regards than the also-imploding UK banks, from what I can see, with a fraction of (say) RBS' leverages by some estimates.

One can argue that Lehman's death was as much a victim of plain old US consumer mortgage fraud as its own overstretch. Many would disagree with me – including Lehman's own mortgage-origination desk traders, who had been increasingly complaining to management about the increasing stench wafting from their books...

Were there signs from inside that things were going wrong that most could see?

The Times on the day Lehman Brothers went under

The Times, on the day that Lehman went under

For me? No, sadly. I am no Mystic Meg, and would have bought Lehman stock right up to my last day there but never did through idleness*, and I didn't see doom coming.

Though clearly towards the very end, at the start of 2008 money was exceptionally tight, even for a bank that had always professed to clamp down on NPE (Non-Payroll Expense) as I was.

In fact it was so bad at the end, and my boss was spending so much effort trying to renew my contract and not getting his work done, that I said that we should just let it lapse so that he could get back to something productive.

Thankfully, the timing of my relatively graceful exit meant that my last invoices got paid in full – unlike for many others. LB went bust owing me a grand total of 1p on my canteen card. I still received a form from the US inviting me to file a proof of claim!

From what I can tell, most people fairly high up didn't see the trouble coming, so I was not alone. Maybe a very small group near the top, on a "need to know" basis, and a few more exposed traders, had a clearer picture.

But backing up a little, were there any other signs?

GNAB GIB: where vision gets bilked

Snazzy project acronyms seemed to have become more important than content a couple of years before the gnab gib. Big organisations always do stupid things, but looking back, this should have been a really big clue since LB was normally, somehow, quite good at concentrating on substance.

Worthy but dull projects were just not happening, such as replacing ageing but fairly key bits of infrastructure like the global build system that I had been working on almost since day 1.

I have to say that I found the corporate strapline "Where Vision Gets Built" pretty crap too, but I can't remember exactly when it was promulgated relative to the general slide towards insanity and dissolution.

In my own corner, I was finding it increasingly hard to get satisfactory, meaningful work done and was talking myself in circles trying to resolve contradictions in “management”. Having spent many years in one group turning out solid work, I found myself shuttling from group to group and then off the trading floor entirely, to a morgue-like (and slowly emptying) development floor.

I'd said for a long time that if got pushed away from the buzz and excitement of the trading floor it would signal the end of my time in the City, though I did not imagine that it would be LB quitting on me rather than the reverse!

So with hindsight the place did feel as if it was becoming unhinged, closing in, getting harder to do good work done in, or get anything signed off. Some of the better people were drifting away, leaving behind also-rans such as yours truly.

Others reported that the dealing floor was unusually febrile on certain market events beyond the (usually rather fun) adrenalin rush of the US "non-farm payroll" announcements. In particular, whenever the US Fed moved rates outwith the normal cycle, things would go mad.

After Bear Stearns imploded in March, LB was pretty quiet until its next quarter's earnings emerged, which provided some respite and a big jump in stock price. But in the week before 15th September the prop traders (“proprietary” traders who try to make money with the bank's money, rather just the ebb and flow of clients' business) simply couldn't do anything or trade with anyone. Maybe LB was going to be bought by Bank of America or otherwise survive, but no counterparty was going to take that risk.

After the fall

There was some particularly grim comedy in events and behaviour as things started to fall apart: LB Europe arguably being sacrificed by head office in New York on the morning of 15th September, not "sweeping" back several billion quid sent to them out of hours for management in the usual way. NYC staff turned their back on their "brothers" in London thereafter. There was also bizarre stuff about who grabbed which elements of the computer systems for example, and suppliers found they suddenly had two contracts with "warring" factions.

Lehman mug and hat left on the ground

Memories

All mergers are horrible; forced mergers of the kind the corpse of LB was experiencing even more so. People I knew who were assimilated into Barclays Capital/BarCap reported that part of the process as being especially unpleasant.

I didn't hear nearly so many complaints from the Nomura side, even though I had a little more insight, having worked at Nomura before Lehman's. That wasn't roses and unicorns either, and although it initially resulted in some rapid promotions, lots of senior people have now moved on.

Broken back

There were times when the “backbone” (ie network) connecting the various tentacles of LB looked likely to be cut. The techies were primed to do it at least once in the first few days, from what I heard. LB Europe noticed that LB New York was sucking out huge quantities of data westward across the Atlantic, which might have been of strategic value to Europe as a bargaining chip later on.

I managed to pop into the living-dead LB building in Bank Street to catch up with friends still in there, see my old desk (with the machine still off and as I'd left it months before, as I recall), have lunch in the canteen and generally revel in some gallows humour.

It was very odd for me, subsequently on contract to BarCap, to see parts of my Lehman's handiwork in use there, and to realise that other fragments might live on at Nomura and elsewhere.

I regularly walk past 25 Bank Street on my way to other work in Canary Wharf, and the building (and even some of the security staff) is superficially almost unchanged. Someone now working in there pointed out to me that before Lehman's the building had been earmarked for Enron: clearly not a workplace for the superstitious! ®

Bootnote:

* Logically you should short the stock/shared of your employer or client to reduce your exposure to them, but it's considered rude. One of my client's permanent and contract staff did it anyway, shockingly.

References:

[1] Russian bond default:

http://www.nytimes.com/1998/09/25/business/the-holders-of-russian-debt-begin-blocking-overseas-assets.html

http://news.bbc.co.uk/1/hi/business/179439.stm