He's got the all the resources of the city of San Francisco directed at getting into the network he was responsible for securing and they can't manage it.

And the network still appears to be running OK.

If he's denying changing the passwords, what's the be that he's just defaulted them. I wonder if they've tried Admin and Admin?

You would have thought someone in San Francisco would know how to reset the admin password on a cisco router. Ten minute job with a terminal and a reboot. Even Paris could do it with a little help from Darva Conger:

http://routergod.com/darvaconger/

Is he their only sysadmin? What are the other sysadmins doing about it?

They have physical access to the machines and can't get in? Fire them all.

he deleted/disabled default admin account ( good practice) and set up a new one and when he was suspended some tw@ disabled or deleted his account to prevent him gaining acces and effectively blocked everyone. The password he gave wouldn't work because the account no longer exists. Not guilty m'lud.

Good point.

To the SF council: I'll solve your network woes for $5Million. So you'll get it back if he's out on bail.

than standard procedure to get into a CISCO router.

A network to me is all the individual hosts in the network including the routers and switches.

If the term network here is being used to refer to only the router, then they have to only be worrying about the router configuration (odd there is no backup).

I am guessing it is the admin control over the entire system (where system is not one host but the collection of hosts), it has to go deeper than just one or a few routers. If it doesn't then whoo this is day 3.

Physical access is not game over as far as security is concerned, if the systems are running off an encrypted backing store, then that would still need to be defeated, of course they could get the liquidN and try to hotswap the memory :)

And hey San Fran has got the tech community on its doorstep, why haven't the simple solutions worked yet, there has to be some reason.

Their thinking could be, as long as the system is working, then they will just take the more cautious approach of doing nothing, once it requires admin access then they will probably start throwing the solutions at it. That is a possible scenario, but of course they don't know for sure everything is ok apart from the access.

And he is claiming innocence, the access codes given could have been genuine as far as he knew it. And it could just be coincidence, some cybercriminal just hijack'd his account, that could explain the monitoring of the other admins. You are not going to gain too much monitoring your colleagues, much simpler to chat to them, and unlikely they will slag you off in an internal email, they will use the water cooler for that. But, they will email about technical mechanisms in the company, something he probably would have already been privy to but a cracker wouldn't, and a cracker would want that info.

And here is another idea, the password he gave may have only been correct for that time period, therefore the access code was valid for say 5 minutes but not after that.

Something really doesn't stack up here, 3 three days is too long not to have regained control, or at least regained control of key elements to the system.

Once the city started throwing its weight around, its become more difficult for them to back down. Jail and $5m seems over the top for a question of ego, but then I'm not American. Just imagine the fun if he told them the password was say.

"Adm1n" and they wrote it down and tried to use "admin", my what red faces they would have, my they could be sued for lots of wonga, and so the cover-up begins.

Still a defence would be, I gave you the right passwords, now prove that you actually typed them in correctly. I have noticed that panic stricken Sysadmins tend not to log everything they do in their haste to get the system to do what they want it too.

Also don't all network devices have a hard reset switch that lets you put them back to factory settings, which naturally destroys the configured setup and any evidence that they might contain.

A final point is that the devices could have been configured to use LDAP, so there would be one central database with a super admin password, which is how it should be set up. That password should be written down and locked in a safe accessible by the head of security and not used for day to day access and only used when your sysadmin gets run over by a bus.

Personally I think this over reaction is SF making up for the fact that it has been incompetent in its own management.

"very good at what he did", then perhaps they shouldn't have suspended him.

I usually find that "run-ins" with "superiors" are actually cases of "telling it like it is" to "overpaid morons".

If these people are so "superior", I suggest they fix their network themselves.

a PFY's wages that, in the rush to gain control back, some SysAdmin opened the network with the password he gave and let some script kiddies in....

Either that or the SF Mayor will be getting an email from some Russian guy saying:

"All you're passwords are belonging to us! You give 100,000 of you're American dollars to us. We give you good working password. p.s. you want to buy the Viagra?"

If his superiors are so dam superior, then why is it that he still knows the password, and they do not. Who's superior now??

and now they're surprised when he not only hacked their system but seems to have made it hacker-proof.

router1(config)#no service password-recovery

WARNING:

Executing this command will disable password recovery mechanism.

Do not execute this command without another plan for

password recovery.

rommon security is the same as locking the door and throwing away the key to a device. Without the access password, there is only one way to get into the router -- return the device to Cisco to reflash the IOS.

Why doesn't SF just get a copy of LoPh7CR4ck and use Brute Force?

"I_cannot_answer_that_question"?

SF city officials have officially ordered that all servers are to be replaced with Microsoft servers after this debacle is over. By doing so they will never be locked out of their system again.

A few years back we were looking at buyng a supplier company and I was on the team that got to do the "review of their personnel, systems and resources" AKA "play God with people's jobs". Their head admin was a real BOFH and had seen the issue coming from a long way out, and he'd basically made himself fireproof by ensuring the company had signed up to a security policy that meant he effectively controlled everything. Virtually nothing about the company's systems were documented, it was all in his head. He was quite calm and open about it all, and seeing as he seemed to be the only one who actually knew how the company systems worked, he had his directors over a barrel. As part of the risk appraisal, I wrote something along the lines of "Mr X is your number one risk - if Mr X should leave, be removed, or gets hit by a bus, the company will continue to operate for a period but without control of the systems". I got a ticking off for not using a more serious approach to an appraisal, but two weeks later, Mr X actually did get hit by a bus! My then boss did have a sense of humour and pasted a picture of Mystic Meg over my desk.

This guy will end up as a high priced security consultant; after a public flogging of course.

I think you have good theory. It would be a classic if they disabled his access centrally when they suspended him. Logically they'd have done it just before!

I recall confusing some people when I altered a system so you did not login as root to do normal daily monitoring, and lots of stuff ran as "admin" rather than root. It made the production server a little tougher against finger trouble and made you think about using root's special powers. It was really alien to people. So if he removed the standard account they could be really locked out.

...and then there will be cake.

Not entirely relevant, I just felt like saying it.

I have one, 10 mins should suffice to retrieve the correct password assuming the account hasnt been deleted, GItmo his ass!

Much better than all of the posts from the 'master security consultants' who know exactly how to get into the SF system.

<pedant alert>

... unless you're suggesting he turned their network into a conservatory.

</>

My coat, the one with (n) after it.

Visit the computer club at the local high school. Offer $50 and a copy of Playboy to the first one to crack the password.

Ten minutes. Job done.

If recovering access to the system was as easy as some people here seem to think, I'm pretty sure they would have done it by now, if only to avoid the embarrassment. So it seems he has truly managed to secure the network that was under his control. He'll probably serve time for tell overpaid idiots to go fuck themselves, but I'm guessing he'll have a job when he gets out, if not before.

sry. couldn't resist.

Has anyone thought that maybe they are too scared to break into their own network as many of the ways of resetting a password essentially involve resetting routers and switches or reflashing them which trash the running configs.

If the sysadmin was the only guy who actually knew how everything was configured and had made a few changes recently which weren't backed up etc. they might be trying to work out how it all hangs together prior to breaking back into their own network....

That'd do it.

Shirely they'd have a backup copy of the router configs somewhere?

No?

Oh bugger!

(kind of explains why it took cisco 3 days to re-configure the network)

I hear that Cisco and other experts are all over this thing, days later, still trying to hack their way back in. Give this guy credit for securing his systems so well!

No matter what he did, it is stupid if they cannot make it work unless he tells them how to. What about if he had a heart attack?

Paris because... well, it's in the title

To make use of "physical access" to crack into a system usually means a reboot to some kind of standalone recovery OS. I suspect they're afraid to reboot-- for one, they'd probably have to pull the plug on things to do so, and things that are currently successfully running.

The guy is no doubt holding out until they become desperate enough to let him off the hook for it and possibly is even dreaming of being reinstated and with an increase in salary... But he's delusional-- we know governments really do not like to negotiate with terrorists, data or otherwise.

Clearly though, the admin has little confidence in his own ability if he thinks he has to resort to such antics in order to keep a job. Methinks such positions ought to be subject to the same sort of psychological testing that the GIs sitting on the launch buttons in missile silos do-- it's not a good idea to allow unstable personalities to hold such critical job positions-- someone can "go postal" with your data with far less resistance from a conscience than using an AK47 on his office mates...

If the evidence against you can't be accessed without your consent, would you be wise to give that consent? By refusing to disclose a password, aren't you effectively pleading the 5th (amendment)? There's also the matter of plausible deniability, "my password used to work, but someone's hacked it", etc., etc. Especially when there's no recoverable evidence to show otherwise.

Bad input, bad output !

In other words, the press release don't give enough informations about the problem for you to propose a logical solution.

Let's wait the end of the story to start to comment on facts and not on suppositions ?

"Both of them"

It is after all the punchline to the only joke that's ever been written about San Franciscans.

Exactly!

"You have the right to remain silent. Anything you say CAN and WILL be used against you in the court of law."

It is a requirement by US police that these are the very first words spoken to you when arrested. If he were to give information that was either used incorrectly but was interpreted as malicious due to the shakedown staff, then he is in even more trouble. He gave them the first password, correct or incorrect - it didn't work, and now his lawyer is probably telling him to keep his mouth shut so he doesn't get in any more trouble.

So many good insights and comments for this one on El Reg. I'd like to see Mr. Childs give an exclusive interview to this fine publication once his ordeal is over!

Heard Joe Fay on R4 yesterday. Is that a first?

Now, if only someone would do this to a (preferably US) military network...

The password is <drumroll> "I'm_not_telling" !

No need to thank me, just donate any reward monies to my favorite charity; Hookers For DaFt.

" but I'm guessing he'll have a job when he gets out, if not before."

Would you trust this guy with anything?

Real justice would be for the jailers to find themselves unable to let Childs out of his cell because they'd misplaced the key.

The local articles are sort of vague. It sounds like database admin account passwords are really what was changed.

It works and its secure, no wonder he locked it! It sounds obvious that everyone else there is a fool and I wouldn’t want them making changes to my systems either.

This fate could be waiting for anyone who annoys the people in power. You'll be hauled in, your computer taken away for forensic analysis. A file will be 'discovered' (even if it's random deleted sectors) and you'll be required to provide the password. Then you get locked up for failing to provide the password even though it never existed.

ebbg and ebbg

ROT13. when was the last you used it ?

or from above

I'm_not_telling

is

V'z_abg_gryyvat

http://www.theregister.co.uk/2007/04/17/chocolate_password_survey/

Hey, I wanted to offer the perfect solution from a safe distance too!

DEViANCE or RELOADED.....

heheeheheh

Drop the charges, give him immunity from legal action for this alleged offence, take him on as a one-time contractor for a ludicrous amount of money (that idiotic $5m bail should do), get him to open it, change the password, and give it to the new Admin. He / she can then change it to something else, and all is well.

You get a BULLETPROOF system (as proven by your many days of attempting to fix it), and he gets recognition for building a system the suppliers couldn't even get into without reflashing appliances and rebuilding your entire network infrastructure from scratch..

If I was you, though, i'd take him back full time on double pay, no hard feelings; The guy is OBVIOUSLY not slacking on the job. If he was, it's because he's done his job to the best of his ability, and that ability seems to be better than anyone elses. Get some humility, FFS.

If he wants to take the hard road, keep the passwd secret and screw SF city for fun, I'm already enjoying it..

After all, Sysadmins have above average IQ's, I trust he was probably stiffed by some corporate w4nk3r and took revenge - All BOFH wannabees can take pleasure from this.

On the other hand, IT IS WRONG. He was employed to manage, he doesn't own the equipment, and having complete control over the network isn't his right, it belongs to whoever SF City nominates. (they were stupid to let it get like this in the first place)

I reckon he should pony up now, get whatever leniency he can for cooperation and get on with his life.

Can't really criticize the city for throwing the book, but I can't help but enjoy the fact that their ineptitude has been shown to the world for what it is.

.... Here's hoping for a lenient sentence. But no matter how good he is, who will trust him with their network now?

Mines the password protected one.

quote "been willing to hand over the password since Tuesday".

Looks like paranoia brought on by overwork to me.

All started off with a Audit.

T3rry1z4w350m3!!!

?

Infoworld's published an anon insider's account of the situation, along with some personality sketches. As usual, slack jaw IT management had screwed the pooch in letting this situation begin - and persist for 5 years.

http://www.infoworld.com/archives/emailPrint.jsp?R=printThis&A=/article/08/07/18/30FE-sf-network-lockout_1.html

In re the chattering class's opinion that "SF/Cisco/Bigfoot/etc. must be idjitz if they can't reset the password on a router within 3 days," apparently Mr. Childs never wrote the config to flash for any of the routers. What, did your certification textbook(s) say this was illegal to do for mission critical infrastructure on UPSes?

"Combat tactics, Mr. Ryan."

You can Ctrl-Break through a console, and then reset to factory defaults and reload the image, although they probably have no backups, It used to be a one way deal, but if it has a recent working IOS you can recover, the IOS has 10 seconds now instead of 5 to decompress which was one of the problems.

As this guy is crazy, they have to wipe and reset everything anyway, he could be using a custom IOS or installing wireless links with a timer for external access in case he gets fired. I would use a scorched earth policy anyway, lock him up, nuke the network, go through every bit of kit and software and upgrade to VIsta as a wipe method.

How come he's 43 years old in the earlier articles and 42 in the latest one? I demand answers!

The configs have not been written to flash, password-recovery is disabled, and the morons running the show in SF didn't have a policy for maintaining configuration backups.

a BOFH t-shirt from the reg store?

Seems so simple right? Some higher up just "deleted" his account? The one account, apparently, used to gain superuser access to an LDAP/Active Directory backed network of systems. So much talk of resetting Cisco routers, and network configuration issues. If there was any user in the network with the ability to "Delete" superuser accounts, then there is a user with the ability to CREATE the same.

Now, let me get this straight here. The BOFH is locked up, and the.. engineers.. can't get in. Of COURSE the PFY is assisting the engineers perfectly right? Well trained I'd say.

Mine's the one with the cattle prod in the pocket.....

The guy did his job and was terminated. This fuss about the network being locked up tight while still running means he did his job. If they had asked for a smooth turnover to his successor all this would have been avoided. If they had redundancy in the sysadmin position this would have been avoided. If they had required documentation of routine operational procedures, system tweaks, and passwords, this would have been avoided. Bean counters with tight budgets mess up systems, too.

I took over a system (not SF) from a guy who left no documentation and I had to hack into every machine to regain control. When I left there was a 60 page manual with all the details of how to run the system. If I had been suddenly dismissed there could have easily been a similar crisis for the next guy but that did not happen because reasonable employment practices were followed.

It looks to me like SF is a place sysadmins should avoid.