Dev's telnet tinkering lands him on out-of-hour conference call with CEO, CTO, MD
'Please stop... you're doing something to the radios'
Who, me? Welcome all, to the merry world of Who, Me?, our weekly trip down memory lane for techies who want to get something off their chest.
This week, we hear from "Gavin", who was working at an ISP that had a few thousand point-to-multipoint radio links and had been asked to write a script to backup their configs.
"They already had a bare config backup going every night, but the bosses wanted to get some extra diagnostics out as well, which involved running a telnet command on the units," he said.
"They were rough 'n' ready Russian-made things with their own firmware, so the commands were pretty straightforward and the same across base stations and CPEs."
Gavin wrote the script in Perl – "the telnet module saw good use in other projects just fine" – and set about testing it on a few hundred devices at a time.
College PRIMOS prankster wreaks havoc with sysadmin manualsREAD MORE
It looked fine for a handful, Gavin said, so he started forking the process so it was quicker and would finish through the night.
"Then I started noticing my script kept hanging, seemingly randomly."
Gavin said he would kill the process, make a change to his telnet login or commands, and then try again.
"This seemed to progress it every time to a larger batch of devices."
Gavin kept doing this, out of hours, for about 20 minutes, until his personal phone rang.
"I picked up to realise I was in a conference with the MD (who was fairly technical, especially around these radios) the operations director, the CTO and the CEO – essentially all top brass," he recalled.
"Rather politely (compared to their usual behaviour during priority one incidents) they asked whether I was 'doing something' with the radios."
Gavin explained that they'd assigned him this project, to which he was told to "please stop for now" as it was "causing problems with some of the units".
After facing the walk of shame the next morning, "under the glaring eyes of colleagues and poor installation technicians about to be dispatched or returning from jobs", Gavin found the source of the problem.
"Because the firmware on a lot of the devices wasn't updated for years, their telnet client had a number of quirks, such as it wouldn't ask for a password or would only ask for a password," he said.
"In some of those cases, my script would happily keep sending the password/user in a loop, even if it didn't see the right prompt, effectively flooding the device."
And, if the extra traffic wasn't enough, the CPU would crash and then they had to dispatch the technicians – to four different cities in the UK from just one base. During the night. To dozens of sites.
"I had a stern talking to by the bosses but received no punishment," Gavin said.
He added that he remained an employee for years, as "adaptability to shifty infrastructure and business knowledge was what kept us going as a business. Or maybe because I was the only dev at the time…"
Gavin says his story became part of his company's lore, told at dinners and parties. Has anything you've ever done ended up outlasting your time at a firm?
Tell Who, Me? about it by emailing us here. ®
Sponsored: Becoming a Pragmatic Security Leader