Data centres: what are the new skill sets?
Commodity technology = commodity staff?
You the expert The capabilities of modern servers offer in-principle benefits of more dynamic management, workload balancing and so on. How do these capabilities impact on the skillsets required of data centre operations staff today?
Infrastructure Support Engineer
The commoditisation of certain technologies once found only in high-end servers has changed the face of IT. Once upon a time only the most high end systems had the kinds of remote management, live workload transfer, high availability and "sandboxing" that all but the smallest IT shops have started to take for granted.
Not long ago the ability to remote-manage a commodity server outside the OS was a pipe dream. Today the purchase of hardware without such considerations is almost inconceivable. IPKVMs and Lights Out Management (LOM) solutions are increasingly affordable and becoming a standard component in all business class systems. (Think VPro and AMT.)
Virtualization management suites can give you the ability to turn your collection of x86 and x64 servers into something that behaves very much like one big computer. You can fill this "meta computer" with VMs, each of which can serve as a sandbox for a given workload. VMs can be moved from node to node on-the-fly. Neat tricks like mass clone deployment or high-availability are now commonplace.
While all of the above has many consequences, the take-home message here is that the technology available today is enabling any given number of administrators to care for a greater number of servers than ever before. For all intents and purposes the traditional "grunt work" of IT has been largely automated away.
In large organisations this often means IT administration services become concentrated in a central location which no longer has to be physically near a datacenter. Small numbers of "rack monkeys" are left to man the datacenters and swap out the dead bits. As new technologies allow IT departments to become leaner, the altered administrator-to-server ratio merely increases the burden of responsibility placed on each administrator.
For the moment, smaller organisations still cling to "jack of all trades" administrators. Though features like LOM and virtualisation are increasingly common, the management tools to really make them shine are often outside the reach of an SME budget.
A management tool licensing stack costing half again as much as the hardware of your 2-socket server is still finding a hard sell here. Costs of these tools are however dropping, and effective remote management of SME IT services by third-parties is increasingly a viable consideration.
These technologies are inducing a massive change in the demand for skills among administrators. As smaller numbers of specialist administrators are increasingly able to manage larger numbers of more complex systems, the "jack of all trades" administrators have become an endangered species. The commoditisation of server management tools has thus begun a true commoditisation of the various skill sets of administrators.
@Sir Runcible Spoon
Oh, no denying there's still (currently) a market for us old-timers. Jack of all trades admins are still hugely relevant in the SME space.
There will always be a limited need for jack of all trades types...but I fear the number of us that are required is shrinking. I am sure it’s cyclical…all IT job patterns have seemed so far to be, however what seems to be happening is a higher demand for narrowly-focused, (and most importantly cheap) specialists.
Smaller organisations are more and more outsourcing their IT. Those outsourcing companies, be they Dell, HP, or your local consultancy are slowly evolving along the same path. "Customer support staff" at the edges calming the customer down and finding out the problem, passing the information back to their actual "doers" who then push the button to fix the issue remotely. In the case of a dead bit, the backup is activated, and the customer sends it back to the outsource company for replacement. (Commodity hardware has gotten to the point that this is easily doable.) It is all very process-managed and sterile.
I don’t claim that this is the best way to do things…but simply that this is the direction that our industry is heading. IT is largely no longer about innovating or solving a problem with a unique solution. Almost everything you could want comes as a pre-packaged appliance now, or can be solved with easy-to-manage nifty virtualisation tricks.
Like the handyman of the physical world, the IT generalist will probably be the only of the "IT trades" that a company keeps on staff. The handyman can fix your doorknob, change the lightbulb or coax that creaky boiler back to life. If he needs something done outside his particular expertise, he calls up a plumber to have new pipe run, or an electrician to wire a room to code. We’ll be calling up a cable monkey to run lines, a datacenter specialist to install rack/power/cooling or a programmer to add functionality to a website.
Pay attention to how many handymen even large companies employ…and what those fellows get paid. They are what IT generalists are set to become. (On a totally selfish note however, it gives me no end of glee to picture several of the "specialists" I know as the digital plumbers of the future. Sadly, the "IT Specialist" types in my neck of the woods really look down on us generalists quite a bit.)
Good luck to us all!
ssh, bash perl?
If your seriously using ssh bash and perl alone to manage large numbers of systems, even smallish numbers(more than 50) then you really should look at better tools like cfengine or puppet.
ssh bash and perl alone are far too limited to handle real distributed admin work, unless of course your writing something along the lines of cfengine or puppet in perl ..
Just last week as an example my company built a new edge location in London(I'm in Seattle). We sent one guy out with 1 server running ESX and a VM with enough data to seed the rest of the network. He configured all of the remote management cards, the network guy got the VPN up, and I remotely installed ~55 physical and virtual machines in roughly 6.5 hours over the WAN including all application configuration and everything. And that was using real remote installs, not stupid image-based crap.
It could of been done faster though I decided to throw a wrench into things and revamp my kickstart setup at the same time which made me have to go troubleshoot some things and re-install some systems to fix them.
But all in all it was a pretty good day.
Wouldn't of been possible in that time line to do it without cfengine though(I haven't used puppet hear it's pretty good). My cfengine configuration is about 20,000 lines, to give an idea of the number of variations and tests that the system runs to ensure compliance, and that compliance is re-checked every single hour of the day.
I suppose I consider myself a jack of all trades guy whether it's systems, storage, networking, automation, monitoring, and architecture. It's really rare that I physically touch hardware these days.
I don't personally know anyone else that has my skill sets, though they weren't easy to come across, made a lot of sacrifices over the past decade(well worth it in the long run I believe, no regrets).
Pick the right products and you can scale pretty damn well.
lots of parallels
My old school physics teacher complains that he cannot understand his new Honda - even though he taught auto engineering.
Same with the data center, how on earth do you know what is happening with hundreds of apps interacting on top of an dynamic, ever changing infrastructure. Talk about a house built on sand. Virtualisation/elastic environment = sharp knife = blood on floor.
Complexity has forced the auto manufacturers produce analysis boxes to pinpoint problems in their vehicles. I need one for my e2e processes but unfortunately, we are not there yet and there is little on the horizon.
Come back squawk box, all is forgiven.
re:Jack of all trades
Whilst it's true that there aren't as many positions for jack of all trades operators (thus most people seem to specialise) it does leave a small, but workably signifcant market for people who specialise in being able to do a bit of everything.
Perhaps not employed in the actual field of their knowledge, but a general background of all things technical, coupled with a good range of 'soft' skills can make all the difference when contracting. It's still difficult to secure a position, but anyone who can organise and relate to various parts of the business and have a good understanding of budgetry requirements, time constraints and internal politics (and how to avoid them) will more than likely get the renewals that make contracting viable.
This is the path I have tried to take and I'm still only on my fourth contract in 8 years, being fully employed throughout (apart from the 4 month 'holiday' I took this summer :)