Automation - one step closer to lights out?
Do you trust the tools?
Workshop In an earlier article in this workshop we considered the challenges associated with procuring system management technologies. We know from our research amongst IT administrators that making a business case for systems management tools is rarely straightforward, even when major changes in the infrastructure or the way services need to be delivered to users is undergoing change.
Increasing the automation of systems management by using these management tools holds out the promise of significant benefits and aids the creation of a business case, but are we ready to trust software to automate the administration of critical, or even major components, of the IT service delivery infrastructure?
Insight we have gathered around the management of x86 server estates illustrates that IT staff are under considerable pressure. Over half of all organisations with more than 50 staff reported that management operations are challenging due to over stretching operations staff. The same report tells us that only about a quarter of organisations have management capabilities standardised on one or two integrated systems management suites.
Automating routine processes has the potential to free up valuable staff time as well as opening the possibility to relieve the “overstretch” we mentioned earlier. It can also delay significantly, or even remove, the need to employ additional staff. The latter benefit is clearly a matter that helps at a time when IT spending is under very tight control. It also assists when making a business case when increases in operational budgets are difficult to justify, and for which it is even harder to garner approval.
This is interesting when we consider the current use of tools to automate systems management processes. The figure below highlights that few organisations make extensive use of automation. As can be seen only around ten percent of organisations consider that they have sorted out the automated response to problems with obvious solutions. It is likely that the number automating the response to more complex operational issues is even smaller, especially as the implementation of “root cause” analysis tools and service level monitoring systems is still at a relatively low level.
Many IT vendors are now talking up their ability to help automate routine tasks and even to help “repair” simple service management problems without the manual intervention of IT staff. Indeed, talk of the “lights out” running of systems is sometimes posited to be the ultimate “nirvana” state of the future for IT operations.
Clearly this state of affairs is nowhere near ready for practical implementation in the daily IT operation of any organisation. But the question to be asked is just how much manual labour is required today if best practice and modern management tools are to be employed to the full? The answer is that many routine tasks can be operated with little or no manual intervention. Indeed, it is even possible for system provisioning to be almost fully automated in response to a user request, a matter well illustrated by the way that many consumers gain access to certain web services. This automation of service provisioning is gaining a foothold in business but there does appear to some resistance to be overcome, with a fair dose of technological objections or issues combined with both business inertia and political wrangling.
I know from my own experiences as a systems manager that few IT professionals are comfortable with the full automation of any process run by management tools. And nor should they be comfortable – such an approach would constitute a big risk, and systems manager are paid to manage the risk effectively. However, it is possible to automate where it makes sense and still maintain good oversight of risk.
Even when the tools have been used without incident for some time, there are many administrators who wish to still hit the “yes” key to allow a systems management tool to make any change or initiate a process, thereby still avoiding “full automation”. A good approach to take would be to combine operations experience with automation to allow managers to maintain a watching eye on operations and control without always having to say “yes”, i.e. raising the flag where appropriate. In this way it should be possible to gain confidence in the automation tools and then gradually shift responsibility to them over time, always keeping an effective process on hand for exception handling and manual escalation. ®
Perhaps you have stumbled upon why sysadmins take 6-12mo to update/patch their software! WinXP+IE6 for everyone! Why? Because we have automated scripts to manage that (and you don't get a block-all popup requesting authorization to make system changes, Vista/7).
Over my rotting corpse. Users (and especially managers) don't give a flight-enabled euphemism about anything except that which directly affects them. If you happen to work in a business where licensing costs are “ITs problem” then the concept of automated provisioning is essentially like asking all members of IT to please go an slit their wrists with an HIV infected needle. The very concept is ludicrous.
Users are the “people” (and I use the term loosely) who /demand/ with raised voices, red faces and other tantrum-like symptoms a copy of the latest full Adobe CS suite so they can open a JPEG. “Because the colour is rendered better in Photoshop.” Not so they can EDIT a JPEG mind you, but so that they can VIEW it. Why they need the whole rest of the suite is bloody beyond me.
Don’t forget that apparently saving a word document to PDF absolutely REQUIRES the latest Adobe CS Suite as well. Apparently using PDF 995 would /end the world/. Also; every single new task, render engine or what-have-you apparently requires a separate VM. The capacity for which supposedly grows on trees.
AUTOMATED PROVISIONING? Like hell. The day that the resources to provide the licenses and the storage/network/server capacity come out of someone ELSE’S budget, they can automatically provision anything they want. As it is, I have trouble just keeping critical business functions (such as e-mail, data storage, backups, web services, etc.) running under the existing budget.
I should add that training/”educating” users doesn’t help. At all. You can’t make someone actually /care/ about something they don’t view as their problem. There is no education platform, corporate policy or incentive package in existence that will ever cause the average joe to grow two spoonfuls of give-a-shit. If the cost of what they are “automatically provisioning” doesn’t hit them /personally/ and /directly/, then they will simply look at it as “something the company/IT/my manager/someone-who-is-not-me” has to worry about and push the button to spawn another instance.
If you don’t restrain users, they will feast upon IT services long past the point where they have gorged themselves.
Automatic provisioning? Over my rotting corpse.
Change is the enemy of automation
So you've got a bunch of screen-based processes and procedures you have to carry out: Whether it's adding new users, updating the helpdesk status of support requests or reporting the consumption of storage space across the enterprise. All fine and dandy, then some numpty in another part of the organisation goes and changes something.
They might do a software upgrade to the storage manager, or patch the helpdesk software or change the default width of the user's address field. Whatever it is, no matter how far ahead they published the change notice or how innocuous it seemed at the time, things like this break automated processes. You can't test for them before deploying your automated scripts as the nature of the change is unknowable (even down to correcting a spilling mistake that you were screen-scraping for, or creating a new one that you now miss). Hence you now have to spend the time you saved automating something to fix the automation script to account for the new environment.
The end result is that as long as people make software changes, we will always be playing catchup with our automation suites. So all that happens is sysadmins go from spending boring days doing repetitive tasks to boring days debugging faulty automation scripts.