2.5 million trades a DAY: How ISE admins became Puppet masters
Penguins and Windows kids pull strings
Sysadmin blog The International Securities Exchange  (ISE) has just completed the first phase of a Puppet Enterprise  deployment. Their approach and the lessons they have learned are a teaching tool for all of us. Even in the cases where two organisations seek to accomplish the same task with the same product, the rationale and methodology can be radically divergent.
Talking to Rob Cornish - Technology Strategy and Operations Officer at ISE - and some of his team, the biggest thing that jumped out at me was the use of the past tense in describing a separation between his Linux and Windows IT teams. Cornish was eager to talk about both the operational time savings and the new version deployment time savings the organisation got from its Puppet implementation. That's well and good, but the bit that stuck with me was the casual way a cross-platform management tool brought together previously siloed entities within his organisation.
The increasing interoperability between the Windows and Linux groups happened organically. Linux administrators learned about Windows; through Puppet, they have started setting configurations and policies on Windows machines. The reverse holds true as well; Windows administrators have started to learn more about Linux. In what some may view as the sin of sins, configuration changes to Linux systems are slowly being performed by Windows admins!
This has affected ISE's large development staff as well. Not only has Puppet upped software releases from two per year to five, the mechanics of application deployment Puppet enables has begun a development/operations integration on a path towards DevOps. 
ISE didn't pick Puppet at random, it evaluated numerous products to fulfill the configuration management niche. Eventually, the exchange settled on Altiris  to manage what ISE calls the "bootstrap level". They also chose Puppet Enterprise  to manage the operating system configuration. These products are technically competitors, each offers functionality in the other product's role. But for ISE, neither is quite ready to step out of its core territory.
What makes Puppet stand out for ISE is the concept of configuration states. ISE configures a setting to a given state. Regardless of the operating system, Puppet will then ensure that setting is in that state. If Puppet discovers the setting is not in the configured state, it will change the setting so that it is - and if it cannot, it will let you know.
ISE has done its homework. Downtime is big money here and Puppet Enterprise passed muster. In an environment where every minute change must be exhaustively tested, the idea of a configuration tool that rigidly enforces a known good state is powerfully appealing.
ISE's take on the true power of Puppet Labs' enterprise offering is an interesting eye-opener. I've always used Puppet as a cross-platform version of group policy objects. It's appealing to me in that this approach doesn't require me to step too far outside my core background of Microsoft training. In my environment, I don't get to deploy big arrays of identical systems; over 70 per cent of my servers are "special needs" cases, whether I like it or not.
But the same product can be used for different needs. When I look at Puppet, I see a way to do across multiple operating systems what I had always done in Windows: GPOs. ISE looked at Puppet and saw a way to dramatically streamline its approach to IT and bring three different IT groups together.
I receive a steady stream of emails from start-ups offering not simply "one more application for $niche", but legitimately better mousetraps for the particular set of problems they identified - and which drove them to create a company. Puppet Labs is just one example of smaller tech companies that are trying to change how we practise IT. ISE maintains that Puppet Enterprise's defined-state approach reflects long-standing best practices while making implementation cheaper.
Using all tier-1 tools, the cost of accomplishing this same goal is often high enough - or complicated enough - that it is never fully realised. For how many other areas of best practice beyond those covered by Puppet might this be true? Our collective attachment to "preferred vendors", their ecosystems, ISVs and consultants often blind us to alternate providers.
The next time you casually dismiss a vendor because they are smaller - or because they have an approach that is outside of your comfort zone - stop and think. Engage with that vendor and find out why they feel different is better.
ISE did. For its efforts it has not only saved on administrator time, but has gained a massive competitive advantage by reducing time-to-market for new software versions. Seeing if "the little guy" offers a better route to best practice was good enough for a company with daily trading volumes exceeding 2.5 million contracts. It's a message the rest of us sysadmins should take to heart. ®