Academia battles forces of IT anarchy
Unending security headaches
Academic institutions who have to add, manage, and secure thousands of new users within a period of just a few days face political and social issues on top of the immense technical ones, suggests Scott Granneman .
I really enjoy traveling about the country speaking to various groups about security, technology, and other issues, because it gives me a chance to learn first-hand about life in the trenches for IT folks. I doubt if anyone reading this will be surprised to learn that it's a rough world out there, or that most organizations are having a harder and harder time keeping up with the unending security headaches we're seeing everywhere around us. Recently I spoke at a small, private, liberal arts university, and it got me thinking about several key issues facing the folks who work maintaining and securing academic institutions.
Located in cozy Lewisburg, Pennsylvania, Bucknell University has an enrollment of 3500 undergrads and a few hundred graduate students. Add on to that hundreds of faculty and staff, and you begin to see just how many network-connected devices IT has to oversee. The folks who manage Bucknell's hardware, software, network, and data do a superlative job, but in some ways their task is impossible. Major issues lurk around every corner, and though IT has a good track record in staving off disasters, the potential is never far off.
Let's start with the beginning of each new school year. A large corporate network might have to add a few new machines every day or so, but Bucknell has to assimilate 3500 students and their computers over three days as they move in to the dorms. Every machine is wildly different, of course - if it didn't seem possible to find 3500 machines that were all installed, set up, and patched differently, then head over to Bucknell in late August 2005 for proof. Almost immediately, the fun begins... for the bad guys.
We all know that the period between Christmas and New Year's is a prime time for attacks on corporate networks, but I'm guessing that late August and early September is often the time of year that college network admins dread. During Winter holidays crackers can count on the quiet to cover their tracks, since people are focused on family and a skeleton crew is often left to watch over everything on a business' network, but the start of a new school year provides just the opposite - where there's chaos and confusion and 3500 varieties, a hacker sees happy hunting grounds, a smorgasbord of unpatched and vulnerable PCs just waiting to be owned.
Bucknell does everything in its power to keep student computers safe. It has a corporate license for a very common anti-virus product, one that pushes updates onto all the students' machines. Kids are encouraged to set Windows Update to automatically download and install patches. They're told not to engage in risky online behaviors - you know, opening every attachment they get, installing file-sharing software, downloading shady software, and so on. You and I both know the result: Bucknell still has problems with student machines getting compromised and causing major complications.
A particularly sharp new hire hacked together some perl scripts that constantly scan the residential network looking for signs of compromise: port scans, worms, IRC bots, abnormal pinging activity, spambots, and more. When a machine is found to exhibit the telltale signs of ownership by someone other than the student, then what one tech calls "the power of willy-nilly port-killing abilities" kick in, and the student's machine is automatically quarantined. Prior to the new scripts, a student's machine could be taken offline, but it was a tedious process that could take up to a half-hour for each incident; now it takes a minute or two, and it's all automated.
The idea is that once a student's machine is quarantined, he must either demonstrate to campus IT that he has fixed the problem himself, or bring the machine in to be patched and repaired by IT. At that point, full rights are restored, and away we go. I was surprised to learn that quarantine doesn't mean the student's machine is completely disconnected from the network. Instead, it means that the student cannot access the outside world of the internet. He can still view Bucknell's library website, and use any of the student forms located on campus servers, and access local resources; he just can't get to eBay, or IM his buddies at Whatsamattawith U., or email Mom and Dad.
You may be wondering, as I did, why the student's machine is not summarily removed completely from all network access. After all, his computer is still scanning the LAN and potentially causing headaches. The reason is painfully obvious once you hear it, and it has everything to do with the economic situation that higher education finds itself in today: because parents will raise holy hell if Johnny can't use his computer for school. The fact that Johnny's brand new Dell is spewing garbage all over the campus network isn't a concern shared by Mom and Dad. Removing access to the Net is one thing - IT can justify that (temporarily) to parents --but not a complete lockdown. Turning that $1500 PC into a machine good solely for playing Minesweeper just won't fly with parents paying five figures each year for their kid's education.
So PCs are quarantined, and then they come out of quarantine, and you can guess the rest. Some of them are back in quarantine within a few hours, or days, or even weeks if the stars are aligned right and the moon is in the seventh house. In fact, right after school started, Bucknell's IT staff faced a nasty problem.
The University is located in a remote rural area of Pennsylvania, so it's had to pay hefty fees for a nice fat pipe to service its 6000 residents. In spite of the bucolic location, Bucknell's techies have ensured that the network was up 24/7 over the past several years, solid as a rock. That changed a few days ago, when the network was brought to its knees. Massive slowdowns were reported by faculty, staff, and students, and they were not happy.
The IT team sprang into action, and the problem was rapidly tracked to five compromised student machines that were involved in an attempted DDOS attack on an outside server located at a hosting facility unaffiliated with Bucknell. They weren't getting outside of the network, but the incredible volume - 41,031,641 packets in 15 minutes - was leading to the angry phone calls from the University community. Once the five machines were removed from the network, things returned to normal. Not surprisingly, two of the five machines had already been in and out of quarantine several times since the start of school a few days before.
University students cause problems, but so do other adults. I was interested to hear that there is a big difference between the attitudes and perceptions held by faculty and staff. The staff - the folks who work day-to-day in the admissions and financial aid departments, the secretaries, the maintenance people, even the University officials - spend a large part, if not most, of their time in front of their computers. Their computers are their main tools they use for their jobs. Consequently, they feel some ownership for their machines. They tend to pay attention to security directives issued by Bucknell IT, and they try - as best they can - to take precautions. Not everyone, of course, but most.
Contrast that with the faculty. Again, the following description does not apply to everyone with a Ph.D. ... but it does apply to most. Faculty see computers and networks as tools that are, for the most part, not really central to their jobs as teachers. A computer is about the same as lights in the building, or heating and cooling, or plumbing: something necessary for the work environment, but outside of their realm of responsibility. But boy, it had better work, all the time, flawlessly! If the lights go off, it's not their fault; conversely, if the computers go down, it's IT's problem, not theirs.
The faculty expect computers that always work, and they also expect administrative privileges on those machines. IT had tried taking away admin rights, but there was a vocal hue and cry, a near rebellion by the professors. Worse, Tech Support was inundated by calls from users who needed to run as Administrator in order to install the various pieces of software that academics need (and as a sideline, this makes uniform patching of faculty machines crazily difficult, as it is virtually guaranteed that patches are going to break software right and left). So, admin rights went back to the faculty.
As we learned from Spider-Man, with great power comes great responsibility. Unfortunately, the faculty want super-sophisticated tools and want those tools to work perfectly, but they don't want to take any part in the care of those tools. I'm not just blaming Bucknell's faculty - this is just a part of human nature.
And that brings us to the big questions - what can Bucknell do to improve these situations? How do you change the culture at Bucknell (and at every other university) so that everyone - students, faculty, and staff - understands that there is a social value to infosec? And do colleges and universities have a responsibility to teach security, as a set of skills or as a value?
This last question might sound silly to some, but remember that some schools of higher learning used to require that students pass a swimming proficiency test before graduation. Virtually every college I know requires some set of classes, in a variety of disciplines, as a requirement for matriculation. In fact, Bucknell, like many universities, now has all incoming students go through a month-long 'Alcohol 101' course so they are aware of the dangers and responsibilities (there's that word again) associated with beer and liquor.
Colleges understand that they need to instruct students in more than just Anthropology, Physics, and English Literature, so maybe computer usage and security should be added to the list. After all, we expect college grads to know how to read, write, and think - to be literate, in other words - so perhaps we should also expect computer and security literacy as well. Bucknell's IT staff has debated the idea internally. What do you think?
Scott Granneman is a senior consultant for Bryan Consulting Inc. in St. Louis. He specializes in Internet Services and developing web applications for corporate, educational, and institutional clients.