Original URL: http://www.theregister.co.uk/2008/05/24/switch_switchnap_rob_roy/

Welcome to Las Vegas - Home of the technology superpower you've never heard of

Rob Roy seizes Enron's busted dream

By Ashlee Vance

Posted in Servers, 24th May 2008 01:00 GMT

Exclusive Drive a couple of blocks past the Loose Caboose and the Carburetor Shop on E. Sahara Avenue in Las Vegas, and you'll find one of the world's leading technology companies. The name of the company - Switch Communications - will go unrecognized by almost all of you. That's because it has operated in near total secrecy for the last few years. Switch has preferred to keep its gold mine a need-to-know type of affair. "Pay no attention to the secure fortress in the strip mall."

A few months ago, word of Switch's apparently fantastic operations started to reach my in-box. Most of the people who visited the Switch facility were bound by non-disclosure agreements, but that failed to stop them from leaking out a few choice details. "This is the most advanced computing center in the world," I was told. "It's like the internet superhighway wrapped up in one package. All the heavies are there."

Ever a cynic, I struggled to match these claims with the total lack of public information available on Switch. Companies fall all over themselves to issue press releases about things as a minor as cost-savings achieved by changing toilet paper suppliers. If a technology giant really existed in Las Vegas of all places, then it should be patting itself on the back and then letting city officials finish off the job with celebrations of their own.

As Switch's CEO Rob Roy tells it, however, the company had good reason to avoid publicity.

Legend has it that the company managed to acquire what was once meant to be Enron's broadband trading hub for a song. This gave Switch access to more than twenty of the primary carrier backbones in a single location. Switch tied this vast network to existing data center hosting facilities and attracted military clients, among others, to its Las Vegas shop.

Roy argues that drawing undue attention to this facility would go against the military customers' best wishes. There are rooms at the Switch facility that require top secret clearance, preventing even Roy from entering them.

But Switch has now decided to forgo the code of silence as its business expands on a massive scale.

In the next couple of months, Switch will open a new facility located just a few miles from the McCarran International Airport called the SuperNAP. Roy describes the 407,000 square foot facility as the most energy efficient, tightly packed data center on the planet. He expects it to be filled by the world's most prominent companies, including just about every technology heavyweight you can think of and the major media conglomerates. The SuperNAP monstrosity looks to stand as just a starting point for Switch with the company's investors urging it to build close to 10 similar centers around the globe. Such an undertaking could strap actual muscle to the cloud and utility computing buzzwords that have become commonplace in the technology industry.

Come November, Switch will throw a huge party at the SuperNAP for its investors, top customers, employees and the local bigwigs in the real estate and casino businesses. The event was pushed back to November, so that Nevada's two US senators could attend the gala.

So much for staying quiet.

You'll Need a Rob Roy to Face the Security

"Those companies (Google, Microsoft and others) spending $500m on their data centers can only do one fourth of what we can do at the new site," Roy told me. "What we are building has three or four times the power and cooling of the other guys."

This is how Roy talks. Everything is on a massive scale. Everything clobbers the competition. Everything is "Look at how much ass we kick. The competitors have no hope. We do a damn good job, and our customers know it."

Roy exhibits such unabashed enthusiasm for Switch's accomplishments that it seems almost miraculous for him to have rejected public accolades over the past 8 years.

The 38 year-old CEO insists that he stayed quiet to protect his customers and because businesses was so good that Switch didn't need the attention.

Part of me believes this line. I spent weeks calling and e-mailing Switch, trying to secure an interview with Roy or, in fact, anyone at the company. The polite receptionist would take the same message every time, but nothing resembling a returned phone call materialized.

Then I learned that Sun Microsystems had a significant presence inside of Switch's existing hosting facility. On a lark, I sent a note to Chairman and co-founder Scott McNealy to see if he knew anything about this Rob Roy fella. Twelve hours later, Roy was on the phone.

Roy and I worked out a deal. He would give me full access to Switch's operations, if I agreed to keep some information - mostly on clients - out of this publication. We rarely make such bargains, but since Roy would rather talk to a door-to-door salesman than a hack, it seemed the best way to attack the story at hand. I can tell you that Switch's data centers are packed with Fortune 100 companies you would recognize. These companies make everything from servers and switches to search engines, bombs, cartoons, MP3 players and hotels. We'll let you fill in the blanks for now.

Even with McNealy's kind introduction in place, Roy seemed apprehensive about my arrival on-site. He came into the conference room and sat at the opposite end of the table from me and my audio recorder. (Note to self: Bring the boom microphone next time.)

Roy's distancing himself from me gesture was about the last thing I needed at that point.

People visiting Switch must go through a bit of a security ordeal. You buzz in at a main gate that's located in an alley and then drive around to the front of the building. By the time your car arrives, there's a guy with a gun waiting for you. I'm not talking about a sleepy rent-a-cop here. I'm talking Data Center Rambo. The muscle-bound dude who greeted me had something between a smile and a grimace on his face, which I assumed was the result of gas from the prowler he killed that morning and then ate for breakfast.

Most up-scale businesses have a nice enough receptionist, some couches and magazines waiting behind their front door. Not Switch. It has something known as a mantrap.

If you're living the good life in Las Vegas, then a mantrap is a busty waitress who will take half your money many years down the road. If you're hunting Co-Lo facilities, then a mantrap is a room encased in 12 gauge steel that can only be bested by biometric authentication.

On one side of this steel-clad room - where the nice receptionist should be - are more former military personnel and a bunch of screens. One of the gun-toting guys takes your government issued identification, which Switch keeps during your stay, while the other monitors the monitors.

Switch has one screen dedicated to tracking weather across the globe. It has about 15 more that can pull up pictures from what must be hundreds of security cameras. Then there's another one that seems to track power and networking grids, and then there's the really creepy one.

Shot of the Switch Man Trap - a steel cage

Switch's Man Trap

"That screen monitors global terrorist activity," I was told. "If someone even mentions the word 'bomb' on a plane, we find out about it."

Quizzical look from reporter.

"These guys have access to databases that you and I don't even know anything about."

Sure, sure. We'll get into a discussion about how realistic that is some other time far, far away from the guy with the gun and the gas grimace.

Some contacts have described Switch's security as "overblown" and on a level that would "make Ross Perot look almost negligent." Roy insists that such protection is necessary for the work at hand, so close to 70 per cent of the employees have military backgrounds, including a COO who used to wire fighter jets.

My mistake was thinking that the security part would be the most unnerving part of the morning.

After being led down a hallway - God, everything here is painted red, gray or blue - I went into the meeting headquarters.

The conference room was almost intimidating. The lights were dimmed, which provided an ominous aura for the thirty black leather chairs filling up the space. I kept expecting Darth Vader to spin around for a grand reveal. The room also had a number of tables made out of ladder racks and brushed aluminum, which I'm told that Switch designed itself. Then, there were screens and some more screens - the largest of which had a paused video of the SuperNAP.

Switch's attention to things like the custom tables carries throughout the complex. Even the bathroom has backlit sinks with webby, Spiderman like designs. "I just washed my hands and am now full of Justice."

Given Switch's penchant for flair, it again seems surprising that the company has denied overtures from the press. But this is all for the customers - part of the show meant to convey that Switch means business.

The King of Co-Lo

Roy eventually seemed to warm up to me, moving progressively closer to the recorder, as we spent close to three hours together in the facility. I even weaseled my way into some personnel details.

He and his younger brother grew up in Wisconsin. When Roy was 12, his father was paralyzed - a devastating injury that Roy described as being similar to actor Christopher Reeve's condition. Roy's father decided to obtain a computer science degree from the University of Wisconsin, using a dumb terminal to do the work from his home. This left Roy needing to help with tasks such as holding books, turning pages and listening to lectures.

"You're sitting there in every meeting," he said. "I grew up just being very involved in the whole process. I think I have been 30 since I was 13."

Mike Borden, Roy's brother, is President of Switch, and the different last name reflects "a difficult divorce between our parents" that Roy preferred not to discuss.

As the years went by, Roy shifted away from this early interaction with computer science into commercial real estate, moving to Las Vegas during his twenties. Eventually, he ended up getting into the co-location business, where a company will host servers and storage systems for other businesses.

An early incarnation of Switch was already up and running when Roy caught wind of the Enron facility across the street that was going up for sale.

"Enron wanted to expand from trading energy," Roy said. "They wanted to use the same type of conceptual algorithm and go trade bandwidth."

The ever overzealous folks at Enron poured millions of dollars into creating a single facility that would serve as a hub for all the major US carriers. Their pipes would funnel into the Enron building, and Enron would move bandwidth around like a commodity.

"I don't think that was ever going to be real, but they thought it was," Roy said.

Las Vegas will not jump out as the likely spot for such a hub to most people. The city, however, presents certain advantages over places such as New York, Los Angeles, Atlanta, Dallas or Silicon Valley. There are almost no natural disasters that threaten Las Vegas. The city has cheaper power and real estate than other major cities. There are tons of flights going into Las Vegas, making it easy to do business. And, as Roy says, "No one lives in Nevada. So, you don't have locals pulling on the bandwidth."

Enron had already built a lot of the infrastructure needed for its facility and brought the major carriers on board just as its business started to collapse. So, the broadband center went up for sale.

"We were the only ones that bid on it," Roy said. "It should have been the $200bn companies that owned it. We got it for a Cinderella story type of figure."

Shot of Rob Roy in his office, which is filled with cartoon and movie character action figures

Rob Roy's Fun Filled Office

And other folks back up Roy's take on the situation.

"More people should have bid on the Enron Super POP auction," Sun's McNealy told me. "I think they got a steal there."

It should be noted that there seems to be some dispute over who bought what from the Enron facility. One contact tells me that the Enron network actually went to Wilhagan Ventures, which now operates the system as WVFiber. And, in fact, there are a number of stories noting that Wilhagan did buy the Enron assets during bankruptcy proceedings to fuel a nationwide fiber backbone.

It's more difficult to dispute what Switch actually offers to customers, which is access to the networks of companies such as Verizon, AT&T, Sprint, Cox, XO, Qwest, Time Warner, Global Crossing and Level 3 from a single location. You can see the full scope of the operation here. The end result is that Switch can provide bandwidth to customers on a scale that rivals the major hubs in New York and Los Angeles.

According to Roy, this gives Switch some unique advantages as far as pricing and capacity go. The company can underbid more expensive facilities in California and create more competition among the carriers.

"We built this huge valve system, and we plug all the carriers - 10 and 40 gig - into our big gateway," Roy said. "Our biggest customers are saying we are selling space to them at 44 per cent less than all of their connections. And my agreements are not just about Vegas. I can order a link in Germany cheaper than anyone else can.

"I can do this because we have some of the world's biggest companies looking to get into the SuperNAP. And the carriers are hearing about this and seeing the volume of bandwidth these customers want. And I can go and price out that bandwidth with a bunch of different carriers. So, carrier X comes and says, 'Okay, we'll play ball.'

"You know, you can take wholesale, and we're 30 per cent below that. We are just alone in a very unique spot in the world because of the Enron building. We have clients that come in and save more on connectivity than they pay for the entire data center in a month. So, it almost makes their data center free."

I told you he was proud of the network, right?

Ah, but that's just where the fun begins.

Look! There in the desert! It's SuperNAP!

The networking operation has helped Switch carve out a significant place in the Nevada technology landscape, but it's the SuperNAP that threatens to take the company global.

Switch has spent years banging away on a data center design that runs contrary to conventional thinking. Most large scale data centers utilize raised floors where cold air is pumped through holes in the floor and picked up by servers and storage systems. This type of architecture often results in the top parts of the computing systems getting very hot. So, companies have developed all manners of weird contraptions ranging from liquid cooling systems, heat socks and even data center robots that seek out warm spots to try and avoid concentrations of warm air.

By contrast, Switch has created its own system called a T-SCIF (Thermal Separate Compartment in a Facility), which lets Switch abandon expensive raised floors altogether.

You basically back a server or storage system into the T-SCIF container and create a seal that separates the cool air pulled in by the front of the system from the hot air exhaled from the rear of the system. Unlike most data centers, Switch does not need to deal with intermingling cool and warm air. The hardware receives only cool air, while the hot air travels out through ducts above the T-SCIF. This design lets Switch run servers at full power and allows it to stack as many systems as possible into a given space whereas many data centers must avoid filling up racks with hardware because of heat and power concerns.

Shot of the TSCIF container

Meet T-SCIF

(For more on the T-SCIFs and the SuperNAP design see this story.)

The SuperNAP will make use of the T-SCIF approach on a grand scale.

The $350m facility will stand as one of the largest data centers on the planet. It will rival anything being built by the likes of Microsoft and Google, which often say that their new centers cost about $500m a pop. Roy, however, believes that he can stack four times as much gear as those companies do in his center.

Switch is bringing in 30 cooling towers and its own power station to fill the SuperNAP with 7,000 cabinets of hardware and 1,500 watts per square foot of energy. All told, the SuperNAP will chew through more energy than three mega casinos. And Switch has room for four SuperNAP-sized buildings on the land that it owns.

While the scale of the center is impressive, it's Roy's ambitions that prove more remarkable.

Rather than selling space in the SuperNAP to a handful of clients, which he could probably do, Roy wants to divide the building up among a number of the world's thriving businesses that operate in various fields. Roy seems to see himself as a type of rainmaker for the internet age, bringing together disparate parties to create digital happiness.

Roy talks about linking the major casinos with the major content providers and carriers to make IPTV a reality. Only a company such as Switch with the bandwidth, data centers and connections could get all these folks to agree on a shared agenda, Roy says.

"This building will be the first super-scale data center where all the new technology in the world merges and therefore creates a better layer of technology than anyone else has been looking at," he said. "I can make this work because I can do things no one else can."

A man must dream big, right?

Today, organizations turn to Switch for black-ops projects, spam filtering of the most serious proportions, utility computing projects, data warehouses at casinos, modeling, online games and old-fashioned e-commerce. Roy wants to convince these clients to keep their most cutting-edge projects at Switch and then wants to work as a go between, telling the various clients where they can help each other out rather than duplicating effort.

This will strike many of you as fanciful talk. Although, as Roy described his vision, I was reminded of Samuel Insull, who created what amounted to the first true electricity grid in Chicago by pulling in customers of different stripes to keep his generators humming away all day and night. By achieving this scale, Insull could lower the price of electricity and make it more feasible for companies to abandon their own power systems.

Could it be that a figure like Roy will be this generation's Insull and actually deliver a utility computing infrastructure? Such an idea seems almost more likely to me than a company like Google, Microsoft, Amazon or IBM pulling it off given Switch's rather more neutral role.

Then again, maybe Roy talks way too big.

Over the years, Roy has acquired quite a number of detractors.

Some of the nay-sayers attack Switch straight on with practical complaints. One of our contacts said that Switch fails to beat out the usually far more expensive Bay Area providers such as Equinix, Savvis and AT&T, which "is pretty sad considering that power in Las Vegas is half of Silicon Valley." The same contact - a data center expert - suspects that Switch will struggle to power the SuperNAP. "They're in the middle of the desert and will need almost 3 million gallons of water per day for blowdown and evaporation for their 30,000 ton evaporative cooling plant."

It's the data center old-hands that find Switch's security operations over-the-top and more show than substance. (I had a gun toter follow me around the entire data center tour, which Roy said was a requirement of some of the government contracts. The guard may have been over-the-top, but he was very polite and handy, opening every door.)

Those with closer, rather acrimonious personal relationships with Roy say he tends to over-promise by a country mile. There's plenty of polish to go around, we're told, but the product often fails to live up to its shine. (Look past the Men in Black, and you'll find this guy as the EVP of network operations.)

Having barely opened the door onto Switch, I'm in no position to judge in which direction the see-saw of truth tilts.

I can say that the company's attention to detail comes off as more impressive than any other similar facility that I've encountered. There's a great deal to be said for any company that's willing to apply unique engineering to a large scale problem such as data center operations. And there's a great deal to be said for a company that's vibrant and aggressive.

In addition, the investors behind Switch include some of the most respected institutions and individuals in Las Vegas. This company has plenty of capital to tap as it tries to turn dreams into reality. Large investors outside of Las Vegas have obviously taken note of this and of Switch's success because I spied an investment proposal from one of the world's top banks sitting on Roy's desk.

The workers at Switch - many of them multi-millionaires, including Roy's assistant - look like they've bought into the program as well. I've never seen a happier group of t-shirt and short wearing staff.

Ultimately, it won't matter what outsiders think of Switch once the SuperNAP opens. The computing center will either be just another huge co-location facility or it will turn into something much more significant - a utility computing landmark in the land of Elvis impersonators.

I can assure you that The Register will be monitoring Switch's progress in the months to come. Goodbye golden silence. Hello World. ®

Bootnote

People who work at Switch long enough are awarded their very own comic book character as part of the Switch Blades series. The underlying premise of the comic is that the Dark Ethereals have taken control of the Earth's ionosphere with the intention of controlling all natural disasters and stopping human advancement by disabling the internet. It's the Switch Blades mission to destroy these Dark Ethereals.

Roy, in addition to designing furniture and cooling systems, wrote the plot for the comic strip.