Secrets Revealed: Inside the AOL Dulles Data Center

Okay, first off, I'm not actually going to be revealing any secrets (I'm generally happy with my current employment situation) -- I just wanted to make a few people nervous.

Last week, as part of February's focus on showing employees existing and emerging Internet technologies (the Valentine's Day thing was also part of it), the company conducted a couple of tours of the Dulles Technology Center (DTC).

A data center is pretty much a building with lots and lots of computer hardware in it. AOL has three of them in Northern Virginia (Dulles, Reston and Manassas).

I'd never been to an AOL data center, though I had visited someone else's colocation center.

Now, because Internet companies are nothing without their data, they tend to be very cagey about their data centers, and security is really tight. So I wasn't able to take any photographs. Of anything.

(I mean it -- they're really strict. A year or two ago, someone in Ops told me that one of the Journals servers was going to be decommissioned. I asked if I could get a picture. It didn't even have to be in the facility proper -- I would have been happy with it sitting on a loading dock or something. It took a while, but they finally gave me an answer: NO.)

So I don't have any pictures. Otherwise, I would have maxed out my memory card, because I saw a ton of really cool things.

(I was going to grab a photo of a power plant from 'The Matrix', and use that, but decided it was too much of a hassle to do it legal-like.)

After about 20 minutes on a shuttle bus, rounding up all the folks on the tour (the better to keep tabs on us), we got to the data center.

Physical security is paramount (Seen 'Sneakers'? Kind of like that, only with fewer shotguns.) so we weren't going anywhere until we were greeted by Alan, the facility manager, who gave us a quick briefing in the lobby. Then we went inside.

Loud, Clean and Cold
The DTC is solid. It's very spare and sparse-- lots of white-painted concrete. It's also very much a "form follows function" kind of place -- lots of big blue chilled-water pipes and cable runs overhead. It's very industrial-chic.

Also, most everything is neatly labeled.

Alan led us through a couple of server rooms. Each one was cavernous, noisy and filled with racks (so, like a Costco), only instead being stocked with cat food and paper towels, they were filled with switches, servers and other kinds of hardware.

Here's a public domain photo of a typical data center setup:

Not sure whose it is, but it's not ours.

The machines are mounted in racks, some like the ones you see above. The floors are raised so they can run air conditioning (gotta keep the machines cool) and power underneath. It's also the reason why you can't wear open-toed shoes. (It's posted. I asked. Answer: You don't want to stub a toe on the edge of a open floor tile.)

Fire is also a big concern. Overhead sprinklers are required by code (though it would be really, really bad if they ever went off), so there's another chemical fire suppression system (I forget what gas it is, but it's good for machines, bad for humans).

Since we're near the flight path of Dulles airport (just down the road), I asked if there were any additional special precautions for that. Alan said no, so I guess there aren't dudes with Stinger missiles on the roof. (When it was built, the prevailing rumor was that if a plane crashed into the building, the machines would be okay, but the people would be toast. Because you can always replace people. I know, not funny. And yet... )

Along the way, we learned a lot about the air conditioning and watercooling units, uninterruptible power supplies, battery backups, generators, fuel and power consumption, and even weather monitoring (in the facility control room, there's a weather monitoring system with a memorial plaque, dedicating it to the guy who originally set it up) -- everything you need to keep the machines happy.

(One of the mechanical contractors has a project page about the Manassas Data Center, with a few photos of the big, colored pipes and some very impressive sounding specs.)

And redundancy. They're very big on redundancy in systems. Primary systems. Backup systems. Backups to the backups. Spares for the backups. Backups for the spares. I was reminded of the saying "One is none and two is one" -- take that and double it a few times.

Of course, none of this comes cheap; according to the press release, the Manassas facility (which is newer, and incorporates some of the lessons learned from the other data centers) cost a total of $520 million (including the land, building, and all the computer hardware and other gear.)

NOC, NOC
After the facilities tour, we went to the NOC (Network Operations Center), and got handed off to Dave and Jennifer. The NOC is a room filled with computers they use to monitor the health of the hardware. This wasn't the big NOC (I've seen pictures: it's well on its way to being the bridge of the USS Enterprise), but there were still lots of screens, with lots of usage graphs.

Dave said, "People like the graphs."

They talked a little bit about what they do and how they're basically the gold standard when it comes to running a NOC (especially in terms of efficiency). They apparently give a lot of tours to folks (including the then-Prince of Jordan, NBC execs, and Ozzy Osbourne) who want to improve their own data centers.

I can't really talk about specifics, but they said the Pentagon uses one person to watch 7 machines, and that we used... significant fewer people.

Anyway, that's about all I can talk about without getting into RAIDs and hard drive failure rates and Linux adoption and support tiers and leasing vs. buying and custom monitoring tools and other interesting stuff.

I Know Where I'll Be for the Zombie Apocalypse
So, overall, I was very impressed with both the facility, and the professionalism of the people who run it.

If tours of the DTC or similar facilities are offered again, I think that every product manager should sign up for one, so they know what goes on when there's a hardware problem, or they wonder why someone can't just run to CompUSA to get more hard drives when they need more storage space.

Finally, when the zombie apocalypse or other catastrophe occurs, I'm going to head to the data center (well, first, I'm going to raid the Wegman's), because according to the short story 'When Sysadmins Ruled the Earth' [strong language], it seems like the best place to ride it out.

Thanks -- Joe

Tags: aol, dulles, dtc, dulles technology center, data center, redundancy, redundancy, zombie apocalypse