Communications and data network ran out the clock, no problem
For electricians it wasn’t the Ravens’ close victory over the 49ers, and it wasn’t Beyoncé’s halftime spectacular, that made Super Bowl XLVII unforgettable.
It was that 34 minutes…when the lights went out.
The outage on February 3, 2013 left a stadium full of fans and nearly 110 million TV viewers in the dark, wondering what happened. The problem was later blamed on a malfunctioning relay device in the Superdome’s power supply. Ironically, it was a device intended to protect the power system.
But for Bill Lipscomb, owner of WBL Services in Seattle, the “Blackout Bowl” was a kind of endorsement. Powered by redundant generators fully independent of the stadium, and rigorously tested in advance, Lipscomb’s Super Bowl communications and data network ran out the clock, no problem.
”Justin Oliver and Christopher Lipscomb, technology managers with WBL Services, use the Fluke 233 Remote Display Digital Multimeter to check the power supply to a UPS backup of the data network in the IDF room. The uninterruptible power supply sits between the power drop and the network rack. If it isn’t running at one hundred percent, and the power in the stadium has a glitch, down goes the network with all communications. A bit of preventive maintenance goes a long way.”
When the spotlight’s glare is brightest, the systems Lipscomb provides count most. WBL Services provides communications and technology infrastructure for big NFL events like the Super Bowl and other mega-events around the globe—the London Games, the World Cup, concerts, trade shows, and more.
“We don’t limit ourselves to telecommunications,” Lipscomb said. “We do the instant replay, the coaches’ intercoms, coaches’ video, PA announcement system, power to the sidelines, power distribution, two-way radio—pretty much anything to do with technology.”
“Small company doing big things”
“A small company doing big things” is how Lipscomb describes his firm. WBL Services has seven full-time employees and 10 on retainer. When subcontractors and vendors like AT&T or Verizon join in a big project, the team Lipscomb oversees can grow to seven hundred people. Why so many? It’s a huge job to build the technology base for an event like the Super Bowl—and not much time to do it.
“For a Super Bowl we have one month,” Lipscomb said, “to put in as much technology as an Olympics or World cup has two years to put in. Our venue is a football stadium that is in use up until the end of December. We have a crowd of 120 million people in 146 different countries, and we have less than 30 days to build that infrastructure.”
That last minute rush is preceded by extensive planning. The year before the event we are there for a week every month up until the January timeframe, when we’re going to be on site full time.”
Though stadiums are fully equipped with the lights, power, and communications they need during the playing season, a Super Bowl demands the whole new layer of infrastructure and services that Lipscomb builds and manages. “We’ll take the whole outside, the parking lots of the stadium, and turn it into the contractor compounds and the broadcast compounds,” he said. “We put in another 70 to 80,000 feet of 144-strand optical fiber. Then you have all the venues around the city for NFL Honors, NFL Media Day—another twenty-plus venues that have to be cabled up and wired.”
How does a small company manage such complexity on so short a schedule? “Experience helps,” Lipscomb said with a laugh. “We know our subcontractors, we know where our broadcast compounds will be. We have a good idea from year to year that we’ll need this much stuff.”
Careful planning is essential. “What is our backup plan to our backup plan? If this fails, what is my first process going to be to make it come back up? We monitor ourselves a lot and quite often we will take care of a problem before anybody knows it is going on. We’ve got our checks and balances in there, and we build in redundancy.”
One guiding principle is unity: broadcasters can’t install their own networks. They must use the NFL facilities that Lipscomb builds. “If it’s not managed as a single source, then you’re going to lose your ability to manage it,” he explained. “If I can’t control it, then I can’t be responsible.”
“Everybody expects it to work”
Though their work takes place behind the scenes, the expectations faced by Lipscomb’s team are very high. “We run like a utility,” Lipscomb said. “Everybody expects it to work . . . and nobody realizes how much work goes into it.”
“We will do a complete overlay of a building’s network,” Lipscomb explained. “Their data network, their cloud, their telephony network. We will tie into their network, but we do a complete overlay. So we will take two semi-loads of equipment from Seattle. We’ve done a site survey and a design, so we’ll load the equipment that we need for the job.”
Once on site, a crucial part of the job is ensuring that the power supplies all else depends on are adequate, clean, and reliable. “We will give our power provider our load calculations, and from there they will determine the size of transformer—if it’s on house power or utility power—or size of generators,” Lipscomb said. “Then we’ll do a load test. Then we’re going to verify that the power is clean, that we have the correct voltages to the power plugs, and that we have correct grounds before we plug anything in.”
With the Fluke 117 Electrician’s Digital Multimeter with Non-Contact Voltage, Justin checks the line voltage coming into the intercom distribution box. Low voltage has its own challenges and stable line voltage is a must.
Though he uses utility power on occasion, Lipscomb normally obtains power from generators supplied by Aggreko plc, the world’s largest temporary power generation company. Location is one reason: It can be easier to temporarily place a pair of generators in a stadium parking than to terminate utility power there.
“Power is critical”
“Power is a big part of what we do,” Lipscomb said. “Obviously power is critical to us. We’re always making sure our main distribution voltages are correct, using the Fluke voltage meter. We’ll use the amp meter to verify our loads. I want 20 to 30 percent headroom at least, on all of our loads. Also grounding is critical to us, to meet the TIA (Telecommunications Industry Association) grounding requirements. We’ll use the Fluke meters to make sure our grounds are correct and to certify that we’re meeting the TIA specs for grounding.” Lipscomb also uses a Fluke meter to check phase balance.
Some issues pop up again and again. “The biggest problem we run into is GFIs,” Lipscomb said. “We don’t like GFIs and our equipment doesn’t like GFIs. I know they’re a safety thing, but that’s quite often why we’ll use our own generator power, and not utility power. Our equipment is constantly tripping those things. It’s just like the coaches’ intercom systems. I will never put that on GFI. The coaches are dragging those cables or they’ll throw a headset on the ground. Obviously it’s going to short out a little bit, but that’s a low voltage short. But somehow the transmit will get onto the AC side of it and trip the GFI.”
Chris does a continuity test with the Fluke 117 DMM on the back side of the Wall Field panel in the central office of the stadium. Faulty grounds can cause noisy communications or no communications at all. Proper grounding is the first thing to check before investigating the system further, and often is the root cause of many network and communication problems.
Asked to discuss some lessons learned, Lipscomb talked about control—in both the narrow sense, and in broader terms.
“Make sure that you can control what you’re plugging into,” he advised. “If we’re plugging into utility power we want to make sure we know where the breakers are. We always test, with some Fluke device, that the outlets are grounded properly and that the hot and neutral are in correct phase.”
More broadly, it’s a matter of who can do it right: “The NFL is going to come to me if something doesn’t work. My livelihood’s on the line,” he said. “We work very closely with the stadium folks, and they’re very instrumental in helping us, but in an event like this they don’t have the amount of staff required to do this work.”
As the kickoff approaches the pressure grows. But so does a sense of accomplishment.
“What we do is so unique. There’s a lot of pressure,” Lipscomb said, “but when I walk out on that field Saturday night before the game is ready to start, I look around and say yeah, we pulled another one off. That’s a lot of satisfaction.”