The Starz Entertainment building with its satellite dish farm.
It’s an old saying in the entertainment business, but HVAC technician and chief engineer Bob Axelson and the maintenance team at Starz Entertainment, LLC live it every day: The show MUST go on. They watch over Denver-area data storage and production facilities vital to ensure that 17 Starz Entertainment, LLC premium channels are available. Fluke HVAC News sat down with Axelson to learn how he approaches a job where one mistake can put you center stage—in exactly the wrong way.
Bob, please tell us about your facility and your team.
We have a team of four that includes two evening-shift building engineers, a master electrician, and me (the HVAC tech and chief engineer). We take care of a 350,000-square-foot (32,500-square-meter), class A facility that includes three floors of offices and two technical floors. The technical floors include three data centers, two uninterruptible power supply (UPS) and battery rooms, the transmitter room that serves our four satellite uplink dishes, the broadcast operations center, IT services, and our graphics, audio, and editing bays.
Starz has a state-of-the-art amplifier room that generates quite a bit of heat, requiring some serious air movement to keep everything cool.
We have four 175-ton rooftop units to serve the office areas and more than 25 Liebert computer room air conditioning (CRAC) units serving the technical space. The Liebert units are water cooled and served by five cooling towers, two of them redundant. Three of the towers are newer high efficiency units with VFDs [variable frequency drives] driving the cooling fans. The two older towers were converted to VFDs for efficiency. Xcel Energy, our power provider, paid half the cost of the VFDs in the form of rebates for the energy savings.
How long have you been there, and how did you get started in this work?
I have been employed by Starz Entertainment for close to nine years. I came out of the trades as a journeyman HVAC tech specializing in data center maintenance and construction.
How is your team organized?
The master electrician and I work the day shift and our second shift works evenings. The electrician and I work in conjunction as needed, and we can rely on backup from our broadcast engineers or bring in contractors. We do 95 percent of the service work in-house, and we follow a strict maintenance schedule to prevent failures.
What specific issues are your responsibility?
We are responsible for all the HVAC, electrical, and plumbing. We maintain and repair the cooling towers, remote terminal units, CRAC units, building automation, and the general building operation.
Bob takes a temperature measurement from one of the air conditioner units. The servers in this room generate quite a bit of heat. To address this problem, Starz uses individual cooling units that patch into a main Liebert CRAC unit through a system of cooling lines. If more servers are added to any rack in the room and more cooling is needed, Bob can simply add (or move) one of these small units.
Data centers can generate a lot of heat. How do you deal with that, in a “green” way?
We have three data centers, two UPS [uninterruptible power supply] and battery rooms, a transmitter room, three labs, and twelve IDF [individual distribution frame] rooms [containing cable racks for telecommunications cabling] requiring over 300 tons of critical cooling. Because of the dispersed nature of the spaces being cooled and the Colorado environment, we use a closed glycol loop for heat rejection on all our critical spaces. Two years ago we overran the capacity of our original glycol loop and added three additional high efficiency towers (one being redundant) with accommodations for a fourth future tower. Along with the build-out of the second loop, we retrofitted the existing cooling towers with VFDs to improve operating efficiency.
The new cooling loop uses a dry chemical water treatment that is much more environmentally friendly than older water treatment methods. The existing towers are being converted to the same system. Our IT data center was also upgraded to Liebert’s new XDC/ XDV high density cooling system to provide higher efficiency and more precise cooling. We are in the process of converting our broadcast data center over to the same systems. We received a significant rebate from our power provider for power savings on the original system, and expect the same or more for the next system.
Any other “green” initiatives?
That is very important to this company. We at one point we were going to go for LEED Certification, but we found that we were better off spending the money we would use for certification on different projects we had. We changed out eleven-hundred 50-watt halogen bulbs with three ½–watt LEDs. We just completed the retrofit of our elevators, and went from 25-watt bulbs to 2½-watt bulbs. There are 250 of those. We’ve gotten considerable rebates from our power company over the past four or five years. We spent around $11,000 in an energy audit, and the suggested modifications ended up saving around $65,000 the first year.
What is your greatest challenge?
Our largest challenge is to provide uninterrupted cooling and power. Downtime is not an option. If we lose our ability to transmit our signal we lose revenue and can be subject to fines. Do that and you can get real famous, real fast.
What issues affect how you get the job done?
One of the largest issues is what parts to stock for repairs and replacement. A lot of manufacturers do not keep a sizable inventory on hand, and neither do we.
What approach does it take to succeed?
MAINTENANCE! We’ve gone into the operations manuals on all the equipment we’ve got, and used the manufacturers’ recommendations as far as maintenance schedule. Every month we have a schedule that pops up and lists all the equipment that needs service that month. Then we’ve modified that over the years and either lengthened the span of maintenance on something or condensed it, to make sure we don’t have any issues. We log what we’ve done on a web-based record and keep track of it that way.
What tests do you typically perform?
We perform routine maintenance on all equipment and do daily walkthroughs. The routine maintenance includes voltage and amperage reading on motors and pumps, and pressure and temperature readings on cooling equipment. We do an annual vibration analysis, along with thermographic inspections on breaker panels and electrical connections.
A real asset is our building automation system. We monitor our three incoming power feeds, water pressure, and gas pressure from the utilities, all our cooling/heating equipment, UPS power and distribution, CRAC units and so on, even down to the temperature of the hot water into the faucets serving the kitchen. This system has saved our bacon more than once.
Why are these tests important in keeping your facility up and running?
We have a recorded benchmark, and any variance is noted and the cause is determined and dealt with in a timely manner to prevent failures.
Bob Axelson has several rooftop air conditioners to look after. Here he’s at an electrical panel on one of those rooftop HVAC units, taking a quick current measurement with his Fluke clamp meter.
What test tools do you use, and which do you reach for most often?
Where do I start? The majority of our tools, and those of our broadcast engineers, come from Fluke. My tool collection includes a 116 multimeter, a 922 Airflow Meter, a 79-lll multimeter, an LVD2 Volt Light, a 61 Infrared Thermometer, a 53 ll Thermometer, an RLD2, and in my tool pouch a 334 Clamp Meter. Every time I send it out to be recalibrated I go nuts that week that it’s gone. I’ve carried that since I got into the trades. We do have some tools from other manufacturers, because Fluke does not make them yet. We trust our lives and safety to our tools—what is that worth? Not to mention accurate, precise measurements.
What constitutes a tough day on the job?
A contractor, working on a build-out, forgot to phase tape a switch leg. The next day that leg got tied in with the other neutrals. When the 277 volt switch leg got fed to the neutrals by turning on the light switch, the main breaker tripped and we lost all but UPS power to that side of the building. We had no idea what caused the failure but had to get fans set up in the data centers and other critical areas to stay online.
What has been your experience with Fluke tools?
Early in my career I saw a co-worker experience a failure of a less expensive multimeter. It exploded like a grenade in his hand, fortunately not injuring him. That created a lasting impression. You will see more Fluke tools throughout this facility.
Is there something about the entertainment business that adds to the challenge?
People want their TV.