MASTERS OF DISASTERS

Spring almost never comes early in the Upper Great Plains. Most years, in fact, spring is nothing more than a rumor. But in 1997, spring was impatient, and when the premature thaw came, those who lived along the Red River were ecstatic.

Until the flood. The Red River, which divides North Dakota and Minnesota, is rare among North American tributaries in that it flows north, not south. Unfortunately, Canada was still in winter's icy grip, which meant that all the water filling the Red River's banks further downstream had nowhere to go. So it came back. That quickly became a problem for Qwest Communications, which has a central office in Grand Forks, N.D.

“The water was waist-deep in the basement,” said Jim Sampson, director of Qwest's disaster preparedness program. Fortunately for Qwest, critical equipment such as switches and the power plant were located on upper floors. But the carrier suffered considerable damage to copper cable. The event became the catalyst for Qwest's acceleration of its fledgling disaster recovery program.

Disaster recovery and business continuity have become industry buzzwords, seemingly uttered ad infinitum since the terrorist attacks of Sept. 11, 2001 — particularly by carrier personnel who watched Verizon Communications struggle with the monumental restoration of its Lower Manhattan central office for nearly two years.

But while it would be easy to point to Sept. 11 as the point at which disaster recovery became a critical action item for carriers, it also would be wrong, said Suzy Henderson, executive director of national security emergency preparedness for SBC Communications. SBC's footprint is particularly prone to the Big Three of natural disasters: earthquakes, tornadoes and hurricanes. “Disaster recovery has been on our radar screen at least since the [1984] divestiture,” she said. “It's been an integral part of our corporate philosophy.”

Each RBOC has made sizeable commitments to the effort. Though they wouldn't go into specifics, Sampson and Henderson said Qwest and SBC both budget several million dollars annually to disaster response preparation, and each carrier has dedicated a cadre to the effort — 13 employees at Qwest and seven at SBC — plus a battalion of hundreds of volunteers.

Qwest maintains 11 in-region teams, one out-of-region team, one corporate-level team that handles multi-state incidents, and an executive-level team. SBC maintains one team for each major market it serves, but beefs up seasonally; in the spring, tornadoes are a nuisance in the Midwest, while late summer brings hurricane threats to the Texas coast.

SBC renewed its commitment to disaster recovery about four years ago, with technology becoming more complex and networks becoming the lifeblood of customers, Henderson said. “So much depends on the network that we have to be very fluid at this,” Henderson said.

The key to fluidity is preparation. In football, coaches develop game plans, drill their players until they can perform the required actions by rote and then test what they've learned under game conditions. Afterward, they assess what worked and what didn't, and then tweak the game plan for the next week. It works the same way for telephone service providers.

Qwest, SBC, BellSouth and Verizon all run their emergency response teams through simulations and drills. Qwest, for instance, annually conducts one planned four-hour disaster drill for each in-house team. All team members also must participate in two hours of training annually. Three times each year, team members receive pages that instruct them to report for a disaster event — and they must do so within 20 minutes.

At SBC, all team members have to participate in a policy training session at least once each year. SBC also conducts at least two exercises in each of its three regions annually. Not every drill is pre-announced; SBC occasionally will spring an exercise on its team to test their preparedness. SBC even stages mock news bulletins that are broadcast over the company's closed-circuit television system, reporting that a disaster has occurred but providing little detail on the extent of the damage. “They don't get a lot of information,” Henderson said.

Verizon Communications even runs its media-relations team though the gauntlet. “They spring it on us. We'll get a call from building security telling us that there are news cameras in the lobby,” said Mark Marchand, a media-relations director for Verizon Network Operations.

The one aspect upon which Sept. 11 had a direct influence concerns the types of disasters for which the RBOCs prepare. Prior to the attacks, the Bells' efforts were restricted to preparing for what Mother Nature might throw at them. Now they also prepare for manmade calamities. “Everyone says that after Sept. 11 the world changed, and for us it did,” said Francis “Fran” Dramis, executive vice president and chief information officer for BellSouth. “As we looked at our plans, we found ourselves to be a little naive. All of [the plans] were built around natural disasters. None of them was based on wanton or malicious acts.”

For instance, the Bells are now drilling for biohazard events. “You have to be ready to evacuate your personnel,” said Sampson. “But you also have to know what to do when you can't evacuate. Employees have to be sheltered and made as comfortable as possible.”

It takes three to four months to develop a disaster simulation, said Henderson. “It's like writing a novel,” she said. “You start at the beginning and work your way through it. At the end, we want to have tested as many critical services, and put as much stress on our systems, as possible.” Once the scenario is constructed, a tabletop exercise ensues — SBC's is patterned after models created by the Federal Emergency Management Administration.

But as well developed as the Bells' programs are, they don't come close to the effort AT&T puts into its program, which has been a dozen years in the making. AT&T not only is the patriarch of today's disaster recovery programs, it also is the kingfish.

John Kearn, AT&T's director of network continuity, leads the program, as he has done for the past five years. About 90 employees are involved in the effort, with 50 of those dedicated full time, numbers that pale when compared with the RBOCs. But what makes AT&T unique is not how many of its employees are involved, but what those employees do.

AT&T has spent more than $300 million on the effort since 1991, and much of that money has been spent on 20-foot semi-tractor trailers that house network equipment — both inside and outside plant — operations support systems equipment and power equipment, capable of replicating any domestic or international transport scenario (AT&T does business in 54 countries and 140 markets outside the U.S.). The program started with two trailers; currently, AT&T has 150 trailers and adds 10 to 12 each year.

The trailers are loaded and ready to go at a moment's notice. They are stored at four nondescript, secret and secure locations scattered geographically in such a way that AT&T can have equipment up and running at a disaster scene within 72 hours. “That's the bogey,” said Kearn.

AT&T has on call in each region two trucking companies plus two national companies on backup at all times. AT&T employees accompany the caravan when the trucks roll. “We have a good relationship with these companies, but we don't trust them that much,” Kearn said. “There are millions of dollars in equipment on board. Plus, if something goes wrong along the way, we want our employees interfacing with the government, not some trucking company.”

Four times a year, once in each region, AT&T rolls the trucks and conducts disaster simulations. The scenarios are staged in different locations each time to give AT&T an opportunity to work directly with local governments, each of which has its unique agenda.

Deploying the equipment to conduct the simulations gives AT&T critical hands-on opportunities to analyze how existing systems and processes work, as well as test new theories, said Kearn. For instance, AT&T discovered a more effective and less costly way to ground its equipment at a disaster scene as a result of its simulations.

Its original approach was to drive 8-foot copper rods into the ground every eight feet to create a ring to which the trailers could be grounded. It was time-consuming and backbreaking work, and once the exercise ended, the rods had to be pulled out of the ground, which was also problematic. “Once the rod is driven, if you take it out, it's scrap,” Kearn said. “Drive it again and it will bend.”

Eventually, an engineering firm with which AT&T regularly works suggested to Kearn that he look at a grounding system developed by the U.S. Military that consisted of steel braiding affixed to a single 10-inch aluminum stake. Once the lone stake was driven, it was a relatively simple matter to connect each trailer to the steel braiding.

According to Kearn, it was the perfect solution in that it provided the same grounding as the copper rods, could be reused, required only two people to deploy — compared with 10 to 15 people for the copper rod system — and takes just a couple of hours to set up, compared with a full day. “Every exercise we come up with a new trick or toy that makes it easier,” Kearn said.

Setting up such elaborate simulations also provides a corollary benefit for AT&T: The company can showcase its disaster recovery abilities to the media and to its customers. “It's important that our customers know they made the right choice,” Kearn said. “We want them to see for themselves that we have a robust network that will perform for them on good days and bad.”