WHEN DISASTER STRIKES

There are times when no amount of planning is enough to negate the effects of a disaster, whether it's by an act of God or an act of terrorism. But having a plan is better than having no plan at all.

Verizon Communications discovered this truism when the World Trade Center was attacked Sept. 11. The collapse of the twin towers and, later, 7 World Trade Center caused a great deal of chaos for the carrier, whose central office in Lower Manhattan is across the street from Ground Zero (Telephony, Nov. 19, page 36).

In addition to a great deal of structural damage caused by vibration shock and flying debris — including steel girders that javelined into the building — the facility suffered the ravages of 10 million gallons of water — a torrent created by a broken water main, ruptured sprinkler system and the New York City fire department — which flooded the building's five sublevels and took 10 days to pump out.

Located in these sublevels was the facility's primary power grid and its backup diesel generators, all of which were submerged and destroyed. Commercial power was knocked out during the catastrophe as well. Consequently, with its primary and backup power systems kaput, Verizon had to get creative — and in a hurry.

Fortunately, the carrier had a plan. Verizon brought in three temporary generators that replaced a good portion of the 16,000 amps normally provided by Consolidated Edison. It built an adequate amount of redundancy into the temporary system to handle any challenges that would pop up should one of the generators have failed. And it prioritized — also quickly — what needed to get power right away and what could wait, in order to avoid overloading the system until the commercial power could be restored.

Having a good disaster recovery plan is paramount for any telecommunications carrier, according to Steve Roy, director of product line management for Emerson Energy Systems. “You've got to make sure you've planned for the worst,” he said.

Unfortunately, not enough carriers do, said Angelo Mandarino, vice president of sales for Invensys. Or at least not enough did before the tragedies. “Before 9-11, the money being spent toward disaster recovery was really dropping,” he said. “The disaster plans that were in effect, and probably are still in effect, are woefully out of date.”

While Verizon was forced to deal with what everyone hopes is a once-in-a-lifetime event, carriers encounter significant disasters on a fairly regular basis, including hurricanes, tornadoes, floods and earthquakes. Though redundancy didn''''t help Verizon in New York, it is the key to coping with such calamities, said John Barnicle, president and CEO of competitive carrier Focal Communications.

“It's a design issue. You want to build in… better backup systems, multiple UPS systems and multiple grid feeds,” he said. “Anything you can do to build in that redundancy up front [will help] you to react in an emergency.”

Controlling one's destiny also would help. According to Barnicle, Focal leased space in New York City for its equipment in a carrier hotel. Because it didn't own the building, the carrier found itself in the unenviable position of being unable to gain access to the generator when disaster befell Lower Manhattan. “Let's just say that it is the only situation of its kind in Focal, and it will be the last,” said Barnicle. Focal is now negotiating with the landlord to put in its own dedicated generator, “or at least an umbilical — a big pipe down to the street to which you could roll up a portable generator and plug it in,” he added.

Redundancy and control are just parts of the answer. Familiarity with the unique nuances of each type of disaster is another. “In an earthquake, roads may be damaged… so you want the company that maintains your generator and stocks your fuel supply in close proximity,” Barnicle said.

Qwest Communications found itself in the middle of an earthquake last February when a temblor centered near Olympia, Wash., shook the Seattle/Tacoma region. Almost immediately, the company dispatched teams into the area to assess the damage. Qwest also relied heavily on the local power company, which was able to identify the grids where the carrier lost power.

Another important tool was the company's maintenance and surveillance centers in Denver — where Qwest is headquartered — and Minnesota. Circuits on the carrier's switches provided important data on load and traffic on the network, which was useful in determining where power had been lost. In addition, an automatic paging system kicks into gear when trouble occurs. “If it''''s three in the morning and we lose power, someone knows about it right away,” said Robert Jones, Qwest's senior vice president of network operations for the Pacific Northwest. All of this information and activity helped Qwest deploy mobile generators from Oregon to where they were needed in less than three hours. “We didn't lose any central offices,” said Jones.

Remote monitoring is an important tool, said Emerson's Roy. “This is the one feature that would be beneficial in becoming better prepared for a failure,” he said. “More and more large telecom companies have these networks to manage the status of nodes and power plants.”

However, preparation and maintenance are the most important tactics for any carrier trying to develop a disaster recovery plan, Jones said. Consequently, Qwest has developed a standard service assurance plan. Generators are checked each week, batteries are checked on a regular basis and fuel supplies are monitored routinely. The carrier also conducts three disaster recovery exercises each year, one of which is coordinated with city and county agencies. The exercises are designed to test Qwest's ability to quickly coordinate its internal efforts as well as its ability to coordinate with the efforts of local government. A variety of scenarios — including earthquakes, floods and fires — are simulated.

Jones predicted these activities would be stepped up in the aftermath of Sept. 11. “The tragedies brought home how critical all of this preparation and maintenance work is, and we're probably more sensitive to it now,” he said. “This is not discretionary activity. You have to be prepared for every eventuality.”

Stan Taylor, staff manager for network operations at BellSouth, agreed. According to Taylor, BellSouth's COs have been designed and reinforced to “withstand just about anything,” though he admits no BellSouth facility has taken a hit from a category F4 or F5 tornado. Nevertheless, all COs are wired for backup power and have permanent diesel generators, large fuel tanks, battery backups and sump pumps. The company also owns its own fleet of diesel fuel trucks.

“We believe we have all the bases covered. I can't remember ever losing power to a central office,” Taylor said. “We're prepared for ‘act of God’ situations. But if you're talking about a smoking black hole where a CO used to be, that's a whole different story.”

Like Qwest, BellSouth also employs “regional pool generators” that can be transported to wherever they're needed. However, BellSouth hedges its bets by positioning these generators in regions where they are most likely to be required, depending on the time of year. In the winter, they are housed in the carrier's northern territory, in preparation for ice storms. In the spring and summer they are stationed near the Atlantic and Gulf coasts, in anticipation of hurricanes.

“Preparation is more important than response” when dealing with a disaster situation, said Taylor, who added he expects preparation levels “will be expanded to some degree” in the aftermath of Sept. 11. “Awareness is heightened,” he said.

The events of Sept. 11 have underscored for many the intrinsic vulnerability of carrier facilities. For that reason, Invensys' Mandarino suggests that carriers should begin to decentralize their power plants. “With decentralization, all your eggs are never in one basket. There's never one incident that can hinder you.”

Though decentralization is a costlier approach, Mandarino said there are other benefits that make it worth the effort. One is that decentralized sites are rarely used to capacity. “It allows you to take on more work… because you're not running sites at top workload,” he said. “And if there's a spike in one area, you're typically able to offload that work. It allows you to handle more peaks and valleys in your business.”

---

With additional reporting by Jennifer D'Anastasio and Toby Weber in Chicago.