CARRIERS ASSESS NETWORKS IN WAKE OF DEVASTATING FIRES

California wildfires did much damage, but recovery is in full swing

Facilities-based carriers serving Southern California are well on the path to recovery after devastating wildfires that scorched more than 750,000 acres across their footprints, but not before experiencing some harrowing moments.

SBC Communications suffered the greatest damage by a wide margin, losing more than 1 million feet of fiber and copper cable. Most of that cable was aerial, due to the mountainous terrain typical of the region. SBC also lost 936 telephone poles, and 117 remote terminals lost their ability to provide service.

As of the middle of last week, SBC had replaced more than 250,000 feet of cable, replaced or restored 258 poles and had 99 remote terminals back in service. Though most of the temporary fixes will be complete in about three weeks, it will be three to six months before permanent restoration is completed.

Verizon Communications, which has several local networks serving areas of southern California from its acquisition of GTE, also suffered considerable damage, losing 222,000 feet of fiber and copper cable and suffering damage to 58 cable locations. However, no remote terminals were lost. As of last week, 44 of the cable locations had been restored.

Other carriers were more fortunate. AT&T lost a couple of Sonet rings and some aerial fiber, and Cox Communications and MCI reported no damage at all. In MCI's case, that was partly because nearly all its fiber cable is buried three feet underground, and partly because the fire moved so quickly that ground temperatures were never high enough to damage them.

While all carriers are experienced in disaster recovery and practice simulations several times a year, executives said nothing they previously experienced could have prepared them for what they encountered over the past couple of weeks. “It's unlike anything I've seen in my career,” said Bill Huber, president of network services for SBC. “I've been part of a lot of disasters, unfortunately, and each is unique, but the plant is just gone.”

California native Becky McCurdy, an area manager for Verizon, agreed, noting that previous fires generally have been confined to much smaller areas. “I've lived here all my life, and I've never seen anything like this,” she said. “You never knew what to expect in the morning.”

At one point, the fire leapt over eight lanes of Interstate 15 near San Diego, said John Kern, director of AT&T's disaster recovery program. “Usually, freeways make pretty good fire breaks,” Kern said.

Constantly changing wind direction proved as difficult an obstacle as the fires. In one instance, SBC crews replaced a fiber link that had been lost on a Sonet ring three separate times because the wind direction shifted and the fire returned to burn up the replacement cable. “Our guys would be there, and all of a sudden the fire trucks would show up again and tell them it was time to get back out,” Huber said.

McCurdy said she had to evacuate a vehicle yard containing several four-wheel-drive bucket trucks that cost $100,000 each. A firefighting plane made a sweep of the area and dumped fire-retardant foam, much of which landed on the yard. “It saved the yard,” McCurdy said. “It was a total stroke of luck.”

Though few want to experience such a disaster again, all say they've learned important lessons. McCurdy already is thinking about how to bury more cable as a way to prevent future damage. That's difficult in much of Southern California given the mountainous terrain and the fragile environmental conditions throughout the region.

“They don't want us to go to these beautiful mountain areas and start digging them up,” McCurdy said. “But now is a good time to go for the permits, because the forest is burned away.”

And while the affected carriers were justifiably doing some chest thumping over the speed of the restoration, some customers simply couldn't wait for their service to come back. Harrah's Rincon Casino near San Diego enlisted CapRock Communications, which installed a satellite-based ISDN system inside of 48 hours.

“We considered dispatching a satellite truck, but the drive from Houston to San Diego would have taken two days by itself,” said Steve Newstead, account manager for CapRock.

Instead, the carrier dispatched two engineers who worked with internal engineers from Harrah's to assemble a satellite antenna on a roof, which sent its signal to a multiplexer, which then handed off the signal to the casino's PBX via three T-1 feeds. Once up and running, the casino had 72 voice trunks that were crucial in getting its automated teller, credit/debit and check-cashing machines back in operation. Later, CapRock added a 256 kb/s Internet circuit and a dedicated virtual private network for Harrah's corporate office.

Though CapRock has deployed systems more quickly, according to CEO Peter Shaper, it had never attempted a custom application as big as this one in such a short amount of time. Making the endeavor more challenging was that Harrah's was not a previous CapRock client. “We weren't familiar with their networks or their people,” Shaper said. “But they were very supportive and appreciative of what we did for them.”