AT&T spotlights disaster recovery

AT&T doesn’t normally train for disasters when “it’s 75 degrees and sunny outside,” according to Ken Smith, team lead of AT&T’s Network Disaster Recovery unit.

But next week, the unit will make an exception, putting its 16 years of disaster recovery experience on exhibit as part of the TelecomNext trade show in Las Vegas. The company will have 20 of the 150 self-contained trailers it uses in real-life disaster recovery on display at the trade show and will run a demonstration that shows part of what goes on behind-the-scenes at AT&T four times a year, when the company does its real disaster recovery training.

Begun in the early ‘90s, AT&T’s disaster recovery training program is unique in the industry, Smith said, because of the resources it devotes to preparing on a national scale for almost any type of disaster, manmade or natural. The company houses the 150 trailers and other NDR vehicles as well as another 250 trailers that provide backup and logistical support in four undisclosed locations around the country, ready to mobilize at a moment’s notice, as they did in the fall for 2005 for Hurricanes Katrina and Rita.

The main trailers are designed provide full Central Office functionality with everything from Class 5 switching to digital cross-connect, ATM and frame relay gear and electrical power equipment. More importantly, however, the response effort can be configured to provide exactly the capability of the specific CO affected by the disaster, Smith said.

“We have proprietary software developed by our labs that contains information about every one of our AT&T COs,” he said. “For example, for Jacksonville, Fla., [our team] can tell me what is in the office, what trailers I would need, the closest trailers stored, and all of the cabling needed to recover that office. That allows me to show up at a site and recover an office, regardless of the size, within seven days or less.”

After the September 11, 2001 attacks on the World Trade Center, AT&T had its NDR trailers up and running in New Jersey within 48 hours to replace the switch it lost under the collapsing towers, Smith said.

AT&T also has emergency communications vehicles that it supplies to local emergency services personnel to help them maintain their communications. In response to Hurricane Katrina, the company provided five such vehicles – its maximum – for the first time ever, Smith said.

“Normally, when we see a situation developing, like Katrina, we will have two vehicles in position ready to deploy – one for us and one for emergency services,” he said. “For Katrina, we deployed all five for the first time.”

Among other services provided was the capability for local law enforcement to check on the prior records of individuals arrested while looting, to determine if they had previous criminal histories and should be held.

AT&T puts its NDR capabilities to use for its corporate customers as well, doing disaster recovery assessments for them, to determine how prepared they are in case of major problems.

In Las Vegas, AT&T will have a total of 29 trailers, including its command center, a Lucent 5ESS class 5 switch, digital cross connect capability, other technology trailers and those providing power and other support. The company will be conducting tours and demonstrations of its disaster recovery capability.

“My hope is that they walk away understanding that AT&T takes reliability very seriously and has probably the best disaster recovering program in the industry,” Smith said. “Whether they are an AT&T customer or they are not, they can ask themselves about their company’s disaster recovery program. Individual customers need to ask themselves about their own plan. We have seen the devastation that can happen.”

AT&T’s NDR includes both full-time workers and other AT&T employees who have other jobs in addition to disaster recovery. While the program is a major financial investment, Smith considers it to be insurance.

“You either pay it up front and then you recover, or if you don’t pay it up front, you wind up paying later,” he said.