IT execs plan, don't test, business continuity

Business-continuity planning is a growing priority among Chicago-area businesses, but convincing them to fully test their plans remains a challenge, according to AT&T’s annual business continuity survey.

In conjunction with a network disaster recovery drill AT&T conducted in Chicago over the past week, the company released the Chicago portion of its annual survey in advance of the full survey results, which will come out in June. The Chicago results are based on a telephony survey of 100 IT executives in the Chicago metro area, drawn from companies with at least $10 million in revenue.

The survey showed 75% of IT executives consider business-continuity planning to be a priority, compared with 60% in 2007. Eighty-one percent of those surveyed said their companies have a business-continuity plan, up from 76% a year ago.

But only 43% have fully tested their plans within the last 12 months, an improvement over the 37% that did so in 2007. Almost one-fifth -- 12% -- admitted they have never tested their business-continuity plans, up slightly from 10% in 2007. Businesses were much more likely to update their plans -- 54% had done so within the last year.

As part of its drill, conducted in a Soldier Field parking lot, AT&T brought business customers through 60 technology trailers and other vehicles, representing $500 million in network assets, that are normally warehoused in various undisclosed locations around the country, said Mark Francis, vice president of global network operations.

“We bring them through here to show them that we are ready, so we can ask if they are ready,” Francis said in an interview Tuesday. “Because if your business continuity plan consists of a set of viewgraphs and a pretty Word document, and you are counting on your vendors to come in at the last minute and save you, you don’t have a real plan.”

Other key findings in Chicago showed 72% of Chicago executives have included cyber security as part of their business continuity plan.


“We are also concerned with cyber attacks within our group,” Francis said. “The beauty of aggregating all traffic at the network level is that we can see when hackers are at work. They are like any other software makers, they have bugs. We can see their attempts, and we can see the spike in traffic when they fix the bugs. That enables us to respond more quickly in shutting them down.”

AT&T conducts these drills up to four times a year, in differing locations, both to keep its national team on its toes but also to train local personnel, many of whom volunteer for this service, Francis said. While in Chicago, the AT&T team also met with local officials engaged in disaster recovery planning, to discuss things such as logistical management, Francis said.

“Our first goal is exercising our people, because a lot of them are volunteers for disaster recovery,” Francis said. “They need to keep up to speed, and we need to be able to leverage their skills and their knowledge of the local market, so we use local personnel in conjunction with the [network disaster recovery] leadership team.”

As all services converge on an IP core, plans for business continuity in disasters become even more important, Francis said, as the converged network impacts everything from point-of-sale terminals in stores to internal networks within companies.

AT&T believes it is unique in the scope of its network disaster recovery (NDR) effort, which began in 1992 but has been fully deployed only once, following the Sept. 11 attacks, when the NDR team set up in Jersey City, N.J.

“This is a huge investment in assets and in personnel,” Francis said. “We don’t see anyone investing at this level.”

The scope of AT&T’s response includes support for any service that is in the field today, from basic copper services through legacy data such as frame relay and ATM, right up to IP and high-speed optical services, with each technology deployed in a different vehicle. AT&T’s immediate response vehicles include a communications van or SUV that is a self-contained unit that can establish immediate satellite communications to help emergency responders, among others, and is completely self-sufficient, providing its own power. The satellite truck can be up and running in as little as four to six hours.

The AT&T NDR effort also includes COWs – cell sites on wheels – and COLTs – cell sites on light trucks – that can not only provide cell service in the event of a disaster but also are used to boost cell capacity at special events such as the SuperBowl or NASCAR races, said Kevan Parker, a member of the NDR team. It takes about 24 hours to configure the mobile cellular units to match local spectrum requirements, he said.

While multiple trucks that just carry power generators are part of the NDR response team, each individual truck also contains its own generators and battery backups for failover power, said Kelly Morrison, public information officer for the NDR team. The NDR effort also travels with its own machine shop and tool shop. “We take everything we need to be self-sufficient,” Morrison said. “We assume there’s nothing available where we are going.”