Rapid Response

AT&T Wireless ensures adequate capacity during disasters and special events.

Since AT&T entered the wireless industry in 1981, it has been tweaking its network to accommodate disasters and special events. From hurricanes to plane crashes to large conventions, the company has learned to act quickly to handle large surges in network usage.

Although special events usually allow the company more time to plan for an influx of wireless users than natural or manmade disasters allow, AT&T Wireless' strategy is almost identical in all situations: Look at what the network handles right now; analyze busy-hour traffic; figure out where the most people will enter the area; find out where the hot spots will be; design a new plan to handle the increased traffic.

Dealing With Disasters After adding necessary capacity and responding to the needs of emergency personnel during the ValuJet Flight 592 and TWA Flight 800 plane crashes in 1996, as well as during the fall of Alaska Airlines Flight 261 last January, AT&T Wireless has proved its network is ready to handle increased call volumes when disaster strikes.

Bill Brugger, AT&T Wireless Network Services director of business continuity, said in plane crash situations, the company immediately tries to determine where capacity issues will crop up due to investigative bodies such as the FBI and the Bureau of Alcohol, Tobacco and Firearms (ATF) coming into a market.

"Typically, we mobilize cell sites on wheels (COWs) before we even receive a request for them," Brugger said. "From our experience, we know that there is a need for our services in times of tragedy, whether the people there locally know they will need our services or not."

Not only does AT&T Wireless mobilize COWs in order to provide free wireless services to emergency personnel in times of disaster, it also does so to pro-actively circumvent capacity problems that may arise.

"Usually, our local customers run into traffic jams if we don't do that, so we protect our business at the same time that we help officials deal with the incident," added Ken Woo, AT&T Wireless spokesperson.

AT&T Wireless then gets in touch with the ranking official or manager of the incident. Although Federal Emergency Management Agency, ATF and Special Weapons & Tactics teams are experts in disasters, the local people who have not experienced such events often are first on to the scene.

"Sometimes the local people say they won't need our services, so we give them a history of what has happened in the past," Brugger said. "Typically, the local office personnel just haven't experienced such a disaster and they don't know what will befall them in the next 24 hours. If they refuse, we say we are standing by. Inevitably, in the next few hours they ask for help."

To get a sense of how much capacity to add, AT&T Wireless technicians find out where officials plan to set up the disaster field offices and command centers. The local AT&T Wireless technicians and managers know what the network can handle on a daily basis and what the busy-hour traffic looks like.

"From our track record, we know that X people typically come in during these times of disaster," Brugger said. "The people in charge also help us determine what agencies are coming in and how many folks they are bringing so we can identify where we need to set up towers. We then develop a new temporary plan to cover the amount of people."

The decision about where to set up extra sites is incident-specific. It could be that emergency personnel set up centers in two separate places, so AT&T Wireless must set up two separate COWs. Other mobile sites include a CALF, which is a mobile cell site smaller than a COW; a cell site on light truck (COLT); and microwave equipment on wheels (MEOW). MEOWs include large microcells and small microcells.

"If we just need to increase coverage in one building because there will be a lot of officials at desks, we probably will put a microcell in that building," Brugger said. "However, for something like the Oklahoma City bombing, we brought COWs in to cover the overall area and the local area for the increased population around there, then we put a microcell in a building where the command center was."

AT&T Wireless takes several things into consideration when figuring out the best combination of mobile sites. First, the company decides how much capacity it will need, then it looks at where it will need it. Next, it analyzes whether it must work around any geographic obstructions.

"For example, if you need coverage in downtown Manhattan, usually radios work off of line-of-sight, so you have some issues there," Brugger explained. "The technical folks are used to designing this network day-to-day and will look at all of those considerations. Then, we create a solution based on what the needs are."

Besides rolling COWs, CALFs, COLTs and MEOWs, AT&T Wireless takes advantage of its modular digital system, which allows it to slide more radios into cell sites to quickly add capacity.

Planned Events When planning ahead for special occasions that draw large crowds to certain areas, Brugger said AT&T Wireless goes through the same process as during disasters.

"The difference is that with planned events we have the lead time to work with the parties that are involved to discover what their needs are going to be," he said.

Usually, AT&T Wireless meets with event coordinators and leaders well in advance of large events. Each year, the NFL contacts AT&T Wireless before the Super Bowl, and the company always knows a year or two ahead of time where the next annual game will be. Brugger added that AT&T Wireless has been planning for the 2002 Winter Olympics in Salt Lake City for more than a year. The company will add capacity in areas around town where competitions will be held, at tourist attractions and resorts, and in places where it anticipates a higher call volume such as media and security headquarters.

It also keeps on top of other regular events. For example, prior to the recent Republican National Convention and Democratic National Convention, the company met with convention leaders, as well as the U.S. Secret Service and local authorities, in order to plan ahead for extra capacity in its Philadelphia and Los Angeles markets.

As for conventions and smaller events in various markets, local public-relations offices in each market keep abreast of events and notify the company of necessary actions.

"It just depends on the magnitude of the event and how much lead time we need to plan for it," Brugger said. "We learn a lot from the planned events that help us shorten our response time for our unplanned events."

Crossing Competitive Lines Woo noted that during emergency situations, it sometimes is necessary to work with competitors in order to get a damaged network up and running. For example, a switching center that flooded in Rochelle Park, NJ, during Hurricane Andrew happened to be next door to a Bell Atlantic switch connecting AT&T Wireless to Bell Atlantic's landline network. Bell Atlantic's switching center also flooded, and once AT&T Wireless fixed its own facility, it worked on the Bell Atlantic switch, too.

"We had to get them up in order for us to reconnect to our customers, and that is the interdependencies that exist between our communication network and the local exchange carrier (LEC)," Brugger said.

In addition, Bell Atlantic's generator was located in the flooded basement, so the company had no power. AT&T Wireless had extra power in its building, so it ran 200-foot cables from its power plant to Bell Atlantic's equipment and kept the LEC running for about three weeks.

"This is a very competitive business, but when it comes to disasters, it seems the carriers come together and forget about the divisions and there is a lot of coordination between technical and engineering groups," Woo said.

Whether it is an unplanned emergency or a large special event that attracts an influx of wireless-phone users, networks do fail. It is important to have a survivability plan that will get your network back on track if it takes a beating from forces beyond your control.

AT&T Wireless' survivability plan is based on four pillars: prevention, mitigation, response and recovery.

Bill Brugger, AT&T Wireless Network Services director of business continuity, said the prevention piece means the company tries to design a reliable, survivable and recoverable network from the start. It also tries to locate its key network elements in geographic areas that are less exposed to natural disasters.

"If one location is directly on a fault line or a 100-year flood plane and the other is not, we pick the one that is not," he said.

However, sometimes it is necessary to collocate equipment, and sometimes those locations are in exposed areas. In these cases, mitigation is paramount. In potential flood planes, for instance, a mitigation tactic simply could mean locating power and generator equipment above the fifth floor.

A written incident-management plan helps the company preplan its response to natural disasters and network outages. AT&T Wireless continuously updates its response tactics as it learns from each new situation and as it trains employees. For example, during Hurricane Andrew, several employees flew an airplane full of equipment down to Florida. However, damage was so extensive that whole buildings had disappeared, rendering their map useless and forcing them to use landmarks to find their sites. Now, the company uses GPS to locate its cell sites.

AT&T Wireless also conducts routine exercises in each market, during which employees react to hypothetical situations while implementing the company's recovery plan.

Last, the company performs recovery, which is a damage assessment on the network that determines what is broken and the extent of the damage. Different disasters require different recovery strategies, which AT&T outlines in its survivability plan. Once it reviews the damage statistics, it puts the plan into action in order to recover all of its assets.

"Every disaster is different, so you need a comprehensive plan that really focuses on the worst-case scenario instead of developing a plan for every scenario you could have," Brugger advised.