Networks Under Test

Tuesday, Sept. 11, 2001, tested the wireless industry and its networks like never before.

Most networks are designed to accommodate a predictive growth model as new subscribers are added. But the worst terrorist attack in U.S. history pushed those networks serving 121 million subscribers beyond the limits of even the best models.

Capacity more than doubled most carriers' peak traffic periods, which typically last a few hours during rush hours. Carriers whose sites were directly affected by the devastation in New York City and Washington, DC, were forced to redirect traffic, restore service and even redesign networks.

Despite the losses and outages, wireless networks not only held up, they also provided a vital lifeline for authorities, rescue workers and friends and families across the country. In the process, carriers learned a lot about their networks, where they will go from here and how to prepare them for future crises that seemed unthinkable only two short months ago.

Shocks to the System

Almost immediately after the attacks on the World Trade Center (WTC) and a subsequent attack on the Pentagon, wireless networks were inundated with traffic, which surged to unprecedented heights.

During the peak of the crisis, Verizon Wireless experienced 50% to 100% more traffic than normal, nationwide.

Sprint PCS logged 2.5 times the normal traffic volume for its New York City network.

In the Piscataway, NJ, wireless telephone switching office that serves New Jersey and greater New York, Cingular Wireless experienced a 1,000% increase in attempts to use the network between 10 a.m. and 11 a.m. that Tuesday. During the height of the crisis overall, Cingular saw a 200% to 250% increase in call attempts nationwide.

According to Toby Seay, vice president of field operations, AT&T Wireless saw a “tremendous lift” in its total network traffic on and around Sept. 11. In Manhattan, the carrier experienced an 800% increase from one hour to the next in the number of attempted calls.

Nextel won't reveal exact figures but reported its digital cellular network experienced “extraordinary call volume” along the East Coast and sharp activity increases nationwide, resulting in congestion on its wireless network.

“The landlines were flooded, and the wireless system had usage that is probably historic,” said Audrey Schaefer, Nextel spokesperson.

There was little time for shock — the first thing most carriers did after hearing about the attacks and after making sure all of their people were safe, was to assess their networks' stability while developing plans to restore service, reroute traffic and add capacity wherever necessary.

Kevin Haynes, Cingular Wireless executive director, network development and operations support, quickly called regional vice presidents in the New England area and laid out plans to set up a command center in Washington. Then all of the other vice presidents were conferenced in to assess the entire Cingular network, region by region.

“We had a conference call every hour to assess the network impact and come up with best practices that were going on that we could share with the other regions and implement across the country,” he said.

AT&T Wireless' implemented its instant-management plan, which goes into effect during a crisis. A national emergency operations center, as well as local centers, were opened and staffed 24/7 to monitor and manage the network and flow information and resources. Seay said a conference bridge connected “duty officers” across the country who accounted for people, resources and network status on an hourly basis and assessed what was happening in every area — which resources were needed and what issues had come up since the last “roll call.”

The issues included tremendous traffic spikes in airports around the country because of stranded subscribers. The network also experienced large demands in Chicago, where people evacuated downtown and headed for the suburbs.

Service Despite Damage

Wireless service was maintained for the most part, despite the fact that most carriers lost cell sites or experienced outages.

Verizon Wireless said 10 of its cell sites were knocked out of service initially.

Four Sprint PCS cell sites in the immediate disaster area were rendered useless. Many others across Manhattan were temporarily without service due to commercial power outages. Some lost their landline connections from other telecom carriers whose systems and facilities were damaged seriously.

AT&T experienced outages, and individual sites in and around the WTC were damaged or lost. Although the carrier never totally lost service in Manhattan, it did experience network outages in the 2- to 3-mile southern tip of the island, caused by power and landline problems.

“There was a tremendous amount of damage done to the landline network, specifically in Manhattan,” Seay said. “Multiple city blocks were without power for extended periods of time, and that had impacts on us in terms of sites and our ability to keep those sites on air and processing traffic.”

Cingular Interactive lost telephony and/or power to eight of its 60 cell sites in lower Manhattan as a result of a power outage, but fully functional base stations in New Jersey and Brooklyn provided backup. No network equipment was destroyed, but the Mobitex messaging service was hampered by outages in lower Manhattan, mostly due to power problems after the towers collapsed.

“Once we were allowed back in there, we were able to get the sites back up with temporary power,” Haynes said. “In the event that we'd lost a data line, we were able to bring it back up with a dial up.”

Cingular Interactive, which operates the nation's largest dedicated data network, said traffic on its system surged almost 60% in the hours after the attacks. AT&T Wireless also experienced increased traffic in its 2-way messaging service, and Verizon Wireless said text-messaging traffic on its network surged by 50% during the attacks.

Redesigning, Rerouting & Restoring

Cingular's Washington coverage didn't experience any outages, but a high spike in traffic required additional capacity and cell sites on wheels (COWs).

“There was a 400% increase in attempts in Washington, DC, (on that Tuesday) … mostly in the few hours after the occurrence,” Haynes said. “It didn't cause any congestion on our control channels or signaling system, but … there was just not a channel to provide every one of those calls, unfortunately.”

However, Cingular did handle an increased amount of traffic — about 25% more minutes than normal. It deployed six COWs in the Washington area to increase network capacity, and engineers made use of channels, too.

“We had some channels in our warehouse for (our) build plan — what we were going to build our cell sites with for the next quarter,” Haynes said. “So we took channels that were slated for new sites and brought them out to existing sites and put them in. They increased the channel capacity in almost every site in the area to handle the off-load.”

More than 250 members of Verizon Wireless' network organization from the New England area worked around the clock to add capacity to handle the high demand. Neighboring cells picked up the slack from downed Manhattan sites. Engineers also set up 21 COWs in the New York City area to replace sites knocked out by explosions and fires, and to provide much-needed capacity for emergency teams. COWs also were installed at the Pentagon and in Pennsylvania.

Two of Sprint PCS' cell sites located near the disaster location that remained in service were redirected to add capacity for Ground-Zero efforts. Sprint moved two COWs and additional network equipment into Manhattan and put them into service. In addition, Sprint redirected a cell located in New Jersey to provide service for the NYPD. Four additional COWs were brought in to serve New York City subscribers. Five days after the strikes, Sprint PCS reported that 93% of its network throughout the greater New York City area had been restored.

Seay said AT&T Wireless deployed a temporary site at Ground Zero within hours of the WTC attacks. It also added extra capacity to sites serving the Brooklyn Bridge as “hundreds of thousands” of people walked across to leave the area after the attacks. When it couldn't get landline service, AT&T deployed point-to-point microwave equipment to provide backhaul facilities to its cell sites. It also brought in COWs near the Pentagon to boost coverage and capacity.

But COWs weren't always enough. Carriers also had to reroute traffic and redesign networks on the fly, which required a lot of creativity.

“What you end up doing is seeing where there are shifts in traffic patterns in the network that you don't normally see, and you look for alternate ways to provide that,” Seay explained. “So we might go out and put additional capacity in the radio network in that location. We go and add additional equipment to cell sites, do rerouting in the public network — to get calls out to long-distance carriers and local exchange carriers, we find alternate routes to deliver that traffic.”

Schaefer said Nextel's network-engineering teams added capacity to areas of New York City and Washington. They also rerouted traffic and employed other technologies such as spread-spectrum microwave and LMDS.

“When your main switching office is out, you've got to come up with other ways,” she said. “Situations like this spur need and creativity that people might not have imagined they had.”

According to Schaefer, Direct Connect and 2-way messaging offset a lot of traffic. During the heightened usage period, many customers relied on Direct Connect, the digital 2-way radio in Nextel's wireless phones, because it isn't connected to a public-switched network.

Seay said other creative methods of handling more traffic include shutting down non-critical network functions.

“When you open a big Excel spreadsheet on your computer, sometimes it has trouble processing,” he said. “If we have a large lift in processor occupancy, you can go in there and shut down your ability to measure traffic, and it gives more capability to the network to handle calls. You can turn off your ability to capture billing records. We would rather in this sort of instance say we're going to allow calls to go through as opposed to making sure we can capture traffic data.”

Preparing for the Unthinkable

Most carriers' network backup plans performed extraordinarily well, but this situation required a whole lot more than that.

“It's a situation that requires the ability to see the situation as it is right now and knowing that anything that you might have planned is going to be very helpful, but it isn't enough,” Schaefer said. “You have to solve for the scenario now, and you have to draw upon resources that before you might not have thought about drawing upon.”

Seay said you can provision a network to handle almost anything, but some things you just can't forecast.

“We spend time making sure we design our network to handle anything that we can anticipate and try to err on the side of overprovisioning it,” he said. “(But) you don't provision something to handle eight times the normal traffic in a small time period in a very localized area.”

Seay said no one had anticipated the need for providing protective gear such as masks and special clothing to engineers working in the asbestos-saturated air around Ground Zero.

Nor did carriers foresee another challenge: moving equipment to besieged areas without commercial air travel. Seay said AT&T's technicians were resigned to driving cross-country with equipment.

Haynes stayed up through the night to provide equipment lists to other carriers in the affected regions and help deliver it.

“I actually had a trucking company bill me directly to get equipment to another carrier because they couldn't get a trucking company,” he said. “There's a lot of things that went on that are just second nature, and you worry about later.”

But there's a bigger worry carriers have now: the idea that this event has changed their jobs forever.

“One of the real concerns we have is this is a different kind of event than what we've normally thought about,” Seay said. “At some point, you run out of resources and the ability to respond to those sorts of things. Unfortunately, we're living in a bit of a different country than we were three (months) ago, and we have to anticipate different kinds of things. You can prepare for a hurricane, but hurricane season comes and goes.”

Crash-Site/Course Coverage

When Flight 93 smashed into a field in rural Pennsylvania, rescue workers, law-enforcement officials and investigators who arrived at the scene soon discovered that the plane had also crash-landed on the fringe of wireless coverage.

Three Nortel Networks technicians responded to a request from Cricket Communications to deploy an F2 CDMA 1900 Metrocell immediately. They traveled from Pittsburgh to within eight miles of the crash site in Somerset, PA, arriving at 5:30 p.m. that Tuesday. The team was allowed onto the site at 9 p.m. and worked for the next eight hours installing a permanent-type cell site.

According to Kevin Bunger, Nortel Networks field installation supervisor, they installed a more permanent site instead of a cell site on wheels because of the area's sparse wireless coverage. The team used a 110-foot antenna tower on a trailer and deployed a normal cell site on a temporary platform they built.

“CDMA is easier to deploy because it all operates on the same RF frequency and uses spread-spectrum technology versus the other technologies like TDMA and analog AMPS,” he explained. “They use specific channels and have frequency plans, and the reuse on that has to be calculated out so that they don't interfere with other cell sites they have.”

Bunger said Nortel didn't have to coordinate any frequencies or move channels around at additional sites for the installation. It only had to work out the handoff borders.