Crisis puts capacity in spotlight

The dramatic spike in call volume that taxed telecom systems in the wake of last week's terrorist attacks in New York and Washington--and the resulting blocked calls, wireless outages and rampant "fast busy" signals--raises important questions about the capacity and durability of networks. As important is whether carriers can prepare for the worst.

Cover image: AP Photo/ Stuart Ramson Special Report Main Page

Given the circumstances, the nation's telecom network performed as well as-or better than-expected, and callers seemed to take the frustrations in stride, instinctively sensing what telecom service providers were up against.

"It's just sort of understood that this is what happens at times such as these," said Andrew McCormick, senior analyst for Aberdeen Group.

But should customers have had to make such concessions? Is it unreasonable to expect--particularly in a crisis situation--to pick up a handset, dial a number and reach the intended party on the first try? And shouldn't telecom service providers be required to build their networks to ensure that expectation is met?

Yes, yes and no, said Art Deacon, vice president of network operations for AT&T, which received 101 million more call requests on Sept. 11--431 million total--than its previous busiest day. Last week's circumstances, Deacon said, went "well beyond any standard telecommunications engineering practice."

Real Time Network Routing Real Time Network Routing [RTNR], a system patented by AT&T, plays a primary role when it comes to ensuring network reliability. read more

To manage capacity, AT&T employs an algorithm that establishes a value for the anticipated busiest hour. The network is then engineered to meet that load. The company also employs techniques such as call gapping and real-time network routing (RTNR).

Call gapping allows AT&T to terminate calls at the originating end when those calls are routed to end locations the company knows is offline. "That allows us to use network resources for calls that we can complete," Deacon said. RTNR gives AT&T the flexibility to send calls down paths that have available capacity and the highest probability of terminating the call to the intended recipient.

The result, added Deacon, is that customers were virtually assured last week of reaching their intended party within three attempts. However, he concedes that isn't good enough for a parent desperately trying to find out whether her child is safe.

'Internet usage was up maybe 20% from where it would be normally because of the crisis but not enough to interfere with our traffic in any way.' -Tom Evslin, ITXC

But Deacon isn't sure if it's feasible to reach the Holy Grail of being able to pick up a phone and--regardless of the circumstances--reach the intended party on the first try, every time.

"At the end of the day, I would agree that-on a personal level-[three attempts] isn't a reasonable performance parameter," he said. "But the issue is, what is the reasonable extent to go through to [achieve the ideal]?"

Deacon is sure the answer isn't an unbridled buildout of existing networks. To handle the peak call volumes of last week, AT&T would have to increase the size of its network by a third, something that simply isn't feasible-even if the capital required to do so were readily available.

"It would be like increasing the loop around Chicago to 70 lanes just to make sure no one ever again has to sit in a traffic jam," he said.

A recent Aberdeen study estimates carriers will spend $168 billion this year on their networks, primarily to replace equipment and conduct incremental buildouts. To construct the type of network that could handle the peak call volumes of last week, "you'd have to look at what has been spent on the network over the past 20 to 25 years and double that," McCormick said. "It would be in the hundreds of billions of dollars."

Even if a massive buildout were economically feasible, it probably wouldn't work, said Paul Bishop, market development director of telecom, media and networks for Cap Gemini Ernst & Young. "We could raise the percentage of calls that could get through by adding capacity, but you could never guarantee that everyone will get through. I can't think of any improvement target...that would make sense as a response to this incident."

Others can. Given its superior bandwidth characteristics, fiber might be part of the answer. But it also has cost and deployment limitations, and McCormick believes it could be decades before it has any kind of meaningful impact.

Voice compression might be a more feasible approach, said Russ McGuire, chief strategy officer for TeleChoice. Much of the work done with voice compression is related to voice-over-packet networks that can proportionally increase compression as the network load increases beyond what the transport infrastructure is designed to handle.

Glen MacDonald, a vice president for Adventis, believes next-generation technologies such as photonic switching and intelligent switching in real time might help, but they will only go so far. "This will enable calls to be routed with more efficiency and speed," he said. "But I don't know if a network service provider would engineer a network for an event that probably happens once in a century."

Another option is voice over IP. Christin Flynn, a senior analyst for The Yankee Group, was still able to get onto the Internet via a DSL connection and was busily communicating via e-mail from her office in Boston at the height of the crisis. She suggests that a VoIP connection would have allowed her to take those communications to the next level.

"People always talk about a POTS line as being a backup to voice over IP, but what about voice over IP being a backup to your POTS line?" she asked.

Flynn may be onto something as the country's data network appears to have performed well during the crisis. Many people had trouble accessing sites managed by news organizations, but only because those servers were jammed.

"Our frame [relay], ATM, IP and private line networks performed flawlessly the entire day," Deacon said.

However, VoIP is no panacea because calls eventually must terminate at the public network, said Tom Evslin, chairman and CEO for wholesale carrier ITXC. "Internet usage was up maybe 20% from where it would be normally because of the crisis but not enough to interfere with our traffic in any way," he said. "On the other hand, when calls were terminating in N.Y., the phone network itself was jammed."

Telecom service providers undoubtedly will assess last week's performance. In the end, they might be better served to accept the fact that there likely was nothing more they could have done and take pride in what was accomplished, suggested Bob Taylor, chairman and CEO of Focal Communications, which had some equipment and customers in the World Trade Center (Click here to see a graphic that shows telecom equipment and office locations in the World Trade Center).

"You can't plan for something like this. You couldn't solve this problem even if you knew it was going to happen," he said. "Some of the challenges are out of the industry's hands... but the network survived and it has done what it has been asked to do."

Said McDonald: "Without that communications infrastructure, the nightmare could have been worse."
Jim Barthold, Amalia D. Parthenios and Toby Weber contributed to this report.