Code blue: network in distress:

Surging data traffic poses increasingly complex network management challenges. Testing will not only keep networks from flat-lining, but it might help differentiate carriers from the competition

Growing demands for Internet and corporate intranet access are forcing the test and measurement industry to address the most complex issues posed by the public network.

Industry vendors are now designing testing tools to meet the challenge of moving from an analog world to digital. Those tools provide greater network access for technicians and monitor dynamic traffic situations such as protocol matching, peak data loads and patterns indicating potential problems.

These traffic complexities will become more important as data traffic and quality of service (QOS) demands intensify. Traffic ultimately will have to be monitored according to its QOS priority level, says Steve Solberg, research associate at The Giga Group, Cambridge, Mass.

Yet carriers face a real paradox when it comes to QOS, especially on the public network that serves residential users. On one hand, carriers are being pushed to provide a higher level of service in a competitive environment. Yet long-standing regulations force carriers to file flat-rate tariffs for lifeline and second-line services.

Depending on how the network is engineered, the dilemma also means that the flat-rate consumer users may interrupt access for high-end business users.

Even before QOS becomes an issue, one industry analyst believes more work is needed to ensure that existing technology and equipment adhere to designed specifications.

"First, there is pure equipment reliability," says Michael Speyer, program manager in the data communications group at The Yankee Group, Boston. "Then we can talk about getting data from point A to point B. This brings up the whole question of service level agreements, which brings up the question, 'How do you offer differentiated classes of services, and how do you measure and feed back to customers?'"

The bad news is that none of these issues has been solved.

Perhaps because such critical questions remain, many carriers are reluctant to discuss the topic. Bell Atlantic formally declined to talk about it. BellSouth, SBC Communications and Pacific Bell did not return calls.

One of carriers' most immediate concerns involves state public utility commission mandates on minimum data service levels, which could require that carriers deliver 14.4 kb/s or higher speeds. That would be no problem if clear 6 kHz bandwidth could be obtained, says Peter Schmitt, division manager of Teradyne Midnight Networks, a Waltham, Mass.-based vendor of telecommunications test systems.

"But much of the network is not giving one clean kilohertz end to end," he says.

Bandwidth shortfalls Indeed, bandwidth shortfalls are among the most important data communications challenges.

"One of the biggest problems is to at least get the bandwidth that is advertised for modems," says Patrick White, a principal in the telecommunications, information, media and electronics practice of consulting firm Arthur D. Little Inc., Cambridge, Mass.

Testing can help ensure network capacity and operation. Carriers must identify long local loops, which tend to be noisy, and crosstalk problems that can cause humming. They also must ensure that equipment is operating at specified levels, says White, a former vice president of research and development at Bell Atlantic.

With data transmission, of course, any errors in the network are more serious than for voice transmission. "In data networks, errors tend to [affect] the performance a lot [more] than in the world of voice," says Steve Greer, director of marketing at test equipment vendor Telecommunications Techniques Corp., Germantown, Md. That means carriers need to test for data reliability to ensure at least the same quality level that is provided for voice transmission.

"There is a very large economic incentive to be able to get a higher data rate through existing plant," says Rex Stevens, general manager of transmission tests and cable network analyzers at Tektronix, Wilsonville, Ore. "If you can establish clear testing parameters, for example on a twisted pair line, to certify that it will carry DSL, there's an economic advantage to use that existing plant."

Because of network testing's importance, carriers and end users are fueling healthy demand for test equipment. Revenues of $840 million in 1996 are expected to surge to $1.2 billion by 2000, according to a study by research firm Frost & Sullivan (Table 1).

Peak load testing Carriers must predict how their networks will run at peak loads, which vary tremendously for data communications.

A small company on the West Coast whose Web site address was placed prominently in an Internet search engine provides a perfect example. Virtually overnight, the volume of traffic into the site increased by 300%. The company struggled to add capacity, but its site ended up being inaccessible for at least one day.

"If you want to provide service perceived as world-class, you have to be able to handle peak loads gracefully," Schmitt says. "If one of your customers has some level of success [with its Web site or data traffic], nothing is going to kill that faster than the network breaking down under the load."

Historically, some carriers have provisioned for future capacity by using rough guidelines of, for example, 10 users per dial-in port-a "black art" process, as Schmitt calls it. Today, estimates aren't enough.

Testing for future projected peak loads requires an emulation approach. New systems can emulate activity on hundreds or thousands of lines with an arbitrary profile of those connections. "You throw those at any part of the network, turn up the load until you see what breaks," says Schmitt.

Taming the switch The switched nature of the public network poses a dilemma in itself. "Because these are switched services, it is hard to do long-term monitoring when services are either on or off," says TTC's Greer.

Carriers are responding by monitoring and gleaning information from switches. They also are using tools that test for protocol mismatching and misconfiguring when technology such as ISDN is deployed.

Today, more carrier customers with high-volume data needs are directly accessing the backbone, which typically operates over frame relay or asynchronous transfer mode. In these environments, LAN/WAN analysis tools have added value, says Greer.

Carriers are also coping with the complexity of end users' environments. "A lot of people are using the network in ways it was never designed [to be used]," Greer says.

This overload of users is straining voice networks, according to Hewlett-Packard Co. Working with four local exchange carriers in the U.S., HP conducted studies earlier this year using its AcceSS7 network monitoring system to pull out SS7 network data that revealed some disturbing trends in switch use.

For example, a typical call to an Internet service provider averaged 30 minutes, but some calls lasted 24 hours or more. For some switches, the collected data indicated that a small number of heavy Internet users accounted for a high percentage of the switches' minutes of use.

"Clearly, there is a divergence between voice and Internet traffic patterns," says Tom White, vice president and general manager of HP's Communications Test Solutions Group. "By using SS7 data, telcos can demonstrate to regulatory agencies the level to which Internet traffic affects [QOS], thus paving the way for implementing solutions to alleviate congestion."

QOS is coming to the forefront, according to Todd Benjamin, vice president of sales and marketing at testing vendor Ellipsys Technologies Inc., Soquel, Calif. "With deregulation and competition, more [companies] are involved in providing these services," he says. "There is more attention on quality of service."

Furthermore, the explosion in the number of lines and data traffic has not been matched by increases in engineering and technician staffing (Tables 2 and 3).

"You can hear the traditional analog network engineers screaming bloody murder every day-every time a new customer is sold a line primarily for data or fax use," says Paul Zagaeski, senior industry analyst for The Giga Group.

Test and measurement companies are responding by developing tools that are sophisticated but not as complex as the protocol analyzers that engineers use in carriers' test centers.

TTC's second generation tools let field technicians turn up service, and they play a strong role in installation, maintenance and troubleshooting, Greer says.

Tekelec also focuses on powerful yet easy-to-use systems, according to Dan Bantukul, a product manager for the company's Intelligent Network Diagnostics unit, Calabasas, Calif. "The [transmission] technology is going to get more complex," he says. However, he points out that testing systems are being designed to shield users from having to know all the bits and bytes of the technology.

Tektronix's Stevens says his company has added the ability to conduct signal-level tests, including protocol testing for its equipment. The company is adding greater sophistication to its testing tools so they will be simpler to operate and require a minimum number of operations yet test for a variety of complex problems.

Smart systems Carrier networks have evolved from circuit-switched links and now are overlaid with technology such as ISDN. Consequently, testing equipment must identify not only transmission problems-such as breaks and signal fluctuations-but also protocol and interface problems.

The first priority is the transmission itself, says Cecil Boyd, vice president and general manager of Tekelec's Intelligent Network Diagnostics unit. Typical problems that require the higher-level transmission testing involve local number portability or wireless users who cannot roam, he says.

Carriers also need test equipment that can give a network-wide view, Boyd says. The next generation of testing technology will provide that capability-along with sophisticated signaling and protocol testing-at a cost of about one-half that of systems on the market today.

HP's White agrees. "It's not enough to make sure that the transmission path is there," he says. "You have to make sure that the protocols between the two end points are operating properly."

Analyzers now can pinpoint problems with protocols or data internetworking, and testing tools simultaneously monitor transmission and protocols.

"On a day-to-day basis, if there is a problem turning up systems or connecting two LANs over a WAN, [the equipment] gives complete insight over every layer of the protocol stack," Greer says.

"Equipment and networks are getting a lot smarter," agrees Steve Adams, director of market planning at CrossKeys Systems Corp., a Kanata, Ontario-based software vendor that develops and markets telecom management systems. He cites frame relay and ATM switches as examples of today's intelligent networks.

Carriers must store the performance test measurements to find patterns that could cause trouble over time. That requires data warehousing capabilities associated with testing and monitoring equipment. Some data can be analyzed in near real time by today's testing systems. Other data can be stored, enabling carriers to detect trends that may indicate conflicts, degradation or incidents that occur at infrequent intervals. Data warehousing systems can store hundreds of millions of records and still deliver sub-second reporting.

Once data communications problems have been detected in any part of the network, carriers respond immediately. Testing systems can automatically call for re-routing around trouble spots, a process dubbed "cross-control."

Testing: the new revenue builder With data demands growing even before digital subscriber line speeds are deployed, carriers face the proverbial moving target as they try to meet those demands and prepare for the future. That is forcing carriers todeploy whatever they can to get the job done, says The Yankee Group's Speyer.

In this environment, tools that test a network's ability to handle projected future traffic loads are essential. "The point of our products is to proactively identify capacity or performance problems with the network at any given point in time, so you can effectively plan to avoid customers having poor service because of it," says Teradyne's Schmitt.

As data traffic continues to grow and service quality becomes critical, some believe carriers will see testing not so much as an added expense but as a marketplace differentiation and a way to build revenue. Carriers can use test results to show users proven levels of service.

"There is a real need to understand the performance of a network before it is engaged in a service," says Barry Phelps, president and CEO of Netcom Systems, a provider of performance analysis tools.

"These tools are increasingly important to both manufacturers and carriers from a [quality assurance] standpoint," he says. "They are being used in production tests to demonstrate how new technologies interoperate with legacy products. The results of these tests become an important part of the marketing effort, and they can indeed help bring products to the market sooner."

Circuit

* Bit errors

* Call durations

* Drop-offs or cut-offs

* Levels

* Intermittent hits

Data

* Retraining

* Retransmission

* Cyclic redundancy check errors

* Connection speeds

* Response time

Network

* Busy routes (interoffice)

* Failed call attempts (station)

* "Hung" or out-of-service trunks