NETWORK VISIBILITY: TAKE TWO

After years of shooting network troubles, employing the hard-won best practices of technicians from generations gone by and weaving in the latest in software sleuthing, today's script-slinging techs must sometimes feel about network management vendors as Anakin Skywalker did when he said to Obi-Wan: “I have nothing left to learn from you.”

But there is always room for improvement. The basics of troubleshooting haven't changed much over the years: acknowledge, assess, isolate, identify, minimize and fix the problem.

Most network technicians are familiar with how young Skywalker's attitude served him and are willing to give new techniques a whirl, as long as those techniques don't remove them from the process. They like more processing power, a little more memory or commonality of language. They really like reports, and the younger ones love the bells and whistles — the graphics.

But what all network technicians like the most is visibility. In the following two examples of small yet essential improvements in network troubleshooting, it's evident that the next greatest thing is often an exercise in subtlety.

First, BellSouth and Lucent Technologies teamed to improve the troubleshooting capabilities of BellSouth's wholesale customers by providing visibility into its packet network. Next, Agilent Technologies completed the third stage of a distributed analysis platform that gives technicians network-wide visibility into multiple technologies simultaneously.

BELLSOUTH AND LUCENT TECHNOLOGIES

Until now, BellSouth's wholesale customers, primarily interexchange carriers (IXCs), had no visibility into the elements that comprise its ATM and frame relay circuits once those circuits disappeared into the local cloud. Their first indication of a service-affecting problem usually came after it had already affected customers. The IXC would then have to engage the LEC in helping it troubleshoot such problems.

No more, says Jose Blanco, director of marketing for BellSouth Interconnections Services. BellSouth recently introduced its Network Visibility Service, an optional service package that carriers can buy along with new ATM and frame relay circuits that provides partitioned access to performance data on BellSouth's network. “Being able to quickly isolate trouble, particularly in the IXC environment with RBOCs or CLECs on either end, is a big deal for them,” Blanco said.

It's such a big deal that BellSouth is hoping visibility has a price. For $12 to $14 per circuit per month, a carrier can monitor real-time fault and alarm activity, view real-time network performance and historical reporting on BellSouth's Fast Packet network. BellSouth will roll out the service in stages beginning with two Service levels. The first level includes fault management and on-demand statistics; the second adds performance reports.

“We aren't interested in being a low-cost provider,” Blanco said. “We want to offer world-class, predictable performance and, frankly, charge top dollar for it.”

Over the next six months BellSouth will collect data to determine the time and expense saved by faster troubleshooting, and the carrier will build that into service level agreements that its wholesale customers can offer their enterprise users.

For Lucent, which co-developed the NVS, this requirement is growing. And the driver, unlike the cost cutting which drove most buying decisions last year, is to generate revenue and retain customers by adding indispensable features such as visibility, said Tom Blackie, Lucent's director of product architecture.

“BellSouth has always been aggressive and remains aggressive in pursuing new opportunities based on either technology [improvements] or generating new revenue and they have embraced this customer network management feature,” said Bob Logan, director of optical network management for Lucent.

The customer network management feature is not a custom product from Lucent. It is part of Lucent's Navis iOperations software suite of network management solutions. “We use different modules in different ways to develop individual solutions for different service providers,” Blackie said.

NVS is currently available for the following Fast Packet services — identified by nomenclature only a mother could love and only the IXCs and the FCC's tariff department could appreciate: XAFRS, MSFRS, XAATMS and MSATMS. (To the uninitiated, those stand for “exchange access frame relay service,” “managed shared frame relay service,” and “exchange access ATM service and managed shared ATM service.”)

“As networks evolve toward [Layers 2 and 3], the network management tools have to be there as well,” Blanco said. BellSouth's NVS will soon include customer provisioning of private virtual circuits, automated trouble ticketing, alarm correlation for both Layer 1 and Layer 2 services, and operating level agreements.

DEUTSCHE TELEKOM AND AGILENT TECHNOLOGIES

With Deutsche Telekom as a development sponsor, Agilent introduced last week a software tool called the Network Troubleshooting Center. The name elicits visions of a grand solution with hundreds of square feet of monitors, servers, alarm panels and technicians. But it's quite the opposite: The NTC is a software solution that provides a network-wide view of performance to mobile technicians using a single remote device — a capability usually reserved for the network operations center.

Agilent developed the NTC to give technicians access to distributed analyzers from any location on the network through reconfigurable interfaces such as T-1, Sonet and gigabit Ethernet.

“Typically, a portable tool gives you a great segment view of the network, but it doesn't give you a network-wide view and doesn't let you get to inter-related problems on the network as fast,” said Steve Witt, vice president and general manager of Agilent's network systems test division.

Agilent claims that field technician with such broad visibility into network elements and analyzers can reduce the time it takes to start solving a problem from about four hours down to 20 minutes. The NTC also can reduce the overall mean time to repair to 55 to 145 minutes down from 240 to 720 minutes.

DT engineer Jeurgen Hansen said having to troubleshoot end-to-end applications rather than segment-by-segment technologies made the job more difficult. “[However], with this central troubleshooting tool, we will be able to identify and fix network problems from a central point more effectively and faster.”

Centralized and remote testing is nothing new, Witt admits. But providing mobile technicians with a single PC-based tool with interfaces that can be reconfigured “makes network troubleshooting not just fast and simple, but available all the time,” he said.

NTC uses Windows-based software than monitors multiple analyzers simultaneously and correlate RMON1, RMON2 and MIB-II information from various network elements. The NTC gathers this information from a number of different access points depending on how a network operator deploys it. The software can be deployed on a PC or on a rack-mounted device connected to a T-1 or E-1 interface, as well as 10/100 Ethernet, gigabit Ethernet, ATM, frame relay or packet over Sonet. It then aggregates performance across the network and across time to aide in trouble isolation.

It's the ability to aggregate performance information at different network layers across multiple network technologies that DT's Hansen said is what makes the NTC such a strong tool. DT has implemented NTC in two locations for troubleshooting both its carrier and enterprise networks. Both networks have been migrated from token ring and FDDI technologies to Ethernet and gigabit Ethernet backbones. NTC will eliminate the need for on-site technicians and improve customer satisfaction through fewer outages. “The more efficient use of our technicians will reduce the troubleshooting time significantly,” Hansen said.

Access and visibility are not by themselves problem solvers. “Just looking at statistics bubbling out of the network can be overwhelming,” Witt said. So the goal of the NTC is not to inundate the technician with information, but to help guide them to where they must look to identify a source of a trouble.

Witt said network engineers like to solve problems and like troubleshooting, but with so many work force reductions, there is more pressure on engineers to solve problems for which they previously were not responsible. “They can't do it without having access to information or visibility into the network,” he said.

As reliable as current practices and capabilities have been, the networks and services running over them continue to increase in complexity. New tools and services must do the same while seemingly doing the opposite.