More with less

Rapid advancements in computing hardware and software have revolutionized much of the business world, making difficult tasks easier to handle and creating new capabilities.

For telecommunications providers, more powerful hardware and software is streamlining - and, to a degree, combining - two crucial functions for success: network management and test and measurement.

Telcos have long been able to monitor their networks for signs of trouble. Alarm systems that notify providers of trouble conditions in the network so that technicians can be dispatched to identify and fix the problem have been a mainstay of network operations for many years.

But recently, equipment manufacturers have begun to offer greater intelligence in network infrastructure as well as monitoring and management tools. This has allowed telcos not only to identify problems as they occur, but also to troubleshoot more problems remotely so that, if necessary, technicians can be dispatched with more detailed information in hand.

In some cases, the newer network elements and management tools can exchange and correlate data to identify possible problems before they occur. Hewlett-Packard began developing intelligent network management and testing equipment in the late 1970s when it hooked space division multiplexing equipment to a computer for testing and monitoring in Scotland, says Tom White, vice president of HP's communications testing systems group.

"What we did was marry our information expertise with our computer expertise," White says.

Much has changed since that initial effort. Perhaps most critical for HP has been the development of SS7. HP's modern network monitoring and testing systems use SS7, gathering data on network performance and performing testing functions "out-of-band," freeing bandwidth on the customer network.

HP's method is to "distribute data gathering sites throughout the network and gather data in real time," White says.

Another company that believes monitoring and testing via the signaling system is the most efficient method is Inet Inc., Plano, Texas. Inet's GeoProbe system also takes advantage of telcos' SS7 networks to continuously monitor and gather data.

Development of intelligent networks has exploded within the last 10 years, says Kevin Keough, Inet's vice president for new business development. Before the late 1980s or early 1990s, testing and monitoring functions were primarily localized, he says.

"It's likely you had a box, meant to be lugged around by a technician, to monitor certain equipment," Keough says. "That was really all you needed back then." But with increasing interconnection of networks and with all the new services that have gained popularity in the last decade, the demand for keeping networks up and running has increased.

With the addition of enhanced services and huge growth in data traffic, not only has the potential for problems grown, but network downtime also can mean significant losses in network traffic. And that means lost short-term revenue and lost long-term customer confidence for the service provider whose network fails.

Keough points out that even as recently as the late 1980s, the major interconnections were between a few interexchange carriers and local service providers.

"Today, you've got little companies plugging in, Internet service providers plugging in - everybody from mom-and-pop operations to major telcos," Keough says. "It's very difficult now to localize a problem.

Due recognition Aiming to bring a higher level of consistency to network management methods, some telcos, including BellSouth, are reorganizing their testing and network management divisions.

Until 1993, the company had handled testing, monitoring and network management out of 42 centers spread throughout its service territory, according to Eddie White, BellSouth's network surveillance manager. In 1993, the company kicked off a re-engineering project that culminated in 1995, with all such functions concentrated in two network reliability centers and one network monitoring center.

"Now we're regionalized," White says, adding that the network reliability functions, for which he is responsible, can be handled from a single center. "But for redundancy and reliability, we have two," he says.

BellSouth's network reliability centers primarily monitor and respond to alarms within the system, but they also perform tests on newly deployed equipment before it is made available to customers. They also work hand-in-hand with the network monitoring center that gathers cumulative reports generated from network elements, White says. Both arms constantly communicate to alert each other of potential problems and to keep everyone apprised of what's being done about existing trouble.

One crucial change is that switch and facilities monitoring are now done side by side, whereas under the 42-center structure, they had been separate, he says. And by centralizing the identification and solution of problems, BellSouth has essentially centralized quality control.

"We try to carry best-case scenarios throughout the network," White says.

The methods that equipment providers are developing to allow carriers to deploy intelligent networks aren't all the same, though. While companies such as HP and Inet rely on the out-of-band signaling network to perform their functions, other companies have developed systems that work in-band.

Applied Digital Access of San Diego is one such company.

"We've built the company on a family of network elements," says Kevin Pope, ADA's vice president of development engineering, network systems. "Our focus has been on test access and test resources as well as network performance monitoring.

ADA's equipment performs full-time performance monitoring on DS-3 signals as well as embedded channels, and it stores the data for as long as 30 days within the network itself, Pope says. And although the functions are performed in-band, they are non-intrusive to customers, he says.

The company has been developing its technology for five or six years, says Kirk Moir, ADA's director of systems engineering for network management. He believes its functions take place at the key points in the network: interconnections between customer networks and telco networks.

"It sits at that key border and allows the telco to identify if a problem is in the customer network or the broad carrier network," Moir says. "Only about 30% of problems occur in the customer network.

Pope says there are two key elements in ADA's products: The placement at key network intersections is one, and the ability to correlate the data gathered into a usable format is another. The latter is not only a key element for ADA, but it's also crucial for all such systems in order to make sense of the data they gather (Figure 1).

"You've got hundreds of elements in the network sending information," Moir says. "It's hard to look at that information and say how it's affecting the customer.

Controlling information overload Developing a system that can monitor, troubleshoot and produce effective reports spurred a recent joint development agreement between Telecommunications Techniques Corp., Germantown, Pa., and Chicago-based Clear Communications, which makes customer network management and intelligent surveillance software.

The Telecommunications Act of 1996 radically changed the outlook for performance management software and equipment, says Clear Vice President Steve Subar. Between 1984 and 1996, 12 companies - the Bell regional holding companies and the major IXCs - spent the majority of money, he says. And they had three primary sources on which they spent those dollars: Bellcore, Bell Labs and their own in-house initiatives.

But the telecom act "has created a whole new set of requirements," Subar says.

To meet the act's stipulations and to keep up with the demands of customers - including other telcos - providers are being forced to open access to portions of their networks that previously had been used in-house only. That opens up a new can of worms for network management.

"They need to find a way to do what's called flow-through," Subar says, referring to provisioning some services through the operations support system. "They need to add new services to their legacy systems.

While Clear had developed a management system in its Clearview Network Window application, it didn't have a testing component to help the telcos identify and repair problems.

"It's a lot easier to say than to do," Subar says of performing test functions remotely. "The idea of one vendor providing all such services is not a realistic expectation.

So rather than add a testing module developed in-house, Clear is integrating TTC's Centest 650 centralized test system into its Clearview application. That way, TTC's test functions can be initiated from the same window that accesses Clear's management functions (Figure 2). Under the agreement, TTC will distribute Clear's products, targeting competitive access and wireless service providers.

"Carriers can integrate individual products and know they have the support of TTC and Clear for a turnkey solution," Subar says.

TTC has built its remote testing equipment to take advantage of the presence of digital cross-connects, says Rush Johnson, solutions program manager at TTC. "We use them as test access points," he says.

Competitive issues and the need for efficiency in deploying and monitoring new services are driving the move to centralized testing and monitoring, Johnson says. Through the deal with Clear, "we're going to be able to impact both of these drivers," he says. "With the relationship with Clear, we've got the proactive and the reactive ends covered.

Even as preventive measures become more widespread and effective, Johnson says he doesn't expect testing to fall by the wayside.

"I think what's proven is that you need both," he says. "To fix troubles, you're always going to need to test, and you always need to test when you turn up new services.

New definitions for measuring service availability from the Network Management Forum also are beginning to find their way into network management systems. Hekimian, which offers a family of operations support systems used by telcos for network testing, is incorporating the new NMF standards into its PM Integrator, says Director of Marketing Bill Duncan. The PM Integrator is a software package that can be bundled into Hekimian's service management architecture.

Developing an intelligent network for performance monitoring is so important to U S West that it has undertaken a program, called Network 21, to address the issue in a series of projects.

Contrary to the traditional alarm/response method of network upkeep, U S West is developing an advanced warning system, says Dave Stubbert, director of the Network 21 program. The equipment it buys now, he says, includes continuous performance monitoring capabilities.

"It's telling us, 'This equipment is showing signs of being in trouble,'" says Stubbert. "It's like a nervous system. Now, instead of waiting for something to suddenly happen, we can see what's happening on an ongoing basis.

But no RHC can be expected to have top-of-the-line capabilities throughout its network simply because of its sheer size, says Stubbert.

"We've got a mixed bag out there," he says. "From a practical standpoint, you still have to use testing" to identify and repair problems.

But efficiency is improving. BellSouth's White says a field technician needs to be sent out on just 13% of the carrier's alarms. Still, the company maintains an army of technicians throughout its territory.

Perhaps the most difficult task for telcos as they further centralize testing and management is finding the optimum number of field technicians to handle problems that can't be identified or fixed remotely.

While White declines to comment on how the centralization of monitoring and testing has affected BellSouth's force of technicians, he admits that it doesn't take as much manpower to do the job as it once did.

"We're doing more with less," he says.