Integrating Performance & Fault Monitoring

Your network is the vehicle that drives your business. By providing constant care, you think you are doing all you can to keep it performing at top speed. But perhaps your network could be doing a more efficient job.

Integrating two important elements of network management -- performance and fault monitoring -- can mean the difference between staying in the lead and fading in the backstretch, according to Ed Kurzenski, Objective Systems Integrators (OSI) director, wireless solutions, operational support systems (OSS).

"Integrating real-time and historical analysis capabilities into something you can use across the enterprise is one of the critical issues for carriers," he said. "[If you do not], it is analogous to driving a car with no speedometer, no tachometer, no oil-pressure gauge and no water-temperature gauge. That is the way many networks are being run. Someone is strapped in a seat and their foot has the gas pedal to the floor, but they have no idea how the engine is performing, and they are just pouring gas in."

Streamlining performance and fault monitoring allows systems to share information faster and more efficiently, and more effectively monitor alarms in a multivendor environment. Increasingly, many carriers are simplifying their networks by combining these systems. The biggest benefit is a centralized monitoring system that provides relevant, real-time information you can use to keep your network running smoothly.

It's All About Information Why are more carriers interested in integrating performance- and fault-monitoring systems? Because getting the right information in the right form and at the right time can mean the difference between a functioning network and a fault-prone one.

"[You must] make sure you have systems that ensure the highest quality of the network," said Deepak Kanwar, Lucent Technologies communication software division director of network surveillance.

Network managers need both kinds of information (fault and traffic) coming together at one pass. They need integrated systems that share information.

Doug Strombom, US West director of operations, said it is critical to provide information to everyone in your company, and streamlining network-management systems helps considerably.

"This is not information for a couple of engineers," he said. "It is information for the business, and it needs to be married to all of the other information, such as who are your customers, what are your sales distribution channels, what kinds of trouble customers are reporting."

Strombom said carriers need information in centralized repositories so that everyone can see how the network is performing and what faults are occurring on it.

"It is not just the local specialist anymore who needs to know that information," he said. "It is like the central nervous system -- you've got people responding in real time to service problems, thinking more about how they are going to invest in the future, but everyone needs information. We do not subscribe to the idea that network management is technical or is something for a few people sitting in a network-management center."

Carriers that have integrated performance and fault management know the benefits. One-2-One is a service provider in the United Kingdom that employs US West's WatchMark network-management software solutions. The carrier has nearly 100 employees across all departments that share a common view of the network, and everyone can retrieve information about the network with the click of a mouse.

"Initially, this was risky that you would let the director of operations or the vice president look at what's really going on out there," Strombom said. "But it really helps everyone because even the COO can short-circuit problems. The network-management center is responsible for fixing all of those problems, but the communications with the rest of the organization is handled automatically."

When fault- and performance- management systems share information, you can leverage your real-time response capabilities (the fault side) with the historical and trend-analysis capabilities of performance-management systems.

Most carriers have too much data from too many sources. Kurzenski said that most network operators have anywhere from two to 12 different data repositories storing parallel or even the same information. Integrating systems can minimize that storage capability to just one location.

"Having only one place is more efficient than having to manipulate from every place," he said.

Because most network-management systems encompass a billing system, a configurations system, an inventory system, and a fault and performance system, changing a network element means changing several more systems on the backend. According to Kurzenski, the capability to run one application is more efficient than what most people are doing today.

To Integrate or Not to Integrate? Lucent's Kanwar said the ability to integrate provides a complete view of the network and correlation of information, which helps locate the root cause of problems. It also allows carriers to pro-actively address faults as well as traffic congestion before a major outage occurs.

"You need to be able to look at the entire network and see if there is a fault," he said. "And in case there is, you want to move your traffic around that fault. That is a critical need all carriers have."

Kanwar said fault and performance monitoring needs to allow carriers to integrate all of the various pieces of equipment and present a seamless view. With many service providers' (wireless and wireline) technologies converging, you want a vehicle that will allow you to manage the network from an end-to-end perspective, he added.

Aaron Glass, AirTouch Cellular director of systems engineering, agreed that simplifying the network is an obvious benefit. For example, if you have a cell site or some other network node that detects an alarm, you can click on that node and then telescope down to the specific element that is in the alarm state.

"The advanced systems will even allow you to go beyond that, telescope into that component and look at it at the board level or component level," he said. "You also can integrate the response or call-out procedures for that particular component right there in the system, so it is accessible from your mouse."

Most integration solutions are much faster alternatives for carriers because they provide both fault and performance data on one screen. Glass said a status screen can list a circuit identifier, the carrier, the trouble-reporting number, the account number and the information that whoever reports the trouble needs in order to streamline the recovery process.

"Outages affect performance, and if the systems are integrated, then you can use the actual data to generate your performance statistics," he said. "When you evaluate your performance statistics, the answer to 'Why aren't they better?' is right there because you can examine the outages that you had that impacted them."

This, in turn, makes the improvement process easier and faster.

"Single-screen access is like 1-stop shopping because you do not have to get up and run around to troubleshoot the network," Glass explained. "All of your documentation, diagrams and call-out procedures are in a single location."

Another plus is that integration makes comparing network information easier.

Strombom said US West's CDMA customers can second-guess the information presented by the network by comparing faults to performance and configuration data to customer trouble reports, and they actually can pinpoint service-quality impairment instantaneously with an integrated system.

Curt Whitney, BellSouth Cellular's director of operations, said integration makes relating data easier.

"If you have a common methodology of handling faults that is also used to display and report performance degradation, you do not have two or three ways of looking at the same information," he said.

Cause for Alarm? More and more operators are looking for umbrella solutions that can span multivendor environments. Whitney said integrating multivendor performance- and fault-monitoring functions would enable carriers to respond to alarms more effectively. For example, BellSouth Cellular has systems in place that collect alarms from various systems, but it has problems correlating the alarms in terms of time and applications.

"Because we have different vendors, we have to work with each vendor to understand what an alarm really means, what it is indicating, how serious it is, and then correlate that with other alarms that might come in from other systems," Whitney said. "One of the fundamental issues we have is alarm correlation, either within a single vendor's product or across vendors. You don't want to respond incorrectly, but you might get multiple alarms from the same fault."

Often carriers do not receive one clean alarm that alerts network technicians to the exact problem. And when service providers get an alarm, a second alarm message is critical to inform network operators that the fault that caused the alarm cleared, whether by manual intervention or some automated process.

"You've got to correlate an alarm-clearing message with the alarm-generation message, and the degree to which they do not match up -- and they never do -- is another measure of how well you can manage faults," Whitney said.

BellSouth Cellular manages its alarms through a decentralized method that effectively uses the paging function of the fault-management system. BellSouth Cellular has built intelligence into the local alarm-monitoring system, so it can notify the people responsible for that network element, and they can respond directly and quickly, he said.

Whitney said BellSouth Cellular has worked with its vendors to enhance the paging functionality and flexibility to make it effective. For example, the service provider uses time-of-day-based paging, functionality-based paging and geographical-based paging.

"Rather than paging a person at midnight to get them out of bed to come find a problem, the alarm calls a separate, intelligent fault-management system that fixes the problem automatically," he said. "Where we're going in the future is to bring more intelligence to fault management."

Other carriers find navigating through multivendor systems just as challenging.

"[AirTouch] has a wide variety of different network elements that do not share fault or performance information," said AirTouch Cellular spokesperson Patti Finley. "One of the challenges carriers face is to have one type of intelligent network system that can take all of the different inputs from a variety of network elements and come up with one root cause."

Currently, Finley said, the (network manager) has to make the correlation between different elements and try to figure out what is happening. Ideally, in the future there will be a network device that will consider different alarms and display the root cause for the network technician.

Whitney explained that service providers with a central monitoring system can better integrate potential alarms from the hardware with the performance degradation that they may cause.

According to AirTouch's Glass, the value of integration is that carriers in a multivendor environment can have single-screen control. This means one screen that monitors the entire network and all of the key components, no matter what vendor they come from.

"The real objective is a single platform that gives total network visibility," Glass said.

If carriers are operating in a multivendor environment, the goal is to have a platform that will allow visibility to all network components from a single window, beyond just the blinking lights or a text message that says a certain site is down, he explained. It is important to be able to click on an icon and zoom in to a specific network component and get the information necessary for speedy resolution without referring to other documents or people.

Kanwar said Lucent integrates fault monitoring and performance monitoring at two levels. One level ensures that equipment from different vendors is integrated. The other level of integration is making sure systems have the same level of reliability as the network.

Although Kanwar admitted that things can get tricky if you run two systems from two different vendors, he said it is not a big problem. Lucent, and many other vendors, can provide applications that allow integration regardless. And many service providers can integrate systems by themselves.

Glass said AirTouch's implementation of such an integrated management system is "in progress."

BellSouth Cellular also may implement a centralized performance and fault-management system in the near future, Whitney said. The carrier has integrated systems in some regions, but Whitney said that integration has not been a universal effort across all of the carrier's networks.

"We treat faults and performance degradation as an alarm-generating condition, but we have not integrated them into a single system," he explained.

But, he said, integrating monitoring systems is something BellSouth Cellular is evaluating, along with building a central network-operating center to administer alarms and fault management for the entire network.

Integrating your performance- and fault-monitoring systems is not automatic, nor is it the only way to improve network efficiency, but it is a growing trend.

"[Integration] is something that will be more common in the future," Whitney predicted. "As we become more network-based in terms of providing services, integrating alarms and performance measures will become more critical."

Because integration is customer-specific, cost varies, said Deepak Kanwar, Lucent Technologies director of network surveillance.

"Many vendors have integration capabilities built into systems already," he explained. "So if a carrier bought both of Lucent's systems, there is no additional cost involved in integration."

Streamlining network-monitoring systems can be more cost-effective than running two systems. With integration, you can reduce your costs because you do not need as many people or machines to manage your network, Kanwar explained. And once you have integrated fault and performance monitoring, you can run diagnostics automatically and dispatch technicians to fix a problem instantly.

These benefits do not come free.

"The only con is the cost of integrating -- if you buy a system that does not have the capability," he said. "But most carriers find that the payback analysis is very positive from an integration standpoint."