Tolerant to a fault

Jan 1, 2002 12:00 PM, Dan O'Shea

Fault tolerance is one of those sneaky little puns. To most telecom people, it describes a network's reliability and resilience to faults. But in discussion about wireless networks, the temptation is to turn the phrase on its ear, and apply it instead to wireless users and their saintly tolerance for the network's persistent faults.

It is difficult to say who is more willing to accept that users will experience a certain number of dropped calls on wireless networks: the carriers, or the users themselves. Either through Pavlovian conditioning by their carriers or because of their own persistent nature, users seem to live with network imperfection. If the call drops, they just call back.

"The price for mobility has always been a higher tolerance of disruptions in the connectivity," said Joseph Neil, Technical Marketing Engineer at Cisco Systems. And for the most part, wireless carriers have thrived despite their services' patterns of imperfection. However, the emergence of wireless data may force the industry to think again about fault tolerance and what it really means.

"Wireless networks are not currently designed to be fault-tolerant data networks," said Neil. "Data over wireless to this point has been much more like a 9.6 kb/s connection is over wireline. It's really no fun at all, and there are not really any interesting data applications or content designed for wireless users."

That promises to change in the future with the transition to IP-based 2.5G and 3G data and Internet services. And in tandem, any theories about wireless users' higher tolerance for disruptions are likely to crumble: Whereas mobile voice initially was seen as a luxury, mobile data will, from the start, be viewed as a utility.

"Dropped calls are so frequent with mobile voice that it's considered an inherent part of mobile architectures, but users have been trained on data as a reliable desktop medium," said Neil. With very little data traffic so far, wireless users have not begun to change their attitudes toward network problems--but they will.

"Subscribers have come to have an expectation of immediate service from their data devices," said Jeff Rinscheid, director of product marketing and network operations software at Lucent Technologies. "They will be more demanding with data. They won't put up with outages and delays that stop them from sending files or e-mail. They will jump carriers more often if this happens. The churn rate will go up, and they will look for the carriers with the best promise of data reliability."

At the same time that users are getting demanding about data, wireless carriers likely will be struggling to integrate and manage new IP network overlays to support these services.

"Wireless carriers will encounter more network faults as they bring IP technology into their networks and introduce packet data services," said Neil, adding that these problems will not be caused by quality issues in regard to the packet technology itself, or by carriers' inexperience in trafficking packets.

Rather, the bigger fault tolerance problem wireless carriers will face with packet-based data services is more rooted in the traditional network shortcomings. They have not been engineered from the beginning with reliability in mind. Network coverage is not consistently strong over large areas, and air-interface connectivity can never be guaranteed during peak-usage hours in popular locations.

In addition to the engineering issues, carriers also have had problems in managing over-subscription to their services and handling the increase in traffic created by influxes of new users and services.

Two early-phase data services already have shown where fault tolerance falls short.

SMS, a pre-packet service leveraged on SS7 technology, took off as an initial data medium. But even though its capabilities limit the number of characters in each message, the service still has a tendency to choke up signal transfer points.

"The number of people getting on the network at any given time is what most figures into bandwidth congestion," said Matt Standish, product market manager at Concord Communications, a vendor of network reporting software. "When everyone is using SMS at the same time, that leaves quality of service a challenge."

NTT DoCoMo's i-Mode is another transitional content service whose popularity has given the wireless industry reason to believe in data's promise, but like SMS, it also has proved that carriers are not able to manage all the aspects of even an early-phase data service.

"DoCoMo's service was actually too successful, which led to fault problems. Older routers passing the data traffic to old MSCs couldn't handle the traffic. Reliability was not designed into the configuration," said Neil, whose company is working with DoCoMo to help improve service reliability.

These early experiences leave the industry in a precarious position as it begins the near-term move to packet technologies, such as GPRS and CDMA 1X. "The vision is for wireless carriers to eventually move to all-IP infrastructures to get better services," Neil said. "Building in 2.5G data capabilities is the first step in that direction. As 2.5G becomes ubiquitous, it will become clear that most mobile networks are under-engineered in terms of over-the-air interfacing, even in their hot spots. There's congestion on wireless networks because they are not engineered for wireline reliability. When you get dropped data packets, it will not necessarily be because there's a problem with IP."

In some ways, introduction of packet technologies will actually ease the lot of wireless carriers seeking to control fault tolerance as they launch data services. For instance, IP architectures are inherently more intelligent than traditional TDM networks because with IP, intelligence exists in each network element. This makes IP architectures easier to manage and packet network elements easier to tap for important performance data. Also, CDMA 1X actually allows more voice users to use the network along with data traffic.

Cisco's Neil added, "Once you get past the congested points in the network, IP overlays can be engineered to be extremely robust. IP no longer has generic problems of unavailability and dropped packets."

Yet, if those congested points exist, dropped voice calls and scrambled data packets will be a fact. Both IP technology and network management vendors believe the only way for carriers to improve fault tolerance on their data overlays is by better engineering their network foundations and existing air-interfaces to handle variable traffic loads.

"Carriers have to come up with design rules for data, and they are not the same as for voice," said Des Owens, president of Actix, a network management vendor. Still, he admitted they face an uphill climb into thick fog as they do so. "With the introduction of packet data, there are a lot of new terminals and applications, a lot of new complexity. We're all flying blind. There's a lot of new stuff to juggle."

For example, Owens said a carrier running a GPRS trial in Frankfurt, Germany, recently discovered how a planeload of Germans landing at the airport can quickly lead to a network bottleneck as all of their GPRS phones get turned on simultaneously. "That's why will see soft launches of wireless packet data. Right now, companies can't market it as an always-on solution."

Standish added, "With 2.5G and 3G, there will be content trends that occur at different times of day, just as there are network trends. Fault and performance management is very critical to a progressive rollout of content services."

Lucent's Rinscheid also said the learning curve will make it difficult for carriers to provision services as quickly as they do today.

"With data, you will have a harder time with service provisioning because you will be offering customers many more choices and services. Reps are already overwhelmed by the volume of new service activations, and operations people are probably not ready for that volume of service provisioning today."

Wireless carriers traditionally have been more reactive than proactive in handling network problems that arise from traffic overloads, service provisioning problems and the like. The era of packet data is not beginning much differently. Owens said Actix is usually called in after carriers have built data capability and done drive tests, but still have no real visibility into the performance of these services or the network elements running them.

"They are still in a troubleshooting and improving stage."

Said Rinscheid, "On the surveillance side, carriers can generously engineer their networks in preparation for 3G, but everybody will still get it wrong because we don't have the experience yet to know how much capacity data traffic will require. It will be hard to predict when a data call will be made or when data traffic will peak. Are carriers' routing table optimized for data? We don't really know yet."

Lucent Bell Labs has created a system called Oscelot that hypothetically optimizes a network based on the introduction of data services and traffic into the existing network. Rinscheid said the product was created with data in mind, and accounts for how it affects network fault tolerance overall. On the network management end, carriers also have to invest in surveillance solutions for packet network overlays.

"Knowing what is really broken is difficult. If you cut a fiber, Sonet lets you know about it. Packet gear can let you know when a connection has been lost, if an IP tunnel is lost," said Rinscheid. However, wireless carrier may have a tendency to keep their overlay experts separated and focused only on management of their own overlays. Instead, carriers should not invest in separate surveillance centers for IP, ATM, SS7 and other overlays. They should invest in a comprehensive NOC.

"What you need is a coordinated effort-centralized network management," said Rinscheid.

As packet overlays become more widespread, Rinscheid said packet controls will help carriers better manage changing data traffic patterns. These controls include call gapping, which spaces traffic to allows calls only once every 10 seconds, and advanced call routing, in which calls are sent through alternate facilities to balance the load.

Concord's Standish agrees with the importance of having integrated fault management, and said carriers will have to establish congestion thresholds and learn to perform root cause analysis. "You need to correlate groups of users and certain segments of the network over certain times of day. Wireless data networking will have usage trends, and that's what you need to pay attention to."

But, as Cisco's Neil said, the big questions for carriers seeking improved fault tolerance remain 'What's the cost?' and 'Will carriers want to pay it?' "Carriers are cutting costs. Are they willing to pay for a more robust network? Traditionally, they haven't been."