Closing in on QOS

It wasn't supposed to be like this. IP was supposed to make network management and maintenance so simple. What, after all, could be easier than capturing a single protocol layer of packets, correlating them, then performing some fancy algorithmic interpretation?

Just about anything, as it turns out. IP is not delivering on its promise of simplicity; it's making things more complicated, both from a technical and a business perspective, said Karl Erik Traberg, director of product marketing for NetTest.

His NetTest colleague Thomas Jensen, vice president of sales and marketing, said marketing efforts in the late 1990s fooled us when they said we would have one network with one IP layer and one optical layer onto which we could throw services, and everything would work seamlessly with all distance and time removed.

“We thought it would all collapse into something very simple, but it never did,” Thomas said. “If you look at IP today, you have more protocols in the network than you ever had.”

And the protocols are just the eyes of the hippo. The value chain is a convoluted mess, and the measure of service quality has changed from being a network-centric ideal to a customer-centric perception.

“But that's all good for NetTest because we make solutions to solve those problems,” Traberg said.

It's also good for companies such as Anritsu, Comarco, Tektronix and other vendors of wireless test gear. But saying service quality is measured from a customer-centric point of view might be taking the concept a little far, despite the spin vendors put on their product capabilities. They have made strides with new tools: note Comarco's QualiPoc solution, which uses actual customer handsets to test the network and services, and NetTest's probe-based solutions. Both are essential tools in the never-ending battle to prove out the network. But testing service quality is a multi-pronged process that begins long before a service hits the customers' hands.

The wireless network testing process is complex. There's benchmark testing, load testing, drive testing, simulation testing, network performance testing, application performance testing and device testing. There's testing in a pre-deployment phase, a post-deployment phase and the phase in between called installation. It all contributes to giving the network infrastructure the best chance of delivering on its promise.

However, as Francis Sideco, Comarco product manager, said, “The only metric that really means anything for carriers is the customer experience.”

That's where test equipment companies differ. They don't all necessarily go head-to-head trying to beat each other on capabilities, although behind closed doors, that's certainly the case. Instead, test companies seem to concentrate on being the leader in some of the test categories mentioned above.

Comarco has come close to providing tools that measure actual end-user experience. The company's Seven. Five family of products tests customer-perceived voice quality, call completion, call retention, data transmission rates and other quality metrics. It is a set of drive-test products that includes a single-phone optimization solution called Solo; an advanced optimization tool for network deployments called Duo; and Multi, a multi-chassis solution for multi-technology optimization and benchmarking.

The products were adopted last month by Cofetel, Mexico's government agency for overseeing telecommunications carriers, to monitor quality of service (QOS) for all of the country's wireless operators.

That's a tall order, but Sideco has a few tricks for achieving that close approximation of the end-user experience.

“You keep your test equipment as close to what Joe Schmoe is going to be using,” he said. “You don't introduce additional degradation with your own test equipment. You use the most commonly used applications like Internet Explorer and Outlook. And you make sure that for each phone you are testing, you have an independent PC controlling it.”

The independent PC — or single-board computer — that Comarco uses in a 1-to1 fashion with the phones and applications they use in drive tests is one of the things that sets its product suite apart, Sideco said.

However, “There is only so much drive testing you can do and only so much indoor testing you can do before your customer goes places you can't go,” he said.

That's where Comarco's new QualiPoc test system comes in. With more people using mobile phones as primary lines in the home, Sideco said it's a good way to get testing into the house.

The concept behind QualiPoc is enabled by a new generation of smartphones. The QualiPoc test platform lets wireless operators use their employees' phones and, in some cases, their customers' phones, to test the quality of voice, data, browser and messaging services. QOS tests can be scheduled remotely and executed from a browser-based control center and are conducted in the background. QualiPoc-enabled phones upload their QOS data to a central data server where it gets analyzed.

As close as QualiPoc gets to the true end-user experience, it won't replace drive testing. Nothing will. “It's complementary,” Sideco said. “You can deploy 1000 QualiPoc-enabled phones and use that information to understand where your weaknesses are, but then send out more expensive drive-test optimization gear to focus on those weak points.”

NetTest, too, gets close to the customer, but in a more traditional way. The company uses software probes distributed throughout the network to collect signaling and user-plane data. NetTest combines testing gear with its MasterQuest operations support system (OSS) platform to correlate and interpret the data. It has deployed the system in 17 new wireless operators' networks over the last year, including Hutchinson 3G in Italy and Tier 1 providers in Europe and North America. Its name has been bandied about in relation to the 3G deployment of Cingular Wireless, but nothing formal has been announced.

MasterQuest is a comprehensive OSS solution that includes modules for fraud detection, interconnect management, service assurance, network and service performance monitoring and troubleshooting. NetTest combines MasterQuest with testing because while network quality alone can be determined from a small amount of data collected from the network with an analyzer, “When it comes to customer quality of service, you have to know everything about a specific customer.” Traberg said. “The amount of data you need is much bigger.”

It also needs to be varied from sources throughout the network.

“One of the basic assumptions in OSS is that the information you need to do it can just be found lying around somewhere, but you have to capture that data and correlate it, and that's what MasterQuest does,” Traberg said.

Despite being behind on actual 3G deployments, Traberg said North American wireless operators are more interested in and farther along in deploying service-centric QOS solutions.

Benchmarking is another criteria for determining QOS that takes another step back from the end user. Before an operator can know what to look for to measure quality in its live network, it must have a benchmark by which to measure it. Using benchmarks, quality measurements can ultimately be understood by everyone, not just engineers.

One way to determine benchmarks is in load testing network elements before they are deployed. Tektronix launched a new load test system last month called the LTS21, which it claims is the first automated load test system for UMTS and GPRS network equipment that combines both control-plane traffic and user-plane traffic.

Load testing is a new market for Tektronix, driven by the 3G paradigm shift in how load test systems should be designed. The LTS21's claim to fame is the ability to create real-world conditions under which to stress test a network.

“Determining what conditions break a network or impair network stability is not much of a challenge with voice traffic,” said Othmar Kyas, director of product line marketing for Tektronix' monitoring and protocol test unit. “But you have to be able to test network elements under real-world conditions to see how they will behave.”

That includes injecting unexpected and non-standards traffic into the network to see how it responds. The LTS21 creates realistic traffic scenarios by using caller profiles that include the call models operators use to define a service. It's more art than science, Kyas said, but it does create greater realism.

Tektronix has also launched a new network and service analyzer for optimizing the UMTS Terrestrial Radio Access Network that Kyas said almost makes drive testing obsolete. The system uses a Cell Overlapping Matrix test feature and service-specific key performance indicator subsets to identify problems with neighbor lists; extensive or insufficient cell overlap; service-specific, call-specific and user-specific performance issues; and to provide the capability to prioritize them.

“Depending on what KPIs one picks, the network can look really good or really bad,” Kyas said.

Kyas also said that while operators are getting closer to providing true service-centric QOS across their networks, capability breaks down in multi-network environments.

“No one is there yet,” he said. “Even simple handoffs between operators are not happening.”

That problem will continue to plague operators as multimedia and data services make interoperability a more urgent requirement. Interoperability is not something Anritsu typically worries about when going about its job of proving service-centric QOS. Where Anritsu looks for KPIs and proofs of performance, the metrics do not differ from operator to operator. Anritsu stays purely in the physical, testing channel power and frequency error. It measures the noise floor and the pilot-time tolerance. It looks at waveform quality and occupied bandwidth.

In other words, everything it needs to know about determining the QOS that a network is capable of delivering can be found in the 3GPP specifications.

“Most problems are associated with the physical layer — the cable or the transmitter,” said Christopher Bosso, project manager for Anritsu. “The 3GPP spec is very clear about what test equipment needs to do to be confident in measuring QOS.”

Anritsu joined its fellow test companies last week in launching new services. Theirs is aimed at portability and wrestling even more QOS measurements from the radio frequency (RF) channel. The company introduced the UMTS Master hand-held Node B analyzer, which conducts both RF and demodulated measurements for the UMTS frequency ranges of 824 MHz to 894 MHz, 1710 MHz to 2170 MHz and the 2400 MHz to 2700 MHz WLAN range.

Anritsu also got into the drive-test environment in a roundabout way through a partnership with Nemo Technologies, which will use Anritsu's portable hand-held UMTS Master Node B analyzer in its Nemo Outdoor drive test system.

From mean opinion scores to Mb/s, there are 101 ways to measure performance. Which ones matter in 3G wireless networks depends on your point of view, or at least the point of view of your available test tools.