Do You Measure Up?

Knowledge is power, but it doesn't come cheap. Case in point: benchmarking, which assesses all the carriers in a market based on performance parameters such as coverage, access failures and voice quality. It's important to know how you stack up against the competition, but when you're one of five carriers in a market, drive testing your network and four others once a quarter suddenly doesn't seem like the most effective use of resources.

Enter independent benchmarking services, which go beyond traditional drive-testing-and-optimization services to offer J.D. Powers- or Consumer Reports-type analyses useful for marketing and engineering alike. The services can be especially attractive to smaller carriers, which often can't justify buying a broad range of test equipment, much of it for technologies they don't use, just to check how their competitors are doing. They also eliminate the need for in-house know-how necessary to make apples-to-apples comparisons: It's tough to assess your CDMA competitor's Ec/Io when you barely know what Ec/Io is.

"As more competitors come on board and not necessarily with the same technology that you're using and maybe (at) different frequencies, you just don't have the ability to do that any longer," said George Sidman, Alltel vice president of network support. "We wouldn't necessarily have the equipment to be able to do all of the technology that's there. From that perspective, it is more cost-effective."

Another benefit is that personnel and equipment normally devoted to drive testing now can work on other tasks, such as determining why the report identified problem areas.

"Outsourcing allows them to keep their internal resources focused on network-improvement projects," said Bob Joslin, Safco manager, wireless-information services. "Benchmarking really isn't considered a network-improvement project. It's a snapshot of them and their competitors."

Snapshot probably isn't the most apt description, however.

"A lot of reports by the competition tend to be focused on the specifics of the engineering parameter," said Paul Christoforou, Telephia vice president of network quality. "You end up with a report that's 10 inches thick, which is completely indigestible by just about anyone. So one of our focuses is to make the presentation of the data a lot easier to understand and interpret."

That's helpful because performance reports aren't just for engineers anymore. Senior managers with MBAs but not MSEEs likely won't know whether -100dBm is good or bad, but they can appreciate a low call-drop rate. As a result, benchmarking reports often are available in many formats: executive summaries for management, detailed Consumer Reports-type rankings for marketing and several inches of technical minutia for engineers.

"It's moving to something that the carriers are interested in marketing," said Alan Brune, a former Cellular One vice president of marketing and strategic planning who's now Telephia vice president, marketing. "The network is a valuable asset that should be leveraged. The marketing group now wants to see that information in a way that they can understand and in a way that they can communicate to the customers."

SOUND METHODOLOGY? Outsourcing benchmarking can result in anxiety: Do they really understand our technology? Will they drive enough streets? Are they testing all the networks simultaneously or consecutively?

A clear, up-front understanding of the benchmarker's methodology is key both to alleviating that angst and ensuring that its approach is sound from an engineering standpoint. After all, a standardized industry benchmark isn't much good if its methodology isn't applied consistently.

"Providing a standardized benchmark across all networks is very difficult to do, especially due to the different technologies of PCS and cellular," said Paul Steckbeck, BellSouth Cellular senior manager, network management. "Dropped calls and call completions are pretty straightforward metrics, but obtaining accurate apples-to-apples comparisons of these metrics across the different technologies is sometimes very difficult."

No two markets are alike, so road choices are key. Bench- markers often use federal highway-classification codes as guides.

"We do testing based on road and street classifications," said Michael Burdiek, Comarco Wireless Technologies senior vice president. "We cover all the major roadways and U.S. highways and important state and county roads and major thoroughfares. These designations are based on traffic counts. It's not worth it for us to go into every little street. It's a lot of extra expense for no benefit."

Statistical analyses support that claim.

"We did a study where we selected 100% of the available roads within a suburban area and an urban area," said Jim Anderson, Safco manager, wireless-information services. "Then we randomly stripped out 10% of the data 100 different ways and looked at the standard deviations in the means of the data that was left. Then we stripped out 20% 100 different ways. We found that once you reached about the 50% level, the means and standard deviations started falling apart. If you lump that together with the way we select the roads, it makes a good match."

COST-EFFECTIVE? Despite the advertised benefits, not all carriers are sold on the concept of independent benchmarking.

"There's really not enough value in obtaining these types of services just to understand your competitors' coverage," Steckbeck said. "These services must also provide value by improving the performance of our network. In our eyes, using them just for competitive analysis really isn't justifiable."

Cost can vary widely depending on the size of the network and whether another carrier has already paid to have that market benchmarked.

"It's a highly specialized service, and it can be very expensive," Steckbeck said. "The fact that you can't just do it once and forget about it means that the price really has to come down. Some of the initial quotes we got were pretty pricey. That's why we started looking at tools and doing it ourselves."

But for small and mid-size carriers, benchmarking may be just the next part of running a network to be outsourced.

"I expect to see more carriers using these types of services mostly because of the substantial investment required in purchasing the tools, the manpower needed for drive-testing and the specialized training required to use the tools," Steckbeck said. "These tools can be millions of dollars. For some carriers, it absolutely makes sense to outsource that."