Analysis: A closer look at BTIG's 4G speed tests

BTIG Research’s recent tests of Verizon Wireless and Sprint’s 4G networks is causing quite the dust-up. If you haven’t seen the results, BTIG summarizes them on its site (registration required), but they basically boil down to VZW’s long-term evolution (LTE) network kicking the pants off Sprint’s Clearwire-provided WiMAX service in both smartphone and laptop card tests within a building in New York. There’s been a huge media storm surrounding the results with some like VentureBeat calling the Sprint network a joke, while others like InformationWeek being a bit more judicious in their analysis, pointing out that the LTE network is brand new and relatively unloaded while Sprint and Clearwire have been piling EVOs, mobile broadband routers and USB cards onto the WiMax network for more than a year. Sprint didn’t let this one pass, as it’s done with other speed rating stories. It called the BTIG’s tests plain inaccurate, while also referencing the relative youth and inexperience of VZW’s LTE network.

BTIG’s Walter Piecyk and his team did a thorough job in their tests and, conducting 1000 separate measurements over a week and keeping the parameters consistent with all of its previous tests. But what a lot of the stories written so far fail to point out is that those tests were conducted from a single location, BTIG’s offices in New York. The huge nearly 9X discrepancy (9.1 Mb/s for VZW vs. 1.2 Mb/s for Sprint) could have had much more to do with the relative locations of Clearwire and VZW’s towers to the building than the capabilities of the network itself. The results could be a completely different if you moved the test two blocks in any direction, to say nothing of across the country.

In my view, tests like these are pretty inconclusive. They test single a location within a single market under a singular circumstance. Don’t get me wrong. I think they’re completely valid for testing those specific set of circumstances, but they’re hardly comprehensive (which Piecyk readily points out in his analysis) and can’t be used to make any sweeping generalizations about the two networks as a whole.

BUT BTIG’s results can be taken as part of the wider body of emerging tests, all of which show that Verizon’s network is the fastest thing on the airwaves today.

PCWorld conducted its own ad hoc tests of 4G phones in five west coast markets, and found an even bigger disparity than BTIG between Verizon and Sprint’s connection speeds (a whopping 18.3 Mb/s for the VZW Thunderbolt compared to meager 1.65 Mb/s for Sprint’s EVO Shift). In PCWorld’s case the tests were more geographically comprehensive—it tested 10 separate locations in each market—though it only performed one speed measurement at each location.

The numbers I tend to give the most weight to are those of RootMetrics, an independent testing company that has been gaining a lot of attention in the last year. It did its own limited testing in Seattle of the Thunderbolt when it first went on sale, collecting hundreds of different data points while circumventing Lake Washington. Again, Verizon blew the doors off those tests, averaging download speeds of 17.6 Mb/s and achieving average peak througputs of 22.2 Mb/s, but Sprint didn’t come off as shabbily in those tests, averaging 4.4 Mb/s, dead center in the 3-6 Mb/s range Sprint and Clearwire have always advertised. Root shared with me data from a much more comprehensive report on Seattle it plans to release next week, which collected millions of data points over several months (though obviously still limited to one market) and found Sprint’s network to deliver an average 3.78 Mb/s. Root even tested smartphone speeds under the less than optimal conditions of CES in January and found that Sprint far outperformed the competition (though there was no Thunderbolt to test at that point), clocking speeds in excess of 7 Mb/s on average (CP: Mobile data triage: testing the operators at CES).

More: Verizon LTE is clearly faster, but...