IMS software bug caused Verizon LTE outage

A software bug in Verizon Wireless’ IP Multimedia Subsystem (IMS) architecture caused last month’s 4G network outage, shutting off access to VZW’s long-term evolution (LTE) and 3G networks for hundreds of thousands of subscribers for at least 24 hours (CP: Verizon LTE back online), Verizon Communications (NYSE:VZ) chief technology officer Tony Melone said today.

An element in the IMS core experienced a software problem, which rapidly escalated, affecting the core’s back-up systems and eventually shutting off all access to either mobile data network, Melone said. Normally a software bug would only cause a “hiccup” in the network, which has multiple redundant systems, but this bug persisted, affecting the same processes in the back-up systems, Melone said. Verizon and the element’s vendors identified the problem immediately and developed a fix, but it brought the network up gradually to ensure no other problems remained, Melone said.

Melone didn’t identify the specific IMS element; nor did he identify the vendor. VZW has multiple vendors in the IMS core, each supplying different elements. Alcatel-Lucent (NYSE:ALU) and Nokia Siemens Networks (NYSE:NOK, NYSE:SI) are the primary vendors providing the key call session control function (CSCF) and home subscriber system (HSS) respectively. Verizon also uses Acme Packet session border controllers (SBCs) on the edge of its network as well as Tekelec’s policy control platform.

The PCRF and SBC primarily handle the VoIP-based call traffic—services Verizon has been testing but has not yet launched (CP: Verizon shifting to VoIP only phones in 2013), while the policy manager defines quality of service (QoS) levels and charging rules. That’s led Morgan Keegan analyst Simon Leopold to suggest that the NSN-supplied HSS, which tracks and authenticates subscribers across the network, as the culprit.

Verizon implemented IMS primarily as means to support voice-over-LTE (VoLTE) on its LTE network, so IMS doesn’t touch its 2G voice or its 3G data customers. But the 3G network still went down for Verizon’s 4G customers, even though its 3G customers remained unaffected. Melone said that this was a result to the way the data network is designed. When 4G customers move out of LTE coverage into the 3G radio network, they remain on the 4G mobile packet core, rather than move onto the high-rate packet data (HRPD) core that services 3G-only subscribers. This ‘evolved’ HRPD core allows Verizon Wireless to maintain continuity between the two data networks for its 4G subscribers, Melone said. So when the IMS core glitch shut off LTE network access to those 4G subscribers, it also shut off their 3G access, even though the 3G radio network was working just fine. “Our 3G customers don’t interact with that IMS core,” Melone said.

Melone offered up the explanation during a press conference at TIA 2011 after his keynote address, delivering the most detailed account of what happened to Verizon’s network on April 26^th and 27^th to date. But Melone also fielded questions on AT&T’s (NYSE:T) proposed acquisition of T-Mobile USA. While operators like Sprint have railed against the deal Verizon has remained relatively neutral. Verizon officials have said they believe the deal will be approved on its merits but have expressed concerns that regulators may impose conditions on the merger that would be bad for the wireless industry.

Melone said that from a competitive point of view Verizon is indifferent whether the merger goes through (Related, CP: Verizon’s take on the AT&T merger: is the enemy of enemy my friend?). If AT&T didn’t buy T-Mobile, they would find some other way to scale or acquire spectrum, Melone said.
“Quite clearly AT&T is a formidable competitor no matter what they do,” Melone said. “Regardless of whether they acquire T-Mobile or not they’re not going to just lay down in the market.”

Melone pointed to Verizon Wireless’ recent acquisition of Alltel, which propelled it over AT&T in terms of subscribers. The acquisition may have made Verizon bigger, but becoming bigger doesn’t give Verizon any inherent competitive advantage. “These mergers in and of themselves do not make a difference,” he said.