UUNET hit by outage

UUNET experienced a major service outage today practically shutting down Internet traffic on several parts of WorldCom’s vast IP backbone network.

The outage crippled many businesses that depended on WorldCom as their sole backbone provider was shut off completely from the Internet. The outage also hit service providers and enterprises with multiple transport contracts who found their IP traffic severely constricted, said Tom Ohlsson, vice president of marketing for Matrix NetSystems, a network monitoring and measurement firm with numerous large enterprise customers.

“At about 7 a.m. [EST] we saw a sharp vertical spike in Internet packet loss on WorldCom’s network,” Ohlsson said. “A normal packet loss rate is around 0.01% to 0.05%. Between 7 a.m. and 7:15 a.m., WorldCom went from nominal packet loss to 22%.

Ohlsson said once packet loss reaches 5% to 7% the network is essentially inoperable. “For about six to eight hours today, WorldCom isolated itself from the Internet,” Ohlsson said.

A WorldCom spokeswoman today confirmed that the carrier was experiencing network problems but couldn’t name the cause or the exact extent of the outage. She said service had been restored to some customers by this afternoon and WorldCom staff was working to correct the remaining problems.

“Our technicians and vendors are investigating the problem as we speak,” the spokeswoman said.

Sockeye Networks, which optimizes IP flow for companies and carriers multi-homed to the Internet, said WorldCom’s outage has effected practically the whole IP network with enough destinations shutting down to make most of UUNET bog down completely.

“We have 14 customers with UUNET links, and all of them are having problems,” said Brendan Hannigan, vice president of marketing for Sockeye. “We’ve had to route traffic completely off WorldCom’s network. We’re seeing latency spikes from 50 to 700 milliseconds and almost 100% packet loss in some cases.” The problems, however, extended well beyond UUNET. Since AT&T and Sprint have extensive peering relationships with WorldCom, traffic bottlenecked at the carriers’ peering points, causing Sprint and AT&T to experience similar high latency and packet loss rates, Ohlsson said. Providers and businesses with relationships with Qwest, Level3 and Cable&Wireless, however, were able to avoid much of the problem, by shutting down circuits linking to UUNET, Ohlsson continued. Those three carriers share no peering relationships with WorldCom and were able to carry the extra IP traffic without running into logjams at UUNET routers.

Ohlsson said the problem was caused by software upgrades WorldCom conducted on all of its border routers -- the routers at all peering points and hosting locations. WorldCom most likely bought some bad software from a vendor, and when the software went live at 7 a.m. it caused thousands of routers to shut down, he said.