AT&T CTO lauds network performance during MS-SQL attack

Last weekend’s MS-SQL worm attack slowed global Internet traffic to a crawl for millions of users on hundreds of ISP networks. However, AT&T Chief Technology Officer, Hossein Eslambolchi, pointed to a near negligible impact on the company’s IP backbone as testament to not only its secure infrastructure design, but to the growing need for more robust security across the industry.

The worm, also known as Sapphire or SQL-Hell, began shortly after midnight on January 25, allegedly originating in the Asia Pacific region. Within hours it had spread across the globe impacting approximately 22,000 servers and increasing download times by 50% on major U.S.-based Web sites while shutting down many Internet nodes and crashing Internet backbone switches. The attack exploited a known vulnerability in Microsoft’s SQL server software, for which the company had distributed patches, but not all systems operators applied, apparently including Microsoft, which was affected by the worm.

AT&T’s Eslambolchi said many system managers do not apply available patches because they are difficult to apply in a production systems and it is not always clear that a patch will solve a potential problem.

Yet despite this being the fastest spreading work he has seen, Eslambolchi said the systems and techniques applied in the AT&T network helped identify and respond to the attack within minutes and resulted in a performance impact he termed negligible and non-customer affecting. He said AT&T network managers knew within two seconds which peering link and port the worm came in and how it was propagated through access control lists.

“We performed better than any other network because of the way [our] network has been architected,” Eslambolchi said. “Security cannot rest on a single layer of defense. I don’t believe there is a magic bullet for these types of worms or viruses. In fact, it is going to get a lot more complicated and complex.”

He added that AT&T has put innovative technologies and techniques such as various forms of detection and sampling in order to understand what is going on in the network and be predictive in managing network traffic.

However, one network operator’s performance is not enough to ensure the security of the Internet and Eslambolchi offered this prescription for the entire industry.

“Security has to be ongoing,” he said. “It will never be perfect given the nature of the Internet so you just cannot rest for a single second. You can never stop these types of incidents from happening on the IP network and anybody that believes they can is mistaken.” On containing damage, Eslambolchi said, “I believe there has to be some kind of minimum requirement across the design of the various IP networks. And I do not believe there is a common concept of one set of standards across the industry. Everybody is designing [their networks] with a different set of capabilities and tools in mind. There has to be one [standard] if we are looking at this from a sense of national security, which I believe we should. I think the industry has to rally around in defining--[either] from the standards perspective or driven directly from the Office of Homeland Security--minimum requirements for every ISP across the globe so we can ensure the impact of these [attacks] is absolutely contained.”

As to how the continuation of attacks and the perception of an insecure Internet affect AT&T’s plans to migrate its voice traffic to IP, Eslambolchi said, “I have been a proponent of moving to voice over IP because I believe IP is going to eat everything by the end of the decade. It’s like a Pac man. Today a lot of applications cannot move to VoIP because of the reasons we saw on Saturday and the security of IP. But if any company is going to lead the industry in moving its voice telephony traffic to an IP infrastructure it’s going to be AT&T because we just demonstrated a prime example of how we can do that without having any impact in terms of performance.

“Whether the whole industry can move to that is going to come down to the will of the entire industry to make the Internet more robust, more reliable and more secure. It’s going to come down to the people actually writing the code, engineering, architecting and operating the network. If all those parameters happen, I truly believe that we will have a flawless and resilient network infrastructure to be reckoned with, comparable to any network that has been built over the past decade or so.”