HACKING OFF THE HACKERS

Back in the 1930s, one of the era's most popular radio programs asked the rhetorical question, “Who knows what evil lurks in the hearts of men?” The answer was “The Shadow,” who each week would cleverly and doggedly track down evildoers and bring them to justice.

Today's equivalent of The Shadow is the information technology and security professionals who feverishly labor to keep computer hackers at bay. They know what hackers can do, and it scares them. The threat is so serious, in fact, that everyone interviewed for this story downplayed their successes in warding off hackers to avoid putting themselves in harm's way.

“Once you thump your chest, the hackers say, ‘Let's see how good they really are,’” said Tom Montauri, director of information services for Verizon Communications.

For related metrics on the growing vulnerability assessment market, click here.

Fortunately, IT and security pros are very good, and getting better all the time. Carriers are learning more about hacker tendencies through increasingly more sophisticated surveillance and preventative measures, a greater depth of experiences and a stronger, more effective information flow between carriers, security consultancies and the government.

But security teams must continue to improve because hacker incidents are on the rise, with no slowdown in sight. According to the CERT coordination center, an Internet security reporting organization operated by Carnegie Mellon University in Pittsburgh, more than 73,000 hacking incidents were reported nationwide through the first nine months of 2002, up from about 53,000 incidents for all of 2001. Comparatively, CERT reported about 10,000 incidents in 1999 and about 22,000 incidents in 2000.

Alan Fitzpatrick, senior vice president of engineering for competitive carrier US LEC, said the hacking problem is “big and will only get bigger.” The fear is that as the number of applications and servers grows — and as new technologies develop — hackers will always stay at least one step ahead of service providers.

“They'll always find new ways to attack us,” Fitzpatrick said.

One of the biggest problems for carriers is that their attention is divided between protecting their own network assets and those of their customers, and vital resources are invariably spread too thin.

According to Fitzpatrick, hackers tend to attack easy IT targets, such as car dealerships, which may have great mechanics but tend to have ineffective IT staffs.

“One [dealership] was having some problems and asked us to look into it,” Fitzpatrick said. “We discovered that someone had hacked in and set it up so that [the computer game] Doom could be played on the dealer's server by players all over the world.”

Sometimes customers think they've been hacked, but find that something quite different, albeit equally damaging, is going on, Fitzpatrick said.

“One of our customers thought they were under a denial-of-service attack, but we found out an employee was downloading a bunch of movies,” he said.

Such occurrences keep carriers scrambling. The good news is that less technically savvy customers have created an opportunity for carriers such as US LEC, Fitzpatrick said, because it's more effective and less expensive to rely on carriers for anti-hacking expertise than it is to maintain their own IT departments.

Despite this, US LEC thus far has not marketed this capability as a new service but instead uses it as a value-added enticement to get customers to change over from the incumbent. US LEC even provides port scanning to determine a customer's vulnerabilities at no charge, which Fitzpatrick said has been an effective sales tool.

The bad news is that every minute spent protecting customers is one less minute the carrier can spend protecting its own networks, including the back office, where sensitive and valuable customer proprietary information resides. Nevertheless, considerable efforts are being applied to the cause.

All carriers employ some combination of intrusion detection devices, vulnerability scans and sophisticated firewalls to determine when hackers attack and where networks are most vulnerable. They then use that information to place as many obstacles as possible to bar entry.

The public switched network generally is less of a concern because the switches are so complex and most hackers don't possess the technical expertise to breach them. But carriers still err on the side of caution.

“The more layers, the more obstructions, the better,” said Billy Potter, manager of IT security for rural local exchange carrier Alltel. “We try to make sure that a lot of protocols have to be met at each level.”

Some employ network operations centers that coordinate security efforts as they manage switch activities. The largest carriers, including Verizon and Qwest Communications, employ computer intrusion response teams (CIRTs) that cross all company areas and jump into the battle when all else has failed.

“It's our SWAT team, if you will,” said Bill Oswald, Qwest's senior director of risk management. “Once all of our proactive and reactive steps fail to work, these people get involved in managing the intrusion. They repair the damage and get us back to recovery pretty quickly.”

Often the CIRTs employ former “white hat” hackers — those who hack as a hobby, not profit — who bring an attractive knowledge base and zeal to the effort.

“It's a game to them,” Potter said. “White hats don't want black hats to beat them.”

Because black hats are typically more insidious of the two, one may think carriers would be eager to have a few on staff, if only to pick their brains. Not so, said Jim Roberts, Alltel's vice president of network operations.

“Once a black hat, always a black hat,” Roberts said. “At what point do you turn them loose in a switch room and believe that you can trust them? We would rather have a white hat.”

When Alltel needs black hat insights, it turns to one of the many security consultancies, a number of which employ ex-black hats, Roberts said.

Another tactic is to regularly log on to chat rooms and bulletin boards to see what people are saying about the company, Verizon's Montauri said. “We try to identify people who have proprietary information they shouldn't have. When we do, we bring it to the attention of law enforcement,” he said.

As bright as hackers often are, they're not infallible. At one hacker Web site, Verizon's security experts noticed that the site's owner had posted a picture of his car. Upon blowing up the photo, the license plate was visible, which made it easy to track down the suspect. “But the more sophisticated hackers are mobile and not easy to catch,” Montauri said.

So the effort continues. To make matters easier, carriers are setting aside fears surrounding sensitive market information leaks in order to share information vital to the anti-hacker crusade with one another. Carriers typically are members of several security organizations where information is shared. Since last year's terrorist attacks, the federal government has become more active as well.

Because timing is everything in the security business, this benefits carriers greatly. Patches are often developed in a matter of hours, so the earlier a carrier is warned of an impending attack, the better chance it has of heading it off, or at least shutting down the affected part of the network before it crashes.

With intense financial pressures in the telecom industry, there is concern that carriers cannot keep up these efforts.

“With all of the layoffs, security has been relegated to second string because management has other priorities,” said Steve Crutchley, chief of security for consultant Forefront Security.

But Rick Felts, vice president of operations for SBC Communications, said there is little to worry about because carriers know they must protect their customers' sensitive information.

“Hackers are just part of the challenge,” Felts said. “From a service provider perspective, our main job is to maintain the security of our networks and customer information. We will continue to do everything we can, whether hackers are on the rise or in decline."