A new challenge

Everyone loves a good story, and the Starr Report has been no exception. Well, perhaps it is an exception, considering the high-profile independent counsel's report made history as being the first of its kind to be released on the Internet.

The distribution of the report created a new challenge for the nation's telephone network, Internet service providers, the Internet backbone network and Web sites. Never before had the Internet been used by so many to access the same information at the same time.

This mass calling event involved the interaction of public network, ISP networks and the Internet. Examining how these interconnected networks handled this event will lead to a better understanding of what can be done to better cope with similar incidents in the future.

Internet traffic originated by dial-up modems affects both telephone networks and the Internet because telephone network switches and lines are used to connect the call from the user's PC to an ISP location. ISPs then digitize the calls and route the packets via the Internet to the appropriate Web sites.

The effect on the Internet is best understood by separating it into two categories:

* The impact on the Web sites that hosted the report-which handled a majority of the traffic.

* The overall impact on the Internet, including the users who were not connected to the Starr-related Web sites, but instead were using other Web sites.

A unique situation A number of factors were involved with this particular posting that further added to its uniqueness and that would not unnecessarily be collectively apparent in other seemingly similar postings.

The sheer size of the document-445 pages-presented the first challenge. A large number of people accessing this amount of information would certainly tax the system.

Little or no precedent existed to understand viewing preferences of Internet users. Whether users chose to download the report or read portions of the report on-line would make a huge difference in call holding times. This was of particular importance to telecom companies because long holding times can harm the telephone network more than the Internet because of circuit- switching technology used in the telephone network.

Network managers had little notice before the report was posted on the Web. Telephone companies typically require two weeks' notice for mass calling events-such as ticket sales for a popular concert, or TV or radio advertisements for a prize-to prepare for the unusually high volume of calls. Increasingly, time is of the essence and advanced warning is not always possible.

But unlike other mass calling events, which are typically local or regional in nature, the interest in the Starr Report was global. Calls originated from throughout the country, and Internet queries arrived from many foreign countries. While some may attempt to compare the Web traffic related to the 1998 World Cup soccer games in France with this event, the two were, in fact, quite different. The most notable distinguishing factor is that traffic for World Cup Web sites was distributed over a much longer time (one month), while the posting of the Starr Report caused a sudden surge of traffic at the exact time when the report was released.

A report card One thing that helped performance during this event was the number of available access points. Unlike traditional focused overload events when a large number of customers dial a single number, in this case users could dial thousands of access numbers to get connected to the ISPs and the Internet. This greatly diffused the concentration of traffic at the originating segment of the network and did not result in very many hot spots.

There was, however, a general increase in the volume of calls to the ISP access numbers. The potential problems surface when the increase in call volume is greater than the capacity of ISP modems, thereby sharply increasing the percentage of call blocking to ISP modems. This is further complicated by the fact that customers will reattempt the call, and the load generated by each further attempt increases the demand on network resources. In normal situations, Bellcore measurements show that Internet users reattempt an average of five to six times, while this rate is less than one time for normal voice calls.

Bellcore placed monitoring systems at three different metropolitan areas to gather measurements on the traffic levels, monitor the performance of the networks and analyze the behavior of users.

Traffic data was examined to obtain insights into three sets of issues:

* An analysis of the time spent on-line by users

* The increase in call volume to ISPs

* How often users received busy signals when dialing the ISP's access number.

Call volume analysis. As depicted in Figure 1, the hourly call volume increased by as much as 60% to 70% compared with previous Fridays (measurements included two major ISPs).

Call failure rate analysis. The percentage of calls receiving a busy signal increased significantly for ISPs at these sample metropolitan areas during the hours of 6 to 10 p.m. compared with previous Fridays (Figure 2). Call blocking increased to 20% for ISP 1 and was as high as 55% for ISP 2 after the release of the Starr Report.

Call holding time analysis. As expected, the call holding time measured during the entire 24-hour period on Friday, Sept. 10, did not noticeably increase compared with previous Fridays. However, call holding time measured during the hours after the release of the report shows an increase of 2% over the same periods on previous Fridays (Figure 3). This means users logged on to the Internet for a short duration (approximately 18 to 20 minutes). The release of the report did not increase the average holding time, but the total call volume increased.

Based on measurements reported by Keynote Systems, the overall average delay measured for 40 popular sites was 8.55 seconds, compared with 8.02 seconds, measured during the same hours on the previous afternoon. This represents a degradation of only 6.6%. Using an overall average hides the fact that certain Web sites may have had tremendous delays while others may have had only slightly poorer performance and still others may not have been affected at all. To fully understand the impact on the Internet backbone, more detailed statistical analysis is needed. It is possible that routers and network access points in the east corridors of the Internet were heavily congested while other areas were less loaded due to the fact that government sites are on the East Coast.

Because the targets of all this traffic were the few Starr-related Web sites, it is not surprising that these targeted sites did rather poorly. Again according to measurements reported by Keynote, the access failure rate of the Web sites of the U.S. House of Representatives, where the full report was posted, was as high as 89% during the one-hour period between 4 and 5 p.m. EST. Other Web sites that also posted the report experienced failure rates of 30% to 50% depending on the particular sites.

A matter of timing Timing for the release of this report was actually a reason for the congestion levels not being as bad as they could have been. Telephone network busy hour is either between the business hours of 10 and 11 a.m. or 2 and 4 p.m. In contrast, the Internet use peaks at night between 9 and 11 p.m.

Ideally, there is a window between telephone and Internet busy hours during which both networks are not at their peak load. This window can be exploited for scheduling such events. Of course, care should be given with respect to the time of day and which time zones the targeted Web sites are located in. Also, the fact that many people connected to the Internet via a corporate connection at work further reduced the impact on the public network.

Several steps can be taken to better cope with such events in the future.

* Making the information available on multiple Web sites at the same time greatly diffuses the flow of traffic to a single point in the network. Sites should be selected that are physically located at different segments of the Internet backbone to distribute the traffic as much as possible. For example, the use of an alias address, such as www.starrreport.com, would allow the network to distribute the traffic to multiple physical sites based on a uniform load distribution.

* Procedures similar to the ones available in telephone networks to manage focused overload events need to be expanded and applied to Internet use.

* Greater coordination among local exchange carriers, ISPs and the provider of information is needed to plan for such events in the future. For example, if telephone companies are to invoke manual real-time call throttling controls on ISP access numbers, the list of such numbers and the size of control filters must be determined in advance of such events and there must be an agreement to invoke these controls at certain congestion levels.

* Finally, even with the best planning efforts in place, there are still times when traffic fluctuations will far exceed network capacity. To assure service quality to critical services, automatic real-time controls must be in place to block some of the calls from entering the network in the first place. These controls must become more sophisticated because of the complexity of these interconnected networks and the relationship between telephone traffic and Internet traffic.

Getting access to the Internet the day the Starr Report was posted was not impossible, but it did require a lot of patience. The extent of users' frustration varied based on the particular ISP's ability to answer the incoming calls. Call failure rates varied substantially in our sample results from 20% to 50%, depending on the ISP type. Once users successfully logged in, the government sites were under-engineered for the volume of traffic, and larger than normal delays were experienced. More powerful traffic management capabilities and greater coordination among different network managers can potentially help with similar events in the future.