Advanced peering: A better alternative to the traditional Internet peering model

With the rapid growth of Internet traffic, larger retail ISPs (with traffic loads exceeding 1 Gb/s) are faced with the challenge of efficiently handling IP traffic increases. These companies, which include dial-up providers, DSL providers, RBOCs, PTTs and cable companies, are evaluating the traditional Internet peering model as a way of achieving a leading cost structure and higher level of quality. This alternative approach is known as advanced peering.

Advanced peering leverages the Martini Draft implementation of MPLS. It enables ISPs and content providers to use a shared, IP-based transport network to establish advanced peering links to business partners. It allows each provider to maintain its own AS network and achieve high quality and secure performance for all traffic types. And it does this while allowing the providers to avoid the cost and operational implications of the traditional model. The result is a more eloquent and cost-efficient option for Internet interconnection.

Because the traditional Internet peering model involves settlement-free peering and the achievement of Tier 1 peering status, it is viewed as a low-cost solution. However, the actual cost structure that can be achieved through the traditional Internet peering model is highly dependent on traffic volume and mix. Only the largest ISP backbones will achieve a cost structure that is equal or better than purchasing transit.

When taken to the extreme, the traditional Internet peering model results in reverse economies of scale that are detrimental to the industry as a whole. This is due in part to inefficiencies that slow the pace of technological innovation while stifling cost improvements. On the other end of the spectrum, a model based solely on the use of a transit provider fails to deliver the most efficient network architecture. Alternative models such as advanced peering create a robust networking environment coupled with the right cost structure.

The fiber analogy

To better understand the challenge presented by the traditional Internet peering model, one only needs to look at the fiber-based carrier market. Over the last few years, more than 20 carriers in the United States and more than 15 in Europe pursued business plans that involved intercity fiber-based networks. In the United States, these companies included traditional IXCs (such as AT&T, Sprint, Qwest and WorldCom), next generation carriers that focused on intercity backbone business (such as Global Crossing, Williams Communications, Genuity, Level 3 Communications and 360 Networks) and a host of others (such as Savvis, McLeod Communications, Core Express, XO Communications, MFN, Aeries and Telia). A similar number and mix of companies pursued fiber-based Pan-European business plans.

Of the U.S. carriers, only AT&T, Level 3 and Sprint do not face the prospect of bankruptcy and/or retrenchment from the U.S. intercity fiber business. Most of the remaining companies have completely abandoned their intercity fiber business--in most cases after spending many $100 millions. A select few might re-emerge from bankruptcy and perhaps become viable. Most will not.

It seems that each company's goal was to achieve a better-cost position relative to lit services. They also valued the perception created among customers and investors that they were "facilities-based" providers. There was a positive view that the companies' cost structure would not be as variable with bandwidth growth--and building could be capitalized while buying would be an expense item. They convinced themselves that their network engineering and operational capabilities would enable them to achieve a higher level of service quality (Table 1).

Table 1: Costs associated with building and operating a dark fiber network

Operating fiber-based intercity backbone networks is a scale business. Cost efficiency and technical performance result from achieving economies of scale. Network and OSS innovations can drive the cost of bandwidth down, but only if high traffic levels can be achieved. In hindsight, these observations seem obvious.

But why was so much money invested in such a large number of fiber-based business plans? Certainly the companies were faced with the desire to be part of the Internet boom, and this skewed decision-making. Build-vs.-buy models tended to be highly biased toward building. Companies told themselves the financial markets would reward them based on how much capital was spent on the network. The only question was, "What will be the ultimate multiple of invested capital--3 times, 4 times or 5 times?" Business judgment was drowned out by exciting business plans involving a large number of facilities and fanciful stories about technical innovation.

While the decisions of the late 1990s cannot be reversed, the industry can learn from them.

Rethinking the solution

There are more than 30 companies pursuing the traditional Internet peering model (see sidebar below). Dozens more are pursuing more limited versions of it. This isn't surprising considering the only alternative to date has been Internet transit. But what are the results of so many providers implementing this solution, and what is the ultimate viability for the industry?

The primary problem with the traditional model is that it produces substantial reverse economies of scale. Consider the tremendous overlap of OC-n and waves between cities; consider the permutations of peering links, most of which involve dedicated metro circuits, transport equipment, and router ports. Networks operate at lower speeds because traffic is spread across so many networks. Higher bandwidth MPLS and IP technologies are slow to emerge. Utilization is lower due to statistical inefficiencies. The operating resources to operate the networks and stay current on technology and to build systems and processes are replicated many times over.

The better alternative clearly is advanced peering (Figure 1).

Figure 1: Internet exchange point depiction

In the traditional peering model, each peer uses waves or OC-n circuits to reach each of the major Internet cities. In these cities, the companies often meet at a common switch matrix, such as the MAE-West ATM switch shown in Figure 1. A transport link connects them up to the MAE-West building. A port is leased on the switch. Once an agreement is struck with a peer for peering on this MAE-West ATM switch, a virtual line is established between the two providers. Traffic is then exchanged. There is efficiency in the fact that one switch port into MAE-West can be used to establish virtual links with many peers. But there is inefficiency in the fact that the same scenario must be replicated in many locations across the country, while numerous wavelength and SONET links must be leased to provide the underlying connectivity. Each company must also build and operate its own MPLS network.

The advanced peering concept was introduced in late 2002. Advanced peering is similar to the Internet Exchange model in that it enables networks to interconnect with one another in order to exchange traffic through a shared switching fabric. The difference is that the shared switching fabric is a MPLS network (Figure 2).

Figure 2: Advanced peering switching fabric

Each ISP or content company connects up to the MPLS network wherever convenient. An extensive intercity and metro network usually provides for cost-effective, fiber-based connections. Each company then determines with whom they want to peer. They enter into bilateral agreements, just as they would with traditional peering. They then establish a virtual link across the MPLS backbone. That's all there is to it (Figure 3).

Figure 3: Advanced peering link

The advantaged of advanced peering

Advanced peering is much more efficient than traditional peering arrangements because it minimizes cost replication. By concentrating traffic across fewer networks, higher utilization levels are achieved. These efficiencies accelerate the development of high-bandwidth IP technologies. The combined effect of these factors is the acceleration of the IP unit cost reduction trend, which, in turn, fuels the growth of high-bandwidth applications.

Advanced peering also exploits the advantages of MPLS. The virtual links can be established at MPLS layers to ensure better service quality and security.

The advanced peering infrastructure can be used for multiple purposes and traffic types, including Internet transit traffic and the interconnection of non-contiguous properties. For example, a cable company could establish a voice or a video distribution network between its properties. Each of these links could use an MPLS layer that is optimized for voice or video. The cable company also could establish links for peer-to-peer traffic. An RBOC or PTT might wish to design a core ATM or Frame Relay network to interconnect the major markets in the U.S. and Europe.

In addition, Advanced Peering would allow companies to establish links with their trading partners for non-internet traffic. One example would be to allow a media company to distribute video content to broadcasters or satellite up-link sites. Another would be to enable an interactive gaming network, such as Microsoft's X-Box, by establishing links between a cable company and the gaming provider's servers. In each of these examples, the virtual links would leverage the MPLS queue that is best suited for the service type.

Just as important, advanced peering links can be established between the United States and Europe, which can be used for any of the purposes above.

One of the most intriguing aspects of the advanced peering solution, however, is its ease of implementation. To take advantage of advanced peering, a port would need to be homed to a MPLS edge switch instead of the Internet router. Initially, this would need to be a separate port from the Internet transit port. Within months, a shared port would be used. In the latter case, the incremental cost of enabling MPLS is trivial.

Once attached to the MPLS cloud, virtual connections can be easily established.

Immediate savings

The cost to support advanced peering traffic is lower than Internet traffic. Internet traffic often traverses peering links, which means that the provider of advanced peering will carry the cost for both origination and termination of the traffic, but will be compensated entirely by one customer. With advanced peering, the provider originates and terminates the traffic between two customers, who share the cost of the connection. Each customer benefits in the form of a lower per-megabit price.

Many of the details of a pricing model that reinforces the combined economies of advanced peering, Internet transit and other traffic types currently are being developed. The price per megabit for advanced peering and other traffic is materially lower than transit. The amount of traffic in aggregate (across all ports and traffic types) determines the overall pricing discount. As the aggregate traffic grows, the price per megabit of each type is pushed downward.

Each time an advanced peering link is established, traffic that otherwise would be subject to transit pricing would be priced at a lower rate resulting in immediate savings. And, since aggregated volume drives price down, advanced peering relationships can be established between more and more trading partners. The network can be used for other applications. The hidden costs of traditional peering--metro links, co-location facilities, excess equipment and operational overhead--can be eliminated. All this adds up to long-term savings.

Advanced peering does not require that all ISPs and content companies agree on a single advanced peering provider. Vendor management should be used to drive pricing and performance levels over time, ensuring that the ISP and content communities benefit from the economic efficiencies of advanced peering. Over time, interoperability between advanced peering providers would become desirable.

While the industry continues to change rapidly, cash flow performance will decide who will survive and who will prosper. The health of the industry will depend on the industry's ability to develop solutions that will continue to drive IP bandwidth costs downward while increasing the overall capability of IP networks to carry all forms of communication.

Many lessons are being learned from the telecom meltdown. The industry will prosper again, but fundamentals of business--centered on the effective management of cash flow--will need to drive network decisions. The industry needs to be honest about when it is appropriate to build and operate vs. when it makes more sense to leverage someone else's investment and core competency. And, it needs to be creative in finding solutions that produce win-win results. Advanced peering represents one way the industry can work more closely together to achieve a better business result for the industry as a whole.

 

SIDEBAR: The traditional Internet peering model

An ISP implementing the traditional Internet peering model would do the following:

1. The ISP would decide which cities are needed as POP locations. San Jose, L.A., Seattle, Denver, Chicago, Dallas, Atlanta, Washington D.C., New York, Miami and Boston comprise a typical U.S. list.

2. In each of these cities the ISP would locate a technical site that could serve as their core IP node. Commonly chosen sites include Level 3 gateways, AboveNet PAIX facilities, Equinix sites and traditional carrier hotels.

3. The ISP would lease or purchase OC-n or wavelength bandwidth to connect the cities and to extend the intercity bandwidth to its POP.

4. It would then enter into peering agreements with its trading partners. This process would begin with Tier 2 ISPs and content providers. These agreements describe the business and technical terms that will govern the exchange of traffic.

5. Physical links would be procured to connect the ISP to each of its peering partners. Some of the peering partners would be in the same technical site. These connections are relatively easier and less expensive since they don't involve metro loops. However, many of them are outside the POP locations. These include MAE-East, other co-location facilities and private facilities on the partner's network. High bandwidth (and costly) OC-n or Ethernet metro links are required to interconnect to the peering partners. The nextgen peering initiative is an attempt to encourage ISPs to locate in common sites. Several neutral co-location facilities have been identified by the initiative. If successful, it will provide some help in this area of Internet peering.

6. The next step would be to build out an Internet network on top of the leased or purchased wavelengths or private lines. Most new implementations are likely to use MPLS technology. Routers and switches would be deployed to support both the core backbone and the metro connections to the peers. Dedicated ports (or shared switches such as the MAE-West ATM switch) are needed to connect directly to peers.

7. Operational capabilities associated with traffic management, routing tables, circuit provisioning, network management and monitoring, etc. would be developed.

8. Network architecture, engineering and IT capabilities would need to be maintained. IP and optical technology will continue to progress. The ISP would need to maintain the expertise to know what new technologies to deploy and when.

Lisa King-Guillaume is Vice President of (3)Packet Services for Level 3 Communications.

Visit Level 3 Communications online.