Future An inside look at the design and operations of a terabit ATM system

The question is not whether asynchronous transfer mode will take over switching, but when.

ATM already has a toehold in central and headend offices. As the only standardized international protocol on the horizon for very high-speed switching, it's in commercial service or undergoing trials at most major telephone and cable TV companies in the U.S. and around the world.

ATM's advantages for local exchange carriers, interexchange carriers and cable TV service providers are clear. ATM switching can handle voice, data and multimedia, as well as any new services likely to be developed-including those for which traffic patterns can't be forecasted.

But to take on the central role in switching offices, ATM systems will have to be faster than even today's products. The time will come when gigabits aren't enough-when throughput from voice, data, multimedia and other signals will engulf switching offices by the hundreds of terabits.

As part of a Department of Defense program to investigate the potential of future information technologies, Lucent Technologies has demonstrated the feasibility of an ATM switch capable of switching aggregate traffic in excess of one terabit per second. The 20 Gb/s switching module

The terabit switch is based on a 20 Gb/s capacity switching module that is essentially the same as AT&T's GlobeView-2000 ATM switch. Researchers at Lucent's Bell Laboratories have shown that modules of this size can be compiled into a 160 Gb/s switch, eight times larger than today's commercially available model. The 160 Gb/s systems can in turn be combined to produce a terabit system, which switches traffic at 1280 Gb/s.

Three of these 20 Gb/s switches are currently working in a laboratory testbed network at Bell Labs in Holmdel, N.J. One is set into a system designed to accept 64 2.5 Gb/s signals, demonstrating the feasibility of the 160 Gb/s switch. Testbed experience and computer simulations indicate that the terabit switch can be successfully constructed from such systems with negligible degradation in performance.

Also part of the testbed is a network operations center, which offers a single point of control for the network, as well as software that incorporates a layered approach to call processing, creating the flexibility required by future services.

The system is not only feasible but can be built from components available today. The 20 Gb/s switches and the components for the prototype 160 Gb/s system are constructed in the 0.5 micron complementary metal oxide semiconductor (CMOS) technology used in widely available products such as Intel's Pentium chip. The switching fabric is the same as that offered in Lucent's ATM switch. Because the test network is based on commercially available and deployed products, it demonstrates that the scalability of current ATM networks to handle such speeds is entirely viable.

Memory sharing The terabit switch incorporates design strategies that offer not only very high-speed switching but also the best possible delay throughput characteristics, the ability to accept input signals at multiple bit rates, the ability to grow with demand and a high degree of system integration that reduces cost and maintains speed.

Three major developments contribute to the switch's performance: a concentrator-based scalable switching architecture, memory sharing and hierarchical multiplexing.

Memory sharing is crucial to the performance of the core switching module. This innovative memory design ensures optimal protection against cell loss and high throughput.

The 20 Gb/s electronic switching module-the terabit switch's core-takes in eight 2.5 Gb/s signals and produces eight 2.5 Gb/s outgoing signals. This size module achieves optimum density and performance-benefits that continue to accrue to the larger configurations as modules are added together.

A key question in designing an ATM switch is how to store individual cells. Each cell contains 53 bytes-48 bytes of data and five of address information. When large numbers of cells arrive, many of them destined for the same exit output port, there must be a means of caching and queuing them to await their turn at the output ports.

There are three locations in a switch where cells can be stored: at the input, somewhere in the middle and at the output. Storage at the output is the only efficient choice-if cells are stored at the input or in mid-switch, those waiting to leave by one port can slow down cells routed for others.

The question then is how to allocate enough memory for each output so that cells are not dropped, without the switch being prohibitively large or costly. The 8 x 8 switching module uses a shared memory-all ports use a common memory, and any storage location can retain a cell intended for any port.

This memory operates on the linked-list, first-in, first-out principle typical of computer memory. Cells are written into the next available memory location. All locations storing cells waiting in the queue for a given port are kept on a list. Cells in this virtual queue are then read out in order and routed on their way.

The shared memory technology used in the terabit demonstration is patented and should prove cost-effective. It is scalable to large sizes and minimizes space requirements-because no memory need reside on the line cards, each card can contain more ports, reducing the number of cards needed and resulting in a smaller footprint for the system. The memory-sharing technique requires memory devices that operate faster than the speed at which the switch actually switches cells. However, RAM devices now available in 0.5 micron CMOSs are fast enough to store and retrieve data at the rate needed to create a terabit switch.

In fact, some memory devices already achieve slightly faster speeds than those in the shared-memory prototype, and future switches will ride the curve of decreasing size and increasing speed of semiconductor devices. Therefore, the speed available should easily keep up with the speed needed at even higher switching rates. Line interface cards at both the inputs and outputs can handle traffic at OC-3c (155 Mb/s) and OC-48 (2.5 Gb/s) speeds in both Sonet and synchronous digital hierarchy (SDH) formats (Figure 1). Hierarchical multiplexers and demultiplexers take care of multiplexing the lower-rate signals up to the 2.5 Gb/s rate and back. The line cards, multiplexers and demultiplexers are combined with the switching module in one integrated system. This level of integration lowers costs and increases the efficiency of the switch. Concentrated signals To create a 160 Gb/s switch, it is feasible now to link eight 8 x 8 (20 Gb/s) modules together in an architecture that combines them into one integrated system. Specifically, the modules can be connected by a concentrator, which funnels incoming 2.5 Gb/s signals into groups of eight 2.5 Gb/s output signals that create the inputs for the 8 x 8 module (Figure 2). The concentrator used in the present system thus takes in 64 2.5 Gb/s signals and produces eight 2.5 Gb/s signals. Bell Labs has built a functional 64-to-8 concentrator. In the laboratory testbed, it funnels signals to one of the 20 Gb/s switches and demonstrates the feasibility of a concentrator network that can process incoming signals on a terabit scale.

Like the 8 x 8 module, the concentrator employs shared memory-if 64 cells arrive at the same instant, the memory retains those that don't get routed to the switching module in the first time slot. It is, in effect, a high-capacity storage system, sized so there's little danger of congestion. Although the concentrator is fed by 64 lines, at any given time only a fraction of the incoming cells in these lines are destined for the 8 x 8 module the concentrator supports. Generally there will be no queue for most ports. Queues that do exist will fluctuate over time like the checkout lines in a grocery store.

Because cells can be stored for indefinite lengths of time, the ATM switch is better at handling sudden peaks of usage than circuit-switched systems. For example, when demand exhausts all the available lines to the hot spot-such as a ticket service offering seats at the next Hootie and the Blowfish concert-circuit-switched systems merely return a busy signal to callers. The ATM switch, in contrast, can retain cells to the limits of its shared memories, up to the capacity specified by the purchaser.

Concentrators can theoretically be made that take in 512 2.5 Gb/s signals and funnel them into eight output signals. However, present memory devices aren't fast enough to handle this scale of concentration.

Instead, the terabit switch design uses a cascaded plan in which signals are funneled through two sets of 64-to-8 concentrators (Figure 3). Such devices can be built with today's memory technology, and the two-step approach entails virtually no loss in speed or capacity.

Basing the switch design on the 64-to-8 concentrator also contributes to its scalability because one of these concentrators serves one 8 x 8 switch module. If only 40 Gb/s of capacity is needed, one can install two 20 Gb/s switching modules and two concentrators. Capacity can be added in 20 Gb/s increments, up to the full 160 Gb/s. More modules could also be linked together, in which case a cascaded concentrator would be used. Physically, the concentrator takes up half a cabinet shelf.

Network control In the Bell Labs testbed, the three prototype switches are linked to a network operations control station as well as a variety of peripherals. Each switch is also governed by a local controller (Figure 4). The switches perform errorless manual fabric and line switching. Planned or manual switching, done with a PC for maintenance purposes, produces no errors because the switching fabric and line cards are duplicated. During testing, traffic can be switched over to the duplex components with no loss of cells.

The testbed is a real and complete operating network. The network operations control center, for example, has a fully developed PC/workstation graphical user interface (GUI) and a broad array of operations support tools.

Through the GUI, users can provision and monitor virtual paths and virtual circuits across equipment from multiple vendors, as well as restore the network in case of outage. They can also collect maintenance-related diagnostic information from the switches such as alarms indicating circuit pack failures and signal loss.

In addition, network personnel can call up a map of the network that emulates traffic among four cities linked by OC-48 facilities. Because one of the three testbed switches is duplex, it can act in the simulation as two switches serving two different cities. Users click on the graphical representation of lines between nodes to call up status information on facilities and to request information on specific virtual paths or circuits to isolate network faults.

The network operations center is based on a client-server model so more nodes can be added as the network grows.

The final push The push for ATM switching may come from network operators or end users, perhaps both. And end users are already pursuing ATM private local and wide area networks, adding impetus to get ATM into the public switching infrastructure. Network operators may deploy ATM switches where needed today. The systems can be integrated into any Sonet/SDH network-there's no need to build overlay or other special networks.

A key factor in ATM's acceptance, however, may be how clearly the path is defined from present switching speeds to future terabit rates. Demonstration of manufacturing and operational feasibility will be crucial, as will a clear evolutionary path from present systems to protect a network operator's capital investments.

Scalability will also be crucial. Scalability allows network operators to match switching capacity to customer demand-especially important for a new technology such as ATM switching.

Finally, operations support and software will be the capstone. ATM is different enough from circuit switching-and high-bandwidth services are different enough from lower-speed voice and data services-that new call processing and operational techniques will be needed to take advantage of ATM's flexibility. A terabit switch needs to be not just a piece of hardware but a truly comprehensive, planned system.

Kai Y. Eng is Head of the Broadband Systems Research Department at Lucent Technologies' Bell Laboratories, Murray Hill, N.J.