Precise synchronization: How to achieve circuit-switching performance in IP/MPLS networks

The low-cost and ubiquitous nature of IP continues to provide a compelling motivation for carriers and network operators to expand the scope and role of their IP networks to carry circuit-switched as well as real-time voice and video traffic. However, while packet networks are excellent at protecting data and guaranteeing that it eventually arrives, they are not so good at ensuring that data arrives in a timely fashion. In order for IP networks to perform carrier-class transport of data, additional mechanisms to reduce delay are required.

Two generic approaches to improving the quality and responsiveness of IP networks – Multi-Protocol-Label-Switching (MPLS) and Quality and Service (QoS) – offer promising results. However, without precise timing and synchronization, they fail to offer comparable performance quality to that provided by traditional circuit-switched networks. In order for IP-MPLS networks to accurately mimic legacy circuit-switched networks, precise synchronization via a Primary Reference Source (PRS) is required.

Time Delay Variation

Several network impairments prevent IP networks from carrying emulated circuit-switched traffic such as a T1 (1.544 Mb/s signal). A T1 line delivers a constant bit rate stream from node A to some node B on the other side of the network. (Note: the following discussion also applies to real-time voice and video traffic.) As packets travel through the network, delay accumulates at each intermediary node. In order to compensate for this delay, node B must use a jitter buffer to further delay packets in order to guarantee that there will always be a packet ready to be transmitted out. Where problems start to arise, however, is that each packet arrives with a different delay. The range of delay in which packets arrive is known as time delay variation (TDV) or jitter. Typically, with a large enough jitter buffer, packets will arrive in time to be useful. In the extreme case, a packet may be discarded or synthesized (erroneously) if the time delay variation exceeds the maximum the jitter buffer can accommodate.

TDV is a measure of the consistency of delay between packets. The larger the variation, the larger the buffer that is required to smooth it out since the buffer must be able to handle the maximum tolerated variation. Note that the buffer size is not determined by how much end-to-end delay there is but rather by how much variance there is in delay between packets. Buffers are inexpensive to implement because memory is cheap. Where large buffers become a problem is in the additional latency that they add to already delayed packets. Note that this additional latency is bounded by the most delayed packet; slow packets delay every other packet in the buffer, even those that have arrived quickly. An emulated T1 line may already be stressed because of delays inherent in network routing, and additional delay of an adequate jitter buffer may be enough to affect the delivery of real-time data.

MPLS

MPLS reduces TDV by creating a tunnel through the IP network. Instead of packets taking different routes through the network – a course that could introduce significant time delay variation between packets – an MPLS tunnel emulates a circuit-switched connection by fixing a route through the network. The fixed route adds consistency and narrows the variation in overall delay. Additionally, when packets travel over a fixed route, they arrive in sequence, eliminating the need to sort out-of-order packets and simplifying buffer management.

It is important to note that MPLS can change the fixed route when there's a problem in the network, such as a router going down. This is done in response to a problem, and is therefore an infrequent occurrence. When such a routing change takes place, there is a momentary change in TDV, which may result in a glitch in service. However, circumstances such as when a router goes down always result in some kind of glitch, regardless of the protocols or network topology in place. This feature of MPLS restores the tunnel quickly to provide the level of reliability required for carrier-class networks.

MPLS + QoS

While MPLS is effective at reducing TDV, oftentimes it is not enough. Packets must still pass through routers, which have their own queuing buffers and can introduce significant variable delay. For this reason, Quality of Service (QoS) mechanisms are often also implemented as well. QoS can be implemented in a variety of ways but in all cases, the key notion of QoS is prioritization. Within a router, for example, packets associated with a circuit-switched stream can be given higher priority than packets associated with non-real-time services. This results in packets from an emulated T1 line moving to the front of the router queue, further reducing the delay added at this router and therefore reducing the overall TDV.

While QoS also substantially reduces overall delay, it still may not be enough, for several reasons. While users may consider their T1 connection the highest priority traffic, it may not be the router's highest priority. For example, control packets have the highest overall priority, as do customers who have paid more for their bandwidth than others have. However, even if traffic has the highest priority, it must still contend with other traffic of the same priority.

In any case, high-priority traffic can be delayed by lower priority traffic if it arrives after a lower priority packet has already been placed in the output queue (see Figure 1). Typically, the largest packets traveling through a network are file transfers of the lowest priority. The router will place these packets in the output queue only when there is no other higher priority traffic to precede it. However, since routers don't interrupt packets once they start transmission, a high priority packet that arrives after a low priority packet has already begun to transfer will simply have to wait until the transfer is complete. For example, over a 1 Gbps link, a 1.5 Kbyte packet, regardless of the mechanisms in place, will add between 0 to 12 ms of delay, thereby increasing overall TDV by 12 ms.

Not Enough

While MPLS and QoS do much to reduce overall delay, together they still are often not enough to keep TDV within the limits required for reliable circuit-switched emulation. When packets enter and leave the IP network, how the underlying frequency of the circuit emulation stream is managed has a significant relationship to TDV. Consider two different circuit emulation streams meeting at a router. To simplify the discussion, assume that both streams transmit the same packet size and that no other transfers are taking place (see Figure 2).

In this example, Stream A is at a slightly faster frequency than Stream B. Over time, Streams A and B will progress through a cycle where 1) Stream A experiences varying delay because of Stream B, 2) there is no contention, 3) Stream B experiences varying delay because of Stream A, 4) there is no contention, and then the cycle repeats. What is important to note here is the introduction of delay variation introduced by interaction between the two streams, even though there is no other traffic, and even if the router load is miniscule. Even though the packets come routinely, the interaction is not consistent.

In the case with two streams of the same frequency, whatever the initial delay is – whether Stream A is delayed by Stream B or vice versa or there is no contention at all – will continue to propagate consistently for a delay variance of zero, even if delay is added (see Figure 3). However, a router typically handles more than two streams. If the streams are synchronized (see below), they will tend not to interact. However, if even one stream has a different frequency it will begin to interact with the others, knocking them out of synchronization until each pair of streams is interacting with the others. As a result, each stream will experience significant TDV.

Minimizing Variance through Synchronization

The mechanism for minimizing interaction between streams is precise synchronization. While streams will still experience delay, the streams won't collide or interact with each other and the delay will be constant, resulting in close to zero TDV.

Synchronization is most important at the periphery of the IP network; i.e., at both the ingress and egress ports because this is where the packet rate is determined. If the packets enter the network irregularly, then the frequency of the stream will be irregular. Precisely controlling the rate at which packets enter the network makes the stream frequency consistent. This in turn lowers the TDV and enables consistent control of the rate at which packets leave the network (see Figure 4).

In many networks, the edge is the only place where one can effectively control the packet rate. IP routers tend to employ FIFO architectures, and so offer little control of packet rate, per se. Synchronization at the edge of the IP network also serves to control TDV in “other people's” networks. Effectively, timing a packet before it enters the network and then retiming it as it comes out minimizes TDV and the impact it has on the egress serial signal. This also serves to ensure that the bit rate at egress matches the bit rate at ingress.

In order to minimize TDV through synchronization, it is critical to use a precise clock source, such as a primary reference source (PRS), also known as a primary reference clock (PRC). A PRS ensures that the maximum frequency difference between any two streams is less than two parts in 1011, or effectively a Df of zero. Common sources for a PRS include stand-alone cesium clocks and LORAN- or GPS-based wireless systems. Note that network timing signals sent over the IP network cannot be used for synchronization because they are as corrupted as the signals they are expected to retime. However, a PRS can be propagated a short number of hops through a network provided the transmission method supports frequency-transfer, such as SONET; the resulting timing signal is referred to as PRS-traceable. Any two nodes that are PRS-traceable, even if they reference different clock sources, can be considered synchronized to a common clock because each PRS is so accurate.

If every circuit-switched-to-IP edge boundary employs synchronization, then TDV can be effectively addressed. Note that if a packet enters the IP network without being retimed, it is no longer PRS traceable and has already accumulated variable time delay. With synchronization, TDV is reduced to only those parts of the IP network that are not retimed. By retiming packets and tunneling them through the network, it is possible to control variance.

Precise synchronization at the edges also relieves the network from having to transfer a sync reference. Packets would otherwise require time stamping and adaptive clock recovery (ACR) mechanisms. ACR mechanisms are typically not effective because it is difficult to achieve the jitter and wander specs required to transport T1 traffic. They are possible to achieve with a high-performance oscillator but this is not cost-effective. ACR is only required because packets lack consistency; i.e. have a high TDV. Even though the frequency of the T1 stream is “known”, the high TDV of IP networks dynamically changes that frequency from packet to packet. Put another way, the frequency is based on the actual rate that packets arrive, i.e., a short-term average and not a theoretical average.

With consistency through synchronization, there is no need for ACR since TDV is moot. Because packets are timed to a PRS at the ingress, timing is known: packets are timed to a PRS at the egress. Additionally, the low TDV reduces the minimum size of the jitter buffer and the impact it has on delay, as well as eliminates slips and buffer over/under runs. This method of synchronization is important enough that it is included in a recognized standard: ITU-T Rec. Y.1413.

Today we have parallel IP and circuit-switched networks. However, as more and more traffic moves to IP, there will be an increase in the amount of circuit-switched emulation and real-time traffic that IP networks will need to be able to carry reliably and in a timely fashion. Next-generation networks, while moving more data, still require precise synchronization and timing beyond MPLS and IP mechanisms in order to implement carrier-class reliability and performance by minimizing end-to-end delay. Without strict control of packet delay and time delay variance, migration to IP will be adversely affected and the touted cost savings of moving to IP will not be fully realized.

Kishan Shenoi is Chief Technologist, R&D, for Symmetricom.

Visit Symmetricom online.