X-ray vision: CEC gets visibility into last-mile provisioning

When utility provider Consolidated Edison diversified four years ago to provide optical fiber-based communication services for Metro New York, the new company, Con Edison Communications, created a legacy-free infrastructure management and provisioning system--and gave customers a unique view deep into quality-of-service provisioning.

That deregulation has spurred electric and gas companies to jump into the communications sector shouldn't be surprising: They frequently rely on the same physical routes above, below and in between city streets and buildings to reach right to the desks of corporate employees.

One of the biggest barriers that CEC and all like-minded entrants in today's telecommunications market face is how to establish a network services management and provisioning system that minimizes what is otherwise significant cost overhead and potential source of customer dissatisfaction.

CEC's fresh start in the market hinged on finding reliable answers to all the requirements of an operations support system (OSS) and fastest provisioning (see Table 1 below and Figure 1).

Table 1: Significant OSS requirements that CEC addressed
Customer Management	Maintaining information on customers, buildings and contacts.
Contract Management	Maintaining information about customer contracts as well as contracts entered into by CEC for outside services and building leases.
Order Management	Managing the order lifecycle, status, implementation time, and non-standard terms and conditions, which involves generating on-demand status and project reports, tracking due dates, issuing work orders and escalating late tasks.
Trouble Management	Managing the service troubles lifecycle from creation to customer acceptance of trouble clearance. This involves accepting troubles as they are reported by customers and the NOC, as well as automatic trouble reports from partner companies and the CEC network management system, and then notifying SLA management of any service outage.
Performance Management	Recording performance measurements from the network equipment, which involves detecting threshold violations, creating trouble tickets and providing performance reports.
Physical Plant Management	Managing the physical equipment in the network including GIS location information.
Network Engineering	Supporting the design of products and services.
Service Level Management	Determining if service outages or degradations have violated SLAs, and passing this information to Billing and Finance.
Fault Management	Collecting fault information from the network, determining if it affects service and providing remote access to diagnose and correct faults.
Configuration Management	Providing configuration requests to the network equipment to implement customer service orders. This involves maintaining a logical inventory and topology information for the network.
Billing	Billing customers for services delivered.
Financials	Managing the financial information for CEC.
EPortal	Interfacing customers and internal users to a predefined set of functions via Web access from a browser.

Getting a new handle on provisioning

To enable these objectives, CEC crafted what it calls a "Smart-Build" approach. Smart-Build is a strategy for automating order management and provisioning an entire telecommunications network. Crucial to its success is an emphasis on automating functions that other service providers still typically perform manually.

The focal point of this tactic with Smart-Build is a new network hub built in New York City. The 22,000 square-foot facility provides a major point of interconnection with carriers (216 fibers to each of 12 major carriers), a collocation facility for customers, and a site for CEC's network operations center (NOC). Three separate hardware platforms have been set up to separate the development, test and production environments.

The network hardware includes:

• A Cisco 15454 Optical Transport Platform, which supports Sonet and SDH networking standards

• A Nortel OPTera Metro DWDM Platform, which is bit-rate and protocol independent, while supporting Sonet and SDH networking standards.

The Production Hardware Platform consists of:

• HP-Compaq, UNIX, Alpha and WIN2K Intel-based servers, with all servers configured in high availability clusters with no single point of failure

• A HP-Compaq Storage Works SAN (storage area network).

The Development and Disaster Recovery Platform includes:

• HP-Compaq, UNIX, Alpha and WIN2K Intel-based servers

• A Compaq Storage Works SAN

• Oracle Database replication.

Smart-Build was designed to closely match the above capital expenditures with actual business growth. Hoping to avoid the uneven infrastructure growth and inflated expectations that are the bane of the industry, CEC's IT planning group carefully considers the order in which every component of the overall OSS (operations support system) is deployed. Then, when an order does come in, CEC moves quickly, usually furnishing the requested services to a building within 90 days.

Using software to look beneath manhole covers

By June 2001, CEC had completed its OSS planning requirements for Smart-Build, and the IT group had mapped required functionality against vendor technology, yielding a short list of potentially useful solutions. A successful track record of integrator consulting was also essential, leading to selection of HP TeMIP (telecommunications management information platform), a network and services management solution for multi-vendor and multi-technology environments. It augments standard monitoring and problem resolution capabilities with high levels of automation and customer access to service-related information. The solution was available through Compaq (now merged with Hewlett-Packard), which acted as systems integrator for the project.

Currently in use at CEC for Phase 1-A testing, the new management framework connects to Cisco and Nortel network elements to contribute to a high-availability configuration through standard management of faults, alarms and problem resolution, including trouble ticketing. In Phase 1-B, TeMIP will link with powerful mapping technology that can show the precise physical location of all cables and fibers in the network. Pairing management with mapping will enable NOC operators to identify and locate problems so they can dispatch repair crews quickly, maintaining maximum network availability.

Subsequent Phases 2 and 3 are planned to enhance many of the first-round capabilities, and CEC configured the solution so that other software applications can be plugged into TeMIP as middleware. This is a "plug-and-play" framework permitting integration of different software applications in the network and service-level areas, which is necessary for true flow-through provisioning slated for Phase 3 and completion all within two and a half years.

Telco systems typically comprise millions of dollars of assets that are believed to be active in the infrastructure, but in reality are not. This is a problem that CEC resolved now that NOC personnel can use the element management system to retrieve inventory information from working network elements and reconcile with the GE Smallworld inventory management system automatically (Figure 2).

In contrast, CEC models a Transport Service Controller (TSC), which connects to access modules. Given access to the interface of the network elements, it can activate circuits into these elements and build circuits across multiple elements, networks, vendors and so on. The IT engineering staff simply identifies the end points of the circuit. Then uses inventory provisioning and activation to seek out the affected Sonet rings and target the components making up the circuit. The components in the circuit are tagged and the equipment in the network is activated. This enables the software to automatically configure the circuit within moments by directing commands through a workflow process to the network elements.

This rapid circuit-building process is coupled with powerful mapping capabilities that show the precise physical location of all cables and fibers in the network. This has enormous service and maintenance implications. For example, say a service pack goes bad in the field and a technician is dispatched. With all the network components connected to a single, central location, the technician can look at a street map and tell where the circuit is by looking at a NOC screen.

The screen also directs technicians to the correct manhole and identifies which card and rack is bad. After removing the faulty card and inserting the new one, technicians use an IPAC to scan the card's code. The scan is automatically sent back to TeMIP's physical inventory database, which electronically updates its card information in real time. This keeps the database pristine, preventing the TSC from building circuits on equipment that is no longer or is already in use. A geographic information system (GIS) component is also updated and the circuit route is highlighted on the map.

These mapping capabilities can serve as the basis for prioritizing trouble tickets. For example, a service provider working for a priority customer can tag a faulty circuit to be repaired first in the inventory and GIS databases. All affected components are physically marked so that they are not interfered with while the repairs are taking place.

Mapping capabilities are also valuable for network planning. As already mentioned, the provisioning process tags the equipment required to build a circuit. Every time that equipment is tagged, it shows that it is being reserved from the inventory database. IT staff can then determine what capacity is being taken up and what idle capacity is left. Network engineers can use this information to activate another fiber cable, install another rack, or otherwise plan for future growth.

Dealing with performance irregularities is the crux of the second phase of CEC's OSS deployment. TeMIP issues alarms based on performance traits alone. A higher level of performance management is possible, however, by integrating TeMIP with ADC Metrica.

CEC achieves this by enhancing existing TeMIP TL1 Access Modules for Cisco CTM and Nortel Optera to retrieve performance metrics from CTM and the Optera network elements on demand. TeMIP periodically collects data from the network equipment and deposits it in a specified directory on the Metrica NPR system. Metrica NPR periodically scans this directory and incorporates new data into its Oracle repository. TeMIP then formats the data for NPR.

Operators at the network operating center, launching a Web GUI provided by Metrica for Windows NT workstations, access the NPR reports. Reports can be generated on demand or periodically. As Metrica NPR processes incoming data and generates reports, performance management data can be examined for threshold crossings and, when necessary, an alarm sent to TeMIP. Threshold-crossing criteria is defined by an administrator working within the Metrica Performance Alarm Server (PAS).

When a threshold crossing is detected, an alarm is sent to a TCP port on the TeMIP system. The TeMIP-Metrica Liaison product on the TeMIP system contains a GAT (Graphical ASCII Toolkit) module that listens on this port, receives the alarm and converts it into a standard TeMIP alarm.

A key benefit of integration with ADC Metrica is the ability to trend performance management alarms. It's one thing for a self-healing network to clear its own alarms, but Metrica provides trending insight into the performance data based on rules-based thresholds that tip off the system to take action long before an alarm is issued.

Provisioning through portals

Depending on access levels, clients can securely and conveniently access back-office network systems from the Web, including systems for managing trouble tickets, orders, bills and payments. Premium-level clients are able to get additional access to alarms, maps of network engineering, as well as performance management tools, reports and graphs (Figure 3).

Additionally customers can import data and specify how it should be integrated into their own reporting and troubleshooting systems. Alarms, threshold notifications, and trouble tickets can all be customized to support unique reporting requirements. As such, there is continuous assurance that SLAs are being met on every level. Moreover, customer and service provider are viewing the same data used to formulate billing statements.

Ultimately customers may gain direct access to the solution architecture as an outcome of true, flow-through provisioning. They will be able to order services and manage bandwidth externally through the portal service. Say, for example, a television news network wants to activate a video link for a sports event. The network could simply use a PC to login to a portal, activate the necessary circuit themselves, and go directly on the air.

Larry Linkens is Vice President of IT and Support Systems at Consolidated Edison Communications. He was one of the founding technical architects of the TeMIP-based IT telecom management infrastructure now used by CEC. A widely recognized spokesman within the telecommunications industry, Larry makes frequent presentations to user groups and forums. He will address the United Telecommunications Conference in October 2002.