DSL, 2, 3, 4

Flow-through provisioning isn't always a cake dance Participants in the frantic DSL provisioning contest are filling out their dance cards and choosing partners as fast as they can. Compared with more sedate telecom contests such as the voice line two-step, DSL provisioning usually involves three participants: an ISP, a data competitive local exchange carrier and an incumbent LEC.

These three, which take turns partnering in some pretty dizzying hand-offs, haven't had much opportunity to practice working together. Thus, communication - an essential element if each participant is to make the right moves and not step on somebody else's foot or even trip over one's own - is tough.

This article dissects the three major elements of DSL provisioning: loop qualification, ordering and network element provisioning, and customer premises equipment (CPE) pre-qualification and installation. But to see how each of these elements is supposed to work, it's first necessary to look at what can go wrong.

The qualifying round The first step in DSL provisioning is loop qualification. It begins when a customer contacts his ISP to request DSL service. The customer can call or log onto the ISP's Web page and provide the salient service information - name, address and telephone number."In the worst-case scenario, a customer service rep takes the order by phone or over the Web and creates a work order," says Steve Roberts, vice president of Broadband Information Systems for ISP EarthLink. The work order can be batched with others sent to the data CLEC for service availability determination, which can take 24 hours or more. "The data CLEC responds in about 24 hours with service availability," Roberts says. "But we can't tell the customer on the phone or via the Web whether or not service is available."

From the data CLEC side of things, it's not so pretty, either. "ISPs might use anything from a sophisticated order-entry system to an Excel spreadsheet," says John Moshier, senior vice president of marketing at Austin-based DSL provider Vectris Communications. None of these manual methods works well.

Moreover, automation is key (Figure 1). "Non-automated requests take three to five days," says Vinu Sundaresen, Covad Communications' vice president of software engineering."The ISP gets a call from a customer, someone takes the information down on paper and hands it off to another department. Finally, it is passed on to Covad. Each manual step introduces errors. Order-entry errors create lots of delays."

Work orders now are automated at Earth-Link, but they're still done manually throughout the industry. Thus, just transmitting a request from the ISP to the data CLEC can trip up DSL provisioning.

Once the data CLEC has a request, real loop qualification can begin. Of all the elements of DSL provisioning, this one offers perhaps the greatest opportunity for missteps. Why? In a word: history.

According to DSL Forum board member Tom Starr, the questions that determine loop qualification include:

- From what central office (CO) is the customer served?

- Does a DSL access multiplexer (DSLAM) reside in the CO?

- What kind of copper loop does the customer have - how long is it and does it contain bridged taps or loading coils?

The first two questions are easy for a data CLEC to answer. "[The customer's address] and phone number determine the location of the CO and if it's part of Covad's network," Sundaresen says. "The network footprint for Covad is the structure of our DSLAM locations."

But the third question is a different matter. ILEC databases are not accurate about copper loops. Line lengths were often just incorrectly recorded, and bridged taps and loading coils made no difference for voice service. In fact, loading coils enhanced voice service on long lines.And voice service was, of course, historically the only service. Over time, as customers and lines changed, details about modifications to specific loops were lost.

"It was not necessary to keep good records at the time," says Dave Gellerman, vice president of technology and corporate development for Rockville, Md.-based service assurance vendor Hekimian. So, human nature being what it is, nobody bothered.

In practice, this means that even though records may indicate a loop is fine for DSL - information that typically gets back to the inquiring data CLEC via the ever error-prone e-mail or fax - it may not be. "We find out there is a problem when the technician goes out to install the service," says Eric Boyer, vice president of product and sales support for ASI, SBC Communications' data subsidiary. This results in a very frustrated customer, whom the data CLEC or ISP must now "manage."

Test system developer Teradyne has a slightly different angle on loop pre-qualification. The company's Celerity system allows incumbents to do single-ended insertion loss estimates over the narrowband test access path into their voice switches, which takes provisioning testing to on-demand status.

"Incumbents are wasting money on field personnel sent out to test lines that won't support service and losing revenue on lines they don't know can support service," says Chris Barton, product manager for Teradyne.

There is yet another spin to loop qualification. "Sometimes we hear anecdotal evidence that customers who might actually be able to receive DSL service are being told by providers that they cannot," says Pat Hurley, DSL analyst for TeleChoice. "There's enough demand for DSL from people whom the providers know can get service to keep the providers busy. So they may try not to provision more marginal customers - that is, marginal from the perspective of provisioning."

Taking orders Let's assume that a loop supposedly qualifies. The message limps back via e-mail or fax to the data CLEC, the ISP and finally - perhaps via an irritated phone call to inquire about the request - to the customer. This sets off a new conga-line of events. There is a work order for service, which will likely go through a sequence just like that of the initial loop qualification request. But now both the data CLEC and the ILEC have to begin provisioning equipment. And today "this is done by going to each network element - ATM switches, DSLAMs, modems, etc. - and provisioning each one," says Greg Bathrick of the DSL Forum's operations and network management group.

"We know the DSLAM and the copper pair," Sundaresen says."So now we assign the pair to a card in the DSLAM. We place the loop order with the ILEC: Here's the pair number we want to cross-connect." The next step is for the ILEC to verify the pair number. But recall that records aren't too accurate. What happens when an ILEC goes to verify the pair number and discovers some kind of line problem - the wrong pair number, a loop that's too long or the presence of bridged taps or loading coils?

Provisioning can be slowed or stopped, depending on what it takes to fix the loop. "For past data, we look where there are problems and we can find a group of loops and fix them," says Ed Glotzbach, chief information officer of SBC."For bridged taps and coils, we find loops that seemed to qualify properly, and then we go in and fix the data. We roll trucks to fix the loops," he adds.

Indeed, the problem may not even be with the end-user's loop. "ISPs interface with the world via an ATM switch in the ATM cloud," Glotzbach says."The assignment of that ISP's circuit on the ATM cloud may be mis-designated. [The ISP has] no continuity - they're not connected to the network where they're supposed to be. It's like building COs with transposed frame numbers. We can use statistical sampling to fix this, but it's too big a job to do manually. So we go back to the source records for the ISP, track down the problem and solve it."

ILECs can have other troubles as well."The technology bridges internal organizations that haven't traditionally been interfaced," says Eric Fogle, director of the wholesale DSL division for BellSouth. The organization is used to installing ATM or frame relay circuits for businesses, not residential customers. "Everyone is struggling with the complexity, and we try to leverage all of the people to let them work together," he says.

For the data group, this might involve adapting to a totally new user profile with all its ramifications for planning. For the tech group, it involves learning how to install data lines. For all groups, it involves building new organizational interfaces and enhancing those that exist.

While all this behind-the-scenes activity is occurring, customers are waiting. And they might be told at the end,"Sorry, you can't get service after all." It's not that anyone wants to tell a customer this; it's simply that there is a limit to resources (Figure 2).

"In the first two quarters of the year, we saw 50% quarter-to-quarter growth in installations," Hurley says."And if you have to roll a truck for every one of those installations, you simply run out of trucks and techs and time." Adding in truck rolls and tech time to fix lines before they can be rolled out for installations exacerbates the shortage.

Ultimately, the ILEC will transmit provisioning data to the data CLEC. The good news would be that the loop is ready, even though, as Sundaresen says, "an ILEC can take 10 to 15 days to do loop provisioning."And the bad news? "In some cases, they say they can't do it at all."

Sometimes, the news is so-so. Customers who are too far from a CO may qualify for ISDN DSL (IDSL)."It's a re-use of transceiver technology from ISDN to carry a DSL service," Bathrick explains. "The good part is that it uses a mature, low-cost technology that works over long loops. It can work with ISDN modems and mid-span repeaters. The bad news is that the best bandwidth is 144 kb/s. Even at that speed, it's always on, and that is better than dial-up."

If the final report from the ILEC is good, the rest is up to the data CLEC,which has to answer two more of Starr's provisioning questions:

- Is the customer served from the DSLAM or from a digital loop carrier (DLC)?

- If the customer is served from either one, is there spare capacity?

If the customer has service from either a DSLAM or DLC with spare capacity, then he or she can get service. If not, then the only options are IDSL or dial-up... for now.

CPE concerns Finally, we come to the last step in the worst-case scenario of DSL provisioning. It occurs only if the customer's line has really and truly qualified, the copper pair has been properly connected at a DSLAM or DLC that has available capacity, the ISP's ATM switch connection is good and the order didn't get lost along the way. It's the point where the customer's computer must be fitted with a DSL modem, cards must be installed, etc. This step also is rich with opportunities for trip-ups.

"Customers will try to get broadband without a computer that can support it," says Boyd Peterson, product marketing director of Austin-based DSL self-installation firm BroadJump. "Sometimes they will try to get broadband without a computer at all."

But if a customer's computer can support broadband, then the ISP or its DSL provider will send a representative to set up hardware such as network interface cards, line filters and whatever else is required."The technician also must configure the computer's OS, ensure that the computer has enough RAM and processing power and check to see if there are any other network conflicts," Peterson says.

Sometimes there's more to it. Joe Senesac, a Covad/EarthLink residential customer, had asymmetrical DSL installed at his Houston apartment in April."I waited about six weeks for the installation," he says."SBC had to provision the phone line. That took about five weeks. Covad came out, and they worked on the line at the same time as they set up the computer. There was a snag in this. We set up an appointment time, but Covad re-scheduled seven or eight times, then ultimately rescheduled to the original time."

The Covad tech who arrived was knowledgeable, Senesac says. Unfortunately, the tech was not psychic. A sixth sense would have helped because SBC did provision Senesac's line for DSL but didn't label it.

This was a problem because Senesac's apartment complex includes 12 buildings with about 10 or 12 apartments in each one. The entire complex is served from one phone room. Thus, Senesac says, "the Covad tech had to trace the line from my apartment to the snarl of wires in the phone room." Once there, the tech had to isolate Senesac's line from that snarl. "The Covad tech spent two hours figuring out what SBC had done," Senesac says. Once that happened, though, the process was straightforward, and Senesac's line was up and running immediately.

And there you have it: the best possible result of the worst-case scenario. Fortunately, a much better scenario is emerging.

Duende In flamenco,"duende" is a state in which the dance and the dancer become one, achieving moves that are virtually impossible. Flow-through provisioning certainly hasn't reached that peak of seamless grace yet, but it isn't from lack of trying.

What is"flow-through"provisioning?"It's an efficient, electronic data interchange method so that when an ISP keys in an order, the information can flow to the other companies' systems without human intervention," Starr says. "It's all automatic - `one-touch'provisioning."

It begins with loop qualification and ordering, Sundaresen says."We didn't want a phone tree for this." So Covad designed an extensible markup language (XML) application programming interface (API) called Crosslink that an ISP can incorporate into its own Web site. Seattle-based Speakeasy Network was the first ISP to implement Covad's API.

"Via the Covad API, service qualification takes seven seconds. Then the customer can pick the service, speed and so forth," says Speakeasy President and CEO Mike Apgar.

Once a loop qualifies and the order is placed, the next step is loop provisioning. To do so machine-to-machine via electronic data exchange and without human intervention requires a solution such as NightFire Software's, chosen by both Covad and Vectris.

"NightFire's key value proposition is change management," NightFire founder Venkates Swaminathan says. "The ILECs change interfaces a lot - four or five times a year - because of changes in specs or subsystems, increased levels of behind-the-scenes automation or new services. There are eight major ILECs - despite the mergers, they still operate as eight companies."

Each ILEC uses slightly different versions of standardized forms - different fields, values, ordering codes."One ILEC may use `X' as an ordering code for a service, whereas another ILEC uses `A' as its code for the same service. Each of these is a little thing, but each must be tracked," Swaminathan says.

So NightFire automatically translates a Covad or Vectris loop query or line order into the format the ILEC's own provisioning system can understand - and then translates the response back. NightFire also provides XML interfaces for its customers.

Electronic data interchange (EDI) isn't a perfect solution. At Vectris, the decision to bond electronically with an ILEC depends on the volume and difficulty of the transactions involved. If an ILEC's EDI is incomplete or immature, and if transaction volume is low, Vectris may not bond electronically. "Not all EDI formats and structures are standard, but the protocols are," Covad's Sundaresen says. "I'd rather have a non-standardized EDI solution than nothing."

At EarthLink, an ISP that interfaces directly with some ILECs and with several DSL providers that don't have flow-through provisioning, it's been necessary to develop an EDI system in-house. "Data CLECs and ILECs haven't offered flow-through on the ISP side," Roberts says. "We built our middleware because there are no standards, and there is a different interface for every company."

What about the problem of inaccurate data? It's a matter of "purifying" the data and ensuring its accuracy - finding errors and fixing them via the methods described previously, SBC's Glotzbach says."For new data," he says, "we can be 100% accurate. We are getting better by the day."

Inaccurate line data also can be solved with testing - before trucks roll. Hekimian offers line testing via its TestDSL product. Probes located in the CO co-location cage test copper loops, DSLAMs, ATM switches and aggregation routers.

"For pre-provisioning, we can measure line length and imperfections in the wire [such as bridged taps and loading coils]," Hekimian's Gellerman says. If that step is successful, "we look for the DSLAM on the line, the wide area network and the aggregation router to make sure the services provisioned match the services ordered and that all the links connect."

For unqualified lines, the only approach is to fix them. "A lot of ILECs - SBC, Verizon, BellSouth - are putting out remote access multiplexers and DLCs to neighborhoods," Starr says."Thousands of these are going out now and next year. It's well in progress already. We will see thousands of neighborhoods that will get high-speed DSL access in the next couple of years."

SBC's Project Pronto also is "rehabbing" loops by removing bridged taps and loading coils. "Month by month, it will become less true that you can't get access," Starr says.

The last step in DSL provisioning is the CPE qualification and line installation. "Self-installation is really the Holy Grail for providers," TeleChoice's Hurley says. "There has been some good progress in this recently, with ILECs like SBC and Verizon announcing self-install programs. [They're] getting a great response from customers, as well as 80%-plus rates when a customer tries to do the install." The DSL Forum just announced new self-installation standards in October.

BroadJump offers two solutions for CPE: Virtual Truck Qualifier and Virtual Truck Installer. VT Qualifier can be downloaded from the Web by potential customers or, if a service provider knows it has a DSLAM in a given area, the provider can mail the software to everyone.

"With Virtual Truck Qualifier, the software checks everything [on the computer] and tells the customer what they need," Broad-Jump's Peterson says. DSL providers [have different requirements], and our software is customized to the service provider and to the computer, according to the provider's needs." VT Installer comes as a kit, complete with DSL modem, line filters, required interface cards, software disk and instruction guide. "VT Installer automates everything a technician needs to do so the customer can do it," Peterson says. The company claims a 98% success rate with VT Installer.

Internal tracking is crucial to managing the internal flow of work orders, checking their status and moving that information down the chain to customers. It applies equally to ISPs, data CLECs and ILECs. Vectris uses MetaSolv Software for work flow management and for handling network inventory, capacity management and customer service. MetaSolv performs its tasks by acting as the in-house translator for internal provisioning applications - application A speaks to MetaSolv, which translates to a format application B can understand. MetaSolv won't speak directly to an ISP or to an ILEC, but it can translate information into formats that the gateways to ISPs or ILECs - such as a NightFire - can understand.

"Our software can take you all the way through the provisioning process, depending on your level of automation," says Beth Meeks, MetaSolv's senior product manager.

The final steps There are two final elements in flow-through provisioning that actually underlie solutions such as NightFire's or MetaSolv's.

The first of these is a foundation where business rules can be captured and documented. At Covad, that foundation is BEA Systems' open-standards, Java-based platform for developing provisioning applications: WebLogic.

"WebLogic models the entire provisioning system as a chain of items in a work flow, and it tells what needs to processed at any place in that flow," says Alfred Chuang, BEA president and chief operations officer. Each chain of items represents a business process such as the creation of an order or a change to an order. Business rules dictating the nature of these processes are captured within WebLogic, and these rules usually are fixed. However, Chuang says, "the flow of the process can be changed, depending upon the results of a particular event or on changes imposed from the outside." Thus, rules about loop queries or creating orders remain stable, but the information about either one can be called up in the provisioning process whenever it is needed.

The second underlying element captures business rules about the way processes relate to one another. At Covad and Vectris, this piece is provided by Vitria's Businessware for DSL. "Vitria is the e-business platform that automates the transactions and manages them," says Amitabh Shah, Vitria's telco solutions manager.

"Low-level middleware ties business applications down to hardware, equipment and operating systems," says Dale Skeen, Vitria's chief technology officer. "You need to acknowledge the order, the pre-qualification and the order acceptance. These interactions are based on business processes and rules." It's these high-level processes and rules that Vitria's solution captures. And because Businessware for DSL separates process logic from the functional code, the former can be changed whenever necessary without re-writing a lot of lines of code.

Dancing the night away Will it ever be possible to achieve true one-touch provisioning for DSL? SBC's Glotzbach believes the whole process will be relatively painless by the summer of 2001, but Covad's Sundaresen says, "flow-through just means you're going through the pain faster. It doesn't mean there's no pain."

Billing isn't well-integrated into the DSL provisioning process at all. "[It's] later in the food chain," Roberts says.

Some technology developers are campaigning for a vastly different approach to the provisioning process. Sedona Networks, for example, is developing an architecture based on concepts known as domain switching and distributed subscriber management, which are designed to put more control over provisioning for services such as DSL in the hands of the end customers.

"We're taking subscriber management from the core and putting it at the CPE," says Joseph Elchakieh, president and CEO of Sedona. "The advantage of that is scalability - services can be scaled as the customer needs it."

Domain switching will allow carriers simple methods of transfering control between access devices and switches and and back again, Elchakieh says, which lets them grant control over service customization to the customers. "You achieve mass customization by virtue of the customer choosing their own services," he says.

And at NightFire, Swaminathan is looking beyond DSL to broader provisioning issues. "The big challenge in the industry is not just DSL, but broadband," he says. "Automated service fulfillment is a huge challenge. The most important service is voice over broad-band, then VPNs." And eventually, no doubt, 100 Gb/s fiber to the home.

It looks as though the DSL contest may become a broadband marathon.