Data Perfect

As technology races forward, carriers are urged to seize the data.

New tools and technologies can offer you benefits in cost reduction and improved customer relations. Customer-relationship management (CRM) technologies, for example, offer a means for you to coordinate customer service over the Web, by telephone and in person. To ensure the success of a CRM system, you must invest in a complete data-integration infrastructure in order to tie together various data sources into a consolidated customer framework, forming the foundation for a variety of customer-acquisition, -retention and -management programs.

Creating a consolidated customer-information infrastructure requires five key components: assessing source data quality levels; integrating disparate data; de-duplicating and cleansing; consolidating data definitions; and publishing data definitions.

By implementing an information infrastructure that addresses these issues, you can be more confident that corporate data will support the business decisions, customer-acquisition and customer-retention initiatives. Without these components, you will build customer-data stores ripe with duplicate information or inaccurate or incomplete data that will result in wasted customer-retention and -capture campaigns and, more importantly, lead to company-integrity issues.

Assessing Source Data Quality Levels
Prior to integrating data into a common customer database, information technology (IT) must assess the data-quality levels of the various source systems. Given the diversity of data available in most companies — custom transactional applications, packaged applications, departmental data, external data — data-quality levels will vary. Although all sources may be "reliable," they probably were created at different times and for different reasons. As such, these sources may not share a common format and set of business rules. It is crucial to understand these differences and resolve them prior to or during the data-integration phase.

One of the best ways to perform this data-quality assessment is through the evaluation of the content and structure of data against the business rules and data structures that govern business practices. These rules and structures can be represented through comprehensive data filters that are applied to samples or entire data sources. The results of this analysis highlight data-source adherence to these rules or even pinpoint specific data-quality issues, providing a thorough understanding of the data-quality condition that can help drive decisions on data integration, de-duplication and cleansing. More sophisticated systems allow you to assign values or costs to specific data-quality issues resulting in reports that can show overall data-quality-problem costs from various systems. Furthermore, once these filters and costs are established, tests can be repeated over time intervals to monitor the trends and variations in data-quality issues as new data is introduced or efforts are made to improve the data quality.

Integrating Disparate Data
Typically, the most time-consuming and fundamental task of creating a common customer data warehouse or data store is integrating the data from the various source systems. A variety of commercial products exist to ease this step, providing tools to extract the data from source systems, transform and integrate the data and load the data into the target data warehouses or marts. When used properly, these products can offer significant productivity advantages over traditional hand-coding methods for data integration, but you first need to ensure that the products fit your needs.

You'll have your own list of core requirements for data-integration tools, but the overarching requirement for any tool in this category is that it fits the existing environment and does not require significant changes to accommodate the product. This includes natively supporting all existing data sources, platforms and skill sets. Tools that require significant new skill sets, introduction of new platforms or that do not natively support the data sources will introduce more complexity and costs into the equation than necessary. Other critical success factors include scaleability — products that can extend to handle the tremendous volumes of data typical of telecommunications customer information — and usability. Finally, it is important to look for products that integrate with data-quality assurance products or technologies to modify or improve data found to be errant in the data-quality-analysis phase.

De-Duplicating & Cleansing
Often considered part of the data-integration step, data cleansing is especially critical when integrating wireless-customer information. Although many data-integration tools can perform basic data cleansing as part of the transformation phase, there's a specific set of data-cleansing routines centered on customer information that can improve the data's quality drastically. Basic data-cleansing routines center around de-duplicating and consolidating data from the various data sources to ensure there is only one record for each customer. More sophisticated systems also identify possible data-entry errors and automatically correct ZIP codes, verify street numbers and ensure compliance with the U.S. Postal Service requirements.

Lucent InterNetworking Systems (INS), formed as part of Lucent Technologies' merger with Ascend Communications, leveraged data-cleansing software from Firstlogic as part of its data integration and consolidation task.

"It helped us build a customer warehouse featuring refined data that produces clean, reliable information and most importantly a single customer view, which we use to drive marketing decisions," said Arturo Munoz, Lucent INS knowledge management solutions manager. Munoz found data-cleansing technology from Firstlogic to be "essential to the success of our customer warehouse, (as it) matches and standardizes customer data as it enters the consolidated database."

Consolidating Data Definitions
The final part of consolidating data from various sources is consolidating data definitions. Source systems may have different definitions of common attributes such as name, address, telephone number, age or demographic information. To ensure consistency, these definitions must be reconciled and a single set of definitions must be determined.

The first step in this process is extracting these definitions from the source systems. More commonly called meta data, or data about the data, these definitions may exist in the source database systems, or products or applications that create or use the source information. This meta data needs to be extracted and loaded into a consolidated meta-data directory, creating a corporate "yellow pages" of the data definitions. IT then must work with business managers to agree upon a set of standard definitions. Once standard definitions have been created, the product and technologies used in data integration and data cleansing can transform the data to these common definitions.

Publishing Data Definitions
Once the consolidated data store has been created with integrated, cleansed information, it is ready for use. A variety of analytical tools exist for this purpose, but one additional step, critical to the success of these tools, is sharing the consolidated data definitions with these tools and their users. As marketing managers prepare campaigns or customer-service personnel respond to customer issues, they must completely understand the definitions of the data they are viewing.

It is not enough to indicate that a subscriber has a "good" credit rating or that the subscriber is in the "upper middle class" economic bracket. You must get a comprehensive definition of these identifiers. A comprehensive definition comes from the meta data created during the data-definition-consolidation phase and should be fully exposed to the user, in business terms that he understands. In addition to defining what a "good" credit rating means, it is useful to understand where the supporting data for that rating came from, when it was last updated, and if there are any known data-quality issues with that data.

Integrated Best-of-Breed Technology
Although many of the steps identified here can be accomplished by hand-coding the necessary routines together, or using point technology products, there is a significant advantage to using an integrated set of products. An integrated set of products ensures that data and meta data can pass seamlessly between the products, and that the products have been proved to work together to solve the broader business problem. No longer relegated to mundane support operations, the time has come to embrace innovative technology to revolutionize business practices and propel companies to the forefront of the wireless industry. "Know thy customer" is an age-old adage that is perhaps more relevant today than ever. Technology now exists to make this a reality. Carpe data.

Boehm (jeff.boehm@ardentsoftware.com) is Ardent Software director of product management for data warehousing.