Your business intelligence arsenal

Data mining and data warehousing may seem like sexy models for organizing and leveraging information but only if they enable carriers to solve specific problems

The business intelligence market will be worth $148 billion by 2003, according to Survey.com. It is growing at exponential rates; is critical to every business, regardless of size; has every major technology company as a player; features e-everything delivery and interaction; offers more capability at much lower prices than a year ago; is commercializing top-secret technology from formerly off-limits government programs; has some of the hottest IPOs and best performing companies of the last two years; and has nowhere to go but up.

Business intelligence consists of all activities related to organizing and delivering information and analysis to businesses. This includes data mining, knowledge management, analytical applications, reporting systems and data warehousing. The business intelligence space is an exciting place to be today but only if service providers leverage it to provide solutions that solve specific business problems.

Bottoms up

The primary components in a business intelligence infrastructure are data warehousing systems. These systems integrate data from a variety of operations systems. They cleanse the data to remove errors. They also standardize data so that products, customers, profit and revenue are consistent across the system. Data warehousing systems integrate the data so that information from different systems can be combined to yield information and analysis such as lifetime value of a customer or profit by product.

In the early to mid-1990s, many organizations attempted to build their business intelligence infrastructure data warehouse elements in a top-down monolithic fashion (Figure 1). This approach attempted to model the enterprise, then incrementally build a central mega-data warehouse resource.

This has become known as the "dream of homogeneity" because it assumes and demands a consistency of systems, data and architectures inconsistent with the heterogeneous nature of a business environment. These large-scale, enterprise-class projects had trouble delivering value to the business, with studies showing failure rates from 30%, according to the Meta Group, to 80%, according to DWN and OTR.

Enterprise data warehouses aren't the only large-scale projects having trouble. A recent Boston Consulting Group study showed a 70% lack of success rate in large-scale enterprise projects involving enterprise resource planning (ERP) and customer relationship management systems. These high failure rates led to the development of an alternative approach to the enterprise data warehouse: "bottom-up." It involves the creation of highly targeted, architected data marts that are integrated into the resulting data warehouse system. The BCG study found that small, targeted solutions are five times more likely to be rated as a success by the business.

Surprisingly, some organizations still adhere to the top-down, monolithic data warehousing approach. However, both methods - top-down and bottom-up - are viable given a suitable political and cultural environment. Top-down monolithic approaches are sure death in organizations that lack the senior-level support and the political and communications skills required to be successful. In addition, top-down monolithic approaches cannot accommodate today's heterogeneous mix of custom data warehouses; turnkey, packaged data warehouses; data mining; and analytical applications (Figure 2).

Technological considerations, such as architectures, approaches, tools and technologies, are meaningless to the business - it is a fast, measurable impact on the business that counts. Finally, the business makes the rules, not the technologists.

Federated architecture

The current business intelligence market is built on the foundation of a modern infrastructure, consisting of a federated architecture accommodating all the components of a contemporary business intelligence system.

This system includes:

- Packaged/turnkey data warehouses and data marts

- Packaged/turnkey analytical applications

- Custom-built data warehousing and data marts

- Custom-built analytical applications

- Data mining

- Online analytical processing tools

- Query and reporting tools

- Production reporting tools

- Data-quality tools

- Extraction transformation and load (ETL) tools

- System management tools

- Information delivery tools

- Enterprise information portals

- Reporting systems

- Knowledge management systems

- Database systems

The federated business intelligence architecture provides the foundation and environment to facilitate and enable business information flow, analysis and decision-making.

Similar to the Internet being a network of networks, a federated data warehousing architecture is an architecture of architectures (Figure 3). It provides a framework for the integration of disparate data warehouses, data marts and analytical application systems. A federated data warehousing architecture is the most pragmatic route to providing the maximum amount of architecture possible given the political and implementation realities of real-world sites.

The federated architecture shares as much core information among the various systems as possible. This is accomplished by sharing critical master files or dimensions, common metrics and measures and other high-impact data across all systems that can make use of the information. It usually is accomplished via an enterprise-class ETL tool, which provides a common meta data repository and the use of common data staging areas.

Apps for telecom

A packet-based telecommunications company has high demands for business intelligence and often has a business model based on core business intelligence functionality, such as bandwidth-/usage-based billing and real-time configuration. To accommodate these needs, a federated business intelligence architecture is required for the heterogeneous requirements inherent in providing the near real-time analysis required by the networking organization, along with service and support team requirements and the billing, usage and analysis needs (Figure 4).

Telephony and packet business intelligence systems face special challenges in the areas of data volume and real-time data streams. While typical data warehouse systems are considered large if they contain a terabyte of data, a packet system easily can contain 10 terabytes or more.

To provide support for provisioning, support and dynamic billing, the system also must manipulate large volumes of data in near real time. The data must be gathered from a worldwide network of devices, cleansed, integrated and aggregated within minutes. These requirements are well beyond run-of-the-mill architectures, ETL tools and server systems found in everyday data warehouse systems and therefore require special expertise, experience, techniques and technologies to be successful.

Regardless of its technical elegance or purity of design, no business intelligence system will survive if it does not provide direct business value and solve a specific business problem. The most popular ways to achieve this goal is via analytical applications and data mining.

The most popular form of business intelligence usage from the business perspective is via packaged, turnkey analytical applications. A true, high-impact, analytical application is defined by the following characteristics:

Architected, integrated data from multiple sources. An analytical application can include information from multiple sources, both native online transaction processing (OLTP) applications, as in the case of an analytical application offered by an ERP vendor, and external information from heterogeneous OLTP systems or third-party vendors. Note that many ERP vendor-supplied analytical application offerings cannot capture, leverage or use any external data. This shortcoming cannot be emphasized enough as service providers consider the implications of an environment composed of disparate, non-architected analytical applications, each with its own semantics.

Flexible, multidimensional analysis, drill-through and reporting. Analytical applications allow business users a flexible environment to view business metrics and measurements by pertinent dimensions, with any required number of members. Analytical applications allow seamless drill-through into pertinent detailed transactions and flexible and easy movement across dimensions and measures. They also provide the capability to view and report information in all forms required by the applicable business processes, including detailed lists and summary cross tab.

Turnkey package/short time to market. Analytical applications feature rapid deployment with easy data extraction and integration into OLTP packages and data sets; indigenous online analytical processing (OLAP) or native support for industry standard OLAP engines; pre-formatted, pre-defined relevant business metrics and key performance indicators; and implementation-ready agents, reports and aggregations.

Integrated business processes. Analytical applications provide domain-specific solutions to particular business challenges, including internal representations of relevant business processes. Analytical applications provide an interactive environment to interact with the business process by presenting applicable metrics and measurements of processes and the ability to interact with, and alter, process values and measures.

Self-measuring. Analytical applications provide internal value measurements of relevant business processes. They monitor the ongoing use of the analytical application itself and its effects on the business process. In doing so, they provide ongoing return on investment analysis of the business process. In addition, they monitor the use of the analytical application and provide an active monitor into the propagation of the tool throughout the organization, the relative sophistication of the usage of the system, optimization of the system and identification of best practices regarding usage of the system.

Closed loop system. An analytical application provides a closed loop, feeding new inputs back into the host OLTP or data warehouse system. As users interact with the business process, they introduce new information or alter existing information, as in a budgeting and forecasting system. These new values then are fed back into the source systems as new or modified information for use by users of the source system and downstream business intelligence systems.

This new or altered information must flow back into the analytical application in real time or near real time. This places challenges on the technical infrastructure of data warehouse systems that are more accustomed to monthly, weekly or daily information refreshes. It also places heavy demands for massive recalculation and re-allocation of data, as in budget vs. actual calculations or performance vs. plan. An even greater challenge is that these write-back, flow-through prerequisites require a level of process rigor and structure diametrically opposed to the free-form flexibility required of a successful business intelligence system. This is a key technological and cultural hurdle that many teams cannot overcome.

The data mining weapon

Data mining solutions are a key weapon in the business intelligence arsenal. They reveal trends and relationships and predict future outcomes. They are built on variations of artificial intelligence such as neural networks, machine learning and genetic algorithms. Data mining tools are a powerful technological and competitive weapon and form the underpinnings of powerful product offerings and infrastructure and support capabilities for packet companies.

Most organizations use data mining tools to discover previously unknown relationships, trends and anomalies and to predict future outcomes. On the customer side of the house, these capabilities are used for target marketing, churn management, fraud detection and promotion management. Packet content business intelligence systems also can use data mining tools to track, trend and predict network volumes, spot significant outlier behavior, optimize system configuration and performance and optimize the structure and design of customer offerings.

A federated business intelligence system is a prerequisite to survive and thrive in today's fast-changing and evolving market. Without the capabilities provided by integrated data, powerful analytical tools and insightful data mining applications companies are at a tremendous disadvantage and find themselves unable to compete with their better informed and capable competitors. With the players, the customers and the fundamental possibilities of the market changing daily, no organization can afford to deny itself the power of business intelligence.