http://www.debtechint.com/dg2010/

which this year is in the attractive setting of San Diego (the place with perhaps the best climate in the USA). Naturally as a conference delegate you will be influenced solely by the agenda and the speaker quality rather than the prospect of a sunny location, but I just thought I’d mention it.

There will be some excellent speakers, and also me giving the keynote. As a reader of this blog I am happy to offer you a discount should you be able to attend. Just quote the following code when booking: IDDG100 – please be aware that this code expires on May 7th.

Yet there is a considerable source of rules that are obscured from view. Some are tied up within business applications, while there is another class that are also opaque: those locked up within extract/transform/load ETL rules, usually in the form of procedural scripts. If several source files need to be merged, for example to load into a data warehouse, then the logic which defines what transformations occur are important rules in their own right. Certainly they are subject to change, since source systems sometimes undergo format changes, for example if a commercial package is upgraded. Yet these rules are usually embedded within procedural code, or at best within the metadata repository of a commercial ETL tool. Mike’s article proposes a repository that would keep track of the applications, data elements and interfaces involved, the idea being to get the rules as (readable) data rather than buried away in code.

The article raises an important issue: rules of all kinds concerning data should ideally be held as data and so be accessible, yet ETL rules in particular tend not to be. It is beyond the scope of the article, but for me there is a question of how the various sources of business rules: ETL repository, MDM repository, data quality repository, database catalogs etc can be linked together so that a complete picture of the business rules can be seen. Those with long memories will recall old fashioned data dictionaries, which tried to perform this role, but which mostly died out since they were always essentially passive copies of the rules in other systems, and so easily became out of data. Yet the current trend towards managing master data actively raises questions about just what the scope of data rules should be, and where they should be stored. Application vendors, MDM vendors, data quality vendors, ETL vendors and database vendors will each have their own perspective, and will inevitable will each seek to control as much of the metadata landscape as they can, since ownership of this level of data will be a powerful position to be in.

From an end user perspective what you really want is for all such rules to be stored as data, and for some mechanism to access the various repositories and formats in a seamless way, so that a complete perspective of enterprise data becomes possible. This desire may not necessarily be shared by all vendors, for whom control of business metadata is power. An opportunity for someone?

Another recurring theme is the lack of attention given to measuring the costs of poor data quality. Lots of respondents fail to make any effort to measure this at all, and then complain that it is hard to make a business case for data quality. “Well duh”, as Homer Simpson might say. Estimates given by survey respondents seemed very low when compared to our experience, and also to anecdotes given in the very same survey. One striking one was this: “Poor data quality and consistency has led to the orphaning of $32 million in stock just sitting in the warehouse that can’t be sold since it’s lost in the system.” This company at least has no difficulty in justifiying a data quality initiative. The survey had plenty of other interesting insights too.

The full survey and analysis, all 33 pages of it, can be purchased from here.

The cat was put firmly among st the pigeons this week when Informatica bought Address Doctor. From their viewpoint this secures a key provider of address data, and follows their prior acquisitions of Similarity Systems and, more recently, Identity Systems. Informatica, via these purchases, has established itself as one of the major data quality vendors. Given its competitive position, the data quality vendors who use Address Doctor will, at the least, be feeling nervous. I spoke to an executive from Informatica this week and was told that Informatica intended to honour the existing arrangements, but who knows how long this state will last? As Woody Allen said, the lion may lay down with the lamb, but the lamb won’t get much sleep.

The problem for the other vendors is that there is no obvious place to go. Global Address is already in the hands of Harte Hanks, and while Uniserv in particular has its own name and address data, it is mainly strong in this area in Europe. Address Doctor was a convenient neutral player and is now in the hands of a major market competitor, and other vendors may have little choice but to look at building up their own networks of address data providers if they are to sleep easy. Of course it is not clear that they have to worry; for example Pitney Bowes Business Insight (who have what was Group 1 software) use Global Address, and this arrangement has continued without incident despite Harte Hanks Trillium’s ownership of them.

It will be interesting to see what measures the current Address Doctor users take, or whether they will just cross their fingers and hope Informatica plays nice.

Story after story confirms some really shocking data errors that lurk beneath most operational systems. These are the same operational systems that are used to generate data for the end-year accounts which senior executives happily sign off on pain of jail-time these days. I hope no one shows these sames execs the data inside some of these systems, or they might start to get very nervous indeed.

Yet in a survey we did last year, only about a third of companies in the survey have invested in data quality tools at all! Does anyone else find this in any way scary? Do you have any entertaining data quality stories you can share?