We have now completed our survey of data quality. Based on 193 responses from IT and business staff from around the world, there were some very interesting findings. Amongst these was that 81% of respondents felt that data quality was much more than just customer name and address, which is the focus of most of the vendors in the market. Moreover, customer name and address data ranked only third in the list of data domains which survey respondents found most important. Both product and financial data was felt to be more important, yet product data is the focus of barely a handful of vendors (Silver Creek, Inquera, Datactics) while of all the dozens of data quality vendors out there, few indeed focus on financial data. Name and address is of course a common issue and conveniently is well structured and has plenty of well-established algorithms out there to attack it. Yet surely the vendor community is missing something when customers rate other data types as higher in importance?

Another recurring theme is the lack of attention given to measuring the costs of poor data quality. Lots of respondents fail to make any effort to measure this at all, and then complain that it is hard to make a business case for data quality. “Well duh”, as Homer Simpson might say. Estimates given by survey respondents seemed very low when compared to our experience, and also to anecdotes given in the very same survey. One striking one was this: “Poor data quality and consistency has led to the orphaning of $32 million in stock just sitting in the warehouse that can’t be sold since it’s lost in the system.” This company at least has no difficulty in justifiying a data quality initiative. The survey had plenty of other interesting insights too.

The full survey and analysis, all 33 pages of it, can be purchased from here.

As part of our ongoing research program, we are conducting a major survey into the state of data quality today. If you have a few minutes it would great if you could participate in this (all participants get a free summary of the survey results).

The survey can be found here:

In addition your e-mail address will be entered for a prize draw offering you the chance to win one of ten free annual subscriptions to The Information Difference website (worth USD $ 550).

Thanks in advance.

Most data quality vendors have their roots in name and address checking, even if their software can go beyond this. What is less well known is that the actual business of getting street level address data (to verify postal codes etc) is a tedious business that varies dramatically by country (the UK post office database covers almost every address in the UK, but Eire has no post code system, for example). Software vendors do not typically want to be in the business of updating street address databases, and there is a patchwork of local information providers that fill the gaps. If you have any international aspirations, though, just discovering who does what by country, and licensing the various data sources is in itself a non-trivial task, and so companies exist that do this. One was a UK company called Global Address, bought some time ago by Harte Hanks (who market Trillium), while the other was Address Doctor. Many data quality vendors use Address Doctor, including some that might superficially appear to compete. These include Dataflux, IBM, and even QAS. Some MDM platform vendors also use Address Doctor, who provide at least basic name and address data for 240 countries and territories.

The cat was put firmly among st the pigeons this week when Informatica bought Address Doctor. From their viewpoint this secures a key provider of address data, and follows their prior acquisitions of Similarity Systems and, more recently, Identity Systems. Informatica, via these purchases, has established itself as one of the major data quality vendors. Given its competitive position, the data quality vendors who use Address Doctor will, at the least, be feeling nervous. I spoke to an executive from Informatica this week and was told that Informatica intended to honour the existing arrangements, but who knows how long this state will last? As Woody Allen said, the lion may lay down with the lamb, but the lamb won’t get much sleep.

The problem for the other vendors is that there is no obvious place to go. Global Address is already in the hands of Harte Hanks, and while Uniserv in particular has its own name and address data, it is mainly strong in this area in Europe. Address Doctor was a convenient neutral player and is now in the hands of a major market competitor, and other vendors may have little choice but to look at building up their own networks of address data providers if they are to sleep easy. Of course it is not clear that they have to worry; for example Pitney Bowes Business Insight (who have what was Group 1 software) use Global Address, and this arrangement has continued without incident despite Harte Hanks Trillium’s ownership of them.

It will be interesting to see what measures the current Address Doctor users take, or whether they will just cross their fingers and hope Informatica plays nice.

We have recently completed a major survey into the deployment styles used in MDM implementations. My colleague Dave Waddington has recently posted a summary of the results here. As can be seen, MDM projects are turning out to be quite meaty in size, but encouragingly the sucess rates were higher than I was expecting.

There were several quite interesting results that came out, and we will be doing further research into this area. The full report can be purchased from our website.

I have recently been spending some time looking at the data quality market, and a few things seem to pop up time and again. The first thing is, in talking with customers, just how awful the quality of data really is within corporate systems. One major UK bank found 8,000 customers whose age was over 150 according to their systems. All seemingly academic (if you are taking money out of your account, who cares what your age is?) until some bright spark in marketing decided that selling life insurance to these customers would be a fine idea.

Story after story confirms some really shocking data errors that lurk beneath most operational systems. These are the same operational systems that are used to generate data for the end-year accounts which senior executives happily sign off on pain of jail-time these days. I hope no one shows these sames execs the data inside some of these systems, or they might start to get very nervous indeed.

Yet in a survey we did last year, only about a third of companies in the survey have invested in data quality tools at all! Does anyone else find this in any way scary? Do you have any entertaining data quality stories you can share?

There are a number of approaches of “styles” to tackling MDM within a company, at least in terms of what to do first. If your most pressing issues are improving the quality of business reporting you may opt for “analytical” MDM, or if your issues are mostly with the data in transaction systems you would go for “operational MDM”. Within such categories there are different degrees of invasiveness, from a “registry” style where you leave the master data intact within operational systems, to an extreme root and branch approach “transaction style” where you rip out the ability of operational systems to maintain master data and put in new MDM system(s) to do this and feed the rest of the enterprise, while “co existence” recognises the reality that it may be necessary to live with a mix of approaches. But just how many companies go with which style, and how successful are they?

To answer this The Information Difference is launching a major piece of market research. This survey, sponsored by Microsoft and with media sponsors DM Review and CIO Magazine, is an in-depth look at this topic. To take the survey please click here.

When the survey is complete and the analysis is done I will let you know. Participants will receive a summary of the finding.

We recently completed the Q4 “Market Landscape” for MDM. As part of this we looked at all the vendors in the market and obtained figures for software revenues and growth of each. One interesting aspect of this is that the MDM software market so far appears to be holding up well. Indeed it is currently growing at an annualised rate of 30% according to our research,a healthy clip. It should be noted that the market size figures that we use exclude systems integrator revenues associated with MDM – these are estimated at around three times the size of the software market. As an aside, it is these kind of assumptions that can lead to seemingly wide discrepancies between market size estimates from different firms; typically you see a figure quoted in the press, but what does it include and exclude? Our figures for MDM software exclude data quality vendors, which are handled in a separate twice yearly update.

These figures, which are admittedly retrospective, confirm our November market research study looking at the effect of the financial crisis on MDM spending. This found that about as many companies were planning to accelerate their MDM spend as were planning to slow it down or defer projects (admittedly nearly a third of respondents were undecided).

So far at least, then, both actually Q4 spend as seen by vendors, and spending intentions in our survey are telling the same story. MDM software revenues are holding up well. We will continue to track the market closely, with a Q2 2009 Market Update to be published in July.

I am sure most of us have experienced some dismal experience when calling a technical helpdesk, being put through a maze of automated menus before finally getting through to some bored, half-trained “engineer”. An example of this from my own past is here. However usually the one bit of an organisation that is fairly responsive is the sales function, for obvious reasons. I relate the following personal example of the risks that a high-tech company can take when it decides to “economise” in this area.

I recently needed to buy a new printer and rang up Dell to do this. Dell now pushes its “consumer” (for which read “cattle class”) sales though to an offshore centre. I placed my order and awaited the confirmation email: nothing. The following day I called, went through the menus again and explained what had happened to another Dell sales person. He told me that there was no trace of my order in the system, do I duly went through the whole thing again, wondering quite what happened to the original details, which after all included my credit card details. I specifically said to he salesman “please make sure that the order was not duplicated”, and I even gave him the name of the sales lady I had originally spoken to. “No problem” I was assured.

A week or so a printer duly arrived, and all seemed well. A couple of days later there was another knock on the door, and, surprise surprise, a second identical printer arrived. I declined delivery, explaining the situation. Well, you can probably guess by now that, despite me sending back the duplicate printer, I was charged twice, so I duly rang up and spoke to the original sales lady who took the initial, seemingly lost order. At least I did this after gong through no less than five separate people at Dell, each of them insisting that I tell the story again and each taking full details (name address, order number, …) every time, as if computers had never been invented.

The thing that reduced me to apoplexy at the end of this was that the sales lady (I could not make this up) suggested that in order to get a refund, I would have to take delivery of a fresh printer, uninstall the one I have sitting on my desk, rebox the current one and sent it back to Dell. Oddly enough, I have declined to do this – pesky customers eh?

So, if anyone from Dell is reading this, please give my money back. For any company executives considering outsourcing their sales operation to a cheap location, just consider what effect this may have on your customers. I can’t say I am exactly itching to order another Dell product at the moment. A customer’s perception of a brand, after all, is significantly derived from their personal interactions with the company. The irony is that Dell used to have an award winning call centre based in Ireland, but seemingly decided that this was all a bit expensive. I am not suggesting that the precipitous decline in their share price over the last eight years can be specifically linked back to their decision to save money on their call centres, but I suspect that it will not have helped.

In doing the research for our six-monthly update of the MDM market I came across something which surprised me. Generally master data can be complex (such as bill of materials structures) but is not generally very large in volume, at least compared to transaction volumes. The exception is the “customer” dimension in a B2C company, where it is easy to see that 50 million or so records may be needed. Luckily “customer” data is usually quite simple compared to, say, product data, which may have hundreds of attributes.

However I have come across three cases now where the volume of master data records being managed is claimed to be around 500 million records. One vendor I spoke to said they had a customer planning a billion record MDM system. Dealing with hundreds of millions of records rather than tens of millions is a lot more challenging, especially where the data need to be dealt with in real time e.g. if you are adding a new customer account then you need to check whether that apparently new customer account is really a duplicate of an existing account; this should ideally be done straight away.

If anyone reading this has come across one of these really large MDM implementations then I’d be interested to hear your experiences.

Usually when economic times are tough then there are a series of things that the bean counters do to rein in costs. First they ban business travel except for customer facing situations, followed by freezing training budgets and recruitment, before sharpening their knives more seriously. The first two of these mean that conferences are usually at the sharp end of corporate spending cuts. I have just returned from speaking at a BI/MDM conference in Amsterdam, and was pleasantly surprised to see a healthy attendance of paying customers, a large number of which appeared in the last few days. This must have been a considerable relief to the conference organisers, and it was certainly nice to be speaking a packed room rather than one with rows of empty seats.

It is certainly hard to be sure just how deep recession is likely to be, with seemingly contradictory data all around. A scary figure is the cost of shipping (the Baltic Dry Index) which is a reasonable predictor of the flow of trade. This has dropped a little matter of 95% since the peak in June, with shipping companies cancelling orders and talking of mothballing ships. This kind of broader economic data would seem to suggest a fairly sharp recession is on the cards, and this must feed through into lower IT expenditure eventually, just as night follows day. Of course some areas will be hit harder than others, but I am always suspicious of claims that a certain area is “strategic” and so will be unaffected. Usually this is whistling in the dark by vendors. We shall see.