Category: Master data management

In doing the research for our six-monthly update of the MDM market I came across something which surprised me. Generally master data can be complex (such as bill of materials structures) but is not generally very large in volume, at least compared to transaction volumes. The exception is the “customer” dimension in a B2C company, where it is easy to see that 50 million or so records may be needed. Luckily “customer” data is usually quite simple compared to, say, product data, which may have hundreds of attributes.

However I have come across three cases now where the volume of master data records being managed is claimed to be around 500 million records. One vendor I spoke to said they had a customer planning a billion record MDM system. Dealing with hundreds of millions of records rather than tens of millions is a lot more challenging, especially where the data need to be dealt with in real time e.g. if you are adding a new customer account then you need to check whether that apparently new customer account is really a duplicate of an existing account; this should ideally be done straight away.

If anyone reading this has come across one of these really large MDM implementations then I’d be interested to hear your experiences.

I noticed an interesting blog post by Andrew Brooks about a possible side effect of the credit crunch, that of an increasing interest in MDM and data quality by recruiters. This may sound paradoxical, but it makes sense. Companies struggle to get good management information (for example about levels of counterparty risk for trading organisations such as investment banks) due to inconsistent master data across multiple systems. When times are booming this may be glossed over, but with prestigious companies going to the wall on a daily basis, being certain of the information that you rely on gets a higher priority. The blog resonated with me since I have just had a couple of recruitment agencies call me in the last few days asking what this “MDM thing is all about” in response to recent client inquiries.

Every major company I talk to struggles with getting reliable enterprise-wide data, and in every project I have been involved with, the data quality in corporate systems is worse than people think it is. If the current tough financial conditions prompt a new focus on fixing these issues, then indeed perhaps there will be a modest silver lining at the end of the credit crunch cloud.

There is an encouraging increase in MDM education, reflecting the growing interest in the subject. As well as Aaron Zornes’ pioneering work at the MDM Institute, we had a dedicated TDWI MDM conference in Savannah earlier in 2008. Now TDWI is running a dedicated MDM track on its next week-long conference, in New Orleans in early November. Just in case you didn’t have enough reason to attend, I will running a half day workshop as part of the conference.

For more information or to download a complete copy of the brochure, visit: www.tdwi.org/neworleans2008

Register before October 4, and receive an early registration discount, if you use this code when booking.

Priority Code: IN34

By early November it is nearing the end of hurricane season, and you can take advantage of New Orlean’s famous easy-going atmosphere and fine music and cuisine (at least after you have finished the workshop sessions). You may even learn something.

Hope to see you there!

Increasingly when I talk to customers about their plans for master data management or data quality, the area of data governance comes up. Data governance is the set of business processes and controls that surround the lifecycle of data, rather than necessarily involving technology. Some MDM vendors provide sophisticated support for data governance, others less so, but I believe that it is key to the success of a data initiative. Without thinking about the processes, it is of limited use to just put in a piece of technology, or carry out a once-off data cleansing exercise without considering what happens next.

Indeed in talking recently with data quality vendors, a number have commented on how the increasing profile that data governance is getting has enabled them to, in turn, get a higher profile. Previously data quality was seen by many companies as strictly something for IT or those direct mail geeks in their marketing department, but now data quality is getting an airing as part of broader initiatives to improve data across the board.

In order to put more flesh on the data governance bone, The Information Difference is conducting a major piece of primary market research into data governance. You can be part of this by taking the survey Completed surveys will receive a full copy of the research, and there is even a prize draw with exciting and valuable prizes (this view may depend a little on your taste) to tempt you.

In a survey of telecoms companies by Yankee Group, it would appear than master data management has made it to the second highest investment prioity amongst the wireless, wireline cable, and satellite companies. This is interesting since it appears to confirm the generally very bullish predictions for MDM as a market, which I wondered whether may be affected by the general economic climate. After all, MDM is not a trivial matter on an enterprise scale, so it could be tempting to defer an MDM project and put it into into the “too difficult” pile for a year or two to see how the economy recovered.

Later in the year the Information Difference will conduct some research amongst MDM vendors to see how the predictions from analyst firms such as Forrester and Yankee are stacking up to the reality of a tricky economy.

There are some good points in the DM Review article on MDM by William McKnight. In particular, I like that he clearly highlights some issues that some parts of our industry are still in denial over:

(a) that customer and product data are just a small, though important, part of the master data picture for an enterprise.
(b) in reality, most companies will end up with more than one approach in MDM “style” or architecture, due to sheer complexity of the enterprise application environments out there. I feel that too often IT architects spend time agonising over the most “elegant” approach when in fact the tools are still evolving, and will no doubt go through more than one iteration of architecture over the next decade. Given the savings to be made out there by improved master data management, waiting for a perfect solution may be a costly and missed opportunity.
(c) data quality is a key part of an MDM project. Indeed, as my colleague Dave Waddington said the other day: “if you don’t have any data governance processes, what exactly is the point of doing a data quality project?”. One-off data quality initiatives may have value, but without the processes to fix the disease rather than treat the symptoms, the patient is never going to get truly better.

Although it might seem obvious that “The best place to manage master data is in the operational environment” as it says in the article, I am not sure that in practice this is always right. Some companies with a very centralised approach may have the discipline to really drive home a single master data system across the enterprise (I was speaking to one just the other day) but many do not. For de-centralised companies it may well make sense to consider alternative architectures, as it may simply be impractical or go against the grain of the corporate culture to have a central, harmonised set of operational master data.

Other than that caveat, it seemed like generally sage advice.