Andy on Enterprise Software


June 14, 2010

Our research shows that building a business case is one of the major barriers when putting together an MDM initiative. Some of this is due to people not knowing the correct information to present in order to make a convincing business case, and in other cases it is due to the difficulty in estimating costs and benefits. This is actually not rocket science.

Join me for a webinar this Wednesday (at 11 EST = 16:00 UK time) on how to build the business case for an MDM project. It contains lots of practical information to make a proper business case, including data from our latest research e.g. on how much MDM projects cost to maintain once live.

You can register for it here:

It is even free!

Governing Data

June 5, 2010

This week I will be delivering the keynote speech at the IDQ Data Governance Conference in San Diego (funny how they never hold technology conferences in Detroit or Duluth). This promises to be an excellent event, with over 350 registered attendees, and plenty of movers and shakers in this emerging field. Data governance is the business-led strand that is beginning to bring together the hitherto curiously separate worlds of MDM and data quality, and it will be interesting to see what leading end-user companies are doing in this field.

Something for nothing

February 25, 2010

In early June there is the annual Data Governance Conference:

which this year is in the attractive setting of San Diego (the place with perhaps the best climate in the USA). Naturally as a conference delegate you will be influenced solely by the agenda and the speaker quality rather than the prospect of a sunny location, but I just thought I’d mention it.

There will be some excellent speakers, and also me giving the keynote. As a reader of this blog I am happy to offer you a discount should you be able to attend. Just quote the following code when booking: IDDG100 – please be aware that this code expires on May 7th.

Sunlight is the best disinfectant

December 15, 2009

I read a very interesting article today by independent data architecture consultant Mike Lapenna about ETL logic. Data governance initiatives, MDM and data quality projects are all projects which need business rules of one kind or another. Some of these may be trivial, and as much technical than business e.g. “this field must be an integer of most five digits, and always less than the value 65000”. Others may be more clearly business-oriented e.g. “customers of type A have a credit rating of at most USD 2,000” or “every product must be part of a unique product class”. Certainly MDM technologies provide repositories where such business rules may be stored, as (with a different emphasis) do many data quality repositories. Some basic information is stored within the database systems catalogs e.g. field lengths and primary key information. Databases and repositories are generally fairly accessible, for example via a SQL interface, or some form of graphical view. Data modeling tools also capture some of this metadata.

Yet there is a considerable source of rules that are obscured from view. Some are tied up within business applications, while there is another class that are also opaque: those locked up within extract/transform/load ETL rules, usually in the form of procedural scripts. If several source files need to be merged, for example to load into a data warehouse, then the logic which defines what transformations occur are important rules in their own right. Certainly they are subject to change, since source systems sometimes undergo format changes, for example if a commercial package is upgraded. Yet these rules are usually embedded within procedural code, or at best within the metadata repository of a commercial ETL tool. Mike’s article proposes a repository that would keep track of the applications, data elements and interfaces involved, the idea being to get the rules as (readable) data rather than buried away in code.

The article raises an important issue: rules of all kinds concerning data should ideally be held as data and so be accessible, yet ETL rules in particular tend not to be. It is beyond the scope of the article, but for me there is a question of how the various sources of business rules: ETL repository, MDM repository, data quality repository, database catalogs etc can be linked together so that a complete picture of the business rules can be seen. Those with long memories will recall old fashioned data dictionaries, which tried to perform this role, but which mostly died out since they were always essentially passive copies of the rules in other systems, and so easily became out of data. Yet the current trend towards managing master data actively raises questions about just what the scope of data rules should be, and where they should be stored. Application vendors, MDM vendors, data quality vendors, ETL vendors and database vendors will each have their own perspective, and will inevitable will each seek to control as much of the metadata landscape as they can, since ownership of this level of data will be a powerful position to be in.

From an end user perspective what you really want is for all such rules to be stored as data, and for some mechanism to access the various repositories and formats in a seamless way, so that a complete perspective of enterprise data becomes possible. This desire may not necessarily be shared by all vendors, for whom control of business metadata is power. An opportunity for someone?

MDM and Spaghetti

September 14, 2009

When looking at the business case for MDM it is normal to look at the kind of business initiatives that can be enabled by better master data. For example with higher quality, consistent customer data it is possible to run more efficient marketing campaigns, or by having a complete picture of a customer it is possible to cross-sell products effectively or better manage an account. However such things tend to rely on having MDM as a piece of infrastructure, so it is hard to claim all the benefits directly for an MDM project. Perhaps it is time to take a look at some of the more murky and less sexy areas that can benefit from MDM, specifically by lowering the cost of maintaining application interfaces.

Large companies have hundreds of applications, even after they have finished implementing an ERP system (and then re-implementing it to reduce the number of ERP instances). One company I work with owns up to 600 applications, another to 3,500. In many cases data needs to be shared across applications, and of course the very fact of having so many systems can cause master data issues to occur, since each application frequently generates and maintains at least some master data that it needs rather than being fed such data by a consistent enterprise-wide master data repository.

One key difference between MDM hubs and a data warehouse is that a warehouse needs to have clean, pure data; this is achieved by an extensive process of data cleansing and validation that is conducted outside the warehouse prior to data being loaded, perhaps through a mix of data quality tools and ETL processing. Indeed one major issue is that in order to come up with high quality data for the warehouse, business rules end up being embedded in sometimes complex ETL scripts, which are opaque and hard to get business people to engage with. A good master data hub should be able to take on much of this burden of strong and managing business rules, and may be a more productive place to carry this out. For example it may be more effective to use probabilistic techniques to help determine matches angst potential data sources (say, multiple product masters) rather than needing to hard-code business rules, as usually happens with ETL scripts. If this is the case then you may be able to get away with a much smaller set of business rules in an MDM hub than were typically necessary in ETL scripts. In turn this reduction of complexity may be able to cause a lot of the maintenance effort needed in maintaining such scripts to be go away.

I have not seen any quantitative analysis out there of the relative productivity of MDM hubs v ETL processing for storing business rules, or the potential effect that this could have on the support effort needed to maintain interfaces. However it was always the case that a high proportion of overall support effort in an enterprise was associated with interfaces, so even a small effect here could have a significant saving in terms of IT costs. I think there is an opportunity here for someone to do some serious research into this area, getting hard data rather than making hand-waving benefits statements. If followed through, it would not surprise me to see this as an area where properly implemented MDM could have a significant effect on IT support costs. Given that so many people claim that making a business case for MDM is one of the biggest obstacles, this would seem to me a fruitful area of further research.

Something for Nothing

August 31, 2009

I have now completed the second of my on-line courses on master data management for eLearning Curve. This one goes into considerable detail on how to evaluate an MDM vendor, based around an in-depth MDM functionality model which I have developed (and which has been through a significant review process by some serious MSM experts). The course also looks at the MDM market and talks about the current vendor Landscape in some depth, and finally goes through a suggested process for software procurement, including some tips and hints I have learnt by being on both sides of the negotiating fence.

The course can be accessed here:

Its price is what seems to me almost absurdly cheap (eLearning Curve is new and they are trying to promote things), but as a reader of this blog you can take advantage of a special offer as well. When buying the course just quote the following voucher code:


and you will get a further discount of 20% off the already amusingly low list price. Seriously, this is a real bargain. Over five hours of chunky, in-depth material, to absorb at your leisure.

As the old Derek Bok saying goes, if you think education is expensive, try ignorance.

Training without travelling

July 31, 2009

I have spent some time recently in building up an MDM course. Normally such things are done at conferences, so unless you happen to be at some distant venue, they pass you by. However this one is different. It is an on-line course, marketed by a new company called eLearning (who have some well-known founders). The company reckons that it is harder and harder for people to justify trips to conferences for education, and this is certainly true at the moment from what I have observed and heard about technology conferences. Hence its model is to sell courses on-line.

The course is “MDM Fundamentals and Best Practice”, and you can see more about it here. It is actually quite a lot of work to put together such a course, but I am pleased with the result, and you can now get over three hours of my views on MDM for a very fair price indeed, all from the comfort of your desk or living room, and at a pace that you can control. Of course you do miss out on the trip to Las Vegas or similar, but you can’t have everything.

Stylish MDM

April 9, 2009

We have recently completed a major survey into the deployment styles used in MDM implementations. My colleague Dave Waddington has recently posted a summary of the results here. As can be seen, MDM projects are turning out to be quite meaty in size, but encouragingly the sucess rates were higher than I was expecting.

There were several quite interesting results that came out, and we will be doing further research into this area. The full report can be purchased from our website.

MDM Styles

February 2, 2009

There are a number of approaches of “styles” to tackling MDM within a company, at least in terms of what to do first. If your most pressing issues are improving the quality of business reporting you may opt for “analytical” MDM, or if your issues are mostly with the data in transaction systems you would go for “operational MDM”. Within such categories there are different degrees of invasiveness, from a “registry” style where you leave the master data intact within operational systems, to an extreme root and branch approach “transaction style” where you rip out the ability of operational systems to maintain master data and put in new MDM system(s) to do this and feed the rest of the enterprise, while “co existence” recognises the reality that it may be necessary to live with a mix of approaches. But just how many companies go with which style, and how successful are they?

To answer this The Information Difference is launching a major piece of market research. This survey, sponsored by Microsoft and with media sponsors DM Review and CIO Magazine, is an in-depth look at this topic. To take the survey please click here.

When the survey is complete and the analysis is done I will let you know. Participants will receive a summary of the finding.

Running Against the Tide

January 9, 2009

We recently completed the Q4 “Market Landscape” for MDM. As part of this we looked at all the vendors in the market and obtained figures for software revenues and growth of each. One interesting aspect of this is that the MDM software market so far appears to be holding up well. Indeed it is currently growing at an annualised rate of 30% according to our research,a healthy clip. It should be noted that the market size figures that we use exclude systems integrator revenues associated with MDM – these are estimated at around three times the size of the software market. As an aside, it is these kind of assumptions that can lead to seemingly wide discrepancies between market size estimates from different firms; typically you see a figure quoted in the press, but what does it include and exclude? Our figures for MDM software exclude data quality vendors, which are handled in a separate twice yearly update.

These figures, which are admittedly retrospective, confirm our November market research study looking at the effect of the financial crisis on MDM spending. This found that about as many companies were planning to accelerate their MDM spend as were planning to slow it down or defer projects (admittedly nearly a third of respondents were undecided).

So far at least, then, both actually Q4 spend as seen by vendors, and spending intentions in our survey are telling the same story. MDM software revenues are holding up well. We will continue to track the market closely, with a Q2 2009 Market Update to be published in July.