Microsoft generally likes to acquire software companies when they are quite small, with a dozen or two employees. In this way they can assimilate the development staff into Redmond and into the Microsoft way of doing things. An example of this was last week, when they decided to acquire a data quality technology. There are literally dozens of data quality vendors out there, most fairly small, and so there was plenty of choice. They opted for Zoomix, a small Israeli company which I first encountered in 2006, though they were founded in 1999. Zoomix had some quite clever marketing, claiming “self learning” technology as a way of making data profiling in particular more productive. In this way it could be compared to Exeros, although the technology underpinnings are quite different.

In this case the R&D team will move into the Microsoft technology centre already in Israel. This is a logical move by Microsoft, who acquired Stratature in order to give them an MDM capability. This product is currently being retooled under the code-name Bulldog, and a data quality offering to complement this is a natural fit. The timing around Bulldog’s release are unclear at this point, as it is folded into the SQL Server release timeframe.

At the Information Difference we continue to add new research on the MDM vendors. One of the things that is useful to know about vendors when drawing up a shortlist is which platforms the various vendors support e.g. which database, which web server, and perhaps more minor but useful technical information about whether they have double byte character support, do they have 7 x 24 helpline support etc.

There doesn’t seem to be a place where this kind of information is gathered together but there is now.

http://www.informationdifference.com/vendor_technical_details.html

This also has information on the level of SOA support (if any), which non-English languages are supported in the user interface etc.

There are some good points in the DM Review article on MDM by William McKnight. In particular, I like that he clearly highlights some issues that some parts of our industry are still in denial over:

(a) that customer and product data are just a small, though important, part of the master data picture for an enterprise.
(b) in reality, most companies will end up with more than one approach in MDM “style” or architecture, due to sheer complexity of the enterprise application environments out there. I feel that too often IT architects spend time agonising over the most “elegant” approach when in fact the tools are still evolving, and will no doubt go through more than one iteration of architecture over the next decade. Given the savings to be made out there by improved master data management, waiting for a perfect solution may be a costly and missed opportunity.
(c) data quality is a key part of an MDM project. Indeed, as my colleague Dave Waddington said the other day: “if you don’t have any data governance processes, what exactly is the point of doing a data quality project?”. One-off data quality initiatives may have value, but without the processes to fix the disease rather than treat the symptoms, the patient is never going to get truly better.

Although it might seem obvious that “The best place to manage master data is in the operational environment” as it says in the article, I am not sure that in practice this is always right. Some companies with a very centralised approach may have the discipline to really drive home a single master data system across the enterprise (I was speaking to one just the other day) but many do not. For de-centralised companies it may well make sense to consider alternative architectures, as it may simply be impractical or go against the grain of the corporate culture to have a central, harmonised set of operational master data.

Other than that caveat, it seemed like generally sage advice.

Just to let you know that the Information Difference has released its first piece of primary market research, as reported by IT Pro. There are some intriguing snippets in the survey results, as well as some rather more expected results, or the “well, duh” results as Homer Simpson might say.

13% of the (mostly larger) 112 companies surveyed had over 100 systems that hold and maintain customer data, which gives some idea of the scale of the problem that MDM is tackling. It is bit more than “just put in a hub” when you have systems at this level of complexity.

Given the generally flaky level of data quality reported, I found it surprising that nearly a third of the companies in the survey had not purchased an automated data quality tool.

The good thing was that plenty of companies seem to have begin measuring the costs of poor master data, and those costs are high, which should make it easier to justify master data management initiatives.

I have wondered for some time about Informatica’s intentions in the MDM space. There had been some market rumours about them possibly buying Siperian, but it seems as if their not so secret meetings with Siperian were actually to form a partnership, which was just announced.

This is an eminently sensible partnership in my view. In most MDM projects there is going to be much gather of data from multiple places, and so MDM vendors typically need to rely on integration, or at least data movement technologies that enterprises have already deployed. Assuming that this relationship will be exclusive (unclear from the press release), Siperian has locked in a relationship with the last remaining independent ETL and integration player of note (Ab Initio is also still out there, but is so secretive that it is part vendor, part cult). Informatica gains a foothold into the fast growing and important MDM market, so it is a win-win as far as I can see.
Perhaps one day this relationship will lead to something more binding, but for now these two vendors appear to be dating rather than tying the knot.

Today sees the launch of the Information Difference, a boutique market research and analyst firm specialising in the master data management market. This reflects the increasing interest in this fast-growing area. The company has developed detailed profiles of all the vendors in the MDM space, as well as of some of the major and most interesting players in the related data quality space. The company will shortly announce its first piece of primary research (into MDM adoption) and will produce white-papers on key issues in the MDM market.

Its principals are Dave Waddington (ex Chief Architect at Unilever Foods) and myself, with some part-time assistance from a number of other talented individuals. It is nice to see some positive reactions from some serious industry luminaries (see press release).

We hope to bring a more in-depth perspective to this emerging market than is common today, and have some exciting research in preparation.

For more information see the company website.

There is an interesting web forum which seeks to bring an open source approach to the world of data management. Of interest are topics involving the creation of open source de-duplication, profiling, matching and cleansing tools (hat tip to CW for pointing this out).

No doubt the tools here are at an early stage and won’t directly compare in broad functionality with a major data quality vendor. However, for many people with less sophisticated requirements that may not matter. The rise of products like MySQL has shown how influential an open source product can become given the right circumstances.

I would be very interested as to whether any readers of the blog have any experience with the tools here, or any views on the merits or otherwise of an open approach to data quality and data integration.

I was amused by a piece regarding data quality in which a data quality initiative at a chemical manufacturer was kicked off only after a warehouse burnt down and the company discovered that they had no way of tracing which customers would be affected. I recall a similar example at Shell, where a data quality initiative received a serious management boost when an oil well was drilled into an existing well (fortunately it was not in use at the time) due to faulty positional data in a computer system.

These kind of incidents demonstrate that management care about data quality when there is a crisis, but when things are running smoothly it is usually a long way down the priority lists of businesses. This is a puzzle in many ways, and frustrating for the data quality software industry, where few vendors see widespread deployment in enterprise initiatives. Usually data quality software is implemented (if ti is at all) in a piecemeal project by project fashion, and many companies have no data quality software deployed at all.

Perhaps it is just one of those subjects that few get excited over. Author Kurt Vonnegut once said: “Another flaw in the human character is that everybody wants to build and nobody wants to do maintenance.” and perhaps this is an inherent problem with data quality – it is hard to make it appealing to executives. Some more creative marketing by the industry could perhaps change this perception.

I wrote recently about likely further consolidation in the MDM market. A further example, albeit on a small scale, happened today as FullTilt, a PIM provider, was bought by QAD. QAD is a public company selling ERP software, with around 1,500 employees which has been listed since 1997 though its history goes back to 1987. FullTilt was known by industry insiders to be “in play” for many months, and has been openly for sale for some time. It is a relatively small company that has struggled to get scale, and so a deeper pocketed parent makes some sense for it.

This is another example of companies in more mature markets seeking to get exposure to the fast growing MDM market. It will not be the last such move.

The MDM market, from a term barely used before 2003, has grown up into a market where literally dozens of vendors compete. As with any fast growing market, the big players have tended to acquire rather than build their own technology. IBM bought Trigo and DWL, Oracle bought Siebel and Hyperion, SAP bought A2i, TIBCO bought Velosel, Teradata licensed the code to I2’s MDM offering, D&B bought Purisma, Microsoft has bought Stratature and the list goes on.

In my view this consolidation as vendors seek to fill out their product lines is far from complete. For any data integration vendor it makers sense to have an MDM offering. TIBCO has already made its move, but this still leaves Informatica (so far just buying a data quality tool in Similarity) and even Ab Initio, though it is hard to figure out anything much about that secretive vendor. This also leaves Sun (with SeeBeyond) and other EAI vendors, and perhaps even a dark horse like EMC from a storage perspective.

While a number of technologies have already been purchased, as noted earlier, there are still several MDM independents out there that could potentially be bought up: Initiate, Siperian, Kalido, Visionware and, in France,, Orchestra Networks and Amalto. Some of these are big mouthfuls, but there are plenty of vendors with deep pockets who may like the look of a fast growing market (over 50% annual growth in the coming years predicted by Forrester).

MDM platform vendors themselves should consider whether to plough their own furrow or be vendor-neutral with regards to data quality. Some have their own algorithms, some partner with one or more data quality vendors, some offer a choice. There are a scary number of data quality vendors out there, many of them quite bite sized (few have more than USD 10M in revenue) and so I would be surprised if nothing happened here n the next twelve months. Moreover MDM platform vendors from a CDI heritage often get a rabbit in the headlights look when asked about data governance and complex work-flow (a screen or two to handle customer address mis-matches does not count as business process workflow support in my book). By contrast vendors from the PIM or analytic MDM world often have quite sophisticated offerings here, but may lack raw high volume performance. Hence there is room for some of the smaller vendors with tasty technology offerings to be snapped by those wishing to fill out their technology stack.

If there was a big move e.g. Informatica buying Initiate, then this may trigger a response from other MDM platform vendors who at the least could then decide to gobble up a data quality vendor like SilverCreek or Exeros in order to expand their own technology stack.

All in all 2008 I expect the next 18 months will an interesting time in the MDM market from a mergers and acquisitions perspective. After all, those silver tongued investment bankers will need something to do with their time now they can’t jam dodgy credit derivatives down gullible corporate treasurer’s throats any more.