I have recently been spending some time looking at the data quality market, and a few things seem to pop up time and again. The first thing is, in talking with customers, just how awful the quality of data really is within corporate systems. One major UK bank found 8,000 customers whose age was over 150 according to their systems. All seemingly academic (if you are taking money out of your account, who cares what your age is?) until some bright spark in marketing decided that selling life insurance to these customers would be a fine idea.

Story after story confirms some really shocking data errors that lurk beneath most operational systems. These are the same operational systems that are used to generate data for the end-year accounts which senior executives happily sign off on pain of jail-time these days. I hope no one shows these sames execs the data inside some of these systems, or they might start to get very nervous indeed.

Yet in a survey we did last year, only about a third of companies in the survey have invested in data quality tools at all! Does anyone else find this in any way scary? Do you have any entertaining data quality stories you can share?

There are a number of approaches of “styles” to tackling MDM within a company, at least in terms of what to do first. If your most pressing issues are improving the quality of business reporting you may opt for “analytical” MDM, or if your issues are mostly with the data in transaction systems you would go for “operational MDM”. Within such categories there are different degrees of invasiveness, from a “registry” style where you leave the master data intact within operational systems, to an extreme root and branch approach “transaction style” where you rip out the ability of operational systems to maintain master data and put in new MDM system(s) to do this and feed the rest of the enterprise, while “co existence” recognises the reality that it may be necessary to live with a mix of approaches. But just how many companies go with which style, and how successful are they?

To answer this The Information Difference is launching a major piece of market research. This survey, sponsored by Microsoft and with media sponsors DM Review and CIO Magazine, is an in-depth look at this topic. To take the survey please click here.

When the survey is complete and the analysis is done I will let you know. Participants will receive a summary of the finding.

We recently completed the Q4 “Market Landscape” for MDM. As part of this we looked at all the vendors in the market and obtained figures for software revenues and growth of each. One interesting aspect of this is that the MDM software market so far appears to be holding up well. Indeed it is currently growing at an annualised rate of 30% according to our research,a healthy clip. It should be noted that the market size figures that we use exclude systems integrator revenues associated with MDM – these are estimated at around three times the size of the software market. As an aside, it is these kind of assumptions that can lead to seemingly wide discrepancies between market size estimates from different firms; typically you see a figure quoted in the press, but what does it include and exclude? Our figures for MDM software exclude data quality vendors, which are handled in a separate twice yearly update.

These figures, which are admittedly retrospective, confirm our November market research study looking at the effect of the financial crisis on MDM spending. This found that about as many companies were planning to accelerate their MDM spend as were planning to slow it down or defer projects (admittedly nearly a third of respondents were undecided).

So far at least, then, both actually Q4 spend as seen by vendors, and spending intentions in our survey are telling the same story. MDM software revenues are holding up well. We will continue to track the market closely, with a Q2 2009 Market Update to be published in July.

I am sure most of us have experienced some dismal experience when calling a technical helpdesk, being put through a maze of automated menus before finally getting through to some bored, half-trained “engineer”. An example of this from my own past is here. However usually the one bit of an organisation that is fairly responsive is the sales function, for obvious reasons. I relate the following personal example of the risks that a high-tech company can take when it decides to “economise” in this area.

I recently needed to buy a new printer and rang up Dell to do this. Dell now pushes its “consumer” (for which read “cattle class”) sales though to an offshore centre. I placed my order and awaited the confirmation email: nothing. The following day I called, went through the menus again and explained what had happened to another Dell sales person. He told me that there was no trace of my order in the system, do I duly went through the whole thing again, wondering quite what happened to the original details, which after all included my credit card details. I specifically said to he salesman “please make sure that the order was not duplicated”, and I even gave him the name of the sales lady I had originally spoken to. “No problem” I was assured.

A week or so a printer duly arrived, and all seemed well. A couple of days later there was another knock on the door, and, surprise surprise, a second identical printer arrived. I declined delivery, explaining the situation. Well, you can probably guess by now that, despite me sending back the duplicate printer, I was charged twice, so I duly rang up and spoke to the original sales lady who took the initial, seemingly lost order. At least I did this after gong through no less than five separate people at Dell, each of them insisting that I tell the story again and each taking full details (name address, order number, …) every time, as if computers had never been invented.

The thing that reduced me to apoplexy at the end of this was that the sales lady (I could not make this up) suggested that in order to get a refund, I would have to take delivery of a fresh printer, uninstall the one I have sitting on my desk, rebox the current one and sent it back to Dell. Oddly enough, I have declined to do this – pesky customers eh?

So, if anyone from Dell is reading this, please give my money back. For any company executives considering outsourcing their sales operation to a cheap location, just consider what effect this may have on your customers. I can’t say I am exactly itching to order another Dell product at the moment. A customer’s perception of a brand, after all, is significantly derived from their personal interactions with the company. The irony is that Dell used to have an award winning call centre based in Ireland, but seemingly decided that this was all a bit expensive. I am not suggesting that the precipitous decline in their share price over the last eight years can be specifically linked back to their decision to save money on their call centres, but I suspect that it will not have helped.

In doing the research for our six-monthly update of the MDM market I came across something which surprised me. Generally master data can be complex (such as bill of materials structures) but is not generally very large in volume, at least compared to transaction volumes. The exception is the “customer” dimension in a B2C company, where it is easy to see that 50 million or so records may be needed. Luckily “customer” data is usually quite simple compared to, say, product data, which may have hundreds of attributes.

However I have come across three cases now where the volume of master data records being managed is claimed to be around 500 million records. One vendor I spoke to said they had a customer planning a billion record MDM system. Dealing with hundreds of millions of records rather than tens of millions is a lot more challenging, especially where the data need to be dealt with in real time e.g. if you are adding a new customer account then you need to check whether that apparently new customer account is really a duplicate of an existing account; this should ideally be done straight away.

If anyone reading this has come across one of these really large MDM implementations then I’d be interested to hear your experiences.

Usually when economic times are tough then there are a series of things that the bean counters do to rein in costs. First they ban business travel except for customer facing situations, followed by freezing training budgets and recruitment, before sharpening their knives more seriously. The first two of these mean that conferences are usually at the sharp end of corporate spending cuts. I have just returned from speaking at a BI/MDM conference in Amsterdam, and was pleasantly surprised to see a healthy attendance of paying customers, a large number of which appeared in the last few days. This must have been a considerable relief to the conference organisers, and it was certainly nice to be speaking a packed room rather than one with rows of empty seats.

It is certainly hard to be sure just how deep recession is likely to be, with seemingly contradictory data all around. A scary figure is the cost of shipping (the Baltic Dry Index) which is a reasonable predictor of the flow of trade. This has dropped a little matter of 95% since the peak in June, with shipping companies cancelling orders and talking of mothballing ships. This kind of broader economic data would seem to suggest a fairly sharp recession is on the cards, and this must feed through into lower IT expenditure eventually, just as night follows day. Of course some areas will be hit harder than others, but I am always suspicious of claims that a certain area is “strategic” and so will be unaffected. Usually this is whistling in the dark by vendors. We shall see.

According to the Celts, today is the day of the year when the boundaries between the living and the dead dissolve. While the ghosts of the departed such as Lehman Brothers stalk the earth, the living put on masks in order to mimic or placate the evil spirits. I’m a little unclear as to what would be the most suitable mask to don to mimic a deceased investment bank (do Armani make masks?), but software vendors across the globe will be nervously hoping that the spirit of Lehman has been thoroughly placated by the kindly sprites of Hank Paulsen, Gordon Brown et al. Fear is stalking the enterprise software market on a scale not seen since the aftermath of the millenium party in 2001.

Ghouls and goblins in the form of financial controllers in large companies are, as we speak, preparing a witches brew of budget cuts sufficient to make the bravest software salesman quail. Companies look at each other nervously and sing around the campfires to keep the spirits up, saying that their particular type of software does such an important job that it won’t be affected, and indeed that in times of adversity, perhaps companies will actually spend more money on critical IT projects? After all, data quality/MDM/(insert sector) is more important than ever now right? Right?

If you believe that you probably also believe in fairies and that derivatives make our financial systems more stable. As sure as night follows day, in times of economic downturn the finance department bring out their trusty red pen and seek out advertising budgets,travel allowances, training and information technology projects to carve up. At present there is a delayed reaction, as IT projects lumber on like zombies, unaware of the carnage waiting ahead. Perhaps our particular sector or project will be unaffected? Perhaps, but few will escape unscathed.

There will shortly be a lot more tricks than treats out there on offer for the enterprise software salesman as he makes his calls this winter.

Happy halloween!

Like many of us, I am curious as to what extent the meltdown in the banks will affect the rest of the economy and, in particular, enterprise software spending. Random conversations over the last few weeks with vendors have been varied, with only those exposed heavily to financial services seemingly seeing a real decline in spend (one company had Lehman Brothers on its Q4 sales forecast). Certainly some sectors may barely be affected e.g. the public sector, or perhaps pharmaceuticals (people still get ill and will need to pay for their pills) and maybe law (imagine all the fun the lawyers will have as banking positions unwind and contracts cannot be fulfilled, never mind the shareholder class action suits). However there are only so many of these and I wonder whether we are in the situation that you get in cartoons, where Wile E Coyote or Bugs Bunny runs off a cliff and happily progresses forward until he actually looks down and notices there is no ground any more.

A troubling sign of this is in a sector a long way from the world of credit default swaps, that of trucks. Volvo is one of the largest suppliers of commercial lorries and trucks to continental Europe, and sold 41,970 trucks in the third quarter of 2007, when admittedly things were booming. They just announced their Q3 2008 results. How many trucks do you reckon that they sold? Less than 41,970 for sure, but what what sort of reduction might you expect? Maybe a 10% drop in sales, perhaps 20%, or even 30% if things had become really bad? Well, they actually sold 115 trucks in the last three months. That is not a typo. It is a 99.7% reduction in sales.

Now that is a scary number.

The Information Difference will shortly be conducting a survey of enterprise software buyers, looking specifically at their spending plans for master data management and data quality. I’ll keep you updated when we have some results (a few week’s time). If your company would like to sponsor this survey, you have four days left to do so (just contact me and I’ll send you details).

Just in case you ever wondered about whether data quality in spreadsheets was really a big deal, then this little story should change your mind. A lawyer on the Barclays buyout of Lehman Brothers defunct assets had drawn up an Excel spreadsheet detailing the particular trading contracts that Barclays was prepared to accept as part of the deal. The lawyer took out 179 particularly undesirable contracts, but rather than deleting the cells, marked them as “hidden” cells in Excel. This was all fine until a first year legal associate who was helping to put together the final paperwork converted the spreadsheet to a PDF file and, you guessed it, managed to include all the hidden cells as well as the items Barclays was supposed to be agreeing to buy. The deal has now gone through, and Barclays are frantically trying to reverse this mistake. I imagine that the Lehman’s administrators will be most sympathetic when the case gets to court.

The original article in “Above the Law” does not detail the financial consequences of this and, let’s face it, Lehman’s well-paid traders seem to have only a sketchy notion of the value and risk of their trading positions or they would not have gone bust in the first place. However I am guessing that the ones that Barclays intended to exclude were not exactly the most attractive, juiciest ones.

The lesson here is that you can have all the fanciest data quality tools and controls applied to your mainstream transaction processing systems, but data quality is something that needs to be applied to Excel as well, yet is widely ignored. When I was at Shell I was in an office opposite a team of bright young things that used to draw up sophisticated Excel models that did all kinds of things, such as deciding on the right levels to bid on contracts, and how much to invest in certain projects in order to make a decent return. Some of these spreadsheets were fiendishly complex. At least this group had various standards for Excel model development, and a well established spreadsheet audit process (with code reviews by other team members), but Excel formulae can be pretty opaque, and I am betting that the odd error slipped through. We are used to applying fairly elaborate testing and code reviews to C++ code, but how many tools are there for Excel testing, and how widely are they applied?

Hat tip to Stephen for pointing out the article.

I am currently at the SAP Teched conference in Berlin. I will write in a separate publication about the forthcoming version 7.1 of SAP MDM, but have a couple of quite separate observations to mention here. The first is a confirmation of what i have long believed:that going towards an SOA world is going to be very hard work. One customer here, Volkswagen Financial Services, described an ambitious project where they have taken a part of their business, which deals with fleet car hire, and moved wholesale to an SOA-based infrastructure. This project has been live a few months and is already showing some genuine benefits compared to the rather manually intensive system they had before, in terms of faster processing time for certain common business processes (which used to involve agents dealing with multiple applications) and in terms of improved data quality. However it is interesting that no formal cost/benefit analysis appears to have been done. Moreover this project, which involved 100 IT staff and 50 business people, took over five years to complete. I do not think this is much to do with the technology, but rather the sheer complexity of taking a cross functional view, involving different business lines agreeing on common terminology and data definitions, agreeing on the way in which the many new web services behave. There has also been a lot of change management needed to effectively get the front-line business users to accept the new system, which automates many tasks that they used to have direct control of.

I suspect that few companies have been quite so aggressive in their move to SOA as VW. A more typical conversation was with a gentleman at a German utility and resources company, who have been looking actively into SOA since 2006. They are only just putting their toe in the water now, putting in a very limited project with just a handful of web services, across a single process, in just one small subsidiary of their organisation. Even this limited pilot has not been entirely without its issues. One problem which has reared its head is how much more difficult it is to do debugging across a web services application which touches a whole series of different applications in its wake. If something goes wrong, then they have found it is a lot more fiddly to trace where exactly the fault lies, given the cross-application nature of the project. Again, this is a project driven by the IT department as an exercise in proving technology, rather than one with a quantified business case. I do not pretend that a few random conversations at a conference is a remotely scientific sample, but it seems clear that SOA is far from mainstream in many companies thus far, and that there are new issues to address compared to traditional applications. Not least of these is the need to sort out common master data definitions across the multiple applications affected.

On a separate note, those who read my blog regularly will know that a bugbear of mine is conferences that do not run on time or are disorganised – yes ETRE, that means you. By contrast, this conference is a testament to stereotypical Teutonic efficiency. Sessions start on time to the minute, and finish on time, to the minute. There are plenty of staff around to guide people around the large congress centre, and the pre-conference administration was exemplary. When I arrived I was handed not just a conference schedule, but a suggested set of lectures and meetings that were likely to be of interest specifically to me based on my MDM interests. If only all conferences could be run by Germans.