On software and relationships

An article in Intelligent Enterprise asks “why can’t vendors and customers just get along?” after explaining the many issues at which they are usually at loggerheads. Having been both a customer and a software vendor, I think Joshua Greenbaum points to one key point in his article: honesty. As a customer I found that software vendors frequently made patently absurd claims about their software “this tool will speed up application development by 1000%” being one memorable example. Release dates for software were another issue: vendors fail to grasp that businesses have to plan for change all the time, so a release date slipping by three months is rarely an issue provided that you are told about it in good time. However if you have lined up a load of resources to do an upgrade, it just not turning up on the appointed day does cause real cost and annoyance.

Another bug-bear is testing, a tricky subject since it is impossible to fully test all but the most trivial software (see the excellent book “the art of software testing“) . However vendors vary dramatically on the degree of effort that they put in. At Kalido we have extensive automated test routines which run on every build of the software, which at least means that quite a lot of bugs get picked up automatically, though bugs of course still get through. Yet according to someone who used to work at Oracle, there it was policy to do minimal testing of software, where the testing strategy was described as “compile and ship”. This certainly avoids the need for lots of expensive testers, but is hardly what customers expect.

However, customers can be unreasonable too. Their legal departments insist on using sometimes surreal contract templates that were often designed for buying bits of building equipment rather than software, resulting in needless delays in contract negotiation (but in-house corporate lawyers don’t care about this, indeed it helps keep them busy). They can also make absurd demands: we recently lost a contract after refusing to certify that we would fix ANY bug in the software within eight hours, something which patently cannot be guaranteed by anyone, however responsive. A small vendor who won the bid signed up to this demand, and so will presumably be in breach of contract pretty much every day. Quite what the customer thinks they have gained from this is unclear. It is not clear why some customers behave in such ways, perhaps they feel like exacting revenge for previous bad experiences with vendors, or maybe some corporate cultures value aggressive negotiating.

From my experience on both sides of the fence, the best relationships occur when both parties understand that buying enterprise software is a long-term thing, in which it is important that both sides feel they are getting value. Vendors are more inclined to go the extra mile to fix a customer problem if that customer has been doing lots of reference calls for them, and actively participates in beta test programs, for example. As with many things in life, there needs to be a spirit of mutual respect and co-operation between customers and vendors if both are to get the best out of their relationship.

Real Time BI – get real

I permitted myself a wry smile when I first heard the hype about “real time” business intelligence, which is hyped again this week . The vision sounds appealing enough: as soon as someone in Brazil types in a new sales order, the ultra-swish business intelligence system in central office knows and reacts immediately. Those who have worked in large corporations will be entertained by the naivety of this, since most large companies would be grateful just to know who their most profitable global accounts are.

The mismatch between fantasy and reality is driven by two factors. The first is that business rules and structures (general ledgers, product classification, asset hierarchies, etc) are not in fact uniform, but are spread out among many disparate transaction system implementations – one survey found that the average Global 2000 company has 38 different sources of product master data alone. Yes, this is after all that money spent in ERP. Large companies typically have dozens or even hundreds of separate ERP implementations, each with a subtly different set of business structures from the next (plus the few hundred other systems they still have around). The second problem is that the landscape of business structures is itself in constant flux, as groups reorganize, subsidiaries are sold or new companies acquired.

Today’s business intelligence and data warehouse products try to sweep this reality under the carpet, producing tools to convert the source data into a lowest common denominator consistent set that can be loaded into a central data warehouse. This simplification is understandable, but means that local variations are lost, and many types of analyses are not possible. Worse, if the business structures change in the source systems, then the data warehouses and reports built on top of them are undermined, with any changes to the structure of the data warehouse taking typically months to bring about. In these intervening months, what happens to the “real time” business intelligence?

The problem comes down to fundamental truth: databases do not like having their structure changed. Adding data is fine, but something which affects the structure of a database (a major reorganization will usually do the trick) will cause pain. If you doubt this, ask a CFO how long it will take him or her to integrate an acquisition just enough to be able to run the management accounts as one combined entity. For some companies acquisitions are a way of life, with several undertaken a year. Such companies are always chasing their tail in terms of trying to get a complete picture of their business performance. This is not just inconvenient but also costly: one company built a large and well-used conventional data warehouse, costing USD 4 million to build. When they properly accounted for all aspects of maintenance, including business user time (which few companies do) they found it was costing USD 3.7 million per year to maintain. There was nothing wrong with the warehouse design; they were operating in a volatile business environment, with 80% of the maintenance cost caused by dealing with business change.

What is needed, and generally what the industry has failed to deliver, are technology solutions that are comfortable dealing with business change: “smarter” software. Today few IT systems can cope with a change in the structure of the data coming into the system without significant rework. The reason for this is in the heart of the way that databases are designed. They are usually implemented to reflect how the business is structured today, with relatively little regard to how to deal with future, possibly, unpredictable, change. Introductory courses on data modeling show “department” and “employee” with a “one-many” relationship between them i.e. a department can have many employees, but a person can be only in one department (and must be in one department). This is easy to understand and typical of the way data models are built up, yet even this most basic model is flawed. I have myself been in between departments for a time, and at another time was briefly part of two departments simultaneously. Hence the simple model works most of the time, but not all of the time: it is not resilient to exceptional cases, and IT systems built on this model will break and need maintenance to cope when such special cases arise. This is a trivial example, but it underlies the way in which systems, both custom built and packaged, are generally built today. Of course it is hard (and expensive) to cater for future and hence somewhat unknown change, but without greater “software IQ” we will be forever patching our systems and discovering that each package upgrade is a surprisingly costly process. If you are the CFO of a large company, and you know that it takes years to integrate the IT systems of an acquired company, and yet you are making several acquisitions each year, then getting a complete view of the business performance of your corporation requires teams of analysts with Excel spreadsheets, the modern equivalent of slaughtering a goat and gazing at its entrails for hidden meaning.

Some techniques in software are emerging that tackle the problem in a more future-oriented way, but these are the exception today. Unfortunately the vendor community finds it easier to sell appealing dreams than to build software to actually deliver them. “Real-time business intelligence” comes from the same stable as those who brought you the paperless office and executive information systems (remember those?) where the chief executive just touches a screen and the company instantly reacts. Back in reality, where it takes months to reflect a reorganization in the IT systems, and many months more just to upgrade a core ERP system to a new version, “real time” business intelligence remains a pipe dream. As long as people design data models and databases the traditional way, you can forget about true “real-time” business intelligence across an enterprise: the real world gets in the way. It is interesting that the only actual customer quoted in the techworld article, Dr Steve Lerner of Merial, had concluded that weekly data was plenty: “The consensus among the business users was that there was no way they were prepared to make business decisions based on sales other than on a weekly basis”.

On SAP and Zombies

On SAP and Zombies

I worked for many years in Exxon and Shell and noticed something curious about large (are there any other kind?) SAP implementations: something odd happens to the people on them. Previously reasonable people would start to view the world entirely through the eyes of SAP, as though by using the software they had joined some secret society or cult. Despite any evidence to the contrary, it was as if these people had their critical faculties removed when discussing SAP applications – which could do no wrong even when there were clear problems or issues. If, for example, you mentioned some issue with the software or project, a glazed look would come into their eyes as of they were extras in “Invasion of the Body Snatchers” http://www.imdb.com/title/tt0049366/combined

I noticed this for the first time after being involved in an SAP roll-out at Exxon in the late 1980s, one of the first large-scale SAP projects outside of Germany. The project was justified because Esso UK (where I worked) had very old transaction systems that needed replacing with something, and a previous attempt to implement software from Walker had been a fiasco. The business case rested on getting rid of all the accounts clerks who raised invoices and processed orders, the idea being that the rest of us employees would do this instead using SAP. So if you wanted anything from stationery to a new part for petrol station, you would use the system instead of involving people from finance. Unfortunately SAP at the time was still partly in German, and had a quite complex user interface that involved remembering various esoteric codes to do anything like post an invoice, so although we were all sent on training, things did not go well. After a period of denial, it emerged that Esso’s suppliers were not being paid to such an extent that there were issues about the company’s credit rating, and so all the old finance clerks (and more) were re-hired to sort out the mess. To save face, they were distributed around the business lines in order to not make it look like there were now more admin and finance staff than there were before the system. This was the first instance of denial around SAP that I observed – the project was “too big to fail”.

A little later I moved to Shell UK and was invited to a presentation by a gentleman from Shell Centre who will remain nameless; let’s call him Roger. We were to hear about Shell’s new IT strategy. Flanked by consultants from PWC, Roger proceeded to explain that Shell was going to implement SAP. The bulk of the presentation was done by PWC, and was light on details e.g. the only business case seemed to be “er, other big companies are doing it”. When I mentioned that the Esso UK implementation had not delivered its promised benefits there was another zombie-like experience, with Roger saying that he had been on exotic trips to all sorts of companies in warm locations in order to research the area, and there would be no problem. I asked “Have you ever seen an SAP screen” to which the reply “I don’t need to” was not the comforting response I had hoped for.. If you were about to spend a large but unspecified amount of money on a system and you were in charge of the recommendation, would it not have been prudent to have cast a quick glance at what was being bought? Apparently not. Fortunately the consultants from PWC, who at the time got 11% of their worldwide revenue from SAP implementations and so were utterly objective advisors, saw no problem at all either.

The space-out stares continued when we were selecting a finance, time-writing and billing system for Shell Services International, the internal IT arm of Shell at the time (1998). Despite that fact that SAP was designed for manufacturing companies and not services companies, and despite the fact that none of the consultancy firms implementing SAP used it for their internal tracking, SSI management selected SAP for this purpose. What was required was very simple and could have been done in a dozen commercial packages, or indeed probably an Excel spreadsheet on steroids, but SAP it was. The cost of this, for an organisation with under USD 1 billion in total revenue (and loss making at that) was estimated at USD 50 million i.e. 5% of total revenues, which should have set off some alarm bells (most big companies spend less than 2% of their revenue on IT, never mind one project). Not a bit of it, and the project duly clanked into life. What did it cost? Not a popular question, and eventually someone admitted that it had cost USD 70 million “up to the time they stopped counting”. Did anyone get into trouble over this? Far from it. Were the customers happy with their new billing systems? I can assure you they were not.

So what is it that causes such odd behaviour? I can only speculate that once a project reaches a certain size then it indeed cannot be seen to fail – too many senior people had their name on the decision to implement, and so problems dare not be acknowledged. The people on the project, comforted by the sheer scale of the thing going on around them and concentrating on delivery, have trouble seeing the wood for the trees. The scary thing was the inability to even raise issues for fear of being labelled “not a team player”. I’m not sure if this is something that occurs on any very large IT project, as I doubt there is anything peculiar about SAP that induces such lack of objectivity. However it is an odd experience, seeing those around you who you formerly trusted seemingly oblivious to all issues and suddenly incapable of critical reasoning. I felt like Kevin McCarthy’s character Miles Bennel in the “Invasion of the Body Snatchers” movie at the end, running around shouting: “you’re next, you’re next!”