In a surprise move HP has snapped up Knightsbridge in a move to bolster its technology services business.  Knightsbridge had carved out a strong reputation for handling large data warehouse and BI projects for US corporations, and had grown to over USD 100M in revenue.  It was up with IBM as one of the two leading data warehouse consulting organisations.  This in itself makes it clear why it was attractive to HP, who do not have anything like such a strong reputation in this area.  Knightsbridge was growing strongly in 2006, and the financial terms of the deal are not public, but one would assume HP paid a good price for such a good business.  This will no doubt provide a happy retirement for the Knightsbridge founders, but it is less clear as to how well the Knightsbridge culture, which was quite fiercely vendor-independent, will sit within a behemoth like HP, which has its own technology offerings.  It was revealing that Knightsbridge CEO Rod Walker had dismissed service company acquisitions in an interview just a year ago, and for reasons which sounded pretty sensible.   No doubt this will present an interesting spin challenge for the Knightsbridge PR staff, but perhaps they will have other things on their minds, such as dusting off resumes.

“If the cultures of the two companies are not a near-perfect match, people will leave, and services is a people business.”  I couldn’t have put it better myself Rod.

I have wondered for some time why business intelligence has been so slow to come up with software as a service solutions.  Celequest has done so, and this week sees the launch of another, called LucidEra.  This company aims to offer a ciomplete BI suite including ETL, data quality, database schema, OLAP server and reporting.  Given that enterprises are prepared to trust their customer data to third parties e.g. salesforce.com, there is no reason I can see why they would not do the same with business intelligence. 

The advantages of a service offering is seem to me twofold,  First is the easier and more reliable deployment.  Many problems in software stem from environmental incompatibilities e.g. some weird combination of releases of Oracle and Tomcat and something else that cause obscure bugs which the vendor could never have tested for, and which are hard to reproduce.  This problem goes away with hosted solutions, where the web browser is just about the only software the client can screw around with.  Secondly, though this is a commercial rather than technical issue, the leasing that software as a service typically uses means an easier point of entry.  One mid-ranking customer can sign off on a few months of leasing in a way that they could not for a multi-hundred thousand dollar software purchase, which would end up in steering committees and a formal procurement process.  

Salesforce has shown what can be done with this approach if well executed.  It will be interesting to track the progress of LucidEra, Celequest and others that emerge into this space.

An article on “the future of Business intelligence” is always a bold undertaking, but I think the one just out by Brian Watson could be a lot bolder.  I don’t think that just making BI “more real time” or plugging a load of reports into Google is really going to change the world of BI, and indeed to some extent it is disappointing just how unimaginative the software community has been in recent years with respect to BI.  Although it is a large and growing market, there are many pretty fundamental issues that have barely improved in a decade.

Starting with data quality, everyone agrees that data quality it pretty horrible in most companies, yet what has really come on the market to address it?  No vendor is making more than USD 50 million in revenue (Trillium is about the largest) and yet every big company has a large, expensive, data quality problem.  I like the more automated discovery approach taking by US start-up Exeros, and indeed something similar can be seen (but is not articulated in its marketing) by Uk software vendor Datanomic, and yet these companies are still pretty small. Surely there is room for compelling innovation here?

Getting data out of source systems has become somewhat commoditised.  Products like Ab Initio have increased throughput, but in general the technology is slipping into the database (as with IBM buying Ascential and Oracle buying Sunopsis).

When it comes to the data warehouse itself, this is a cottage industry, with few true packages.  Most data warehouses are built by hand, which suits systems integrators just fine (all those yummy billable hours) but does not serve customers well.  TDWI reckon an average data warehouse takes 16 months to deploy, USD 3 million to build and costs 72% of its development costs in support every year.  This is a dismal state of affairs, yet other than one new design approach (Kalido) and adding ODS functionality to ERP (SAP BW) there has been little to move things forward here. 

On the database side of things there has been more activity, with Teradata carving out a proftable niche at the top end, and now Netezza biting at its ankles.  There are one or two software solutions in the works also e.g. Kognitio.  So here at least is some sign of life.

The reporting suites have mostly consolidated around a few vendors: Business Objects, Cognos and Hyperion, with a few smaller players like Microstrategy and Actuate. There are only so many ways you can display a report, so it is not surprising that this area is showing consolidation rather than a lot of innovation. 

Master data management is at least coming out of the closet as an issue, but here we see a flood of companies rebadging some tired old products as “MDM”, yet relatively few companies with genuinely new approaches.  At least here there seems to be some genuine customer interest, if not heavy spending so far.

Data visualisation tools still lurk in the shadows, with no vendors really breaking out of niches, though Spotfire is doing a good job, especially in pharmaceuticals.  Yet companies like Fractal Edge and others which have genuinely interesting user interfaces are still very small. 

I think there is an opportunity for a more hosted approach to BI, as is being taken by Celequest.  If people are prepared to trust their customer information to be stored outside their enterprise (salesforce.com) then why not their BI data?  I am surprised that more has not happened so far here.

All in all, I think that the BI industry has a lot of potential for improvement in innovation, yet is showing few signs of bold thinking right now. Customers should not have to live with the relatively poor status quo. Although venture capital is now scarce for enterprise software plays, a large multi-billion dollar market with 10% annual growth and a generally pretty low standard of solutions is a market crying out for innovation, rather than incremental improvement. 

 

In IDC’s latest annual report, it gives the size of the “data analysis” market, which includes data warehousing, business intelligence and generally anything analytic, as being worth a chunky USD 16.5 billion in software (systems integration related to this would be greater than this), up 11% from last year.  They also reckon that this market will grow at a healthy clip of 10% a year for the next five years, based on the fact that “analysis” is now one of the top two spending items for IT executives.

Recently I pointed out that the specialist players in what is more commonly called the business intelligence market grew revenues at 23% in calendar 2005 over 2004, though the IDC figures include the BI offerings of the industry giants Oracle, SAP, IBM and Microsoft, as well as a broad set of other companies and categories e.g. data mining offerings. 

A large and growing market not only causes the behemoths to want to gobble up smaller players with good technology, as Oracle recently did with Sunopsis, but in principle should interest venture capital firms to back innovative start-ups in the area.  However VCs seem too starry-eyed at the moment over social networking web sites to want to return to anything as tedious as enterprise software, with all its long sales cycles, costly software development and grumpy and conservative enterprise buyers.  Still, fashions change, and who in 2002 (when venture firms turned firmly against the internet in the wake of the crash) would have been betting that a web site company set up in 2005 for consumers, with no obvious mechanism for making money, would be snapped up just over a year later for USD 1.65 billion?  Perhaps it is time to get ahead of the curve and look ahead to when enterprise software will be fashionable again.  Any VCs feeling brave?

 

Kalido has now announced revised positioning targeted at selling solutions to business problems (and will soon announce a new major product release). The key elements are as follows. The existing enterprise data warehouse and master data management product offerings remain, but have been packaged with some new elements into solutions which are effectively different pricing/functionality mechanisms on the same core code base.

The main positioning change is the introduction of pre-built business models on top of the core technology to provide “solutions” in the areas of profitability management, specifically “customer profitability” and “product profitability”. This move is, in many ways, long overdue, as Kalido was frequently deployed in such applications but previously made no attempt to provide a pre-configured data model. Given that Kalido is very strong at version management, it is about the one data warehouse technology that can plausibly offer this without falling into the “analytic app” trap whereby a pre-built data model, once tailored, quickly becomes out of synch with new releases (as Informatica can testify after their ignominious withdrawal from this market a few years ago). In Kalido’s case its version management allows for endless tinkering with the data model while still being able to recreate previous model versions.

Kalido also announced two new packaging offerings targeted at performance management/business intelligence, one for data mart consolidation and one for a repository for corporate performance management (the latter will be particularly aimed at Cognos customers, with whom Kalido recently announced a partnership). Interestingly, these two offerings are available on a subscription basis as an alternative to traditional licensing. This is a good idea, since the industry in general is moving towards such pricing models, as evidenced by salesforce.com in particular. In these days of carefully scrutinised procurement of large software purchases, having something the customers can try and out rent rather than buy should ease sales cycles.

The recent positioning change doesn’t, however, ignore the IT audience – with solution sets geared toward “Enterprise Data Management” and “Master Data Management.” The enterprise data management category contains solutions that those familiar with Kalido will recognize as typical use cases – departmental solutions, enterprise data warehouse and networked data warehouse. The key product advance here is in scalability. Kalido was always able to handle large volumes of transaction data (one single customer instance had over a billion transactions) but there was an Achilles heel if there was a single very large master data dimension of many million of records. In B2B situations this doesn’t happen (how many products do you sell, or how many stores do you have – tens or hundreds of thousands only) but in B2C situations e.g. retail banking and Telco, it could be a problem given that you could well have 50 million customers. Kalido was comfortable up to about 10 million master data items or so in a single dimension, but struggled much beyond that, leaving a federated (now “networked”) approach as the only way forward. However in the new release some major re-engineering underneath the covers allows very large master data dimension in the 100 million range. This effectively removes the only real limitation on Kalido scalability; now you can just throw hardware at very large single instances, while Kalido’s unique ability to support a network of linked data warehouses continues to provide an effective way of deploying global data warehouses.

Technologically, Kalido’s master data management (MDM) product/solution is effectively unaffected by these announcements since it is a different code base, and a major release of this is due in January.

This new positioning targets Kalido more clearly as a business application, rather than a piece of infrastructure. This greater clarity is a result of its new CEO (Bill Hewitt), who has a strong marketing background, and should improve the market understanding of what Kalido is all about. Kalido always had differentiated technology and strong customer references (a 97% customer renewal rate testifies to that) but suffered from market positioning that switched too often and was fuzzy about the customer value proposition. This is an encouraging step in the right direction.

I write recently about the emergence of software as a service as one of the few bright spots in enterprise software at present.  With perfect timing, today a vendor came along and announced a software as a service offering in the business intelligence field.  Celequest is a start-up and it is certainly early days to see how well this idea takes off, but this is certainly an interesting development.  Celequest has the credibility of being run by Dias Nesamoney, who was founder iof Infomatica, and is backed by VCs Bay Partners and Lightspeed Ventures, who both have long track records.  The company was set up in 2002, and has some good customers like Citigroup, Cendant and Brocade, though it is not clear from the company’ website what scale these deployments are.  The application covers dashboards, analytics and data integration technology.  As far as I am aware the company uses an in-memory database “appliance” though from what I can gather the volume of data dealt with by this application so far is modest.  However this is not the point and no doubt will imcrease over time as the concept gains acceptance.  Celequest has made an astute partnership with salesforce.com, with a bridge to AppExchange.  There is also a connector to SAP. 

Certainly, there are barriers to the widesprea acceptance of this approach.  Large enterprises will be naturally conservative about the idea of letting their data out of the corporate firewall, particularly when it is key performance data of he type that BI applications use.  It is also unclear what sort of scale issues come into play when data is being accessed from beyond the coirporate network.  However for many companies, and especially SMEs, such issues will seem less important than the convenience of being able to deploy a business intelligence solution without the usual hassle of complex software installation and an army of systems integrators.  No doubt where Celequest has begun to tread, others will follow, and it will be a healthy new area of competition in the business intelligence industry.

 

 

 

 

It has just been announced that Oracle has bought Sunopsis, one of the few remaining independent ETL vendors.  Since Oracle’s existing ETL tool (the rather inaccurately named “Data Warehouse Builder”) is pretty weak, this makes a lot of sense for Oracle.  I suspect that their statement about “integrating” the two tools will involve much use of the delete key for the Warehouse Builder code. Sunopsis is a good product, a French company that had been around for some time but had recently made more visible market progress in the US.  No numbers are public, but my information is that Sunopsis revenues were about USD 10M and the purchase price was just over USD 50M, which at a price/sales ratio of over five makes a quite healthy price for the company.  Sunopsis was 80% owned by the founder, who had spurned venture capital, so this is very good personal news for him also. 

Sunopsis made a virtue of using the DBMS functions where possible rather than re-inventing transformation code, so is particularly compatible with Oracle (or other relational databases). This deal should also put paid to the loose marketing relationship Oracle had with Informatica. 

In my view this is a rare case where the deal is good for both companies.  Oracle finally gets a decent ETL capability and Sunopsis gets Oracle’s massive sales channel. 

My prize for the most creative marketing jargon of the week goes to IBM, who announced that they now consider their offerings to be a “third generation” of business intelligence.  Come again?  In this view of the world, first generation BI was mainframe batch reporting, while the second generation was data warehousing and associated BI tools like Cognos, Business Objects etc.  So, as you wait with bated breath for the other shoe to drop, what is the “new generation”?  Well, it would seem that this should include three things:

(a) pre-packaged applications

(b) focus on the access and delivery of business information to end users, and support both information providers and information consumers

(c) support access to all sorts of information, not just that in a data warehouse.

Well (a) this is certainly a handy definition, since IBM just happens to provide a series of pre-built data models (e.g. their banking data model) and so (surprise) would satisfy the first of these criteria.  It is in fact by no means clear how useful such packages are outside of a few specific sectors that lend themselves to standardisation.  Once you take a pre-existing data model and modify it even a little (as you will need to) then you immediately create a major issue for how you support the next vendor upgrade.  This indeed is a major challenge that customers of the IBM banking model face.  Nothing in this paper talks about any new way of delivering these models e.g. any new semantic integration and versioning capability.

Criteria (b) is essentially meaningless since any self respecting BI tool could reasonably claim to focus on information consumers.  After all, the “universe” of Business Objects was a great example of putting user-defined terminology in front of the customer rather than just presenting tables and columns.  Almost any existing data warehouse with a decent reporting tool could claim to satisfy this criteria.

On (c) there is perhaps a kernel of relevance here, since there is no denying that some information needs are not always kept in a typical data warehouse e.g. unstructured data.  Yet IBM itself does not appear to have any new technology here, but merely is claiming that DB2 Data Joiner allows links to non-DB2 sources. All well and good, but this is not new. They haven’t even done something like OEM an unstructured query product like Autonomy, which would make sense.

Indeed all that this “3rd generation” appears to be is a flashy marketing label for IBM’s catalog of existing BI-related products.  They have Visual Warehouse, which is a glorified data dictionary (now rather oddly split into two separate physical stores) and scheduling tool, just as they always have.  They talk about ETI Extract as an ETL tool partner, which is rather odd given their acquisition of Ascential, which was after all one of the two pre-eminent ETL tools, and given ETI’s near-disappearance in the market over recent years.  They have DB2, which is a good database with support for datatypes other than numbers (just like other databases).  They also have some other assorted tools like Vality for data quality.

All well and good, but this is no more and no less than they had before. Moreover it could well be argued that this list of tools actually misses several important points that could be regarded as important from a “next generation” data warehouse architecture.  The paper is oddly silent on the connection between this and master data management, which is peculiar given IBM’s buying spree in this area and its direct relevance to data warehousing and data quality.  There is nothing about time-variance capabilities and versioning, which are increasingly important.  What about the ability to handle a federation of data warehouses and synchronise these?  What about truly business model-based data warehouse generation and maintenance?  How about the ability to be embedded into transactional systems via SOA?  What about “self discovery” data quality capabilities, which are starting to appear in some start ups.

Indeed IBM’s marketing group would do well to examine Bill Inmon’s DW 2.0 material, which while not perfect at least has a decent go at setting out some of the capabilities which one might expect from a next generation business intelligence system.

There is no denying that IBM has a lot of technology related to business intelligence and data warehousing (indeed, its buying spree has meant that it has a very broad range indeed).  Yet there is not a single thing in this whitepaper that constitutes a true step forward in technology or design.  It is simply a self-serving definition of a “3rd generation” that has nothing to do with limitations in current technology or new features that might actually be useful.  Instead it just sets out a definition which conveniently fits the menagerie of tools that IBM has developed and acquired in this area. To put together a whitepaper that articulates how a series of acquired technologies fits together is valid, and in truth this is what this paper is.  To claim that it represents some sort of generational breakthrough in an industry is just hubris, and destroys credibility in the eyes of any objective observer.  This is by no means unique in the software industry, but is precisely why software marketing has a bad name amongst customers, who are constantly promised the moon but delivered something a lot more down to earth.

I suppose when presented with the choice of developing new capabilities and product features that people might find useful, or just relabelling what you have lying around already as “next generation”, the latter is a great deal easier.  It is not, however, of any use to anyone outside a software sales and marketing team.

 

 

Those of you with long memories will recall that the first three real ETL vendors were Prism, Carleton and ETI.  The others were acquired but ETI survived as an independent company, though with an ever-diminishing profile.  Early this year they were apparently down to just one salesman, but having the US Department of Defense as a customer does wonders for maintenance revenue.  In recent years the company had been pared down to a minimum, and I had assumed that, like an old soldier, they might just fade away.  However in the summer the original investors were bought out and a new capital injection happened in a USD 6.5 million round from investors Appian Ventures of Denver, Access Venture Partners and Osprey Ventures, and a $5M line of credit was negotiated with Comerica bank.  Consequently the company is effectively born again, with new money, owners and management, but with established technology.

ETI’s software used to be strong at dealing with extracting data from esoteric sources, generating code against things like COBOL workbooks and assorted mainframe file systems, as well as having the usual transformation capabilities.  It suffered from being rather complex to use and from some weak marketing. 

So, the interesting question is whether this old warhorse can be dusted off, repainted and revitalised.  Judging by the seven vice presidents that have appeared in the management ranks, the new board is not afraid to spend some of that new money.  They have also licensed in some data quality offerings from a couple of small British companies, which is a logical step to broaden the product range from just ETL.  This is important because ETL on its own is a tough market, as ETI has discovered.  More and more ETL functionality is being thrown into the database (MSFT with SSIS – previously DTS – and Oracle with the poorly named Warehouse Builder tool, which is really an ETL tool) which makes it hard work to persuade a customer to buy your technology.  Only Sunopsis has really made much headway here in recent times, with a clever pitch built around using rather than competing with the database capabilities.  Other pure plays like Sagent have withered and died.  Informatica is really the only ETL player of size left standing, and they have broadened their appeal by going for a wider integration message.  So what has changed that will allow ETI to flourish now when it clearly has not done fo some time?  Perhaps new the investors have noticed a flurry of companies being bought out in the data quality space recently and so can see a fairly quick exit, perhaps there is just too much venture capital around, or maybe they have more ambitious plans for the company.

ETI certainly has some well proven technology, and its foray into data quality looks logical.  Good luck to them.  Yet relaunching a company is hard work, and it will take some impressive sales and marketing execution to turn breath new life into this particular body.

Sreedhar Srikant writes about the importance of the logical data model in a data warehouse project in DM Review. This well-written article describes the process of building a model, highlights five pitfalls and suggests some ways of avoiding them.  It is in this last area that I feel the article could be enhanced.  In my experience there are two serious dangers that a data warehouse project faces that go beyond issues of project problems like trouble agreeing on a model.  These are:

(a)   The project gets insufficient business buy-in through lack of a well-articulated and robust business case

(b)   The project takes too long to deliver, making it vulnerable to budget cutbacks since it has not shown tangible benefit early enough.

I am constantly surprised how often IT projects in major corporations seem to get off the ground without a strong business case.  IT projects compete for capital in a company with many other project proposals, and so it can be a Darwinian process when times get tough: projects with the strongest business case and sponsorship will survive.  As a minimum, the project needs to set out the expected returns that it will make, set against the project costs, over a three year (sometimes five year) period.  A simple example is shown below:

Costs:  $3M one-off, $2.16M annual

Benefits: $5M from year 2 onwards, with $2M only in year 1.

In this instance the project costs $3M to deliver and just over $2M to support each year (a Data Warehouse Institute survey showed that the average data warehouse costs 72% of its build costs to support every year).  Against this are some benefits as shown.  In this instance the project is tolerably attractive, since it has a positive net present value (USD $536k using a typical 18% discount rate) and a decent 27% IRR, though its payback period is a little slow.  However, while not stellar, it is respectable as cases go, and it is at least written in the language of business. 

What might the project benefits be?   These will vary from project to project and from industry to industry, but examples might include either profit-enhancing benefits, such as reduced customer churn or improved pricing ability, or cost reductions such as fewer misplaced deliveries due to improved data quality, or better procurement margins to due to improved understanding of supplier spend.  In order to articulate these you need to find a business sponsor, preferably one who has a problem related to poor information.  Trust me; you should not have to look too far in a big company for one of these. 

Having a business case that is properly set out will act as a safety net when project reviews happen, and reduce the chances of a project being cancelled when the knives come out. 

The second thing that can help your project is to deliver something tangible early.  Traditional waterfall methodologies, often used by large systems integrators, are not always well suited to data warehouse projects, where requirements are often rather loose.  The average data warehouse project takes 16 months to deliver according to TDWI, and that is a long time in this turbulent world when management has its budgets adjusted and people are looking for projects to cut.  If your project can deliver something meaningful quickly i.e. a piece of the overall problem, then your project sponsor has a lot better chance of defending the project.  If all the review committee can see is costs then things will be harder.  Many real projects I have been involved with have been killed in this way.

One way to improve your odds of delivering something quickly is to use a data warehouse package, where at least some of the functionality is already pre-built for you.  Packages may or may not be cheaper than custom-build, but they should be quicker.  If you can pick off a chunk of the project and deliver reports back to the sponsor that add value early on then the project is much more likely to survive than one that is still delivering a grand enterprise logical data model. These days there are several packaged or semi-packaged alternatives to custom build.  A good overview of the packaged data warehouse market was done this year by Bloor and can be downloaded for free here.  

By developing a robust business case and by delivering benefits iteratively your project greatly improves its chances of survival.  When the budget sharks come circling, it is nice to have a life raft.