Case Studies

There is a significant demand for technology case studies, from would-be adopters and practitioners of specific technologies. There is also a considerable supply of technology case studies, mostly from vendors.

But I don't see the supply meeting the demand. There seems to be a gap between what people want to know and what people are willing to publish.

Most so-called case studies take this form:

Gringotts Bank (NYSE:GOBL) needed a fast yet secure customer response across multiple legacy vaults so they installed WebHogz Enterprise Edition Version 6.66. Gringott's Chief Architect Bill Weasley said Our productivity has whizzed up by 31.4%. WebHogz CTO Harry Potter said This application clearly demonstrates the magical superiority of our product over muggle alternatives.

I regard these as press releases rather than genuine case studies. What purpose can they possibly serve apart from name-dropping? We are told that a respectable large organization is happy with the product, but we are given few if any details on how the product was used. Impressive numbers may be quoted, but we have no idea how they are measured or what to compare them with.

One thing we really want to know is about practical lessons - difficulties and pitfalls. An alternative perspective perhaps?

However, Dr Aco Malfoy, a security consultant with Arthur T. Riddle, voiced some concerns about the WebHogz solution. A number of our clients have had problems with the stability and performance of the product. Because of these concerns we have advised Gringotts Bank to install a separate firewall system and invest in additional dragonware, which will add considerably to the overall cost of the WebHogz project.

That's not much better. It may identify some areas of concern, but still doesn't give us a rounded evaluation of a project, successful or otherwise.

And academic studies often aren't much use either.

Professor Hermione Granger, a fellow of Halloween College at Hogsfjord University, has completed a three-year comparative study of vaulting technologies in collaboration with a number of industrial partners including WebHogz Labs. Our preliminary findings, based on an action research paradigm, do seem really quite promising, but we obviously need much more research funding before we can produce more reliable and detailed figures.

What we need is a lot more detailed, warts-and-all case studies, in plain English. But there are never enough organizations willing to expose themselves to independent public scrutiny. Any organizations willing to volunteer?



Update: this blogpost originally referred to SOA case studies. I have removed the specific references to SOA, as I believe my argument applies to any technology or product.

Flight from Quality

TIBCO (TIBX:NSQ) shares soared in December, following impressive financial results. This week they have fallen to a 52-week low, following a Sell advisory from Goldman Sachs (Euro2day via Tim Bass). Goldman Sachs analyst Derek Bingham predicts a flight from quality.

Customers will move away from buying more expensive “best-of-breed” offerings, like Tibco’s products, and more toward buying less expensive “good enough” substitutes that are bundled with broader solutions from the likes of IBM, Oracle Corp. and SAP AG.

This is not about whether TIBCO has the best products, but about the buying behaviour of TIBCO's customers.

The SOA mantra of Loose Coupling works both ways here. On the one hand, greater standardization and interoperability could mean there is less reason to buy everything from a single supplier, and so “best-of-breed” offerings become more viable, even during an economic downturn. On the other hand, some companies may feel that it is less critical to get the highest quality infrastructure from the beginning, since greater flexibility makes it easier to contemplate changing things later.

The traditional economics of the software industry has generally favoured the giants, which is why IBM and Microsoft have seen off so many rivals. Meanwhile, the ecology of SOA favours diversity and heterogeneity. Traditional investors such as Goldman Sachs and its Wall Street clients may not appreciate this yet. The important question for TIBCO and other niche vendors is the extent to which the economics of SOA can overcome the economics of software.

Update and Correction

The TIBCO shareprice recovered on January 16th, and has risen further since, perhaps helped by buy-out speculation following Oracle's acquisition of BEA.

But Tim and I were misled by the original story (apparently from Thomson Financial) linking the shareprice fall to the Goldman Sachs advisory. According to Yahoo (another company that knows something about buyout speculation!), the Goldman Sachs advisory was last April.

I stand by my original point, however, that stock market investors and city analysts don't necessarily appreciate the economics of SOA.