Case Studies

There is a significant demand for technology case studies, from would-be adopters and practitioners of specific technologies. There is also a considerable supply of technology case studies, mostly from vendors.

But I don't see the supply meeting the demand. There seems to be a gap between what people want to know and what people are willing to publish.

Most so-called case studies take this form:

Gringotts Bank (NYSE:GOBL) needed a fast yet secure customer response across multiple legacy vaults so they installed WebHogz Enterprise Edition Version 6.66. Gringott's Chief Architect Bill Weasley said Our productivity has whizzed up by 31.4%. WebHogz CTO Harry Potter said This application clearly demonstrates the magical superiority of our product over muggle alternatives.

I regard these as press releases rather than genuine case studies. What purpose can they possibly serve apart from name-dropping? We are told that a respectable large organization is happy with the product, but we are given few if any details on how the product was used. Impressive numbers may be quoted, but we have no idea how they are measured or what to compare them with.

One thing we really want to know is about practical lessons - difficulties and pitfalls. An alternative perspective perhaps?

However, Dr Aco Malfoy, a security consultant with Arthur T. Riddle, voiced some concerns about the WebHogz solution. A number of our clients have had problems with the stability and performance of the product. Because of these concerns we have advised Gringotts Bank to install a separate firewall system and invest in additional dragonware, which will add considerably to the overall cost of the WebHogz project.

That's not much better. It may identify some areas of concern, but still doesn't give us a rounded evaluation of a project, successful or otherwise.

And academic studies often aren't much use either.

Professor Hermione Granger, a fellow of Halloween College at Hogsfjord University, has completed a three-year comparative study of vaulting technologies in collaboration with a number of industrial partners including WebHogz Labs. Our preliminary findings, based on an action research paradigm, do seem really quite promising, but we obviously need much more research funding before we can produce more reliable and detailed figures.

What we need is a lot more detailed, warts-and-all case studies, in plain English. But there are never enough organizations willing to expose themselves to independent public scrutiny. Any organizations willing to volunteer?



Update: this blogpost originally referred to SOA case studies. I have removed the specific references to SOA, as I believe my argument applies to any technology or product.