Thursday, January 10, 2008

Case Studies

There is a significant demand for SOA case studies, from would-be adopters and practitioners of SOA. There is also a considerable supply of SOA case studies, mostly from vendors.

But I don't see the supply meeting the demand. There seems to be a gap between what people want to know and what people are willing to publish.

Most so-called case studies take this form:

Gringotts Bank (NYSE:GOBL) needed a fast yet secure customer response across multiple legacy vaults so they installed WebHogz Enterprise Edition Version 6.66. Chief Architect Bill Weasley said "Our productivity has whizzed up by 31.4%." WebHogz CTO Harry Potter said "This application clearly demonstrates the magical superiority of our product over muggle alternatives."

I regard these as press releases rather than genuine case studies. What purpose can they possibly serve apart from name-dropping? We are told that a respectable large organization is happy with the product, but we are given few if any details on how the product was used. Impressive numbers may be quoted, but we have no idea how they are measured or what to compare them with.

One thing we really want to know is about practical lessons - difficulties and pitfalls. An alternative perspective perhaps?

However, Dr Aco Malfoy, a security consultant with Arthur T. Riddle, voiced some concerns about the WebHogz solution. "A number of our clients have had problems with the stability and performance of the product. Because of these concerns we have advised Gringotts Bank to install a separate firewall system and invest in additional dragonware, which will add considerably to the overall cost of the WebHogz project."

That's not much better. It may identify some areas of concern, but still doesn't give us a rounded evaluation of a project, successful or otherwise.

And academic studies often aren't much use either.

Professor Hermione Granger, a fellow of Halloween College at Hogsfjord University, has completed a three-year comparative study of vaulting technologies in collaboration with a number of industrial partners including WebHogz Labs. "Our preliminary findings, based on an action research paradigm, do seem really quite promising, but we obviously need much more research funding before we can produce more reliable and detailed figures."

What we need is a lot more detailed, warts-and-all case studies, in plain English. But there are never enough organizations willing to expose themselves to independent public scrutiny. Any organizations willing to volunteer?


Mike Kavis said...

I agree. We need honest assessments from practitioners. I did posted a few lessons learned, good and bad, from my SOA initiative while it was still in-flight.

The problem is, most corporations don't want their people talking about what is going on inside their company. I did start getting a lot of pressure from my previous company as my blog grew in popularity. If you want real cases studies, you need to build your network and start collaborating with practitioners where they can talk freely with you about lessons learned without being under the watchful eye of big brother.

James McGovern said...

Everyone has it twisted. Did you know that our media relations department is measured based on how many times our company name appears in the media?

If an analyst firm has a policy to anonymize the name of the company such that I am employed by a Fortune 200 in New England then why would they bother?

If you want to do a formal SOA case study (not blog entry or twitter) on our shop, I can arrange.

James McGovern said...

The last analyst firm that approached us for a case study was Forrester in 2006 where they wanted to peak into how we do EA. No other firm has even asked us lately.