Showing posts with label business continuity. Show all posts
Showing posts with label business continuity. Show all posts

Monday, May 23, 2022

Risk Algebra

In this post, I want to explore some important synergies between architectural thinking and risk management. 


The first point is that if we want to have an enterprise-wide understanding of risk, then it helps to have an enterprise-wide view of how the business is configured to deliver against its strategy. Enterprise architecture should provide a unified set of answers to the following questions.

  • What capabilities delivering what business outcomes?
  • Delivering what services to what customers?
  • What information, process and resources to support these?
  • What organizations, systems, technologies and external partnerships to support these?
  • Who is accountable for what?
  • And how is all of this monitored, controlled and governed?

Enterprise architecture should also provide an understanding of the dependencies between these, and which ones are business-critical or time-critical. For example, there may be some components of the business that are important in the long-term, but could easily be unavailable for a few weeks before anyone really noticed. But there are other components (people, systems, processes) where any failure would have an immediate impact on the business and its customers, so major issues have to be fixed urgently to maintain business continuity. For some critical elements of the business, appropriate contingency plans and backup arrangements will need to be in place.

Risk assessment can then look systematically across this landscape, reviewing the risks associated with assets of various kinds, activities of various kinds (processes, projects, etc), as well as other intangibles (motivation, brand image, reputation). Risk assessment can also review how risks are shared between the organization and its business partners, both officially (as embedded in contractual agreements) and in actual practice.

 

Architects have an important concept, which should also be of great interest for enterprise risk management - the idea of a single point of failure (SPOF). When this exists, it is often the result of a poor design, or an over-zealous attempt to standardize and strip out complexity. But sometimes this is the result of what I call Creeping Business Dependency - in other words, not noticing that we have become increasingly reliant on something outside our control.


There are also important questions of scale and aggregation. Some years ago, I did some risk management consultancy for a large supermarket chain. One of the topics we were looking at was fridge failure. Obviously the supermarket had thousands and thousands of fridges, some in the customer-facing parts of the stores, some at the back, and some in the warehouses.

Fridges fail all the time, and there is a constant processes of inspecting, maintaining and replacing fridges. So a single fridge failing is not regarded as a business risk. But if several thousand fridges were all to fail at the same time, presumably for the same reason, that would cause a significant disruption to the business.

So this raised some interesting questions. Could we define a cut-off point? How many fridges would have to fail before we found ourselves outside business-as-usual territory? What kind of management signals or dashboard could be put in place to get early warning of such problems, or to trigger a switch to a "safety" mode of operation.

Obviously these questions aren't only relevant to fridges, but can apply to any category of resource, including people. During the pandemic, some organizations had similar issues in relation to staff absences.


Aggregation is also relevant when we look beyond a single firm to the whole ecosystem. Suppose we have a market or ecosystem with n players, and the risk carried by each player is R(n). Then what is the aggregate risk of the market or ecosystem as a whole? 

If we assume complete independence between the risks of each player, then we may assume that there is a significant probability of a few players failing, but a very small probability of a large number of players failing - the so-called Black Swan event. Unfortunately, the assumption of independence can be flawed, as we have seen in financial markets, where there may be a tangled knot of interdependence between players. Regulators often think they can regulate markets by imposing rules on individual players. And while this might sometimes work, it is easy to see why it doesn’t always work. In some cases, the regulator draws a line in the sand (for example defining a minimum capital ratio) and then checks that nobody crosses the line. But then if everyone trades as close as possible to this line, how much capacity does the market as a whole have for absorbing unexpected shocks?

 

Both here and in the fridge example, there is a question of standardization versus diversity. On the one hand, it's a lot simpler for the supermarket if all the fridges are the same type, with a common set of spare parts. But on the other hand, having more than one type of fridge helps to mitigate the risk of them all failing at the same time. It also gives some space for experimentation, thus addressing the longer term risk of getting stuck with an out-of-date fridge estate. The fridge example also highlights the importance of redundancy - in other words, having spare fridges. 

So there are some important trade-offs here between pure economic optimization and a more balanced approach to enterprise risk.

Tuesday, July 24, 2018

Evidence-Based Planning

Everybody's favourite internet-book-retailer-cum-cloud-computing-giant is planning for a wide range of outcomes after Brexit.
"Like any business, we consider a wide range of scenarios in planning discussions so that we’re prepared to continue serving customers and small businesses who count on Amazon, even if those scenarios are very unlikely," a spokesperson said.

However, a Government spokesperson dismissed speculation about civil unrest, saying
"Where is the evidence to suggest that would happen?"

To which one might counter

"Where is the evidence to suggest that wouldn't happen?"



There is a methodological gulf between these two positions. One is planning for things you can't prove won't happen. The other is NOT planning for things you can't prove WILL happen.

The political problem with planning for things that might not happen, is that people may criticize you for wasting time and money on something that didn't happen. Whereas if you fail to plan for something that is unlikely to happen, and then it does happen, you can appeal to bad luck. Or the wrong kind of snow.

As with other modes of decision-making, planning simply to avoid censure is not necessarily conducive to good outcomes.


Gareth Corfield, I predict a riot: Amazon UK chief foresees 'civil unrest' for no-deal Brexit (The Register, 23 July 2018)

Rob Davies, No-deal Brexit risks 'civil unrest', warns Amazon's UK boss (The Guardian, 23 July 2018)

Related Post: Decision-Making Models (March 2017)

Saturday, April 28, 2018

Be the Change

Anyone fancy a job as Head of Infrastructure? Here is the job description, posted to Linked-In earlier this week.

We're responsible for "IT Change", including the end to end architecture, deployment and maintenance of IT infrastructure technologies across [organization]. We’re the first technical point of contact for people in [organization] who want to speak to the CIO function. We take business requirements and architect solutions, then work with [group IT] to input the solution into our data centres.
We provide direction, thought leadership, guidance and subject matter expertise on our IT estate to make sure we get the maximum value from our investment in our IT. We do this by defining our IT strategy and aligning it with Group IT, producing technology roadmaps and identifying and recommending IT solution opportunities, supporting business initiatives and ideas, and documenting and managing our architecture assets.
The Head of Infrastructure is a key leadership role in the CIO and critical to the delivery of both customer and partner facing technology. Working closely with our technology supplier, group IT, CISO and Service Management teams, this leader will be accountable for the end to ends analysis, design, build, test and implementation of; Platforms and Middleware, Network and Communications, Cloud Services, Data Warehouse and End User Services.
https://www.linkedin.com/jobs/view/633114556/

The job description contains a number of key words and phrases that architects should be comfortable with - direction, strategy, alignment, thought leadership, roadmaps, architecture assets.

But perhaps the first clue that there may be something amiss with this position is the fact that "IT Change" is in quotes. (As if to say that in IT, nothing really changes.)

The Register has contacted the person who is (according to Linked-In) currently holding this position. Is he moving on, moving up? Could this vacancy be connected in any way with recent IT difficulties facing the organization? (No answer reported. Curious.)

The recent IT difficulties facing this particular organization have come to the attention of politicians and the media. After the chair of the Treasury Select Committee described the situation as having "all the hallmarks of an IT meltdown", the word "meltdown" is now the descriptor of choice for journalists covering the story.

But help is at hand: IBM has kindly volunteered to help sort out the mess. So we can guess what "working closely with our technology supplier" might look like.




Karl Flinders, TSB IT meltdown has the makings of an epic (Computer Weekly, 25 April 2018)

Samuel Gibbs, Warning signs for TSB's IT meltdown were clear a year ago – insider (The Guardian, 28 April 2018)

Kat Hall, Newsworthy Brit bank TSB is looking for a head of infrastructure (The Register, 27 April 2018)

Stuart Sumner, TSB brings in IBM in attempt to resolve IT crisis (Computing, 26 April 2018)

Saturday, March 10, 2018

Fail Fast - Why did the Chicken cross the road?

A commonly accepted principle of architecture and engineering is to avoid a single point of failure (SPOF). A single depot for a chain of over 850 fast food restaurants could be risky, as KFC was warned when it announced that it was switching its logistics from Bidvest to a partnership with DHL and QSL, to be served out of a single depot in Rugby. We may imagine that the primary motivation for KFC was cost-saving, although the announcement was dressed up in management speak - "re-writing the rule book" and "setting a new benchmark".

The new system went live on 14th February 2018. The changeover did not go well: by the weekend, over three quarters of the stores were closed. Rugby is a great location for a warehouse - except when there is a major incident on a nearby motorway. (Who knew that could happen?)

After a couple of weeks of disruption, as well as engaging warehouse-as-a-service startup Stowga for non-food items, KFC announced that it was resuming its relationship with Bidvest. According to some reports, Burger King also flirted with DHL some years ago before returning to Bidvest. History repeating itself.

However, the problems faced by KFC cannot be attributed solely to the decision to supply the whole UK mainland from Rugby. A just-in-time supply chain needs contingency planning - covering business continuity and disaster recovery. (Good analysis by Richard Priday, who tweets as @InsomniacSteel.)



KFC revolutionizes UK foodservice supply chain with DHL and QSL appointment (DHL Press Release, 11 Oct 2017)

Andrew Don, KFC admits chicken waste as cost of DHL failure grows (The Grocer, 23 Feb 2018)

Andrea Felsted, Supply chains: Look for the single point of failure (FT 2 May 2011)

Adam Leyland, KFC supply chain fiasco is Heathrow's Terminal 5 all over again (The Grocer, 23 Feb 2018)

Charlie Pool (CEO of Stowga), Warehousing on-demand saves KFC (Retail Technology 26 February 2018)

Richard Priday, The inside story of the great KFC chicken shortage of 2018 (Wired 21 February 2018) How KFC ended the great chicken crisis by taking care of its mops (Wired 2 March 2018) The KFC chicken crisis is finally over: it's (sort of) ditched DHL (Wired 8 March 2018)

Carol Ryan, Stuffed KFC only has itself to blame (Reuters, 20 February 2018)

Su-San Sit, KFC was 'warned DHL would fail' (Supply Management, 20 February 2018)

Matthew Weaver, Most KFCs in UK remain closed because of chicken shortage (Guardian 19 Feb 2018) KFC was warned about switching UK delivery contractor, union says (Guardian 20 Feb 2018)

Zoe Wood, KFC returns to original supplier after chicken shortage fiasco (Guardian 8 March 2018)

Wikipedia: Single Point of Failure

Related posts: Fail Fast - Burger Robotics (March 2018)

Tuesday, September 06, 2016

The Cruel World of Paper

Some airlines have blamed recent service disruptions on power cuts. In contrast, recent disruptions to service at BA have been blamed on a system upgrade in the check-in department. Among other things, it appears that BA staff could not access their computers to see which passengers had gone through security.




The cruel world of paper remains a constant threat in some sectors, particularly healthcare. In the same week of February 2017, those afflicted included a hospital in Melbourne and a hospital trust in Tyneside. Healthcare also seems particularly vulnerable to virus and ransomware attacks.

In contrast, when media giant WPP was hit by a ransomware attack in June 2017, although some creatives turned to "good old-fashioned pen and paper", it seems that other departments simply went out, bought some Macs, and went to work in local coffee shops. Oh, cruel world.



Related posts

Single Point of Failure (Airlines) (August 2016)



Chris Baraniuk, BA apologises for check-in delays at Heathrow and Gatwick (BBC News,18 July 2016)

Kate Gibbons, Cyberattack sent marketing giant back to pen and paper (Times, 7 December 2017) (paywall)

Jeremy Lee, Safety first: Lessons from the cyberattack on WPP (Campaign, 7 July 2017)

Kate McDonald, EMR crash causes Austin Health to revert to paper temporarily (Pulse+IT, 17 February 2017)

Laura Stevens, Failure of core network at Northumbria downs IT systems (Digital Health, 23 February 2017)

Evan Sweeney, Buffalo hospital returns to pen and paper after a virus shuts down IT systems (12 April 2017)

British Airways passengers delayed by computer glitch (BBC News, 6 September 2016)

Who was hit by the Cyber attack? (BBC News, 13 May 2017)




Post updated and extended 28 December 2017

Friday, September 02, 2016

Single Point of Failure (Comms)

Large business-critical systems can be brought down by power failure. My previous post looked at Airlines. This time we turn our attention to Telecommunications.




Obviously a power cut is not the only possible cause of business problems. Another single-point of failure could be a single rogue employee.




Gavin Clarke, Telecity's engineers to spend SECOND night fixing web hub power outage (The Register, 18 November 2015)


Related Post: Single Point of Failure (Airlines) (August 2016)

Monday, August 08, 2016

Single Point of Failure (Airlines)

Large business-critical systems can be brought down by power failure. Who knew?

In July 2016, Southwest Airlines suffered a major disruption to service, which lasted several days. It blamed the failure on "lingering disruptions following performance issues across multiple technology systems", apparently triggered by a power outage.
In August 2016 it was Delta's turn.

Then there were major problems at British Airways (Sept 2016) and United (Oct 2016).



The concept of "single point of failure" is widely known and understood. And the airline industry is rightly obsessed by safety. They wouldn't fly a plane without backup power for all systems. So what idiot runs a whole company without backup power?

We might speculate what degree of complacency or technical debt can account for this pattern of adverse incidents. I haven't worked with any of these organizations myself. However, my guess is that some people within the organization were aware of the vulnerability, but this awareness didn't somehow didn't penetrate the management hierarchy. (In terms of orgintelligence, a short-sighted board of directors becomes the single point of failure!) I'm also guessing it's not quite as simple and straightforward as the press reports and public statements imply, but that's no excuse. Management is paid (among other things) to manage complexity. (Hopefully with the help of system architects.)

If you are the boss of one of the many airlines not mentioned in this post, you might want to schedule a conversation with a system architect. Just a suggestion.


American Airlines Gradually Restores Service After Yesterday's Power Outage (PR Newswire, 15 August 2003)

British Airways computer outage causes flight delays (Guardian, 6 Sept 2016)

Delta: ‘Large-scale cancellations’ after crippling power outage (CNN Wire, 8 August 2016)

Gatwick Airport Christmas Eve chaos a 'wake-up call' (BBC News, 11 April 2014)

Simon Calder, Dozens of flights worldwide delayed by computer systems meltdown (Independent, 14 October 2016)

Jon Cox, Ask the Captain: Do vital functions on planes have backup power? (USA Today, 6 May 2013)

Jad Mouawad, American Airlines Resumes Flights After a Computer Problem (New York Times, 16 April 2013)

 Marni Pyke, Southwest Airlines apologizes for delays as it rebounds from outage (Daily Herald, 20 July 2016)

Alexandra Zaslow, Outdated Technology Likely Culprit in Southwest Airlines Outage (NBC News, Oct 12 2015)


Related post Single Point of Failure (Comms) (September 2016), The Cruel World of Paper (September 2016), When the Single Version of Truth Kills People (April 2019)


Updated 14 October 2016. Link added 26 April 2019

Tuesday, February 24, 2009

Architecture or Firefighting?

For many organizations, the main worry today is Business Survival.

ZDNet blogger Ian Finley reckons "companies will need architects when the crisis is over, but today, they need fire fighters". (When the building’s on fire, who calls an architect?)

In response, fellow ZDNet blogger Joe McKendrick said that "the problem is we don’t have enough architecture as it is, and even when times are good, we’re always firefighting". (SOA 2009: Do we need architects or firefighters?) Mike Kavis agrees: "The reason there are so many fires is that there is so little architecture!"

Comparing the IT crisis with the financial crisis, my colleague David Sprott talks about a toxic mess of IT assets that can’t be easily cleared up. As he says, "although we may be cancelling projects and reducing activity, unless we get smart on architecture we are increasing the toxic mess". Also known as technical debt or technology debt (HT @jpmorgenthal).

As far as I can tell from Joe's later post, Ian and Joe have found a middle point of agreement: focus-focus-focus on the essentials. (2009: a year of extreme focus)

These are not the only voices arguing that Little SOA is more practical/pragmatic than Big SOA. See for example Jordan Braunstein.



At the bottom of these debates is an important cultural clash.

In many organizations, there are strong cultural divisions within IT - developers versus architects versus project managers. (In previous posts I have used the metaphor of planets: Mars, Venus, Saturn, or Uranus, while of course Enterprise Architects are from Pluto.) The current climate maybe seems to favour the project managers and the developers over the architects. But that doesn't mean that the architects should abandon their principles and expertise, and pretend to be project managers or developers. As if that's what anyone needs.

What architects do need to do is find a new way of engaging with the business and the rest of IT, in a way that achieves a better balance between the different cultures.

Some organizations have always been strongly focused on delivery - following the JFDI management style. In other organizations, the emphasis has often been on careful planning and inclusive consensus-building.

There are problems with both extremes. As a consultant, I once had the experience of working with two organizations at the opposite ends of this spectrum at the same time - trying to get the first organization to slow down and the second organization to speed up. As a result of this experience, I came up with a novel approach to working with organizational culture, based on the five elements of ancient Chinese thought. See my Slideshare presentation: Elements of Change.


Further reading

Martin Fowler, Technical Debt (Oct 2003)
Ruth Malan, Technical Debt (March 2013)
Dharmesh Shah, Understanding Technology Debt (August 2005)


Updated 14 March 2017

Saturday, September 06, 2008

Tangential Service

Sometimes a service can be provided as a cheap or unimportant side-effect of something else.

I was struck by this thought when I saw a news story indicating shortages of Molybdenum-99, a radioactive isotope used for medical scans. There are five nuclear reactors in the UK that normally produce this isotope as a by-product, but two of the five are currently down for scheduled maintenance and a third has encountered an unexpected problem. [Isotope problem may delay scans, BBC News, 5 Sept 2008] Nuclear medicine is dependent on rare isotopes produced in nuclear reactors; however, the nuclear reactors' primary purpose is the generation of electricity and not the supply of radioactive materials to hospitals.

Here's another example. Sally is planning to drive her daughter to a swimming gala, so she offers to drive her daughter's school friend as well. In the morning of the gala, Sally's daughter has a fever and cannot swim. It is now too late for the friend's parents to make alternative arrangements. Is Sally morally obliged to stand by her promise to take her daughter's friend, although she would prefer to stay at home with her sick daughter?

Here are the common factors in these examples
  • A is providing a service to B
  • This service is not (and never will be) the primary mission or purpose of A.
  • B is dependent on this service
  • Under normal circumstances, the cost to A of providing the service is negligible.
  • If A's circumstances change, it may become inconvenient, expensive or impossible for A to provide the service to B.
I'm going to call services that fit this pattern tangential services, and I think there are important considerations for both providers and consumers.
  • A may be unwilling to invest any resources in monitoring or improving the service. B might be willing to contribute some resources to improving the service (especially improving its reliability), but this may be difficult to manage.
  • B needs to monitor the service, but may not have access to information that would provide advance warning of service problems (because this is private to A).
  • Service level agreements may be weakly specified or completely absent. A may have no formal obligations to B.
  • Under normal circumstances, everyone is happy. There may be a sudden jump from happy (mutally convenient, value-adding) to unhappy (inconvenient to A and/or B, unexpected cost).
There is of course nothing wrong with providing or consuming tangential services. For the provider, it may be a bit of extra revenue or an opportunity to provide some social value. For the consumer, it may represent a significant cost-saving, or provide access to something that might otherwise have been impossible. However, it is important to understand the implications of a service's being tangential, and to think through questions of service levels, liabilities and contingency plans.

Wednesday, May 16, 2007

Service Escrow - Iron Mountain

Last month I riffed with Gianpaolo Carraro about Service Escrow. A few days later, a company called Iron Mountain launched an SaaS escrow service, covering the source code, system documentation and data.
Gianpaolo (who is one of Microsoft's SaaS experts) refers to companies providing this service as SaaS undertakers. But it might be better to call them SaaS support. By acting as a safety net for SaaS providers and their customers, they may reduce the risk associated with SaaS and contribute indirectly to SaaS market growth.

I phoned Iron Mountain to find out more about their SaaS Escrow service, and the extent to which it differed from the traditional software escrow service. Here are some of the main points of our conversation.

Storage Model

Iron Mountain stores both the software (source code, object code and documentation - all of which belongs to the SaaS provider) and the data (which belongs to the SaaS user). Whereas traditional software escrow can often operate on a fairly slow cycle, with new software versions deposited in a fairly leisurely manner, SaaS escrow generally calls for live data backup.

If the escrow conditions are triggered, Iron Mountain will release the software and data to the SaaS user. To avoid any perceived conflict of interest, Iron Mountain does not operate the software - even on an emergency basis. But this means that the SaaS developers (or some appointed third party) must install and test the software on a new server, load and test the data, and then restore the service.

This is probably not something that can or should be done overnight - so it is not going to provide much protection against a sudden and unexpected failure of a business critical service. A more likely scenario is a gradual worsening of the relationship between the SaaS provider and the SaaS user, and a growing dissatisfaction with service levels and support arrangements, approaching the point where the SaaS provider is in breach of contract. SaaS escrow means that the SaaS user has a reasonable exit from an unsatisfactory relationship, and cannot be held to ransom because the SaaS provider controls a key business asset.

The economic benefits of escrow therefore fall under the economics of governance - making sure the SaaS user has proper control of the relationship.

Charging Model

Although the SaaS provider may benefit from the existence of SaaS escrow, the prime beneficiary is the SaaS user. Historically, it has always been the user who has negotiated and paid for escrow. However, Iron Mountain is increasingly seeing more complex arrangements whereby the software provider pays for the escrow and passes these charges onto the sofware user.

In the case of SaaS escrow, a typical arrangement would be that the SaaS provider pays for the deposit of the software, while the SaaS user pays for the deposit of the data (depending on the data volumes).

It is in the interests of the SaaS user that Iron Mountain always holds the latest version of the software. Iron Mountain therefore encourages the SaaS provider to deposit software upgrades as frequently as necessary - preferably online - and does not charge on the basis of frequency. (Remember that SaaS software may undergo a faster improvement cycle than traditional software packages.)

Management Process

SaaS escrow only works if the SaaS provider has adequate software configuration management and data management, so that it becomes a matter of routine to send controlled copies to Iron Mountain. There is a certain amount of ongoing verification and audit that needs to be carried out, involving all the parties to the escrow arrangement, and Iron Mountain sees this as an important aspect of its own role.

Remember that the SaaS user does not see the software unless and until the escrow conditions are triggered - so it isn't possible to test the escrow arrangements in a trial run exercise. Instead, it is necessary to test the process at a higher level of abstraction, to provide some reassurance to the SaaS user that the escrow would work adequately.

Future

This is early days for the SaaS escrow market, and I shall be interested to see how the market develops ...

Monday, April 16, 2007

Service Escrow

A small SaaS company bites the dust, reports Gianpaolo Carraro, Director of SaaS Architecture at Microsoft.


Obviously this is not just an SaaS problem. Companies fold all the time. If you are dependent on some external capability, you'd better have a plan for business continuity. And if you have entrusted your supplier with important assets (e.g. data) you'd better be able to get your assets back quick.

A pessimist might regard this risk as a reason to avoid SaaS altogether. But I don't think Microsoft employs many pessimists. For his part Gianpaolo sees this risk as an opportunity for some enterprising SaaS undertaker - selling services to those whose beloved supplier has gone to the great SaaS graveyard.

I prefer to see this as an architectural challenge:
  • How can we design a robust network of services, one not affected by a single point of failure? (This is of course a classic problem of distributed systems.)
  • Are there patterns of collaborative networks that will stand up to the loss of any single organization in the network?
  • What are the appropriate SLAs to support these patterns?
  • And what are the interoperability requirements to make this work?
Structural problems call for structural solutions. That's what architects are for.

Update

Gianpaolo's initial response here: SaaS Undertaker (April 2007). Shortly after my exchange with Gianpaolo, a company called Iron Mountain launched an escrow service. See my post Service Escrow - Iron Mountain (May 2007).

Thursday, November 24, 2005

Efficiency and Robustness - Processing New Orleans mail after Katrina

New Orleans resident and blogger Ernie the Attorney posts an interesting diagram on Flikr.

Processing New Orleans mail after Katrina

Obviously this is not an efficient way of delivering mail. But when the direct method is not available, an inefficient workaround is usually better than nothing. I wonder what service level is being achieved by this workaround, and who is paying the extra cost? I also wonder how quickly it was possible to put this workaround into operation, and how long it will persist.

From a systems engineering point of view, I am interested to note the architectural characteristics of the postal system that makes this kind of workaround possible. Among other things, this assumes a high degree of interoperability and consistency between the various locations involved.

See my previous posts on Efficiency and Robustness