Data Estimation - Stockpile Reports

My attention has been drawn to a company called Stockpile Reports, which provides a data estimation service for piles of material. As I understand it, the calculations are based on visual images of a pile of material, possibly obtained using either a drone or a handmobile phone app, from which a detailed 3D model of the pile is produced, allowing the volume, weight or value of the material to be estimated.

I don't have any information about how good these estimates are, but they must be a lot more accurate than simply gauging the height of the pile, and I guess they are probably good enough for the organizations that use the service. In this blogpost, I want to make some observations about this service and its potential value to user organizations, as well as some general thoughts about aggregation, variety and governance

The top benefit promised by Stockpile Reports is to eliminate painful write-offs. Because inventory is shown as an asset in a company's accounts, and companies don't like having to make large adjustments when these asset values turn out to be grossly inaccurate.

But this makes it seem as if the primary motivation for accurate inventory data is financial accounting. While I understand that many execs may be concerned about this, especially CFOs, financial accounting is a largely retrospective process. Whereas operational excellence and other forms of strategic advantage have a more immediate need for accurate inventory data.

And this is not just about the relative importance of financial reporting versus operational excellence etc. It is also about top-down versus bottom-up. Because a write-off essentially means that the total figure is incorrect. This plays into a top-down view of data quality and trustworthy data. Stockpile provide a service whose primary purpose appears to be to reassure investors that the financial accounts are correct.

This leads me to talk about aggregation. Top-down aggregation is about calculating a number, from a given body of data, for a given purpose. For example, the purpose might be financial accounting, management accounting, or billing. (Where billing essentially means aggregating everything that has happened during the past month in order to determine how much the customer owes us.) With some exceptions (such as Activity-Based Costing), accounting is largely premised on reducing everything to a standard set of numbers.

So this implies three capabilities. 

1. The capability of producing the correct number 
2. The capability of inspecting the source data and calculation in order to explain or verify the number. (For example, the reason for itemized billing is to justify the number at the bottom of the invoice.) 
3. The capability of preserving the source data and calculation, so that inspection and audit will be possible into the future. (Also mechanisms to allow external parties to trust the integrity of the numbers.) 

All of these capabilities are simplified if we can eliminate as much variation as possible. If Stockpile’s drones are solely assessing the size of different heaps of stuff in order to aggregate them, then the fact that the heaps are all different is simply an annoying difficulty that the technology is designed to solve. Thus the variation between the piles is smoothed over.

Some degree of standardization may also be needed to support and enable Statistical Process Control. 

But what if the differences between the heaps are interesting and strategically significant? Now we start to look at the data in a different way - no longer top-down aggregation but something else. The Stockpile Reports claim is that estimating the size of the piles more frequently produces more accurate data. The immediate benefit is more consistent data and fewer nasty surprises. Presumably there is also a feedback loop, so that when a pile of gravel is loaded onto trucks, we can see if the quantity on the trucks matches the quantity we thought we had in the pile. So this is what might generate greater accuracy (over time), assuming other conditions remains unaltered. 

Where is the variety in this business problem?

• Different shapes and sizes of pile
• Vegetation or other extraneous material on top of the pile
• Different kinds of material - e.g. gravel versus sand
• Lack of consistency in the material - e.g. the quality of the gravel may depend on the quality of the original rock, plus the operational characteristics of the extraction machinery. Even within a single pile, the gravel may not be completely homogeneous.
• Weather. Presumably wet gravel is heavier than dry gravel.
• Control. What difference does it make if the pile of gravel is at your customer’s or supplier’s site rather than your own? How transparent are inventory levels across organizational boundaries? Do companies really want to share this information, or might they have reasons for hiding or distorting the true state of their inventory? 

These are all factors that may need to be considered when estimating the pile. But there is a more fundamental question, which is about the meaning and use of the number that is produced by this estimation process, given the variety of industry operating contexts - so I wonder whether there is a universal understanding of what counts as a pile. When estimating the size of a pile of gravel, does it matter what the gravel is going to be used for? What about the end-to-end journey of the gravel: if a pile of gravel is moved to a new location, is it still the same pile?

Depending on the frequency of viewing the pile, it may be possible to track changes in the shape of the pile, and this might provide information about stock movements (including unauthorized ones). For example, it may be possible to infer something about the movement vehicle (truck, wheelbarrow) from the difference made to the shape of the pile. For some materials, there may also be natural processes that change the volume of the pile over time - for example organic material may dry out or decay.

But this discussion of variety brings us to an important question - variety for whom. If some degree of requisite agility is needed to provide requisite variety, where does this agility need to be situated? How much does Stockpile Reports know (need to know) about its customer’s context of use? 

There may be variation within a single customer, to provide economies of scale and scope (to those customers), and to allow these organizations to operate with requisite agility in relation to a dynamic demand. Stockpile Reports might also wish to manage variation across different customers. For example, there may be some value in aggregating or comparing data from different customers - for example, calibrating the estimates of similar materials, allowing a new customer to get up to speed more quickly. 

There may be some value to both Stockpile Reports and its customers from closer coordination and mutual agility. But this raises some important questions. Firstly, how this value is shared between them. Secondly what kind of mutual trust does this require. And therefore how this coordination is to be governed.

See also my post on Testing Multiples (July 2022). Philip Boxer's analysis of the Stockpile example can be found here Accelerating demand tempos and the double-V cycle and here The dialectics implied by the Q-sectors (April 2023).

Data Governance Overview

In many organizations there is a growing awareness of the need for data governance. This is often driven by a perception that something is lacking - perhaps related to data quality or accountability. So this leads to a solution based on data ownership, and the monitoring and control of data quality. But is that all there is to data governance?

Data management operates at many levels, and there may be governance issues at each of these levels.

 

 

In some organizations, we may be constrained to work bottom-up, starting from level 1 and 2, because this is where the perceived issues are focused, and where we can engage people to establish the correct activities and control mechanisms. Some level 3 issues may be perceived, but it may initially be difficult to find people willing to commit any effort to resolving them. Level 4 and 5 issues are "above the ceiling", at least within these forums. 

There may be some clashes between the five levels, especially the higher ones. For example, at Level 5 the organization may assert a vision of a data-driven strategy, delivering strategic advantage to the business, while at Level 4 the data-related policies of an organization might be predominantly defensive, for example concerned with privacy, security and other compliance issues. If these two levels are not properly aligned, this is a matter for data governance, but one which bottom-up data governance is unlikely to reach. 

So what would top-down data governance look like? 

Purpose (final cause) - for example, to drive the reach, richness, agility and assurance of enterprise data - see my posts on DataStrategy

Approach (efficient cause) - to establish policies and processes that align the enterprise to these purposes 

Structure (formal cause) - an understanding of the contradictions and conflicts that might enable or inhibit change - what kinds of collaboration and coordination might be possible - and how might these structures themselves be altered 

Problem space (material cause) - finding issues that people are willing to invest time and energy to address 

Bottom-up data governance is driven by the everyday demands of case workers and decision-makers within the organization, the data fabric not being fit for purpose for fairly mundane tasks. Top-down data governance would be driven by senior management trying to push a transformation of data management and culture - always provided that they can give this topic much attention alongside all the other transformations they are trying to push. (I have some experiences from different organizations, which I plan to mash together into a generic example. Watch this space.)

But what if that's not where the true demand lies? 

Philip Boxer and I have been looking at alternatives to these top-down/bottom-up hierarchical forms, which we call edge-driven governance. If the traditional top-down / bottom-up can be thought of as North-South, then this is East-West. We recently got together at the Requisite Agility conference to discuss how far our thinking had developed (separately but largely in parallel) since we wrote our original papers for the Microsoft Architecture Journal. During this time, Phil spent a few years at the Software Engineering Institute (SEI) while I was mostly working at the architectural coal-face. There's a lot of really interesting work that has emerged recently, including further work by David Alberts, and a book on Bifurcation edited by Bernard Stiegler.

So how can we make practical interventions into these data governance issues from an East-West perspective? There are some critical concepts that are in play in this world, including agility, alignment, centricity, flexibility, simplicity, variety. Everyone pays lip service to these concepts in their own way, but they turn out to be highly unstable, capable of being interpreted in quite different ways from different stakeholder positions. 

So to pin these concepts down requires what John Law and Annemarie Mol call Ontological Politics. This include asking the Who-For-Whom question - agility for whom, flexibility for whom, etc. Philip has developed an abstract framework he calls Ontic Scaffolding, to support practical efforts to move “Across and Up”.

Note - videos from the Requisite Agility conference are currently in post-production and should be available soon. I'll post a link here when it is.

---

Philip Boxer, East-West Dominance (Asymmetric Leadership, April 2006)

Philip Boxer, Pathways across the 3rd epoch domain (Slideshare, November 2019)

Philip Boxer and Richard Veryard, Taking Governance to the Edge (Microsoft Architecture Journal 6, August 2006) 

Annemarie Mol, Ontological Politics (Sociological Review 1999)

Bernard Stiegler and the Internation Collective (eds), Bifurcate: There Is No Alternative (trans Daniel Ross, Open Humanities Press, 2021) 

Richard Veryard and Philip Boxer, Metropolis and SOA Governance: Towards the Agile Metropolis (Microsoft Architecture Journal 5, July 2005) 

 

Related topics on Philip's blog: Agility 

Related topics on Richard's blog: EdgeStrategy, Top-Down, RequisiteVariety

NEW eBook How to do things with data (Leanpub 2022)

What does "Top-Down" mean?

There seem to be several different ways people use the term top-down.

• Management hierarchy. Top-down means traditional command-and-control. See my post on Multiple Styles of EA.

• Decomposition/Refinement. Top-down means starting from broad vision and principles; bottom-up means starting with concrete detail and evidence. This is sometimes linked to the distinction between Analytic and Synthetic. In my post on SOA Chaos, I disagreed with Jeff Schneider's idea that analytic intelligence ("brains") was superior to synthetic intelligence ("grunts").

• Large versus small. Top-down means starting with the big pieces and assuming that the small pieces can be fitted into the gaps left by the big pieces. See Nick Malik's piece Should SOA be Top Down or Bottom Up, which I discussed in my post on Service Planning.

• Strategic versus tactical. See my post on Forest and Trees.

• Planned versus emergent. Top-down means directed and planned, bottom-up means collaborative and emergent. See my post on Emergent Architecture.

• TO-BE versus AS-IS. Top-down means starting from the future requirements of the business; bottom-up means starting from the available assets.

• Generic versus Specific. Top-down means starting from an apriori (generalized, enterprise-wide or industry-wide) schema (Kant); bottom-up means starting from the specific local requirements (Hume). This is a possible interpretation of Ali Arsanjani's posts on Kant versus Hume. See my post on The General and the Particular.

In very simple cases (such as the examples given in the books and training courses), these overlapping but logically distinct meanings of top-down may happen to coincide; but as things get more complex, the tension between competing notions of “top-down” will become more significant.

---

In a debate on community regeneration, Matthew Mckeague writes What I don’t think can be forgotten in the ‘bottom up’ ‘top down’ debate is that it shouldn’t be a binary choice. There’s a balance to be struck and professional skills and networks can’t be underestimated. It’s where the balance sits that we need some further debate.

See also The Politics of "Top-Down"


Updated 25 October 2019

The Planning Dilemma

One of the key problems faced by planning (in IT and elsewhere) has been the dilemma – top-down or bottom-up. Top-down methods produce grand schemes without addressing the problems on the ground (including legacy), while bottom-up methods produce local solutions without any overall order, coherence or reuse.

Bottom-Up Approach (Point Projects)	Top-Down Approach (Area Projects)
Local short-term initiative. No mandate to pay attention to broader, longer-term opportunities and effects.	Broader, longer-term initiative
Building a solution against immediate requirements (where “building” means design, construct or assemble)	Focus on system properties across a whole area (e.g. business domain, technical domain, infrastructure)
Strongly aligned to local objectives. Direct link between (local) benefits, costs and risks.	Indirect links between benefits (across area), costs and risks Often difficult to create/maintain business case for adequate investment in resources and infrastructure Often difficult to demonstrate return on investment
Cost-effective use of conveniently available resources (improvisation or “bricolage”)	Creating value by establishing (procuring or building) conveniently available resources

One way of addressing this dilemma is to introduce a twin track process, involving a top-down stream of activity and a bottom-up stream of activity.

Obviously for this twin-track process to be effective, we need clear allocation of responsibility, authority, expertise and work (RAEW). This is an aspect of governance - making sure the right things are done in the right way. Twin-track development exposes the inevitable tensions between business goals and service needs. And in federated/distributed development, these tensions are replicated across multiple business entities; governance then becomes a question of negotiation between two separate organizations, rather than simple management resolution within a single right framework.

---

This is a modified extract from an article on Business-Driven SOA published in the CBDI Journal, June 2004. For further extracts from this article, please see my Slideshare presentation on Organic Planning. See also our papers in the Microsoft Architecture Journal on Metropolis and SOA Governance (July 2005) and Taking Governance to the Edge (August 2006).

The notion of twin-track development is included in the Practical Guide to Federal SOA, published by the CIO Council in 2008. See also

• Stuart Charlton, On Effective Architecture (October 2007)
• Jeevak Kasarkod, CBDI SAE Reference Framework Continues to Evolve (InfoQ November 2010)
• Dave Mayo, Agile IT (June 2011)
• Mike Papazoglou, Engineering Services (??)
• Ronald Schmelzer, Why Service Consumers and Service Providers Should Never Directly Communicate (Zapthink, Feb 2008) with my response on Agility and Variation (March 2008)
• David Sprott, Beware the new silos (March 2011)

See also: What does Top-Down mean? (September 2011)

For other posts on twin-track: browse, subscribe.