Four indistinguishable layers of figures
Design rules for information systems used to plan and monitor internal oversight
Summary
An annual oversight plan holds figures that look alike but are not. Some state what was planned, some state what was actually carried out, some state how much was paid, and some state how the results were assessed. All four are routinely mixed in one file with nothing to tell them apart. As a result, a question as simple as "has this assignment started?" cannot be answered without asking the person responsible.
This note calls the problem four indistinguishable layers of figures. It argues that the four merge not through carelessness but because the medium offers no place to keep them separate. From there it sets out five design rules that any inspectorate can use when building or assessing its oversight planning tools. The note states the rules, not the results of testing them. A before-and-after measurement is under way, and its results will be reported separately.
1. A question nobody can answer
Start with the most ordinary situation. A senior manager points to one row in the list of assignments and asks whether that assignment has started.
The row is there. The amount is there. The dates are there. Yet to answer, someone has to phone the assistant inspector in charge (Inspektur Pembantu, or Irban, who leads one of the inspectorate's audit groups), open another file, or remember last week's meeting. The figure on the screen is not enough. Not because it is wrong, but because nobody knows what kind of figure it is.
This is easily mistaken for a problem of incomplete data, as if the file simply had not been updated yet. That is not the problem. A file updated every day still cannot answer the question, as long as one amount column is used interchangeably for four different things.
2. The four layers
| Layer | Question it answers | Owner | How often it changes |
|---|---|---|---|
| Original plan | What the inspectorate intends to do this year | Assistant inspector (Irban) | Once a year, then frozen |
| Actual execution | Which assignments were actually carried out and completed | Assistant inspector (Irban) | With each assignment |
| Actual spending | How many rupiah were actually paid out | Finance unit | Follows the expense settlement cycle |
| Assignment evaluation | What the results were, and what follow-up is needed | Irban and the Inspector (head of the inspectorate) | After reporting |
What matters is not the table but the one thing it shows. These four layers are not four columns of one fact. They are four different facts, with four different owners, changing at four different rhythms.
Putting them in one place without labels is like asking four people to write on the same whiteboard with markers of the same colour, then hoping someone can tell who wrote what.
3. Why the four merge
This is nobody's carelessness, and it is worth saying so plainly, because a fix that begins by blaming people will not be used by those people.
The cause lies in the medium. A spreadsheet is a flat surface of rows and columns with no layers. Nor does it offer a place to record who entered a figure, and when. As a file passes from the assistant inspector to the planning unit and then to the finance unit, whoever holds it last overwrites what the previous holder entered. The record of where each figure came from is not erased. It was never kept in the first place.
The different rhythms make it worse. The plan is frozen once a year, execution moves with every assignment, and spending follows an expense settlement cycle that is in step with neither. A single file that has to serve three rhythms will always be right for one of them and out of date for the other two.
So the four layers were not combined by anyone's decision. They ran together, because there was no wall to keep them apart.
4. What breaks
Four consequences, from the mildest to the most dangerous.
First, every question needs a person. A figure that carries no context forces someone to act as its interpreter. This cost is rarely counted because it arrives in small, scattered amounts: five minutes here, a phone call there.
Second, variances cannot be explained. When the plan and the actuals differ, there are four possibilities that cannot be told apart.
- The assignment was not carried out
- It was carried out but not yet paid for
- It was paid for but not yet settled with receipts
- It was carried out in a different form from the plan
Without layers, all four look exactly the same.
Third, calculated figures get promoted to input figures. A calculated result, such as the remaining budget (the budget ceiling less spending), the share of the budget spent, or the average number of days per assignment, is copied into another file. In the new file it loses any sign that it was calculated. After a few rounds of copying, nobody knows any longer which figures were once typed in by a person.
Fourth, and this is the one internal auditors in government most need to watch for, conclusions can come to rest on calculated figures. Once an audit conclusion relies on a figure computed by a tool, what gets challenged is not the figure but the tool. Defending a tool costs far more than defending a figure, and a defence of the tool cannot be prepared after the fact.
5. The design rules
The following five rules are written so that they can be used outside the place where they were developed. None of them requires a particular technology. All of them can be applied to a well-kept spreadsheet just as they can to a database.
R1. Every figure carries a layer and an owner. No figure goes without both. If a figure's layer cannot be identified, it is not ready to be stored.
R2. Figures are never added up across layers. Adding the plan to the actuals produces nothing meaningful, even if the result is a tidy number. A tool that allows it is producing figures that have no name.
R3. Input figures are kept distinct from calculated figures, on screen and in every export. The distinction has to survive export, because that is where figures start to change hands.
R4. Formal conclusions rest only on input figures. Calculated figures may be used to direct attention, set priorities, and choose audit samples. They are not used as the basis for a conclusion.
R5. Every output states the tool version and the date the data was extracted. This is the cheapest defence available, and the only one that cannot be added later. By the day it is needed, it must already have been there for a long time.
Alongside the five rules stands one stance that is harder to practise than to state.
The tool refuses to guess. When a figure's layer cannot be established, the tool does not show a total. It states that no total can be given. Showing a guessed total is far more dangerous than showing a blank, because a wrong total still looks like an answer.
This stance is nearly always resisted when first proposed, because it makes the screen look emptier than the spreadsheet it replaces. The counterargument takes one sentence. The spreadsheet was never right; it simply never admitted it.
6. How to tell whether the rules work
Design rules that cannot be tested are only a matter of taste. The indicators below were chosen because they can be measured without relying on the judgement of whoever designed the tool.
- The time needed to answer a question from management about a unit's remaining budget as of that day
- The share of completed assignments that actually appear in the original plan
- The number of revision rounds between those who draft the plan and the planning unit
- The number of people double-booked on the same dates
A before-and-after measurement of these indicators is under way, and its results will be reported in a separate note. This note claims no results. It only sets out the problem and the rules drawn from it, so that others can check them, dispute them, or put them to use before the results are in.
7. Limits of the claims
Four things should be stated openly rather than hidden.
These rules come from a single provincial inspectorate, not from a comparison across many institutions. The person who formulated them also designed the tool and uses it. That role conflict cannot be removed. It can only be disclosed, and counterbalanced by indicators that do not depend on the author's own judgement. Any improvement measured later may also have other causes, such as a change of leadership, a change in regulations, or simply fresh attention to planning. Finally, rules formulated for oversight planning may not hold for other kinds of planning.
This note is worth publishing now, before the results exist, because the problem is more general than the tool. An inspectorate with no intention of building software at all can apply R1 to R4 tomorrow morning to the spreadsheet it already uses.
References
- Hevner, A. R., March, S. T., Park, J., & Ram, S. (2004). Design science in information systems research. MIS Quarterly, 28(1).
- Peffers, K., Tuunanen, T., Rothenberger, M. A., & Chatterjee, S. (2007). A design science research methodology for information systems research. Journal of Management Information Systems, 24(3).
About the author
Faizul Abrori is an auditor at Inspektorat Aceh, the internal audit office of the Aceh Provincial Government in Indonesia. The work is mostly reviews and audits of regional government finances. The rest goes into designing and building, personally, the tools used in that work, and into writing about oversight practice from personal experience and views. This note is one of them.
Email halo@fairi.id. Other writings and a full profile at https://fairi.id.
The views expressed in this note are the author's own and do not represent the official position of Inspektorat Aceh. This note does not name any audited entity, assignment letter number, audit result, or individual's data.