182. Firmware lifecycle and the cost of a defect

Parts one through twenty-one are about making the hardware work. This part is about turning a working prototype into a product that is built in batches and maintained for years. We start with the question that costs the most money: why the same defect costs an hour on paper and two weeks in the field.

The numbers below are not "it is well known that", but a model with explicit assumptions: _verify/soft182/defects.py. Every assumption is named in the text, and any of them can be changed in one line and recomputed.

Terms used in the documentation

Term	Meaning
software lifecycle	the stages of development from requirements to the field
requirement	what the device must do
traceability matrix	a table linking each requirement to its check
design / code review	reviewing the architecture / the code
unit / integration / acceptance test	a test of one module / of modules together / of the finished product
defect, bug	an error in the product
rework	redoing work that was already done
regression	an old bug coming back
technical debt	shortcuts that will have to be paid for later
shift left	moving checks towards the start
incremental delivery	building and delivering in pieces

182.1 Stages and what each one leaves behind

What matters is not the name of the stage but one simple property: each stage builds on the previous one. A mistake in the measurement range requirement survives the architecture, the code and the tests — and each of them will have to be redone once the requirement is fixed.

182.2 Why a defect gets more expensive

The model's assumption: one hour for the fix itself plus rework of everything that was done between the moment the defect was introduced and the moment it was found. Rework per stage: requirements 1 h, architecture 2 h, code 6 h, unit tests 2 h, integration 3 h, acceptance 4 h, field 58 h (investigating the case plus reflashing the batch).

A requirements defect that survives to the field costs 76 times more than one found at its own stage. A code defect — 68 times more. Neither of these figures is about how hard the fix itself is: in both cases the fix takes an hour. What is expensive is everything around it that has to be redone.

182.3 Two ways of working on the same 100 defects

Assumptions: of 100 defects, 30 are introduced in requirements, 15 in architecture, 55 in code. Each gate catches its own share of what reaches it; the shares below are for requirements / architecture / code defects:

Whatever is not caught goes to the field.

A. We check only at the end, with acceptance testing.

B. Design reviews, code review, unit, integration and acceptance tests.

The checks themselves also take time: requirements review 8 h, architecture review 8 h, setting up code review 40 h, the unit test framework 80 h, the integration bench 40 h — 176 h of one-off investment. Including them, process B costs 848 h against 3969 h, that is 4.7 times less, and the investment pays for itself after 5.3 defects — by the sixth defect found the checks have already earned their keep.

It is useful to look not at the total but at the last row: in process A 45.5 defects out of a hundred go to the field, in process B 4.9. That is the row that turns into complaints, site visits and unplanned updates.

182.4 The requirement → check matrix

A traceability matrix is a two-column table that is filled in from the start, not just before delivery. On the left is a verifiable requirement, on the right is exactly how it was checked:

A requirement that cannot be checked is not a requirement but a wish: "the device must be reliable" cannot be checked, while "after 1000 power-on cycles there must be no failures" can.

The matrix gives two answers you cannot get otherwise: which requirements are not yet checked, and which tests check nothing.

182.5 A fix brings a fix

A fix is also a change to the code, and it can introduce a defect too. Suppose this happens in 7 % of fixes. Then one defect takes on average 1.075 fixes, and ten fixes in a row go without a new defect only in 48 % of cases.

Two practical consequences follow:

1. After every fix, run the whole test suite, not only the test that caught the original defect — otherwise the customer will find the regression.
2. A fix right before delivery is the most dangerous: nobody is left to catch it. Hence the change freeze before a release and the rule "only fixes for found defects go into the candidate".

A test suite and a build on every change are exactly what CI from section 110 does; defensive programming from section 129 reduces the share of fixes that break something next door.

182.6 Increments instead of a waterfall

The model above does not require going through the stages once, one after another. On the contrary: the cheapest place for a gate is as close as possible to where the defect is introduced, and that means short loops.

A working technique for firmware: from the first week have a skeleton that builds, blinks an LED, answers over UART and can be flashed onto the board. After that each week adds one piece — a driver, a task, a frame format — and each piece passes its own gates. Then integration does not become a separate disaster stage at the end: it happens a little every week.

What the skeleton gives you right away:

• the firmware is always in a buildable state — "broke the build" is visible the same day;
• there is a place to add tests: the unit test framework is set up once;
• the budget is visible: how much Flash and RAM is used — from the first week, not a week before delivery;
• hardware and firmware move towards each other: the board can be checked with the skeleton while the logic is not written yet.

182.7 Pitfalls

"The requirements are in our heads." Then there is nothing to check: acceptance turns into an argument about what was meant. Twenty lines in a file next to the code are better than a perfect document that does not exist.

Tests are written at the end. Gates placed after acceptance catch defects at field prices — the model above shows a 4.7-fold difference.

A fix without a test for it. The defect will come back: same code, same situation. A test for a fixed defect is the cheapest test of all — its condition is already known.

Fixing something next door "while we're at it". In a release candidate every extra fix is a 7 % chance of a new defect with no gates ahead.

Believing that process discipline is only for large teams. A solo developer pays the same 68 hours for a defect in the field — just out of their own time.

182.8 Summary

• Each stage builds on the previous one, so the cost of a defect grows not with the difficulty of the fix but with the amount of rework: 1 h at its own stage against 76 h in the field for a requirements defect and 68 h for a code defect.
• On the same 100 defects, checking along the way gives 848 h against 3969 h and 4.9 defects in the field instead of 45.5.
• The investment in checks — 176 h in the model — pays for itself after 5.3 defects.
• A requirement without a way to check it is a wish; the requirement → check matrix shows both unchecked requirements and useless tests.
• A fix introduces a new defect in 7 % of cases, so after every fix the whole test suite is run, and changes are frozen before a release.

Next: section 183 "Schedules and people: the team, the late project, three-point estimates".