Evidence by design: producing development evidence before the project ends

Four weeks before the audit, many projects begin the same exercise: hunting for test reports, reconstructing approvals, matching up versions, retrofitting documents. The effort is considerable and the result is weaker than what already existed during the work – it simply was not kept.

1. The problem with after-the-fact evidence

Evidence assembled after the fact has three weaknesses. It is expensive, because people produce it. It has gaps, because intermediate states were overwritten. And it is weak, because it shows that somebody had to look for the records – a process that produces them automatically is more convincing in any inspection.

Evidence by design reverses the order: the evidence does not follow the work, the work produces the evidence as a by-product.

2. What evidence concretely is

“Evidence” sounds abstract but is very concrete. An artefact carries weight when it meets four properties:

• Generated automatically – not written by hand.
• Attributed – to an exact software version and to an acting person or process.
• Kept unchanged – for the required period, outside volatile tool interfaces.
• Findable – through an identifier, not through the memory of individuals.

3. The artefacts per stage

What stands out is how little of this has to be produced additionally. Almost all of it exists anyway – it is simply not collected.

4. Generating them in the pipeline

The decisive step is mundane: do not merely display reports, store them as artefacts and link them to the build.

# Collect evidence in the same job that produces the build.
# Reports must outlive the CI retention period.

evidence:
  script:
    - ./gradlew test jacocoTestReport
    - syft dir:. -o cyclonedx-json > evidence/sbom.json
    - git log --format='%H %s%n%b' $PREV..$HEAD > evidence/changes.txt
    - sha256sum build/app.jar > evidence/checksums.txt
  artifacts:
    paths: [evidence/, build/reports/]
    expire_in: never          # retention follows the record-keeping policy

Retention is what matters. CI systems clear artefacts away after weeks by default, which is useless for evidence. The period has to follow the record-keeping obligation, not the storage budget.

5. The release package

In the end a release should amount to a single, self-contained package that answers what was shipped without further questions:

• version, commit state and checksum of the artefact,
• the requirements it contains, with identifiers,
• test reports with results and requirement references,
• reports from security and dependency checks,
• a software bill of materials (SBOM),
• approvals with person and timestamp,
• target environment, time and outcome of the deployment.

The package is not a document somebody writes. It is a directory the pipeline fills – and one that can be handed to an inspecting body unchanged if needed.

6. Retention and readability

After the retention period, format is the second stumbling block. Evidence has to stay readable even once the tool that produced it has been replaced. Practical rules:

• Prefer open formats – text, JSON, XML, PDF instead of proprietary exports.
• Store them self-contained: a report that points to a tool URL is worthless once that tool is gone.
• Secure integrity: checksums or signatures, so later modification would be detectable.
• Plan for migration. Changing systems means taking the evidence along – that belongs in the decommissioning plan, not at the end.

7. Limits of the approach

• Automatic evidence proves execution, not adequacy. That 400 tests are green says nothing about whether they check the right things.
• Not everything can be automated. Risk assessments, validation plans and approval decisions remain human artefacts – but they should be referenced in the same package.
• Storage and order. Keeping everything means finding nothing. The structure of the package is part of its value.

Common mistakes

• Leaving CI artefact retention at its default.
• Keeping evidence only inside the tool instead of as exported artefacts.
• No requirement reference in test reports – which makes any coverage statement impossible.
• Approvals given verbally or in chat.
• Defining the package shortly before the audit instead of at the start.

Checklist: evidence by design

Conclusion

Evidence by design is not a new methodology but a shift: the evidence is created where the work happens anyway. The technical effort is small – a few artefact paths, an honest retention period, a requirement reference in test reports.

The return shows up twice: in an inspection, because the questions can be answered without searching. And day to day, because the same artefacts are needed for every defect analysis and every change assessment. How the underlying chain is built technically is described in the deep dive traceability: from requirement to release.