Same Bytes, Every Time: Reproducible Builds as Risk Management, Not Ritual

There is a question that surfaces more often in UK procurement calls than it used to: if we rebuild from the same commit, do we get the same artefact? A couple of years ago only another engineer would have asked it. Now it arrives in a security questionnaire, in a customer's due diligence pack, occasionally from an insurer reviewing a supplier's posture.

That shift is the whole story. Reproducible builds — the property that a given source tree, compiled with a given toolchain, produces byte-identical output every time — stopped being a curiosity somewhere along the line. It became evidence. Evidence that what you shipped is what you reviewed, that your SBOM is not fiction, and that a rebuild during an incident hands you the same bytes rather than a fresh set of unknowns.

Why this stopped being a nice-to-have

Two pressures are doing most of the work. The first is supply chain scrutiny. Software bills of materials are now a routine ask, and an SBOM that cannot be tied back to a verifiable build is just a spreadsheet with good intentions. If you cannot demonstrate that the binary in production corresponds to the source you audited, the document proves very little.

The second is quieter: hygiene frameworks. Cyber Essentials and its siblings do not demand bit-for-bit reproducibility, and it would be wrong to claim they do. But the surrounding culture — controlled builds, pinned dependencies, signed releases, documented change — pulls teams in exactly that direction. Once you are already pinning and signing, the remaining gap is smaller than it looks.

And there is a third reason, the honest one. Reproducibility makes debugging cheaper. When a build fails on a colleague's laptop but passes in CI, and both are supposed to be producing the same thing, you have narrowed the problem to the environment rather than the code. That is a gift on a Friday afternoon.

The three things that actually break it

Most failed reproducibility attempts die on the same handful of causes, and none of them are glamorous.

Timestamps and embedded metadata. Archives record creation times. Compilers embed build dates. Left alone, every build is unique by definition. The fix is the SOURCE_DATE_EPOCH convention, supported by a growing set of toolchains, plus a build step that normalises archive metadata.

Paths and ordering. Absolute build directories leak into debug symbols. Filesystem ordering leaks into archive contents. Neither matters to the program and both matter enormously to the hash. Building from a fixed path inside a container, and sorting before you package, removes most of this.

Floating dependencies. A lockfile that resolves differently tomorrow gives you a different build tomorrow. Pinning is table stakes, but pin the toolchain too — the compiler version is a dependency like any other, and it is the one teams most often leave floating.

A small-team path that does not stall the roadmap

You do not need a dedicated reproducibility programme. You need a sequence that lets you stop halfway and still be better off.

Start with container digests. Reference base images by digest, not by tag. This is a five-minute change per Dockerfile and it removes an entire class of drift overnight. It is also the single easiest thing to point at when someone asks what you have done.

Next, make the build hermetic in the modest sense: no network access during compilation, dependencies fetched beforehand, everything running inside a pinned image. You are not chasing formal hermeticity theorems. You are removing the ability for the build to reach out and grab something different.

Then add a double-build check. In CI, build twice from a clean checkout and compare hashes. If they differ, fail. Expect this to fail often at first — that is the point. Each failure is a source of nondeterminism you now know about by name.

Finally, sign what you produce. Sigstore or cosign, an attestation linking artefact to commit, published alongside the release. Signing without reproducibility gives you a tamper-evident record of something you cannot verify. Signing with it gives you a chain someone else can follow.

What CI should actually check

Keep the gate narrow or it will be disabled within a month. Two builds, hash comparison, fail loudly. A separate job that rebuilds a tagged release and confirms the published digest. And a diff of the SBOM between builds, so a new transitive dependency shows up as a review item rather than a surprise.

Everything else — per-platform reproducibility, cross-compilation parity — can wait until the basics are boring.

The honest trade-offs

Reproducible builds cost time. Builds get slower because you build twice. Engineers get frustrated because a change breaks a hash for reasons that feel cosmetic. Some vendor toolchains simply will not cooperate, and you will spend an afternoon on a compiler flag that emits a build ID.

Be upfront about that. The payoff is not that builds become deterministic everywhere. It is that nondeterminism becomes visible, attributable and, eventually, rare.

The conversation with your product manager

Bring evidence, not ideology. Something like: we can now prove a released binary matches a reviewed commit, which shortens the next security questionnaire; we can rebuild an old release after a CVE and know exactly what changed; and we have cut a recurring class of 'works on my machine' debugging. Frame the CI time as a cost with a named benefit attached.

A short checklist helps: base images pinned by digest; toolchain version pinned; dependencies locked and cached; SOURCE_DATE_EPOCH set; double-build hash comparison in CI; releases signed with an attestation; SBOM diffed between builds.

None of that is exotic. It is just discipline, written down — which is roughly what craftsmanship has always meant when you strip the ceremony away.