Keeping a shared golang package in sync across repos with a golden-file hash test
Three services carried their own copy of the same logger package and one copy quietly drifted. A small Go test that hashes the shared files against a golden file now fails the build when a copy changes on its own.
We have three Go services that all log through the same small logger package. Not a shared module, just the same folder copied into each repo. Two copies were byte for byte the same. The third had drifted: a different field name and a different time format. Nobody did it on purpose, it just happened one edit at a time.
That is how you end up with three log formats that you can't query together. In this blog we will look at the cheap fix we put in: a test that hashes the shared files and compares them with a golden file. It does not stop anyone from editing the package. It stops the edit from shipping in one repo only.
Why not a shared module
The proper answer is a Go module that every service imports. I thought about it and decided against it for now. A module means a separate repo (or a multi-module repo), tags, go get bumps in every service and some pipeline to publish it. For a few files of logging code that don't change often, that felt like a lot of ceremony.
Copying is fine. Copying without a check is the problem. So the plan was simple: keep copying, but make a forgotten or local edit fail the tests.
The shared package
For this post I wrote a small version of the package so you can run everything. It lives at internal/logx in each service. The logger itself is a thin wrapper over log/slog that fixes the field names, which is exactly the thing that drifted for us.
// Package logx is the shared logger. The same files are copied into every
// service; sync_test.go fails when a copy drifts from shared.golden.
package logx
import (
"io"
"log/slog"
"os"
"time"
)
// New returns a JSON logger with the field names every service agrees on.
func New(service string, w io.Writer) *slog.Logger {
if w == nil {
w = os.Stdout
}
h := slog.NewJSONHandler(w, &slog.HandlerOptions{
ReplaceAttr: func(_ []string, a slog.Attr) slog.Attr {
switch a.Key {
case slog.TimeKey:
// One time format everywhere, in UTC.
return slog.String("ts", a.Value.Time().UTC().Format(time.RFC3339Nano))
case slog.MessageKey:
return slog.Attr{Key: "msg", Value: a.Value}
}
return a
},
})
return slog.New(h).With(slog.String("service", service))
}
There is also a redact.go with a couple of masking helpers. The content doesn't matter for this post. What matters is which files are "shared" and must be identical everywhere.
The golden-file hash test
The test reads every shared file, takes a SHA-256 of it and compares with a line in shared.golden. With a flag it rewrites the golden file instead.
package logx
import (
"crypto/sha256"
"encoding/hex"
"flag"
"os"
"path/filepath"
"slices"
"strings"
"testing"
)
var update = flag.Bool("update-logx-golden", false, "re-record shared.golden")
// sharedFiles must be byte-identical in every service. The test file is in
// the list on purpose, so nobody can quietly relax the check in one copy.
var sharedFiles = []string{"logx.go", "redact.go", "sync_test.go"}
// localFiles may differ per service (service specific helpers).
var localFiles = []string{}
func TestSharedFilesInSync(t *testing.T) {
got := map[string]string{}
for _, name := range sharedFiles {
b, err := os.ReadFile(name)
if err != nil {
t.Fatalf("reading %s: %v", name, err)
}
// Windows checkouts can turn \n into \r\n; that is not drift.
b = []byte(strings.ReplaceAll(string(b), "\r\n", "\n"))
sum := sha256.Sum256(b)
got[name] = hex.EncodeToString(sum[:])
}
if *update {
var sb strings.Builder
sb.WriteString("# sha256 of the shared logx files. Same in every repo.\n")
for _, name := range sharedFiles {
sb.WriteString(name + " " + got[name] + "\n")
}
if err := os.WriteFile("shared.golden", []byte(sb.String()), 0o644); err != nil {
t.Fatal(err)
}
t.Log("shared.golden re-recorded; copy the package into the other repos")
return
}
want := readGolden(t)
for _, name := range sharedFiles {
if want[name] == "" {
t.Errorf("%s has no line in shared.golden", name)
continue
}
if got[name] != want[name] {
t.Errorf("%s drifted from the shared copy\n golden: %s\n actual: %s\n"+
"copy the package from the source repo, or re-record there with -update-logx-golden",
name, want[name], got[name])
}
}
}
// TestNoUnlistedFiles catches a new file added to one copy only.
func TestNoUnlistedFiles(t *testing.T) {
entries, err := filepath.Glob("*.go")
if err != nil {
t.Fatal(err)
}
for _, name := range entries {
if !slices.Contains(sharedFiles, name) && !slices.Contains(localFiles, name) {
t.Errorf("%s is not in sharedFiles or localFiles", name)
}
}
}
func readGolden(t *testing.T) map[string]string {
t.Helper()
b, err := os.ReadFile("shared.golden")
if err != nil {
t.Fatalf("reading shared.golden (run with -update-logx-golden): %v", err)
}
out := map[string]string{}
for line := range strings.Lines(strings.ReplaceAll(string(b), "\r\n", "\n")) {
line = strings.TrimSpace(line)
if line == "" || strings.HasPrefix(line, "#") {
continue
}
name, sum, ok := strings.Cut(line, " ")
if !ok {
t.Fatalf("bad shared.golden line: %q", line)
}
out[name] = strings.TrimSpace(sum)
}
return out
}
A few choices in there that I would keep even in a bigger setup.
The test file hashes itself. sync_test.go is in sharedFiles, so if someone in one repo "fixes" a failing test by loosening it, that copy no longer matches the golden and fails anyway. It sounds a bit cute but it closes the most obvious escape hatch.
Line endings are normalised before hashing. Without that a Windows checkout with core.autocrlf on reads as a drifted copy, and people learn to ignore the test.
The golden file is plain text with comments. When a diff in a pull request shows a changed hash, you can see which file changed without running anything.
Recording and running it
The first run fails because there is no golden yet:
--- FAIL: TestSharedFilesInSync (0.00s)
sync_test.go:49: reading shared.golden (run with -update-logx-golden): open shared.golden: no such file or directory
Then record it in repo that owns the package:
go test ./internal/logx/ -run InSync -update-logx-golden -v
cat internal/logx/shared.golden
# sha256 of the shared logx files. Same in every repo.
logx.go 155d9f0f58108c246171fd7a549740a7251c742c7891da29c8e93f19f0be9d56
redact.go 7a1ef471ec29117f302e8e7b2b04337e70754abb997cb80eac5cd99f8772eb7d
sync_test.go 32e9b6676dca6052852b786596aff782b7f6a361271f26f0f17e3cdc49abcae3
Copy the whole folder, golden included, into the other service and its tests pass. To check the test really bites, I changed the time key in the worker copy, the same kind of edit that caused our drift:
sed -i '' 's/"ts"/"time"/' internal/logx/logx.go
go test ./internal/logx/
--- FAIL: TestSharedFilesInSync (0.00s)
sync_test.go:56: logx.go drifted from the shared copy
golden: 155d9f0f58108c246171fd7a549740a7251c742c7891da29c8e93f19f0be9d56
actual: fbc3a8aba329e7ec790d1047483c44078f8727c7931eee18178292a6aee2883d
copy the package from the source repo, or re-record there with -update-logx-golden
FAIL
I also dropped a empty extra.go into one copy. TestNoUnlistedFiles caught it straight away. And I converted redact.go to CRLF with perl, the test still passed, which is what I wanted.
The hole: re-recording in the wrong repo
This is the part that bothered me. If a developer in the worker repo edits logx.go, sees the failure and runs -update-logx-golden right there, the test passes again. I tried it and it did. The golden is only the same everywhere by agreement.
So the rule is "re-record only in the owning repo", and I don't trust rules that live in people's heads. The fix is a small sync script that copies the package and then compares the golden files byte for byte:
#!/usr/bin/env bash
# Copy the shared logx package from the source repo into other repos and
# run their tests. Usage: ./sync-logx.sh ../orders-api ../billing-worker ...
set -euo pipefail
src="$1"; shift
for repo in "$@"; do
dst="$repo/internal/logx"
mkdir -p "$dst"
for f in logx.go redact.go sync_test.go shared.golden; do
cp "$src/internal/logx/$f" "$dst/$f"
done
echo "== $repo"
(cd "$repo" && go test ./internal/logx/)
cmp -s "$src/internal/logx/shared.golden" "$dst/shared.golden" || {
echo "shared.golden differs in $repo" >&2
exit 1
}
done
./sync-logx.sh orders-api billing-worker
In CI you can run the same cmp against the owning repo's golden file (a checkout of that repo in the job, or a raw download with a read token). Then a local re-record shows up as a failed check on the pull request, which is where someone will actually look at it.
When to stop doing this
If the shared code grows past a handful of files, or more than a couple of teams edit it, publish a module. The golden test is a stop gap that works because the package is small and changes rarely. For us it turned "three log formats" back into one with about 100 lines of test code, and the same trick works for any small package you copy around: error types, a retry helper, the redaction rules from the post on failing the build when someone logs a password or token with zap.
I ran all of this with Go 1.27.3. Same advice as in the govulncheck post though: keep the go line in each repo's go.mod current, or the copies can behave differently even when the bytes match. The only newer bit is strings.Lines, which needs Go 1.24 or later. If you are stuck on something older, a strings.Split on "\n" does the same job.