Envelope encryption of PII columns in golang with AWS KMS data keys
Encrypt sensitive columns with AES-GCM under a KMS data key, store the wrapped key inside each value, reuse data keys for a short while to save KMS calls, and backfill old rows safely at startup. Tested in Go against Postgres 18.
Encrypting a few columns in Postgres is easy. Managing the key is the hard bit. If the AES key sits in an environment variable, anyone who can read the task definition or the process memory has it, rotating it means re-encrypting everything, and you can't revoke it.
Envelope encryption with AWS KMS moves the key problem to KMS. The master key never leaves KMS. Your app asks it for a data key, encrypts locally with that, and stores a wrapped (encrypted) copy of the data key next to the value. In this blog we will look at a small Go package that does this per column value, how I keep KMS calls down, and a backfill that is safe to run on every startup.
The format of one value
Each encrypted value is self describing. To decrypt it you need nothing but the string and permission to call kms:Decrypt.
kms1:base64( uint16 len(wrapped key) | wrapped key | 12 byte nonce | AES-GCM ciphertext+tag )
The kms1: prefix does two jobs. It tells the reader how to decrypt, and it lets SQL find rows that are not encrypted yet with NOT LIKE 'kms1:%'. If you ever change the format, write kms2: and support both on read for a while.
The KMS part
I keep KMS behind a two method interface. The real one is a few lines over the AWS SDK for Go v2, and tests use a local fake.
type KeyService interface {
GenerateDataKey(ctx context.Context) (plaintext, wrapped []byte, err error)
Decrypt(ctx context.Context, wrapped []byte) ([]byte, error)
}
package envelope
import (
"context"
"github.com/aws/aws-sdk-go-v2/aws"
"github.com/aws/aws-sdk-go-v2/service/kms"
"github.com/aws/aws-sdk-go-v2/service/kms/types"
)
// KMSKeys talks to a customer managed KMS key.
type KMSKeys struct {
Client *kms.Client
KeyID string // key ARN or alias/...
Ctx map[string]string // encryption context, e.g. {"purpose": "pii"}
}
func (k KMSKeys) GenerateDataKey(ctx context.Context) ([]byte, []byte, error) {
out, err := k.Client.GenerateDataKey(ctx, &kms.GenerateDataKeyInput{
KeyId: aws.String(k.KeyID),
KeySpec: types.DataKeySpecAes256,
EncryptionContext: k.Ctx,
})
if err != nil {
return nil, nil, err
}
return out.Plaintext, out.CiphertextBlob, nil
}
func (k KMSKeys) Decrypt(ctx context.Context, wrapped []byte) ([]byte, error) {
out, err := k.Client.Decrypt(ctx, &kms.DecryptInput{
CiphertextBlob: wrapped,
KeyId: aws.String(k.KeyID), // only accept our key
EncryptionContext: k.Ctx,
})
if err != nil {
return nil, err
}
return out.Plaintext, nil
}
Two small things I'd always do here. Pass KeyId on Decrypt even though KMS can work it out from the blob for symmetric keys, so a blob from some other key gets refused. And use an encryption context. It must match on decrypt, and it shows up in CloudTrail, so you can see in the logs what each decrypt was for. The key ARN in a real config looks like arn:aws:kms:ap-south-1:111122223333:key/....
The task role needs kms:GenerateDataKey and kms:Decrypt on that one key. A read only reporting job only needs kms:Decrypt.
Reusing a data key, for a while
The simple version calls GenerateDataKey for every value. That is a network round trip and a billed request on every write. The other extreme, one data key forever, throws away most of what KMS gives you.
So the sealer reuses one data key for at most 5 minutes or 500 values, whichever comes first. Each value still gets a fresh random nonce and carries its own wrapped copy of the key. Nonce reuse is the thing that breaks AES-GCM, and 500 random 96 bit nonces per key is nowhere near the danger zone.
func (s *Sealer) dataKey(ctx context.Context) (*dataKey, error) {
s.mu.Lock()
defer s.mu.Unlock()
if s.cur != nil && s.cur.uses < s.MaxUses && s.now().Sub(s.cur.created) < s.MaxAge {
s.cur.uses++
return s.cur, nil
}
plain, wrapped, err := s.keys.GenerateDataKey(ctx)
if err != nil {
return nil, err
}
if len(plain) != 32 {
return nil, errors.New("envelope: data key is not 32 bytes")
}
s.cur = &dataKey{plain: plain, wrapped: wrapped, created: s.now(), uses: 1}
return s.cur, nil
}
Encrypt and decrypt
// Encrypt seals plaintext. aad ties the value to where it lives (for
// example "contacts.phone:42"); the same aad must be passed to Decrypt, so a
// value copied into another row will not decrypt.
func (s *Sealer) Encrypt(ctx context.Context, plaintext string, aad []byte) (string, error) {
if plaintext == "" {
return "", nil
}
dk, err := s.dataKey(ctx)
if err != nil {
return "", err
}
gcm, err := newGCM(dk.plain)
if err != nil {
return "", err
}
nonce := make([]byte, gcm.NonceSize())
if _, err := rand.Read(nonce); err != nil {
return "", err
}
if len(dk.wrapped) > 0xFFFF {
return "", errors.New("envelope: wrapped key too long")
}
buf := binary.BigEndian.AppendUint16(nil, uint16(len(dk.wrapped)))
buf = append(buf, dk.wrapped...)
buf = append(buf, nonce...)
buf = gcm.Seal(buf, nonce, []byte(plaintext), aad)
return Prefix + base64.StdEncoding.EncodeToString(buf), nil
}
The aad (additional authenticated data) is the bit people skip. Without it, someone with write access to the table can copy an encrypted value from one row into another and it decrypts fine. With the table, column and row id as AAD, it fails. There is a test for exactly that.
Decrypt is the reverse: strip the prefix, read the length, split out the wrapped key, nonce and ciphertext. The one optimisation is a cache from wrapped key to plaintext key with a TTL, so a page of rows that share a data key costs one KMS call and much less round trips. I made sure expired entries get deleted, otherwise a long running process would keep every data key it ever saw in memory.
func (s *Sealer) unwrapKey(ctx context.Context, wrapped []byte) ([]byte, error) {
id := string(wrapped)
now := s.now()
s.cacheMu.Lock()
if c, ok := s.unwrap[id]; ok && now.Before(c.expires) {
s.cacheMu.Unlock()
return c.plain, nil
}
s.cacheMu.Unlock()
plain, err := s.keys.Decrypt(ctx, wrapped)
if err != nil {
return nil, err
}
s.cacheMu.Lock()
for k, c := range s.unwrap { // drop expired entries so the map can't grow forever
if !now.Before(c.expires) {
delete(s.unwrap, k)
}
}
s.unwrap[id] = cachedKey{plain: plain, expires: now.Add(s.CacheTTL)}
s.cacheMu.Unlock()
return plain, nil
}
The tests run against a fake KeyService that wraps keys with a local AES key and counts calls. With a fake clock, 1200 encryptions made exactly 3 GenerateDataKey calls (500 + 500 + 200), and moving the clock 6 minutes forward made a 4th. Reading 100 values written with one data key made 1 Decrypt call.
Backfilling existing rows
When you turn this on, older rows are still plain text. I run the backfill at startup, before the server takes traffic, the same place I'd hook things in after the one-off migration task. It has to be safe to run every time and safe when two tasks start together during a deploy.
func Contacts(ctx context.Context, db *pgxpool.Pool, s *envelope.Sealer) (int, error) {
var lastID int64
done := 0
for {
rows, err := db.Query(ctx, `
SELECT id, phone FROM contacts
WHERE id > $1 AND phone <> '' AND phone NOT LIKE 'kms1:%'
ORDER BY id LIMIT $2`, lastID, batchSize)
if err != nil {
return done, fmt.Errorf("load batch: %w", err)
}
type row struct {
id int64
phone string
}
var batch []row
for rows.Next() {
var r row
if err := rows.Scan(&r.id, &r.phone); err != nil {
rows.Close()
return done, err
}
batch = append(batch, r)
}
rows.Close()
if err := rows.Err(); err != nil {
return done, err
}
if len(batch) == 0 {
break
}
for _, r := range batch {
lastID = r.id
enc, err := s.Encrypt(ctx, r.phone, AAD(r.id))
if err != nil {
slog.Error("backfill: encrypt failed", "id", r.id, "err", err)
continue
}
// Only overwrite the value we read. If another task got here
// first (or the user changed it), this updates 0 rows.
tag, err := db.Exec(ctx,
`UPDATE contacts SET phone = $1 WHERE id = $2 AND phone = $3`,
enc, r.id, r.phone)
if err != nil {
slog.Error("backfill: update failed", "id", r.id, "err", err)
continue
}
done += int(tag.RowsAffected())
}
}
return done, nil
}
The example table is a made up contacts.phone; use your own columns. Two details are worth copying.
It walks by id (id > $1) instead of re-reading first page. Re-reading the same page with no offset looks neat, since encrypted rows drop out of the result. But if one row fails every time and you continue, that loop never ends. With a moving cursor the bad row gets logged and skipped.
The update has AND phone = $3. That is what makes two tasks at once safe: whoever writes second matches 0 rows, and a value the user changed in between never gets overwritten with an old one.
I tested it against a throwaway postgres:18-alpine container (18.7) with 1000 rows, 100 of them empty, and two goroutines running the backfill at the same time:
docker run -d --name pg-test -e POSTGRES_PASSWORD=pw -p 127.0.0.1::5432 postgres:18-alpine
export TEST_DATABASE_URL="postgres://postgres:pw@127.0.0.1:$(docker port pg-test 5432 | cut -d: -f2)/postgres?sslmode=disable"
go test ./backfill/ -v
backfill_test.go:76: task counts [231 669], GenerateDataKey calls 4, Decrypt calls 4
--- PASS: TestContacts (11.14s)
231 plus 669 is 900, every non-empty row once. A second run encrypted 0 rows, and every value decrypted with its own row id. My first thought was that 4 data keys was one too many for 900 rows. It isn't. The task that lost a race had already encrypted the row before its update matched nothing, so both tasks together did more than 900 encryptions. Harmless, but its why the counts don't add up neatly.
(I asked for a random host port with -p 127.0.0.1::5432 because the fixed ports I tried first were already taken on my machine.)
Things I'd check before turning it on
- Searching. You can't
WHERE phone = ?on ciphertext. If you need lookups, store a separate keyed hash (HMAC) of the value for equality matches. - Logs. Decrypted values should not be logged; the zap denylist test helps.
- Key policy. Who can call
kms:Decrypton that key is now the real access list for the data, so keep it short.
Built and tested with Go 1.27.3, aws-sdk-go-v2 KMS v1.63.0 and pgx v5.13.0. I didn't call real KMS for this post; the fake stands in for it, and the KMS adapter is only compiled.