Subscribe
AWS

Daily EBS snapshots with Data Lifecycle Manager in Terraform

Data Lifecycle Manager takes a snapshot of every EBS volume with a given tag once a day and deletes the old ones for you. Here is the Terraform for the policy and its IAM role, how I tag the root volume, and why I still keep a separate database backup.

Daily EBS snapshots with Data Lifecycle Manager in Terraform

The EC2 instance that ran my blog from 2020 until this month had never had a snapshot taken. Not once in six years. I only found out when I started moving it to a new server, and the first thing I did was take a snapshot before touching anything else.

For the new server I wanted that to happen on its own, every day, and I wanted old snapshots to go away without me cleaning up. Amazon Data Lifecycle Manager (DLM) does exactly that. You give it a tag and a schedule, it snapshots every volume with that tag and deletes snapshots past the retention count. In this blog we will look at setting it up in Terraform.

Tagging the volume

DLM picks volumes by tag. So first step is putting a tag on the root volume of the instance. With the AWS provider you can tag the root volume right inside root_block_device.

resource "aws_instance" "ghost" {
  ami           = data.aws_ami.ubuntu_2404.id
  instance_type = "t3.small"
  # subnet, security group, instance profile and so on left out here

  root_block_device {
    volume_type           = "gp3"
    volume_size           = 40
    encrypted             = true
    delete_on_termination = true
    tags = {
      Name   = "ghost-root"
      Backup = "ghost-daily"
    }
  }
}

The tag is Backup = ghost-daily. I picked a name that says both what it is and how often, so when I look at a volume in the console I know what will happen to it without opening DLM.

Note it's the volume that need the tag. Tagging only the instance does nothing for a policy that targets volumes. That's an easy one to miss.

delete_on_termination = true might look scary next to backups. It only affects the volume. Snapshots are separate objects and stay around when the instance and its volume are gone, until DLM's retention removes them.

The IAM role

DLM needs a role it can assume to create and delete snapshots. AWS has a managed policy for this, AWSDataLifecycleManagerServiceRole.

resource "aws_iam_role" "dlm" {
  name = "ghost-dlm"
  assume_role_policy = jsonencode({
    Version = "2012-10-17"
    Statement = [{
      Effect    = "Allow"
      Principal = { Service = "dlm.amazonaws.com" }
      Action    = "sts:AssumeRole"
    }]
  })
}

resource "aws_iam_role_policy_attachment" "dlm" {
  role       = aws_iam_role.dlm.name
  policy_arn = "arn:aws:iam::aws:policy/service-role/AWSDataLifecycleManagerServiceRole"
}

If you've ever made a DLM policy in the console, there may already be a role called AWSDataLifecycleManagerDefaultRole in your account. You could point at that. I'd rather have the role in Terraform with a name I chose, so it's obvious what it's for and nothing depends on a role someone clicked into existence.

Mind the service-role/ part of the policy ARN. Without it the attach fails because the policy isn't found.

The policy

Now the lifecycle policy itself.

resource "aws_dlm_lifecycle_policy" "ghost_daily" {
  description        = "Daily snapshots of the Ghost host volume"
  execution_role_arn = aws_iam_role.dlm.arn
  state              = "ENABLED"

  policy_details {
    resource_types = ["VOLUME"]
    target_tags = {
      Backup = "ghost-daily"
    }

    schedule {
      name      = "daily"
      copy_tags = true
      create_rule {
        interval      = 24
        interval_unit = "HOURS"
        times         = ["20:30"] # UTC = 02:00 IST
      }
      retain_rule {
        count = 7
      }
    }
  }
}

A few things about it.

times is in UTC and it's a start time, not an exact time. DLM starts snapshots within an hour of that. So don't schedule two things at 20:30 and expect them to happen in a fixed order.

copy_tags = true copies the volume's tags onto each snapshot. So the snapshots also carry Name = ghost-root, and in the snapshot list I can see what they are without opening each one.

retain_rule { count = 7 } keeps the last seven snapshots. When the eighth gets made, the oldest gets deleted. You can also keep by age with interval and interval_unit in the retain rule. For a daily schedule it comes out about the same, I went with count because it's easier to read.

Apply it.

terraform plan -out=dlm.tfplan
terraform apply dlm.tfplan

Checking that it works

The next day, list the snapshots for the volume.

VOL=$(aws ec2 describe-volumes \
  --filters Name=tag:Backup,Values=ghost-daily \
  --query 'Volumes[0].VolumeId' --output text)

aws ec2 describe-snapshots --owner-ids self \
  --filters Name=volume-id,Values="$VOL" \
  --query 'sort_by(Snapshots,&StartTime)[].[SnapshotId,StartTime,State]' \
  --output table

DLM adds its own tags to every snapshot it makes, like aws:dlm:lifecycle-policy-id. That's how it knows which snapshots are its to delete. It also means it only ever deletes snapshots it created.

That turned out to matter for me. Before the migration I took a manual snapshot of the old server, and when I terminated the old server I took a final one of its full disk. Those don't have the DLM tags. DLM will never touch them. They stay until I delete them myself. Which is what I want for a "last copy of the 2020 box" snapshot.

You can also check the policy state from the CLI.

aws dlm get-lifecycle-policies \
  --query 'Policies[].[PolicyId,Description,State]' --output table

Getting data back from a snapshot

A snapshot you can't restore from is just a line on your bill. The simplest test is to make a volume from the latest snapshot and look inside it from some instance in the same availability zone.

aws ec2 create-volume \
  --snapshot-id snap-0123456789abcdef0 \
  --availability-zone ap-south-1a \
  --volume-type gp3 \
  --tag-specifications 'ResourceType=volume,Tags=[{Key=Name,Value=ghost-restore-test}]'

The new volume is encrypted because the snapshot is. Attach it to a scratch instance and mount it read only.

sudo mkdir -p /mnt/restore
lsblk
sudo mount -o ro /dev/nvme1n1p1 /mnt/restore
ls /mnt/restore/opt/ghost/data

Mount by device path. A copy of an Ubuntu cloud image root disk has the same filesystem label as the scratch instance's own root, cloudimg-rootfs, so mounting by label can pick the wrong disk. Clean up the test volume after, it costs money for as long as it exist.

Why I still have a separate database backup

A DLM snapshot is crash consistent. It's what the disk looked like at one moment, as if someone pulled the power cord. MySQL's InnoDB is built to recover from that, and it usually does. Usually.

I also run a nightly mysqldump plus a tarball of the Ghost content folder to S3, at 20:00 UTC, half an hour before the snapshot. That one is checked (the script fails if the dump didn't finish), it's portable, and I can restore it into any MySQL anywhere. A snapshot only helps if I want the same disk back in the same region.

So they cover different failures. The snapshot is for "the server is gone, give me the whole machine back fast". The dump is for "the data is bad or I need it somewhere else". I wrote about the dump side in my post on nightly MySQL and content backups to S3 with a systemd timer.

What it costs

Snapshots are incremental. The first one is a full copy of the used blocks. After that, each snapshot only stores blocks that changed since the one before. On a blog server that mostly sits idle, very little changes in a day, so seven of them cost much less than seven full copies. Check the snapshot line in Cost Explorer after the first week if you want a real number.

DLM itself has no charge, you only pay for the snapshots.

That's it. A tag on the volume, a role, and one policy, and the server now has seven days of history it never had before.