Running a Docker host on EC2 with SSM instead of SSH
How I build the EC2 host for this blog with Terraform: IMDSv2, an encrypted disk, cloud-init for Docker and only ports 80 and 443 open. Changes go in through SSM Run Command, so port 22 stays closed.
This blog used to run on an EC2 box I set up by hand in 2020. Ubuntu 18.04, Ghost-CLI, Nginx, and SSH open to my IP. It worked for years. But nothing about it was written down, and when I finally moved the blog to Docker Compose this month I wanted the new host to be built by Terraform and reached without SSH at all.
In this blog we will look at how that host is set up and how I push changes to it with AWS Systems Manager (SSM) instead of logging in.
Why I stopped wanting SSH
The cutover is what convinced me. I moved the Elastic IP from the old instance to the new one, which is the whole point of an Elastic IP: same address, no DNS change. Then I tried to SSH to that address and got this.
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@ WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED! @
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
Of course it changed. Same IP, brand new machine, new host key. SSH is doing its job here. The fix is one command.
ssh-keygen -R 203.0.113.10
But I was in the middle of a cutover, the old host was already detached from the IP, and sorting out the host key is where most of the downtime went. About five minutes in total. Not a disaster for a personal blog. Still annoying, because it had nothing to do with Ghost or Docker.
I had added a temporary SSH rule to the security group just for the migration. Once SSM was working I removed it. Port 22 has been closed since.
The instance
The instance is a plain aws_instance in Terraform. Ubuntu 24.04 from Canonical's AMI, an encrypted gp3 root disk and IMDSv2 required.
data "aws_ami" "ubuntu_2404" {
most_recent = true
owners = ["099720109477"] # Canonical
filter {
name = "name"
values = ["ubuntu/images/hvm-ssd-gp3/ubuntu-noble-24.04-amd64-server-*"]
}
}
resource "aws_instance" "ghost" {
ami = data.aws_ami.ubuntu_2404.id
instance_type = "t3.small"
vpc_security_group_ids = [aws_security_group.ghost.id]
iam_instance_profile = aws_iam_instance_profile.ghost.name
user_data = file("${path.module}/ghost/cloud-init.sh")
metadata_options {
http_endpoint = "enabled"
http_tokens = "required"
http_put_response_hop_limit = 2
}
root_block_device {
volume_type = "gp3"
volume_size = 40
encrypted = true
}
lifecycle {
prevent_destroy = true
ignore_changes = [ami, user_data]
}
}
A few notes on this. http_tokens = "required" is IMDSv2. The hop limit of 2 matters on a Docker host, because a container is one extra network hop away from the metadata service and with a limit of 1 the token request from inside a container just times out.
The ignore_changes on ami and user_data is there because most_recent = true finds a newer AMI every few weeks. Without it, terraform plan wants to replace the instance, which is the last thing I want for a box with a database on it. prevent_destroy is the second lock on the same door.
The security group
Only 80 and 443. Port 80 has to stay open for the Let's Encrypt HTTP challenge and the redirect to HTTPS, Caddy handles both.
resource "aws_security_group" "ghost" {
name = "ghost"
description = "Ghost docker host: public HTTP/HTTPS only"
vpc_id = aws_default_vpc.default.id
ingress {
description = "HTTP (ACME + redirect)"
protocol = "tcp"
from_port = 80
to_port = 80
cidr_blocks = ["0.0.0.0/0"]
ipv6_cidr_blocks = ["::/0"]
}
ingress {
description = "HTTPS"
protocol = "tcp"
from_port = 443
to_port = 443
cidr_blocks = ["0.0.0.0/0"]
ipv6_cidr_blocks = ["::/0"]
}
egress {
protocol = "-1"
from_port = 0
to_port = 0
cidr_blocks = ["0.0.0.0/0"]
}
}
The egress rule is needed. The SSM agent talks out to the SSM endpoints over 443, it never needs anything inbound.
The instance profile
SSM needs a role on the instance with the AWS managed policy AmazonSSMManagedInstanceCore. That's it.
resource "aws_iam_role" "ghost" {
name = "ghost-host"
assume_role_policy = jsonencode({
Version = "2012-10-17"
Statement = [{
Effect = "Allow"
Principal = { Service = "ec2.amazonaws.com" }
Action = "sts:AssumeRole"
}]
})
}
resource "aws_iam_role_policy_attachment" "ghost_ssm" {
role = aws_iam_role.ghost.name
policy_arn = "arn:aws:iam::aws:policy/AmazonSSMManagedInstanceCore"
}
resource "aws_iam_instance_profile" "ghost" {
name = "ghost-host"
role = aws_iam_role.ghost.name
}
The Ubuntu AMIs ship with SSM agent as a snap, so there is nothing to install. A couple of minutes after boot the instance shows up here.
aws ssm describe-instance-information \
--query 'InstanceInformationList[].[InstanceId,PingStatus,PlatformName]' \
--output table
If it says Online, you can get a shell with aws ssm start-session --target i-0123456789abcdef0. You need the Session Manager plugin installed locally for that one.
Cloud-init does the rest
User data is a bash script. It adds 2 GB of swap, installs Docker from Docker's own apt repo (the Ubuntu docker.io package lags behind and doesn't have the compose plugin), and turns on unattended upgrades.
#!/bin/bash
set -euxo pipefail
export DEBIAN_FRONTEND=noninteractive
if [ ! -f /swapfile ]; then
fallocate -l 2G /swapfile && chmod 600 /swapfile && mkswap /swapfile && swapon /swapfile
echo '/swapfile none swap sw 0 0' >> /etc/fstab
fi
apt-get update
apt-get install -y ca-certificates curl gnupg unzip
install -m 0755 -d /etc/apt/keyrings
curl -fsSL https://download.docker.com/linux/ubuntu/gpg -o /etc/apt/keyrings/docker.asc
chmod a+r /etc/apt/keyrings/docker.asc
echo "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.asc] https://download.docker.com/linux/ubuntu $(. /etc/os-release && echo "$VERSION_CODENAME") stable" > /etc/apt/sources.list.d/docker.list
apt-get update
apt-get install -y docker-ce docker-ce-cli containerd.io docker-compose-plugin
systemctl enable --now docker
usermod -aG docker ubuntu
apt-get install -y unattended-upgrades
mkdir -p /opt/ghost && chown ubuntu:ubuntu /opt/ghost
touch /var/lib/cloud/instance/ghost-bootstrap-done
The last line drops a marker file. It's an easy thing to check over SSM before copying anything onto the box, so I know cloud-init is finished.
The stack itself starts from a small systemd unit, so a reboot brings the blog back without me doing anything.
[Unit]
Description=Ghost docker compose stack
Requires=docker.service
After=docker.service network-online.target
[Service]
Type=oneshot
RemainAfterExit=yes
WorkingDirectory=/opt/ghost
User=ubuntu
ExecStart=/usr/bin/docker compose up -d --remove-orphans
ExecStop=/usr/bin/docker compose down
[Install]
WantedBy=multi-user.target
Pushing a change with Run Command
The first real change after cutover was small. I needed an ads.txt served at the root of the site, which is a few lines in the Caddyfile. No SSH, so it went through aws ssm send-command with the AWS-RunShellScript document.
The trick is getting a file onto the host. I base64 encode it locally and decode it on the other side, that way quotes and newlines in the file can't break the command.
INSTANCE_ID=i-0123456789abcdef0
B64=$(base64 < Caddyfile | tr -d '\n')
cat > /tmp/params.json <<EOF
{"commands": [
"set -e",
"echo $B64 | base64 -d > /opt/ghost/Caddyfile",
"cd /opt/ghost",
"docker compose exec -T caddy caddy validate --config /etc/caddy/Caddyfile --adapter caddyfile",
"docker compose exec -T caddy caddy reload --config /etc/caddy/Caddyfile --adapter caddyfile"
]}
EOF
CMD_ID=$(aws ssm send-command \
--instance-ids "$INSTANCE_ID" \
--document-name AWS-RunShellScript \
--comment "update Caddyfile" \
--parameters file:///tmp/params.json \
--query Command.CommandId --output text)
echo "$CMD_ID"
The commands run as one shell script, so set -e at the top stops it if caddy validate fails, and a broken Caddyfile never gets reloaded. The running config stays as it was.
One thing to watch with bind mounts. The Caddyfile is mounted into the container as a single file. Writing over it with > keeps the same inode and the container sees the new content. If you write a temp file and mv it in place, the container keeps the old file and you'll be very confused why the reload did nothing.
Then poll for the result.
sleep 2
while true; do
STATUS=$(aws ssm get-command-invocation \
--command-id "$CMD_ID" --instance-id "$INSTANCE_ID" \
--query Status --output text)
case "$STATUS" in
Pending|InProgress|Delayed) sleep 3 ;;
*) break ;;
esac
done
aws ssm get-command-invocation \
--command-id "$CMD_ID" --instance-id "$INSTANCE_ID" \
--query '[Status,StandardOutputContent,StandardErrorContent]' \
--output text
The sleep 2 at the start is there because calling get-command-invocation right away sometimes returns InvocationDoesNotExist. There is also aws ssm wait command-executed, which does the same loop for you, but I like seeing stdout and stderr at the end anyway, so I kept the loop.
After that, check it from outside with curl https://fenixara.com/ads.txt.
What I'd still change
Run Command output is cut off at 24,000 characters in the API response. For a Caddyfile reload that never matters. For anything chatty, like docker compose pull, send the output to an S3 bucket with --output-s3-bucket-name or pipe it through tail.
And .env file with the database and mail passwords still lives only on the host. Moving it to Parameter Store and rendering it at boot is next on the list, but it hasn't happened yet.
That's it. Same IP, no port 22, and no more host key surprises.