Subscribe
AWS

Running goose migrations as a one-off ECS task before updating the service

Running goose up in the container start command means every new ECS task migrates on boot. Here is how I moved migrations into a one-off ECS task in GitHub Actions that must exit 0 before the service gets the new image.

Running goose migrations as a one-off ECS task before updating the service

For a long time our golang API ran its database migrations when the container started. The image had goose in it and the start command was goose up && ./app. It was simple and it worked, so nobody touched it.

The trouble shows up the day a migration goes wrong. In this blog we will look at why migrating on boot is a bad idea on ECS, and how I moved goose into a one-off ECS task that runs from GitHub Actions before the service gets the new image.

What was wrong with migrating on boot

The service ran two tasks. During a rolling deploy ECS starts the new tasks before it stops the old ones, so for a short while there were four tasks, and every new one ran goose up against the same postgres database at the same moment.

goose keeps a version table, so most of the time the late starters find nothing to do. Most of the time. The bad case is a migration that fails. The container exits, ECS starts it again, it fails again, and now you have a service that crash loops with no clean way back. Rolling back the image doesn't undo a half applied migration.

I wanted to see the race with my own eyes before writing this, so I built a small example image and started four copies of goose up against an empty postgres 18.7 database at once:

for i in 1 2 3 4; do
  docker run --rm --network demo -e DATABASE_URL="$URL" notes:dev \
    /bin/sh -c 'goose -dir ./migrations postgres "$DATABASE_URL" up; echo "task '$i' exit=$?"' &
done; wait
OK   20261119090000_create_notes.sql (2.8ms)
goose run: ERROR 20261119090000_create_notes.sql: failed to run SQL migration: ...
  ERROR: duplicate key value violates unique constraint "pg_class_relname_nsp_index" (SQLSTATE 23505)
OK   20261119093000_add_notes_author.sql (2.01ms)
task 4 exit=1
task 2 exit=0
task 3 exit=0
task 1 exit=0

One of the four lost the race on CREATE TABLE and exited with 1. With goose up && ./app that task never starts the app, ECS marks it unhealthy and replaces it. On a good day the retry finds nothing to do. On a bad day you spend the evening reading the deployment events.

So the plan became simple. Run the migration exactly once, from the new image, and only point the service at the new image when that run exits 0.

The image

The image still carries goose and the migration files. Only the start command changes. Here is the Dockerfile I used for the example, with Go 1.27.3 and goose v3.30.1:

FROM golang:1.27.3-alpine3.23 AS build
WORKDIR /src
COPY go.mod go.sum ./
RUN go mod download
COPY . .
RUN CGO_ENABLED=0 go build -ldflags="-s -w" -o /out/app .
RUN CGO_ENABLED=0 GOBIN=/out go install github.com/pressly/goose/v3/cmd/goose@v3.30.1

FROM alpine:3.23
WORKDIR /srv
COPY --from=build /out/app ./app
COPY --from=build /out/goose /usr/local/bin/goose
COPY migrations ./migrations
EXPOSE 8080
# CMD, not ENTRYPOINT: the migration task replaces this with a command override.
CMD ["./app"]

If multi-stage builds are new to you, I wrote about them in creating a multi-stage Dockerfile for a golang web service.

That last line matters more than it looks. My first version of this change had ENTRYPOINT ["./app"] and the migration task quietly started the API server instead of goose. That one deserves its own post, so I'll only say here: keep it CMD.

You can run the same thing locally that the ECS task will run:

docker run --rm -e DATABASE_URL="$URL" notes:dev \
  /bin/sh -c 'goose -dir ./migrations postgres "$DATABASE_URL" up'

Register the task definition on its own

The usual GitHub Actions setup renders a new task definition and deploys it in one step. That doesn't work here. The migration has to run the new image, and run-task overrides can change the command but not the image. So first we register the new revision by itself and keep its ARN:

aws ecs describe-task-definition --task-definition "$TASK_FAMILY" \
  --query taskDefinition --output json > current.json

jq --arg name "$CONTAINER_NAME" --arg image "$IMAGE" '
  (.containerDefinitions[] | select(.name == $name) | .image) = $image
  | del(.taskDefinitionArn, .revision, .status, .requiresAttributes,
        .compatibilities, .registeredAt, .registeredBy, .deregisteredAt)
' current.json > new.json

ARN=$(aws ecs register-task-definition --cli-input-json file://new.json \
  --query taskDefinition.taskDefinitionArn --output text)

describe-task-definition returns read-only fields that register-task-definition refuses, so the del(...) removes exactly those and keeps everything else. I like that better than building the JSON from scratch, because nothing configured on the task (secrets, ephemeral storage, runtime platform) gets dropped by accident.

Every deploy now adds a revision, which is normal for ECS. Old revisions pile up though, and they keep their contents. I covered what that means in deregistering an ECS task definition does not delete it.

Run the migration task

The database isn't reachable from the internet, so the task has to sit in the same subnets as the service. Instead of hardcoding subnet IDs in the workflow I read them from the service itself:

NETWORK=$(aws ecs describe-services \
  --cluster "$ECS_CLUSTER" --services "$ECS_SERVICE" \
  --query 'services[0].networkConfiguration' --output json)

OVERRIDES=$(jq -cn --arg name "$CONTAINER_NAME" \
  '{containerOverrides: [{name: $name,
    command: ["/bin/sh", "-c", "goose -dir ./migrations postgres \"$DATABASE_URL\" up"]}]}')

TASK_ARN=$(aws ecs run-task \
  --cluster "$ECS_CLUSTER" \
  --task-definition "$TASK_DEF_ARN" \
  --launch-type FARGATE \
  --count 1 \
  --started-by "migrate-${GITHUB_SHA:-local}" \
  --network-configuration "$NETWORK" \
  --overrides "$OVERRIDES" \
  --query 'tasks[0].taskArn' --output text)

Look at the quoting around $DATABASE_URL. The jq program is in single quotes, so the runner never expands it. The string goes to ECS as is and the shell inside the container expands it, where the task definition has already injected the secret. The CI runner never sees the database password, which is how it should be.

Poll, don't wait

My first version used aws ecs wait tasks-stopped. It looks perfect, but the waiter checks every 6 seconds for 100 attempts, so it gives up after about 10 minutes. It prints nothing while it waits and when it gives up the task is still running. The next morning I replaced it with a plain loop that prints every status change and stops the task when it runs out of time:

deadline=$(( SECONDS + TIMEOUT_SECONDS ))
last=""
while true; do
  status=$(aws ecs describe-tasks --cluster "$ECS_CLUSTER" --tasks "$TASK_ARN" \
    --query 'tasks[0].lastStatus' --output text)
  if [ "$status" != "$last" ]; then
    echo "  [$(date -u +%H:%M:%S)] $status"
    last="$status"
  fi
  [ "$status" = "STOPPED" ] && break

  if [ "$SECONDS" -ge "$deadline" ]; then
    echo "::error::migration task still $status after ${TIMEOUT_SECONDS}s, stopping it. Service not updated."
    aws ecs stop-task --cluster "$ECS_CLUSTER" --task "$TASK_ARN" \
      --reason "migration timed out in CI" >/dev/null || true
    exit 1
  fi
  sleep "$POLL_SECONDS"
done

exit_code=$(aws ecs describe-tasks --cluster "$ECS_CLUSTER" --tasks "$TASK_ARN" \
  --query 'tasks[0].containers[0].exitCode' --output text)
if [ "$exit_code" != "0" ]; then
  echo "::error::migrations failed (exit=$exit_code). Service not updated."
  exit 1
fi

The printed states are more useful than they look. A task stuck in PROVISIONING or PENDING is a network problem (no route to ECR, no free IPs in the subnet). A task stuck in RUNNING is a slow migration. Those need very different fixes and the old waiter showed them the same way, as silence. I set the timeout to 30 minutes because a long backfill shouldn't fail an otherwise healthy deploy.

I didn't want to test this against a real account so I put a fake aws script first on the PATH that plays back a list of states and an exit code. Three cases: success, a failing migration and a task stuck in PROVISIONING. All three behaved, and the stuck one called stop-task like it should.

Update the service last

Only now does the service get the new revision:

      - name: Run migrations
        env:
          TASK_DEF_ARN: ${{ steps.register.outputs.arn }}
          MIGRATION_TIMEOUT_SECONDS: "1800"
        run: scripts/run-migration.sh

      - name: Update service
        env:
          TASK_DEF_ARN: ${{ steps.register.outputs.arn }}
        run: |
          aws ecs update-service --cluster "$ECS_CLUSTER" --service "$ECS_SERVICE" \
            --task-definition "$TASK_DEF_ARN" --query service.serviceName --output text
          aws ecs wait services-stable --cluster "$ECS_CLUSTER" --services "$ECS_SERVICE"

If the migration step fails, the job stops and the old tasks keep serving traffic. Nothing to roll back. The logs of the migration task land in the same CloudWatch log group as the service, under its own task ID, so finding them is easy.

One rule comes with this setup and the team has to know it: migrations must be backwards compatible with the version that is still running. For a few minutes the old code runs against the new schema. Add a column, deploy, use it; drop the old one in a later release. That was true before too, honestly, we just got away with it more often.

What the deploy role needs

The OIDC role GitHub Actions assumes now needs ecs:RunTask, ecs:DescribeTasks, ecs:StopTask and ecs:DescribeServices on top of the usual deploy permissions. It also needs iam:PassRole for the task role and the execution role, because run-task passes them just like update-service does. If PassRole is missing, run-task fails straight away with an access denied error that names the role, so at least that one is quick to spot.

That's it. The migration runs once, from the image you are about to ship, and a bad migration now costs you a red build instead of a crash looping service.