Subscribe

Upgrading Ghost from 4 to 6 one major version at a time with Docker

Ghost has to be upgraded to the latest minor of each major before the next one. Here is the script I used to step a Ghost 4 database through ghost:4, ghost:5 and ghost:6 containers, and the "Ghost booted" trap that made me run it twice.

Upgrading Ghost from 4 to 6 one major version at a time with Docker

My blog was on Ghost 4.1.2 until this month. The target was Ghost 6. That's two majors and a lot of minors in between, and Ghost does not let you jump. You have to be on the latest minor of a major before you move to the next one, because each version only knows how to run the database migrations from where the previous one left off.

With Ghost-CLI that means ghost update a few times on the live server. I was moving off Ghost-CLI anyway (the whole move is in the post on migrating a Ghost-CLI blog to Docker Compose), so I did it with Docker instead. Boot ghost:4, let it migrate, stop it. Then ghost:5. Then ghost:6. All against a copy of the data, in containers I could throw away.

It sounds simple. It mostly was. Except for one log line.

The version path

Docker Hub tags make this easy. ghost:4-alpine is the last 4.x release, ghost:5-alpine the last 5.x, and ghost:6-alpine the current 6.x. For me that was this path.

4.1.2  (old Ghost-CLI install)
4.48.9 (ghost:4-alpine)
5.130.6 (ghost:5-alpine)
6.75.0 (ghost:6-alpine)

You can check what a tag really contains before you trust it.

docker run --rm ghost:5-alpine node -p 'require("/var/lib/ghost/current/package.json").version'

Making the image version a variable

In compose.yml the Ghost image takes its major from an environment variable, with 6 as the default. So normal docker compose up gives Ghost 6, and the migration script can set GHOST_VERSION=4 for one run.

services:
  ghost:
    image: ghost:${GHOST_VERSION:-6}-alpine
    restart: unless-stopped
    environment:
      url: ${GHOST_URL}
      database__client: mysql
      database__connection__host: db
      database__connection__user: ${MYSQL_USER}
      database__connection__password: ${MYSQL_PASSWORD}
      database__connection__database: ${MYSQL_DATABASE}
    volumes:
      - ./data/ghost:/var/lib/ghost/content

During the migration I also load a second file, compose.migrate.yml, on top. It only changes MySQL so that Ghost 4 can log in at all. That story is in the MySQL 5.7 to 8.4 post, I won't repeat it here.

MIG="docker compose -f compose.yml -f compose.migrate.yml"
GHOST_VERSION=4 $MIG up -d --force-recreate ghost

--force-recreate matters. Without it, Compose may decide the existing container is fine and not switch images.

First attempt

My first version of the script was naive. Start the container, follow the logs, wait for "Ghost booted", stop the container, move to the next version.

That log line shows up quite early, Ghost logs it when the HTTP server is up, and on a big jump the knex migrations can still be running in the background at that point. My script saw the line, stopped the Ghost 4 container, and killed the migrations halfway.

That left a half migrated database. It was in a state no Ghost version expects, and there is no clean way to roll half done migrations back.

So I wiped data/mysql, restored the dump again and started over. The script actually refuses to run if data/mysql is not empty, which I'd added for safety and which made this rerun a deliberate step instead of an accident.

[ -z "$(ls -A data/mysql 2>/dev/null)" ] || { echo "data/mysql not empty - refusing to overwrite" >&2; exit 1; }

Honestly this was the most annoying part of the whole migration. Not hard. Just a full restore and rerun because I trusted one log line.

What "done" actually means

Ghost uses a migrations_lock table while it migrates. The row with lock_key = 'km01' has locked = 1 while migrations run and goes back to 0 when they finish. That is a much better signal than a log line.

mysql_root() {
  docker compose exec -T db sh -c 'exec mysql -uroot -p"$MYSQL_ROOT_PASSWORD" "$@" 2>/dev/null' -- "$@"
}

lock_state() {
  mysql_root -N "$MYSQL_DATABASE" -e "select locked from migrations_lock where lock_key='km01'" | tr -d '[:space:]'
}

In the fixed script, a step only counts as OK when all of these are true, and still true 15 seconds later:

  • the "Ghost booted" line is in the logs
  • migrations_lock is released
  • the container is running and has restarted zero times
  • no ERROR, MigrationsAreLockedError or DatabaseIsNotOkError in the logs

And the restart count is there because of restart: unless-stopped. If a migration crashes Ghost, Docker starts it again, and from the outside the container looks "running" the whole time. The 15 second wait catches a crash that happens right after the lock is released.

Here is the loop, a little trimmed.

boot_ghost() {
  local v=$1
  GHOST_VERSION=$v $MIG up -d --force-recreate ghost
  for i in $(seq 1 180); do
    sleep 5
    status=$(docker inspect -f '{{.State.Status}}' ghost-ghost-1)
    restarts=$(docker inspect -f '{{.RestartCount}}' ghost-ghost-1)
    if [ "$restarts" -gt 0 ] || docker logs ghost-ghost-1 2>&1 \
        | grep -q -E "MigrationsAreLockedError|DatabaseIsNotOkError|ERROR"; then
      break
    fi
    if [ "$status" = running ] && docker logs ghost-ghost-1 2>&1 | grep -q "Ghost booted" \
        && [ "$(lock_state)" = 0 ]; then
      sleep 15
      if [ "$(lock_state)" = 0 ] && [ "$(docker inspect -f '{{.RestartCount}}' ghost-ghost-1)" = 0 ]; then
        echo "ghost:$v OK"
        GHOST_VERSION=$v $MIG stop ghost
        return 0
      fi
    fi
  done
  echo "ghost:$v FAILED"
  GHOST_VERSION=$v $MIG stop ghost
  docker logs ghost-ghost-1 2>&1 | tail -60
  exit 1
}

boot_ghost 4
boot_ghost 5
boot_ghost 6

180 tries of 5 seconds is 15 minutes per step. My database was about 285 MB, so that is a lot of room, but a timeout that is too short is just the same bug again.

Logging the result of each step

On success, the real script also prints the Ghost version inside the container and the last migration that ran. It made the log of the run readable afterwards, and I kept those logs next to the dump.

docker exec ghost-ghost-1 node -p 'require("/var/lib/ghost/current/package.json").version'
mysql_root -N "$MYSQL_DATABASE" -e 'select version from migrations order by id desc limit 1'
docker logs ghost-ghost-1 2>&1 | grep -i -E "WARN|theme" | tail -10

The last line is for themes. Ghost logs theme warnings at boot, and I wanted to see those before the cutover rather than on the live site.

Things that change on the way

A few differences showed up when I compared the old and new databases. They are expected.

User id 1 becomes an ObjectId. The Ghost 5 migration does that, so any query that joins on user id 1 from the old dump will find nothing. The old tokens table is emptied. And Ghost 6 adds new built-in roles, settings and integrations, so row counts for those tables go up. Posts, tags, the post tag links and revisions stayed exactly the same, checksum for checksum.

The second run went through all three steps without a stop. After ghost:6 passed, script switched the MySQL user back to the normal auth plugin and started the stack with the plain compose.yml, which is what runs today.

If you are doing the same thing, wait for the lock, not the log line.