CI/CD explained: from git push to production
What happens between git push and production, stage by stage: build, test, package and deploy, the difference between continuous delivery and continuous deployment, and a diagram of the pipeline shorty will use in this series.
"We need CI/CD" gets said in a lot of meetings, and people mean very different things by it. Some mean "tests run on pull requests". Some mean "every merge goes to production by itself". Those are years apart in how a team works.
In this blog we will walk through what happens between git push and code running in production, one stage at a time, using shorty, the URL shortener from the design post, as the example. Then we'll draw the pipeline shorty will use for the rest of this series and write a first version of it. Level is beginner. You should know basic git (branch, commit, push, pull request) and roughly what a Docker image is. If not, Docker from zero covers it.
Versions I used for the bits I ran: Go 1.27.3, Docker 29.8, actionlint v1.7.12.
The words first
Continuous integration (CI) means everyone merges small changes into the main branch often, and every change is built and tested automatically before and after it gets there. The point is that main always builds and always passes. The tool that runs the checks is only half of it. The other half is the habit of small, frequent merges.
Continuous delivery means every change that passes CI ends up as something you could deploy to production with one button. The image is built, it's been to staging, it's waiting. A person decides when.
Continuous deployment goes one step further: no button. If it passes, it ships.
So "CD" is two different things. A common setup is continuous deployment to staging and continuous delivery to production, and I think that's a sensible place to be. Going fully automatic to production needs good tests, good monitoring and a fast rollback, and if any of the three is weak, the button is cheap insurance.
What a pipeline actually does
A pipeline is a list of stages that run on some machine that isn't your laptop, started by a git event. For a Go service like shorty there are four stages, and the order matters.
1. Build and check
Compile it, and run the quick checks that don't need anything else running. For shorty:
go mod tidy -diff
go vet ./...
go test -race -count=1 ./...
go mod tidy -diff fails if go.mod or go.sum are out of date, so nobody merges a change that only builds on their machine. go vet catches mistakes the compiler allows. go test -race runs the unit tests with the race detector. On my Mac the three together took 4.6 seconds:
ok github.com/fenixara/shorty/internal/config 2.741s
ok github.com/fenixara/shorty/internal/httpapi 1.463s
ok github.com/fenixara/shorty/internal/links 2.331s
ok github.com/fenixara/shorty/internal/shortcode 1.895s
That's the shape you want for this stage. Fast enough that people don't learn to ignore it. Security checks belong here too, like govulncheck for known vulnerable dependencies, or project specific checks like failing the build when someone logs a password.
2. Package
Turn the code into one artifact that will never change again. For shorty that's a Docker image, built from a multi-stage Dockerfile:
docker build --build-arg VERSION=ci-test -t shorty:ci .
docker run --rm shorty:ci /shorty version
With a warm cache that took about 10 seconds, the image was 25.6 MB and it printed ci-test, so the version made it into the binary. In pipeline the tag is the git commit SHA, never latest, and the image is pushed to a registry.
The important rule of this stage: build once. Staging and production must run the exact same image, referred to by its digest. If you rebuild for production, you are shipping something you never tested.
3. Deploy to staging
Put the new image somewhere that looks like production, run the database migrations, and check it works. Migrations are their own topic. I wrote about running goose migrations as a one-off ECS task, and the idea is the same anywhere: migrate first, then roll out, and only use migrations the old code can live with.
4. Deploy to production
Same image, same steps, different target. With continuous delivery there's an approval in front of it. Afterwards, the deploy has to wait and check the new version is healthy, and roll back if it isn't. A deploy step that just fires update service and exits green is lying to you. Apps need to cooperate here too, for example by shutting down gracefully so a rollout doesn't drop requests.
shorty's pipeline
Here's the whole thing as a picture:
developer CI system environments
git push branch
|
v
pull request ------> +----------------------+
| test (every PR) | go mod tidy -diff, go vet,
| | go test -race; fail = no merge
+----------------------+
|
review + merge to main
|
v
+----------------------+
| test (on main) | same checks again
+----------------------+
|
v
+----------------------+
| build | docker build, tag = commit SHA,
| | push, output = image@digest
+----------------------+
|
v
+----------------------+ automatic +----------------------+
| staging | -------------> | staging: migrate, |
| | | roll out, wait |
+----------------------+ +----------------------+
|
v
+----------------------+ approval +----------------------+
| production | -------------> | prod: same image, |
| | (a person) | roll out or back |
+----------------------+ +----------------------+
Pull requests only get the test job. Merges to main get everything. Staging is continuous deployment, production is continuous delivery.
A first version of it
The series will use GitHub Actions for this, and GitLab CI as well later on. Here is a first version as a GitHub Actions workflow, so you can see the diagram as code:
name: pipeline
on:
pull_request:
push:
branches: [main]
permissions:
contents: read
jobs:
test:
runs-on: ubuntu-latest
timeout-minutes: 15
steps:
- uses: actions/checkout@v7
- uses: actions/setup-go@v7
with:
go-version-file: go.mod
- run: go mod tidy -diff
- run: go vet ./...
- run: go test -race -count=1 ./...
build:
needs: test
if: github.event_name == 'push'
runs-on: ubuntu-latest
permissions:
contents: read
packages: write
outputs:
image: ${{ steps.ref.outputs.image }}
steps:
- uses: actions/checkout@v7
- uses: docker/setup-buildx-action@v4
- uses: docker/login-action@v4
with:
registry: ghcr.io
username: ${{ github.actor }}
password: ${{ secrets.GITHUB_TOKEN }}
- id: push
uses: docker/build-push-action@v7
with:
context: .
push: true
tags: ghcr.io/${{ github.repository }}:${{ github.sha }}
build-args: VERSION=${{ github.sha }}
- id: ref
run: echo "image=ghcr.io/${{ github.repository }}@${{ steps.push.outputs.digest }}" >> "$GITHUB_OUTPUT"
staging:
needs: build
runs-on: ubuntu-latest
environment: staging
steps:
- uses: actions/checkout@v7
- name: Deploy
env:
KUBECONFIG_DATA: ${{ secrets.KUBECONFIG_DATA }}
IMAGE: ${{ needs.build.outputs.image }}
run: ./scripts/deploy.sh staging
production:
needs: [build, staging]
runs-on: ubuntu-latest
environment: production
steps:
- uses: actions/checkout@v7
- name: Deploy
env:
KUBECONFIG_DATA: ${{ secrets.KUBECONFIG_DATA }}
IMAGE: ${{ needs.build.outputs.image }}
run: ./scripts/deploy.sh production
The production job uses a GitHub environment called production. Add a required reviewer to that environment in the repo settings and the job pauses until someone approves. That one setting is the difference between delivery and deployment.
The deploy script points the Deployment at the new image and waits for it, and rolls back if the rollout doesn't finish:
#!/usr/bin/env bash
set -euo pipefail
ns="${1:?namespace required}"
: "${IMAGE:?IMAGE required}"
: "${KUBECONFIG_DATA:?KUBECONFIG_DATA required}"
export KUBECONFIG="$(mktemp)"
trap 'rm -f "$KUBECONFIG"' EXIT
echo "$KUBECONFIG_DATA" | base64 -d > "$KUBECONFIG"
kubectl -n "$ns" set image deploy/shorty shorty="$IMAGE"
if ! kubectl -n "$ns" rollout status deploy/shorty --timeout=180s; then
echo "rollout failed, rolling back" >&2
kubectl -n "$ns" rollout undo deploy/shorty
exit 1
fi
What I checked: actionlint passed on the workflow, bash -n on the script, and the build and test steps ran locally as shown above. I didn't push this to a real repo with a cluster behind it, so treat the deploy part as a sketch. A kubeconfig stored as a secret is the simplest thing that works, but it's a long-lived credential. Short-lived OIDC credentials are better, and that's coming up in the series.
Things that go wrong early
Flaky tests are the first problem. A test that fails one run in twenty teaches everybody to press "re-run" without reading, and then a real failure gets re-run too. Fix or delete a flaky test the same week.
Slow pipelines are the second. If the PR check takes 25 minutes, people batch up changes to wait less, and that's the opposite of CI. Caching and running jobs in parallel help, but mostly it's about keeping the first stage small.
And "it worked in staging" with a different image than prod. Already covered, build once. I'm repeating it because it's the one I see broken most.
A smaller example closer to home
My own blog has a tiny version of this. Posts are Markdown files in a repo, and a build script refuses to produce the post if it finds an em dash, an excerpt over 300 characters or a post shorter than a three minute read. Nothing reaches the publish step until that passes. It isn't fancy, but it's the same idea as stage 1: cheap checks that run every time, so the expensive step only ever sees good input.
There we have it. Next in the series, the same pipeline gets built properly step by step, first in GitHub Actions and then in GitLab CI.