Subscribe

Graceful shutdown of a golang HTTP server on ECS Fargate behind an ALB

gin's r.Run() has no timeouts and no signal handling, so every ECS deploy cut in-flight requests. Here is a golang http.Server with sane timeouts, a short drain delay and a shutdown budget that fits inside Fargate's 30 seconds, with tests.

Graceful shutdown of a golang HTTP server on ECS Fargate behind an ALB

Our golang API started with gin's r.Run(":3000") for a long time. That one line has no timeouts and no signal handling. When ECS stops a task during a deploy, Fargate sends SIGTERM, the process dies on the spot and every request in flight dies with it. The ones I worried about most were payment callbacks, which are exactly the requests you don't want cut in half.

In this blog we will replace r.Run() with an http.Server that has real timeouts, waits a few seconds before it stops listening, and then gives in-flight requests a fixed budget to finish. All of it has to fit inside the 30 seconds Fargate gives you.

What r.Run actually does

r.Run(addr) in gin is a thin wrapper around http.ListenAndServe(addr, r). So it's the zero value http.Server: no ReadHeaderTimeout, no IdleTimeout, and nobody listening for SIGTERM. Go's default reaction to SIGTERM is to exit right away.

I wanted to see what the client gets, so I wrote the smallest "before" version, a handler that sleeps 8 seconds to stand in for a slow callback:

func main() {
    http.HandleFunc("/slow", func(w http.ResponseWriter, r *http.Request) {
        time.Sleep(8 * time.Second)
        fmt.Fprintln(w, "done")
    })
    http.ListenAndServe(":8081", nil)
}

Start it, call /slow, send SIGTERM a second later:

./naive & NP=$!; sleep 1
curl -sS localhost:8081/slow & sleep 1
kill -TERM $NP
curl: (52) Empty reply from server

That's what a deploy looked like for any request that was unlucky enough to be running. Behind an ALB the client sees a 502 instead.

Timeouts first

Since we're building the server by hand anyway, set the timeouts. Two of them matter more than the rest:

const (
    readHeaderTimeout = 10 * time.Second
    readTimeout       = 60 * time.Second
    writeTimeout      = 60 * time.Second
    idleTimeout       = 75 * time.Second
    maxHeaderBytes    = 1 << 20
)

func newServer(addr string, h http.Handler) *http.Server {
    return &http.Server{
        Addr:              addr,
        Handler:           h,
        ReadHeaderTimeout: readHeaderTimeout,
        ReadTimeout:       readTimeout,
        WriteTimeout:      writeTimeout,
        IdleTimeout:       idleTimeout,
        MaxHeaderBytes:    maxHeaderBytes,
    }
}

ReadHeaderTimeout is the slowloris guard. Without it a client can open a socket and send one header byte at a time forever, and each of those holds a goroutine and a file descriptor.

IdleTimeout must be higher than the ALB's idle timeout, which is 60 seconds unless you changed it. The ALB keeps connections to your targets open and reuses them. If your server closes an idle keep-alive connection first, the ALB can send a request down a socket that is just going away, and the client gets a random 502. AWS says the same thing in the ALB docs. Those random 502s are miserable to debug because nothing in your own logs shows them.

A *gin.Engine is an http.Handler, so you pass the router straight in as h. Nothing else in the gin code changes.

The shutdown sequence

Now the main part. When the context is cancelled (that's our SIGTERM), we keep serving for a short drain period, then call Shutdown with a deadline:

func serve(ctx context.Context, srv *http.Server, ln net.Listener, drain, grace time.Duration) error {
    errc := make(chan error, 1)
    go func() {
        errc <- srv.Serve(ln)
    }()

    select {
    case err := <-errc:
        // The listener died on its own. Nothing to drain.
        return err
    case <-ctx.Done():
    }

    slog.Info("stop signal received, draining", "drain", drain, "grace", grace)

    // Keep serving while the load balancer stops sending us new requests.
    time.Sleep(drain)

    sctx, cancel := context.WithTimeout(context.Background(), grace)
    defer cancel()

    if err := srv.Shutdown(sctx); err != nil {
        return err
    }

    if err := <-errc; !errors.Is(err, http.ErrServerClosed) {
        return err
    }

    slog.Info("shutdown complete")
    return nil
}

Shutdown closes the listener, closes idle connections, and waits for active requests to finish or for sctx to expire. If the deadline hits first it returns context.DeadlineExceeded, and that is worth an error log line because it means a request outlived your budget.

And main wires up the signal. If channels and select are new to you, Golang Tutorial 8: Concurrency in GoLang covers them.

ctx, stop := signal.NotifyContext(context.Background(), syscall.SIGINT, syscall.SIGTERM)
defer stop()

srv := newServer(":"+port, mux)
ln, err := net.Listen("tcp", srv.Addr)
if err != nil {
    slog.Error("listen failed", "err", err)
    os.Exit(1)
}

drain := secondsFromEnv("SHUTDOWN_DRAIN_SECONDS", defaultDrain)
grace := secondsFromEnv("SHUTDOWN_GRACE_SECONDS", defaultGrace)

if err := serve(ctx, srv, ln, drain, grace); err != nil {
    slog.Error("server stopped with error", "err", err)
    os.Exit(1)
}

I take the listener as a parameter so tests can pass 127.0.0.1:0 and get a free port. secondsFromEnv reads whole seconds and falls back to the default on anything odd; zero is allowed and switches the drain off, which is handy locally.

Why the drain delay

This is the part I had to correct while writing this post. The comment in our code said ECS deregisters the target and sends SIGTERM at the same moment. When I read the ECS task lifecycle page again, that's not quite how a normal service stop works. The task goes to DEACTIVATING first, where the target group deregistration happens, and only then to STOPPING, where the agent sends SIGTERM.

I kept the drain delay anyway, at 5 seconds. Deregistration on the ALB is not instant on every load balancer node, so a few requests can still arrive just after the target starts draining. And not every way a task stops goes through that neat order; Fargate Spot interruptions for example have their own two minute notice and the docs over the years haven't agreed about the order there. Five seconds of extra serving is cheap. A refused connection during a payment callback is not.

Here is the new version with the same test as before:

PORT=8082 ./app & AP=$!; sleep 1
curl -sS localhost:8082/slow & sleep 1
kill -TERM $AP; sleep 1
curl -sS -o /dev/null -w 'new request during drain: %{http_code}\n' localhost:8082/healthz
INFO stop signal received, draining drain=5s grace=20s
new request during drain: 200
done
INFO shutdown complete

The slow request finished after its full 8 seconds and the process exited 0. A request that arrived during the drain still got a 200.

Fitting into 30 seconds

Fargate waits stopTimeout between SIGTERM and SIGKILL. The default is 30 seconds and you can raise it to 120 in the container definition. Our drain plus grace is 5 + 20 = 25, which leaves a bit of room for the logger to flush. I put that rule in a test so nobody raises the grace to 30 without noticing:

const (
    albIdleTimeout = 60 * time.Second // AWS default, we never change it
    ecsStopTimeout = 30 * time.Second // Fargate default stopTimeout
)

func TestShutdownBudgetFits(t *testing.T) {
    if total := defaultDrain + defaultGrace; total >= ecsStopTimeout {
        t.Errorf("drain + grace = %v, does not fit in stopTimeout %v", total, ecsStopTimeout)
    }
}

func TestServerTimeouts(t *testing.T) {
    srv := newServer(":0", http.NewServeMux())
    if srv.ReadHeaderTimeout <= 0 {
        t.Error("ReadHeaderTimeout must be set")
    }
    if srv.IdleTimeout <= albIdleTimeout {
        t.Errorf("IdleTimeout %v must be above the ALB idle timeout %v", srv.IdleTimeout, albIdleTimeout)
    }
}

They look a bit silly. But they turn a comment ("must stay above 60s") into something that fails the build. If you ever change the ALB idle timeout or the stopTimeout in Terraform, change the constants in the test in the same pull request.

Testing the shutdown itself

The more useful test starts a slow request, cancels the context while it runs, and checks the response still arrives:

<-started
cancel() // the "SIGTERM"

r := <-res
if r.err != nil || r.body != "done" {
    t.Fatalf("in-flight request was cut: body=%q err=%v", r.body, r.err)
}
if err := <-done; err != nil {
    t.Fatalf("serve returned %v", err)
}

A second test uses a handler that sleeps longer than the grace and expects serve to return context.DeadlineExceeded instead of hanging. Both pass with go test -race on Go 1.27.3, and the whole package runs in under 3 seconds because the tests use 200ms drain instead of 5s.

A few things that still need care

Shutdown does not wait for hijacked connections like websockets. If you have those, register a hook with srv.RegisterOnShutdown and close them yourself.

Background goroutines (queue workers, tickers) are not covered either. They need to watch the same ctx and stop on their own.

Also check the target group's deregistration delay. It ties in with how you deploy, for us that's the pipeline from running goose migrations as a one-off ECS task before updating the service. The default is 300 seconds, and ECS waits for it during DEACTIVATING, so a deploy can feel very slow. Lower it to something just above your longest request, but don't drop it below what your handlers really need.

That's it. A few lines more than r.Run(), and deploys no longer cut requests in half.