Designing a URL shortener from requirements to scale
Part 0 of the shorty series: designing a URL shortener before writing code. Requirements, rough numbers, random base62 codes vs counters and hashes, 301 vs 302, caching hot links in Redis and counting clicks without slowing redirects.
I wanted one small app that I could carry through a whole series of posts: write it in Go, put it in Docker, run it on ECS and Kubernetes, deploy it with CI and Terraform. A URL shortener is the classic pick, and for good reason. It is tiny, but it has a real database, a real cache and a traffic pattern that is very lopsided.
So this is part 0 of the series. The app is called shorty. In this blog we will look at the design first, before any framework or cloud gets in the way: requirements, a few rough numbers, how to make the short codes, how the redirect works, where Redis fits and what to do with click counts. Level is beginner. You only need to know what a database table and an HTTP redirect are.
I wrote the code alongside this post and ran it locally with Go 1.27, Postgres 18.7 and Redis 8.10. The Go snippets below come from that code.
Requirements
First, what shorty has to do. The list are short on purpose.
- Take a long URL and give back a short code, like
http://localhost:8080/fms9pj6. - Let the user pick their own code (
/golang) if it is free. - Visiting the short URL redirects to the long one.
- Count clicks per link.
- Delete a link.
Then the things that are not features but still matter. Redirects must be fast, because a person is waiting on a blank tab. Redirects must keep working even if something less important (the cache, the click counter) is having a bad day. And the codes should not be guessable, so nobody can walk through every link by counting up.
What I left out: user accounts, expiry dates, QR codes, a dashboard. All good ideas. None of them change the design much, and the posts get long enough as it is.
Some rough numbers
I like to put numbers on paper before choosing anything. They don't need to be right, they need to be the right size. Let us assume shorty gets popular:
New links: 100,000 per day -> 100000 / 86400 ~ 1.2 writes/s
Read:write 100 : 1 -> 10,000,000 redirects/day ~ 116 reads/s
Peak 10x average -> ~1,200 redirects/s
Row size ~500 bytes (url + code + index overhead)
5 years 100000 * 365 * 5 = 182.5 million rows ~ 91 GB
Two things jump out. Writes are tiny, one Postgres can do that while sleeping. Reads are a hundred times more, and the same popular links get hit again and again. That shape is the whole design really: make the read path cheap, and keep the write path simple.
91 GB in five years is also fine for a single Postgres. No sharding, no fancy store. I would start with one database and a cache and only change that when the numbers say so.
Making the short code
This is the part people argue about. There are three common ways.
A counter. Postgres gives every row an ID (1, 2, 3...). Encode that number in base62 (digits plus upper and lower case letters) and you get short, unique codes with no collisions at all.
const Alphabet = "0123456789ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz"
func Encode(n uint64) string {
if n == 0 {
return string(Alphabet[0])
}
var b []byte
for n > 0 {
b = append(b, Alphabet[n%62])
n /= 62
}
for i, j := 0, len(b)-1; i < j; i, j = i+1, j-1 {
b[i], b[j] = b[j], b[i]
}
return string(b)
}
Encode(125) is 21 and Encode(3521614606207) is zzzzzzz, I have both in a table test. The problem is that the codes are sequential. If my link is aB3, yours is probably aB4. Anyone can scrape every link you made, and they can also tell how many links service has. For a shortener that people paste private document links into, that is a real leak.
A hash of the URL. Take MD5 or SHA-256 of the long URL and keep the first 7 base62 characters. The same URL always gets the same code, which is neat for dedupe. But you still have to handle collisions (7 characters of a hash can clash), and two users who shorten the same URL now share one link. If one of them deletes it, the other one's link breaks.
Random. Pick 7 random base62 characters. That's 62^7, about 3.5 trillion codes. Check the database, and if the code is taken, roll again.
I went with random. It is the simplest one that doesn't leak anything.
var base = big.NewInt(int64(len(Alphabet)))
func Random(n int) (string, error) {
b := make([]byte, n)
for i := range b {
x, err := rand.Int(rand.Reader, base) // crypto/rand, not math/rand
if err != nil {
return "", err
}
b[i] = Alphabet[x.Int64()]
}
return string(b), nil
}
How often does it collide? With the 182.5 million rows from above, a new code hits an existing one with chance 182.5M / 3.5T, about 1 in 19,000. So once in a while the insert finds the code taken and we try again. Three tries in a row failing is around 1 in 10^13. I stopped worrying there.
The important bit is that the check and the insert are one statement. If you do a SELECT first and then an INSERT, two requests can pick the same code between those two steps. A unique index plus ON CONFLICT DO NOTHING lets Postgres decide:
INSERT INTO links (code, url)
VALUES ($1, $2)
ON CONFLICT (code) DO NOTHING
RETURNING id, code, url, clicks, created_at;
No row back means the code was taken, and the Go side loops:
for range maxAttempts {
c, err := shortcode.Random(shortcode.DefaultLength)
if err != nil {
return db.Link{}, err
}
l, err := s.store.CreateLink(ctx, db.CreateLinkParams{Code: c, URL: rawURL})
if errors.Is(err, pgx.ErrNoRows) {
continue // collision, try another code
}
return l, err
}
Custom codes go through the same insert, just without the loop. If golang is taken the user gets a 409. I also keep a small reserved list (links, healthz, readyz) because those words are API routes and a link with that code would never be reachable.
The redirect: 301 or 302
A redirect is a Location header and a status code. Which code matters more than it looks.
A 301 means "moved permanently". Browsers cache it, often for a very long time. The second time someone clicks, the browser goes straight to the long URL without asking shorty at all. That's great for load and terrible for click counts, and you can never change where the link points for that user.
A 302 means "found, but ask me again next time". Every click comes back to us. shorty uses 302 and also sends Cache-Control: private, max-age=0 so no proxy holds on to it. If you don't care about analytics, 301 is a valid choice and saves you a lot of traffic. I wanted the clicks.
High-level picture
POST /links, DELETE /links/{code} GET /{code} (100x more)
| |
v v
+-------------------------------------------------+
| shorty (Go), N copies behind a load balancer |
| in memory: only a few seconds of click counts |
+-------------------------------------------------+
| | |
INSERT / DELETE 1. GET key (hit: done) every 5s, one transaction:
SELECT on a miss 2. miss: SET key, 24h UPDATE links SET clicks = clicks + n
| | |
v v v
+-----------+ +---------+ +-----------+
| Postgres | | Redis | | Postgres |
| (truth) | | (cache) | | (same db) |
+-----------+ +---------+ +-----------+
The Go processes keep nothing that matters in memory, except a few seconds of click counts (more on that below). So you can run one or ten of them behind a load balancer. Postgres is the source of truth. Redis is only a cache, and shorty must work if Redis is gone.
Caching hot links
The read path is "code in, URL out", and the mapping almost never changes. That's the easiest thing in the world to cache. shorty does cache-aside: look in Redis, on a miss read Postgres and put the answer in Redis with a TTL of 24 hours.
func (s *Service) Resolve(ctx context.Context, code string) (string, error) {
u, ok, err := s.cache.Get(ctx, code)
if err != nil {
s.log.WarnContext(ctx, "cache get failed, using postgres", "code", code, "err", err)
}
if !ok {
l, err := s.store.GetLinkByCode(ctx, code)
if errors.Is(err, pgx.ErrNoRows) {
return "", ErrNotFound
}
if err != nil {
return "", err
}
u = l.URL
if err := s.cache.Set(ctx, code, u, s.cacheTTL); err != nil {
s.log.WarnContext(ctx, "cache set failed", "code", code, "err", err)
}
}
if s.clicks != nil {
s.clicks.Inc(code)
}
return u, nil
}
Look at the error handling. A Redis error is logged and then ignored. That's deliberate, a cache that can take your service down is worse than no cache. I have a unit test that makes the fake cache return "connection refused" and checks the redirect still works.
When I ran it, the key showed up in Redis right after the first redirect with a TTL of 86392 seconds, so about eight seconds had passed since I clicked. Nothing exciting, but it is nice to see the thing do what the diagram says.
Delete is the one place the cache can lie. shorty deletes the row first, then the Redis key. If the Redis delete fails, the old link keeps working until the TTL runs out. For a URL shortener I can live with that. For something like permissions I would not.
How much memory? If a million links are hot and each key plus URL is around 200 bytes, that's about 200 MB. I run Redis with maxmemory and allkeys-lru, so when it fills up it throws out the links nobody clicked lately. That is exactly what we want.
Counting clicks without slowing redirects
The obvious way is UPDATE links SET clicks = clicks + 1 on every redirect. At 1,200 redirects a second during a peak, that's 1,200 writes a second, mostly on the same few popular rows. Every one of them takes a row lock. The redirect waits on the write. I didn't like that for a request that should be a cache hit.
So shorty counts in memory and writes in batches. Each process keeps a map of code to count, and every few seconds it writes all of them in one transaction:
func (c *ClickCounter) Inc(code string) {
c.mu.Lock()
c.pending[code]++
c.mu.Unlock()
}
A link clicked 500 times in five seconds becomes one UPDATE ... clicks + 500. The redirect never touches Postgres for this.
The trade-off is honest though. If the process crashes hard, the last few seconds of clicks are lost. On a normal stop (SIGTERM from Docker or ECS) shorty flushes before exiting. I tested that: five redirects, then docker compose stop within the same flush window, and the row still said 5. If you need every single click (billing, say), write click events to a queue or a log table and add them up later instead. That is a bigger design and a post of its own.
When this stops being enough
With the numbers above, one Postgres and one Redis carry shorty a long way. If it grows past that, the order I would go in is roughly this:
- More shorty instances. They are stateless, so this is free.
- A read replica for the cache misses.
- Put a CDN in front for the very hottest links, with a short
max-age, and accept that clicks are counted less exactly. - Only then think about partitioning the
linkstable by code.
I wouldn't split it into services either. One small Go binary is right for this, which matches what I wrote in monolith first. The routes follow the ideas from my older post on API route design.
What comes next
The next parts build exactly this. First the HTTP API in Go with only the standard library, then Postgres with pgx and sqlc, then a Docker Compose stack with Postgres, Redis and a migration step, then a hardened container image. After that it goes to CI, ECR, ECS and Terraform. The schema migrations will run the same way I described in running goose migrations as a one-off ECS task.
If you want to try the code yourself before then, the only table you need is this one:
CREATE TABLE links (
id BIGINT GENERATED ALWAYS AS IDENTITY PRIMARY KEY,
code TEXT NOT NULL UNIQUE CHECK (length(code) BETWEEN 3 AND 32),
url TEXT NOT NULL CHECK (length(url) <= 2048),
clicks BIGINT NOT NULL DEFAULT 0,
created_at TIMESTAMPTZ NOT NULL DEFAULT now()
);
That's the whole design. A table, a cache and a counter.