Data and infrastructure systems

🔧Key Takeaways

1
URL shortener: hash URL → base62 encode → store mapping. Key decision: hash collision handling, analytics, expiration
2
Rate limiter: token bucket or sliding window algorithm. Distributed: Redis-based counters with TTL
3
Web crawler: BFS with URL frontier queue, politeness (respect robots.txt), deduplication (URL + content hash)
4
Key-value store: design Dynamo — consistent hashing, replication, quorum reads/writes, gossip protocol

Classic Infrastructure Design Problems

These are the most-asked system design questions because they test fundamental concepts with manageable scope. Each one touches specific trade-offs: hashing, distributed state, consistency, throttling, and crawling.

System Designs

Write: hash long URL (MD5/SHA) → take first 7 chars → base62 encode → store mapping in KV store. Handle collisions: if short URL exists, append counter and rehash.

Read: look up short URL in KV store → redirect (HTTP 301/302). 301 = permanent (browser caches), 302 = temporary (server tracks clicks).

Analytics: log each redirect with timestamp, user agent, IP → batch process for click analytics.

Scale: billions of URLs → partition by short URL hash. Read-heavy → CDN for popular links.

Advantages

•URL shortener teaches hashing and distributed KV stores
•Rate limiter teaches throttling and distributed state
•Web crawler teaches BFS, dedup, and politeness

Disadvantages

•These systems seem simple but have subtle edge cases
•Distributed rate limiting has consistency challenges
•Web crawlers face dynamic content (JS rendering) challenges

🧪 Test Your Understanding

Knowledge Check1/1

In a URL shortener, why might you use HTTP 302 instead of 301?