backend-patterns

Distributed system design patterns implemented in Go with Redis, Kafka, and ScyllaDB — benchmarked in single-instance and cluster configurations.

Tech Stack

Go 1.23 · Redis 7.2 · Apache Kafka (KRaft) · ScyllaDB

Overview

Category	Patterns	Technologies	Scale
Rate Limiter	Token Bucket, Sliding Window Log	Redis + Lua	10M requests
Application Cache	Bloom Filter, Cache Invalidation	Redis + ScyllaDB	10M requests
Pub/Sub	Realtime Dashboard, Stock Exchange	Redis PubSub, Kafka	2M participants
Streams	Hinted Handoff, Leaky Bucket	Redis Streams, Kafka	100K+ publishers

All implementations support both single-instance and cluster configurations.

Performance

Rate Limiter

Token Bucket and Sliding Window Log algorithms, each implemented as an atomic Redis Lua script — the entire rate check and update executes as a single operation, eliminating race conditions under high concurrency.

Mode	Algorithm	Requests	RPS	Memory
Single	Token Bucket	10M	3,653	~40 MB
Single	Sliding Window Log	10M	3,551	~40 MB
Cluster	Token Bucket	10M	3,806	~30 MB × N
Cluster	Sliding Window Log	10M	3,631	~30 MB × N

Redis Cluster uses asynchronous replication, so rate limits are eventually consistent across nodes. See rate limiter details for Lua scripts and cluster behavior.

Application Cache

Bloom Filter — Cache Miss Prevention

A cache miss on non-existent data falls through to the database, creating unnecessary load. A Bloom Filter identifies non-existent keys before querying, preventing these wasteful lookups. Tested with ScyllaDB as the backing store:

Requests	RPS	False Positive	False Negative	Memory
10M	16,105	0.0008%	0%	95 MB

Cache Invalidation

Cache-aside write strategy with cache invalidation — eliminates stale reads by invalidating the cache entry before updating the database. In cluster mode, uses Redis WAIT and WAITAOF to tune consistency guarantees across replicas.

See application cache details for cache strategy analysis, filter comparison, and Redis configuration tuning.

Pub/Sub

Scenario	Description	Scale	Implementations
Realtime Dashboard	Live leaderboard with concurrent score updates and rankings	2M participants	Redis · Kafka
Stock Exchange	Market price feed with trader order processing	2K traders	Redis · Kafka

Streams

Pattern	Description	Scale	Implementations
Hinted Handoff	Reliable delivery with fallback hints for unavailable consumers	100K requests	Redis · Kafka
Leaky Bucket	Rate-controlled consumption that smooths out publisher bursts	100K publishers	Redis · Kafka

Getting Started

Prerequisites

git clone git@github.com:rrrrayyyy/backend-patterns.git
cd rsk
go mod tidy

• Go 1.23+
• Redis 7.2+ (brew install redis)
• Docker & Docker Compose

Quick Start

Rate Limiter — Token Bucket, 10M requests:

redis-server --save "" --io-threads 7 --hz 100
go run rate_limiter/main.go -algorithm="token_bucket" -numOfRequest=10000000 -semaphoreSize=10000000
redis-cli shutdown

Bloom Filter — 1M requests with ScyllaDB:

docker run --pull=always --name scylladb -p "9042:9042" -d scylladb/scylla \
  && sleep 8 \
  && docker exec -it scylladb cqlsh -e "CREATE KEYSPACE example WITH replication = {'class': 'NetworkTopologyStrategy', 'replication_factor' : 1};"
go run application_cache/r_filter/main.go -algorithm=bloom_filter -requests=1000000 -hash=xxhash -bits=10000000 -position=7
docker stop $(docker ps -q -f name=scylladb) && docker rm $(docker ps -aq -f name=scylladb)

Cache Invalidation — Redis + ScyllaDB:

redis-server --save ""
docker run --name scylladb -p "9042:9042" -d scylladb/scylla --smp 1 \
  && sleep 5 \
  && docker exec -it scylladb cqlsh -e "CREATE KEYSPACE example WITH replication = {'class': 'NetworkTopologyStrategy', 'replication_factor' : 1};" > /dev/null
go run application_cache/cache_invalidation/main.go
docker stop $(docker ps -q -f name=scylladb) && docker rm $(docker ps -aq -f name=scylladb)
redis-cli shutdown

Realtime Dashboard — Redis PubSub, 2M participants:

redis-server --save "" --io-threads 7 --hz 100
go run pubsub/realtime_dashboard/redis/main.go -semaphore=200000 -participants=2000000 -k=20 -durations=20
redis-cli shutdown

Hinted Handoff — Kafka:

docker run --name kafka -p 9092:9092 -d apache/kafka-native
go run streams/hinted_handoff/kafka/main.go -requests=100000 -semaphore=1000 -bits=1000000 -position=8
docker stop $(docker ps -q -f name=kafka) && docker rm $(docker ps -aq -f name=kafka)

See each subdirectory's README for cluster configurations and additional options.

Cluster Setup

Redis Cluster — 20 nodes (10 masters + 10 replicas)

INSTANCES=20
for port in $(seq 7000 $((7000 + $INSTANCES - 1))); do
  redis-server --port $port --cluster-enabled yes --cluster-config-file nodes-$port.conf --save "" --hz 100 & sleep 0.2
done
redis-cli --cluster create $(for port in $(seq 7000 $((7000 + $INSTANCES - 1))); do echo -n "127.0.0.1:$port "; done) --cluster-replicas 1 --cluster-yes

Teardown:

for port in $(seq 7000 $((7000 + $INSTANCES - 1))); do redis-cli -p $port shutdown; done && rm -rf nodes*.conf dump.rdb appendonlydir

Kafka Cluster — KRaft mode (2 brokers + 2 controllers)

export LOCAL_IP_ADDRESS=$(ifconfig | awk '/inet / && !/127.0.0.1/ {print $2; exit}')
docker compose -f docker-compose.kafka.yml up -d --pull=always
# Kafka UI available at http://localhost:8080

Teardown:

docker compose -f docker-compose.kafka.yml down -v

ScyllaDB Cluster — 2 nodes

docker compose -f docker-compose.scylladb.yml up -d --pull=always
# Wait for nodes to join:
while true; do clear; docker exec scylladb-1 nodetool status; sleep 1; done
# Create keyspace:
docker exec -it scylladb-1 cqlsh -e "CREATE KEYSPACE IF NOT EXISTS example WITH replication = {'class': 'NetworkTopologyStrategy', 'replication_factor' : 1};"

Teardown:

docker compose -f docker-compose.scylladb.yml down -v

Project Structure

rsk/
├── rate_limiter/
│   └── main.go                  # Token Bucket & Sliding Window Log
├── application_cache/
│   ├── r_filter/
│   │   └── main.go              # Bloom Filter & R Filter comparison
│   └── cache_invalidation/
│       └── main.go              # Cache-aside with invalidation
├── pubsub/
│   ├── realtime_dashboard/
│   │   ├── redis/main.go
│   │   └── kafka/main.go
│   └── stock_exchange/
│       ├── redis/main.go
│       └── kafka/main.go
├── streams/
│   ├── hinted_handoff/
│   │   ├── redis/main.go
│   │   └── kafka/main.go
│   └── leakey_bucket/
│       ├── redis/main.go
│       └── kafka/main.go
├── docker-compose.kafka.yml     # Kafka cluster (KRaft)
└── docker-compose.scylladb.yml  # ScyllaDB cluster

Machine Spec

Benchmark results were measured on:

Hardware	OS	Processor	Memory
MacBook Pro 2019	macOS Sequoia 15.0.1	2.3 GHz 8-Core Intel Core i9	32 GB 2667 MHz DDR4

License

MIT