Computer >> Hướng Dẫn Máy Tính >  >> Lập Trình >> Redis

Xây dựng Distributed Rate Limiter Scalable với Redis & Lua – Hướng dẫn Step-by-Step

Xây dựng Distributed Rate Limiter Scalable với Redis & Lua – Hướng dẫn Step-by-Step

Trong comprehensive guide này, bạn sẽ build một distributed rate limiter dùng Redis và Lua scripting để control user requests trong high-traffic environment.

Rate limiting crucial trong bất kỳ system nào để prevent abuse, manage traffic, và protect resources. Bằng cách leverage Redis và Lua, bạn sẽ build efficient, scalable rate limiting system handle large number of requests đồng thời giữ backend services safe.

Chúng ta cũng include interactive demo nơi users simulate traffic, observe rate limits enforced, và view logs blocked requests.

What You Will Learn

  • Cách build rate limiting system dùng Redis.

  • Cách dùng Lua scripts với Redis achieve atomic operations.

  • Hiểu Redis data structures cho efficient request tracking.

  • Techniques handle high traffic trong distributed system.

  • Dùng Docker simulate và scale distributed rate limiter.

Prerequisites

Trước khi bắt đầu, ensure bạn có installed:

  • Node.js (v14 hoặc cao hơn)

  • Redis

  • Docker (cho simulate distributed environment)

  • Basic understanding Node.js, Redis, và Lua scripting.

Mục lục

  • What You Will Learn

  • Prerequisites

  • Project Overview

  • Step 1: How to Set Up the Project

  • Step 2: How to Set Up Redis

  • Step 3: How to Implement the Rate Limiter with Redis and Lua

  • Step 4: How to Create the Node.js API Server

  • Step 5: How to Test the Rate Limiter

  • Step 6: How to Visualize Rate Limiting Metrics

  • Step 7: How to Deploy with Docker

  • Conclusion: What You've Learned

Project Overview

Trong tutorial này, bạn sẽ:

  1. Build rate limiter dùng Redis và Lua enforce request quotas.

  2. Dùng Lua scripts ensure atomic operations, avoid race conditions.

  3. Implement token bucket algorithm cho rate limiting.

  4. Tạo interactive demo simulate high traffic và visualize rate limiting in action.

System Architecture

Build system với components sau:

  1. API Server: Handle incoming user requests.

  2. Redis: Store request data và enforce rate limits.

  3. Lua Scripts: Ensure atomic updates Redis cho rate limiting.

  4. Docker: Simulate distributed environment với multiple instances.

Step 1: How to Set Up the Project

Setup Node.js project:

mkdir distributed-rate-limiter
cd distributed-rate-limiter
npm init -y

Cài required dependencies:

npm install express redis dotenv
  • express: Lightweight web server framework.

  • redis: Interact với Redis.

  • dotenv: Manage environment variables.

Tạo .env file với content:

REDIS_HOST=localhost
REDIS_PORT=6379
PORT=3000
RATE_LIMIT=5
TIME_WINDOW=60

Variables define Redis host, port, rate limit (số requests allowed), và time window (giây).

Step 2: How to Set Up Redis

Trước khi dive vào code, ensure Redis installed và running. Nếu chưa có Redis, dùng Docker quick setup:

docker run -p 6379:6379 --name redis-rate-limiter -d redis

Step 3: How to Implement the Rate Limiter with Redis and Lua

Để efficiently handle rate limiting, dùng token bucket algorithm. Trong algorithm này:

  1. Mỗi user có "bucket" tokens.
  2. Mỗi request consume một token.
  3. Tokens refill periodically at set rate.

Đảm bảo atomicity và avoid race conditions, dùng Lua scripting với Redis. Lua scripts trong Redis execute atomically, nghĩa là không bị interrupt bởi operations khác while running.

How to Create a Lua Script for Rate Limiting

Tạo file rate_limiter.lua:

local key = KEYS[1]
local limit = tonumber(ARGV[1])
local window = tonumber(ARGV[2])
local current = redis.call("get", key)
if current and tonumber(current) >= limit then
    return 0
else
    if current then
        redis.call("incr", key)
    else
        redis.call("set", key, 1, "EX", window)
    end
    return 1
end
  1. Inputs:

    • KEYS[1]: Redis key represent user's request count.

    • ARGV[1]: Rate limit (max allowed requests).

    • ARGV[2]: Time window (giây) cho rate limit.

  2. Logic:

    • Nếu user reached rate limit, return 0 (request blocked).
    • Nếu user within limit, increment request count hoặc set new count với expiration nếu first request.
    • Return 1 (request allowed).

Step 4: How to Create the Node.js API Server

Tạo file server.js:

require('dotenv').config();
const express = require('express');
const redis = require('redis');
const fs = require('fs');
const path = require('path');
const app = express();
const client = redis.createClient({ host: process.env.REDIS_HOST, port: process.env.REDIS_PORT });
const rateLimitScript = fs.readFileSync(path.join(__dirname, 'rate_limiter.lua'), 'utf8');
const RATE_LIMIT = parseInt(process.env.RATE_LIMIT);
const TIME_WINDOW = parseInt(process.env.TIME_WINDOW);
// Middleware for rate limiting
async function rateLimiter(req, res, next) {
    const ip = req.ip;
    try {
        const allowed = await client.eval(rateLimitScript, 1, ip, RATE_LIMIT, TIME_WINDOW);
        if (allowed === 1) {
            next();
        } else {
            res.status(429).json({ message: 'Too many requests. Please try again later.' });
        }
    } catch (err) {
        console.error('Error in rate limiter:', err);
        res.status(500).json({ message: 'Internal server error' });
    }
}
app.use(rateLimiter);
app.get('/', (req, res) => {
    res.send('Welcome to the Rate Limited API!');
});
const PORT = process.env.PORT;
app.listen(PORT, () => {
    console.log(`Server running on port ${PORT}`);
});
  1. Rate Limiter Middleware:

    • Retrieve client's IP address và check within rate limit dùng Lua script.
    • Nếu user exceeds limit, gửi 429 response.
  2. API Endpoint:

    • Root endpoint rate-limited, user chỉ access limited times trong specified window.

Step 5: How to Test the Rate Limiter

  1. Start Redis:

    docker start redis-rate-limiter
  2. Run Node.js Server:

    node server.js
  3. Simulate Requests:

    • Dùng curl hoặc Postman test rate limiter:

      curl https://localhost:3000
    • Gửi multiple requests rapidly xem rate limiting in action.

Step 6: How to Visualize Rate Limiting Metrics

Monitor rate limiting metrics như cache hits và blocked requests, thêm logging vào middleware trong server.js:

async function rateLimiter(req, res, next) {
    const ip = req.ip;
    try {
        const allowed = await client.eval(rateLimitScript, 1, ip, RATE_LIMIT, TIME_WINDOW);
        if (allowed === 1) {
            console.log(`Allowed request from ${ip}`);
            next();
        } else {
            console.log(`Blocked request from ${ip}`);
            res.status(429).json({ message: 'Too many requests. Please try again later.' });
        }
    } catch (err) {
        console.error('Error in rate limiter:', err);
        res.status(500).json({ message: 'Internal server error' });
    }
}

Step 7: How to Deploy with Docker

Containerize application chạy trong distributed environment.

Tạo Dockerfile:

FROM node:14
WORKDIR /app
COPY . .
RUN npm install
EXPOSE 3000
CMD ["node", "server.js"]

Build và Run Docker Container:

docker build -t rate-limiter .
docker run -p 3000:3000 rate-limiter

Bây giờ bạn scale rate limiter bằng cách chạy multiple instances.

Conclusion: What You've Learned

Chúc mừng! Bạn successfully built distributed rate limiter dùng Redis và Lua scripts. Throughout tutorial, bạn học cách:

  1. Implement rate limiting control user requests trong distributed system.

  2. Dùng Lua scripts trong Redis perform atomic operations.

  3. Apply token bucket algorithm manage request quotas.

  4. Monitor rate limiting metrics optimize performance.

  5. Dùng Docker simulate scalable distributed environment.

Next Steps:

  1. Add Rate Limiting by User ID: Extend system support rate limits per user.

  2. Integrate with Nginx: Dùng Nginx as reverse proxy với Redis-backed rate limiting.

  3. Deploy with Kubernetes: Scale rate limiter dùng Kubernetes cho high availability.

Happy coding!

Học code miễn phí. Chương trình open source freeCodeCamp đã giúp hơn 40.000 người có việc developer. Bắt đầu ngay