Khi building large scale web applications, speed là major priority. Users không muốn wait long cho responses nữa, và họ không nên phải. Nhưng một số processes tốn thời gian, và không thể làm nhanh hơn hoặc eliminate.
Message queues giúp solve problem bằng cách cung cấp additional branch cho usual request-response journey. Branch thêm này giúp ensure users get immediate responses, và time-consuming processes done on the side. Everybody goes home happy.
Bài này focus giải thích message queues là gì và cách get started bằng cách build very simple application. Bạn nên familiar với basics Node.js, và Redis installed locally hoặc cloud instance. Học cài Redis here.
Queue là gì?
Queue là data structure cho phép store entities theo order. Queues dùng first-in-first-out (FIFO) principle.
Khái niệm queues trong computer science giống queues đời thường nơi people line up get things. Bạn join queue từ back, wait till your turn, leave queue từ front sau khi attended to.
Trong computer science, khi process như API request running, và bạn cần remove certain task (như sending email) từ current flow, bạn push it to queue và continue process.
Diagram dưới illustrate lifecycle của queue:
Queue Lifecycle | https://optimalbits.github.io/bull/
Job là gì?
Job là any piece of data used on queue, usually JSON-like object.
Như demonstrated trong cover image bài này, bạn nghĩ job như mỗi person trên queue tại airport. Mỗi person hold briefcase chứa specific data, và other instructions (passports và medical papers where required) giúp khi đến lượt attended to.
New people joining queue enter từ back (as last person), và people attended từ front. Jobs cũng processed vậy, mỗi job chứa data dùng cho processing. New jobs added từ back while jobs taken out từ front.
Job Producer là gì?
Job producer là any piece of code adds job to queue. Real life, security guard airport give direction people, telling which queue join cho different purposes.
Job producer có thể exist independently của job consumer. Nghĩa là microservice setup, particular service might just concerned với adding jobs to queue, nhưng không how they're processed after.
Worker (Job Consumer) là gì?
Worker, hoặc job consumer, là process, hoặc function, execute job. Nghĩ worker như bank cashier attending people trên queue tại bank. Khi first person comes in, họ join queue as only one. Cashier call for them và queue emptied.
Cashier request specific details process transaction từ person. While cashier attends customer, four more customers could lined up. Họ remain on queue till cashier done first customer trước call next one. Same process với queue workers — they pick first job in queue, process it.
Failed Jobs là gì?
Thường times, some jobs might fail during processing.
Lý do job fail:
- Invalid hoặc missing input data: Khi data required cho job process missing, job fail. Ví dụ, job send email fail without recipient email address.
- Timeout: Job could be failed by queue mechanism nếu taking longer than usual. Có thể do issue on dependency của job hoặc else, nhưng usually bạn không want single job running forever.
- Network hoặc infrastructure problems: Problems gần như out of control, nhưng happen. Database connection error ví dụ fail job.
- Dependency issues: Sometimes job relies external resources function well. Khi these resources unavailable hoặc unsuccessful, job fail.
Khi jobs fail, configure queue mechanism retry them. Retry immediately hoặc after calculated time. Set maximum attempts, recommended. Nếu không, end up running job always fail infinitely.
Queues useful tạo robust communication channels giữa microservices. Multiple services dùng same queue. Different services tasked different problems. Khi service complete task, push job to another service có workers waiting for that job. Service pick up và do whatever needed với data.
Queues also useful offload heavy tasks từ process. Như bạn thấy bài này, time consuming task như sending email put on queue avoid slowing response time.
Queues help avoid single points of failure. Process có ability fail và can be retried best processed using queue where can be retried after a while.
How to Build a Simple Application that Uses Queues
Trong bài này, build simple project dùng Node.js và Redis. Dùng Bull library vì simplify complexities building queue system. Project có single endpoint send emails.
Create a New Node.js Project
mkdir nodejs-queue-project
cd nodejs-queue-project
npm init -y
Commands tạo folder nodejs-queue-project và package.json file. File package.json trông như:
{ "name": "nodejs-queue-project", "version": "1.0.0", "description": "", "main": "index.js", "scripts": { "test": "echo \"Error: no test specified\" && exit 1" }, "keywords": [], "author": "", "license": "ISC" }
Install Required Dependencies
npm i express @types/express @types/node body-parser ts-node ts-lint typescript nodemon nodemailer @types/nodemailer
Commands cài different packages và dependencies required project.
Sau installation, update scripts section package.json có dev command. Whole package.json trông như:
{ "name": "nodejs-queue-project", "version": "1.0.0", "description": "", "main": "index.js", "scripts": { "dev": "nodemon src/app.ts" }, "keywords": [], "author": "", "license": "ISC", "dependencies": { "@types/express": "^4.17.17", "@types/node": "^20.3.3", "@types/nodemailer": "^6.4.8", "body-parser": "^1.20.2", "express": "^4.18.2", "nodemailer": "^6.9.3", "nodemon": "^2.0.22", "ts-lint": "^4.5.1", "ts-node": "^10.9.1", "typescript": "^5.1.6" } }
File show all installed dependencies. npm run dev command run khi dùng dev script.
How to Build the Endpoint
First thing tạo folder src. Folder chứa all code files. First file là root file application — app.ts defined in package.json.
Dùng app.ts import required packages và create simple server với single endpoint send email như dưới:
import express from "express";
import bodyParser from "body-parser";
import nodemailer from "nodemailer";
const app = express();
app.use(bodyParser.json());
app.post("/send-email", async (req, res) => {
const { from, to, subject, text } = req.body;
// Use test account as this is a tutorial
const testAccount = await nodemailer.createTestAccount();
const transporter = nodemailer.createTransport({
host: "smtp.ethereal.email",
port: 587,
secure: false,
auth: { user: testAccount.user, pass: testAccount.pass },
tls: { rejectUnauthorized: false },
});
console.log("Sending mail to %s", to);
let info = await transporter.sendMail({
from, to, subject, text,
html: `${text}`,
});
console.log("Message sent: %s", info.messageId);
console.log("Preview URL: %s", nodemailer.getTestMessageUrl(info));
res.json({ message: "Email Sent", });
});
app.listen(4300, () => {
console.log("Server started at https://localhost:4300");
});
Bây giờ start server chạy npm run dev terminal. Thấy message Server started at [https://localhost:4300](https://localhost:4300).
npm run dev message
Test endpoint dùng tool như Postman:
Endpoint testing on Postman
Request mất gần 4 giây như screenshot. Rất slow cho endpoint. Xem terminal, cũng thấy URL preview email sent.
Mở link thấy email trông như nào.
Email content
How to Create the Queue
Làm process faster, email queued sent later và response sent user immediately.
Install bull library và @types library dùng create queue:
npm i bull @types/bull
Tạo new queue dùng bull easy as instantiate new Bull object với name cho queue:
// This goes at top of your file
import Bull from 'bull';
const emailQueue = new Bull("email");
Khi queue created với just queue name, nó try dùng default Redis connection URL: localhost:6379. Nếu prefer different URL, pass second object to Bull class as options object:
const emailQueue = new Bull("email", { redis: "localhost:6379", });
Đến đây, tạo simple function serve as job producer add job to queue mỗi request comes in.
type EmailType = { from: string; to: string; subject: string; text: string; };
const sendNewEmail = async (email: EmailType) => {
emailQueue.add({ ...email });
};
Function sendNewEmail accept object chứa details new email sent type EmailType. Sender email (from), recipient (to), subject, content (text). Push new job to queue.
Dùng function này thay vì send email during request. Modify endpoint:
app.post("/send-email", async (req, res) => {
const { from, to, subject, text } = req.body;
await sendNewEmail({ from, to, subject, text });
console.log("Added to queue");
res.json({ message: "Email Sent", });
});
Code simpler và process faster. Request chỉ mất ~40ms — nhanh ~100x trước.
Endpoint testing with Postman
Email added to queue. Sẽ remain on queue until processed. Job processed by same application hoặc another service (nếu microservice setup).
How to Process the Jobs
Cycle incomplete và useless nếu mails never leave queue. Tạo job consumer process jobs và clear queue.
Tạo logic function accept Job object và send email:
const processEmailQueue = async (job: Job) => {
// Use test account as this is a tutorial
const testAccount = await nodemailer.createTestAccount();
const transporter = nodemailer.createTransport({
host: "smtp.ethereal.email",
port: 587,
secure: false,
auth: { user: testAccount.user, pass: testAccount.pass },
tls: { rejectUnauthorized: false },
});
const { from, to, subject, text } = job.data;
console.log("Sending mail to %s", to);
let info = await transporter.sendMail({
from, to, subject, text,
html: `${text}`,
});
console.log("Message sent: %s", info.messageId);
console.log("Preview URL: %s", nodemailer.getTestMessageUrl(info));
return nodemailer.getTestMessageUrl(info);
};
Function accept Job object. Object có useful properties show status và data in job. Ở đây dùng property data.
Đến đây, chỉ có function. Nó không pick up jobs automatically vì không know which queue work with.
Trước connect it to queue, add few jobs to queue bằng cách send some requests. Check email jobs currently queued chạy command in redis-cli:
LRANGE bull:email:wait 0 -1
Check email waitlist, return ids waiting jobs.
Redis CLI
Tôi tạo few jobs show how workers actually work.
Giờ connect worker to queue thêm line code:
emailQueue.process(processEmailQueue);
app.ts file trông như sau sau đó:
import express from "express";
import bodyParser from "body-parser";
import nodemailer from "nodemailer";
import Bull, { Job } from "bull";
const app = express();
app.use(bodyParser.json());
const emailQueue = new Bull("email", { redis: "localhost:6379", });
type EmailType = { from: string; to: string; subject: string; text: string; };
const sendNewEmail = async (email: EmailType) => {
emailQueue.add({ ...email });
};
const processEmailQueue = async (job: Job) => {
// Use test account as this is a tutorial
const testAccount = await nodemailer.createTestAccount();
const transporter = nodemailer.createTransport({
host: "smtp.ethereal.email",
port: 587,
secure: false,
auth: { user: testAccount.user, pass: testAccount.pass },
tls: { rejectUnauthorized: false },
});
const { from, to, subject, text } = job.data;
console.log("Sending mail to %s", to);
let info = await transporter.sendMail({
from, to, subject, text,
html: `${text}`,
});
console.log("Message sent: %s", info.messageId);
console.log("Preview URL: %s", nodemailer.getTestMessageUrl(info));
};
emailQueue.process(processEmailQueue);
app.post("/send-email", async (req, res) => {
const { from, to, subject, text } = req.body;
await sendNewEmail({ from, to, subject, text });
console.log("Added to queue");
res.json({ message: "Email Sent", });
});
app.listen(4300, () => {
console.log("Server started at https://localhost:4300");
});
Sau save, server restart và immediately start sending out mails. Vì worker sees queue và begins processing immediately.
Server sending out queued emails
Bây giờ, cả producer và worker active. Mỗi new API request pushed to queue, worker immediately process trừ khi có pending jobs.
Tóm tắt
Hy vọng bài này helped bạn understand message queue là gì, cách add jobs và create processes run them, và cách dùng build better web applications. Code files dùng bài này trên GitHub.
Nếu questions hoặc relevant advice, get in touch với tôi share them.
Đọc thêm bài viết tôi hoặc follow work, connect tôi trên LinkedIn, Twitter, GitHub. Nhanh, dễ, free!
Học code miễn phí. Chương trình open source freeCodeCamp đã giúp hơn 40.000 người có việc developer. Bắt đầu ngay