Redis có versatile set data structures từ simple Strings đến powerful abstractions như Redis Streams. Native data types đưa bạn xa, nhưng có use cases cần workaround. Ví dụ: requirement dùng secondary indexes trong Redis để go beyond key-based search/lookup cho richer query capabilities. Dù bạn dùng Sorted Sets, Lists, v.v. để get the job done, bạn cần factor in trade-offs.
Enter RediSearch! Available as Redis module, RediSearch cung cấp flexible search capabilities nhờ first-class secondary indexing engine. Nó offer powerful features như full-text Search, auto completion, geographical indexing, và many more.
Để demonstrate power of RediSearch, blog post này offer practical example dùng RediSearch với Azure Cache for Redis với help của Go service built using RediSearch Go client. Designed để give you set of applications let you ingest tweets real-time và query chúng flexibly dùng RediSearch.
Cụ thể, bạn sẽ learn how to:
- Work with RediSearch indexes
- Use different RediSearch data types: TEXT, NUMERIC, TAG, và others
- How to build application show RediSearch capabilities
- How to deploy service components to Azure với just a few commands
- Analyze tweet data bằng querying RediSearch
Application overview
Như mentioned, example service let you consume tweets real-time và makes them available for querying via RediSearch.
Có 2 components:
- Consumer/Indexer: Reads from Twitter Streaming API, creates index, continuously adds tweet data (in Redis HASHes) as they arrive.
- Search service: REST API cho phép search tweets dùng RediSearch query syntax.
At this point, tôi dive into how to get solution up and running để bạn see it in action. Tuy nhiên, nếu interested understanding individual components work, refer to Code walk through section below, và GitHub repo: https://github.com/abhirockzz/redisearch-tweet-analysis.
Prerequisites
- Đầu tiên, bạn cần Microsoft Azure account: get one free here!
- Service components sẽ deploy to Azure Container Instances dùng native Docker CLI commands. Capability này enabled by integration between Docker và Azure.
- Bạn cần Docker Desktop 2.3.0.5 hoặc later, cho Windows, macOS, hoặc install Docker ACI Integration CLI cho Linux. Để dùng Twitter Streaming API, bạn cũng cần Twitter developer account. Nếu chưa có, follow instructions. RediSearch in action!
Start off bằng quick-start tutorial setup Redis Enterprise tier cache trên Azure. Once finish setup, ensure bạn có Redis host name và access key handy:
Cả 2 components service available as Docker containers: Tweet indexing service và Search API service. (Nếu cần build own Docker images, dùng respective Dockerfile trên GitHub repo.)
Bạn sẽ thấy convenient deploy these to Azure Container Instances, cho phép run Docker containers on-demand trong managed, serverless Azure environment.
Deploy to Azure
File docker-compose.yml define individual components (tweets-search và tweets-indexer). Tất cả bạn cần làm là update nó replace values cho Azure Redis instance và Twitter developer account credentials. Here is the file in its entirety:
Create an Azure context:
Clone the GitHub repo:
Deploy both service components as part of a container group:
(Note that Docker Compose commands currently available in ACI context start with docker compose. That is NOT the same as docker-compose with a hyphen.)
You will see output similar to this:
Wait for services to start, you can also check Azure portal. Once both services up and running, check their respective logs:
If all goes well, tweet-consumer service should have kicked off. It will read a stream of tweets và persist them to Redis.
The moment of truth!
Time to query tweet data. Để do so, bạn access REST API in Azure Container Instances với IP address và fully qualified domain name (FQDN) (read more in Container Access). Để find IP, run docker ps và check PORTS section in output (as shown below):
Bạn now run all sorts of queries! Before diving in, here is quick idea of indexed attributes you can use in search queries:
(Note, I use curl in examples below, but highly recommend "REST Client" for VS Code)
Set base URL cho search service API:
Start simple và query all documents (using * ):
You will see output similar to this:
Notice headers Page-Size và Search-Hits: custom headers passed from application, mainly demonstrate pagination và limits. In response to "get me all documents" query, we found 12 results in Redis, but JSON body returned 10 entries. This is because default behavior of RediSearch Go API, which you can change using different query parameter, such as:
Or, ví dụ, search tweets sent from iPhone:
You may not always want all attributes in query result. Ví dụ, this is how to just get back user (Twitter screen name) và tweet text:
How about query on user name (ví dụ starting with jo):
You can also use combination of attributes in query:
How about we look for tweets with specific hashtags? Possible to use multiple hashtags (separated by |)?
Want to find out how many tweets with biden hashtag created recently? Use range query:
If you were lucky to grab some coordinates info on tweets, you can try extracting them và query on coordinates attribute:
These are just a few examples. Feel free to experiment further và try out other queries. This section in RediSearch documentation might come in handy!
Important: After you finish, don't forget to stop services và respective containers in Azure Container Instances:
Use Azure Portal to delete Azure Redis instance you had created.
Code walk through
Section này provides high-level overview of code cho individual components. Should make it easier navigate source code in GitHub repo.
Tweets consumer/indexer:
go-twitter library used để interact with Twitter.
It authenticates to Twitter Streaming API:
And listens to a stream of tweets in a separate goroutine:
Notice the go index.AddData(tweetToMap(tweet))—this is where indexing component invoked. It connects to Azure Cache for Redis:
It then drops the index (và existing documents as well) before re-creating it:
The index và associated documents dropped to allow you start with clean state, makes it easier experiment/demo. You can choose to comment out this part if you wish.
Information for each tweet stored in a HASH (named tweet:<tweet ID>) using HSET operation:
Tweets search exposes REST API to query RediSearch. All options (including query, etc.) passed in form of query parameters. Ví dụ, https://localhost:8080/search?q=@source:iphone. It extracts required query parameters:
The q parameter is mandatory. However, you can also use following parameters for search:
- fields: to specify which attributes you want to return in result, and
- offset_limit: if you want to specify offset from where you want to search và number of documents you want to include in result (by default, offset is 0 và limit is 10 – as per RediSearch Go client).
Ví dụ:
Finally, results iterated over và passed back as JSON (array of documents):
That's all for this section!
Redis Enterprise tiers on Azure Cache for Redis
Redis Enterprise available as native service on Azure in form of 2 new tiers cho Azure Cache for Redis operated và supported by Microsoft và Redis. Service này gives developers access to rich set Redis Enterprise features, including modules like RediSearch. For more information, see these resources:
- Azure Cache for Redis, Enterprise Tiers Are Now Generally Available
- Deepening Our Partnership with Microsoft to Grow Redis Enterprise in the Cloud
- Microsoft and Redis collaborate to give developers new Azure Cache for Redis capabilities
- Redis Enterprise features on Azure Cache for Redis
Conclusion
End-to-end application này demonstrates how to work with indexes, ingest real-time data create documents (tweet information) indexed by RediSearch engine và then use versatile query syntax extract insights on those tweets.
Want to understand what happens behind the scenes when you search for a topic on Redis documentation? Check out this blog post to learn how Redis site incorporated full-text search with RediSearch! Or, perhaps you're interested in exploring how to use RediSearch in a serverless application?
If you're still getting started, visit RediSearch Quick Start page.
Nếu muốn learn more về enterprise capabilities in Azure Cache for Redis, check out following resources:
- Multi Zone redundancy và how to set it up
- Private Link support for network isolation
- Choosing the right caching tier
- High availability for Enterprise tiers