Standard redundant Redis solution: master/slave replication với Sentinel managing failover. Expected followed up với either a) Client support và use Sentinel discover current master hoặc b) TCP proxy front Redis pod managed by Sentinel point to master. Former là way Redis Sentinel designed, latter growing trend - và way ObjectRocket Redis configured.
Effects của failover trên commands trong Redis
Vast majority Redis operations work as expected. On failover, client code expected detect connection failed và either a) look up current master reconnect hoặc b) attempt reconnect based on scenarios above. Proxy-based scenario cũng identical Redis server restart nên applicable case single instance Redis restart.
Tuy nhiên, small handful Redis commands là long-term blocking hoặc non-atomic. PUBSUB subscribe và psubscribe commands non-atomic: command register request have messages sent to given channel(s) sent to it, nhưng actual data arrive much later - sau khi published. BL* commands long-term two-phase commands: như subscribe commands, block và not return until item available in list để removed.
Key commonality: commands issued và registration done on server, nhưng data come later. Giữa command issued và server có data reply, failovers và restarts có thể occur. Case restart hoặc failover, "registration" không còn on server. Điều này bring us to question: what happens regarding these commands in these scenarios.
Trước khi delve details, another aspect bring up sẽ come into play during testing và dramatic effects on results và how to code for failure scenarios. Có two types of failovers. Failover most consider - loss of master. Master non-responsive. Gọi "triggered failover". Second class: "initiated failover".
Initiated failover: administrator sends sentinel failover command đến Sentinel, reconfigures pod, promote master then demote original master. Surface these two scenarios appear same. Vast majority Redis commands treat identically. Nhưng long-block commands must understood in own context.
Key functional differentiator: sequence of events. Triggered failover: master not responding, no data added/modified, no messages published. Initiated failover: very brief window two masters.
Techniques và scenarios discussed here described in general và demonstrated using Python Redis library "redis-py", nhưng apply to any client library. Với background, look at how affects long-block commands. First: PUBSUB.
PUBSUB
Redis PUBSUB: PUBLISH, SUBSCRIBE, PSUBSCRIBE. First command send message, latter two register for và receive messages published via PUBLISH. First, examine PUBLISH command.
Execute PUBLISH: publish message to given channel immediately và return number clients received it. Failure scenario bring relevance như thế nào? Consider failover case. Client phải reconnect as TCP connection goes away. Nếu talk to proxy: direct reconnect get different server. Nếu Sentinel discovery: takes longer vẫn connect different server. Issue lies in timing. Nếu publisher connect new server trước subscribers, message gone - lost. But can that happen? Look to SUBSCRIBE commands.
Code issue subscribe command: server register in-memory data structures rằng specific connection should get messages on channel(s) subscribed. Information này not propagated to slaves. Khi failure happen và client reconnect, phải re-issue subscribe command. Otherwise new server không know particular connection needs receive certain messages. Clear potential subscriber reconnect new master after publisher. Conditions cho message loss exist. How minimize/prevent? Few options.
First: publisher leverage fact PUBLISH return number clients received it. Nếu zero clients received, publisher could retry. Systems static number subscribers hoặc "at least one subscriber" sufficient: prevent message loss. Systems not meet criteria: another, less robust option.
Second: control reconnect windows. Rule of thumb: publisher delay ít nhất 3x subscriber delay. Provides at least 3 opportunities subscribers reconnect prior publishing messages. Miên subscribers online first, messages không go into ether. Tuy nhiên, still possibility message loss.
Third: build lock mechanism. Most complex route: requires building/employing mechanism prevent publishers publishing until clients connect. Basic process: shared location (Zookeeper, database, Consul) subscribers register ready to receive, publishers check valid ready subscribers before publishing. Thêm complexity: subscribers "unset" themselves khi lose TCP connection.
Cần call out: all this cho triggered failover. Works vì master non-responsive thus no messages go to it. Initiated failover scenario: master continue accepting messages until no longer master hoặc disconnected. Above methods none properly fully catch this scenario.
Case initiated failovers (có thể occur during system maintenance): best buttressing employ both first và second options, accept message loss possible. How much message loss dependent on publishing schedule. Publish message average every few minutes: minuscule chance encounter loss window. Larger publishing interval: smaller risk. Conversely publish hundreds/thousands messages/second: certainly lose some. Managed proxy scenario: dependent on how quickly failover completes in proxy. ObjectRocket Redis platform: window up to 1.5 seconds.
Approach on interval, fails at limits: subscribers try reconnect at interval 1/3d publishing interval. Publish once per minute: code/configure subscribers reconnect at least every 20 seconds, publisher every minute or two. As publishing interval approaches 3 second mark: harder achieve as failover reconnect process (lookups, TCP handshake, AUTH, SUBSCRIBE) easily total few seconds.
BLPOP and friends
Question: blocking list commands operate under these conditions như thế nào? News better than publishing. Case này: modifying data structures được replicated. Furthermore, vì modifications to data structures, không có same level risk lost messages due to order of reconnect. Producer reconnect prior consumer: no change expected behavior. Producer PUSH item onto list (hoặc POP và PUSH) và once consumer connects, data there. With one exception.
Case initiated failover: consider brief multiple-master problem. Few moments, order single digit milliseconds, original master during initiated failover vẫn accept data modifications và as slave already promoted to master, modifications không replicate. Note window very small. Testing: order single digit milliseconds, nhưng there. As with publishing: corner condition odds issuing POP command to stale master rare though increases conceivable conditions khi rate producers/consumers making list modification commands reaches thousands per second.
Case worker doing BRPOPLPUSH: result item being moved "back" in original position after failover. Case BLPOP: essentially same. After failover completes, item appear re-queued. Nếu items idempotent jobs: not issue. Defensive coding cho non-idempotent jobs: determine effect of job run twice factoring frequency modifications và likelihood encountering situation. Cũng note: since only happens initiated failover, should be under operational control - advise maintenance at time systems least usage minimize/eliminate possibility.
Triggered failover: no expectation data loss. Triggered failover happens khi master unresponsive, thus no modifications made to master. Tuy nhiên, mỗi scenario có matter handling actual TCP reconnect mandatory any failover scenario.
Client Reconnections
Under either failover scenario, triggered hoặc initiated, client phải detect và reconnect - either after Sentinel lookup hoặc brief window same address. ObjectRocket Redis dùng managed proxy layer: client simply reconnect. Actual failover process up to two seconds complete. Client code should account for this. Ideally immediate retry handle network blips connection dropped somewhere along route. However accompanied back algorithm with retries account cases như server restarts (standalone Redis restarts hoặc proxy restart).
Upon reconnect, subscribers cần re-subscribe as link between request channels và new TCP connection phải established. Vì Redis PUBSUB channels created khi subscriber hoặc publisher attempt access them, no need "recreating" channel.
How to do this: highly dependent on client library và how handles disconnects. Ideal scenario: connection configurable automatic retry với ultimate failure returned nếu boundaries met. Tested couple libraries handle this: varies dramatically. Discuss common one: redis-py.
Good news: redis-py appear retry khi connection drops. Unfortunately immediately retry, too quick reliably recover connection during failover. Further no way configure this. Result: code must catch/detect failed reconnect attempt và manage reconnect yourself.
Examine standard redis-py publish và subscriber code blocks:
### Publisher example code
r.publish(channel,message)
### Subscriber example code
p = r.pubsub()
p.subscribe(channel)
for item in p.listen():
# do stuff with message (item)Straightforward. Tuy nhiên, khi immediate retry during subscriber's for-loop fails, get redis.ConnectionError exception thrown. Tricky bit: happens "inside" for item in p.listen() line. Properly catch: wrap entire for statement in try/except block. Problematic at best và leads unnecessary code complexity.
Alternative route:
### Publisher example code
p = r.pubsub()
p.subscribe(channel)
while True:
try:
message = p.get_message()
except redis.ConnectionError:
# Do reconnection attempts here such as sleeping and retrying
p = r.pubsub()
p.subscribe(channel)
if message:
# do something with the message
time.sleep(0.001) # be nice to the system :)Method này call get_message() directly cho phép catch exception at that point và re-establish 'p' object's connection. Sleep time dependent on code requirements. Nếu subscriber expects handle certain number messages và for-loop works better, vẫn work. Publishers code simpler vì generally not run on iterator.
### Publisher example code
while True:
try:
rcvd = r.publish(channel,message)
if rcvd >0: break
except redis.ConnectionError:
# handle reconnect attemptsMethod này: control over whether, when, how often retry. Critical để transparently handle failure event properly. Same thing necessary nếu code dùng Sentinel directly hoặc even Redis server restarted. As such really all publishing code should follow basic mechanism.
Blocking list commands như BLPOP: order client reconnection không strictly important as data persisted. Try/except method described above needed re-establish connection khi command execution results in "redis.ConnectionError" exception thrown.
Redis-py specifically against ObjectRocket Redis platform dùng techniques này với retry window encompassing 3 seconds: ensure no data loss expectation cho triggered failovers, và approximately 1.5 seconds potential message loss cho non-stop publishers during initiated failover như vertical resize instance.
Dù code examples specific redis-py, basic techniques should used với any client code needs handle server reconnects và uses PUBSUB commands hoặc blocking list operations. Armed with knowledge này, implement techniques dùng highly available Redis pod và know application handle failovers without waking ops team middle of night.