Redis Basics.

Python

import redis
import traceback

_redis = redis.StrictRedis(host='1.2.3.4', port=6379, db=0)
with _redis.pipeline() as pipe:
  pipe.setex('key', 60*2, 'value')
  try:
    pipe.execute()
  except Exception as e:
    traceback.print_exc()

Cluster

Automatically shard across multiple Redis nodes.

Some degree of availability during partitions are provided:

  • automatically split dataset among multiple nodes
  • continue operations when a subset of the nodes are experiencing failures

Port

Every Redis Cluster node requires 2 TCP connections open.

  • 6379 serving the clients
  • 10000+6379 node-to-node communication channel (binary protocol)

Sharding

Redis Cluster does not use consistent hashing.

16384 hash slots in Redis Cluster. Where every key is conceptually part of an hash slot.

hash_slot = CRC16(key)%16384

Every node in a Redis Cluster is responsible for a subset of the hash slots.

Redis Cluster supports multiple key operations as long as all the keys involved all belong to the same hash slot. The user can force multiple keys to be part of the same hash slot by using a concept called hash tags.

If there is a substring between {} brackets in a key, only what is inside the string is hashed. this{foo}key and another{foo}key are guaranteed to be in the same hash slot, and can be used together in a command with multiple keys as arguments.

Master-slave model

In this model, every hash slot has 1 (the master) to N replicas (N-1 slave)

Consistency

Redis Cluster is not able to guarantee strong consistency. (means that Redis Cluster will lose writes that were acknowledged by the system to the client under specific conditions)

  1. Redis Cluster uses asynchronous replication. (writer to master -> master ack OK to client -> master propates the write to its slaves) (trade-off between performance and consistency.)

  2. Redis Cluster has support for synchronous writes, implemented via the WAIT command. (this makes losing writes a lot less likely, however, it’s possible a slave that was not able to receive the write is elected as master)

  3. Redis Cluster will lose writes during a network partition where a client is isolated with a minority of instances including at least a master (write will lost only if the partition lasts enough time for the slave node to be promoted to master in the majority side of the partition) Note that there is a maximum window to the amount of writes send to isolated master. called node timeout. After node timeout has elapsed, a master node is considered to be failing, and can be replaced by one of its replicas.

Redis Cluster Configuration

  • cluster-enabled <yes/no>
  • cluster-config-file <filename>
    • not the user editable config file, but the Redis Cluster persistent configration.
    • generated at startup by the Redis Cluster instances
  • cluster-node-timeout <milliseconds>
    • Maximum amount of time a node can be unavailable before considered as failing.
  • cluster-slave-validity-factor <factor>
  • cluster-migration-barrier <count>
  • cluster-require-full-coverage <yes/no>
    • if yes, the cluster stops accepting writes if some percentage of the key space is not covered by any node

Create Redis Cluster

for port in {7000..7005}; do
  path="cluster-x/$port"
  mkdir -p "$path"
  conf="$path/redis.conf"
  cat > "$conf" << EOF
port $port
cluster-enabled yes
cluster-config-file nodes.conf
cluster-node-timeout 5000
appendonly yes
EOF
  redis-server "$conf"
done

gem install redis
redis-trib.rb create --replicas 1 127.0.0.1:{7000..7005}

# alternate
create-cluster {start, create, stop}

Failover

Debug

DEBUG SEGFAULT

Manual Failover

  1. CLUSTER FAILOVER exec in one of the slaves whose master will be FAILOVER
  2. clients connecting to master will be stopped
  3. slave sync with master
  4. slave and master switch
  5. master redirect clients to new master

Add A New Node

Add a node as master

  1. Add an empty node
    • redis-trib.rb add-node 127.0.0.1:7006 127.0.0.1:7000
    • first check the state of the cluster before operate
    • sent a CLUSTER MEET message to the node
  2. Reshard data to the node

Add a node as replica

  • to random master ./redis-trib.rb add-node --slave 127.0.0.1:7006 127.0.0.1:7000
  • to specific master ./redis-trib.rb add-node --slave --master-id 3c3a0c74aae0b56170ccb03a76b60cfe7dc1912e 127.0.0.1:7006 127.0.0.1:7000

Remove a node

./redis-trib del-node 127.0.0.1:7000 <node-id>`

References