To create high availability solution we can use Sentinel or we can also use Redis Cluster. Today I will show you how create Redis Cluster.
You can find more also in Redis Cluster Specification
Redis configuration example
Let's assume we have 3 servers and we want to have 6 redis instances (3 masters and slave for each master).
- 1. server - master1 (192.168.100.131:6379) and slave3 (192.168.100.131:6380)
- 2. server - master2 (192.168.100.132:6379) and slave1 (192.168.100.132:6380)
- 3. server - master3 (192.168.100.133:6379) and slave2 (192.168.100.133:6380)
Cluster configuration
We must configure conf file for all 6 running instances of redis. On each server is running two instances one for master and second for slave (this slave will replicate data from master from other server).
Edit Redis config file /etc/redis/redis-6379.conf (6379 in conf file name mean port on which is redis instance running)
sudo nano /etc/redis/redis-6379.conf
Set cluster mode (uncomment those lines):
cluster-enabled yes cluster-config-file nodes-6379.conf cluster-node-timeout 15000
Save changes
If your redis instance running as a daemon (fyi Redis instance running as background daemon) restart service:
sudo systemctl redis1 restart
Don't forget configure all redis instances like this.
Create cluster communication
Nodes in cluster need to communicate each other.
Create communication between redis servers by using meet command:
redis-cli -c -h 192.168.100.131 -p 6379 cluster meet 192.168.100.132 6379 redis-cli -c -h 192.168.100.131 -p 6379 cluster meet 192.168.100.133 6379 redis-cli -c -h 192.168.100.131 -p 6380 cluster meet 192.168.100.132 6380 redis-cli -c -h 192.168.100.131 -p 6380 cluster meet 192.168.100.133 6380 redis-cli -c -h 192.168.100.131 -p 6379 cluster meet 192.168.100.131 6380
I don't need to create meet between ...132 and ...133 because cluster nodes communicate by goosip protocol so node ...131 tell node ...132 that exists node ...133 and than communication between ...132 and ...133 will be created automaticaly
Cluster replication
Now we will configure node as a slave of specified master. More info in Cluster replicate documentation.
Before we create slave for master we will need to know node_id of master node. You can get it by cluster nodes command:
redis-cli -h 192.168.100.131 -p 6379 cluster nodes
That will return something similar to belowe and the bold string is node_id:
67ed2db8d677e59ec4a4cefb06858cf2a1a89fa1 192.168.100.131:6379 master - 0 1426238316232 2 connected 5461-10922
292f8b365bb7edb5e285caf0b7e6ddc7265d2f4f 192.168.100.132:6379 master - 0 1426238318243 3 connected 10923-16383
...
Now we can create replication:
Slave (192.168.100.132:6380) on server2 will replicate master (192.168.100.131:6379) on server1
redis-cli -c -h 192.168.100.132 -p 6380 flushall redis-cli -c -h 192.168.100.132 -p 6380 cluster replicate master_node_id_of_192.168.100.131_6379
Slave (192.168.100.133:6380) on server3 will replicate master (192.168.100.132:6379) on server2
redis-cli -c -h 192.168.100.133 -p 6380 flushall redis-cli -c -h 192.168.100.133 -p 6380 cluster replicate master_node_id_of_192.168.100.132_6379
Slave (192.168.100.131:6380) on server1 will replicate master (192.168.100.133:6379) on server3
redis-cli -c -h 192.168.100.131 -p 6380 flushall redis-cli -c -h 192.168.100.131 -p 6380 cluster replicate master_node_id_of_192.168.100.133_6379
Assign hash slots to master nodes
We must determine what hash slots will by assign to cluster node (master). Hash slots are used when client (web server) need get data from cluster node. Client must ask to correct node because data are distributed across the nodes. Redis work with 16384 hash slots (start with index 0)
for slot in {0..5461}; do redis-cli -h 192.168.100.131 -p 6379 CLUSTER ADDSLOTS $slot; done;
for slot in {5462..10922}; do redis-cli -h 192.168.100.132 -p 6379 CLUSTER ADDSLOTS $slot; done;
for slot in {10923..16383}; do redis-cli -h 192.168.100.133 -p 6379 CLUSTER ADDSLOTS $slot; done;
On client (web server) is hash slot number calculate from key by CRC16 (KEY) mod 16384. There is a lot of Redis clients for different languages that can handle it.