Databases / 运维笔记

Redis 5 高可用集群的水平扩展

Einic Yeo · 4月13日 · 2019年 · · ·

水平扩展准备

Redis3.0以后的版本虽然有了集群功能，提供了比之前版本的哨兵模式更高的性能与可用性，但是集群的水平扩展却比较麻烦，今天就来带大家看看redi版权声明：本文遵循 CC 4.0 BY-SA 版权协议，若要转载请务必附上原文出处链接及本声明，谢谢合作！s高可用集群如何做水平扩展，原始集群(见下图)由6个节点组成。

6个节点分布在一台机器上，采用三主三从的模式，以及进行水平新增的2个节点，一主一从。

实际应用中，最好用多台机器，比如说6个节点分布到3台机器上，redis在建立集群时为自动的将主从节点进行不同机器的分配，比如说：master-8001分布在192.168.0.240这台机器上，则它的slave-8004则不会在这台机器上，这是为了如果一台机器挂掉之后，还有其他的机器上的从节点进行替换master，以达到高可用的目的。

  • 启动 
/usr/local/redis-cluster/bin && ./create-cluster start

查询启动情况 

ps -ef | grep redis
  • 建立集群 
/usr/local/redis-cluster/bin/redis-cli --cluster create --cluster-replicas 1 192.168.0.240:8001 192.168.0.240:8002 192.168.0.240:8003 192.168.0.240:8004 192.168.0.240:8005 192.168.0.240:8006

注意：如果之前redis集群给全部停掉了，这时候再建立集群时，会出现如下的情况 

[ERR] Node 192.168.0.240:8001 is not empty. Either the node already knows other nodes (check with CLUSTER NODES) or contains some key in database 0.

这个时候需要将每个节点下的这几个文件给删掉（测试情况删掉，实际应用不要删，这是备份文件以及节点信息，按实际的情况进行处理）： 

appendonly.aof  dump.rdb  nodes-8001.conf

客户端连接8001端口的redis实例 

/usr/local/redis-cluster/bin/redis-cli -c -h 192.168.0.240 -p 8001
  • 查看集群状态 
192.168.0.240:8001> cluster info
cluster_state:ok
cluster_slots_assigned:16384
cluster_slots_ok:16384
cluster_slots_pfail:0
cluster_slots_fail:0
cluster_known_nodes:6
cluster_size:3
cluster_current_epoch:6
cluster_my_epoch:1
cluster_stats_messages_ping_sent:195
cluster_stats_messages_pong_sent:192
cluster_stats_messages_sent:387
cluster_stats_messages_ping_received:187
cluster_stats_messages_pong_received:195
cluster_stats_messages_meet_received:5
cluster_stats_messages_received:387
  • 查看节点信息 
192.168.0.240:8001> cluster nodes
ff2a669fbe4d4714d3425abf68906a5ee40a7c0f 192.168.0.240:8001@18001 myself,master - 0 1555479393000 1 connected 0-5460
a8aee0cf7deb42278f56981e84b9e1bb98336199 192.168.0.240:8005@18005 slave ff2a669fbe4d4714d3425abf68906a5ee40a7c0f 0 1555479392572 5 connected
eb88e518ca04bac2c73c40c6a0b5efc1c9cd82a7 192.168.0.240:8003@18003 master - 0 1555479393584 3 connected 10923-16383
86286c31017bfb9246cd2385aebd6181722bf16e 192.168.0.240:8004@18004 slave eb88e518ca04bac2c73c40c6a0b5efc1c9cd82a7 0 1555479392876 4 connected
c4b7ddd390c3e320e37340ae030b82102c226e5b 192.168.0.240:8002@18002 master - 0 1555479392067 2 connected 5461-10922
768c8d44baf66cef73966709a8260aea66e51ec0 192.168.0.240:8006@18006 slave c4b7ddd390c3e320e37340ae030b82102c226e5b 0 1555479391865 6 connected

这时候看一下这个信息 

ff2a669fbe4d4714d3425abf68906a5ee40a7c0f 192.168.0.240:8001@18001 myself,master - 0 1555479393000 1 connected 0-5460
eb88e518ca04bac2c73c40c6a0b5efc1c9cd82a7 192.168.0.240:8003@18003 master - 0 1555479393584 3 connected 10923-16383
c4b7ddd390c3e320e37340ae030b82102c226e5b 192.168.0.240:8002@18002 master - 0 1555479392067 2 connected 5461-10922

hash槽的概念

从上图可以看出，整个集群运行正常，三个master节点和三个slave节点

  • 8001端口的实例节点存储0-5460这些hash槽
  • 8002端口的实例节点存储5461-10922这些hash槽
  • 8003端口的实例节点存储10923-16383这些hash槽

这三个master节点存储的所有hash槽组成redis集群的存储槽位，slave点是每个主节点的备份从节点，不显示存储槽位

Redis 集群中内置了 16384 个哈希槽，当需要在 Redis 集群中放置一个 key-value时，redis 先对 key 使用 crc16 算法算出一个结果，然后把结果对 16384 求余数

这样每个 key 都会对应一个编号在 0-16383 之间的哈希槽，redis 会根据节点数量大致均等的将哈希槽映射到不同的节点。

  • 使用哈希槽的好处就在于可以方便的添加或移除节点。
  • 当需要增加节点时，只需要把其他节点的某些哈希槽挪到新节点就可以了；
  • 当需要移除节点时，只需要把移除节点上的哈希槽挪到其他节点就行了；
  • 在这一点上，我们以后新增或移除节点的时候不用先停掉所有的 redis 服务。

用了哈希槽的概念版权声明：本文遵循 CC 4.0 BY-SA 版权协议，若要转载请务必附上原文出处链接及本声明，谢谢合作！，而没有用一致性哈希算法，不都是哈希么？这样做的原因是为什么呢？

Redis Cluster是自己做的crc版权声明：本文遵循 CC 4.0 BY-SA 版权协议，若要转载请务必附上原文出处链接及本声明，谢谢合作！16的简单hash算法，没有用一致性hash。Redis的作者认为它的crc16(key) mod 16384的效果已经不错了，虽然没有一致性hash灵活，但实现很简单，节点增删时处理起来也很方便。

为了动态增删节点的时候，不至于丢失数据么？

节点增删时不丢失数据和hash算法没什么关系，不丢失数据要求的是一份数据有多个副本。

还有集群总共有2的14次方，16384个哈希槽，那么每一个哈希槽中存的key 和 value是什么？

当你往Redis Cluster中加入一个Key时，会根据crc16(key) mod 16384计算这个key应该分布到哪个hash slot中，一个hash slot中会有很多key和value。你可以理解成表的分区，使用单节点时的redis时只有一个表，所有的key都放在这个表里；改用Redis Cluster以后会自动为你生成16384个分区表，你insert数据时会根据上面的简单算法来决定你的key应该存在哪个分区，每个分区里有很多key。

开始水平扩展

我们在原始集群基础上再增加一主(8007)一从(8008)，结构如下图

增加redis实例
在/usr/local/redis-cluster下创建8007和8008文件夹，并拷贝8001文件夹下的redis.conf文件到8007和8008这两个文件夹下 

[root@localhost redis-cluster]# mkdir 8007 8008
[root@localhost redis-cluster]# cp 8001/redis.conf 8007
[root@localhost redis-cluster]# cp 8001/redis.conf 8008

按之前的方法修改8007、8008中redis.conf参数，修改完成后进行启动 

/usr/local/redis-cluster/bin/redis-server /usr/local/redis-cluster/etc/8007/redis.conf
/usr/local/redis-cluster/bin/redis-server /usr/local/redis-cluster/etc/8008/redis.conf

查询启动情况： 

ps -ef | grep redis

[root@es-node1 redis-cluster]# ps aux|grep redis
redis      2829  0.2  0.6 161692 12108 ?        Ssl  13:35   0:01 ../bin/redis-server 0.0.0.0:8001 [cluster]
redis      2835  0.2  0.4 155036  8048 ?        Ssl  13:35   0:01 ../bin/redis-server 0.0.0.0:8002 [cluster]
redis      2841  0.2  0.4 155036  8044 ?        Ssl  13:35   0:01 ../bin/redis-server 0.0.0.0:8003 [cluster]
redis      2847  0.2  0.4 155036  8112 ?        Ssl  13:35   0:01 ../bin/redis-server 0.0.0.0:8004 [cluster]
redis      2853  0.2  0.4 155036  8092 ?        Ssl  13:35   0:01 ../bin/redis-server 0.0.0.0:8005 [cluster]
redis      2859  0.2  0.4 155036  8096 ?        Ssl  13:35   0:01 ../bin/redis-server 0.0.0.0:8006 [cluster]
root       2968  0.0  0.3  24044  7544 pts/0    S+   13:42   0:00 ./redis-cli -c -h 192.168.0.240 -p 8001
root       3194  0.1  0.4 152476  7836 ?        Ssl  13:46   0:00 /usr/local/redis-cluster/bin/redis-server 0.0.0.0:8007 [cluster]                      
root       3219  0.6  0.4 152476  7824 ?        Ssl  13:48   0:00 /usr/local/redis-cluster/bin/redis-server 0.0.0.0:8008 [cluster]                      
root       3233  0.0  0.0 103292   900 pts/2    S+   13:48   0:00 grep --color redis

这时候客户端连接8001端口的redis实例，查看节点信息，会发现并无80版权声明：本文遵循 CC 4.0 BY-SA 版权协议，若要转载请务必附上原文出处链接及本声明，谢谢合作！07、8008节点信息 

192.168.0.240:8001> cluster nodes
ff2a669fbe4d4714d3425abf68906a5ee40a7c0f 192.168.0.240:8001@18001 myself,master - 0 1555480043000 1 connected 0-5460
a8aee0cf7deb42278f56981e84b9e1bb98336199 192.168.0.240:8005@18005 slave ff2a669fbe4d4714d3425abf68906a5ee40a7c0f 0 1555480043519 5 connected
eb88e518ca04bac2c73c40c6a0b5efc1c9cd82a7 192.168.0.240:8003@18003 master - 0 1555480043016 3 connected 10923-16383
86286c31017bfb9246cd2385aebd6181722bf16e 192.168.0.240:8004@18004 slave eb88e518ca04bac2c73c40c6a0b5efc1c9cd82a7 0 1555480044024 4 connected
c4b7ddd390c3e320e37340ae030b82102c226e5b 192.168.0.240:8002@18002 master - 0 1555480043519 2 connected 5461-10922
768c8d44baf66cef73966709a8260aea66e51ec0 192.168.0.240:8006@18006 slave c4b7ddd390c3e320e37340ae030b82102c226e5b 0 1555480043000 6 connected

扩展主节点

那么开始配置8007为集群主节点
使用add-node命令新增一个主节点8007(master)，8007为新增节点，8001为已知存在节点，看到日志最后有”[OK] New node added correctly”提示代表新节点加入成功 

[root@localhost 8007]# /usr/local/redis-cluster/bin/redis-cli --cluster add-node 192.168.0.240:8007 192.168.0.240:8001
>>> Adding node 192.168.0.240:8007 to cluster 192.168.0.240:8001
>>> Performing Cluster Check (using node 192.168.0.240:8001)
M: ff2a669fbe4d4714d3425abf68906a5ee40a7c0f 192.168.0.240:8001
   slots:[0-5460] (5461 slots) master
   1 additional replica(s)
S: a8aee0cf7deb42278f56981e84b9e1bb98336199 192.168.0.240:8005
   slots: (0 slots) slave
   replicates ff2a669fbe4d4714d3425abf68906a5ee40a7c0f
M: eb88e518ca04bac2c73c40c6a0b5efc1c9cd82a7 192.168.0.240:8003
   slots:[10923-16383] (5461 slots) master
   1 additional replica(s)
S: 86286c31017bfb9246cd2385aebd6181722bf16e 192.168.0.240:8004
   slots: (0 slots) slave
   replicates eb88e518ca04bac2c73c40c6a0b5efc1c9cd82a7
M: c4b7ddd390c3e320e37340ae030b82102c226e5b 192.168.0.240:8002
   slots:[5461-10922] (5462 slots) master
   1 additional replica(s)
S: 768c8d44baf66cef73966709a8260aea66e51ec0 192.168.0.240:8006
   slots: (0 slots) slave
   replicates c4b7ddd390c3e320e37340ae030b82102c226e5b
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.
>>> Send CLUSTER MEET to node 192.168.0.240:8007 to make it join the cluster.
[OK] New node added correctly.

查看集群状态,可以看到已知节点为7个，nodes中也显示了8007节点，但是现在没有hash槽分配到8007 

192.168.0.240:8001> cluster info
cluster_state:ok
cluster_slots_assigned:16384
cluster_slots_ok:16384
cluster_slots_pfail:0
cluster_slots_fail:0
cluster_known_nodes:7
cluster_size:3
cluster_current_epoch:6
cluster_my_epoch:1
cluster_stats_messages_ping_sent:150
cluster_stats_messages_pong_sent:162
cluster_stats_messages_sent:312
cluster_stats_messages_ping_received:156
cluster_stats_messages_pong_received:150
cluster_stats_messages_meet_received:6
cluster_stats_messages_received:312
192.168.0.240:8001> cluster nodes
7953c14cc80d3e88265d41bfcd55ac5a77122cca 192.168.0.240:8003@18003 master - 0 1555481547000 3 connected 10923-16383
4ce3c04b61228ffb25f71b43c6a6691bf85780e1 192.168.0.240:8001@18001 myself,master - 0 1555481548000 1 connected 0-5460
898fc934e8c82d171ee314e2202064cd5219e672 192.168.0.240:8007@18007 master - 0 1555481548680 0 connected
53e4735bc35c1baa5375eab58ad279f1a74abec7 192.168.0.240:8006@18006 slave 2eb9606aa417c269771da4ebea81f5ee1dc5df61 0 1555481548000 6 connected
c0a062244a7b8be436e1b9dd045115c5ed5cd33b 192.168.0.240:8004@18004 slave 7953c14cc80d3e88265d41bfcd55ac5a77122cca 0 1555481547672 4 connected
79bd4b3013f0ca02d27543983539c3e2a6c41c5e 192.168.0.240:8005@18005 slave 4ce3c04b61228ffb25f71b43c6a6691bf85780e1 0 1555481548000 5 connected
2eb9606aa417c269771da4ebea81f5ee1dc5df61 192.168.0.240:8002@18002 master - 0 1555481548680 2 connected 5461-10922

我们为新节点手工分配hash槽
使用redis-cli命令为8007分配hash槽，找到集群中的任意一个主节点(8001)，对其进行重新分片工作。 

[root@localhost 8007]# /usr/local/redis-cluster/bin/redis-cli --cluster reshard 192.168.0.240:8001
>>> Performing Cluster Check (using node 192.168.0.240:8001)
M: 4ce3c04b61228ffb25f71b43c6a6691bf85780e1 192.168.0.240:8001
   slots:[0-5460] (5461 slots) master
   1 additional replica(s)
M: 7953c14cc80d3e88265d41bfcd55ac5a77122cca 192.168.0.240:8003
   slots:[10923-16383] (5461 slots) master
   1 additional replica(s)
M: 898fc934e8c82d171ee314e2202064cd5219e672 192.168.0.240:8007
   slots: (0 slots) master
S: 53e4735bc35c1baa5375eab58ad279f1a74abec7 192.168.0.240:8006
   slots: (0 slots) slave
   replicates 2eb9606aa417c269771da4ebea81f5ee1dc5df61
S: c0a062244a7b8be436e1b9dd045115c5ed5cd33b 192.168.0.240:8004
   slots: (0 slots) slave
   replicates 7953c14cc80d3e88265d41bfcd55ac5a77122cca
S: 79bd4b3013f0ca02d27543983539c3e2a6c41c5e 192.168.0.240:8005
   slots: (0 slots) slave
   replicates 4ce3c04b61228ffb25f71b43c6a6691bf85780e1
M: 2eb9606aa417c269771da4ebea81f5ee1dc5df61 192.168.0.240:8002
   slots:[5461-10922] (5462 slots) master
   1 additional replica(s)
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.
How many slots do you want to move (from 1 to 16384)?

会询问要分多少个槽出来（1000）？分给哪个节点（8007） 

How many slots do you want to move (from 1 to 16384)? 1000
What is the receiving node ID? 898fc934e8c82d171ee314e2202064cd5219e672

然后有两种方式，一种是all，以将所有节点用作散列槽的源节点，一种是done，这种是你自己选择从哪个节点上拿出来节点分给8007 

Please enter all the source node IDs.
  Type 'all' to use all the nodes as source nodes for the hash slots.
  Type 'done' once you entered all the source nodes IDs.

all是随机的，比如说我们要分出1000个，则3个主节点分别拿出333个，333个，334个节点分别8007，这里我们选择done，从8001拿1000个给8007 

How many slots do you want to move (from 1 to 16384)? 1000
What is the receiving node ID? 898fc934e8c82d171ee314e2202064cd5219e672
Please enter all the source node IDs.
  Type 'all' to use all the nodes as source nodes for the hash slots.
  Type 'done' once you entered all the source nodes IDs.
Source node #1: 4ce3c04b61228ffb25f71b43c6a6691bf85780e1
Source node #2: done

Ready to move 1000 slots.
  Source nodes:
    M: 4ce3c04b61228ffb25f71b43c6a6691bf85780e1 192.168.0.240:8001
       slots:[0-5460] (5461 slots) master
       1 additional replica(s)
  Destination node:
    M: 898fc934e8c82d171ee314e2202064cd5219e672 192.168.0.240:8007
       slots: (0 slots) master
  Resharding plan:
    Moving slot 0 from 4ce3c04b61228ffb25f71b43c6a6691bf85780e1
    Moving slot 1 from 4ce3c04b61228ffb25f71b43c6a6691bf85780e1
    Moving slot 2 from 4ce3c04b61228ffb25f71b43c6a6691bf85780e1
    Moving slot 3 from 4ce3c04b61228ffb25f71b43c6a6691bf85780e1
    Moving slot 4 from 4ce3c04b61228ffb25f71b43c6a6691bf85780e1
     ............
    Moving slot 998 from 4ce3c04b61228ffb25f71b43c6a6691bf85780e1
    Moving slot 999 from 4ce3c04b61228ffb25f71b43c6a6691bf85780e1

这时候我们再看一下节点信息 

[root@localhost 8007]# /usr/local/redis-cluster/bin/redis-cli -c -h 192.168.0.240 -p 8001
192.168.0.240:8001> cluster nodes
7953c14cc80d3e88265d41bfcd55ac5a77122cca 192.168.0.240:8003@18003 master - 0 1555483177000 3 connected 10923-16383
4ce3c04b61228ffb25f71b43c6a6691bf85780e1 192.168.0.240:8001@18001 myself,master - 0 1555483176000 1 connected 1000-5460
898fc934e8c82d171ee314e2202064cd5219e672 192.168.0.240:8007@18007 master - 0 1555483178513 7 connected 0-999
53e4735bc35c1baa5375eab58ad279f1a74abec7 192.168.0.240:8006@18006 slave 2eb9606aa417c269771da4ebea81f5ee1dc5df61 0 1555483178000 6 connected
c0a062244a7b8be436e1b9dd045115c5ed5cd33b 192.168.0.240:8004@18004 slave 7953c14cc80d3e88265d41bfcd55ac5a77122cca 0 1555483177503 4 connected
79bd4b3013f0ca02d27543983539c3e2a6c41c5e 192.168.0.240:8005@18005 slave 4ce3c04b61228ffb25f71b43c6a6691bf85780e1 0 1555483178000 5 connected
2eb9606aa417c269771da4ebea81f5ee1dc5df61 192.168.0.240:8002@18002 master - 0 1555483178917 2 connected 5461-10922

可以看到0-999已经分给8007了，而8001则从1000-5460
这时候我们配置8008为8007的从节点

扩展从节点

添加从节点8008到集群中去并查看集群状态

[root@localhost 8007]# /usr/local/redis-cluster/bin/redis-cli --cluster add-node 192.168.0.240:8008 192.168.0.240:8001
>>> Adding node 192.168.0.240:8008 to cluster 192.168.0.240:8001
>>> Performing Cluster Check (using node 192.168.0.240:8001)
M: 4ce3c04b61228ffb25f71b43c6a6691bf85780e1 192.168.0.240:8001
   slots:[1000-5460] (4461 slots) master
   1 additional replica(s)
M: 7953c14cc80d3e88265d41bfcd55ac5a77122cca 192.168.0.240:8003
   slots:[10923-16383] (5461 slots) master
   1 additional replica(s)
M: 898fc934e8c82d171ee314e2202064cd5219e672 192.168.0.240:8007
   slots:[0-999] (1000 slots) master
S: 53e4735bc35c1baa5375eab58ad279f1a74abec7 192.168.0.240:8006
   slots: (0 slots) slave
   replicates 2eb9606aa417c269771da4ebea81f5ee1dc5df61
S: c0a062244a7b8be436e1b9dd045115c5ed5cd33b 192.168.0.240:8004
   slots: (0 slots) slave
   replicates 7953c14cc80d3e88265d41bfcd55ac5a77122cca
S: 79bd4b3013f0ca02d27543983539c3e2a6c41c5e 192.168.0.240:8005
   slots: (0 slots) slave
   replicates 4ce3c04b61228ffb25f71b43c6a6691bf85780e1
M: 2eb9606aa417c269771da4ebea81f5ee1dc5df61 192.168.0.240:8002
   slots:[5461-10922] (5462 slots) master
   1 additional replica(s)
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.
>>> Send CLUSTER MEET to node 192.168.0.240:8008 to make it join the cluster.
[OK] New node added correctly.

查看节点状态 

[root@localhost 8007]# /usr/local/redis-cluster/bin/redis-cli -c -h 192.168.0.240 -p 8001
192.168.0.240:8001> cluster nodes
7953c14cc80d3e88265d41bfcd55ac5a77122cca 192.168.0.240:8003@18003 master - 0 1555483459000 3 connected 10923-16383
4ce3c04b61228ffb25f71b43c6a6691bf85780e1 192.168.0.240:8001@18001 myself,master - 0 1555483456000 1 connected 1000-5460
898fc934e8c82d171ee314e2202064cd5219e672 192.168.0.240:8007@18007 master - 0 1555483458000 7 connected 0-999
53e4735bc35c1baa5375eab58ad279f1a74abec7 192.168.0.240:8006@18006 slave 2eb9606aa417c269771da4ebea81f5ee1dc5df61 0 1555483457478 6 connected
c0a062244a7b8be436e1b9dd045115c5ed5cd33b 192.168.0.240:8004@18004 slave 7953c14cc80d3e88265d41bfcd55ac5a77122cca 0 1555483457000 4 connected
79bd4b3013f0ca02d27543983539c3e2a6c41c5e 192.168.0.240:8005@18005 slave 4ce3c04b61228ffb25f71b43c6a6691bf85780e1 0 1555483457000 5 connected
dc0a3d9b8861b5024caf63575fd8abee4f657096 192.168.0.240:8008@18008 master - 0 1555483459094 0 connected
2eb9606aa417c269771da4ebea81f5ee1dc5df61 192.168.0.240:8002@18002 master - 0 1555483458000 2 connected 5461-10922

可以看到8008是一个master节点，没有被分配任何的hash槽。
我们需要执行replicate命令来指定当前节点(从节点)的主节点id为哪个,首先需要连接新加的8008节点的客户端，然后使用集群命令进行操作，把当前的8008(slave)节点指定到一个主节点下 

[root@localhost 8007]# /usr/local/redis-cluster/bin/redis-cli -c -h 192.168.0.240 -p 8008
192.168.0.240:8008> cluster nodes
898fc934e8c82d171ee314e2202064cd5219e672 192.168.0.240:8007@18007 master - 0 1555483552000 7 connected 0-999
79bd4b3013f0ca02d27543983539c3e2a6c41c5e 192.168.0.240:8005@18005 slave 4ce3c04b61228ffb25f71b43c6a6691bf85780e1 0 1555483552000 1 connected
4ce3c04b61228ffb25f71b43c6a6691bf85780e1 192.168.0.240:8001@18001 master - 0 1555483552247 1 connected 1000-5460
53e4735bc35c1baa5375eab58ad279f1a74abec7 192.168.0.240:8006@18006 slave 2eb9606aa417c269771da4ebea81f5ee1dc5df61 0 1555483553257 2 connected
c0a062244a7b8be436e1b9dd045115c5ed5cd33b 192.168.0.240:8004@18004 slave 7953c14cc80d3e88265d41bfcd55ac5a77122cca 0 1555483552550 3 connected
dc0a3d9b8861b5024caf63575fd8abee4f657096 192.168.0.240:8008@18008 myself,master - 0 1555483550000 0 connected
2eb9606aa417c269771da4ebea81f5ee1dc5df61 192.168.0.240:8002@18002 master - 0 1555483552000 2 connected 5461-10922
7953c14cc80d3e88265d41bfcd55ac5a77122cca 192.168.0.240:8003@18003 master - 0 1555483552752 3 connected 10923-16383
192.168.0.240:8008> CLUSTER REPLICATE 4ce3c04b61228ffb25f71b43c6a6691bf85780e1
OK
192.168.0.240:8008> cluster nodes
898fc934e8c82d171ee314e2202064cd5219e672 192.168.0.240:8007@18007 master - 0 1555483766000 7 connected 0-999
79bd4b3013f0ca02d27543983539c3e2a6c41c5e 192.168.0.240:8005@18005 slave 4ce3c04b61228ffb25f71b43c6a6691bf85780e1 0 1555483766222 1 connected
4ce3c04b61228ffb25f71b43c6a6691bf85780e1 192.168.0.240:8001@18001 master - 0 1555483765000 1 connected 1000-5460
53e4735bc35c1baa5375eab58ad279f1a74abec7 192.168.0.240:8006@18006 slave 2eb9606aa417c269771da4ebea81f5ee1dc5df61 0 1555483766000 2 connected
c0a062244a7b8be436e1b9dd045115c5ed5cd33b 192.168.0.240:8004@18004 slave 7953c14cc80d3e88265d41bfcd55ac5a77122cca 0 1555483767231 3 connected
dc0a3d9b8861b5024caf63575fd8abee4f657096 192.168.0.240:8008@18008 myself,slave 4ce3c04b61228ffb25f71b43c6a6691bf85780e1 0 1555483764000 0 connected
2eb9606aa417c269771da4ebea81f5ee1dc5df61 192.168.0.240:8002@18002 master - 0 1555483766528 2 connected 5461-10922
7953c14cc80d3e88265d41bfcd55ac5a77122cca 192.168.0.240:8003@18003 master - 0 1555483765712 3 connected 10923-16383

删除从节点

扩展redis集群已经实现，下面进行删除节点

删除8008从节点
用del-node删除从节点8008，指定删除节点ip和端口，以及节点id

[root@localhost 8007]# /usr/local/redis-cluster/bin/redis-cli --cluster del-node 192.168.0.240:8008 dc0a3d9b8861b5024caf63575fd8abee4f657096
>>> Removing node dc0a3d9b8861b5024caf63575fd8abee4f657096 from cluster 192.168.0.240:8008
>>> Sending CLUSTER FORGET messages to the cluster...
>>> SHUTDOWN the node.

192.168.0.240:8001> cluster nodes
7953c14cc80d3e88265d41bfcd55ac5a77122cca 192.168.0.240:8003@18003 master - 0 1555484326070 3 connected 10923-16383
4ce3c04b61228ffb25f71b43c6a6691bf85780e1 192.168.0.240:8001@18001 myself,master - 0 1555484327000 1 connected 1000-5460
898fc934e8c82d171ee314e2202064cd5219e672 192.168.0.240:8007@18007 master - 0 1555484326070 7 connected 0-999
53e4735bc35c1baa5375eab58ad279f1a74abec7 192.168.0.240:8006@18006 slave 2eb9606aa417c269771da4ebea81f5ee1dc5df61 0 1555484326000 6 connected
c0a062244a7b8be436e1b9dd045115c5ed5cd33b 192.168.0.240:8004@18004 slave 7953c14cc80d3e88265d41bfcd55ac5a77122cca 0 1555484327585 4 connected
79bd4b3013f0ca02d27543983539c3e2a6c41c5e 192.168.0.240:8005@18005 slave 898fc934e8c82d171ee314e2202064cd5219e672 0 1555484325565 7 connected
2eb9606aa417c269771da4ebea81f5ee1dc5df61 192.168.0.240:8002@18002 master - 0 1555484327000 2 connected 5461-10922

如上所示，8008这个slave节点已经移除，并且该节点的redis服务也已被停止

删除主节点

删除8007主节点
我们尝试删除之前加入的主节点8007，这个步骤相对比较麻烦一些，因为主节点的里面是有分配了hash槽的，所以我们这里必须先把8007里的hash槽放入到其他的可用主节点中去，然后再进行移除节点操作，不然会出现数据丢失问题(目前只能把master的数据迁移到一个节点上，暂时做不了平均分配功能) 

[root@localhost 8007]# /usr/local/redis-cluster/bin/redis-cli --cluster reshard 192.168.0.240:8007
>>> Performing Cluster Check (using node 192.168.0.240:8007)
M: 898fc934e8c82d171ee314e2202064cd5219e672 192.168.0.240:8007
   slots:[0-999] (1000 slots) master
   1 additional replica(s)
S: c0a062244a7b8be436e1b9dd045115c5ed5cd33b 192.168.0.240:8004
   slots: (0 slots) slave
   replicates 7953c14cc80d3e88265d41bfcd55ac5a77122cca
M: 4ce3c04b61228ffb25f71b43c6a6691bf85780e1 192.168.0.240:8001
   slots:[1000-5460] (4461 slots) master
M: 2eb9606aa417c269771da4ebea81f5ee1dc5df61 192.168.0.240:8002
   slots:[5461-10922] (5462 slots) master
   1 additional replica(s)
S: 53e4735bc35c1baa5375eab58ad279f1a74abec7 192.168.0.240:8006
   slots: (0 slots) slave
   replicates 2eb9606aa417c269771da4ebea81f5ee1dc5df61
M: 7953c14cc80d3e88265d41bfcd55ac5a77122cca 192.168.0.240:8003
   slots:[10923-16383] (5461 slots) master
   1 additional replica(s)
S: 79bd4b3013f0ca02d27543983539c3e2a6c41c5e 192.168.0.240:8005
   slots: (0 slots) slave
   replicates 898fc934e8c82d171ee314e2202064cd5219e672
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.
How many slots do you want to move (from 1 to 16384)? 1000
What is the receiving node ID? 4ce3c04b61228ffb25f71b43c6a6691bf85780e1
Please enter all the source node IDs.
  Type 'all' to use all the nodes as source nodes for the hash slots.
  Type 'done' once you entered all the source nodes IDs.
Source node #1: 898fc934e8c82d171ee314e2202064cd5219e672
Source node #2: done

Ready to move 1000 slots.
  Source nodes:
    M: 898fc934e8c82d171ee314e2202064cd5219e672 192.168.0.240:8007
       slots:[0-999] (1000 slots) master
       1 additional replica(s)
  Destination node:
    M: 4ce3c04b61228ffb25f71b43c6a6691bf85780e1 192.168.0.240:8001
       slots:[1000-5460] (4461 slots) master
  Resharding plan:
    Moving slot 0 from 898fc934e8c82d171ee314e2202064cd5219e672
    Moving slot 1 from 898fc934e8c82d171ee314e2202064cd5219e672
    Moving slot 2 from 898fc934e8c82d171ee314e2202064cd5219e672
    .........
    Moving slot 998 from 898fc934e8c82d171ee314e2202064cd5219e672
    Moving slot 999 from 898fc934e8c82d171ee314e2202064cd5219e672

已经成功的把8007主节点的数据迁移到8001上去了，我们可以看一下现在的集群状态 

[root@localhost 8007]# /usr/local/redis-cluster/bin/redis-cli -c -h 192.168.0.240 -p 8001
192.168.0.240:8001> cluster nodes
7953c14cc80d3e88265d41bfcd55ac5a77122cca 192.168.0.240:8003@18003 master - 0 1555484774000 3 connected 10923-16383
4ce3c04b61228ffb25f71b43c6a6691bf85780e1 192.168.0.240:8001@18001 myself,master - 0 1555484772000 8 connected 0-5460
898fc934e8c82d171ee314e2202064cd5219e672 192.168.0.240:8007@18007 master - 0 1555484772687 7 connected
53e4735bc35c1baa5375eab58ad279f1a74abec7 192.168.0.240:8006@18006 slave 2eb9606aa417c269771da4ebea81f5ee1dc5df61 0 1555484774708 6 connected
c0a062244a7b8be436e1b9dd045115c5ed5cd33b 192.168.0.240:8004@18004 slave 7953c14cc80d3e88265d41bfcd55ac5a77122cca 0 1555484774505 4 connected
79bd4b3013f0ca02d27543983539c3e2a6c41c5e 192.168.0.240:8005@18005 slave 4ce3c04b61228ffb25f71b43c6a6691bf85780e1 0 1555484774000 8 connected
2eb9606aa417c269771da4ebea81f5ee1dc5df61 192.168.0.240:8002@18002 master - 0 1555484774000 2 connected 5461-10922

ok,那么删除8007节点 版权声明：本文遵循 CC 4.0 BY-SA 版权协议，若要转载请务必附上原文出处链接及本声明，谢谢合作！

[root@localhost redis-cluster]# /usr/local/redis-cluster/bin/redis-cli --cluster del-node 192.168.0.240:8007 898fc934e8c82d171ee314e2202064cd5219e672
>>> Removing node 898fc934e8c82d171ee314e2202064cd5219e672 from cluster 192.168.0.240:8007
>>> Sending CLUSTER FORGET messages to the cluster...
>>> SHUTDOWN the node.

192.168.0.240:8001> cluster nodes
7953c14cc80d3e88265d41bfcd55ac5a77122cca 192.168.0.240:8003@18003 master - 0 1555484900569 3 connected 10923-16383
4ce3c04b61228ffb25f71b43c6a6691bf85780e1 192.168.0.240:8001@18001 myself,master - 0 1555484900000 8 connected 0-5460
53e4735bc35c1baa5375eab58ad279f1a74abec7 192.168.0.240:8006@18006 slave 2eb9606aa417c269771da4ebea81f5ee1dc5df61 0 1555484900872 6 connected
c0a062244a7b8be436e1b9dd045115c5ed5cd33b 192.168.0.240:8004@18004 slave 7953c14cc80d3e88265d41bfcd55ac5a77122cca 0 1555484900000 4 connected
79bd4b3013f0ca02d27543983539c3e2a6c41c5e 192.168.0.240:8005@18005 slave 4ce3c04b61228ffb25f71b43c6a6691bf85780e1 0 1555484900000 8 connected
2eb9606aa417c269771da4ebea81f5ee1dc5df61 192.168.0.240:8002@18002 master - 0 1555484899863 2 connected 5461-10922

cluster常用命令

集群(cluster)  
CLUSTER INFO 打印集群的信息  
CLUSTER NODES 列出集群当前已知的所有节点（node），以及这些节点的相关信息。   

节点(node)  
CLUSTER MEET <ip> <port> 将 ip 和 port 所指定的节点添加到集群当中，让它成为集群的一份子。  
CLUSTER FORGET <node_id> 从集群中移除 node_id 指定的节点。  
CLUSTER REPLICATE <node_id> 将当前节点设置为 node_id 指定的节点的从节点。  
CLUSTER SAVECONFIG 将节点的配置文件保存到硬盘里面。   

槽(slot)  
CLUSTER ADDSLOTS <slot> [slot ...] 将一个或多个槽（slot）指派（assign）给当前节点。  
CLUSTER DELSLOTS <slot> [slot ...] 移除一个或多个槽对当前节点的指派。  
CLUSTER FLUSHSLOTS 移除指派给当前节点的所有槽，让当前节点变成一个没有指派任何槽的节点。 
CLUSTER SETSLOT <slot> NODE <node_id> 将槽 slot 指派给 node_id 指定的节点，如果槽已经指派给另一个节点，那么先让另一个节点删除该槽>，然后再进行指派。  
CLUSTER SETSLOT <slot> MIGRATING <node_id> 将本节点的槽 slot 迁移到 node_id 指定的节点中。  
CLUSTER SETSLOT <slot> IMPORTING <node_id> 从 node_id 指定的节点中导入槽 slot 到本节点。  
CLUSTER SETSLOT <slot> STABLE 取消对槽 slot 的导入（import）或者迁移（migrate）。   

//键 (key)  
CLUSTER KEYSLOT <key> 计算键 key 应该被放置在哪个槽上。  
CLUSTER COUNTKEYSINSLOT <slot> 返回槽 slot 目前包含的键值对数量。  
CLUSTER GETKEYSINSLOT <slot> <count> 返回 count 个 slot 槽中的键。
相关文章

ClickHouse 分片副本集群部署

10月29日 · 2022年

Hadoop 高可用HA集群

12月24日 · 2021年

RabbitMQ 镜像队列集群

12月5日 · 2021年

vSphere 7 (NSX-T 3.0 VDS) with kubernetes&Tanzu cluster

7月6日 · 2020年
0 条回应
© 2017~ Infvie Envoy. All Rights Reserved.
IPV6 | GITHUB