背景
系统：CentOS 7.6
Redis Cluster
| IP | 端口 | 角色 |
| 192.168.100.11 | 6380 | Master |
| 192.168.100.11 | 6381 | slave |
| 192.168.100.12 | 6380 | Master |
| 192.168.100.12 | 6381 | Slave |
| 192.168.100.13 | 6380 | Master |
| 192.168.100.13 | 6381 | Slave |
因为某种原因导致（192.168.100.12）操作系统宕机后无法修复。
查看集群节点信息，可以看到redis cluster的3个master节点均正常
[root]# redis-cli -p 6380 -a {redis密码} -c cluster nodes
4f61cdeb684fec56a82cc5f3cc72f0def5df05c3 :0@0 slave,fail,noaddr 297a72d200be658a37658bd0ed9bda01d96af43f 1567608093527 1567608091611 9 disconnected
7d221e66c3bece129bbe013687feaea093cc3714 192.168.100.13:6381@16381 master - 0 1684128078000 11 connected 10923-16383
303ddc6030053fa499e8f5f0c25ec49d1e71c176 192.168.100.13:6380@16380 master - 0 1684128077000 10 connected 0-5460
297a72d200be658a37658bd0ed9bda01d96af43f 192.168.100.11:6380@16380 myself,master - 0 1684128076000 9 connected 5461-10922
3c2fdc0e8e7295b14aff8c6909b35c38f4939907 :0@0 slave,fail,noaddr 7d221e66c3bece129bbe013687feaea093cc3714 1567608092521 1567608091611 11 disconnected
6d250882e9a180fe234d76435fd1b9dc6aa44ab2 192.168.100.11:6381@16381 slave 303ddc6030053fa499e8f5f0c25ec49d1e71c176 0 1684128079071 10 connected处理过程
从集群中剔除ID为：“4f61cdeb684fec56a82cc5f3cc72f0def5df05c3”、“3c2fdc0e8e7295b14aff8c6909b35c38f4939907”的节点
[root]# redis-cli -p 6380 -a {redis密码} -c cluster forget 4f61cdeb684fec56a82cc5f3cc72f0def5df05c3
[root]# redis-cli -p 6380 -a {redis密码} -c cluster forget 3c2fdc0e8e7295b14aff8c6909b35c38f4939907
再次查看集群节点信息，可以看到异常的两个节点从redis cluster中剔除
[root]# redis-cli -p 6380 -a {redis密码} -c cluster nodes
7d221e66c3bece129bbe013687feaea093cc3714 192.168.100.13:6381@16381 master - 0 1684128261000 11 connected 10923-16383
303ddc6030053fa499e8f5f0c25ec49d1e71c176 192.168.100.13:6380@16380 master - 0 1684128261513 10 connected 0-5460
297a72d200be658a37658bd0ed9bda01d96af43f 192.168.100.11:6380@16380 myself,master - 0 1684128260000 9 connected 5461-10922
6d250882e9a180fe234d76435fd1b9dc6aa44ab2 192.168.100.11:6381@16381 slave 303ddc6030053fa499e8f5f0c25ec49d1e71c176 0 1684128259508 10 connected
待192.168.100.12上的redis服务重建后，重新加入节点
[root]# redis-cli -p 6380 -a {redis密码} -c
127.0.0.1:6380> cluster meet 192.168.100.12 6380
OK
127.0.0.1:6380> cluster meet 192.168.100.12 6381
OK
查看redis cluster节点信息，可以看到新增的两个节点是以master的角色加入到集群中的
10.12.31.79:6380> cluster nodes
c0f760cbfbc6314e7273bcb798d2621f932bd240 192.168.100.12:6381@16381 master - 0 1684128613906 0 connected
297a72d200be658a37658bd0ed9bda01d96af43f 192.168.100.11:6380@16380 master - 0 1684128614907 9 connected 5461-10922
303ddc6030053fa499e8f5f0c25ec49d1e71c176 192.168.100.13:6380@16380 master - 0 1684128615909 10 connected 0-5460
7d221e66c3bece129bbe013687feaea093cc3714 192.168.100.13:6381@16381 master - 0 1684128615000 11 connected 10923-16383
6d250882e9a180fe234d76435fd1b9dc6aa44ab2 192.168.100.11:6381@16381 slave 303ddc6030053fa499e8f5f0c25ec49d1e71c176 0 1684128612502 10 connected
84516161919c5b04a11b889ff8750725415902e3 192.168.100.12:6380@16380 myself,master - 0 1684128611000 12 connected
连接10.12.31.79:6380，配置为某个master的slave
10.12.31.79:6380> cluster replicate 297a72d200be658a37658bd0ed9bda01d96af43f
OK
10.12.31.79:6380> cluster nodes
c0f760cbfbc6314e7273bcb798d2621f932bd240 192.168.100.12:6381@16381 master - 0 1684128712119 0 connected
297a72d200be658a37658bd0ed9bda01d96af43f 192.168.100.11:6380@16380 master - 0 1684128711118 9 connected 5461-10922
303ddc6030053fa499e8f5f0c25ec49d1e71c176 192.168.100.13:6380@16380 master - 0 1684128713121 10 connected 0-5460
7d221e66c3bece129bbe013687feaea093cc3714 192.168.100.13:6381@16381 master - 0 1684128711000 11 connected 10923-16383
6d250882e9a180fe234d76435fd1b9dc6aa44ab2 192.168.100.11:6381@16381 slave 303ddc6030053fa499e8f5f0c25ec49d1e71c176 0 1684128711000 10 connected
84516161919c5b04a11b889ff8750725415902e3 192.168.100.12:6380@16380 myself,slave 297a72d200be658a37658bd0ed9bda01d96af43f 0 1684128709000 9 connected
分别连接10.12.31.79:6381，配置为某个master的slave
10.12.31.79:6381> cluster replicate 7d221e66c3bece129bbe013687feaea093cc3714
OK
10.12.31.79:6381> cluster nodes
c0f760cbfbc6314e7273bcb798d2621f932bd240 192.168.100.12:6381@16381 myself,slave 7d221e66c3bece129bbe013687feaea093cc3714 0 1684128760000 11 connected
6d250882e9a180fe234d76435fd1b9dc6aa44ab2 192.168.100.11:6381@16381 slave 303ddc6030053fa499e8f5f0c25ec49d1e71c176 0 1684128762000 10 connected
84516161919c5b04a11b889ff8750725415902e3 192.168.100.12:6380@16380 slave 297a72d200be658a37658bd0ed9bda01d96af43f 0 1684128760000 9 connected
297a72d200be658a37658bd0ed9bda01d96af43f 192.168.100.11:6380@16380 master - 0 1684128763000 9 connected 5461-10922
303ddc6030053fa499e8f5f0c25ec49d1e71c176 192.168.100.13:6380@16380 master - 0 1684128763957 10 connected 0-5460
7d221e66c3bece129bbe013687feaea093cc3714 192.168.100.13:6381@16381 master - 0 1684128763000 11 connected 10923-16383
后续根据节点分布可以重启master节点进行调整。