Data may inconsistent when followers can't connect to leader. #388
Description
Describe the problem
Leader should not switch when the leader data is not fully synchronized with the followers.
Replicas 3，v2.1.1
- Continuous read and write data
- 90% of the network packet loss of the simulated Leader node, lasting 60s
Then, the old leader(sample-mysql-0) becomes invalid, The new leader(sample-mysql-1) is selected in the other two nodes，and the data is inconsistent :
/ $ xenoncli cluster gtid
+------------------------------------------+----------+-------+------------------------------------------------+----------------------------------------------+
| ID | Raft | Mysql | Executed_GTID_Set | Retrieved_GTID_Set |
+------------------------------------------+----------+-------+------------------------------------------------+----------------------------------------------+
| sample-mysql-0.sample-mysql.default:8801 | INVALID | ALIVE | 04b00fc5-9061-11ec-9d22-2646bcbce8bb:1-93768, | 04b00fc5-9061-11ec-9d22-2646bcbce8bb:1-93768 |
| | | | b2c3c5d9-9060-11ec-9817-12c02189a4ce:1-107648 | |
+------------------------------------------+----------+-------+------------------------------------------------+----------------------------------------------+
| sample-mysql-2.sample-mysql.default:8801 | FOLLOWER | ALIVE | 04b00fc5-9061-11ec-9d22-2646bcbce8bb:1-93768, | 04b00fc5-9061-11ec-9d22-2646bcbce8bb:1-93768 |
| | | | b2c3c5d9-9060-11ec-9817-12c02189a4ce:1-106624 | |
+------------------------------------------+----------+-------+------------------------------------------------+----------------------------------------------+
| sample-mysql-1.sample-mysql.default:8801 | LEADER | ALIVE | 04b00fc5-9061-11ec-9d22-2646bcbce8bb:1-93768, | |
| | | | b2c3c5d9-9060-11ec-9817-12c02189a4ce:1-106624 | |
+------------------------------------------+----------+-------+------------------------------------------------+----------------------------------------------+
xenon log of old leader(sample-mysql-0):
2022/02/18 03:06:40.060491 trace.go:37: [ERROR] LEADER[ID:sample-mysql-0.sample-mysql.default:8801, V:41, E:2].send.heartbeat.to.peer[sample-mysql-2.sample-mysql.default:8801].new.client.error[dial tcp: i/o timeout]
2022/02/18 03:06:40.060554 trace.go:32: [WARNING] LEADER[ID:sample-mysql-0.sample-mysql.default:8801, V:41, E:2].heartbeat.acks.granted[1].less.than.quorums[2].lessHtAcks[1].maxLessHtAcks[5]
2022/02/18 03:06:40.060568 trace.go:37: [ERROR] LEADER[ID:sample-mysql-0.sample-mysql.default:8801, V:41, E:2].send.heartbeat.to.peer[sample-mysql-1.sample-mysql.default:8801].new.client.error[dial tcp: i/o timeout]
2022/02/18 03:06:40.361432 xenon.go:34: [ERROR] api.v1.xenon.ping.error:get.client.error[dial tcp: i/o timeout]
2022/02/18 03:06:40 http: panic serving 10.10.242.188:52766: runtime error: invalid memory address or nil pointer dereference
goroutine 1487 [running]:
net/http.(*conn).serve.func1(0xc0002161e0)
/usr/local/go/src/net/http/server.go:1804 +0x153
panic(0x87e1a0, 0xbefaf0)
/usr/local/go/src/runtime/panic.go:971 +0x499
ctl/v1.xenonPingHandler(0xc0001142f0, 0xc0001627e0, 0x993a80, 0xc0000a9f08, 0xc0000a9ef0)
/go/src/github.com/radondb/xenon/src/ctl/v1/xenon.go:37 +0x11c
ctl/v1.XenonPingHandler.func1(0x993a80, 0xc0000a9f08, 0xc0000a9ef0)
/go/src/github.com/radondb/xenon/src/ctl/v1/xenon.go:25 +0x51
vendor/github.com/ant0ine/go-json-rest/rest.(*router).AppFunc.func1(0x993a80, 0xc0000a9f08, 0xc0000a9ef0)
/go/src/github.com/radondb/xenon/src/vendor/github.com/ant0ine/go-json-rest/rest/router.go:56 +0x99
vendor/github.com/ant0ine/go-json-rest/rest.(*AuthBasicMiddleware).MiddlewareFunc.func2(0x993a80, 0xc0000a9f08, 0xc0000a9ef0)
/go/src/github.com/radondb/xenon/src/vendor/github.com/ant0ine/go-json-rest/rest/auth_basic.go:73 +0x1cf
vendor/github.com/ant0ine/go-json-rest/rest.adapterFunc.func1(0x992c88, 0xc0005348c0, 0xc0001ce200)
/go/src/github.com/radondb/xenon/src/vendor/github.com/ant0ine/go-json-rest/rest/middleware.go:70 +0xd3
net/http.HandlerFunc.ServeHTTP(0xc000058460, 0x992c88, 0xc0005348c0, 0xc0001ce200)
/usr/local/go/src/net/http/server.go:2049 +0x44
net/http.serverHandler.ServeHTTP(0xc0000760e0, 0x992c88, 0xc0005348c0, 0xc0001ce200)
/usr/local/go/src/net/http/server.go:2867 +0xa3
net/http.(*conn).serve(0xc0002161e0, 0x9937e0, 0xc00020c700)
/usr/local/go/src/net/http/server.go:1932 +0x8cd
created by net/http.(*Server).Serve
/usr/local/go/src/net/http/server.go:2993 +0x39b
2022/02/18 03:06:42.062034 trace.go:37: [ERROR] LEADER[ID:sample-mysql-0.sample-mysql.default:8801, V:41, E:2].send.heartbeat.to.peer[sample-mysql-1.sample-mysql.default:8801].new.client.error[dial tcp: i/o timeout]
2022/02/18 03:06:42.062048 trace.go:37: [ERROR] LEADER[ID:sample-mysql-0.sample-mysql.default:8801, V:41, E:2].send.heartbeat.to.peer[sample-mysql-2.sample-mysql.default:8801].new.client.error[dial tcp: i/o timeout]
2022/02/18 03:06:42.062186 trace.go:32: [WARNING] LEADER[ID:sample-mysql-0.sample-mysql.default:8801, V:41, E:2].heartbeat.acks.granted[1].less.than.quorums[2].lessHtAcks[2].maxLessHtAcks[5]
2022/02/18 03:06:44.062574 trace.go:37: [ERROR] LEADER[ID:sample-mysql-0.sample-mysql.default:8801, V:41, E:2].send.heartbeat.to.peer[sample-mysql-1.sample-mysql.default:8801].new.client.error[dial tcp: i/o timeout]
2022/02/18 03:06:44.062622 trace.go:32: [WARNING] LEADER[ID:sample-mysql-0.sample-mysql.default:8801, V:41, E:2].heartbeat.acks.granted[1].less.than.quorums[2].lessHtAcks[3].maxLessHtAcks[5]
2022/02/18 03:06:44.062647 trace.go:37: [ERROR] LEADER[ID:sample-mysql-0.sample-mysql.default:8801, V:41, E:2].send.heartbeat.to.peer[sample-mysql-2.sample-mysql.default:8801].new.client.error[dial tcp: i/o timeout]
2022/02/18 03:06:44.386478 api.go:285: [INFO] mysql.slave.status:&{ 0 false false }
2022/02/18 03:06:44.386712 api.go:295: [INFO] mysql.master.status:&{mysql-bin.000005 1072490484 true true b2c3c5d9-9060-11ec-9817-12c02189a4ce:1-106624 0 }
2022/02/18 03:06:46.063395 trace.go:37: [ERROR] LEADER[ID:sample-mysql-0.sample-mysql.default:8801, V:41, E:2].send.heartbeat.to.peer[sample-mysql-2.sample-mysql.default:8801].new.client.error[dial tcp: i/o timeout]
2022/02/18 03:06:46.063402 trace.go:32: [WARNING] LEADER[ID:sample-mysql-0.sample-mysql.default:8801, V:41, E:2].heartbeat.acks.granted[1].less.than.quorums[2].lessHtAcks[4].maxLessHtAcks[5]
2022/02/18 03:06:46.063453 trace.go:37: [ERROR] LEADER[ID:sample-mysql-0.sample-mysql.default:8801, V:41, E:2].send.heartbeat.to.peer[sample-mysql-1.sample-mysql.default:8801].new.client.error[dial tcp: i/o timeout]
2022/02/18 03:06:48.064198 trace.go:37: [ERROR] LEADER[ID:sample-mysql-0.sample-mysql.default:8801, V:41, E:2].send.heartbeat.to.peer[sample-mysql-1.sample-mysql.default:8801].new.client.error[dial tcp: i/o timeout]
2022/02/18 03:06:48.064213 trace.go:37: [ERROR] LEADER[ID:sample-mysql-0.sample-mysql.default:8801, V:41, E:2].send.heartbeat.to.peer[sample-mysql-2.sample-mysql.default:8801].new.client.error[dial tcp: i/o timeout]
2022/02/18 03:06:48.064227 trace.go:32: [WARNING] LEADER[ID:sample-mysql-0.sample-mysql.default:8801, V:41, E:2].heartbeat.acks.granted[1].less.than.quorums[2].lessHtAcks[5].maxLessHtAcks[5]
2022/02/18 03:06:48.064254 trace.go:32: [WARNING] LEADER[ID:sample-mysql-0.sample-mysql.default:8801, V:41, E:2].degrade.to.follower.lessHtAcks[5]>=maxLessHtAcks[5]
2022/02/18 03:06:48.064259 trace.go:32: [WARNING] LEADER[ID:sample-mysql-0.sample-mysql.default:8801, V:41, E:2].degrade.to.follower.stop.the.vip...
2022/02/18 03:06:49.386049 api.go:285: [INFO] mysql.slave.status:&{ 0 false false }
2022/02/18 03:06:49.386328 api.go:295: [INFO] mysql.master.status:&{mysql-bin.000005 1072490484 true true b2c3c5d9-9060-11ec-9817-12c02189a4ce:1-106624 0 }
2022/02/18 03:06:54.386357 api.go:285: [INFO] mysql.slave.status:&{ 0 false false }
2022/02/18 03:06:54.386624 api.go:295: [INFO] mysql.master.status:&{mysql-bin.000005 1072490484 true true b2c3c5d9-9060-11ec-9817-12c02189a4ce:1-106624 0 }
2022/02/18 03:06:59.385731 api.go:285: [INFO] mysql.slave.status:&{ 0 false false }
2022/02/18 03:06:59.385918 api.go:295: [INFO] mysql.master.status:&{mysql-bin.000005 1072490484 true true b2c3c5d9-9060-11ec-9817-12c02189a4ce:1-106624 0 }
2022/02/18 03:07:04.385802 api.go:285: [INFO] mysql.slave.status:&{ 0 false false }
2022/02/18 03:07:04.386025 api.go:295: [INFO] mysql.master.status:&{mysql-bin.000005 1072490484 true true b2c3c5d9-9060-11ec-9817-12c02189a4ce:1-106624 0 }
2022/02/18 03:07:09.386531 api.go:285: [INFO] mysql.slave.status:&{ 0 false false }
2022/02/18 03:07:09.386740 api.go:295: [INFO] mysql.master.status:&{mysql-bin.000005 1072490484 true true b2c3c5d9-9060-11ec-9817-12c02189a4ce:1-106624 0 }
2022/02/18 03:07:14.386306 api.go:285: [INFO] mysql.slave.status:&{ 0 false false }
2022/02/18 03:07:14.386538 api.go:295: [INFO] mysql.master.status:&{mysql-bin.000005 1072490484 true true b2c3c5d9-9060-11ec-9817-12c02189a4ce:1-106624 0 }
2022/02/18 03:07:19.385595 api.go:285: [INFO] mysql.slave.status:&{ 0 false false }
2022/02/18 03:07:19.385783 api.go:295: [INFO] mysql.master.status:&{mysql-bin.000005 1072490484 true true b2c3c5d9-9060-11ec-9817-12c02189a4ce:1-106624 0 }
2022/02/18 03:07:24.386190 api.go:285: [INFO] mysql.slave.status:&{ 0 false false }
2022/02/18 03:07:24.386487 api.go:295: [INFO] mysql.master.status:&{mysql-bin.000005 1072490484 true true b2c3c5d9-9060-11ec-9817-12c02189a4ce:1-106624 0 }
2022/02/18 03:07:29.386144 api.go:285: [INFO] mysql.slave.status:&{ 0 false false }
2022/02/18 03:07:29.386355 api.go:295: [INFO] mysql.master.status:&{mysql-bin.000005 1072490484 true true b2c3c5d9-9060-11ec-9817-12c02189a4ce:1-106624 0 }
2022/02/18 03:07:34.386493 api.go:285: [INFO] mysql.slave.status:&{ 0 false false }
2022/02/18 03:07:34.389835 api.go:295: [INFO] mysql.master.status:&{mysql-bin.000005 1072490484 true true b2c3c5d9-9060-11ec-9817-12c02189a4ce:1-106624 0 }
2022/02/18 03:07:39.367732 cluster.go:98: [WARNING] api.v1.cluster.prepare.to.remove.nodes[sample-mysql-2.sample-mysql.default:8801].from.leader[sample-mysql-0.sample-mysql.default:8801]
2022/02/18 03:07:39.369044 rpc_node.go:60: [WARNING] server.rpc.node.remove:&{From: Nodes:[sample-mysql-2.sample-mysql.default:8801]}
2022/02/18 03:07:39.369232 trace.go:27: [INFO] LEADER[ID:sample-mysql-0.sample-mysql.default:8801, V:41, E:3].check.peers.json.file[/var/lib/xenon/peers.json].stat[name:peers.json, mode:-rwxr-xr-x, size:114, lastmodified:2022-02-18 03:07:39.364133404 +0000 UTC]
2022/02/18 03:07:39.369252 trace.go:32: [WARNING] LEADER[ID:sample-mysql-0.sample-mysql.default:8801, V:41, E:3].removed.peer[sample-mysql-2.sample-mysql.default:8801].from.peers[map[sample-mysql-1.sample-mysql.default:8801:0xc0002030b0]]
2022/02/18 03:07:39.369498 cluster.go:113: [WARNING] api.v1.cluster.remove.nodes.from.leader[sample-mysql-0.sample-mysql.default:8801].done
2022/02/18 03:07:39.386747 api.go:285: [INFO] mysql.slave.status:&{ 0 false false }
2022/02/18 03:07:39.386951 api.go:295: [INFO] mysql.master.status:&{mysql-bin.000005 1072490484 true true b2c3c5d9-9060-11ec-9817-12c02189a4ce:1-106624 0 }
2022/02/18 03:07:39.441159 rpc_node.go:60: [WARNING] server.rpc.node.remove:&{From: Nodes:[sample-mysql-2.sample-mysql.default:8801]}
2022/02/18 03:07:39.454312 trace.go:32: [WARNING] LEADER[ID:sample-mysql-0.sample-mysql.default:8801, V:41, E:3].peer[sample-mysql-2.sample-mysql.default:8801].not.exists.in.peers[map[sample-mysql-1.sample-mysql.default:8801:0xc0002030b0]]
2022/02/18 03:07:39.516594 cluster.go:98: [WARNING] api.v1.cluster.prepare.to.remove.nodes[sample-mysql-2.sample-mysql.default:8801].from.leader[sample-mysql-0.sample-mysql.default:8801]
2022/02/18 03:07:39.517975 rpc_node.go:60: [WARNING] server.rpc.node.remove:&{From: Nodes:[sample-mysql-2.sample-mysql.default:8801]}
2022/02/18 03:07:39.517997 trace.go:32: [WARNING] LEADER[ID:sample-mysql-0.sample-mysql.default:8801, V:41, E:3].peer[sample-mysql-2.sample-mysql.default:8801].not.exists.in.peers[map[sample-mysql-1.sample-mysql.default:8801:0xc0002030b0]]
2022/02/18 03:07:39.518173 cluster.go:113: [WARNING] api.v1.cluster.remove.nodes.from.leader[sample-mysql-0.sample-mysql.default:8801].done
2022/02/18 03:07:39.586465 rpc_node.go:60: [WARNING] server.rpc.node.remove:&{From: Nodes:[sample-mysql-2.sample-mysql.default:8801]}
2022/02/18 03:07:39.586497 trace.go:32: [WARNING] LEADER[ID:sample-mysql-0.sample-mysql.default:8801, V:41, E:3].peer[sample-mysql-2.sample-mysql.default:8801].not.exists.in.peers[map[sample-mysql-1.sample-mysql.default:8801:0xc0002030b0]]
2022/02/18 03:07:39.627511 cluster.go:98: [WARNING] api.v1.cluster.prepare.to.remove.nodes[sample-mysql-2.sample-mysql.default:8801].from.leader[sample-mysql-0.sample-mysql.default:8801]
2022/02/18 03:07:39.628907 rpc_node.go:60: [WARNING] server.rpc.node.remove:&{From: Nodes:[sample-mysql-2.sample-mysql.default:8801]}
2022/02/18 03:07:39.628930 trace.go:32: [WARNING] LEADER[ID:sample-mysql-0.sample-mysql.default:8801, V:41, E:3].peer[sample-mysql-2.sample-mysql.default:8801].not.exists.in.peers[map[sample-mysql-1.sample-mysql.default:8801:0xc0002030b0]]
2022/02/18 03:07:39.629095 cluster.go:113: [WARNING] api.v1.cluster.remove.nodes.from.leader[sample-mysql-0.sample-mysql.default:8801].done
2022/02/18 03:07:39.694519 rpc_node.go:60: [WARNING] server.rpc.node.remove:&{From: Nodes:[sample-mysql-2.sample-mysql.default:8801]}
2022/02/18 03:07:39.694558 trace.go:32: [WARNING] LEADER[ID:sample-mysql-0.sample-mysql.default:8801, V:41, E:3].peer[sample-mysql-2.sample-mysql.default:8801].not.exists.in.peers[map[sample-mysql-1.sample-mysql.default:8801:0xc0002030b0]]
2022/02/18 03:07:39.824667 cluster.go:98: [WARNING] api.v1.cluster.prepare.to.remove.nodes[sample-mysql-2.sample-mysql.default:8801].from.leader[sample-mysql-0.sample-mysql.default:8801]
2022/02/18 03:07:39.826771 rpc_node.go:60: [WARNING] server.rpc.node.remove:&{From: Nodes:[sample-mysql-2.sample-mysql.default:8801]}
2022/02/18 03:07:39.826809 trace.go:32: [WARNING] LEADER[ID:sample-mysql-0.sample-mysql.default:8801, V:41, E:3].peer[sample-mysql-2.sample-mysql.default:8801].not.exists.in.peers[map[sample-mysql-1.sample-mysql.default:8801:0xc0002030b0]]
2022/02/18 03:07:39.827064 cluster.go:113: [WARNING] api.v1.cluster.remove.nodes.from.leader[sample-mysql-0.sample-mysql.default:8801].done
2022/02/18 03:07:39.843363 rpc_node.go:60: [WARNING] server.rpc.node.remove:&{From: Nodes:[sample-mysql-2.sample-mysql.default:8801]}
2022/02/18 03:07:39.843405 trace.go:32: [WARNING] LEADER[ID:sample-mysql-0.sample-mysql.default:8801, V:41, E:3].peer[sample-mysql-2.sample-mysql.default:8801].not.exists.in.peers[map[sample-mysql-1.sample-mysql.default:8801:0xc0002030b0]]
2022/02/18 03:07:44.359816 cluster.go:98: [WARNING] api.v1.cluster.prepare.to.remove.nodes[sample-mysql-2.sample-mysql.default:8801].from.leader[sample-mysql-0.sample-mysql.default:8801]
2022/02/18 03:07:44.361278 rpc_node.go:60: [WARNING] server.rpc.node.remove:&{From: Nodes:[sample-mysql-2.sample-mysql.default:8801]}
2022/02/18 03:07:44.361309 trace.go:32: [WARNING] LEADER[ID:sample-mysql-0.sample-mysql.default:8801, V:41, E:3].peer[sample-mysql-2.sample-mysql.default:8801].not.exists.in.peers[map[sample-mysql-1.sample-mysql.default:8801:0xc0002030b0]]
2022/02/18 03:07:44.361573 cluster.go:113: [WARNING] api.v1.cluster.remove.nodes.from.leader[sample-mysql-0.sample-mysql.default:8801].done
2022/02/18 03:07:44.385577 api.go:285: [INFO] mysql.slave.status:&{ 0 false false }
2022/02/18 03:07:44.385838 api.go:295: [INFO] mysql.master.status:&{mysql-bin.000005 1072490484 true true b2c3c5d9-9060-11ec-9817-12c02189a4ce:1-106624 0 }
2022/02/18 03:07:44.397306 rpc_node.go:60: [WARNING] server.rpc.node.remove:&{From: Nodes:[sample-mysql-2.sample-mysql.default:8801]}
2022/02/18 03:07:44.397343 trace.go:32: [WARNING] LEADER[ID:sample-mysql-0.sample-mysql.default:8801, V:41, E:3].peer[sample-mysql-2.sample-mysql.default:8801].not.exists.in.peers[map[sample-mysql-1.sample-mysql.default:8801:0xc0002030b0]]
2022/02/18 03:07:49.386328 api.go:285: [INFO] mysql.slave.status:&{ 0 false false }
2022/02/18 03:07:49.386485 api.go:295: [INFO] mysql.master.status:&{mysql-bin.000005 1072491705 true true b2c3c5d9-9060-11ec-9817-12c02189a4ce:1-106624 0 }
2022/02/18 03:07:49.416111 cluster.go:53: [WARNING] api.v1.cluster.prepare.to.add.nodes[sample-mysql-2.sample-mysql.default:8801].to.leader[sample-mysql-0.sample-mysql.default:8801]
2022/02/18 03:07:49.417466 rpc_node.go:28: [WARNING] server.rpc.node.add:&{From: Nodes:[sample-mysql-2.sample-mysql.default:8801]}
2022/02/18 03:07:49.417663 trace.go:27: [INFO] LEADER[ID:sample-mysql-0.sample-mysql.default:8801, V:41, E:4].check.peers.json.file[/var/lib/xenon/peers.json].stat[name:peers.json, mode:-rwxr-xr-x, size:157, lastmodified:2022-02-18 03:07:49.411951679 +0000 UTC]
2022/02/18 03:07:49.417685 trace.go:32: [WARNING] LEADER[ID:sample-mysql-0.sample-mysql.default:8801, V:41, E:4].add.peer[sample-mysql-2.sample-mysql.default:8801].to.peers[map[sample-mysql-1.sample-mysql.default:8801:0xc0002030b0 sample-mysql-2.sample-mysql.default:8801:0xc00051f020]]
2022/02/18 03:07:49.418417 cluster.go:68: [WARNING] api.v1.cluster.add.nodes.to.leader[sample-mysql-0.sample-mysql.default:8801].done
2022/02/18 03:07:53.594398 trace.go:32: [WARNING] LEADER[ID:sample-mysql-0.sample-mysql.default:8801, V:41, E:4].leaderStopShellCommand[[-c /scripts/leader-stop.sh]].done
2022/02/18 03:07:53.594432 trace.go:27: [INFO] LEADER[ID:sample-mysql-0.sample-mysql.default:8801, V:41, E:4].check.semi-sync.thread.stop...
2022/02/18 03:07:53.594438 trace.go:27: [INFO] LEADER[ID:sample-mysql-0.sample-mysql.default:8801, V:41, E:4].check.gtid.thread.stop...
2022/02/18 03:07:53.594447 trace.go:32: [WARNING] FOLLOWER[ID:sample-mysql-0.sample-mysql.default:8801, V:41, E:4].leader.state.machine.exit.done
2022/02/18 03:07:53.594452 trace.go:27: [INFO] FOLLOWER[ID:sample-mysql-0.sample-mysql.default:8801, V:41, E:4].start.CheckBrainSplit
2022/02/18 03:07:53.594506 trace.go:32: [WARNING] FOLLOWER[ID:sample-mysql-0.sample-mysql.default:8801, V:41, E:4].state.init
2022/02/18 03:07:53.671424 trace.go:32: [WARNING] FOLLOWER[ID:sample-mysql-0.sample-mysql.default:8801, V:41, E:4].leaderStopShellCommand[[-c /scripts/leader-stop.sh]].done
2022/02/18 03:07:53.671463 trace.go:32: [WARNING] FOLLOWER[ID:sample-mysql-0.sample-mysql.default:8801, V:41, E:4].mysql.waitMysqlDoneAsync.prepare
2022/02/18 03:07:53.671480 trace.go:32: [WARNING] FOLLOWER[ID:sample-mysql-0.sample-mysql.default:8801, V:41, E:4].state.machine.run
2022/02/18 03:07:58.594911 trace.go:32: [WARNING] FOLLOWER[ID:sample-mysql-0.sample-mysql.default:8801, V:41, E:4].ping.responses[1].is.less.than.half.maybe.brain.split
2022/02/18 03:07:58.598309 trace.go:32: [WARNING] FOLLOWER[ID:sample-mysql-0.sample-mysql.default:8801, V:41, E:4].ping.responses[2].is.greater.than.half.again
2022/02/18 03:07:58.912879 trace.go:32: [WARNING] FOLLOWER[ID:sample-mysql-0.sample-mysql.default:8801, V:41, E:4].mysql.SetReadOnly.done
2022/02/18 03:07:58.954602 api.go:385: [WARNING] mysql[localhost:3306].SetSlaveGlobalSysVar[tokudb_fsync_log_period=1000;sync_binlog=1000;innodb_flush_log_at_trx_commit=1]
2022/02/18 03:07:58.954637 trace.go:32: [WARNING] FOLLOWER[ID:sample-mysql-0.sample-mysql.default:8801, V:41, E:4].mysql.SetSlaveGlobalSysVar.done
2022/02/18 03:07:58.954644 trace.go:32: [WARNING] FOLLOWER[ID:sample-mysql-0.sample-mysql.default:8801, V:41, E:4].prepareAsync.done
2022/02/18 03:07:58.960826 trace.go:37: [ERROR] FOLLOWER[ID:sample-mysql-0.sample-mysql.default:8801, V:41, E:4].mysql.StartSlave.error[Error 1200: The server is not configured as slave; fix in config file or with CHANGE MASTER TO]
2022/02/18 03:07:58.960930 api.go:285: [INFO] mysql.slave.status:&{ 0 false false }
2022/02/18 03:07:58.961121 api.go:285: [INFO] mysql.slave.status:&{ 0 false false }
2022/02/18 03:07:58.961176 api.go:295: [INFO] mysql.master.status:&{mysql-bin.000005 1073642996 true true b2c3c5d9-9060-11ec-9817-12c02189a4ce:1-107648 0 }
2022/02/18 03:07:58.961335 trace.go:32: [WARNING] FOLLOWER[ID:sample-mysql-0.sample-mysql.default:8801, V:41, E:4].init.my.gtid.is:{mysql-bin.000005 1073642996 true true b2c3c5d9-9060-11ec-9817-12c02189a4ce:1-107648 0 }
2022/02/18 03:07:58.961367 api.go:295: [INFO] mysql.master.status:&{mysql-bin.000005 1073642996 true true b2c3c5d9-9060-11ec-9817-12c02189a4ce:1-107648 0 }
2022/02/18 03:07:58.961407 trace.go:32: [WARNING] FOLLOWER[ID:sample-mysql-0.sample-mysql.default:8801, V:41, E:4].get.heartbeat.my.gtid.is:{mysql-bin.000005 1073642996 true true b2c3c5d9-9060-11ec-9817-12c02189a4ce:1-107648 0 }
2022/02/18 03:07:58.961770 mysql.go:137: [INFO] mysql.get.local.uuid:[b2c3c5d9-9060-11ec-9817-12c02189a4ce]
2022/02/18 03:07:58.962428 mysql.go:137: [INFO] mysql.get.local.uuid:[b2c3c5d9-9060-11ec-9817-12c02189a4ce]
2022/02/18 03:07:58.962639 api.go:173: [WARNING] follower.gtid[&{mysql-bin.000005 1073642996 true true b2c3c5d9-9060-11ec-9817-12c02189a4ce:1-107648 0 }].bigger.than.remote[&{mysql-bin.000003 1070145394 true true b2c3c5d9-9060-11ec-9817-12c02189a4ce:1-106624 0 }]
2022/02/18 03:07:58.962731 trace.go:32: [WARNING] INVALID[ID:sample-mysql-0.sample-mysql.default:8801, V:41, E:4].get.heartbeat.from[N:sample-mysql-1.sample-mysql.default:8801, V:46, E:4].change.mysql.master
2022/02/18 03:07:58.995761 trace.go:32: [WARNING] INVALID[ID:sample-mysql-0.sample-mysql.default:8801, V:41, E:4].get.heartbeat.change.to.the.new.master[sample-mysql-1.sample-mysql.default:8801].successed
2022/02/18 03:07:58.995790 trace.go:32: [WARNING] INVALID[ID:sample-mysql-0.sample-mysql.default:8801, V:41, E:4].get.heartbeat.from[N:sample-mysql-1.sample-mysql.default:8801, V:46, E:4].update.view
2022/02/18 03:07:58.995797 trace.go:32: [WARNING] INVALID[ID:sample-mysql-0.sample-mysql.default:8801, V:41, E:4].do.updateViewID[FROM:41 TO:46]
2022/02/18 03:07:58.995826 trace.go:32: [WARNING] INVALID[ID:sample-mysql-0.sample-mysql.default:8801, V:46, E:4].follower.state.machine.exit
2022/02/18 03:07:59.073417 trace.go:32: [WARNING] INVALID[ID:sample-mysql-0.sample-mysql.default:8801, V:46, E:4].leaderStopShellCommand[[-c /scripts/leader-stop.sh]].done
2022/02/18 03:07:59.074417 api.go:385: [WARNING] mysql[localhost:3306].SetSlaveGlobalSysVar[tokudb_fsync_log_period=1000;sync_binlog=1000;innodb_flush_log_at_trx_commit=1]
To Reproduce
Expected behavior
Environment:
- RadonDB MySQL version: