etcd: error validating peerURLs or etcd: request sent was ignored ( cluster ID mismatch )
Getting the following?
etcd: health check for peer 83f149dc6ec1b00a could not connect: dial tcp 10.3.0.124:2380: connect: connection refused (prober "ROUND_TRIPPER_SNAPSHOT")
or
etcd: request sent was ignored (cluster ID mismatch: remote[a82b23223d9f684e]=314e5a8f7a211a07, local=47f62724bd585a9)
or
etcd: publish error: etcdserver: request timed out
or
etcd: error validating peerURLs {ClusterID:314e5a8f7a211a07 Members:[&{ID:c470debdfc4607fe RaftAttributes:{PeerURLs:[http://192.168.0.108:2380]} Attributes:{Name:etcd01 ClientURLs:[]}} &{ID:83f149dc6ec1b00a RaftAttributes:{PeerURLs:[http://10.3.0.124:2380]} Attributes:{Name:etcd02 ClientURLs:[http://10.3.0.124:2379]}} &{ID:a82b23223d9f684e RaftAttributes:{PeerURLs:[http://10.3.0.118:2380]} Attributes:{Name:etcd03 ClientURLs:[http://10.3.0.118:2379]}}] RemovedMemberIDs:[]}: unmatched member while checking PeerURLs ("http://10.3.0.118:2380"(resolved from "http://10.3.0.118:2380") != "http://10.3.0.108:2380"(resolved from "http://10.3.0.108:2380"))
or
Error: client: etcd cluster is unavailable or misconfigured; error #0: dial tcp 127.0.0.1:2379: connect: connection refused
; error #1: dial tcp 127.0.0.1:4001: connect: connection refused
error #0: dial tcp 127.0.0.1:2379: connect: connection refused
error #1: dial tcp 127.0.0.1:4001: connect: connection refused
or
etcd: request sent was ignored (cluster ID mismatch: peer[83f149dc6ec1b00a]=314e5a8f7a211a07, local=47f62724bd585a9)
You can solve this by reinitializing the entire cluster. Set the cluster state to new on each node:
[root@psql03 etcd]# cat /etc/etcd/etcd.conf
ETCD_LISTEN_PEER_URLS="http://10.3.0.118:2380"
ETCD_LISTEN_CLIENT_URLS="http://localhost:2379,http://10.3.0.118:2379"
ETCD_INITIAL_ADVERTISE_PEER_URLS="http://10.3.0.118:2380"
ETCD_INITIAL_CLUSTER="etcd01=http://10.3.0.108:2380,etcd02=http://10.3.0.124:2380,etcd03=http://10.3.0.118:2380"
ETCD_ADVERTISE_CLIENT_URLS="http://10.3.0.118:2379"
ETCD_INITIAL_CLUSTER_TOKEN="etcd-c01"
ETCD_INITIAL_CLUSTER_STATE="new"
ETCD_DATA_DIR="/var/lib/etcd/default.etcd"
ETCD_NAME="etcd03"
ETCD_HEARTBEAT_INTERVAL=250
ETCD_ELECTION_TIMEOUT=1250
[root@psql03 etcd]#
[root@psql02 etcd]# cat /etc/etcd/etcd.conf
ETCD_LISTEN_PEER_URLS="http://10.3.0.124:2380"
ETCD_LISTEN_CLIENT_URLS="http://localhost:2379,http://10.3.0.124:2379"
ETCD_INITIAL_ADVERTISE_PEER_URLS="http://10.3.0.124:2380"
ETCD_INITIAL_CLUSTER="etcd01=http://10.3.0.108:2380,etcd02=http://10.3.0.124:2380,etcd03=http://10.3.0.118:2380"
ETCD_ADVERTISE_CLIENT_URLS="http://10.3.0.124:2379"
ETCD_INITIAL_CLUSTER_TOKEN="etcd-c01"
ETCD_INITIAL_CLUSTER_STATE="new"
ETCD_DATA_DIR="/var/lib/etcd/default.etcd"
ETCD_NAME="etcd02"
ETCD_HEARTBEAT_INTERVAL=250
ETCD_ELECTION_TIMEOUT=1250
[root@psql02 etcd]#
[root@psql01 snap]# cat /etc/etcd/etcd.conf
ETCD_LISTEN_PEER_URLS="http://10.3.0.108:2380"
ETCD_LISTEN_CLIENT_URLS="http://localhost:2379,http://10.3.0.108:2379"
ETCD_INITIAL_ADVERTISE_PEER_URLS="http://10.3.0.108:2380"
ETCD_INITIAL_CLUSTER="etcd01=http://10.3.0.108:2380,etcd02=http://10.3.0.124:2380,etcd03=http://10.3.0.118:2380"
ETCD_ADVERTISE_CLIENT_URLS="http://10.3.0.108:2379"
ETCD_INITIAL_CLUSTER_TOKEN="etcd-c01"
ETCD_INITIAL_CLUSTER_STATE="new"
ETCD_DATA_DIR="/var/lib/etcd/default.etcd"
ETCD_NAME="etcd01"
ETCD_HEARTBEAT_INTERVAL=250
ETCD_ELECTION_TIMEOUT=1250
[root@psql01 snap]#
Then start each node:
systemctl start etcd
and verify:
[root@psql03 etcd]# etcdctl cluster-health
member 83f149dc6ec1b00a is healthy: got healthy result from http://10.3.0.124:2379
member 93200353704b2d19 is healthy: got healthy result from http://10.3.0.108:2379
member a82b23223d9f684e is healthy: got healthy result from http://10.3.0.118:2379
cluster is healthy
[root@psql03 etcd]#
Next, go back and change the configuration of each ETCD member back to existing:
ETCD_INITIAL_CLUSTER_STATE="existing"
Restart all nodes by stopping ETCD first then starting it up on all nodes in close succession:
[root@psql01 etcd]# systemctl start etcd
[root@psql02 etcd]# systemctl start etcd
[root@psql03 etcd]# systemctl start etcd
At this point you should be good. Now if you also get this message:
etcd[16312]: the clock difference against peer a82b23223d9f684e is too high [1.20986188s > 1s] (prober "ROUND_TRIPPER…MESSAGE")
Check your NTP configuration, run ntpdate <NTP SERVER> and restart NTPD: systemctl restart ntpd . This should resolve it.
Thx,
TK