Header Shadow Image


etcd: error validating peerURLs or etcd: request sent was ignored ( cluster ID mismatch )

Getting the following?

etcd: health check for peer 83f149dc6ec1b00a could not connect: dial tcp 10.3.0.124:2380: connect: connection refused (prober "ROUND_TRIPPER_SNAPSHOT")

or

etcd: request sent was ignored (cluster ID mismatch: remote[a82b23223d9f684e]=314e5a8f7a211a07, local=47f62724bd585a9)

or

etcd: publish error: etcdserver: request timed out

or

etcd: error validating peerURLs {ClusterID:314e5a8f7a211a07 Members:[&{ID:c470debdfc4607fe RaftAttributes:{PeerURLs:[http://192.168.0.108:2380]} Attributes:{Name:etcd01 ClientURLs:[]}} &{ID:83f149dc6ec1b00a RaftAttributes:{PeerURLs:[http://10.3.0.124:2380]} Attributes:{Name:etcd02 ClientURLs:[http://10.3.0.124:2379]}} &{ID:a82b23223d9f684e RaftAttributes:{PeerURLs:[http://10.3.0.118:2380]} Attributes:{Name:etcd03 ClientURLs:[http://10.3.0.118:2379]}}] RemovedMemberIDs:[]}: unmatched member while checking PeerURLs ("http://10.3.0.118:2380"(resolved from "http://10.3.0.118:2380") != "http://10.3.0.108:2380"(resolved from "http://10.3.0.108:2380"))

or

Error:  client: etcd cluster is unavailable or misconfigured; error #0: dial tcp 127.0.0.1:2379: connect: connection refused
; error #1: dial tcp 127.0.0.1:4001: connect: connection refused

error #0: dial tcp 127.0.0.1:2379: connect: connection refused
error #1: dial tcp 127.0.0.1:4001: connect: connection refused

or

etcd: request sent was ignored (cluster ID mismatch: peer[83f149dc6ec1b00a]=314e5a8f7a211a07, local=47f62724bd585a9)

You can solve this by reinitializing the entire cluster. Set the cluster state to new on each node:

[root@psql03 etcd]# cat /etc/etcd/etcd.conf
ETCD_LISTEN_PEER_URLS="http://10.3.0.118:2380"
ETCD_LISTEN_CLIENT_URLS="http://localhost:2379,http://10.3.0.118:2379"
ETCD_INITIAL_ADVERTISE_PEER_URLS="http://10.3.0.118:2380"
ETCD_INITIAL_CLUSTER="etcd01=http://10.3.0.108:2380,etcd02=http://10.3.0.124:2380,etcd03=http://10.3.0.118:2380"
ETCD_ADVERTISE_CLIENT_URLS="http://10.3.0.118:2379"
ETCD_INITIAL_CLUSTER_TOKEN="etcd-c01"
ETCD_INITIAL_CLUSTER_STATE="new"
ETCD_DATA_DIR="/var/lib/etcd/default.etcd"
ETCD_NAME="etcd03"
ETCD_HEARTBEAT_INTERVAL=250
ETCD_ELECTION_TIMEOUT=1250
[root@psql03 etcd]#

 

[root@psql02 etcd]# cat /etc/etcd/etcd.conf
ETCD_LISTEN_PEER_URLS="http://10.3.0.124:2380"
ETCD_LISTEN_CLIENT_URLS="http://localhost:2379,http://10.3.0.124:2379"
ETCD_INITIAL_ADVERTISE_PEER_URLS="http://10.3.0.124:2380"
ETCD_INITIAL_CLUSTER="etcd01=http://10.3.0.108:2380,etcd02=http://10.3.0.124:2380,etcd03=http://10.3.0.118:2380"
ETCD_ADVERTISE_CLIENT_URLS="http://10.3.0.124:2379"
ETCD_INITIAL_CLUSTER_TOKEN="etcd-c01"
ETCD_INITIAL_CLUSTER_STATE="new"
ETCD_DATA_DIR="/var/lib/etcd/default.etcd"
ETCD_NAME="etcd02"
ETCD_HEARTBEAT_INTERVAL=250
ETCD_ELECTION_TIMEOUT=1250
[root@psql02 etcd]#

 

[root@psql01 snap]# cat /etc/etcd/etcd.conf
ETCD_LISTEN_PEER_URLS="http://10.3.0.108:2380"
ETCD_LISTEN_CLIENT_URLS="http://localhost:2379,http://10.3.0.108:2379"
ETCD_INITIAL_ADVERTISE_PEER_URLS="http://10.3.0.108:2380"
ETCD_INITIAL_CLUSTER="etcd01=http://10.3.0.108:2380,etcd02=http://10.3.0.124:2380,etcd03=http://10.3.0.118:2380"
ETCD_ADVERTISE_CLIENT_URLS="http://10.3.0.108:2379"
ETCD_INITIAL_CLUSTER_TOKEN="etcd-c01"
ETCD_INITIAL_CLUSTER_STATE="new"
ETCD_DATA_DIR="/var/lib/etcd/default.etcd"
ETCD_NAME="etcd01"
ETCD_HEARTBEAT_INTERVAL=250
ETCD_ELECTION_TIMEOUT=1250
[root@psql01 snap]#

Then start each node:

systemctl start etcd

and verify:

[root@psql03 etcd]# etcdctl  cluster-health
member 83f149dc6ec1b00a is healthy: got healthy result from http://10.3.0.124:2379
member 93200353704b2d19 is healthy: got healthy result from http://10.3.0.108:2379
member a82b23223d9f684e is healthy: got healthy result from http://10.3.0.118:2379
cluster is healthy
[root@psql03 etcd]#

Next, go back and change the configuration of each ETCD member back to existing:

ETCD_INITIAL_CLUSTER_STATE="existing"

Restart all nodes by stopping ETCD first then starting it up on all nodes in close succession:

[root@psql01 etcd]# systemctl start etcd
[root@psql02 etcd]# systemctl start etcd
[root@psql03 etcd]# systemctl start etcd

At this point you should be good.  Now if you also get this message:

etcd[16312]: the clock difference against peer a82b23223d9f684e is too high [1.20986188s > 1s] (prober "ROUND_TRIPPER…MESSAGE")

Check your NTP configuration, run ntpdate <NTP SERVER> and restart NTPD: systemctl restart ntpd .   This should resolve it. 

Thx,
TK


     
  Copyright © 2003 - 2013 Tom Kacperski (microdevsys.com). All rights reserved.

Creative Commons License
This work is licensed under a Creative Commons Attribution 3.0 Unported License