Header Shadow Image


etcd: error validating peerURLs or etcd: request sent was ignored ( cluster ID mismatch )

Getting the following?

etcd: health check for peer 83f149dc6ec1b00a could not connect: dial tcp 10.3.0.124:2380: connect: connection refused (prober "ROUND_TRIPPER_SNAPSHOT")

or

etcd: request sent was ignored (cluster ID mismatch: remote[a82b23223d9f684e]=314e5a8f7a211a07, local=47f62724bd585a9)

or

etcd: publish error: etcdserver: request timed out

or

etcd: error validating peerURLs {ClusterID:314e5a8f7a211a07 Members:[&{ID:c470debdfc4607fe RaftAttributes:{PeerURLs:[http://192.168.0.108:2380]} Attributes:{Name:etcd01 ClientURLs:[]}} &{ID:83f149dc6ec1b00a RaftAttributes:{PeerURLs:[http://10.3.0.124:2380]} Attributes:{Name:etcd02 ClientURLs:[http://10.3.0.124:2379]}} &{ID:a82b23223d9f684e RaftAttributes:{PeerURLs:[http://10.3.0.118:2380]} Attributes:{Name:etcd03 ClientURLs:[http://10.3.0.118:2379]}}] RemovedMemberIDs:[]}: unmatched member while checking PeerURLs ("http://10.3.0.118:2380"(resolved from "http://10.3.0.118:2380") != "http://10.3.0.108:2380"(resolved from "http://10.3.0.108:2380"))

or

Error:  client: etcd cluster is unavailable or misconfigured; error #0: dial tcp 127.0.0.1:2379: connect: connection refused
; error #1: dial tcp 127.0.0.1:4001: connect: connection refused

error #0: dial tcp 127.0.0.1:2379: connect: connection refused
error #1: dial tcp 127.0.0.1:4001: connect: connection refused

or

etcd: request sent was ignored (cluster ID mismatch: peer[83f149dc6ec1b00a]=314e5a8f7a211a07, local=47f62724bd585a9)

You can solve this by reinitializing the entire cluster. Set the cluster state to new on each node:

[root@psql03 etcd]# cat /etc/etcd/etcd.conf
ETCD_LISTEN_PEER_URLS="http://10.3.0.118:2380"
ETCD_LISTEN_CLIENT_URLS="http://localhost:2379,http://10.3.0.118:2379"
ETCD_INITIAL_ADVERTISE_PEER_URLS="http://10.3.0.118:2380"
ETCD_INITIAL_CLUSTER="etcd01=http://10.3.0.108:2380,etcd02=http://10.3.0.124:2380,etcd03=http://10.3.0.118:2380"
ETCD_ADVERTISE_CLIENT_URLS="http://10.3.0.118:2379"
ETCD_INITIAL_CLUSTER_TOKEN="etcd-c01"
ETCD_INITIAL_CLUSTER_STATE="new"
ETCD_DATA_DIR="/var/lib/etcd/default.etcd"
ETCD_NAME="etcd03"
ETCD_HEARTBEAT_INTERVAL=250
ETCD_ELECTION_TIMEOUT=1250
[root@psql03 etcd]#

 

[root@psql02 etcd]# cat /etc/etcd/etcd.conf
ETCD_LISTEN_PEER_URLS="http://10.3.0.124:2380"
ETCD_LISTEN_CLIENT_URLS="http://localhost:2379,http://10.3.0.124:2379"
ETCD_INITIAL_ADVERTISE_PEER_URLS="http://10.3.0.124:2380"
ETCD_INITIAL_CLUSTER="etcd01=http://10.3.0.108:2380,etcd02=http://10.3.0.124:2380,etcd03=http://10.3.0.118:2380"
ETCD_ADVERTISE_CLIENT_URLS="http://10.3.0.124:2379"
ETCD_INITIAL_CLUSTER_TOKEN="etcd-c01"
ETCD_INITIAL_CLUSTER_STATE="new"
ETCD_DATA_DIR="/var/lib/etcd/default.etcd"
ETCD_NAME="etcd02"
ETCD_HEARTBEAT_INTERVAL=250
ETCD_ELECTION_TIMEOUT=1250
[root@psql02 etcd]#

 

[root@psql01 snap]# cat /etc/etcd/etcd.conf
ETCD_LISTEN_PEER_URLS="http://10.3.0.108:2380"
ETCD_LISTEN_CLIENT_URLS="http://localhost:2379,http://10.3.0.108:2379"
ETCD_INITIAL_ADVERTISE_PEER_URLS="http://10.3.0.108:2380"
ETCD_INITIAL_CLUSTER="etcd01=http://10.3.0.108:2380,etcd02=http://10.3.0.124:2380,etcd03=http://10.3.0.118:2380"
ETCD_ADVERTISE_CLIENT_URLS="http://10.3.0.108:2379"
ETCD_INITIAL_CLUSTER_TOKEN="etcd-c01"
ETCD_INITIAL_CLUSTER_STATE="new"
ETCD_DATA_DIR="/var/lib/etcd/default.etcd"
ETCD_NAME="etcd01"
ETCD_HEARTBEAT_INTERVAL=250
ETCD_ELECTION_TIMEOUT=1250
[root@psql01 snap]#

Then start each node:

systemctl start etcd

and verify:

[root@psql03 etcd]# etcdctl  cluster-health
member 83f149dc6ec1b00a is healthy: got healthy result from http://10.3.0.124:2379
member 93200353704b2d19 is healthy: got healthy result from http://10.3.0.108:2379
member a82b23223d9f684e is healthy: got healthy result from http://10.3.0.118:2379
cluster is healthy
[root@psql03 etcd]#

Next, go back and change the configuration of each ETCD member back to existing:

ETCD_INITIAL_CLUSTER_STATE="existing"

Restart all nodes by stopping ETCD first then starting it up on all nodes in close succession:

[root@psql01 etcd]# systemctl start etcd
[root@psql02 etcd]# systemctl start etcd
[root@psql03 etcd]# systemctl start etcd

At this point you should be good.  Now if you also get this message:

etcd[16312]: the clock difference against peer a82b23223d9f684e is too high [1.20986188s > 1s] (prober "ROUND_TRIPPER…MESSAGE")

Check your NTP configuration, run ntpdate <NTP SERVER> and restart NTPD: systemctl restart ntpd .   This should resolve it. 

Thx,
TK

Subscribe
Notify of
0 Comments
Oldest
Newest Most Voted
Inline Feedbacks
View all comments

     
  Copyright © 2003 - 2025 Tom Kacperski (microdevsys.com). All rights reserved.

Creative Commons License
This work is licensed under a Creative Commons Attribution 3.0 Unported License

 

0
Would love your thoughts, please comment.x
()
x
The IT Development and Technology Mini Vault | MicroDevSys.com
Privacy Overview

This website uses cookies so that we can provide you with the best user experience possible. Cookie information is stored in your browser and performs functions such as recognising you when you return to our website and helping our team to understand which sections of the website you find most interesting and useful.