Getting the following?

etcd: health check for peer 83f149dc6ec1b00a could not connect: dial tcp 10.3.0.124:2380: connect: connection refused (prober "ROUND_TRIPPER_SNAPSHOT")

or

etcd: request sent was ignored (cluster ID mismatch: remote[a82b23223d9f684e]=314e5a8f7a211a07, local=47f62724bd585a9)

or

etcd: publish error: etcdserver: request timed out

or

etcd: error validating peerURLs {ClusterID:314e5a8f7a211a07 Members:[&{ID:c470debdfc4607fe RaftAttributes:{PeerURLs:[http://192.168.0.108:2380]} Attributes:{Name:etcd01 ClientURLs:[]}} &{ID:83f149dc6ec1b00a RaftAttributes:{PeerURLs:[http://10.3.0.124:2380]} Attributes:{Name:etcd02 ClientURLs:[http://10.3.0.124:2379]}} &{ID:a82b23223d9f684e RaftAttributes:{PeerURLs:[http://10.3.0.118:2380]} Attributes:{Name:etcd03 ClientURLs:[http://10.3.0.118:2379]}}] RemovedMemberIDs:[]}: unmatched member while checking PeerURLs ("http://10.3.0.118:2380"(resolved from "http://10.3.0.118:2380") != "http://10.3.0.108:2380"(resolved from "http://10.3.0.108:2380"))

or

Error:  client: etcd cluster is unavailable or misconfigured; error #0: dial tcp 127.0.0.1:2379: connect: connection refused
; error #1: dial tcp 127.0.0.1:4001: connect: connection refused

error #0: dial tcp 127.0.0.1:2379: connect: connection refused
error #1: dial tcp 127.0.0.1:4001: connect: connection refused

or

etcd: request sent was ignored (cluster ID mismatch: peer[83f149dc6ec1b00a]=314e5a8f7a211a07, local=47f62724bd585a9)

You can solve this by reinitializing the entire cluster. Set the cluster state to new on each node:

[root@psql03 etcd]# cat /etc/etcd/etcd.conf
ETCD_LISTEN_PEER_URLS="http://10.3.0.118:2380"
ETCD_LISTEN_CLIENT_URLS="http://localhost:2379,http://10.3.0.118:2379"
ETCD_INITIAL_ADVERTISE_PEER_URLS="http://10.3.0.118:2380"
ETCD_INITIAL_CLUSTER="etcd01=http://10.3.0.108:2380,etcd02=http://10.3.0.124:2380,etcd03=http://10.3.0.118:2380"
ETCD_ADVERTISE_CLIENT_URLS="http://10.3.0.118:2379"
ETCD_INITIAL_CLUSTER_TOKEN="etcd-c01"
ETCD_INITIAL_CLUSTER_STATE="new"
ETCD_DATA_DIR="/var/lib/etcd/default.etcd"
ETCD_NAME="etcd03"
ETCD_HEARTBEAT_INTERVAL=250
ETCD_ELECTION_TIMEOUT=1250
[root@psql03 etcd]#
 

[root@psql02 etcd]# cat /etc/etcd/etcd.conf
ETCD_LISTEN_PEER_URLS="http://10.3.0.124:2380"
ETCD_LISTEN_CLIENT_URLS="http://localhost:2379,http://10.3.0.124:2379"
ETCD_INITIAL_ADVERTISE_PEER_URLS="http://10.3.0.124:2380"
ETCD_INITIAL_CLUSTER="etcd01=http://10.3.0.108:2380,etcd02=http://10.3.0.124:2380,etcd03=http://10.3.0.118:2380"
ETCD_ADVERTISE_CLIENT_URLS="http://10.3.0.124:2379"
ETCD_INITIAL_CLUSTER_TOKEN="etcd-c01"
ETCD_INITIAL_CLUSTER_STATE="new"
ETCD_DATA_DIR="/var/lib/etcd/default.etcd"
ETCD_NAME="etcd02"
ETCD_HEARTBEAT_INTERVAL=250
ETCD_ELECTION_TIMEOUT=1250
[root@psql02 etcd]#
 

[root@psql01 snap]# cat /etc/etcd/etcd.conf
ETCD_LISTEN_PEER_URLS="http://10.3.0.108:2380"
ETCD_LISTEN_CLIENT_URLS="http://localhost:2379,http://10.3.0.108:2379"
ETCD_INITIAL_ADVERTISE_PEER_URLS="http://10.3.0.108:2380"
ETCD_INITIAL_CLUSTER="etcd01=http://10.3.0.108:2380,etcd02=http://10.3.0.124:2380,etcd03=http://10.3.0.118:2380"
ETCD_ADVERTISE_CLIENT_URLS="http://10.3.0.108:2379"
ETCD_INITIAL_CLUSTER_TOKEN="etcd-c01"
ETCD_INITIAL_CLUSTER_STATE="new"
ETCD_DATA_DIR="/var/lib/etcd/default.etcd"
ETCD_NAME="etcd01"
ETCD_HEARTBEAT_INTERVAL=250
ETCD_ELECTION_TIMEOUT=1250
[root@psql01 snap]#

Then start each node:

systemctl start etcd

and verify:

[root@psql03 etcd]# etcdctl  cluster-health
member 83f149dc6ec1b00a is healthy: got healthy result from http://10.3.0.124:2379
member 93200353704b2d19 is healthy: got healthy result from http://10.3.0.108:2379
member a82b23223d9f684e is healthy: got healthy result from http://10.3.0.118:2379
cluster is healthy
[root@psql03 etcd]#

Next, go back and change the configuration of each ETCD member back to existing:

ETCD_INITIAL_CLUSTER_STATE="existing"

Restart all nodes by stopping ETCD first then starting it up on all nodes in close succession:

[root@psql01 etcd]# systemctl start etcd
[root@psql02 etcd]# systemctl start etcd
[root@psql03 etcd]# systemctl start etcd

At this point you should be good.  Now if you also get this message:

etcd[16312]: the clock difference against peer a82b23223d9f684e is too high [1.20986188s > 1s] (prober "ROUND_TRIPPER…MESSAGE")

Check your NTP configuration, run ntpdate <NTP SERVER> and restart NTPD: systemctl restart ntpd .   This should resolve it. 

Thx,
TK

Getting this?

Conflicting collector combinations in option list; please refer to the release notes for the combinations allowed

It could be because of this parameter in conjunction with the rest of the GC options specified.

ExecStart=/usr/bin/java -Xmx4096M -Xms2048M -XX:+UseG1GC -XX:MaxGCPauseMillis=200 -XX:+AlwaysPreTouch -XX:+UseConcMarkSweepGC -XX:+UseParNewGC -XX:+CMSIncrementalPacing -XX:ParallelGCThreads=4 -XX:+AggressiveOpts nogui -XX:+UnlockExperimentalVMOptions -XX:ActiveProcessorCount=4 -jar server.jar

But that wasn't the only thing.  It was also this option:

-XX:+UseParNewGC

Turns out that:

-XX:+UseG1GC

is a newer option that may conflict with the rest of the options as well. 

Thx,

Getting this?

C:\Users\tom>ping 10.0.0.1

Pinging 10.0.0.1 with 32 bytes of data:
Request timed out.
Request timed out.
Request timed out.
Request timed out.

Ping statistics for 10.0.0.1:
    Packets: Sent = 4, Received = 0, Lost = 4 (100% loss),

Read the rest of this entry »

Getting this?

Error populating transaction after 10 anaconda retries: failure: perl-HTTP-Tiny-0.033-3.el7.noarch.rpm from anaconda: [Errno 256] No more mirrors to try.

from the anaconda installer after rolling a custom ISO?

/tmp/yum.repos.d/anaconda.repo
[anaconda]
name=anaconda
enabled=1
baseurl=file:///run/install/repo/

Checking the primary.gz generated file list, the Packages/ path is included in the file path, but not our newly created primary.gz : 

[root@zeus01 repodata-backup]# zcat 16890efb08ba2667b3cfd83c4d234d5fabea890e6ed2ade4d4d7adec9670a9a5-primary.xml.gz|grep -Ei "Packages/"|head
  <location href="Packages/GeoIP-1.5.0-14.el7.x86_64.rpm"/>
  <location href="Packages/ModemManager-glib-1.6.10-3.el7_6.x86_64.rpm"/>
  <location href="Packages/NetworkManager-1.18.0-5.el7.x86_64.rpm"/>
  <location href="Packages/NetworkManager-adsl-1.18.0-5.el7.x86_64.rpm"/>
  <location href="Packages/NetworkManager-bluetooth-1.18.0-5.el7.x86_64.rpm"/>
  <location href="Packages/NetworkManager-glib-1.18.0-5.el7.x86_64.rpm"/>
  <location href="Packages/NetworkManager-libnm-1.18.0-5.el7.x86_64.rpm"/>
  <location href="Packages/NetworkManager-team-1.18.0-5.el7.x86_64.rpm"/>
  <location href="Packages/NetworkManager-tui-1.18.0-5.el7.x86_64.rpm"/>
  <location href="Packages/NetworkManager-wifi-1.18.0-5.el7.x86_64.rpm"/>
[root@zeus01 repodata-backup]# ls -altri 16890efb08ba2667b3cfd83c4d234d5fabea890e6ed2ade4d4d7adec9670a9a5-primary.xml.gz
806777 -rw-rw-r–. 1 root root 406843 Sep 11  2019 16890efb08ba2667b3cfd83c4d234d5fabea890e6ed2ade4d4d7adec9670a9a5-primary.xml.gz
[root@zeus01 repodata-backup]# zcat ac3a24e2e118c3a3f04d3a3b008c7d8859ee4d39327a542dcf6f46a17604e957-primary.xml.gz|grep -Ei "Packages/"|head
[root@zeus01 repodata-backup]# ls -altrid ac3a24e2e118c3a3f04d3a3b008c7d8859ee4d39327a542dcf6f46a17604e957-primary.xml.gz
884923 -rw-r–r–. 1 root root 410764 Mar 26 00:35 ac3a24e2e118c3a3f04d3a3b008c7d8859ee4d39327a542dcf6f46a17604e957-primary.xml.gz
[root@zeus01 repodata-backup]#

What was being ran is:

[root@zeus01 working]# createrepo -dpo . Packages/ -g /zeus-images/working/repodata/521f322f05f9802f2438d8bb7d97558c64ff3ff74c03322d77787ade9152d8bb-c7-x86_64-comps.xml
[root@zeus01 working]# pwd
/zeus-images/working
[root@zeus01 working]#

To include the package path Packages/, generate the repo using this command and specify the root path ( / ) of the DVD / ISO files instead of Packages/ :

[root@zeus01 working]# createrepo -dpo . ./ -g /zeus-images/working/repodata/521f322f05f9802f2438d8bb7d97558c64ff3ff74c03322d77787ade9152d8bb-c7-x86_64-comps.xml             Spawning worker 0 with 236 pkgs
Spawning worker 1 with 236 pkgs
Spawning worker 2 with 236 pkgs
Spawning worker 3 with 235 pkgs
Spawning worker 4 with 235 pkgs
Spawning worker 5 with 235 pkgs
Workers Finished
Saving Primary metadata
Saving file lists metadata
Saving other metadata
Generating sqlite DBs
Sqlite DBs complete
[root@zeus01 working]# ls -altri
total 176
136726261 -rw-rw-r–. 1 root     root      1690 Dec  9  2015 RPM-GPG-KEY-CentOS-Testing-7
136726260 -rw-rw-r–. 1 root     root      1690 Dec  9  2015 RPM-GPG-KEY-CentOS-7
136726257 -rw-rw-r–. 1 root     root     18009 Dec  9  2015 GPL
136726256 -rw-rw-r–. 1 root     root       227 Aug 30  2017 EULA
136726253 -rw-r–r–. 1 root     root        29 Sep  6  2019 .discinfo
   806768 drwxr-xr-x. 2 root     root        41 Sep  6  2019 LiveOS
202843815 drwxr-xr-x. 3 root     root        33 Sep  6  2019 EFI
136726254 -rw-r–r–. 1 root     root       354 Sep  6  2019 .treeinfo
136726252 drwxr-xr-x. 3 root     root        54 Sep  9  2019 images
136726255 -rw-rw-r–. 1 root     root        14 Sep  9  2019 CentOS_BuildTag
136726262 -r–r–r–. 1 root     root      3101 Dec  1 17:11 TRANS.TBL
213637698 drwxr-xr-x. 2 root     root      4096 Mar 30 01:26 repodata-jic
 68834653 drwxrwxr-x. 2 root     root     77824 Mar 30 01:38 Packages
202843816 drwxr-xr-x. 2 root     root      4096 Mar 30 09:26 isolinux
136726269 drwxr-xr-x. 6 zeusadmin zeusadmin  4096 Apr  4 10:54 ..
135668264 -rw——-. 1 root     root      1050 Apr  4 11:09 ks.cfg
135668262 drwxr-xr-x. 2 root     root      4096 Apr  4 16:47 repodata
136726251 drwxr-xr-x. 9 root     root      4096 Apr  4 16:47 .
[root@zeus01 working]#
[root@zeus01 repodata]# pwd
/zeus-images/working/repodata
[root@zeus01 repodata]#

Now create the ISO:

[root@zeus01 working]# mkisofs -o ../CentOS-7-x86_64-Minimal-1908-Kickstart-Context-V24.iso -untranslated-filenames -b isolinux/isolinux.bin -c isolinux/boot.cat -no-emul-boot -boot-load-size 4 -boot-info-table -V "CentOS 7 x86_64" -R -J -v -T .
Warning: creating filesystem that does not conform to ISO-9660.
I: -input-charset not specified, using utf-8 (detected in locale settings)

Or using genisoimage command:

[root@one01 working]# genisoimage -untranslated-filenames -volid 'CentOS 7 x86_64' -J -joliet-long -rational-rock -translation-table -input-charset utf-8 -x  ./lost+found -b isolinux/isolinux.bin -c isolinux/boot.cat -no-emul-boot -boot-load-size 4 -boot-info-table -eltorito-alt-boot -e images/efiboot.img  -no-emul-boot -o ../CentOS-7-x86_64-Minimal-1908-Kickstart-Context-GV23.iso -R -v -T .
Warning: creating filesystem that does not conform to ISO-9660.
genisoimage 1.1.11 (Linux)

Try booting using the image.  Your RPM packages should now be found. 

Thx,
DN

Getting this?  

repodata/4712268f3e1cd9ec4303a41b290a9932d1ad22a7d21c60311296a0447cea75a1-primary.xml.gz from anaconda: [Errno 256] No more mirrors to try.
file:///run/install/repo/repodata/4712268f3e1cd9ec4303a41b290a9932d1ad22a7d21c60311296a0447cea75a1-primary.xml.gz: [Errno -1] Metadata file does not match checksum

Read the rest of this entry »

So you're stuck in the Anaconda installer and can't copy files to and from to troubleshoot an issue?  

Could not load host key /etc/ssh/ssh_host_rsa_key

or you're getting something like this after some FS XFS corruption due to storage failure:  

[  358.344254] XFS (dm-0): Metadata corruption detected at xfs_inode_buf_verify+0x79/0x100 [xfs], xfs_inode block 0x3c44aa0
[  358.349741] XFS (dm-0): Unmount and run xfs_repair
[  358.352350] XFS (dm-0): First 64 bytes of corrupted metadata buffer:
[  358.355398] ffff92237a9e2000: 35 64 64 39 38 38 37 36 35 34 36 65 39 30 65 66  5dd98876546e90ef
[  358.359107] ffff92237a9e2010: 66 65 36 32 64 38 34 30 37 37 62 65 36 66 33 37  fe62d84077be6f37
[  358.362794] ffff92237a9e2020: 62 62 37 65 62 63 31 66 61 37 31 38 65 35 36 64  bb7ebc1fa718e56d
[  358.366488] ffff92237a9e2030: 34 30 62 32 64 33 63 36 35 30 34 66 33 62 38 66  40b2d3c6504f3b8f
[  358.370229] XFS (dm-0): metadata I/O error: block 0x3c44aa0 ("xfs_trans_read_buf_map") error 117 numblks 16
[  358.375987] XFS (dm-0): I/O Error Detected. Shutting down filesystem
[  358.379058] XFS (dm-0): Please umount the filesystem and rectify the problem(s)

Run the following commands to enable networking and the SSHD daemon:  

1. ssh-keygen
2. cp /root/.ssh/id_rsa /etc/ssh/ssh_host_rsa_key
3. cp /etc/ssh/sshd_config.anaconda /etc/ssh/sshd_config
4. /sbin/sshd -p 22
5. ip addr add 10.0.0.100/24 dev ens3
6. (THIS) ip route add 10.0.0.0/24 via 10.0.0.1 dev ens3
7. (OR) ip route add default via 10.0.0.1 dev ens3
8. ip link set ens3 up

SSH in to troubleshoot:  

[root@remote01 ~]# ssh 10.0.0.100
Warning: Permanently added '10.0.0.100' (RSA) to the list of known hosts.
[anaconda root@localhost ~]#

Particularly handy when troubleshooting boot issues. Running tail -f /tmp/*.log will now be possible remotely while troubleshooting boot problems.  

Use the same procedure to access corrupted filesystems on environments.  

BONUS: Repair XFS Filesystem in Maintenance

Here's how to recover a corrupted XFS filesystem by logging into the system using the above steps.

vgscan -v –mknodes
vgchange -a y
ls -altri /dev/centos/*

Verify LV's exist.  Repair.  Example:

xfs_repair /dev/centos/home
xfs_repair /dev/centos/root

If you run into this:

ERROR: The filesystem has valuable metadata changes in a log which needs to
be replayed.  Mount the filesystem to replay the log, and unmount it before
re-running xfs_repair.  If you are unable to mount the filesystem, then use
the -L option to destroy the log and attempt a repair.
Note that destroying the log may cause corruption — please attempt a mount
of the filesystem before doing this.

Simply follow instructions and mount it:

[anaconda root@localhost ~]# mkdir lv-root
[anaconda root@localhost ~]# mount /dev/centos/root lv-root
[anaconda root@localhost ~]# cd lv-root/
[anaconda root@localhost lv-root]# ls
bin   dev  home  lib64       mds.xyz  misc  n    nix.mds.xyz  proc  run   srv               sys  usr
boot  etc  lib   lost+found  media    mnt   net  opt          root  sbin  sssd-updates.txt  tmp  var
[anaconda root@localhost lv-root]#

This replays the journal log.  After this, you should be able to boot into the system. 

BONUS: Install RPM in recovery.

Once you complete the above, mount the failed systems folders over a directory of your choice.  For example: 

mkdir lv-root; mount /dev/mapper/centos-root lv-root
mkdir lv-home mount /dev/mapper/centos-home lv-home

chroot lv-root

mknod -m 666 /dev/random c 1 8
mknod -m 666 /dev/urandom c 1 9

Install and update a package in an attempt to fix the environment.

[anaconda root@localhost /]# yum search kernel
================== N/S matched: kernel ===========================
<SOME PACKAGE>.x86_64 : The Linux kernel

Then install:

[anaconda root@localhost /]# yum update <SOME PACKAGE>.x86_64
Installed:
  <SOME PACKAGE.x86_64 0:3.10.0-1062.18.1.el7

Dependency Updated:
  <SOME PACKAGE DEPENDENCY>.noarch 0:20190429-72.gitddde598.el7

Complete!
[anaconda root@localhost /]#

BONUS: Install RPM in recovery. ( METHOD 2 )

1. Mount the ISO or DVD.  Ideally the Everything DVD of CentOS.
2. Select Troubleshooting -> Rescue a CentOS Linux System -> Continue.
3. Once on the command prompt, type cd /mnt/install/repo/Packages
4. mkdir /mnt/sysimage/root/kernel; cp -ip kernel* /mnt/sysimage/root/kernel
5. chroot /mnt/sysimage
6. cd /root/kernel
7. yum install kernel*
8. ( Optional )  Edit /etc/default/grub in case custom parameters are needed.
9. grub2-mkconfig -o /boot/grub2/grub.cfg
10. exit
11. sync;sync; reboot
12. Login and Enjoy!

The above steps are a great replacement for single user mode such as using single or init=/bin/bash.  The filesystems on the original OS remain unmounted allowing a full range of troubleshooting options. 

Hope this helps.

Thx,

Getting this?

Thu Mar 26 01:27:44 2020 [Z0][IPM][D]: Message received: GET_ADDRESS SUCCESS 296 QUREUkVTUyA9IFsgSVAgPSAiMTAuMC4wLjEwMiIsIFNJWkUgPSAiMSIgXQo=

Thu Mar 26 01:27:44 2020 [Z0][IPM][E]: Address returned by IPAM are not within AR or in use
Thu Mar 26 01:27:44 2020 [Z0][ONE][E]: Cannot get IP/MAC lease from virtual network 26.
Thu Mar 26 01:27:44 2020 [Z0][ReM][E]: Req:560 UID:0 one.template.instantiate result FAILURE [one.template.instantiate] Error allocating a new virtual machine template. Cannot get IP/MAC lease from virtual network 26.

Read the rest of this entry »

Getting this?

[root@mdskvm-p05 ~]# ping 10.0.0.73
PING 10.0.0.73 (10.0.0.73) 56(84) bytes of data.
From 10.0.0.72 icmp_seq=1 Destination Host Unreachable
From 10.0.0.72 icmp_seq=2 Destination Host Unreachable
From 10.0.0.72 icmp_seq=3 Destination Host Unreachable

Read the rest of this entry »

Getting this?

Mar 15 18:28:12 mysql03.nix.mds.xyz mysqld[17578]: 2020-03-15T22:28:12.453960Z 0 [ERROR] WSREP: It may not be safe to bootstrap the cluster from this node. It was not the last oneave the cluster and may not contain all the updates. To force cluster bootstrap with this node, edit the grastate.dat file manually and set safe_to_bootstrap to 1 .

Read the rest of this entry »

Getting this?

Mon Mar 9 00:08:04 2020 [Z0][VMM][I]: error: operation failed: domain 'one-35' already exists with uuid b1c73d1b-db71-4d5f-a0f2-f12031332bbc
Mon Mar 9 00:08:04 2020 [Z0][VMM][E]: Could not restore from /var/lib/one//datastores/129/35/checkpoint
Mon Mar 9 00:08:04 2020 [Z0][VMM][E]: restore: Command "virsh –connect qemu:///system restore /var/lib/one//datastores/129/35/checkpoint –xml /var/lib/one//datastores/129/35/checkpoint.xml" failed: error: Failed to restore domain from /var/lib/one//datastores/129/35/checkpoint
Mon Mar 9 00:08:04 2020 [Z0][VMM][I]: error: operation failed: domain 'one-35' already exists with uuid b1c73d1b-db71-4d5f-a0f2-f12031332bbc
Mon Mar 9 00:08:04 2020 [Z0][VMM][E]: Could not restore from /var/lib/one//datastores/129/35/checkpoint
Mon Mar 9 00:08:04 2020 [Z0][VMM][I]: ExitCode: 1

Read the rest of this entry »