Header Shadow Image


tcpdump

This is how to get detailed TCP dumps of your network traffic while avoiding your own PuTTY traffic in the output:

tcpdump -w trace.dat -s 0 port not 22
tcpdump -r trace.dat -nnvvveXXS > trace.dat.txt

Cheers,
TK

mount.nfs: mount(2): Permission denied

You're getting this message:

[root@mysql01 /]# mount -v nfs03:/n /m
mount.nfs: timeout set for Thu Nov  8 23:37:04 2018
mount.nfs: trying text-based options 'vers=4.1,addr=192.168.0.125,clientaddr=192.168.0.126'
mount.nfs: mount(2): No such file or directory
mount.nfs: trying text-based options 'addr=192.168.0.125'
mount.nfs: prog 100003, trying vers=3, prot=6
mount.nfs: trying 192.168.0.125 prog 100003 vers 3 prot TCP port 2049
mount.nfs: prog 100005, trying vers=3, prot=17
mount.nfs: trying 192.168.0.125 prog 100005 vers 3 prot UDP port 20048
mount.nfs: mount(2): Permission denied
mount.nfs: access denied by server while mounting nfs03:/n
[root@mysql01 /]#

It's probably because you might be using Haproxy and it's configured like this for the backends when nfs01 is down:

    server      nfs01.nix.mds.xyz    nfs01.nix.mds.xyz:2049 check
    server      nfs02.nix.mds.xyz    nfs01.nix.mds.xyz:2049 check
    server      nfs03.nix.mds.xyz    nfs01.nix.mds.xyz:2049 check

Notice how each one was pointing to the same nfs01 that was down.  Needed to change this to nfs02 and nfs03 for the second and third line.  

Oh well, shit happens.  Worked pretty good after that fix.

Cheers,
TK

Postgres SQL HA Cluster ( Quick Start Guide )

We will be keeping the Postgres SQL HA Cluster configuration / setup very very brief using only the bare essentials to get it up and running.

Before we begin, we need to plan things out on our CentOS 7.X servers.  First, fill in this table of what your cluster is supposed to look like when done:

NAME ADDRESS HOSTNAME SERVICES
psql01 192.168.0.108 psql01.nix.mds.xyz PostgreSQL, ETCD, Patroni, HAPROXY, Keepalived
psql02 192.168.0.124 psql02.nix.mds.xyz PostgreSQL, ETCD, Patroni, HAPROXY, Keepalived
psql03 192.168.0.118 psql03.nix.mds.xyz PostgreSQL, ETCD, Patroni, HAPROXY, Keepalived
psql-c01 (VIP) 192.168.0.112 psql-c01.nix.mds.xyz  

Read the rest of this entry »

Forbidden You don’t have permission to access /repos/ on this server.

So you get the following message when installing and configuring your HTTPD server?  Despite the right configuration you still receive the following:

Forbidden

You don't have permission to access /repos/ on this server.

Read the rest of this entry »

1765328228 Cannot contact any KDC for realm

When seeing this:

krb5_child.log:(Tue May 22 02:06:15 2018) [[sssd[krb5_child[1605]]]] [map_krb5_error] (0×0020): 1657: [-1765328228][Cannot contact any KDC for realm 'MDS.XYZ']

Access denied
Using keyboard-interactive authentication.
Password:

reverse the order of your DNS hosts in /etc/resolv.conf to this:

[root@cm-r01dn07 sssd]# cat /etc/resolv.conf
search mds.xyz nix.mds.xyz
nameserver 192.168.0.224
nameserver 192.168.0.44
nameserver 192.168.0.45
[root@cm-r01dn07 sssd]#

from this:

[root@cm-r01dn07 sssd]# cat /etc/resolv.conf
search mds.xyz nix.mds.xyz
nameserver 192.168.0.44
nameserver 192.168.0.45
nameserver 192.168.0.224

[root@cm-r01dn07 sssd]#

And that solved it.

Cheers,
TK

sssd krb5_child Key table entry not found

When you get this message:

May 21 00:13:31 nfs03.nix.mds.xyz [sssd[krb5_child[1822]]][1822]: Key table entry not found

followed by:

[[sssd[krb5_child[1752]]]] [k5c_setup_fast] (0×0020): 2628: [-1765328203][Key table entry not found]

or similar, dig into the logs further to see this:

(Mon May 21 00:13:33 2018) [[sssd[krb5_child[1824]]]] [find_principal_in_keytab] (0×0400): No principal matching host/nfs02.nix.mds.xyz@NIX.MDS.XYZ found in keytab.
(Mon May 21 00:13:33 2018) [[sssd[krb5_child[1824]]]] [check_fast_ccache] (0×0080): find_principal_in_keytab failed for principal host/nfs02.nix.mds.xyz@NIX.MDS.XYZ.
[root@nfs03 sssd]#

Then check your /etc/krb5.conf file:

[root@nfs03 etc]# grep -Ei nfs02 *
krb5.conf:  nfs02.nix.mds.xyz = NIX.MDS.XYZ

And also here:

[root@nfs03 etc]# grep -EiR nfs02 * 2>/dev/null
sssd/sssd.conf:ipa_hostname = nfs02.nix.mds.xyz
sssd/sssd.conf-new:ipa_hostname = nfs02.nix.mds.xyz
[root@nfs03 etc]#

And change accordingly.  The issue resulted from copying the same files from another host, nfs02.

Cheers,
TK

 

Saving random seed failed. / No kdump initial ramdisk found. / Failed to run mkdumprd

Kdump doesn't start?

[root@mbpc-pc grub]# service kdump restart
Memory for crashkernel is not reserved
Please reserve memory by passing "crashkernel=X@Y" parameter to the kernel
Stopping kdump:                                            [FAILED]
No kdump initial ramdisk found.                            [WARNING]
Rebuilding /boot/initrd-4.8.4kdump.img
Saving random seed failed.
Failed to run mkdumprd
[root@mbpc-pc grub]#

Then create the ramdom-seed file like this:

dd if=/dev/urandom of=/var/lib/random-seed bs=1024 count=1

Run

bash -x /etc/init.d/kdump start

to get the command line or simply:

[root@mbpc-pc grub]# /sbin/mkdumprd -d -f –allow-missing /boot/initrd-4.8.4kdump.img 4.8.4
[root@mbpc-pc grub]# ls -altri /boot/initrd-4.8.4kdump.img
71 -rw——-. 1 root root 8852315 Apr 22 12:46 /boot/initrd-4.8.4kdump.img
[root@mbpc-pc grub]#

to get an initial kdump.img going.  Try to restart the kdump daemon:

[root@mbpc-pc grub]# /etc/init.d/kdump restart
Memory for crashkernel is not reserved
Please reserve memory by passing "crashkernel=X@Y" parameter to the kernel
Stopping kdump:                                            [FAILED]
Memory for crashkernel is not reserved
Please reserve memory by passing "crashkernel=X@Y" parameter to the kernel
Starting kdump:                                            [FAILED]
[root@mbpc-pc grub]#

So I checked /boot/grub/grub.conf and had this:

crashkernel=256M

Instead of:

crashkernel=256M@32M

Make the change and restart the system because we need to reload the kernel to take the change into effect.  Now also be careful what values you pick as many sites will suggest variations:

Picking:

crashkernel=256M@16M

results in:

crashkernel reservation failed – memory is in use.

Using:

crashkernel=auto

results in:

kexec_core: crashkernel: memory value expected

Specifying:

Reserving 256MB of memory at 592MB for crashkernel (System RAM: 4092MB)

actually autoallocates a free 256MB chunk at an auto determined offset.  However still get's us the familiar startup message above.  So we need to dig deeper.  We try bash -x /etc/init.d/kdump start to see what the issue is and to our surprise, things start up just fine:

[root@mbpc-pc ~]# /etc/init.d/kdump start
Memory for crashkernel is not reserved
Please reserve memory by passing "crashkernel=X@Y" parameter to the kernel
Starting kdump:                                            [FAILED]
[root@mbpc-pc ~]#
[root@mbpc-pc ~]#
[root@mbpc-pc ~]#
[root@mbpc-pc ~]# bash -x /etc/init.d/kdump start
+ . /etc/init.d/functions
++ TEXTDOMAIN=initscripts
++ umask 022
++ PATH=/sbin:/usr/sbin:/bin:/usr/bin
++ export PATH
++ '[' -z '' ']'
++ COLUMNS=80
++ '[' -z '' ']'
+++ /sbin/consoletype
++ CONSOLETYPE=pty
++ '[' -f /etc/sysconfig/i18n -a -z '' -a -z '' ']'
++ . /etc/profile.d/lang.sh
++ unset LANGSH_SOURCED
++ '[' -z '' ']'
++ '[' -f /etc/sysconfig/init ']'
++ . /etc/sysconfig/init
+++ BOOTUP=color
+++ RES_COL=60
+++ MOVE_TO_COL='echo -en \033[60G'
+++ SETCOLOR_SUCCESS='echo -en \033[0;32m'
+++ SETCOLOR_FAILURE='echo -en \033[0;31m'
+++ SETCOLOR_WARNING='echo -en \033[0;33m'
+++ SETCOLOR_NORMAL='echo -en \033[0;39m'
+++ PROMPT=yes
+++ AUTOSWAP=no
+++ ACTIVE_CONSOLES='/dev/tty[1-6]'
+++ SINGLE=/sbin/sushell
++ '[' pty = serial ']'
++ __sed_discard_ignored_files='/\(~\|\.bak\|\.orig\|\.rpmnew\|\.rpmorig\|\.rpmsave\)$/d'
+++ cat /proc/cmdline
++ strstr 'ro root=/dev/mapper/mbpcvg-rootlv rd_LVM_LV=mbpcvg/rootlv rd_LVM_LV=VGEntertain/olv_swap rd_LVM_LV=mbpcvg/swaplv rd_NO_LUKS rd_NO_MD rd_NO_DM LANG=en_US.UTF-8 SYSFONT=latarcyrheb-sun16 KEYBOARDTYPE=pc KEYTABLE=us rhgb nomodeset irqpoll pcie_aspm=off amd_iommu=on crashkernel=256M pci=nomsi' rc.debug
++ '[' 'ro root=/dev/mapper/mbpcvg-rootlv rd_LVM_LV=mbpcvg/rootlv rd_LVM_LV=VGEntertain/olv_swap rd_LVM_LV=mbpcvg/swaplv rd_NO_LUKS rd_NO_MD rd_NO_DM LANG=en_US.UTF-8 SYSFONT=latarcyrheb-sun16 KEYBOARDTYPE=pc KEYTABLE=us rhgb nomodeset irqpoll pcie_aspm=off amd_iommu=on crashkernel=256M pci=nomsi' = 'ro root=/dev/mapper/mbpcvg-rootlv rd_LVM_LV=mbpcvg/rootlv rd_LVM_LV=VGEntertain/olv_swap rd_LVM_LV=mbpcvg/swaplv rd_NO_LUKS rd_NO_MD rd_NO_DM LANG=en_US.UTF-8 SYSFONT=latarcyrheb-sun16 KEYBOARDTYPE=pc KEYTABLE=us rhgb nomodeset irqpoll pcie_aspm=off amd_iommu=on crashkernel=256M pci=nomsi' ']'
++ return 1
+ KEXEC=/sbin/kexec
+ BOOTDIR=/boot
+ KDUMP_KERNELVER=
+ KDUMP_COMMANDLINE=
+ KDUMP_IDE_NOPROBE_COMMANDLINE=
+ KEXEC_ARGS=
+ KDUMP_CONFIG_FILE=/etc/kdump.conf
+ MEM_RESERVED=
+ MKDUMPRD_ARGS=
+ CLUSTER_CONFIG_FILE=/etc/cluster/cluster.conf
+ FENCE_KDUMP_CONFIG=/etc/sysconfig/fence_kdump
+ SSH_KEY_LOCATION=/root/.ssh/kdump_id_rsa
+ DEFAULT_DUMP_MODE=kdump
+ LOGGER='/usr/bin/logger -p info -t kdump'
+ standard_kexec_args=-p
+ '[' -f /etc/sysconfig/kdump ']'
+ . /etc/sysconfig/kdump
++ KDUMP_KERNELVER=
++ KDUMP_COMMANDLINE=
++ KDUMP_COMMANDLINE_APPEND='irqpoll nr_cpus=1 reset_devices cgroup_disable=memory mce=off acpi_no_memhotplug'
++ MKDUMPRD_ARGS=–allow-missing
++ KEXEC_ARGS=
++ KDUMP_BOOTDIR=/boot
++ KDUMP_IMG=vmlinuz
++ KDUMP_IMG_EXT=
+ single_instance_lock
+ exec
+ flock 9
+ determine_dump_mode
+ fadump_enabled_sys_node=/sys/kernel/fadump_enabled
+ '[' -f /sys/kernel/fadump_enabled ']'
+ case "$1" in
+ '[' kdump == fadump ']'
+ '[' -s /proc/vmcore ']'
+ start
+ sestatus
+ grep -q 'SELinux status.*enabled'
+ selinux_relabel
+ local _path _i _attr
++ path_to_be_relabeled
++ local _path _target _mnt=/ _rmnt
++ is_dump_target_configured
++ local _target
+++ egrep '^ext[234]|^xfs|^btrfs|^raw|^ssh|^nfs|^nfs4|^net' /etc/kdump.conf
++ _target=
++ '[' -n '' ']'
+++ get_save_path
++++ grep '^path' /etc/kdump.conf
++++ awk '{print $2}'
+++ local _save_path=/var/crash
+++ '[' -z /var/crash ']'
+++ echo /var/crash
++ _path=/var/crash
+++ df ///var/crash
+++ tail -1
+++ awk '{ print $NF }'
++ _rmnt=/
++ [[ / == \/ ]]
++ echo ///var/crash
+ _path=///var/crash
+ '[' -z ///var/crash ']'
+ '[' -d ///var/crash ']'
++ find ///var/crash
+ for _i in '$(find $_path)'
++ getfattr -m security.selinux ///var/crash
+ _attr='# file: var/crash
security.selinux'
+ '[' -z '# file: var/crash
security.selinux' ']'
+ save_raw
++ awk '$1 ~ /^raw$/ { print $2; }' /etc/kdump.conf
+ local raw_part=
+ local kdump_dir
+ '[' '' ']'
+ return 0
+ '[' 0 -ne 0 ']'
+ status
+ '[' kdump == fadump ']'
+ '[' '!' -e /sys/kernel/kexec_crash_loaded ']'
+ in_xen_pv_guest
+ grep -q 'xen-percpu-virq  *timer0' /proc/interrupts
+ in_xen_hvm_guest
+ grep -q xen /sys/hypervisor/type
++ cat /sys/kernel/kexec_crash_loaded
+ rc=0
+ '[' 0 == 1 ']'
+ return 1
+ rc=1
+ '[' 1 == 2 ']'
+ '[' 1 == 0 ']'
+ '[' kdump '!=' fadump ']'
+ check_kernel_parameter
+ '[' -z '' ']'
++ cat /proc/cmdline
+ KDUMP_COMMANDLINE='ro root=/dev/mapper/mbpcvg-rootlv rd_LVM_LV=mbpcvg/rootlv rd_LVM_LV=VGEntertain/olv_swap rd_LVM_LV=mbpcvg/swaplv rd_NO_LUKS rd_NO_MD rd_NO_DM LANG=en_US.UTF-8 SYSFONT=latarcyrheb-sun16 KEYBOARDTYPE=pc KEYTABLE=us rhgb nomodeset irqpoll pcie_aspm=off amd_iommu=on crashkernel=256M pci=nomsi'
++ cat /sys/kernel/kexec_crash_size
+ MEM_RESERVED=268435456
+ '[' 268435456 -eq 0 ']'
+ return 0
+ '[' 0 '!=' 0 ']'
+ check_config
+ '[' kdump == fadump ']'
+ check_kdump_config
+ local modified_files=
+ local force_rebuild=0
+ MKDUMPRD='/sbin/mkdumprd -d -f –allow-missing'
++ grep '^force_rebuild' /etc/kdump.conf
++ cut '-d ' -f2
+ force_rebuild=
+ '[' -n '' ']'
+ '[' -z '' ']'
++ uname -r
+ local running_kernel=4.8.4
++ echo 4.8.4
++ sed s/smp//g
+ kdump_kver=4.8.4
+ kdump_kernel=/boot/vmlinuz-4.8.4
+ kdump_initrd=/boot/initrd-4.8.4kdump.img
+ '[' '!' -f /boot/vmlinuz-4.8.4 ']'
+ '[' '!' -f /boot/initrd-4.8.4kdump.img ']'
+ '[' -z '' ']'
++ stat -c %Y /boot/initrd-4.8.4kdump.img
+ image_time=1524415601
++ grep '^kdump_post' /etc/kdump.conf
++ cut '-d ' -f2
+ EXTRA_FILES=
++ grep '^kdump_pre' /etc/kdump.conf
++ cut '-d ' -f2
+ CHECK_FILE=
+ EXTRA_FILES=' '
++ grep '^extra_modules' /etc/kdump.conf
++ cut '-d ' -f2-
+ CHECK_FILE=
+ EXTRA_FILES='  '
++ grep '^extra_bins' /etc/kdump.conf
++ cut '-d ' -f2-
+ CHECK_FILE=
+ EXTRA_FILES='   '
++ grep '^extra_modules' /etc/kdump.conf
+ FORCE_REBUILD=
+ files='/etc/kdump.conf /boot/vmlinuz-4.8.4    '
+ grep -q '^fence_kdump_nodes' /etc/kdump.conf
+ '[' -f /etc/cluster/cluster.conf ']'
+ for file in '$files'
+ time_stamp=0
+ '[' -f /etc/kdump.conf ']'
++ stat -c %Y /etc/kdump.conf
+ time_stamp=1524414829
+ '[' 1524414829 -gt 1524415601 ']'
+ for file in '$files'
+ time_stamp=0
+ '[' -f /boot/vmlinuz-4.8.4 ']'
++ stat -c %Y /boot/vmlinuz-4.8.4
+ time_stamp=1477845416
+ '[' 1477845416 -gt 1524415601 ']'
+ '[' -n '' -a '!= ' ']'
+ '[' -n '' -a '!= ' ']'
+ in_xen_hvm_guest
+ grep -q xen /sys/hypervisor/type
+ return 0
+ return 0
+ '[' 0 '!=' 0 ']'
+ start_dump
+ '[' kdump == fadump ']'
+ load_kdump
++ uname -m
+ ARCH=x86_64
++ awk '/Slab:.*/ {print $2}' /proc/meminfo
+ KMEMINUSE=152036
++ dc '-e268435456 1024 / p'
+ MEM_RESERVED=262144
++ dc '-e262144 .7 * 10 * 10 / p'
+ MEM_RESERVED=183500
+ '[' x86_64 '!=' i686 -a x86_64 '!=' i386 -a x86_64 '!=' x86_64 ']'
+ '[' x86_64 == i686 -o x86_64 == i386 ']'
+ '[' -f /sys/firmware/efi/systab ']'
+ echo 'irqpoll nr_cpus=1 reset_devices cgroup_disable=memory mce=off acpi_no_memhotplug'
+ grep -q nr_cpus
++ uname -r
+ ver=4.8.4
++ echo 4.8.4
++ cut -d- -f1
+ maj=4.8.4
++ echo 4.8.4
++ cut -d- -f2
+ min=4.8.4
+ min=4
+ '[' 4.8.4 = 2.6.32 ']'
++ prepare_cmdline
++ local cmdline
++ '[' -z 'ro root=/dev/mapper/mbpcvg-rootlv rd_LVM_LV=mbpcvg/rootlv rd_LVM_LV=VGEntertain/olv_swap rd_LVM_LV=mbpcvg/swaplv rd_NO_LUKS rd_NO_MD rd_NO_DM LANG=en_US.UTF-8 SYSFONT=latarcyrheb-sun16 KEYBOARDTYPE=pc KEYTABLE=us rhgb nomodeset irqpoll pcie_aspm=off amd_iommu=on crashkernel=256M pci=nomsi' ']'
++ cmdline='ro root=/dev/mapper/mbpcvg-rootlv rd_LVM_LV=mbpcvg/rootlv rd_LVM_LV=VGEntertain/olv_swap rd_LVM_LV=mbpcvg/swaplv rd_NO_LUKS rd_NO_MD rd_NO_DM LANG=en_US.UTF-8 SYSFONT=latarcyrheb-sun16 KEYBOARDTYPE=pc KEYTABLE=us rhgb nomodeset irqpoll pcie_aspm=off amd_iommu=on crashkernel=256M pci=nomsi'
+++ remove_cmdline_param 'ro root=/dev/mapper/mbpcvg-rootlv rd_LVM_LV=mbpcvg/rootlv rd_LVM_LV=VGEntertain/olv_swap rd_LVM_LV=mbpcvg/swaplv rd_NO_LUKS rd_NO_MD rd_NO_DM LANG=en_US.UTF-8 SYSFONT=latarcyrheb-sun16 KEYBOARDTYPE=pc KEYTABLE=us rhgb nomodeset irqpoll pcie_aspm=off amd_iommu=on crashkernel=256M pci=nomsi' crashkernel mem hugepages hugepagesz
+++ local 'cmdline=ro root=/dev/mapper/mbpcvg-rootlv rd_LVM_LV=mbpcvg/rootlv rd_LVM_LV=VGEntertain/olv_swap rd_LVM_LV=mbpcvg/swaplv rd_NO_LUKS rd_NO_MD rd_NO_DM LANG=en_US.UTF-8 SYSFONT=latarcyrheb-sun16 KEYBOARDTYPE=pc KEYTABLE=us rhgb nomodeset irqpoll pcie_aspm=off amd_iommu=on crashkernel=256M pci=nomsi'
+++ shift
+++ for arg in '$@'
++++ echo ro root=/dev/mapper/mbpcvg-rootlv rd_LVM_LV=mbpcvg/rootlv rd_LVM_LV=VGEntertain/olv_swap rd_LVM_LV=mbpcvg/swaplv rd_NO_LUKS rd_NO_MD rd_NO_DM LANG=en_US.UTF-8 SYSFONT=latarcyrheb-sun16 KEYBOARDTYPE=pc KEYTABLE=us rhgb nomodeset irqpoll pcie_aspm=off amd_iommu=on crashkernel=256M pci=nomsi
++++ sed -e 's/\bcrashkernel=[^ ]*\b//g' -e 's/\bcrashkernel\b//g'
+++ cmdline='ro root=/dev/mapper/mbpcvg-rootlv rd_LVM_LV=mbpcvg/rootlv rd_LVM_LV=VGEntertain/olv_swap rd_LVM_LV=mbpcvg/swaplv rd_NO_LUKS rd_NO_MD rd_NO_DM LANG=en_US.UTF-8 SYSFONT=latarcyrheb-sun16 KEYBOARDTYPE=pc KEYTABLE=us rhgb nomodeset irqpoll pcie_aspm=off amd_iommu=on  pci=nomsi'
+++ for arg in '$@'
++++ echo ro root=/dev/mapper/mbpcvg-rootlv rd_LVM_LV=mbpcvg/rootlv rd_LVM_LV=VGEntertain/olv_swap rd_LVM_LV=mbpcvg/swaplv rd_NO_LUKS rd_NO_MD rd_NO_DM LANG=en_US.UTF-8 SYSFONT=latarcyrheb-sun16 KEYBOARDTYPE=pc KEYTABLE=us rhgb nomodeset irqpoll pcie_aspm=off amd_iommu=on pci=nomsi
++++ sed -e 's/\bmem=[^ ]*\b//g' -e 's/\bmem\b//g'
+++ cmdline='ro root=/dev/mapper/mbpcvg-rootlv rd_LVM_LV=mbpcvg/rootlv rd_LVM_LV=VGEntertain/olv_swap rd_LVM_LV=mbpcvg/swaplv rd_NO_LUKS rd_NO_MD rd_NO_DM LANG=en_US.UTF-8 SYSFONT=latarcyrheb-sun16 KEYBOARDTYPE=pc KEYTABLE=us rhgb nomodeset irqpoll pcie_aspm=off amd_iommu=on pci=nomsi'
+++ for arg in '$@'
++++ echo ro root=/dev/mapper/mbpcvg-rootlv rd_LVM_LV=mbpcvg/rootlv rd_LVM_LV=VGEntertain/olv_swap rd_LVM_LV=mbpcvg/swaplv rd_NO_LUKS rd_NO_MD rd_NO_DM LANG=en_US.UTF-8 SYSFONT=latarcyrheb-sun16 KEYBOARDTYPE=pc KEYTABLE=us rhgb nomodeset irqpoll pcie_aspm=off amd_iommu=on pci=nomsi
++++ sed -e 's/\bhugepages=[^ ]*\b//g' -e 's/\bhugepages\b//g'
+++ cmdline='ro root=/dev/mapper/mbpcvg-rootlv rd_LVM_LV=mbpcvg/rootlv rd_LVM_LV=VGEntertain/olv_swap rd_LVM_LV=mbpcvg/swaplv rd_NO_LUKS rd_NO_MD rd_NO_DM LANG=en_US.UTF-8 SYSFONT=latarcyrheb-sun16 KEYBOARDTYPE=pc KEYTABLE=us rhgb nomodeset irqpoll pcie_aspm=off amd_iommu=on pci=nomsi'
+++ for arg in '$@'
++++ echo ro root=/dev/mapper/mbpcvg-rootlv rd_LVM_LV=mbpcvg/rootlv rd_LVM_LV=VGEntertain/olv_swap rd_LVM_LV=mbpcvg/swaplv rd_NO_LUKS rd_NO_MD rd_NO_DM LANG=en_US.UTF-8 SYSFONT=latarcyrheb-sun16 KEYBOARDTYPE=pc KEYTABLE=us rhgb nomodeset irqpoll pcie_aspm=off amd_iommu=on pci=nomsi
++++ sed -e 's/\bhugepagesz=[^ ]*\b//g' -e 's/\bhugepagesz\b//g'
+++ cmdline='ro root=/dev/mapper/mbpcvg-rootlv rd_LVM_LV=mbpcvg/rootlv rd_LVM_LV=VGEntertain/olv_swap rd_LVM_LV=mbpcvg/swaplv rd_NO_LUKS rd_NO_MD rd_NO_DM LANG=en_US.UTF-8 SYSFONT=latarcyrheb-sun16 KEYBOARDTYPE=pc KEYTABLE=us rhgb nomodeset irqpoll pcie_aspm=off amd_iommu=on pci=nomsi'
+++ echo ro root=/dev/mapper/mbpcvg-rootlv rd_LVM_LV=mbpcvg/rootlv rd_LVM_LV=VGEntertain/olv_swap rd_LVM_LV=mbpcvg/swaplv rd_NO_LUKS rd_NO_MD rd_NO_DM LANG=en_US.UTF-8 SYSFONT=latarcyrheb-sun16 KEYBOARDTYPE=pc KEYTABLE=us rhgb nomodeset irqpoll pcie_aspm=off amd_iommu=on pci=nomsi
++ cmdline='ro root=/dev/mapper/mbpcvg-rootlv rd_LVM_LV=mbpcvg/rootlv rd_LVM_LV=VGEntertain/olv_swap rd_LVM_LV=mbpcvg/swaplv rd_NO_LUKS rd_NO_MD rd_NO_DM LANG=en_US.UTF-8 SYSFONT=latarcyrheb-sun16 KEYBOARDTYPE=pc KEYTABLE=us rhgb nomodeset irqpoll pcie_aspm=off amd_iommu=on pci=nomsi'
++ cmdline='ro root=/dev/mapper/mbpcvg-rootlv rd_LVM_LV=mbpcvg/rootlv rd_LVM_LV=VGEntertain/olv_swap rd_LVM_LV=mbpcvg/swaplv rd_NO_LUKS rd_NO_MD rd_NO_DM LANG=en_US.UTF-8 SYSFONT=latarcyrheb-sun16 KEYBOARDTYPE=pc KEYTABLE=us rhgb nomodeset irqpoll pcie_aspm=off amd_iommu=on pci=nomsi irqpoll nr_cpus=1 reset_devices cgroup_disable=memory mce=off acpi_no_memhotplug'
++ avoid_cdrom_drive
++ local DRIVE=
++ local MEDIA=
++ IDE_DRIVES=(`echo hd{a,b,c,d}`)
+++ echo hda hdb hdc hdd
++ local IDE_DRIVES
++ local COUNTER=0
++ for DRIVE in '${IDE_DRIVES[@]}'
+++ echo 'ro root=/dev/mapper/mbpcvg-rootlv rd_LVM_LV=mbpcvg/rootlv rd_LVM_LV=VGEntertain/olv_swap rd_LVM_LV=mbpcvg/swaplv rd_NO_LUKS rd_NO_MD rd_NO_DM LANG=en_US.UTF-8 SYSFONT=latarcyrheb-sun16 KEYBOARDTYPE=pc KEYTABLE=us rhgb nomodeset irqpoll pcie_aspm=off amd_iommu=on crashkernel=256M pci=nomsi'
+++ grep -q hda=
++ '[' -f /proc/ide/hda/media ']'
++ for DRIVE in '${IDE_DRIVES[@]}'
+++ echo 'ro root=/dev/mapper/mbpcvg-rootlv rd_LVM_LV=mbpcvg/rootlv rd_LVM_LV=VGEntertain/olv_swap rd_LVM_LV=mbpcvg/swaplv rd_NO_LUKS rd_NO_MD rd_NO_DM LANG=en_US.UTF-8 SYSFONT=latarcyrheb-sun16 KEYBOARDTYPE=pc KEYTABLE=us rhgb nomodeset irqpoll pcie_aspm=off amd_iommu=on crashkernel=256M pci=nomsi'
+++ grep -q hdb=
++ '[' -f /proc/ide/hdb/media ']'
++ for DRIVE in '${IDE_DRIVES[@]}'
+++ echo 'ro root=/dev/mapper/mbpcvg-rootlv rd_LVM_LV=mbpcvg/rootlv rd_LVM_LV=VGEntertain/olv_swap rd_LVM_LV=mbpcvg/swaplv rd_NO_LUKS rd_NO_MD rd_NO_DM LANG=en_US.UTF-8 SYSFONT=latarcyrheb-sun16 KEYBOARDTYPE=pc KEYTABLE=us rhgb nomodeset irqpoll pcie_aspm=off amd_iommu=on crashkernel=256M pci=nomsi'
+++ grep -q hdc=
++ '[' -f /proc/ide/hdc/media ']'
++ for DRIVE in '${IDE_DRIVES[@]}'
+++ echo 'ro root=/dev/mapper/mbpcvg-rootlv rd_LVM_LV=mbpcvg/rootlv rd_LVM_LV=VGEntertain/olv_swap rd_LVM_LV=mbpcvg/swaplv rd_NO_LUKS rd_NO_MD rd_NO_DM LANG=en_US.UTF-8 SYSFONT=latarcyrheb-sun16 KEYBOARDTYPE=pc KEYTABLE=us rhgb nomodeset irqpoll pcie_aspm=off amd_iommu=on crashkernel=256M pci=nomsi'
+++ grep -q hdd=
++ '[' -f /proc/ide/hdd/media ']'
++ '[' 0 -eq 0 ']'
++ KDUMP_IDE_NOPROBE_COMMANDLINE=
++ KDUMP_COMMANDLINE='ro root=/dev/mapper/mbpcvg-rootlv rd_LVM_LV=mbpcvg/rootlv rd_LVM_LV=VGEntertain/olv_swap rd_LVM_LV=mbpcvg/swaplv rd_NO_LUKS rd_NO_MD rd_NO_DM LANG=en_US.UTF-8 SYSFONT=latarcyrheb-sun16 KEYBOARDTYPE=pc KEYTABLE=us rhgb nomodeset irqpoll pcie_aspm=off amd_iommu=on crashkernel=256M pci=nomsi '
+++ get_bootcpu_initial_apicid
+++ awk '                                                       \
        BEGIN { CPU = "-1"; }                                   \
        $1=="processor" && $2==":"      { CPU = $NF; }          \
        CPU=="0" && /initial apicid/    { print $NF; }          \
        ' /proc/cpuinfo
++ local id=0
++ '[' '!' -z 0 ']'
+++ append_cmdline 'ro root=/dev/mapper/mbpcvg-rootlv rd_LVM_LV=mbpcvg/rootlv rd_LVM_LV=VGEntertain/olv_swap rd_LVM_LV=mbpcvg/swaplv rd_NO_LUKS rd_NO_MD rd_NO_DM LANG=en_US.UTF-8 SYSFONT=latarcyrheb-sun16 KEYBOARDTYPE=pc KEYTABLE=us rhgb nomodeset irqpoll pcie_aspm=off amd_iommu=on pci=nomsi irqpoll nr_cpus=1 reset_devices cgroup_disable=memory mce=off acpi_no_memhotplug' disable_cpu_apicid 0
+++ local 'cmdline=ro root=/dev/mapper/mbpcvg-rootlv rd_LVM_LV=mbpcvg/rootlv rd_LVM_LV=VGEntertain/olv_swap rd_LVM_LV=mbpcvg/swaplv rd_NO_LUKS rd_NO_MD rd_NO_DM LANG=en_US.UTF-8 SYSFONT=latarcyrheb-sun16 KEYBOARDTYPE=pc KEYTABLE=us rhgb nomodeset irqpoll pcie_aspm=off amd_iommu=on pci=nomsi irqpoll nr_cpus=1 reset_devices cgroup_disable=memory mce=off acpi_no_memhotplug'
+++ local 'newstr=ro root=/dev/mapper/mbpcvg-rootlv rd_LVM_LV=mbpcvg/rootlv rd_LVM_LV=VGEntertain/olv_swap rd_LVM_LV=mbpcvg/swaplv rd_NO_LUKS rd_NO_MD rd_NO_DM LANG=en_US.UTF-8 SYSFONT=latarcyrheb-sun16 KEYBOARDTYPE=pc KEYTABLE=us rhgb nomodeset irqpoll pcie_aspm=off amd_iommu=on pci=nomsi irqpoll nr_cpus=1 reset_devices cgroup_disable=memory mce=off acpi_no_memhotplug'
+++ '[' 'ro root=/dev/mapper/mbpcvg-rootlv rd_LVM_LV=mbpcvg/rootlv rd_LVM_LV=VGEntertain/olv_swap rd_LVM_LV=mbpcvg/swaplv rd_NO_LUKS rd_NO_MD rd_NO_DM LANG=en_US.UTF-8 SYSFONT=latarcyrheb-sun16 KEYBOARDTYPE=pc KEYTABLE=us rhgb nomodeset irqpoll pcie_aspm=off amd_iommu=on pci=nomsi irqpoll nr_cpus=1 reset_devices cgroup_disable=memory mce=off acpi_no_memhotplug' == 'ro root=/dev/mapper/mbpcvg-rootlv rd_LVM_LV=mbpcvg/rootlv rd_LVM_LV=VGEntertain/olv_swap rd_LVM_LV=mbpcvg/swaplv rd_NO_LUKS rd_NO_MD rd_NO_DM LANG=en_US.UTF-8 SYSFONT=latarcyrheb-sun16 KEYBOARDTYPE=pc KEYTABLE=us rhgb nomodeset irqpoll pcie_aspm=off amd_iommu=on pci=nomsi irqpoll nr_cpus=1 reset_devices cgroup_disable=memory mce=off acpi_no_memhotplug' ']'
+++ cmdline='ro root=/dev/mapper/mbpcvg-rootlv rd_LVM_LV=mbpcvg/rootlv rd_LVM_LV=VGEntertain/olv_swap rd_LVM_LV=mbpcvg/swaplv rd_NO_LUKS rd_NO_MD rd_NO_DM LANG=en_US.UTF-8 SYSFONT=latarcyrheb-sun16 KEYBOARDTYPE=pc KEYTABLE=us rhgb nomodeset irqpoll pcie_aspm=off amd_iommu=on pci=nomsi irqpoll nr_cpus=1 reset_devices cgroup_disable=memory mce=off acpi_no_memhotplug disable_cpu_apicid=0'
+++ echo ro root=/dev/mapper/mbpcvg-rootlv rd_LVM_LV=mbpcvg/rootlv rd_LVM_LV=VGEntertain/olv_swap rd_LVM_LV=mbpcvg/swaplv rd_NO_LUKS rd_NO_MD rd_NO_DM LANG=en_US.UTF-8 SYSFONT=latarcyrheb-sun16 KEYBOARDTYPE=pc KEYTABLE=us rhgb nomodeset irqpoll pcie_aspm=off amd_iommu=on pci=nomsi irqpoll nr_cpus=1 reset_devices cgroup_disable=memory mce=off acpi_no_memhotplug disable_cpu_apicid=0
++ cmdline='ro root=/dev/mapper/mbpcvg-rootlv rd_LVM_LV=mbpcvg/rootlv rd_LVM_LV=VGEntertain/olv_swap rd_LVM_LV=mbpcvg/swaplv rd_NO_LUKS rd_NO_MD rd_NO_DM LANG=en_US.UTF-8 SYSFONT=latarcyrheb-sun16 KEYBOARDTYPE=pc KEYTABLE=us rhgb nomodeset irqpoll pcie_aspm=off amd_iommu=on pci=nomsi irqpoll nr_cpus=1 reset_devices cgroup_disable=memory mce=off acpi_no_memhotplug disable_cpu_apicid=0'
++ echo ro root=/dev/mapper/mbpcvg-rootlv rd_LVM_LV=mbpcvg/rootlv rd_LVM_LV=VGEntertain/olv_swap rd_LVM_LV=mbpcvg/swaplv rd_NO_LUKS rd_NO_MD rd_NO_DM LANG=en_US.UTF-8 SYSFONT=latarcyrheb-sun16 KEYBOARDTYPE=pc KEYTABLE=us rhgb nomodeset irqpoll pcie_aspm=off amd_iommu=on pci=nomsi irqpoll nr_cpus=1 reset_devices cgroup_disable=memory mce=off acpi_no_memhotplug disable_cpu_apicid=0
+ KDUMP_COMMANDLINE='ro root=/dev/mapper/mbpcvg-rootlv rd_LVM_LV=mbpcvg/rootlv rd_LVM_LV=VGEntertain/olv_swap rd_LVM_LV=mbpcvg/swaplv rd_NO_LUKS rd_NO_MD rd_NO_DM LANG=en_US.UTF-8 SYSFONT=latarcyrheb-sun16 KEYBOARDTYPE=pc KEYTABLE=us rhgb nomodeset irqpoll pcie_aspm=off amd_iommu=on pci=nomsi irqpoll nr_cpus=1 reset_devices cgroup_disable=memory mce=off acpi_no_memhotplug disable_cpu_apicid=0'
+ grep -q /sys/kernel/debug /proc/mounts
+ mount -t debugfs debug /sys/kernel/debug
+ MNTDEBUG=/sys/kernel/debug
+ /sbin/kexec -p '–command-line=ro root=/dev/mapper/mbpcvg-rootlv rd_LVM_LV=mbpcvg/rootlv rd_LVM_LV=VGEntertain/olv_swap rd_LVM_LV=mbpcvg/swaplv rd_NO_LUKS rd_NO_MD rd_NO_DM LANG=en_US.UTF-8 SYSFONT=latarcyrheb-sun16 KEYBOARDTYPE=pc KEYTABLE=us rhgb nomodeset irqpoll pcie_aspm=off amd_iommu=on pci=nomsi irqpoll nr_cpus=1 reset_devices cgroup_disable=memory mce=off acpi_no_memhotplug disable_cpu_apicid=0' –initrd=/boot/initrd-4.8.4kdump.img /boot/vmlinuz-4.8.4
+ '[' 0 == 0 ']'
+ umount /sys/kernel/debug
+ /usr/bin/logger -p info -t kdump 'kexec: loaded kdump kernel'
+ return 0
+ return 0
+ '[' 0 '!=' 0 ']'
+ echo -n 'Starting kdump:'
Starting kdump:+ success
+ '[' color '!=' verbose -a -z '' ']'
+ echo_success
+ '[' color = color ']'
+ echo -en '\033[60G'
                                                           + echo -n '['
[+ '[' color = color ']'
+ echo -en '\033[0;32m'
+ echo -n '  OK  '
  OK  + '[' color = color ']'
+ echo -en '\033[0;39m'

+ echo -n ']'
]+ echo -ne '\r'
+ return 0
+ return 0
+ echo

+ /usr/bin/logger -p info -t kdump 'started up'
+ exit 0
[root@mbpc-pc ~]#

Weird.  What's going on then?  This is likely a bash conditional issue somewhere.  Why else would it work if we specify bash -x.  But let's investigate further:

/var/log/messages
Apr 22 15:30:21 mbpc-pc kdump: kexec: failed to load kdump kernel
Apr 22 15:30:21 mbpc-pc kdump: failed to start up
Apr 22 15:30:49 mbpc-pc kdump: kexec: failed to load kdump kernel
Apr 22 15:30:49 mbpc-pc kdump: failed to start up

shows us that kexec failed to load the kdump kernel when running without bash -x.

[root@mbpc-pc ~]# cat /etc/init.d/kdump|grep kexec
KEXEC=/sbin/kexec
standard_kexec_args="-p"
        MEM_RESERVED=`cat /sys/kernel/kexec_crash_size`
        $KEXEC $KEXEC_ARGS $standard_kexec_args \
                $LOGGER "kexec: loaded kdump kernel"
                $LOGGER "kexec: failed to load kdump kernel"
        if [ ! -e /sys/kernel/kexec_crash_loaded ]
        rc=`cat /sys/kernel/kexec_crash_loaded`
                $LOGGER "kexec: failed to unload kdump kernel"
        $LOGGER "kexec: unloaded kdump kernel"
[root@mbpc-pc ~]# cat /sys/kernel/kexec_crash_size
268435456
[root@mbpc-pc ~]#

After a few more hours of digging, we're not that much further along.  So we employ the dirty fix for the time being till we can spend more time and figure out the rest.  Here's the result of the testing:

[root@mbpc-pc ~]# /etc/init.d/kdump start
start(): Calling save_raw() …
KDUMP_COMMANDLINE=
MEM_RESERVED=268435456
Running check_config …
start_dump(): DEFAULT_DUMP_MODE=kdump
load_kdump(): KDUMP_COMMANDLINE_APPEND=irqpoll nr_cpus=1 reset_devices cgroup_disable=memory mce=off acpi_no_memhotplug
load_kdump(): Running /sbin/kexec  -p –command-line=ro root=/dev/mapper/mbpcvg-rootlv rd_LVM_LV=mbpcvg/rootlv rd_LVM_LV=VGEntertain/olv_swap rd_LVM_LV=mbpcvg/swaplv rd_NO_LUKS rd_NO_MD rd_NO_DM LANG=en_US.UTF-8 SYSFONT=latarcyrheb-sun16 KEYBOARDTYPE=pc KEYTABLE=us rhgb nomodeset irqpoll pcie_aspm=off amd_iommu=on pci=nomsi irqpoll nr_cpus=1 reset_devices cgroup_disable=memory mce=off acpi_no_memhotplug disable_cpu_apicid=0 –initrd=/boot/initrd-4.8.4kdump.img /boot/vmlinuz-4.8.4 …
+ /sbin/kexec -p '–command-line=ro root=/dev/mapper/mbpcvg-rootlv rd_LVM_LV=mbpcvg/rootlv rd_LVM_LV=VGEntertain/olv_swap rd_LVM_LV=mbpcvg/swaplv rd_NO_LUKS rd_NO_MD rd_NO_DM LANG=en_US.UTF-8 SYSFONT=latarcyrheb-sun16 KEYBOARDTYPE=pc KEYTABLE=us rhgb nomodeset irqpoll pcie_aspm=off amd_iommu=on pci=nomsi irqpoll nr_cpus=1 reset_devices cgroup_disable=memory mce=off acpi_no_memhotplug disable_cpu_apicid=0' –initrd=/boot/initrd-4.8.4kdump.img /boot/vmlinuz-4.8.4
Memory for crashkernel is not reserved
Please reserve memory by passing "crashkernel=X@Y" parameter to the kernel
Then try loading kdump kernel
+ RETV=1
+ set +x
load_kdump(): RETV=1
Starting kdump:                                            [FAILED]
[root@mbpc-pc ~]#
[root@mbpc-pc ~]#
[root@mbpc-pc ~]# bash -x /etc/init.d/kdump status 2>/dev/null
Kdump is not operational
[root@mbpc-pc ~]# /sbin/kexec -p '–command-line=ro root=/dev/mapper/mbpcvg-rootlv rd_LVM_LV=mbpcvg/rootlv rd_LVM_LV=VGEntertain/olv_swap rd_LVM_LV=mbpcvg/swaplv rd_NO_LUKS rd_NO_MD rd_NO_DM LANG=en_US.UTF-8 SYSFONT=latarcyrheb-sun16 KEYBOARDTYPE=pc KEYTABLE=us rhgb nomodeset irqpoll pcie_aspm=off amd_iommu=on pci=nomsi irqpoll nr_cpus=1 reset_devices cgroup_disable=memory mce=off acpi_no_memhotplug disable_cpu_apicid=0' –initrd=/boot/initrd-4.8.4kdump.img /boot/vmlinuz-4.8.4
[root@mbpc-pc ~]#
[root@mbpc-pc ~]# bash -x /etc/init.d/kdump status 2>/dev/null                                                                                 Kdump is operational
[root@mbpc-pc ~]#
[root@mbpc-pc ~]#
[root@mbpc-pc ~]# bash -x /etc/init.d/kdump stop 2>/dev/null
Stopping kdump:                                            [  OK  ]
[root@mbpc-pc ~]# bash -x /etc/init.d/kdump status 2>/dev/null
Kdump is not operational
[root@mbpc-pc ~]#
[root@mbpc-pc ~]#
[root@mbpc-pc ~]# bash -x /etc/init.d/kdump start 2>/dev/null
start(): Calling save_raw() …
KDUMP_COMMANDLINE=
MEM_RESERVED=268435456
Running check_config …
start_dump(): DEFAULT_DUMP_MODE=kdump
load_kdump(): KDUMP_COMMANDLINE_APPEND=irqpoll nr_cpus=1 reset_devices cgroup_disable=memory mce=off acpi_no_memhotplug
load_kdump(): Running /sbin/kexec  -p –command-line=ro root=/dev/mapper/mbpcvg-rootlv rd_LVM_LV=mbpcvg/rootlv rd_LVM_LV=VGEntertain/olv_swap rd_LVM_LV=mbpcvg/swaplv rd_NO_LUKS rd_NO_MD rd_NO_DM LANG=en_US.UTF-8 SYSFONT=latarcyrheb-sun16 KEYBOARDTYPE=pc KEYTABLE=us rhgb nomodeset irqpoll pcie_aspm=off amd_iommu=on pci=nomsi irqpoll nr_cpus=1 reset_devices cgroup_disable=memory mce=off acpi_no_memhotplug disable_cpu_apicid=0 –initrd=/boot/initrd-4.8.4kdump.img /boot/vmlinuz-4.8.4 …
load_kdump(): RETV=0
Starting kdump:                                            [  OK  ]
[root@mbpc-pc ~]#
[root@mbpc-pc ~]# bash -x /etc/init.d/kdump status 2>/dev/null
Kdump is operational
[root@mbpc-pc ~]#

 

And the vimdiff of the strace of each piece gives:

  + /usr/bin/strace /sbin/kexec -p '–command-line=ro root=/dev/mapper/|  + /usr/bin/strace /sbin/kexec -p –command-line=ro root=/dev/mapper/m
  execve("/sbin/kexec", ["/sbin/kexec", "-p", "--command-line=ro root=/|  execve("/sbin/kexec", ["/sbin/kexec", "-p", "--command-line=ro", "roo
  brk(0)                                  = 0x1eea000                  |  brk(0)                                  = 0x1e76000
  mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1,|  mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1,
  access("/etc/ld.so.preload", R_OK)      = -1 ENOENT (No such file or |  access("/etc/ld.so.preload", R_OK)      = -1 ENOENT (No such file or
  open("/etc/ld.so.cache", O_RDONLY)      = 3                          |  open("/etc/ld.so.cache", O_RDONLY)      = 3
  fstat(3, {st_mode=S_IFREG|0644, st_size=114634, ...}) = 0            |  fstat(3, {st_mode=S_IFREG|0644, st_size=114634, ...}) = 0
  mmap(NULL, 114634, PROT_READ, MAP_PRIVATE, 3, 0) = 0x7f0fc92b5000    |  mmap(NULL, 114634, PROT_READ, MAP_PRIVATE, 3, 0) = 0x7f5b26cfb000
  close(3)                                = 0                          |  close(3)                                = 0
  open("/lib64/libz.so.1", O_RDONLY)      = 3                          |  open("/lib64/libz.so.1", O_RDONLY)      = 3
  read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0 !\240z3\0\0\0|  read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0 !\240z3\0\0\0
  fstat(3, {st_mode=S_IFREG|0755, st_size=91096, ...}) = 0             |  fstat(3, {st_mode=S_IFREG|0755, st_size=91096, ...}) = 0
  mmap(0x337aa00000, 2183696, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENY|  mmap(0x337aa00000, 2183696, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENY
  mprotect(0x337aa15000, 2093056, PROT_NONE) = 0                       |  mprotect(0x337aa15000, 2093056, PROT_NONE) = 0
+ +--  4 lines: mmap(0x337ac14000, 8192, PROT_READ|PROT_WRITE, MAP_PRIV|+ +--  4 lines: mmap(0x337ac14000, 8192, PROT_READ|PROT_WRITE, MAP_PRIV
  fstat(3, {st_mode=S_IFREG|0755, st_size=1926480, ...}) = 0           |  fstat(3, {st_mode=S_IFREG|0755, st_size=1926480, ...}) = 0
  mmap(0x3379600000, 3750152, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENY|  mmap(0x3379600000, 3750152, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENY
  mprotect(0x337978a000, 2097152, PROT_NONE) = 0                       |  mprotect(0x337978a000, 2097152, PROT_NONE) = 0
  mmap(0x337998a000, 20480, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|  mmap(0x337998a000, 20480, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED
  mmap(0x337998f000, 18696, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|  mmap(0x337998f000, 18696, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED
  close(3)                                = 0                          |  close(3)                                = 0
  mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1,|  mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1,
  mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1,|  mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1,
  mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1,|  mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1,
  arch_prctl(ARCH_SET_FS, 0x7f0fc92b3700) = 0                          |  arch_prctl(ARCH_SET_FS, 0x7f5b26cf9700) = 0
  mprotect(0x337ac14000, 4096, PROT_READ) = 0                          |  mprotect(0x337ac14000, 4096, PROT_READ) = 0
  mprotect(0x337998a000, 16384, PROT_READ) = 0                         |  mprotect(0x337998a000, 16384, PROT_READ) = 0
  mprotect(0x337941f000, 4096, PROT_READ) = 0                          |  mprotect(0x337941f000, 4096, PROT_READ) = 0
  munmap(0x7f0fc92b5000, 114634)          = 0                          |  munmap(0x7f5b26cfb000, 114634)          = 0
  brk(0)                                  = 0x1eea000                  |  brk(0)                                  = 0x1e76000
  brk(0x1f0b000)                          = 0x1f0b000                  |  brk(0x1e97000)                          = 0x1e97000
  open("/proc/iomem", O_RDONLY)           = 3                          |  open("/proc/iomem", O_RDONLY)           = 3
  fstat(3, {st_mode=S_IFREG|0444, st_size=0, ...}) = 0                 |  fstat(3, {st_mode=S_IFREG|0444, st_size=0, ...}) = 0
  mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1,|  mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1,
  read(3, "00000000-00000fff : reserved\n000"..., 1024) = 1024         |  read(3, "00000000-00000000 : reserved\n000"..., 1024) = 1024
  read(3, "08\n    fc000000-fc7fffff : 0000:"..., 1024) = 1024         |  read(3, "08\n    00000000-00000000 : 0000:"..., 1024) = 1024
  read(3, "       fdb40000-fdb7ffff : 0000:"..., 1024) = 1024          |  read(3, "       00000000-00000000 : 0000:"..., 1024) = 1024
  read(3, "-fe02afff : ohci_hcd\n  fe02b000-"..., 1024) = 610          |  read(3, "-00000000 : ohci_hcd\n  00000000-"..., 1024) = 608
  read(3, "", 1024)                       = 0                          |  read(3, "", 1024)                       = 0
  close(3)                                = 0                          |  close(3)                                = 0
  munmap(0x7f0fc92d0000, 4096)            = 0                          |  munmap(0x7f5b26d16000, 4096)            = 0
  open("/boot/vmlinuz-4.8.4", O_RDONLY)   = 3                          |  fstat(1, {st_mode=S_IFCHR|0620, st_rdev=makedev(136, 1), ...}) = 0
  fstat(3, {st_mode=S_IFREG|0644, st_size=5045696, ...}) = 0           |  mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1,
  mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1,|  write(1, "Memory for crashkernel is not re"..., 39Memory for crashker
  read(3, "\352\5\0\300\7\214\310\216\330\216\300\216\3201\344\373\374\|  ) = 39
  lseek(3, 0, SEEK_CUR)                   = 16384                      |  write(1, "Please reserve memory by passing"..., 75Please reserve memo
  read(3, "1\300\216\330\216\300\216\320\216\340\216\350H\215-\355\375\|  ) = 75
a.txt                                                26,1           Top b.txt                                                26,1           Top

 

And a better visual in image format:

Kexec Strace and Vimdiff

So then a temporary solution for now is to use the following:

[root@mbpc-pc ~]# cat /etc/rc.local |grep -Ei "kdump|random-seed"
# Create a random seed for kdump. – Tom K.
dd if=/dev/urandom of=/var/lib/random-seed bs=1024 count=1
# kexec of kdump can't start when bash -x isn't used.  So this is a hack.
bash -x /etc/init.d/kdump start 2>/dev/null;
bash -x /etc/init.d/kdump status 2>/dev/null
[root@mbpc-pc ~]#

But unfortunately, that didn't stick either.  Only one choice then to resolve a kernel panic seen earlier on my system and that was to install the latest 4.X Kernel:

[root@mbpc-pc ~]# rpm –import https://www.elrepo.org/RPM-GPG-KEY-elrepo.org
[root@mbpc-pc yum.repos.d]# yum –enablerepo=elrepo-kernel install kernel-ml

Cheers,
Tom

NFS reply xid 3844308326 reply ERR 20: Auth Rejected Credentials (client should begin new session)

Getting this? Mounts freezing?  Final verified solution is at the bottom but this can be for any number of reasons.  Keep reading:

tcpdump -i eth0 -s 0 -w dump.dat
tcpdump -r dump.dat |grep -Ei "psql02|nfs-c01"

02:55:48.731360 IP psql02.nix.mds.xyz.33991 > nfs-c01.nix.mds.xyz.nfs: Flags [P.], seq 1:693, ack 1, win 229, options [nop,nop,TS val 166990 ecr 5681495], length 692: NFS request xid 3844308326 688 null
02:55:48.731483 IP nfs-c01.nix.mds.xyz.nfs > psql02.nix.mds.xyz.33991: Flags [.], ack 693, win 238, options [nop,nop,TS val 5681498 ecr 166990], length 0
02:55:48.732644 IP nfs-c01.nix.mds.xyz.nfs > psql02.nix.mds.xyz.33991: Flags [P.], seq 1:25, ack 693, win 238, options [nop,nop,TS val 5681499 ecr 166990], length 24: NFS reply xid 3844308326 reply ERR 20: Auth Rejected Credentials (client should begin new session)
02:55:48.732670 IP psql02.nix.mds.xyz.33991 > nfs-c01.nix.mds.xyz.nfs: Flags [.], ack 25, win 229, options [nop,nop,TS val 166991 ecr 5681499], length 0

Try this patch to bring nfs-utils-1.3.0-0.48.el7_4.1.x86_64 up to nfs-utils-1.3.0-0.48.el7_4.2.x86_64:

http://download.rhn.redhat.com/errata/RHBA-2018-0422.html

Update and enjoy?  Nope!  So let's keep digging some more.  After more of an exhaustive search, the result was to add the following firewall lines and restart autofs.  Appears autofs didn't properly start on account of the missing firewall ports causing everything else to freeze, including any additional mounts:


[root@ovirt01 sssd]# firewall-cmd –zone=public –permanent –add-port=111/udp
success
[root@ovirt01 sssd]# firewall-cmd –zone=public –permanent –add-port=2049/ufp
success
[root@ovirt01 sssd]# firewall-cmd –reload
success
[root@ovirt01 sssd]# systemctl restart autofs
[root@ovirt01 sssd]# mount nfs-c01:/n /m
[root@ovirt01 sssd]# umount /m
[root@ovirt01 sssd]#
[root@ovirt01 sssd]#

The following fix was also used in combination with above: 

https://review.gerrithub.io/#/c/ffilz/nfs-ganesha/+/408756/

[root@nfs02 ~]# /bin/ganesha.nfsd -v
NFS-Ganesha Release = V2.7-dev.10
ganesha.nfsd compiled on Apr 30 2018 at 02:21:35
Release comment = GANESHA file server is 64 bits compliant and supports NFS v3,4.0,4.1 (pNFS) and 9P
Git HEAD = 9cf00dccc9ab92ea4a6ec6f7f1f2c043bdc20a4b
Git Describe = V2.7-dev.10-0-g9cf00dc
[root@nfs02 ~]#

On top of the above, also ensure the following gluster errors are handled:

[2018-05-01 22:43:06.412067] E [MSGID: 114058] [client-handshake.c:1571:client_query_portmap_cbk] 0-gv01-client-1: failed to get the port number for remote subvolume. Please run 'gluster volume status' on server to see if brick process is running.
[2018-05-01 22:43:55.554833] E [socket.c:2374:socket_connect_finish] 0-gv01-client-0: connection to 192.168.0.131:49152 failed (Connection refused); disconnecting socket

 

[root@nfs02 glusterfs]# netstat -pnlt|grep gluster
tcp        0      0 0.0.0.0:24007           0.0.0.0:*               LISTEN      1108/glusterd
tcp        0      0 0.0.0.0:49152           0.0.0.0:*               LISTEN      1432/glusterfsd
[root@nfs02 glusterfs]#


[ CORRECT ]

[root@nfs02 glusterfs]# firewall-cmd –zone=dmz –list-all
dmz
  target: default
  icmp-block-inversion: no
  interfaces:
  sources:
  services: ssh
  ports: 2049/tcp 111/tcp 24007-24008/tcp 38465-38469/tcp 111/udp 22/tcp 22/udp 49000-59999/udp 49000-59999/tcp 20048/tcp 20048/udp 49152/tcp 4501/tcp 4501/udp 10000/tcp 9000/udp 9000/tcp
  protocols:
  masquerade: no
  forward-ports:
  source-ports:
  icmp-blocks:
  rich rules:

[root@nfs02 glusterfs]#


[ INCORRECT ]

[root@nfs01 /]# firewall-cmd –zone=public –list-all
public (active)
  target: default
  icmp-block-inversion: no
  interfaces: eth0
  sources:
  services: ssh dhcpv6-client haproxy
  ports: 24007-24008/tcp 49152/tcp 38465-38469/tcp 111/tcp 111/udp 2049/tcp 4501/tcp 4501/udp 20048/udp 20048/tcp 22/tcp 22/udp 10000/tcp 49000-59999/udp 49000-59999/tcp 9000/udp 9000/tcp 137/udp 138/udp 2049/udp
  protocols:
  masquerade: no
  forward-ports:
  source-ports: 49000-59999/tcp
  icmp-blocks:
  rich rules:

[root@nfs01 /]#


Fix was to remove the source-port by either editing /etc/firewalld/zones/public.xml and removing 

firewall-cmd –zone=public –permanent –remove-source-port=49000-59999/udp
firewall-cmd –zone=public –permanent –remove-source-port=49000-59999/tcp
firewall-cmd –reload


Also ensure haproxy is running on both hosts:


[root@nfs02 systemd]# systemctl status haproxy -l
* haproxy.service – HAProxy Load Balancer
   Loaded: loaded (/usr/lib/systemd/system/haproxy.service; enabled; vendor preset: disabled)
   Active: active (running) since Tue 2018-05-01 23:21:44 EDT; 20s ago
 Main PID: 2405 (haproxy-systemd)
   CGroup: /system.slice/haproxy.service
           |-2405 /usr/sbin/haproxy-systemd-wrapper -f /etc/haproxy/haproxy.cfg -p /run/haproxy.pid
           |-2406 /usr/sbin/haproxy -f /etc/haproxy/haproxy.cfg -p /run/haproxy.pid -Ds
           `-2407 /usr/sbin/haproxy -f /etc/haproxy/haproxy.cfg -p /run/haproxy.pid -Ds

May 01 23:21:44 nfs02.nix.mds.xyz systemd[1]: Started HAProxy Load Balancer.
May 01 23:21:44 nfs02.nix.mds.xyz systemd[1]: Starting HAProxy Load Balancer…
May 01 23:21:44 nfs02.nix.mds.xyz haproxy-systemd-wrapper[2405]: haproxy-systemd-wrapper: executing /usr/sbin/haproxy -f /etc/haproxy/haproxy.cfg -p /run/haproxy.pid -Ds
[root@nfs02 systemd]# sysctl -p
net.ipv4.ip_nonlocal_bind = 1
net.ipv4.ip_forward = 1
vm.min_free_kbytes = 1048560
[root@nfs02 systemd]#

 

[root@nfs01 ~]# systemctl status haproxy -l
â haproxy.service – HAProxy Load Balancer
   Loaded: loaded (/usr/lib/systemd/system/haproxy.service; enabled; vendor preset: disabled)
   Active: active (running) since Tue 2018-05-01 23:21:53 EDT; 7s ago
 Main PID: 21707 (haproxy-systemd)
   CGroup: /system.slice/haproxy.service
           ââ21707 /usr/sbin/haproxy-systemd-wrapper -f /etc/haproxy/haproxy.cfg -p /run/haproxy.pid
           ââ21708 /usr/sbin/haproxy -f /etc/haproxy/haproxy.cfg -p /run/haproxy.pid -Ds
           ââ21709 /usr/sbin/haproxy -f /etc/haproxy/haproxy.cfg -p /run/haproxy.pid -Ds

May 01 23:21:53 nfs01.nix.mds.xyz systemd[1]: Started HAProxy Load Balancer.
May 01 23:21:53 nfs01.nix.mds.xyz systemd[1]: Starting HAProxy Load Balancer…
May 01 23:21:53 nfs01.nix.mds.xyz haproxy-systemd-wrapper[21707]: haproxy-systemd-wrapper: executing /usr/sbin/haproxy -f /etc/haproxy/haproxy.cfg -p /run/haproxy.pid -Ds
[root@nfs01 ~]# sysctl -p
net.ipv4.ip_nonlocal_bind = 1
net.ipv4.ip_forward = 1
vm.min_free_kbytes = 1048560
[root@nfs01 ~]#

The other issue that existed was that you did not have a proper PTR and DNS records for the server.  Add them in IPA server.  This indicates that either there is an IPA server replication issue or PTR records are not created:

[root@psql01 ~]# dig -x psql01

; <<>> DiG 9.9.4-RedHat-9.9.4-51.el7_4.2 <<>> -x psql01
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NXDOMAIN, id: 29853
;; flags: qr rd ra ad; QUERY: 1, ANSWER: 0, AUTHORITY: 1, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
;; QUESTION SECTION:
;psql01.in-addr.arpa.           IN      PTR

;; AUTHORITY SECTION:
in-addr.arpa.           900     IN      SOA     b.in-addr-servers.arpa. nstld.iana.org. 2018013362 1800 900 604800 3600

;; Query time: 95 msec
;; SERVER: 192.168.0.44#53(192.168.0.44)
;; WHEN: Tue May 01 23:39:52 EDT 2018
;; MSG SIZE  rcvd: 116

[root@psql01 ~]#

But that didn't fix it either.   Next thing:

firewall-cmd –direct –permanent –add-rule ipv4 filter INPUT 0 –in-interface enp0s8 –destination 224.0.0.18 –protocol vrrp -j ACCEPT

And that didn't work either.  Some of the packets in the TCP dump show as incorrect.  This can be fixed using the following:

[root@nfs02 ~]# ethtool –show-offload  eth0 > eth0-checksum.txt
[root@nfs02 ~]# ethtool –offload  eth0  rx off  tx off
Actual changes:
rx-checksumming: off
tx-checksumming: off
        tx-checksum-ip-generic: off
tcp-segmentation-offload: off
        tx-tcp-segmentation: off [requested on]
        tx-tcp6-segmentation: off [requested on]
[root@nfs02 ~]#

 

That did not do the trick either.  The only log that changed on the mount attempt is the following on NFS02:

==> /var/log/ganesha/ganesha-rgw.log <==
11/11/2018 10:05:12 : epoch 5be8411c : nfs02.nix.mds.xyz : ganesha.nfsd-28961[svc_2] nfs_rpc_decode_request :D ISP :D EBUG :0x7faa0c0012b0 fd 32 context 0x7faa08000c10
11/11/2018 10:05:12 : epoch 5be8411c : nfs02.nix.mds.xyz : ganesha.nfsd-28961[svc_2] nfs_rpc_process_request :D ISP :D EBUG :Request from ::ffff:192.168.0.125 for Program 100003, Version 4, Function 0 has xid=1069716099
11/11/2018 10:05:12 : epoch 5be8411c : nfs02.nix.mds.xyz : ganesha.nfsd-28961[svc_2] nfs_rpc_decode_request :D ISP :D EBUG :SVC_DECODE on 0x7faa0c0012b0 fd 32 (::ffff:192.168.0.125:46740) xid=1069716099 returned XPRT_IDLE
11/11/2018 10:05:12 : epoch 5be8411c : nfs02.nix.mds.xyz : ganesha.nfsd-28961[svc_2] free_nfs_request :D ISP :D EBUG :free_nfs_request: 0x7faa0c0012b0 fd 32 xp_refs 3 rq_refs 0
11/11/2018 10:05:12 : epoch 5be8411c : nfs02.nix.mds.xyz : ganesha.nfsd-28961[svc_13] nfs_rpc_decode_request :D ISP :D EBUG :0x7faa00001440 fd 34 context 0x7fa9e8002570
11/11/2018 10:05:12 : epoch 5be8411c : nfs02.nix.mds.xyz : ganesha.nfsd-28961[svc_13] nfs_rpc_process_request :D ISP :INFO :Could not authenticate request… rejecting with AUTH_STAT=AUTH_REJECTEDCRED
11/11/2018 10:05:12 : epoch 5be8411c : nfs02.nix.mds.xyz : ganesha.nfsd-28961[svc_13] nfs_rpc_decode_request :D ISP :D EBUG :SVC_DECODE on 0x7faa00001440 fd 34 (::ffff:192.168.0.125:46742) xid=2610062768 returned XPRT_IDLE
11/11/2018 10:05:12 : epoch 5be8411c : nfs02.nix.mds.xyz : ganesha.nfsd-28961[svc_13] free_nfs_request :D ISP :D EBUG :free_nfs_request: 0x7faa00001440 fd 34 xp_refs 3 rq_refs 0

 

SOLUTION (Verified)

This is one of the solutions that did move us forwrad but for the wrong reason.  Still check auditd for the following:

type=AVC msg=audit(1526965320.850:4094): avc:  denied  { write } for  pid=8714 comm="ganesha.nfsd" name="nfs_0" dev="dm-0" ino=201547689 scontext=system_u:system_r:ganesha_t:s0 tcontext=system_u:object_r:krb5_host_rcache_t:s0 tclass=file
type=SYSCALL msg=audit(1526965320.850:4094): arch=c000003e syscall=2 success=no exit=-13 a0=7f23b0003150 a1=2 a2=180 a3=2 items=0 ppid=1 pid=8714 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=4294967295 comm="ganesha.nfsd" exe="/usr/bin/ganesha.nfsd" subj=system_u:system_r:ganesha_t:s0 key=(null)
type=PROCTITLE msg=audit(1526965320.850:4094): proctitle=2F7573722F62696E2F67616E657368612E6E667364002D4C002F7661722F6C6F672F67616E657368612F67616E657368612E6C6F67002D66002F6574632F67616E657368612F67616E657368612E636F6E66002D4E004E49565F4556454E54
type=AVC msg=audit(1526965320.850:4095): avc:  denied  { unlink } for  pid=8714 comm="ganesha.nfsd" name="nfs_0" dev="dm-0" ino=201547689 scontext=system_u:system_r:ganesha_t:s0 tcontext=system_u:object_r:krb5_host_rcache_t:s0 tclass=file
type=SYSCALL msg=audit(1526965320.850:4095): arch=c000003e syscall=87 success=no exit=-13 a0=7f23b0004100 a1=7f23b0000050 a2=7f23b0004100 a3=5 items=0 ppid=1 pid=8714 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=4294967295 comm="ganesha.nfsd" exe="/usr/bin/ganesha.nfsd" subj=system_u:system_r:ganesha_t:s0 key=(null)
type=PROCTITLE msg=audit(1526965320.850:4095): proctitle=2F7573722F62696E2F67616E657368612E6E667364002D4C002F7661722F6C6F672F67616E657368612F67616E657368612E6C6F67002D66002F6574632F67616E657368612F67616E657368612E636F6E66002D4E004E49565F4556454E54

A few lines like this:

grep AVC /var/log/audit/audit.log | audit2allow -M systemd-allow

semodule -i systemd-allow.pp

solved the issue for us.  The error thrown also included this:

May 21 23:53:13 psql01 kernel: CPU: 3 PID: 2273 Comm: mount.nfs Tainted: G             L ————   3.10.0-693.21.1.el7.x86_64 #1
May 21 23:53:13 psql01 kernel: Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 04/14/2014
May 21 23:53:13 psql01 kernel: task: ffff880136335ee0 ti: ffff8801376b0000 task.ti: ffff8801376b0000
May 21 23:53:13 psql01 kernel: RIP: 0010:[]  [] _raw_spin_unlock_irqrestore+0×15/0×20
May 21 23:53:13 psql01 kernel: RSP: 0018:ffff8801376b3a60  EFLAGS: 00000206
May 21 23:53:13 psql01 kernel: RAX: ffffffffc05ab078 RBX: ffff880036973928 RCX: dead000000000200
May 21 23:53:13 psql01 kernel: RDX: ffffffffc05ab078 RSI: 0000000000000206 RDI: 0000000000000206
May 21 23:53:13 psql01 kernel: RBP: ffff8801376b3a60 R08: ffff8801376b3ab8 R09: ffff880137de1200
May 21 23:53:13 psql01 kernel: R10: ffff880036973928 R11: 0000000000000000 R12: ffff880036973928
May 21 23:53:13 psql01 kernel: R13: ffff8801376b3a58 R14: ffff88013fd98a40 R15: ffff8801376b3a58
May 21 23:53:13 psql01 kernel: FS:  00007fab48f07880(0000) GS:ffff88013fd80000(0000) knlGS:0000000000000000
May 21 23:53:13 psql01 kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
May 21 23:53:13 psql01 kernel: CR2: 00007f99793d93cc CR3: 000000013761e000 CR4: 00000000000007e0
May 21 23:53:13 psql01 kernel: DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
May 21 23:53:13 psql01 kernel: DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
May 21 23:53:13 psql01 kernel: Call Trace:
May 21 23:53:13 psql01 kernel: [] finish_wait+0×56/0×70
May 21 23:53:13 psql01 kernel: [] nfs_wait_client_init_complete+0xa1/0xe0 [nfs]
May 21 23:53:13 psql01 kernel: [] ? wake_up_atomic_t+0×30/0×30
May 21 23:53:13 psql01 kernel: [] nfs_get_client+0x22b/0×470 [nfs]
May 21 23:53:13 psql01 kernel: [] nfs4_set_client+0×98/0×130 [nfsv4]
May 21 23:53:13 psql01 kernel: [] nfs4_create_server+0x13e/0x3b0 [nfsv4]
May 21 23:53:13 psql01 kernel: [] nfs4_remote_mount+0x2e/0×60 [nfsv4]
May 21 23:53:13 psql01 kernel: [] mount_fs+0x3e/0x1b0
May 21 23:53:13 psql01 kernel: [] ? __alloc_percpu+0×15/0×20
May 21 23:53:13 psql01 kernel: [] vfs_kern_mount+0×67/0×110
May 21 23:53:13 psql01 kernel: [] nfs_do_root_mount+0×86/0xc0 [nfsv4]
May 21 23:53:13 psql01 kernel: [] nfs4_try_mount+0×44/0xc0 [nfsv4]
May 21 23:53:13 psql01 kernel: [] ? get_nfs_version+0×27/0×90 [nfs]
May 21 23:53:13 psql01 kernel: [] nfs_fs_mount+0x4cb/0xda0 [nfs]
May 21 23:53:13 psql01 kernel: [] ? nfs_clone_super+0×140/0×140 [nfs]
May 21 23:53:13 psql01 kernel: [] ? param_set_portnr+0×70/0×70 [nfs]
May 21 23:53:13 psql01 kernel: [] mount_fs+0x3e/0x1b0
May 21 23:53:13 psql01 kernel: [] ? __alloc_percpu+0×15/0×20
May 21 23:53:13 psql01 kernel: [] vfs_kern_mount+0×67/0×110
May 21 23:53:13 psql01 kernel: [] do_mount+0×233/0xaf0
May 21 23:53:13 psql01 kernel: [] SyS_mount+0×96/0xf0
May 21 23:53:13 psql01 kernel: [] system_call_fastpath+0x1c/0×21
May 21 23:53:13 psql01 kernel: [] ? system_call_after_swapgs+0xae/0×146

However the above did not work for us on a second attempt because we were missing the right principal on that server.  The correct server has the following:

[root@nfs02 ~]# klist -kte
Keytab name: FILE:/etc/krb5.keytab
KVNO Timestamp           Principal
—- ——————- ——————————————————
   1 02/17/2018 20:13:39 host/nfs02.nix.mds.xyz@NIX.MDS.XYZ (aes256-cts-hmac-sha1-96)
   1 02/17/2018 20:13:39 host/nfs02.nix.mds.xyz@NIX.MDS.XYZ (aes128-cts-hmac-sha1-96)
   1 02/17/2018 20:13:39 host/nfs02.nix.mds.xyz@NIX.MDS.XYZ (des3-cbc-sha1)
   1 02/17/2018 20:13:39 host/nfs02.nix.mds.xyz@NIX.MDS.XYZ (arcfour-hmac)
   4 03/04/2018 13:57:16 nfs/nfs-c01.nix.mds.xyz@NIX.MDS.XYZ (aes256-cts-hmac-sha1-96)
   4 03/04/2018 13:57:16 nfs/nfs-c01.nix.mds.xyz@NIX.MDS.XYZ (aes128-cts-hmac-sha1-96)
   4 03/04/2018 13:57:16 nfs/nfs-c01.nix.mds.xyz@NIX.MDS.XYZ (des3-cbc-sha1)
   4 03/04/2018 13:57:16 nfs/nfs-c01.nix.mds.xyz@NIX.MDS.XYZ (arcfour-hmac)
   2 03/04/2018 13:57:32 nfs/nfs02.nix.mds.xyz@NIX.MDS.XYZ (aes256-cts-hmac-sha1-96)
   2 03/04/2018 13:57:32 nfs/nfs02.nix.mds.xyz@NIX.MDS.XYZ (aes128-cts-hmac-sha1-96)
   2 03/04/2018 13:57:32 nfs/nfs02.nix.mds.xyz@NIX.MDS.XYZ (des3-cbc-sha1)
   2 03/04/2018 13:57:32 nfs/nfs02.nix.mds.xyz@NIX.MDS.XYZ (arcfour-hmac)
   4 03/05/2018 22:56:37 nfs/nfs02.nix.mds.xyz@NIX.MDS.XYZ (aes256-cts-hmac-sha1-96)
   4 03/05/2018 22:56:37 nfs/nfs02.nix.mds.xyz@NIX.MDS.XYZ (aes128-cts-hmac-sha1-96)
   4 03/05/2018 22:56:37 nfs/nfs02.nix.mds.xyz@NIX.MDS.XYZ (des3-cbc-sha1)
   4 03/05/2018 22:56:37 nfs/nfs02.nix.mds.xyz@NIX.MDS.XYZ (arcfour-hmac)
[root@nfs02 ~]#

And the bad one in the cluster had the following:

[root@nfs03 ~]# klist -kte
Keytab name: FILE:/etc/krb5.keytab
KVNO Timestamp         Principal
—- —————– ——————————————————–
   1 05/20/18 23:18:01 host/nfs03.nix.mds.xyz@NIX.MDS.XYZ (aes256-cts-hmac-sha1-96)
   1 05/20/18 23:18:01 host/nfs03.nix.mds.xyz@NIX.MDS.XYZ (aes128-cts-hmac-sha1-96)
   1 05/20/18 23:18:01 host/nfs03.nix.mds.xyz@NIX.MDS.XYZ (des3-cbc-sha1)
   1 05/20/18 23:18:01 host/nfs03.nix.mds.xyz@NIX.MDS.XYZ (arcfour-hmac)
[root@nfs03 ~]#

We can see the back and fourth communication between nfs02 (192.168.0.119 ) and nfs03 ( 192.168.0.125 ) in the log files by the following entries:

04:13:32.932068 00:50:56:86:2d:21 > 00:50:56:86:3a:74, ethertype IPv4 (0×0800), length 90: (tos 0×0, ttl 64, id 41490, offset 0, flags [DF], proto TCP (6), length 76)
    192.168.0.119.2049 > 192.168.0.125.46652: Flags [P.], cksum 0x236c (correct), seq 3129116439:3129116463, ack 1682320548, win 238, options [nop,nop,TS val 2303240 ecr 2060939], length 24: NFS reply xid 2610793391 reply ERR 20: Auth Rejected Credentials (client should begin new session)
        0×0000:  0050 5686 3a74 0050 5686 2d21 0800 4500  .PV.:t.PV.-!..E.
        0×0010:  004c a212 4000 4006 1655 c0a8 0077 c0a8  .L..@.@..U…w..
        0×0020:  007d 0801 b63c ba82 8717 6446 2ca4 8018  .}…<….dF,…
        0×0030:  00ee 236c 0000 0101 080a 0023 2508 001f  ..#l…….#%…
        0×0040:  728b 8000 0014 9b9d 8baf 0000 0001 0000  r……………
        0×0050:  0001 0000 0001 0000 0002                 ……….

VERIFIED SOLUTION

NOTE: Don't forget to restart autofs on the clients and test from more then one client.

The requirement is to add a principal for the server.  In the case of FreeIPA, we use the following ( service-allow-retrieve-keytab allows the subsequent activity using -r option ):  

[root@idmipa01 ~]# ipa service-add nfs/nfs03.nix.mds.xyz

[root@idmipa01 ~]# ipa service-allow-retrieve-keytab nfs/nfs-c01.nix.mds.xyz@NIX.MDS.XYZ –groups=admins
[root@idmipa01 ~]# ipa service-allow-retrieve-keytab nfs/nfs-c01.nix.mds.xyz@NIX.MDS.XYZ –hosts={nfs01.nix.mds.xyz,nfs02.nix.mds.xyz,nfs03.nix.mds.xyz} 

[root@idmipa01 ~]# ipa service-allow-retrieve-keytab nfs/nfs03.nix.mds.xyz@NIX.MDS.XYZ –groups=admins  
[root@idmipa01 ~]# ipa service-allow-retrieve-keytab nfs/nfs02.nix.mds.xyz@NIX.MDS.XYZ –groups=admins
  
[root@idmipa01 ~]# ipa service-allow-retrieve-keytab nfs/nfs01.nix.mds.xyz@NIX.MDS.XYZ –groups=admins  

[root@idmipa01 ~]# ipa service-allow-retrieve-keytab nfs/nfs03.nix.mds.xyz –hosts=nfs03.nix.mds.xyz
[root@idmipa01 ~]# ipa service-allow-retrieve-keytab nfs/nfs02.nix.mds.xyz –hosts=nfs02.nix.mds.xyz
[root@idmipa01 ~]# ipa service-allow-retrieve-keytab nfs/nfs01.nix.mds.xyz –hosts=nfs01.nix.mds.xyz

[root@nfs03 ~]# kinit admin    # Or the user you permissioned above.
[root@nfs03 ~]# ipa-getkeytab -s idmipa01.nix.mds.xyz -p nfs/nfs-c01.nix.mds.xyz -k /etc/krb5.keytab -r 

[root@nfs03 ~]# ipa-getkeytab -s idmipa01.nix.mds.xyz -p nfs/nfs03.nix.mds.xyz -k /etc/krb5.keytab -r 

Test using kinit when done:

[root@nfs02 sssd]# kinit -kt /etc/krb5.keytab nfs/nfs-c01.nix.mds.xyz@NIX.MDS.XYZ
[root@nfs02 sssd]# klist
Ticket cache: KEYRING:persistent:0:krb_ccache_t3UCYMN
Default principal: nfs/nfs-c01.nix.mds.xyz@NIX.MDS.XYZ

Valid starting       Expires              Service principal
11/11/2018 15:45:54  11/12/2018 15:45:54  krbtgt/NIX.MDS.XYZ@NIX.MDS.XYZ
[root@nfs02 sssd]#

Note the use of -r above, this preserves the KVNO. In the event you're using a local, non-IPA KDC, issue the following set of commands:

# kadmin.local
kadmin.local:  addprinc host/nfs-c01.nix.mds.xyz@NIX.MDS.XYZ
Enter password for principal "
host/nfs-c01.nix.mds.xyz@NIX.MDS.XYZ": <password>
Re-enter password for principal "
host/nfs-c01.nix.mds.xyz@NIX.MDS.XYZ": <password>
kadmin.local:  addprinc nfs/nfs-c01.nix.mds.xyz@NIX.MDS.XYZ 
Enter password for principal "nfs/nfs-c01.nix.mds.xyz@NIX.MDS.XYZ": <password>
Re-enter password for principal "nfs
/nfs-c01.nix.mds.xyz@NIX.MDS.XYZ": <password>
kadmin.local:

Adjust the above paramaters to the addprinc command accordingly to the below options:

kadmin.local:  addprinc
usage: add_principal [options] principal
        options are:
                [-randkey|-nokey] [-x db_princ_args]* [-expire expdate] [-pwexpire pwexpdate] [-maxlife maxtixlife]
                [-kvno kvno] [-policy policy] [-clearpolicy]
                [-pw password] [-maxrenewlife maxrenewlife]
                [-e keysaltlist]
                [{+|-}attribute]
        attributes are:
                allow_postdated allow_forwardable allow_tgs_req allow_renewable
                allow_proxiable allow_dup_skey allow_tix requires_preauth
                requires_hwauth needchange allow_svr password_changing_service
                ok_as_delegate ok_to_auth_as_delegate no_auth_data_required
                lockdown_keys

where,
        [-x db_princ_args]* – any number of database specific arguments.
                        Look at each database documentation for supported arguments
kadmin.local:

to achieve the below results:

kadmin.local:  getprinc nfs/nfs-c01.nix.mds.xyz@NIX.MDS.XYZ
Principal: nfs/nfs-c01.nix.mds.xyz@NIX.MDS.XYZ
Expiration date: [never]
Last password change: Tue Mar 06 23:24:10 EST 2018
Password expiration date: [never]
Maximum ticket life: 1 day 00:00:00
Maximum renewable life: 7 days 00:00:00
Last modified: Tue Mar 06 23:24:10 EST 2018 (nfs/nfs-c01.nix.mds.xyz@NIX.MDS.XYZ)
Last successful authentication: [never]
Last failed authentication: Sun Nov 11 11:00:08 EST 2018
Failed password attempts: 8
Number of keys: 4
Key: vno 6, aes256-cts-hmac-sha1-96:special
Key: vno 6, aes128-cts-hmac-sha1-96:special
Key: vno 6, des3-cbc-sha1:special
Key: vno 6, arcfour-hmac:special
MKey: vno 1
Attributes: REQUIRES_PRE_AUTH
Policy: [none]
kadmin.local:
kadmin.local:
kadmin.local:  get princ host/nfs-c01.nix.mds.xyz@NIX.MDS.XYZ
kadmin.local: Unknown request "get".  Type "?" for a request list.
kadmin.local:  getprinc host/nfs-c01.nix.mds.xyz@NIX.MDS.XYZ
Principal: host/nfs-c01.nix.mds.xyz@NIX.MDS.XYZ
Expiration date: [never]
Last password change: [never]
Password expiration date: [never]
Maximum ticket life: 1 day 00:00:00
Maximum renewable life: 7 days 00:00:00
Last modified: Wed Dec 31 19:00:00 EST 1969 (principal@UNINITIALIZED)
Last successful authentication: [never]
Last failed authentication: [never]
Failed password attempts: 0
Number of keys: 0
MKey: vno 1
Attributes: REQUIRES_PRE_AUTH
Policy: [none]
kadmin.local:

Write keytabs to a separate file using the ktutil command and preserving the KVNO number you got from above (highlighted) .  Example commands are as follows for the various encryption algorigthms .  Use all or some as applicable:  

[root@nfs03 ~]# ktutil 

ktutil: add_entry -password -p nfs/nfs-c01.nix.mds.xyz@NIX.MDS.XYZ -k 4 -e des3-cbc-sha1-kd 
Password for nfs/nfs-c01.nix.mds.xyz@NIX.MDS.XYZ: 

ktutil: add_entry -password -p nfs/nfs-c01.nix.mds.xyz@NIX.MDS.XYZ -k 4 -e arcfour-hmac-md5 
Password for nfs/nfs-c01.nix.mds.xyz@NIX.MDS.XYZ: 

ktutil: add_entry -password -p nfs/nfs-c01.nix.mds.xyz@NIX.MDS.XYZ -k 4 -e des-hmac-sha1 
Password for nfs/nfs-c01.nix.mds.xyz@NIX.MDS.XYZ: 

ktutil: add_entry -password -p nfs/nfs-c01.nix.mds.xyz@NIX.MDS.XYZ -k 4 -e des-cbc-md5 
Password for nfs/nfs-c01.nix.mds.xyz@NIX.MDS.XYZ: 

ktutil: add_entry -password -p nfs/nfs-c01.nix.mds.xyz@NIX.MDS.XYZ -k 4 -e des-cbc-md4 
Password for nfs/nfs-c01.nix.mds.xyz@NIX.MDS.XYZ: 

 

ktutil: add_entry -password -p nfs/nfs03.nix.mds.xyz@NIX.MDS.XYZ -k 4 -e des3-cbc-sha1-kd 
Password for nfs/nfs03.nix.mds.xyz@NIX.MDS.XYZ: 

ktutil: add_entry -password -p nfs/nfs03.nix.mds.xyz@NIX.MDS.XYZ -k 4 -e arcfour-hmac-md5 
Password for nfs/nfs03.nix.mds.xyz@NIX.MDS.XYZ: 

ktutil: add_entry -password -p nfs/nfs03.nix.mds.xyz@NIX.MDS.XYZ -k 4 -e des-hmac-sha1 
Password for nfs/nfs03.nix.mds.xyz@NIX.MDS.XYZ: 

ktutil: add_entry -password -p nfs/nfs03.nix.mds.xyz@NIX.MDS.XYZ -k 4 -e des-cbc-md5 
Password for nfs/nfs03.nix.mds.xyz@NIX.MDS.XYZ: 

ktutil: add_entry -password -p nfs/nfs03.nix.mds.xyz@NIX.MDS.XYZ -k 4 -e des-cbc-md4 
Password for nfs/nfs03.nix.mds.xyz@NIX.MDS.XYZ: 

Verify the written entries:

ktutil:  l -k -t -e
slot KVNO Timestamp         Principal
—- —- —————– —————————————————
   1    4 11/11/18 13:27:45      nfs/nfs-c01.nix.mds.xyz@NIX.MDS.XYZ (des3-cbc-sha1)  (0x977c7f3b20b5b694dafb8c6b0749a420e32cf29bd96d803d)
   2    4 11/11/18 13:28:02      nfs/nfs-c01.nix.mds.xyz@NIX.MDS.XYZ (arcfour-hmac)  (0x8846f7eaee8fb117ad06bdd830b7586c)
   3    4 11/11/18 13:28:59      nfs/nfs-c01.nix.mds.xyz@NIX.MDS.XYZ (des-hmac-sha1)  (0x2ab6760e97d0672a)
   4    4 11/11/18 13:29:10      nfs/nfs-c01.nix.mds.xyz@NIX.MDS.XYZ (des-cbc-md5)  (0x0efbf225dc201cf8)
   5    4 11/11/18 13:29:23      nfs/nfs-c01.nix.mds.xyz@NIX.MDS.XYZ (des-cbc-md4)  (0x0efbf225dc201cf8)
   6    4 11/11/18 13:29:39        nfs/nfs03.nix.mds.xyz@NIX.MDS.XYZ (des3-cbc-sha1)  (0xfe73237c160e0d7357d96b61fbcbce437fdf7a9e08ab239d)
   7    4 11/11/18 13:29:49        nfs/nfs03.nix.mds.xyz@NIX.MDS.XYZ (arcfour-hmac)  (0x8846f7eaee8fb117ad06bdd830b7586c)
   8    4 11/11/18 13:30:03        nfs/nfs03.nix.mds.xyz@NIX.MDS.XYZ (des-hmac-sha1)  (0x4557862576d9e973)
   9    4 11/11/18 13:30:12        nfs/nfs03.nix.mds.xyz@NIX.MDS.XYZ (des-cbc-md5)  (0x6d54a1abe0194a25)
  10    4 11/11/18 13:30:21        nfs/nfs03.nix.mds.xyz@NIX.MDS.XYZ (des-cbc-md4)  (0x6d54a1abe0194a25)
ktutil:

Now that you've created the entries above, write a keytab file for them: 

ktutil: wkt /some/path/you/choose/nfs.keytab 

Merge the two keytabs on the system: 

cp -ip /etc/krb5.keytab  /etc/krb5.keytab-backup

[root@nfs03 ~]# ktutil 
  ktutil: rkt /some/path/you/choose/nfs.keytab
  ktutil: rkt /etc/krb5.keytab
  ktutil: wkt /etc/krb5.keytab
  ktutil: quit

Verify the newly created keytab file:

[root@nfs03 ~]# klist -kte /etc/krb5.keytab
Keytab name: FILE:/etc/krb5.keytab-new
KVNO Timestamp         Principal
—- —————– ——————————————————–
   1 11/11/18 13:35:28 host/nfs03.nix.mds.xyz@NIX.MDS.XYZ (aes256-cts-hmac-sha1-96)
   1 11/11/18 13:35:28 host/nfs03.nix.mds.xyz@NIX.MDS.XYZ (aes128-cts-hmac-sha1-96)
   1 11/11/18 13:35:28 host/nfs03.nix.mds.xyz@NIX.MDS.XYZ (des3-cbc-sha1)
   1 11/11/18 13:35:28 host/nfs03.nix.mds.xyz@NIX.MDS.XYZ (arcfour-hmac)
   4 11/11/18 13:35:28 nfs/nfs-c01.nix.mds.xyz@NIX.MDS.XYZ (des3-cbc-sha1)
   4 11/11/18 13:35:28 nfs/nfs-c01.nix.mds.xyz@NIX.MDS.XYZ (arcfour-hmac)
   4 11/11/18 13:35:28 nfs/nfs-c01.nix.mds.xyz@NIX.MDS.XYZ (des-hmac-sha1)
   4 11/11/18 13:35:28 nfs/nfs-c01.nix.mds.xyz@NIX.MDS.XYZ (des-cbc-md5)
   4 11/11/18 13:35:28 nfs/nfs-c01.nix.mds.xyz@NIX.MDS.XYZ (des-cbc-md4)
   4 11/11/18 13:35:28 nfs/nfs03.nix.mds.xyz@NIX.MDS.XYZ (des3-cbc-sha1)
   4 11/11/18 13:35:28 nfs/nfs03.nix.mds.xyz@NIX.MDS.XYZ (arcfour-hmac)
   4 11/11/18 13:35:28 nfs/nfs03.nix.mds.xyz@NIX.MDS.XYZ (des-hmac-sha1)
   4 11/11/18 13:35:28 nfs/nfs03.nix.mds.xyz@NIX.MDS.XYZ (des-cbc-md5)
   4 11/11/18 13:35:28 nfs/nfs03.nix.mds.xyz@NIX.MDS.XYZ (des-cbc-md4)
[root@nfs03 ~]#

PROBLEMS SECTION

In case you get this error:

[root@nfs03 ~]# ipa-getkeytab -s idmipa01.nix.mds.xyz -p nfs/nfs03.nix.mds.xyz -k /etc/krb5.keytab
Failed to load translations
Failed to parse result: PrincipalName not found.

Retrying with pre-4.0 keytab retrieval method…
Failed to parse result: PrincipalName not found.

Failed to get keytab!
Failed to get keytab
[root@nfs03 ~]#

Remember to first create the service principal on the IPA KDC as mentioned above using ipa service-add nfs/nfs03.nix.mds.xyz .

If you get the following error when using -r option when trying to import the keys, don't forget to kinit and permission the activity on the IPA server:

[root@nfs03 ~]# ipa-getkeytab -s idmipa01.nix.mds.xyz -p nfs/nfs03.nix.mds.xyz -k /etc/krb5.keytab -r
Failed to load translations
Failed to parse result: Insufficient access rights

Failed to get keytab
[root@nfs03 ~]#

Once you permission retrieval of keytabs / principals using ipa service-allow-retrieve-keytab, your attempt will succeed:

[root@nfs03 ~]# ipa-getkeytab -s idmipa01.nix.mds.xyz -p nfs/nfs03.nix.mds.xyz -k /etc/krb5.keytab -r
Failed to load translations
Keytab successfully retrieved and stored in: /etc/krb5.keytab
[root@nfs03 ~]#

If you accidentally omitted the -r option when getting keytabs from the KDC / IPA server, you'll need to reimport them using -r otherwise the KVNO number won't match and you'll get this:

[root@nfs02 ~]# kinit -kt /etc/krb5.keytab nfs/nfs-c01.nix.mds.xyz@NIX.MDS.XYZ
kinit: Preauthentication failed while getting initial credentials
[root@nfs02 ~]#

In that scenario you'll need to manually edit out the offending keytab entries and reimport:

[root@nfs02 sssd]# ktutil
ktutil: rkt /etc/krb5.keytab
ktutil: l
ktutil: deltent <NUM>
ktutil: wkt /some/temp/path/krb5.keytab-new
ktutil: quit

Check the entries using:

[root@nfs02 sssd]# klist -kte /etc/krb5.keytab-new
Keytab name: FILE:/etc/krb5.keytab-new
KVNO Timestamp           Principal
—- ——————- ——————————————————
   1 11/11/2018 15:22:23 host/nfs02.nix.mds.xyz@NIX.MDS.XYZ (aes256-cts-hmac-sha1-96)
   1 11/11/2018 15:22:23 host/nfs02.nix.mds.xyz@NIX.MDS.XYZ (aes128-cts-hmac-sha1-96)
   1 11/11/2018 15:22:23 host/nfs02.nix.mds.xyz@NIX.MDS.XYZ (des3-cbc-sha1)
   1 11/11/2018 15:22:23 host/nfs02.nix.mds.xyz@NIX.MDS.XYZ (arcfour-hmac)
   4 11/11/2018 15:22:23 nfs/nfs02.nix.mds.xyz@NIX.MDS.XYZ (aes256-cts-hmac-sha1-96)
   4 11/11/2018 15:22:23 nfs/nfs02.nix.mds.xyz@NIX.MDS.XYZ (aes128-cts-hmac-sha1-96)
   4 11/11/2018 15:22:23 nfs/nfs02.nix.mds.xyz@NIX.MDS.XYZ (des3-cbc-sha1)
   4 11/11/2018 15:22:23 nfs/nfs02.nix.mds.xyz@NIX.MDS.XYZ (arcfour-hmac)
[root@nfs02 sssd]#

Alongside the above, we needed to enable Kerberos on nfs03:

[root@nfs02 ganesha]# grep SecType /etc/ganesha/export.conf
    SecType = "sys","krb5","krb5i","krb5p";     # Security flavors supported
[root@nfs02 ganesha]#

[root@nfs03 ganesha]# grep SecType /etc/ganesha/export.conf
    SecType = "sys";                            # Security flavors supported
[root@nfs03 ganesha]#

This is another reason why the tcpdump command showed that nfs03 couldn't authenticate since it didn't know what KRB5 messages from nfs02 were about.  But this did not work either.

Next check the autofs service and try the mount of othe rmachines:

[root@psql01 sssd]# systemctl list-unit-files|grep auto
proc-sys-fs-binfmt_misc.automount             static
auto-net.service                              enabled
autofs.service                                disabled
autovt@.service                               enabled
rhel-autorelabel-mark.service                 static
rhel-autorelabel.service                      static
sssd-autofs.service                           indirect
sssd-autofs.socket                            disabled
[root@psql01 sssd]#

And try to login again.  This time it was a success!  Now let's fix mysql01 and restart the autofs daemon:

[root@mysql01 ~]# systemctl list-unit-files|grep -Ei auto
proc-sys-fs-binfmt_misc.automount             static
auto-net.service                              enabled
autofs.service                                enabled
autovt@.service                               enabled
rhel-autorelabel-mark.service                 static
rhel-autorelabel.service                      static
sssd-autofs.service                           indirect
sssd-autofs.socket                            disabled
[root@mysql01 ~]#
[root@mysql01 ~]#
[root@mysql01 ~]# ps -ef|grep -Ei auto
root     17817 17812  0 22:33 ?        00:00:00 /usr/libexec/sssd/sssd_autofs –uid 0 –gid 0 –logger=files
root     18835 17688  0 22:53 pts/0    00:00:00 grep –color=auto -Ei auto
[root@mysql01 ~]# mount nfs-c01:/n /m
[root@mysql01 ~]# umount /m

[root@mysql01 ~]#

Now restarting autofs on this client yielded our successful mounts!

[root@mysql01 ~]# ps -ef|grep -Ei auto
root     17817 17812  0 22:33 ?        00:00:00 /usr/libexec/sssd/sssd_autofs –uid 0 –gid 0 –logger=files
root     19014     1  0 22:56 ?        00:00:00 /usr/sbin/automount –debug –pid-file /run/autofs.pid
root     19489 19298  0 23:02 pts/0    00:00:00 grep –color=auto -Ei auto
[root@mysql01 ~]#

Retry the mount command using the VIP!  Enjoy!

Good Luck!

Cheers,
TK

REF: https://www.ibm.com/support/knowledgecenter/en/SSZUMP_7.1.2/management_sym/sym_kerberos_creating_principal_keytab.html
REF: https://sourceforge.net/p/nfs-ganesha/mailman/message/30653393/ 

 

rpc mount export: RPC: Unable to receive; errno = Connection refused

For the below errors:

[root@psql02 log]# showmount -e nfs02
rpc mount export: RPC: Unable to receive; errno = Connection refused
[root@psql02 log]#

Apr 16 01:12:37 nfs02 kernel: FINAL_REJECT: IN=eth0 OUT= MAC=00:50:56:86:2d:21:00:50:56:86:3c:c7:08:00 SRC=192.168.0.124 DST=192.168.0.119 LEN=60 TOS=0×00 PREC=0×00 TTL=64 ID=44729 DF PROTO=TCP SPT=978 DPT=20048 WINDOW=29200 RES=0×00 SYN URGP=0

[root@nfs02 log]#
[root@nfs02 log]#
[root@nfs02 log]#
[root@nfs02 log]# firewall-cmd –zone=public –list-all
public
  target: default
  icmp-block-inversion: no
  interfaces:
  sources:
  services: haproxy
  ports: 20048/udp 2049/tcp 111/tcp 111/udp 24007-24008/tcp 38465-38469/tcp 4501/tcp 4501/udp 22/tcp 22/udp 49000-59999/udp 49000-59999/tcp 9000/tcp 9000/udp 137/udp 138/udp
  protocols:
  masquerade: no
  forward-ports:
  source-ports:
  icmp-blocks:
  rich rules:

[root@nfs02 log]#

Ensure you have port 20048 TCP added to your firewall:

  995  firewall-cmd –zone=public –permanent –add-port=20048/tcp
  996  firewall-cmd –reload

Cheers,
TK

psql: error while loading shared libraries: libpq.so.rh-postgresql95-5: cannot open shared object file: No such file or directory

Well heck:

-bash-4.2$ psql
psql: error while loading shared libraries: libpq.so.rh-postgresql95-5: cannot open shared object file: No such file or directory
-bash-4.2$

So let's see what's going on:

[root@ovirt01 ~]# find / -iname libpq.so*
/usr/lib64/libpq.so.5
/usr/lib64/libpq.so.5.5
/opt/rh/rh-postgresql95/root/usr/lib64/libpq.so.rh-postgresql95-5
/opt/rh/rh-postgresql95/root/usr/lib64/libpq.so.rh-postgresql95-5.8
[root@ovirt01 ~]#

So we can see it's in the lib64 path.  And within the root postgres folder, we see that usr/lib is empty that lib points too:

[root@ovirt01 root]# find / -iname psql
/opt/rh/rh-postgresql95/root/usr/bin/psql
[root@ovirt01 root]# pwd
/opt/rh/rh-postgresql95/root
[root@ovirt01 root]# ls -altrid lib
201829110 lrwxrwxrwx. 1 root root 7 Feb 12 11:08 lib -> usr/lib
[root@ovirt01 root]# pwd
/opt/rh/rh-postgresql95/root
[root@ovirt01 root]# ls -altri usr/lib/
total 4
134420487 dr-xr-xr-x.  2 root root    6 Feb 16  2016 .
 67638534 drwxr-xr-x. 13 root root 4096 Feb 12 11:08 ..
[root@ovirt01 root]#

So obviously, nothing that uses usr/lib/ will get anything useful out of it.  But the following path under the same folder above has lot's of usefull things:

[root@ovirt01 root]# ls -altrid lib64
201829111 lrwxrwxrwx. 1 root root 9 Feb 12 11:08 lib64 -> usr/lib64
[root@ovirt01 root]#

Since the lib folder doesn't have anything useful in it, a simple solution is to link lib to usr/lib64 instead.  So let's do that.  Sure enough:

201829110 lrwxrwxrwx.  1 root root    9 Apr 15 00:56 lib -> usr/lib64

And here we go again:

-bash-4.2$ strace psql
execve("/opt/rh/rh-postgresql95/root/usr/bin/psql", ["psql"], [/* 21 vars */]) = 0
brk(NULL)                               = 0x17b5000
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7fa7a1ff5000
access("/etc/ld.so.preload", R_OK)      = -1 ENOENT (No such file or directory)
open("/etc/ld.so.cache", O_RDONLY|O_CLOEXEC) = 3
fstat(3, {st_mode=S_IFREG|0644, st_size=42118, …}) = 0
mmap(NULL, 42118, PROT_READ, MAP_PRIVATE, 3, 0) = 0x7fa7a1fea000
close(3)                                = 0
open("/lib64/tls/x86_64/libpq.so.rh-postgresql95-5", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
stat("/lib64/tls/x86_64", 0x7ffcdfe3c4a0) = -1 ENOENT (No such file or directory)
open("/lib64/tls/libpq.so.rh-postgresql95-5", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
stat("/lib64/tls", {st_mode=S_IFDIR|0555, st_size=6, …}) = 0
open("/lib64/x86_64/libpq.so.rh-postgresql95-5", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
stat("/lib64/x86_64", 0x7ffcdfe3c4a0)   = -1 ENOENT (No such file or directory)
open("/lib64/libpq.so.rh-postgresql95-5", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
stat("/lib64", {st_mode=S_IFDIR|0555, st_size=40960, …}) = 0
open("/usr/lib64/tls/x86_64/libpq.so.rh-postgresql95-5", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
stat("/usr/lib64/tls/x86_64", 0x7ffcdfe3c4a0) = -1 ENOENT (No such file or directory)
open("/usr/lib64/tls/libpq.so.rh-postgresql95-5", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
stat("/usr/lib64/tls", {st_mode=S_IFDIR|0555, st_size=6, …}) = 0
open("/usr/lib64/x86_64/libpq.so.rh-postgresql95-5", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
stat("/usr/lib64/x86_64", 0x7ffcdfe3c4a0) = -1 ENOENT (No such file or directory)
open("/usr/lib64/libpq.so.rh-postgresql95-5", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
stat("/usr/lib64", {st_mode=S_IFDIR|0555, st_size=40960, …}) = 0
writev(2, [{"psql", 4}, {": ", 2}, {"error while loading shared libra"..., 36}, {": ", 2}, {"libpq.so.rh-postgresql95-5", 26}, {": ", 2}, {"cannot open shared object file", 30}, {": ", 2}, {"No such file or directory", 25}, {"\n", 1}], 10psql: error while loading shared libraries: libpq.so.rh-postgresql95-5: cannot open shared object file: No such file or directory
) = 130
exit_group(127)                         = ?
+++ exited with 127 +++
-bash-4.2$

 

So we need to add it to the default library path.  Easy enough:

[root@ovirt01 ld.so.conf.d]#
[root@ovirt01 ld.so.conf.d]# cat postgres-x86_64.conf
/opt/rh/rh-postgresql95/root/lib
/opt/rh/rh-postgresql95/root/lib64
[root@ovirt01 ld.so.conf.d]# ldconfig
[root@ovirt01 ld.so.conf.d]# strings /etc/ld.so.cache |grep -Ei postgresql95
libpq.so.rh-postgresql95-5
/opt/rh/rh-postgresql95/root/lib64/libpq.so.rh-postgresql95-5
libpgtypes.so.rh-postgresql95-3
/opt/rh/rh-postgresql95/root/lib64/libpgtypes.so.rh-postgresql95-3
libecpg_compat.so.rh-postgresql95-3
/opt/rh/rh-postgresql95/root/lib64/libecpg_compat.so.rh-postgresql95-3
libecpg.so.rh-postgresql95-6
/opt/rh/rh-postgresql95/root/lib64/libecpg.so.rh-postgresql95-6
[root@ovirt01 ld.so.conf.d]#

And let's try again.  And sure enough, we have a winner:

-bash-4.2$
-bash-4.2$ psql
psql (9.5.9)
Type "help" for help.

postgres=# \l
                                             List of databases
         Name         |        Owner         | Encoding |   Collate   |    Ctype    |   Access privileges
———————-+———————-+———-+————-+————-+———————–
 engine               | engine               | UTF8     | en_US.UTF-8 | en_US.UTF-8 |
 ovirt_engine_history | ovirt_engine_history | UTF8     | en_US.UTF-8 | en_US.UTF-8 |
 postgres             | postgres             | UTF8     | en_US.UTF-8 | en_US.UTF-8 |
 template0            | postgres             | UTF8     | en_US.UTF-8 | en_US.UTF-8 | =c/postgres          +
                      |                      |          |             |             | postgres=CTc/postgres
 template1            | postgres             | UTF8     | en_US.UTF-8 | en_US.UTF-8 | =c/postgres          +
                      |                      |          |             |             | postgres=CTc/postgres
(5 rows)

postgres=#

Cheers,
TK

 


     
  Copyright © 2003 - 2013 Tom Kacperski (microdevsys.com). All rights reserved.

Creative Commons License
This work is licensed under a Creative Commons Attribution 3.0 Unported License