Header Shadow Image


DNS issue: Can’t ping but nslookup works

DNS issue: Can't ping but nslookup works

You can do several things in this case. Start Services then recycle DHCP Client.  ipconfig /flushdns and netsh int ip reset resettcpip.txt can fix this temporarily as well.

I've elected to simply stop DHCP Client and let the system do all lookups against my internal DNS servers.

This still leaves the problem of the DHCP Client not working correctly which I'm not 100% sure about. 

Can lookup event viewer to determine the issue however there was nothing in event viewer for this.  

Cheers,
TK

REF: https://merabheja.com/fix-nslookup-works-but-ping-fails-in-windows-10/ 

Setup a USB Null Modem for Kernel Dump Captures

We will setup a serial null modem cable for administering and connecting to a physical machine via another in the event that:  

1) We want to capture kernel crashes and dumps.  
2) Login to the machine machine remotely via another linux box to do things like restart the network.  

For this we will need:  

1) One of DB9 RS232 Serial Null Modem Cable F/F
2) Two of USB to RS232 Serial Port DB9 9 Pin Male

Connect the USB to Serial Adapter to both systems.  Following it set the tty specifc settings on ttyUSB0:

6889  stty -F /dev/ttyUSB0 115200 cs8 -cstopb -parenb
6890  stty -F /dev/ttyUSB0 -a

 

Test the serial connection by running the following:

6894  /sbin/agetty -L 115200 ttyUSB0
 

Use minicom from the connecting linux host.  When test running /sbin/agetty -L 115200 ttyUSB0, you should see a prompt:

[root@rfc1178-01 ~]# minicom

Welcome to minicom 2.6.2

OPTIONS: I18n
Compiled on Jun 25 2013, 10:33:48.
Port /dev/ttyUSB0, 11:30:08

Press CTRL-A Z for help on special keys

Scientific Linux release 6.10 (Carbon)
Kernel 4.18.19 on an x86_64

mbpc-pc login: root
Password:
Last login: Fri Apr 19 12:51:19 from 192.168.0.76
0;root@mbpc-pc:~[root@mbpc-pc ~]#
0;root@mbpc-pc:~[root@mbpc-pc ~]#
0;root@mbpc-pc:~[root@mbpc-pc ~]#
0;root@mbpc-pc:~[root@mbpc-pc ~]# uptime
 13:03:19 up 14 min,  1 user,  load average: 0.06, 0.13, 0.18
0;root@mbpc-pc:~[root@mbpc-pc ~]#

 

You should be able to login as above confirming the physical layer (USB to Serial -> Null Modem Female-to-Female -> Serial to USB) functions correctly and root is allowed to login.  Configure the kernel to send messages on the tty:

title Scientific Linux (4.18.19)
        root (hd0,0)
        kernel /vmlinuz-4.18.19 ro root=/dev/mapper/mbpcvg-rootlv rd_LVM_LV=mbpcvg/rootlv rd_LVM_LV=VGEntertain/olv_swap rd_LVM_LV=mbpcvg/swaplv rd_NO_LUKS rd_NO_MD rd_NO_DM LANG=en_US.UTF-8 SYSFONT=latarcyrheb-sun16 KEYBOARDTYPE=pc KEYTABLE=us rhgb nomodeset irqpoll pcie_aspm=off amd_iommu=on crashkernel=0M-2G:128M,2G-6G:256M,6G-8G:512M,8G-:768M pci=nomsi nohpet clocksource=rtc console=ttyUSB0,115200n8 console=tty0

Configure:

[root@mbpc-pc ~]# cat /etc/securetty |grep USB
ttyUSB0
[root@mbpc-pc ~]# cat /etc/init/ttyUSB0.conf
# ttyUSB0 – agetty
#
# This service maintains a agetty on ttyUSB0.

stop on runlevel [S06]
start on runlevel [12435]

respawn
exec agetty -L /dev/ttyUSB0 115200
[root@mbpc-pc ~]#

 

Configure the minicom settings on the external host (CTRL – A, followed by Z.  Look for option cOnfigure Minicom..O or directly using CTRL – A followed by O):

+—–[configuration]——+
| Filenames and paths      |
| File transfer protocols  |
| Serial port setup        |
| Modem and dialing        |
| Screen and keyboard      |
| Save setup as dfl        |
| Save setup as..          |
| Exit                     |
+————————–+

Followed by the settings below:

+———————————————————————–+
| A –    Serial Device      : /dev/ttyUSB0                              |
|                                                                       |
| C –   Callin Program      :                                           |
| D –  Callout Program      :                                           |
| E –    Bps/Par/Bits       : 115200 8N1                                |
| F – Hardware Flow Control : No                                        |
| G – Software Flow Control : Yes                                       |
|                                                                       |
|    Change which setting?                                              |
+———————————————————————–+

Hit ESC when done and save the configuration:

| Save setup as dfl        |

Restart the server to ensure changes take effect.  You should now see messages from the minicom terminal on the secondary system:

Welcome to minicom 2.6.2

OPTIONS: I18n
Compiled on Jun 25 2013, 10:33:48.
Port /dev/ttyUSB0, 12:03:52

Press CTRL-A Z for help on special keys


Scientific Linux release 6.10 (Carbon)
Kernel 4.18.19 on an x86_64

mbpc-pc login:

Next, test restart with the console connected to see restart messages being printed:

Linux version 4.18.19 (root@mbpc-pc) (gcc version 4.4.7 201209
Command line: ro root=/dev/mapper/mbpcvg-rootlv rd_LVM_LV=mbpcvg/rootlv rd_LVM_8
x86/fpu: x87 FPU will use FXSAVE
BIOS-provided physical RAM map:
BIOS-e820: [mem 0x0000000000000000-0x0000000000093fff] usable
BIOS-e820: [mem 0x000000000009f800-0x000000000009ffff] reserved
BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved
BIOS-e820: [mem 0x0000000000100000-0x00000000dfceffff] usable
BIOS-e820: [mem 0x00000000dfcf0000-0x00000000dfcf0fff] ACPI NVS
BIOS-e820: [mem 0x00000000dfcf1000-0x00000000dfcfffff] ACPI data
BIOS-e820: [mem 0x00000000dfd00000-0x00000000dfdfffff] reserved
BIOS-e820: [mem 0x00000000e0000000-0x00000000efffffff] reserved
BIOS-e820: [mem 0x00000000fec00000-0x00000000ffffffff] reserved
BIOS-e820: [mem 0x0000000100000000-0x000000011fffffff] usable
NX (Execute Disable) protection: active
SMBIOS 2.4 present.
DMI: Gigabyte Technology Co., Ltd. GA-890XA-UD3/GA-890XA-UD3, BIOS FC 08/02/2010
AGP: No AGP bridge found

 

Testing can be done using this:

[root@mbpc-pc cores]# echo "This is a ttyUSB0 test from mbpc-pc." > /dev/ttyUSB0
[root@mbpc-pc cores]#

 

Result on the console is:

[root@mbpc-pc ~]# This is a ttyUSB0 test from mbpc-pc.
CTRL-A Z for help |115200 8N1 | NOR | Minicom 2.6.2  | VT102 | Online 08:12

 

If you get a prompt but no kernel messages, ensure you compile the following options into the kernel:

CONFIG_USB_SERIAL=y
CONFIG_USB_SERIAL_CONSOLE=y
CONFIG_USB_SERIAL_EDGEPORT_TI=y
CONFIG_USB_SERIAL_MOS7840=y

You can find the above in the make menuconfig driver sections.  You can find the above by pressing forward slash ( / ) followed by the search string CONFIG_USB_SERIAL which will give you the path of the option:


  |   Location:                                               |
  |     -> Device Drivers                                     |
  |       -> USB support (USB_SUPPORT [=y])                   |
  |         -> USB Serial Converter support (USB_SERIAL [=y]) |

 

If you get kernel messages but no prompt (after enabling additional kernel parameters above) then try adding the following additional parameters:

[root@mbpc-pc linux-4.18.19]# cat /etc/init/ttyUSB0.conf
# ttyUSB0 – agetty
#
# This service maintains a agetty on ttyUSB0.

stop on runlevel [S06] and (
            not-container or
            container CONTAINER=lxc or
            container CONTAINER=lxc-libvirt)

start on runlevel [12435]

respawn
exec agetty -L /dev/ttyUSB0 115200 vt100
[root@mbpc-pc linux-4.18.19]#

 

However for us it was just a matter of restarting against since agetty didn't come up the first time.  If with the addition of the above items in green you now get a console, all is good and you should be all set to capture the kernel messages when crashes happen!

REF: https://wiki.freepbx.org/display/PC/Capturing+Kernel+Panic+via+Serial+Port

Cheers,
TK

com.cloudera.cmf.service.CommandException: java.io.IOException: Cannot create command directory: /var/lib/cloudera-scm-server/temp/commands/114

Getting this?

com.cloudera.cmf.service.CommandException: java.io.IOException: Cannot create command directory: /var/lib/cloudera-scm-server/temp/commands/114

it's because we blow the folder away.  Reinstall the packages:

[root@cm-r01nn01 ~]# yum reinstall cloudera-manager-daemons cloudera-manager-agent cloudera-manager-server -y

Thx,
TK

Kernel Panic and Disabling HPET

Recent kernel panics have pointed to an issue with the HPET timer on some motherboards.  To disable, add the following to the kernel line:

nohpet clocksource=rtc

Then also disable HPET from BIOS.

Thx,
TK

 

locale: Cannot set LC_CTYPE to default locale: No such file or directory

Getting this?

# locale
locale: Cannot set LC_CTYPE to default locale: No such file or directory
locale: Cannot set LC_MESSAGES to default locale: No such file or directory
locale: Cannot set LC_ALL to default locale: No such file or directory

 

Check this:

[root@cm-r01nn02 yum.repos.d]# cat /etc/locale.conf
LANG=en_EN.UTF-8
[root@cm-r01nn02 yum.repos.d]# 

should be:

[root@cm-r01nn01 ~]# cat /etc/locale.conf
LANG="en_US.UTF-8"
[root@cm-r01nn01 ~]#

You may also have empty libraries such as:

[root@cm-r01nn02 ~]# yum reinstall *glibc*
/sbin/ldconfig: File /lib64/libXcursor.so.1.0.2 is empty, not checked.

[root@cm-r01nn02 ~]# ls -altri /lib64/libXcursor.so.1.0.2
203505067 -rwxr-xr-x. 1 root root 0 Oct 30 12:38 /lib64/libXcursor.so.1.0.2
[root@cm-r01nn02 ~]#

What it should be:

[root@cm-r01nn01 ~]# ls -altri /lib64/libXcursor.so.1.0.2
201697422 -rwxr-xr-x. 1 root root 45200 Oct 30 12:38 /lib64/libXcursor.so.1.0.2
[root@cm-r01nn01 ~]#

Run the following to confirm if any files are empty:

[root@cm-r01nn01 ~]# ldconfig
[root@cm-r01nn01 ~]#

on a bad system:

[root@cm-r01nn02 ~]# ldconfig
ldconfig: File /lib64/libdrm.so.2.4.0 is empty, not checked.
ldconfig: File /lib64/libdrm_intel.so.1 is empty, not checked.
ldconfig: File /lib64/libdrm_intel.so.1.0.0 is empty, not checked.
ldconfig: File /lib64/libdrm_nouveau.so.2 is empty, not checked.
ldconfig: File /lib64/libdrm_nouveau.so.2.0.0 is empty, not checked.
ldconfig: File /lib64/libdrm_radeon.so.1 is empty, not checked.
ldconfig: File /lib64/libdrm_radeon.so.1.0.1 is empty, not checked.
ldconfig: File /lib64/libkms.so.1 is empty, not checked.
ldconfig: File /lib64/libkms.so.1.0.0 is empty, not checked.
ldconfig: File /lib64/libdrm.so.2 is empty, not checked.
ldconfig: File /lib64/libdrm_amdgpu.so.1.0.0 is empty, not checked.
ldconfig: File /lib64/libdrm_amdgpu.so.1 is empty, not checked.
ldconfig: File /lib64/libXfixes.so.3 is empty, not checked.
ldconfig: File /lib64/libXfixes.so.3.1.0 is empty, not checked.
ldconfig: File /lib64/libglapi.so.0 is empty, not checked.
ldconfig: File /lib64/libglapi.so.0.0.0 is empty, not checked.
ldconfig: File /lib64/libXdamage.so.1 is empty, not checked.
ldconfig: File /lib64/libXdamage.so.1.1.0 is empty, not checked.
ldconfig: File /lib64/libxshmfence.so.1 is empty, not checked.
ldconfig: File /lib64/libxshmfence.so.1.0.0 is empty, not checked.
ldconfig: File /lib64/libGLdispatch.so.0 is empty, not checked.
ldconfig: File /lib64/libGLdispatch.so.0.0.0 is empty, not checked.
ldconfig: File /lib64/libwayland-server.so.0 is empty, not checked.
ldconfig: File /lib64/libwayland-server.so.0.1.0 is empty, not checked.
ldconfig: File /lib64/libgbm.so.1 is empty, not checked.
ldconfig: File /lib64/libgbm.so.1.0.0 is empty, not checked.
ldconfig: File /lib64/libXcursor.so.1 is empty, not checked.
ldconfig: File /lib64/libXcursor.so.1.0.2 is empty, not checked.
ldconfig: File /lib64/libpcsclite.so.1 is empty, not checked.
ldconfig: File /lib64/libpcsclite.so.1.0.0 is empty, not checked.
ldconfig: File /lib64/libthai.so.0 is empty, not checked.
ldconfig: File /lib64/libthai.so.0.1.6 is empty, not checked.
ldconfig: File /lib64/libgraphite2.so.3 is empty, not checked.
ldconfig: File /lib64/libgraphite2.so.3.0.1 is empty, not checked.
ldconfig: File /lib64/libharfbuzz.so.0 is empty, not checked.
ldconfig: File /lib64/libharfbuzz.so.0.10705.0 is empty, not checked.
[root@cm-r01nn02 ~]#

Query the files using rpm -qf <FILE> then reinstall the package.  Reboot the machine.

This was all due to some XFS corruption that occurred in the past.   Likewise, check if any files on the OS are zero bytes:

for KEY in $( rpm –ql $(rpm -aq) ); do [[ ! -s $KEY && -r $KEY ]] && echo $KEY; done

Reinstall them if they are.  After a FS corruption, many files were zero on our system.  Reinstalling them can help by reinstalling the package itself.  Compare the file output to another host that is working fine.  You can use this command:

for KEY in $( cat t.txt ); do [[ -s $KEY ]] && echo $KEY; done

NOTE: Copy the file list found from corrupt host to the working host and run the above.  

In the event that a file is corrupted but its file size is not zero, it may not be easy to find the said file without a direct comparison with another host.  An alternative is to try and reinstall existing packages:

[root@cm-r01nn02 ~]# yum reinstall $(rpm -aq)

Lookup the LC_TYPE :

[root@cm-r01nn02 ~]# echo $LANG
C.UTF-8
[root@cm-r01nn02 ~]# echo $LC_CTYPE

[root@cm-r01nn02 ~]#

and in a good environment:

[root@cm-r01nn01 ~]# echo $LANG
en_US.UTF-8
[root@cm-r01nn01 ~]# echo $LC_CTYPE
en_US.UTF-8
[root@cm-r01nn01 ~]#

 

Finally, we copied the /usr/lib/locale/locale-archive from a good server to resolve the problem. But this begs the question: How is /usr/lib/locale/locale-archive generated?  The strace, a good one, should stop at the locale-archive and not go any further like this:

[root@cm-r01nn02 ~]# strace locale 2>&1|grep -Ei "open|stat|exec"
execve("/bin/locale", [“locale”], [/* 21 vars */]) = 0
open("/etc/ld.so.cache", O_RDONLY|O_CLOEXEC) = 3
fstat(3, {st_mode=S_IFREG|0644, st_size=37662, …}) = 0
open("/lib64/libc.so.6", O_RDONLY|O_CLOEXEC) = 3
fstat(3, {st_mode=S_IFREG|0755, st_size=2151672, …}) = 0
mmap(NULL, 3981792, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7f371aab0000
open("/usr/lib/locale/locale-archive", O_RDONLY|O_CLOEXEC) = 3
fstat(3, {st_mode=S_IFREG|0644, st_size=106075056, …}) = 0
fstat(1, {st_mode=S_IFIFO|0600, st_size=0, …}) = 0
[root@cm-r01nn02 ~]#

 

But it did on a faulty server, suggesting this file may be generated on the target system.  So I reinstalled glibc-common once more replacing the good copy from the other server.  This time it worked despire generating a different locale-archive file:

[root@cm-r01nn02 ~]# history|grep strace
  196  strace -p 3951
  534  strace locale
  690  strace locale 2>&1|grep -Ei "open|stat"
  827  strace locale  | grep -Ei "open|stat"
  828  strace locale  2>&1 | grep -Ei "open|stat"
  832  strace locale  2>&1 | grep -Ei "open|stat"
  852  strace locale
  855  strace locale|grep -Ei "exec|open|access"
  857  strace -e locale
  858  strace -e open locale
  859  strace -e trace=open,read locale
  860  strace -ff -e trace=open locale
  863  strace -ff -o trace  locale
  979  strace locale
  980  strace locale 2>&1|grep -Ei "open|stat|exec"
 1003  strace locale
 1004  history|grep strace
[root@cm-r01nn02 ~]# strace locale 2>&1|grep -Ei "open|stat|exec"
execve("/bin/locale", [“locale”], [/* 21 vars */]) = 0
open("/etc/ld.so.cache", O_RDONLY|O_CLOEXEC) = 3
fstat(3, {st_mode=S_IFREG|0644, st_size=37662, …}) = 0
open("/lib64/libc.so.6", O_RDONLY|O_CLOEXEC) = 3
fstat(3, {st_mode=S_IFREG|0755, st_size=2151672, …}) = 0
mmap(NULL, 3981792, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7f371aab0000
open("/usr/lib/locale/locale-archive", O_RDONLY|O_CLOEXEC) = 3
fstat(3, {st_mode=S_IFREG|0644, st_size=106075056, …}) = 0
fstat(1, {st_mode=S_IFIFO|0600, st_size=0, …}) = 0
[root@cm-r01nn02 ~]#
[root@cm-r01nn02 ~]#
[root@cm-r01nn02 ~]#
[root@cm-r01nn02 ~]# rpm -qf /usr/lib/locale/locale-archive
glibc-common-2.17-260.el7_6.4.x86_64
[root@cm-r01nn02 ~]#
[root@cm-r01nn02 ~]#
[root@cm-r01nn02 ~]# sha1sum /usr/lib/locale/locale-archive /root/locale-archive
8698125a0ab14cd3ae969d3c21b867b9cb490227  /usr/lib/locale/locale-archive
8698125a0ab14cd3ae969d3c21b867b9cb490227  /root/locale-archive
[root@cm-r01nn02 ~]# yum reinstall glibc-common -y
Loaded plugins: fastestmirror
Loading mirror speeds from cached hostfile
epel/x86_64/metalink                                                                      |  16 kB  00:00:00
 * base: mirror.csclub.uwaterloo.ca
 * epel: mirror.csclub.uwaterloo.ca
 * extras: mirror.csclub.uwaterloo.ca
 * updates: mirror.csclub.uwaterloo.ca
base                                                                                      | 3.6 kB  00:00:00
cloudera-manager                                                                          | 2.9 kB  00:00:00
epel                                                                                      | 4.7 kB  00:00:00
extras                                                                                    | 3.4 kB  00:00:00
updates                                                                                   | 3.4 kB  00:00:00
vmware-tools                                                                              |  951 B  00:00:00
(1/2): epel/x86_64/updateinfo                                                             | 983 kB  00:00:00
(2/2): epel/x86_64/primary_db                                                             | 6.7 MB  00:00:01
Resolving Dependencies
–> Running transaction check
—> Package glibc-common.x86_64 0:2.17-260.el7_6.4 will be reinstalled
–> Finished Dependency Resolution

Dependencies Resolved

=================================================================================================================
 Package                     Arch                  Version                          Repository              Size
=================================================================================================================
Reinstalling:
 glibc-common                x86_64                2.17-260.el7_6.4                 updates                 12 M

Transaction Summary
=================================================================================================================
Reinstall  1 Package

Total download size: 12 M
Installed size: 115 M
Downloading packages:
Delta RPMs disabled because /usr/bin/applydeltarpm not installed.
glibc-common-2.17-260.el7_6.4.x86_64.rpm                                                  |  12 MB  00:00:02
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction
  Installing : glibc-common-2.17-260.el7_6.4.x86_64                                                          1/1
  Verifying  : glibc-common-2.17-260.el7_6.4.x86_64                                                          1/1

Installed:
  glibc-common.x86_64 0:2.17-260.el7_6.4

Complete!
[root@cm-r01nn02 ~]#
[root@cm-r01nn02 ~]#
[root@cm-r01nn02 ~]#
[root@cm-r01nn02 ~]# sha1sum /usr/lib/locale/locale-archive /root/locale-archive
4a40d739c365ddcd3756283b0d4241dfb9b9dfcd  /usr/lib/locale/locale-archive
8698125a0ab14cd3ae969d3c21b867b9cb490227  /root/locale-archive
[root@cm-r01nn02 ~]# locale
LANG=en_US.UTF-8
LC_CTYPE="en_US.UTF-8"
LC_NUMERIC="en_US.UTF-8"
LC_TIME="en_US.UTF-8"
LC_COLLATE="en_US.UTF-8"
LC_MONETARY="en_US.UTF-8"
LC_MESSAGES="en_US.UTF-8"
LC_PAPER="en_US.UTF-8"
LC_NAME="en_US.UTF-8"
LC_ADDRESS="en_US.UTF-8"
LC_TELEPHONE="en_US.UTF-8"
LC_MEASUREMENT="en_US.UTF-8"
LC_IDENTIFICATION="en_US.UTF-8"
LC_ALL=
[root@cm-r01nn02 ~]# reboot
Using username "mds.xyz\tom".
Using keyboard-interactive authentication.
Password:
Last login: Wed Apr 10 07:14:48 2019 from 192.168.0.93
tom@mds.xyz@cm-r01nn02:~] 🙂 $ locale
LANG=en_US.UTF-8
LC_CTYPE="en_US.UTF-8"
LC_NUMERIC="en_US.UTF-8"
LC_TIME="en_US.UTF-8"
LC_COLLATE="en_US.UTF-8"
LC_MONETARY="en_US.UTF-8"
LC_MESSAGES="en_US.UTF-8"
LC_PAPER="en_US.UTF-8"
LC_NAME="en_US.UTF-8"
LC_ADDRESS="en_US.UTF-8"
LC_TELEPHONE="en_US.UTF-8"
LC_MEASUREMENT="en_US.UTF-8"
LC_IDENTIFICATION="en_US.UTF-8"
LC_ALL=
tom@mds.xyz@cm-r01nn02:~] 🙂 $ sudo su –
[sudo] password for tom@mds.xyz:
Last login: Wed Apr 10 07:15:02 EDT 2019 on pts/0
[root@cm-r01nn02 ~]# locale
LANG=en_US.UTF-8
LC_CTYPE="en_US.UTF-8"
LC_NUMERIC="en_US.UTF-8"
LC_TIME="en_US.UTF-8"
LC_COLLATE="en_US.UTF-8"
LC_MONETARY="en_US.UTF-8"
LC_MESSAGES="en_US.UTF-8"
LC_PAPER="en_US.UTF-8"
LC_NAME="en_US.UTF-8"
LC_ADDRESS="en_US.UTF-8"
LC_TELEPHONE="en_US.UTF-8"
LC_MEASUREMENT="en_US.UTF-8"
LC_IDENTIFICATION="en_US.UTF-8"
LC_ALL=
[root@cm-r01nn02 ~]#

If that still doesn't work, consider these two outputs from a correctly working system and an incorrectly working system:

[root@cm-r01nn02 ~]# localectl status
   System Locale: LANG=en_US.UTF-8
       VC Keymap: us
      X11 Layout: us
[root@cm-r01nn02 ~]#

Incorrectly working one:

[root@awx01 ~]# localectl status
   System Locale: n/a

       VC Keymap: us
      X11 Layout: us
[root@awx01 ~]#

Set the system locale:

[root@awx01 ~]# localectl set-locale LANG=en_US.UTF-8

restart, if necessary, then run:

locale

checking further still we see this:

[root@awx01 locale]# strings locale-archive|grep -Ei en_us.utf8
en_US.utf8
[root@awx01 locale]# ls -altri /etc/profile
134299888 -rw-r–r–. 1 root root 1795 Nov  5  2016 /etc/profile
[root@awx01 locale]# scp cm-r01nn01:/etc/profile /etc/profile-cm-r01nn01
profile                                                                100% 1819   280.5KB/s   00:00
[root@awx01 locale]# diff /etc/profile /etc/profile-cm-r01nn01
65c65
< for i in /etc/profile.d/*.sh ; do

> for i in /etc/profile.d/*.sh /etc/profile.d/sh.local ; do
[root@awx01 locale]#

so we update the system but still the same issue.  locale is a perl executable so we check the following:

[root@awx01 etc]# perl -v
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
        LANGUAGE = (unset),
        LC_ALL = (unset),
        LANG = "C.UTF-8"
    are supported and installed on your system.
perl: warning: Falling back to the standard locale ("C").

This is perl 5, version 16, subversion 3 (v5.16.3) built for x86_64-linux-thread-multi
(with 39 registered patches, see
perl -V for more detail)

Copyright 1987-2012, Larry Wall

Perl may be copied only under the terms of either the Artistic License or the
GNU General Public License, which may be found in the Perl 5 source kit.

Complete documentation for Perl, including FAQ lists, should be found on
this system using "man
perl" or "perldoc perl".  If you have access to the
Internet, point your browser at http://www.perl.org/, the Perl Home Page.

[root@awx01 etc]#
 

vs a working system:

[root@cm-r01nn01 locale]# perl -v

This is perl 5, version 16, subversion 3 (v5.16.3) built for x86_64-linux-thread-multi
(with 39 registered patches, see
perl -V for more detail)

Copyright 1987-2012, Larry Wall

Perl may be copied only under the terms of either the Artistic License or the
GNU General Public License, which may be found in the Perl 5 source kit.

Complete documentation for Perl, including FAQ lists, should be found on
this system using "man
perl" or "perldoc perl".  If you have access to the
Internet, point your browser at http://www.perl.org/, the Perl Home Page.

[root@cm-r01nn01 locale]#

Let's look for the locale explicitly:

[root@awx01 etc]# find / -iname en_US.UTF-8
[root@awx01 etc]#

vs a working system:

[root@cm-r01nn01 locale]# find / –iname en_US.UTF-8
/
usr/share/X11/locale/en_US.UTF-8
[root@cm-r01nn01 locale]#

find what installs that file:

[root@cm-r01nn01 locale]# rpm -qf /usr/share/X11/locale/en_US.UTF-8
libX11-common-1.6.5-2.el7.noarch
[root@cm-r01nn01 locale]#

and reinstall that package or install it:

[root@awx01 etc]# yum install libX11-common.noarch

reboot and check if locale assignment worked.  If this still doesn't work, then we need to revisit our steps above since the following may be true when running grep on hidden files:

[root@awx01 ~]# cat .bash_profile |grep LANG
# export LANG="C.UTF-8"
[root@awx01 ~]# grep -ER LANG= *
[root@awx01 ~]#

To avoid the above issue, consider running greps in this manner:

[root@awx01 ~]# grep -rER “LANG=” * .[^.]*
.bash_profile:# export LANG="C.UTF-8"
[root@awx01 ~]# vi .bash_profile
[root@awx01 ~]#

 

And your issue should be solved!    🙂  

Thx,
TK

ERROR scm-web-216:com.cloudera.cmf.model.DbCommand: Command null(clusterHostInspector) has completed. finalstate:FINISHED, success:false, msg:Can only run host inspector when host is healthy.

When receiving this error, look into the worker clouderascm-agent to determine why.  In our case it was:  

[29/Mar/2019 00:11:47 +0000] 800 MainThread agent        ERROR    Error, CM server guid updated, expected f2f1e171-d20d-4425-afe9-58b567b51397, received 18757343-bd5c-4b15-a104-91fd432ebc82

Then follow this page to resolve the above.  

Thx,
TK

Row size too large. The maximum row size for the used table type, not counting BLOBs, is 65535.

If you are getting the following:

2019-03-21 01:04:17,021 FATAL main:org.hsqldb.cmdline.SqlFile: SQL Error at 'UTF-8' line 6:
"alter table SETTINGS
    add column LDAP_USER_SEARCH_BASE varchar(1024),
    add column LDAP_USER_SEARCH_FILTER varchar(1024),
    add column LDAP_GROUP_SEARCH_BASE varchar(1024),
    add column LDAP_GROUP_SEARCH_FILTER varchar(1024)"
Row size too large. The maximum row size for the used table type, not counting BLOBs, is 65535. This includes storage overhead, check the manual. You have to change some columns to TEXT or BLOBs
2019-03-21 01:04:17,021 FATAL main:org.hsqldb.cmdline.SqlFile: Rolling
backSQL transaction.
2019-03-21 01:04:17,023 ERROR main:com.cloudera.enterprise.dbutil.SqlFileRunner: Exception while
executingddl scripts.
com.mysql.jdbc.exceptions.jdbc4.MySQLSyntaxErrorException: Row size too large. The maximum row size for the used table type, not counting BLOBs, is 65535. This includes storage overhead, check the manual. You have to change some columns to TEXT or BLOBs

it's probably because you're running these SQL commands to setup the Cloudera database:

CREATE DATABASE scm DEFAULT CHARACTER SET utf8mb4 DEFAULT COLLATE utf8mb4_unicode_ci;
GRANT ALL ON scm.* TO 'scm'@'%' IDENTIFIED BY 'scm';

Or these:

CREATE DATABASE scm DEFAULT CHARACTER SET utf8mb4 DEFAULT COLLATE utf8mb4_general_ci;
GRANT ALL ON 
scm.* TO 'scm'@'%' IDENTIFIED BY 'scm';

Instead of something like this:

create database scm DEFAULT CHARACTER SET utf8;
grant all privileges on 
scm.* to 'scm'@'%' identified by 'scm';

Appears unicode or general, can throw an error with the clouderascm-server on install.

Thx,
TK

Free IPA Replication Verification Tool

There is a tool available that does a verification of the replication of each FreeIPA host:

yum install git -y; git clone https://github.com/peterpakos/checkipaconsistency.git

# ./cipa -d mws.mds.xyz -W "SECRET"
+——————–+————+————-+——-+
| FreeIPA servers:   | idmipa03   | idmipa04    | STATE |
+——————–+————+————-+——-+
| Active Users       | 1          | 1           | OK    |
| Stage Users        | 0          | 0           | OK    |
| Preserved Users    | 0          | 0           | OK    |
| Hosts              | 2          | 2           | OK    |
| Services           | 11         | 11          | OK    |
| User Groups        | 10         | 10          | OK    |
| Host Groups        | 1          | 1           | OK    |
| Netgroups          | 0          | 0           | OK    |
| HBAC Rules         | 1          | 1           | OK    |
| SUDO Rules         | 0          | 0           | OK    |
| DNS Zones          | 3          | 3           | OK    |
| Certificates       | 17         | 17          | OK    |
| LDAP Conflicts     | 0          | 0           | OK    |
| Ghost Replicas     | 0          | 0           | OK    |
| Anonymous BIND     | ON         | ON          | OK    |
| Microsoft ADTrust  | True       | False       | FAIL  |
| Replication Status | idmipa04 0 | idmipa03 18 | OK    |
+——————–+————+————-+——-+
#

Cheers,
TK

Zabbix: [Z3001] connection to database ‘zabbix’ failed: [2003] Can’t connect to MySQL server on ‘mysql-01.abc.xyz.123’ (13)

Zabbix error:

[Z3001] connection to database ‘zabbix’ failed: [2003] Can't connect to MySQL server on 'mysql-01.abc.xyz.123' (13)

related to:

audit.log:type=AVC msg=audit(1549949080.977:11328): avc:  denied  { name_connect } for  pid=9115 comm="zabbix_server" dest=3306 scontext=system_u:system_r:zabbix_t:s0 tcontext=system_u:object_r:mysqld_port_t:s0 tclass=tcp_socket

is solved by:

# grep AVC /var/log/audit/audit.log | audit2allow -M systemd-allow; semodule -i systemd-allow.pp

Cheers,
TK

Zabbix: cannot start preprocessing service: Cannot bind socket to “/var/run/zabbix/zabbix_server_preprocessing.sock”: [98] Address already in use.

Zabbix error:

 10272:20190212:003104.073 cannot start preprocessing service: Cannot bind socket to "/var/run/zabbix/zabbix_server_preprocessing.sock": [98] Address already in use.
 10239:20190212:003104.078 One child process died (PID:10272,exitcode/signal:1). Exiting …

related to:

# cat ../audit/audit.log|grep -Ei denied|tail
type=AVC msg=audit(1549949530.062:12551): avc:  denied  { unlink } for  pid=10521 comm="zabbix_server" name="zabbix_server_preprocessing.sock" dev="tmpfs" ino=3998803 scontext=system_u:system_r:zabbix_t:s0 tcontext=system_u:object_r:zabbix_var_run_t:s0 tclass=sock_file

is solved by:

# grep AVC /var/log/audit/audit.log* | audit2allow -M systemd-allow; semodule -i systemd-allow.pp

Cheers,
TK


     
  Copyright © 2003 - 2013 Tom Kacperski (microdevsys.com). All rights reserved.

Creative Commons License
This work is licensed under a Creative Commons Attribution 3.0 Unported License