Header Shadow Image


Bash: PowerLine Configuration under a User Account

Let's setup Powerline to make our prompts look like this in CentOS!

https://i2.wp.com/www.microdevsys.com/WordPressImages/powerline-configuration-introduction.JPG?ssl=1

How to do this?  Follow the following set of steps to configure this within a non privilidged user account without having to modify many target server root owned files or install any packages in the target UNIX systems.

  • Install powerline using pip3 Python 3 installer:

    [tom@mds.xyz@awx01:~] :)$ pip3 install –user powerline-status
    Collecting powerline-status
      Using cached https://files.pythonhosted.org/packages/9c/30/8bd3c62642778af9ad813a526c6ff7dd20ad6fab94ca389265/powerline-status-2.7.tar.gz
    Installing collected packages: powerline-status
      Running setup.py install for powerline-status … done
    Successfully installed powerline-status-2.7
    [tom@mds.xyz@awx01:~] :)$

     

  • Find the installed powerline direcctories. This is needed to configure .bash_profile 

    [tom@mds.xyz@awx01:~] :)$ pip3 show powerline-status
    Name: powerline-status
    Version: 2.7
    Summary: The ultimate statusline/prompt utility.
    Home-page: https://github.com/powerline/powerline
    Author: Kim Silkebaekken
    Author-email: kim.silkebaekken+vim@gmail.com
    License: MIT
    Location: /n/mds.xyz/tom/.local/lib/python3.6/site-packages
    Requires:
    [tom@mds.xyz@awx01:~] :)$

    ?

  • Next, add the following lines to your .bash_profile.  It's ok to leave the previous .bash_profile settings in place.  They'll be overwritten.

    [tom@mds.xyz@awx01:~] :)$ cat .bash_profile |tail -n5
    export PATH=$PATH:$HOME/Library/Python/2.7/bin
    powerline-daemon -q
    POWERLINE_BASH_CONTINUATION=1
    POWERLINE_BASH_SELECT=1
    . ./.local/lib/python3.6/site-packages/powerline/bindings/bash/powerline.sh
    [tom@mds.xyz@awx01:~] :)$

     

  • If running on an X Windows system, such as Gnome or KDE, install a set of fonts in the home folder of the user:

    [tom@mds.xyz@awx01:~] :)$ wget https://github.com/powerline/fonts/archive/master.zip
    [tom@mds.xyz@awx01:~] :($ unzip master.zip

    [tom@mds.xyz@awx01:~/fonts] :)$ ./install.sh
    Copying fonts…
    Powerline fonts installed to /n/mds.xyz/tom/.local/share/fonts
    [tom@mds.xyz@awx01:~/fonts] :)$

     

  • This next part occurs in Windows 10.  Grab the set of fonts below and install them in Windows 10.  https://github.com/powerline/fonts :  A few examples:

    Adding Croscore fonts for Powerline (Chrome OS core fonts)
    https://github.com/powerline/fonts/blob/master/Arimo/

    DejaVu Sans Mono for Powerline 
    https://github.com/powerline/fonts/tree/master/DejaVuSansMono

    Droid Sans Mono for Powerline
    https://github.com/powerline/fonts/tree/master/DroidSansMono
     

  • Select the installed fonts in PuTTy:

    Within PuTTy (Putty Configuration) -> Window -> Apearance -> Font settings -> Change

    Select above-installed fonts.

  • Login to a host.

  • Enjoy your new command line!

BONUS

Below is a one line ansible command to update the .bash_profile as root:

ansible 'awx01*' -i /ansible/infra -m shell -a "yum install python3 -y" -become -u root

ansible 'awx01*' -i /ansible/infra -m shell -a "pip3 install –user powerline-status" -become -u root

ansible 'awx01*' -i /ansible/infra -m shell -a "if ! grep -q powerline ~/.bash_profile; then echo -ne \"export PATH=\\\$PATH:$HOME/.local/bin/\\npowerline-daemon -q\\nPOWERLINE_BASH_CONTINUATION=1\\nPOWERLINE_BASH_SELECT=1\\n. /root/.local/lib/python3.6/site-packages/powerline/bindings/bash/powerline.sh\\n\" >> ~/.bash_profile; fi" -become -u root

Modify the host parameter to just '*' once you feel comfortable with the commands. This is how it looks when done:

https://www.microdevsys.com/WordPressImages/powerline-configuration-rootJPG

Have Fun!
TK

Cloudera: No Java JDK is detected on the host.

Getting this?  

"No Java JDK is detected on the host."  

One reason for this is a missing symlink /usr/java/latest :  

[root@cm-awn01 java]# ls -l /usr/java
total 0
drwxr-xr-x 7 root root 245 May 11 00:39 jdk1.8.0_181-cloudera  
lrwxrwxrwx 1 root root  21 May 27 13:27 latest -> jdk1.8.0_181-cloudera  
[root@cm-awn01 java]#  

GL,
SC

 

Cloudera and Azure: WrongHost: Peer certificate subjectAltName does not match host, expected , got DNS:host01.dom.com, DNS:host02.dom.com, DNS:host03.dom.com

So you're getting this while trying to connect Cloud Hosts to your local Cloudera Infrastructure?

WrongHost: Peer certificate subjectAltName does not match host, expected dhcp-100-0-0-100.remote.user.isp.com, got DNS:srv-c01.cdh.local.hst, DNS:cm-r01nn01.cdh.local.hst, DNS:cm-r01nn02.cdh.local.hst

Read the rest of this entry »

ERROR    (10 skipped) Error sending messages to firehose (retry): mgmt-HOSTMONITOR

Getting this?

[24/May/2020 23:08:13 +0000] 5385 MonitorDaemon-Reporter throttling_logger ERROR    (10 skipped) Error sending messages to firehose (retry): mgmt-HOSTMONITOR-a6c8a202b717eae93da5e0a53f184c3a
Traceback (most recent call last):
  File "/opt/cloudera/cm-agent/lib/python2.7/site-packages/cmf/monitor/firehose.py", line 125, in _send
    self._requestor.request('sendAgentMessages', dict(messages=UNICODE_SANITIZER(messages)))
  File "/opt/cloudera/cm-agent/lib/python2.7/site-packages/avro/ipc.py", line 141, in request
    return self.issue_request(call_request, message_name, request_datum)
  File "/opt/cloudera/cm-agent/lib/python2.7/site-packages/avro/ipc.py", line 254, in issue_request
    call_response = self.transceiver.transceive(call_request)
  File "/opt/cloudera/cm-agent/lib/python2.7/site-packages/avro/ipc.py", line 483, in transceive
    result = self.read_framed_message()
  File "/opt/cloudera/cm-agent/lib/python2.7/site-packages/avro/ipc.py", line 487, in read_framed_message
    response = self.conn.getresponse()
  File "/usr/lib64/python2.7/httplib.py", line 1113, in getresponse
    response.begin()
  File "/usr/lib64/python2.7/httplib.py", line 444, in begin
    version, status, reason = self._read_status()
  File "/usr/lib64/python2.7/httplib.py", line 408, in _read_status
    raise BadStatusLine(line)
BadStatusLine: ''

modify the line slightly to see exactly what host or port it's trying:

    try:
      if self._requestor is None:
        self._transceiver = avro.ipc.HTTPTransceiver(self._address,
                                                     self._port)
        self._requestor = avro.ipc.Requestor(FIREHOSE_MESSAGE_PROTOCOL,
                                             self._transceiver)
      initial_requestor_bytes = self._requestor.get_requestor_bytes_sent()
      self._requestor.request('sendAgentMessages', dict(messages=UNICODE_SANITIZER(messages)))
      self._last_message_transmit_duration_gauge.set_value(
        (time.time() – start) * 1000)
      self._message_transmit_succeeded_counter.increment()
      self._requestor_bytes_sent.increment(
        self._requestor.get_requestor_bytes_sent() – initial_requestor_bytes)
      return True
    except BadStatusLine, ex:
      # We've lost our connection. In practice this usually means the server has
      # closed a connection that we expect to be open because of HTTP keep-alive.
      # We will do a single silent retry. If the problem persistest there, we'll
      # log.
      self._reset()
      if retryOnBadStatusLine:
        return self._send(messages, retryOnBadStatusLine=False)
      self._message_transmit_failed_counter.increment()
      # THROTTLED_LOG.exception("Error sending messages to firehose (retry): " +
      #                        self.name)

      THROTTLED_LOG.exception("Error sending messages to firehose (retry): %s .  Address: %s .  Port: %s" % ( self.name, self._address, self._port ))
      return False
    except Exception:
      THROTTLED_LOG.exception("Error sending messages to firehose: " + self.name)
      self._reset()
      self._message_transmit_failed_counter.increment()
      return False

Now when you start things up, you'll get some more meaningfull messages:

[24/May/2020 23:26:07 +0000] 6934 MonitorDaemon-Reporter firehoses    INFO     Creating a connection to the HOSTMONITOR.
[24/May/2020 23:26:08 +0000] 6934 MonitorDaemon-Reporter throttling_logger ERROR    Error sending messages to firehose (retry): mgmt-HOSTMONITOR-a6c8a202b717eae93da5e0a53f184c3a .  Address: cm-r01en02.mws.mds.xyz .  Port: 9995
Traceback (most recent call last):
  File "/opt/cloudera/cm-agent/lib/python2.7/site-packages/cmf/monitor/firehose.py", line 125, in _send
    self._requestor.request('sendAgentMessages', dict(messages=UNICODE_SANITIZER(messages)))
  File "/opt/cloudera/cm-agent/lib/python2.7/site-packages/avro/ipc.py", line 141, in request
    return self.issue_request(call_request, message_name, request_datum)
  File "/opt/cloudera/cm-agent/lib/python2.7/site-packages/avro/ipc.py", line 254, in issue_request
    call_response = self.transceiver.transceive(call_request)
  File "/opt/cloudera/cm-agent/lib/python2.7/site-packages/avro/ipc.py", line 483, in transceive
    result = self.read_framed_message()
  File "/opt/cloudera/cm-agent/lib/python2.7/site-packages/avro/ipc.py", line 487, in read_framed_message
    response = self.conn.getresponse()
  File "/usr/lib64/python2.7/httplib.py", line 1113, in getresponse
    response.begin()
  File "/usr/lib64/python2.7/httplib.py", line 444, in begin
    version, status, reason = self._read_status()
  File "/usr/lib64/python2.7/httplib.py", line 408, in _read_status
    raise BadStatusLine(line)
BadStatusLine: ''
^C
[root@cm-awn01 pki]# nc -vz cm-r01en02.mws.mds.xyz 9995
Ncat: Version 7.50 ( https://nmap.org/ncat )
Ncat: Connected to 108.168.115.113:9995.
Ncat: 0 bytes sent, 0 bytes received in 0.05 seconds.
[root@cm-awn01 pki]#

Notice the text in blue above.  Keeping it in mind, consider this Haproxy configuration:

listen cm9995
        log                             127.0.0.1:514   local0          debug
        bind                            srv-c01:9995
        mode tcp
        option tcplog
        server cm-r01en01.mws.mds.xyz cm-r01en01.mws.mds.xyz check
        server cm-r01en02.mws.mds.xyz cm-r01en02.mws.mds.xyz check

Notice that we have TCP in the HAproxy but perhaps CMA expects HTTP?  Try setting it to HTTP:

 

ERR NSMMReplicationPlugin CSN not found, we aren’t as up to date, or we purged

Getting below errr?

May 24 13:49:11 idmipa03 ns-slapd: [24/May/2020:13:49:11.182396698 -0400] – ERR – NSMMReplicationPlugin – changelog program – repl_plugin_name_cl – agmt="cn=meToidmipa04.mws.mds.xyz" (idmipa04:389): CSN 5dd194af000000040000 not found, we aren't as up to date, or we purged
May 24 13:49:11 idmipa03 ns-slapd: [24/May/2020:13:49:11.183726430 -0400] – ERR – NSMMReplicationPlugin – send_updates – agmt="cn=meToidmipa04.mws.mds.xyz" (idmipa04:389): Data required to update replica has been purged from the changelog. If the error persists the replica must be reinitialized.

or the following error?

[root@idmipa04 ~]# ipa-replica-manage force-sync –from idmipa03.mws.mds.xyz -vvv
ipa: INFO: Setting agreement cn=meToidmipa04.mws.mds.xyz,cn=replica,cn=dc\=mws\,dc\=mds\,dc\=xyz,cn=mapping tree,cn=config schedule to 2358-2359 0 to force synch
ipa: INFO: Deleting schedule 2358-2359 0 from agreement cn=meToidmipa04.mws.mds.xyz,cn=replica,cn=dc\=mws\,dc\=mds\,dc\=xyz,cn=mapping tree,cn=config
ipa: INFO: Replication Update in progress: FALSE: status: Error (18) Replication error acquiring replica: Incremental update transient warning.  Backing off, will retry update later. (transient warning): start: 0: end: 0
[root@idmipa04 ~]#

Read the rest of this entry »

kernel: ns-slapd: segfault at ip sp error 4 in libc-2.17.so

Getting this?

kernel: ns-slapd: segfault at <ADDR> ip <ALPHA> sp <ALPHA> error 4 in libc-2.17.so

Check free memory (/var/log/dirsrv/slapd-MWS-MDS-XYZ/errors):

[root@idmipa04 slapd-MWS-MDS-XYZ]# cat errors|tail -n 30
[23/May/2020:16:33:18.519974074 -0400] – WARN – NSACLPlugin – acl_parse – The ACL target cn=casigningcert cert-pki-ca,cn=ca_renewal,cn=ipa,cn=etc,dc=mws,dc=mds,dc=xyz does not exist
[23/May/2020:16:33:18.522332851 -0400] – WARN – NSACLPlugin – acl_parse – The ACL target cn=casigningcert cert-pki-ca,cn=ca_renewal,cn=ipa,cn=etc,dc=mws,dc=mds,dc=xyz does not exist
[23/May/2020:16:33:18.759212393 -0400] – WARN – NSACLPlugin – acl_parse – The ACL target cn=automember rebuild membership,cn=tasks,cn=config does not exist
[23/May/2020:16:33:18.773571691 -0400] – ERR – cos-plugin – cos_dn_defs_cb – Skipping CoS Definition cn=Password Policy,cn=accounts,dc=mws,dc=mds,dc=xyz–no CoS Templates found, which should be added before the CoS Definition.
[23/May/2020:16:33:18.820082920 -0400] – NOTICE – NSMMReplicationPlugin – changelog program – _cl5ConstructRUV – Rebuilding the replication changelog RUV, this may take several minutes…
[23/May/2020:16:39:06.851785150 -0400] – ERR – memory allocator – malloc of 2152941454 bytes failed; OS error 12 (Cannot allocate memory)
The server has probably allocated all available virtual memory. To solve
this problem, make more virtual memory available to your server, or reduce
one or more of the following server configuration settings:
  nsslapd-cachesize        (Database Settings – Maximum entries in cache)
  nsslapd-cachememsize     (Database Settings – Memory available for cache)
  nsslapd-dbcachesize      (LDBM Plug-in Settings – Maximum cache size)
  nsslapd-import-cachesize (LDBM Plug-in Settings – Import cache size).
Can't recover; calling exit(1).

Regards,
TK

DEBUG The ipa-server-upgrade command failed, exception: ScriptError: CA did not start in 300.0s

Getting this?

/var/log/ipaupgrade.log
2020-05-23T23:32:58Z ERROR IPA server upgrade failed: Inspect /var/log/ipaupgrade.log and run command ipa-server-upgrade manually.
2020-05-23T23:32:58Z DEBUG   File "/usr/lib/python2.7/site-packages/ipapython/admintool.py", line 178, in execute
    return_value = self.run()
  File "/usr/lib/python2.7/site-packages/ipaserver/install/ipa_server_upgrade.py", line 56, in run
    raise admintool.ScriptError(str(e))

?2020-05-23T23:16:22Z DEBUG The ipa-server-upgrade command failed, exception: ScriptError: CA did not start in 300.0s
2020-05-23T23:16:22Z ERROR CA did not start in 300.0s
2020-05-23T23:16:22Z ERROR The ipa-server-upgrade command failed. See /var/log/ipaupgrade.log for more information

/var/log/pki/pki-tomcat/ca/debug
Could not connect to LDAP server host idmipa04.mws.mds.xyz port 636 Error netscape.ldap.LDAPException: Unable to create socket: java.net.ConnectException: Connection refused (Connection refused) (-1)

It's likely because you have the following set:

[root@idmipa04 ca]# grep -Ei "nsslapd-port|nsslapd-security" /etc/dirsrv/slapd-MWS-MDS-XYZ/dse.ldif
nsslapd-port: 0
nsslapd-security: off
[root@idmipa04 ca]#

These need to be set to:

nsslapd-port: 389
nsslapd-security: on

But this did not work.  Checking certs expiration all shows dates in the future:

[root@idmipa04 ~]# getcert list|grep expires
        expires: 2021-02-05 07:37:13 UTC
        expires: 2021-02-05 07:37:42 UTC
        expires: 2021-01-25 03:22:30 UTC
        expires: 2021-01-25 03:21:37 UTC
        expires: 2021-01-25 03:21:36 UTC
        expires: 2021-01-25 03:21:37 UTC
        expires: 2039-02-05 03:21:36 UTC
        expires: 2021-01-25 07:40:56 UTC
        expires: 2021-02-05 07:42:11 UTC
[root@idmipa04 ~]#

Lastly, check for port 636 and 389 through netstat:

[root@idmipa04 pki-tomcat]# netstat -pnltu
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name
tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN      1069/sshd
tcp        0      0 0.0.0.0:88              0.0.0.0:*               LISTEN      1089/krb5kdc
tcp        0      0 127.0.0.1:25            0.0.0.0:*               LISTEN      1537/master
tcp6       0      0 :::22                   :::*                    LISTEN      1069/sshd
tcp6       0      0 :::88                   :::*                    LISTEN      1089/krb5kdc
tcp6       0      0 ::1:25                  :::*                    LISTEN      1537/master
tcp6       0      0 :::8443                 :::*                    LISTEN      16371/java
tcp6       0      0 :::443                  :::*                    LISTEN      15941/httpd
tcp6       0      0 127.0.0.1:8005          :::*                    LISTEN      16371/java
tcp6       0      0 127.0.0.1:8009          :::*                    LISTEN      16371/java
tcp6       0      0 :::8080                 :::*                    LISTEN      16371/java
tcp6       0      0 :::80                   :::*                    LISTEN      15941/httpd
udp        0      0 0.0.0.0:88              0.0.0.0:*                           1089/krb5kdc
udp6       0      0 :::88                   :::*                                1089/krb5kdc

If missing, start the directory server:

[root@idmipa04 pki-tomcat]# systemctl start dirsrv@MWS-MDS-XYZ.service

Check for the IP once started:

[root@idmipa04 pki-tomcat]# systemctl status dirsrv@MWS-MDS-XYZ.service
? dirsrv@MWS-MDS-XYZ.service – 389 Directory Server MWS-MDS-XYZ.
   Loaded: loaded (/usr/lib/systemd/system/dirsrv@.service; enabled; vendor preset: disabled)
   Active: active (running) since Sun 2020-05-24 01:44:55 EDT; 10s ago
  Process: 18618 ExecStartPre=/usr/sbin/ds_systemd_ask_password_acl /etc/dirsrv/slapd-%i/dse.ldif (code=exited, status=0/SUCCESS)
 Main PID: 18625 (ns-slapd)
   Status: "slapd started: Ready to process requests"
   CGroup: /system.slice/system-dirsrv.slice/dirsrv@MWS-MDS-XYZ.service
           ??18625 /usr/sbin/ns-slapd -D /etc/dirsrv/slapd-MWS-MDS-XYZ -i /var/run/dirsrv/slapd-MWS-…

May 24 01:44:55 idmipa04.mws.mds.xyz ns-slapd[18625]: GSSAPI client step 1
May 24 01:44:56 idmipa04.mws.mds.xyz ns-slapd[18625]: GSSAPI client step 1
May 24 01:44:56 idmipa04.mws.mds.xyz ns-slapd[18625]: GSSAPI client step 1
May 24 01:44:56 idmipa04.mws.mds.xyz ns-slapd[18625]: GSSAPI client step 2
May 24 01:44:57 idmipa04.mws.mds.xyz ns-slapd[18625]: [24/May/2020:01:44:57.329920836 -0400] – ERR…d.
May 24 01:44:57 idmipa04.mws.mds.xyz ns-slapd[18625]: [24/May/2020:01:44:57.331112434 -0400] – ERR…d.
May 24 01:45:00 idmipa04.mws.mds.xyz ns-slapd[18625]: [24/May/2020:01:45:00.339593970 -0400] – ERR…d.
May 24 01:45:00 idmipa04.mws.mds.xyz ns-slapd[18625]: [24/May/2020:01:45:00.340490104 -0400] – ERR…d.
May 24 01:45:03 idmipa04.mws.mds.xyz ns-slapd[18625]: [24/May/2020:01:45:03.348216609 -0400] – ERR…d.
May 24 01:45:03 idmipa04.mws.mds.xyz ns-slapd[18625]: [24/May/2020:01:45:03.354849567 -0400] – ERR…d.
Hint: Some lines were ellipsized, use -l to show in full.
[root@idmipa04 pki-tomcat]#

Check the ports are listening:

[root@idmipa04 pki-tomcat]# netstat -pnltu|grep 18625
tcp6       0      0 :::636                  :::*                    LISTEN      18625/ns-slapd
tcp6       0      0 :::389                  :::*                    LISTEN      18625/ns-slapd

Check the error logs for the service:

[root@idmipa04 pki-tomcat]# systemctl status dirsrv@MWS-MDS-XYZ.service -l
? dirsrv@MWS-MDS-XYZ.service – 389 Directory Server MWS-MDS-XYZ.
   Loaded: loaded (/usr/lib/systemd/system/dirsrv@.service; enabled; vendor preset: disabled)
   Active: active (running) since Sun 2020-05-24 01:44:55 EDT; 28s ago
  Process: 18618 ExecStartPre=/usr/sbin/ds_systemd_ask_password_acl /etc/dirsrv/slapd-%i/dse.ldif (code=exited, status=0/SUCCESS)
 Main PID: 18625 (ns-slapd)
   Status: "slapd started: Ready to process requests"
   CGroup: /system.slice/system-dirsrv.slice/dirsrv@MWS-MDS-XYZ.service
           ??18625 /usr/sbin/ns-slapd -D /etc/dirsrv/slapd-MWS-MDS-XYZ -i /var/run/dirsrv/slapd-MWS-MDS-XYZ.pid

May 24 01:45:09 idmipa04.mws.mds.xyz ns-slapd[18625]: [24/May/2020:01:45:09.372741696 -0400] – ERR – agmt="cn=caToidmipa03.mws.mds.xyz" (idmipa03:389) – clcache_load_buffer – Can't locate CSN 5c7bc2730000ffffffff in the changelog (DB rc=-30988). If replication stops, the consumer may need to be reinitialized.
May 24 01:45:09 idmipa04.mws.mds.xyz ns-slapd[18625]: [24/May/2020:01:45:09.373677051 -0400] – ERR – NSMMReplicationPlugin – send_updates – agmt="cn=caToidmipa03.mws.mds.xyz" (idmipa03:389): Missing data encountered. If the error persists the replica must be reinitialized.
[root@idmipa04 pki-tomcat]#

If you see the above, reinitialize the system:

[root@idmipa04 pki-tomcat]# ipa-csreplica-manage re-initialize –from idmipa03.mws.mds.xyz
Directory Manager password:

Update in progress, 3 seconds elapsed
Update succeeded

[root@idmipa04 pki-tomcat]# systemctl status dirsrv@MWS-MDS-XYZ.service -l
? dirsrv@MWS-MDS-XYZ.service – 389 Directory Server MWS-MDS-XYZ.
   Loaded: loaded (/usr/lib/systemd/system/dirsrv@.service; enabled; vendor preset: disabled)
   Active: active (running) since Sun 2020-05-24 01:44:55 EDT; 4min 29s ago
  Process: 18618 ExecStartPre=/usr/sbin/ds_systemd_ask_password_acl /etc/dirsrv/slapd-%i/dse.ldif (code=exited, status=0/SUCCESS)
 Main PID: 18625 (ns-slapd)
   Status: "slapd started: Ready to process requests"
   CGroup: /system.slice/system-dirsrv.slice/dirsrv@MWS-MDS-XYZ.service
           ??18625 /usr/sbin/ns-slapd -D /etc/dirsrv/slapd-MWS-MDS-XYZ -i /var/run/dirsrv/slapd-MWS-MDS-XYZ.pid

May 24 01:49:15 idmipa04.mws.mds.xyz ns-slapd[18625]: [24/May/2020:01:49:15.687759236 -0400] – WARN – NSMMReplicationPlugin – replica_reload_ruv – New data for replica o=ipaca does not match the data in the changelog.
May 24 01:49:15 idmipa04.mws.mds.xyz ns-slapd[18625]: Recreating the changelog file. This could affect replication with replica's  consumers in which case the consumers should be reinitialized.
May 24 01:49:15 idmipa04.mws.mds.xyz ns-slapd[18625]: [24/May/2020:01:49:15.721328728 -0400] – ERR – cos-plugin – cos_dn_defs_cb – Skipping CoS Definition cn=Password Policy,cn=accounts,dc=mws,dc=mds,dc=xyz–no CoS Templates found, which should be added before the CoS Definition.
May 24 01:49:15 idmipa04.mws.mds.xyz ns-slapd[18625]: [24/May/2020:01:49:15.727578549 -0400] – NOTICE – NSMMReplicationPlugin – changelog program – _cl5ConstructRUV – Rebuilding the replication changelog RUV, this may take several minutes…
May 24 01:49:15 idmipa04.mws.mds.xyz ns-slapd[18625]: [24/May/2020:01:49:15.728113208 -0400] – NOTICE – NSMMReplicationPlugin – changelog program – _cl5ConstructRUV – Rebuilding replication changelog RUV complete.  Result 0 (Success)
May 24 01:49:15 idmipa04.mws.mds.xyz ns-slapd[18625]: [24/May/2020:01:49:15.728579987 -0400] – NOTICE – NSMMReplicationPlugin – changelog program – _cl5ConstructRUV – Rebuilding the replication changelog RUV, this may take several minutes…
May 24 01:49:15 idmipa04.mws.mds.xyz ns-slapd[18625]: [24/May/2020:01:49:15.728985312 -0400] – NOTICE – NSMMReplicationPlugin – changelog program – _cl5ConstructRUV – Rebuilding replication changelog RUV complete.  Result 0 (Success)
May 24 01:49:15 idmipa04.mws.mds.xyz ns-slapd[18625]: GSSAPI server step 1
May 24 01:49:15 idmipa04.mws.mds.xyz ns-slapd[18625]: GSSAPI server step 2
May 24 01:49:15 idmipa04.mws.mds.xyz ns-slapd[18625]: GSSAPI server step 3

[root@idmipa04 pki-tomcat]#

Your FreeIPA server should now be back up?  Let's try that and see what happens.

/var/log/ipaupgrade.log
2020-05-24T06:00:06Z DEBUG request POST http://idmipa04.mws.mds.xyz:8080/ca/admin/ca/getStatus
2020-05-24T06:00:06Z DEBUG request body ''
2020-05-24T06:00:06Z DEBUG response status 200
2020-05-24T06:00:06Z DEBUG response headers Server: Apache-Coyote/1.1
Content-Type: application/xml
Content-Length: 168
Date: Sun, 24 May 2020 06:00:06 GMT

2020-05-24T06:00:06Z DEBUG response body '<?xml version="1.0" encoding="UTF-8" standalone="no"?><XMLResponse><State>1</State><Type>CA</Type><Status>running</Status><Version>10.5.17-6.el7</Version></XMLResponse>'
2020-05-24T06:00:06Z INFO The IPA services were upgraded
2020-05-24T06:00:06Z DEBUG Loading StateFile from '/var/lib/ipa/sysupgrade/sysupgrade.state'
2020-05-24T06:00:06Z DEBUG Saving StateFile to '/var/lib/ipa/sysupgrade/sysupgrade.state'
2020-05-24T06:00:06Z DEBUG Loading StateFile from '/var/lib/ipa/sysupgrade/sysupgrade.state'
2020-05-24T06:00:06Z DEBUG Saving StateFile to '/var/lib/ipa/sysupgrade/sysupgrade.state'
2020-05-24T06:00:06Z INFO The ipa-server-upgrade command was successful

Confirming the command now succeeded as expected:

[root@idmipa04 pki-tomcat]# ipactl start
IPA version error: data needs to be upgraded (expected version '4.6.6-11.el7.centos', current version '4.6.4-10.el7.centos.2')
Automatically running upgrade, for details see /var/log/ipaupgrade.log
Be patient, this may take a few minutes.
Existing service file detected!
Assuming stale, cleaning and proceeding
Starting Directory Service
Starting krb5kdc Service
Starting kadmin Service
Starting named Service
Starting httpd Service
Starting ipa-custodia Service
Starting ntpd Service
Starting pki-tomcatd Service
Starting ipa-otpd Service
Starting ipa-dnskeysyncd Service
ipa: INFO: The ipactl command was successful
[root@idmipa04 pki-tomcat]#

RELATED ERRORS:

The following errors were seen alongside the above-mentioned entries.

/var/log/ipaupgrade.log
2020-05-02T12:50:40Z DEBUG The ipa-server-upgrade command failed, exception: CalledProcessError: Command '/bin/systemctl start dirsrv@MWS-MDS-XYZ.service' returned non-zero exit status 1

2020-05-23T21:07:50Z DEBUG The CA status is: check interrupted due to error: Retrieving CA status failed with status 500

/var/log/pki/pki-tomcat/localhost.2020-05-24.log
SEVERE: Exception Processing /ca/admin/ca/getStatus
javax.ws.rs.ServiceUnavailableException: Subsystem unavailable

SEVERE: Servlet.service() for servlet [Resteasy] in context with path [/ca] threw exception
org.jboss.resteasy.spi.UnhandledException: org.jboss.resteasy.core.NoMessageBodyWriterFoundFailure: Could not find MessageBodyWriter for response object of type: com.netscape.certsrv.base.PKIException$Data of media type: application/x-www-form-urlencoded

/var/log/pki/pki-tomcat/ca/debug
Could not connect to LDAP server host idmipa04.mws.mds.xyz port 636 Error netscape.ldap.LDAPException: Unable to create socket: java.net.ConnectException: Connection refused (Connection refused) (-1)

/var/log/dirsrv/slapd-MWS-MDS-XYZ/errors
[24/May/2020:01:02:41.912364232 -0400] – ERR – NSMMReplicationPlugin – send_updates – agmt="cn=caToidmipa03.mws.mds.xyz" (idmipa03:389): Missing data encountered. If the error persists the replica must be reinitialized.

[23/May/2020:00:40:23.025920441 -0400] – ERR – set_krb5_creds – Could not get initial credentials for principal [ldap/idmipa04.mws.mds.xyz@MWS.MDS.XYZ] in keytab [FILE:/etc/dirsrv/ds.keytab]: -1765328228 (Cannot contact any KDC for requested realm)

Cheers,
TK
 

Cloudera: ERROR Heartbeating to :7182 failed. SSLError: unexpected eof

Getting this?

[18/May/2020 18:04:46 +0000] 2849 MainThread agent        ERROR    Heartbeating to srv-c01.mws.mds.xyz:7182 failed.
Traceback (most recent call last):
  File "/opt/cloudera/cm-agent/lib/python2.7/site-packages/cmf/agent.py", line 1387, in _send_heartbeat
    self.cfg.max_cert_depth)
  File "/opt/cloudera/cm-agent/lib/python2.7/site-packages/cmf/https.py", line 139, in __init__
    self.conn.connect()
  File "/opt/cloudera/cm-agent/lib/python2.7/site-packages/M2Crypto/httpslib.py", line 69, in connect
    sock.connect((self.host, self.port))
  File "/opt/cloudera/cm-agent/lib/python2.7/site-packages/M2Crypto/SSL/Connection.py", line 309, in connect
    ret = self.connect_ssl()
  File "/opt/cloudera/cm-agent/lib/python2.7/site-packages/M2Crypto/SSL/Connection.py", line 295, in connect_ssl
    return m2.ssl_connect(self.ssl, self._timeout)
SSLError: unexpected eof

In this case it was because the Cloudera SCM Server was offline.  Started it back up to resolve the above.  Having said this, if your server is up but you still get this, then Cloudera SCM Server hasn't sent an SSL / TLS Certificate.  

Cheers,
BK

Cloudera: WrongHost: Peer certificate subjectAltName does not match host, expected HOST01, got HOST02

Getting the following while connecting TLS enabled Azure, AWS or GCP cloud hosts to Cloudera Manager?

[18/May/2020 13:12:09 +0000] 2413 Thread-13 downloader   INFO     Fetching torrent: https://cm-r01nn01.mws.mds.xyz:7183/cmf/parcel/download/CDH-6.3.0-1.cdh6.3.0.p0.1279813-el7.parcel.torrent
[18/May/2020 13:12:09 +0000] 2413 Thread-13 https        ERROR    Failed to retrieve/store URL: https://cm-r01nn01.mws.mds.xyz:7183/cmf/parcel/download/CDH-6.3.0-1.cdh6.3.0.p0.1279813-el7.parcel.torrent -> /opt/cloudera/parcel-cache/CDH-6.3.0-1.cdh6.3.0.p0.1279813-el7.parcel.torrent Peer certificate subjectAltName does not match host, expected cm-r01nn01.mws.mds.xyz, got DNS:cm-c01.mws.mds.xyz
Traceback (most recent call last):
  File "/opt/cloudera/cm-agent/lib/python2.7/site-packages/cmf/https.py", line 193, in fetch_to_file
    resp = self.open(req_url)
  File "/opt/cloudera/cm-agent/lib/python2.7/site-packages/cmf/https.py", line 188, in open
    return self.opener(url, *pargs, **kwargs)
  File "/usr/lib64/python2.7/urllib2.py", line 431, in open
    response = self._open(req, data)
  File "/usr/lib64/python2.7/urllib2.py", line 449, in _open
    '_open', req)
  File "/usr/lib64/python2.7/urllib2.py", line 409, in _call_chain
    result = func(*args)
  File "/opt/cloudera/cm-agent/lib/python2.7/site-packages/cmf/https.py", line 179, in https_open
    return self.do_open(opener, req)
  File "/usr/lib64/python2.7/urllib2.py", line 1211, in do_open
    h.request(req.get_method(), req.get_selector(), req.data, headers)
  File "/usr/lib64/python2.7/httplib.py", line 1041, in request
    self._send_request(method, url, body, headers)
  File "/usr/lib64/python2.7/httplib.py", line 1075, in _send_request
    self.endheaders(body)
  File "/usr/lib64/python2.7/httplib.py", line 1037, in endheaders
    self._send_output(message_body)
  File "/usr/lib64/python2.7/httplib.py", line 881, in _send_output
    self.send(msg)
  File "/usr/lib64/python2.7/httplib.py", line 843, in send
    self.connect()
  File "/opt/cloudera/cm-agent/lib/python2.7/site-packages/M2Crypto/httpslib.py", line 69, in connect
    sock.connect((self.host, self.port))
  File "/opt/cloudera/cm-agent/lib/python2.7/site-packages/M2Crypto/SSL/Connection.py", line 313, in connect
    if not check(self.get_peer_cert(), self.addr[0]):
  File "/opt/cloudera/cm-agent/lib/python2.7/site-packages/M2Crypto/SSL/Checker.py", line 125, in __call__
    fieldName='subjectAltName')
WrongHost: Peer certificate subjectAltName does not match host, expected cm-r01nn01.mws.mds.xyz, got DNS:cm-c01.mws.mds.xyz

Read the rest of this entry »

Cloudera: SSLError: certificate verify failed

Receiving the following when enabling SSL Certs on remote Cloudera Worker nodes from Azure, AWS or GCP?

[17/May/2020 13:07:32 +0000] 3332 MainThread agent ERROR    Heartbeating to 108.168.115.113:7182 failed.
Traceback (most recent call last):
  File "/opt/cloudera/cm-agent/lib/python2.7/site-packages/cmf/agent.py", line 1387, in _send_heartbeat
    self.cfg.max_cert_depth)
  File "/opt/cloudera/cm-agent/lib/python2.7/site-packages/cmf/https.py", line 139, in __init__
    self.conn.connect()
  File "/opt/cloudera/cm-agent/lib/python2.7/site-packages/M2Crypto/httpslib.py", line 69, in connect
    sock.connect((self.host, self.port))
  File "/opt/cloudera/cm-agent/lib/python2.7/site-packages/M2Crypto/SSL/Connection.py", line 309, in connect
    ret = self.connect_ssl()
  File "/opt/cloudera/cm-agent/lib/python2.7/site-packages/M2Crypto/SSL/Connection.py", line 295, in connect_ssl
    return m2.ssl_connect(self.ssl, self._timeout)
SSLError: certificate verify failed

Read the rest of this entry »


     
  Copyright © 2003 - 2013 Tom Kacperski (microdevsys.com). All rights reserved.

Creative Commons License
This work is licensed under a Creative Commons Attribution 3.0 Unported License