Header Shadow Image


DEBUG The ipa-server-upgrade command failed, exception: ScriptError: CA did not start in 300.0s

Getting this?

/var/log/ipaupgrade.log
2020-05-23T23:32:58Z ERROR IPA server upgrade failed: Inspect /var/log/ipaupgrade.log and run command ipa-server-upgrade manually.
2020-05-23T23:32:58Z DEBUG   File "/usr/lib/python2.7/site-packages/ipapython/admintool.py", line 178, in execute
    return_value = self.run()
  File "/usr/lib/python2.7/site-packages/ipaserver/install/ipa_server_upgrade.py", line 56, in run
    raise admintool.ScriptError(str(e))

?2020-05-23T23:16:22Z DEBUG The ipa-server-upgrade command failed, exception: ScriptError: CA did not start in 300.0s
2020-05-23T23:16:22Z ERROR CA did not start in 300.0s
2020-05-23T23:16:22Z ERROR The ipa-server-upgrade command failed. See /var/log/ipaupgrade.log for more information

/var/log/pki/pki-tomcat/ca/debug
Could not connect to LDAP server host idmipa04.mws.mds.xyz port 636 Error netscape.ldap.LDAPException: Unable to create socket: java.net.ConnectException: Connection refused (Connection refused) (-1)

It's likely because you have the following set:

[root@idmipa04 ca]# grep -Ei "nsslapd-port|nsslapd-security" /etc/dirsrv/slapd-MWS-MDS-XYZ/dse.ldif
nsslapd-port: 0
nsslapd-security: off
[root@idmipa04 ca]#

These need to be set to:

nsslapd-port: 389
nsslapd-security: on

But this did not work.  Checking certs expiration all shows dates in the future:

[root@idmipa04 ~]# getcert list|grep expires
        expires: 2021-02-05 07:37:13 UTC
        expires: 2021-02-05 07:37:42 UTC
        expires: 2021-01-25 03:22:30 UTC
        expires: 2021-01-25 03:21:37 UTC
        expires: 2021-01-25 03:21:36 UTC
        expires: 2021-01-25 03:21:37 UTC
        expires: 2039-02-05 03:21:36 UTC
        expires: 2021-01-25 07:40:56 UTC
        expires: 2021-02-05 07:42:11 UTC
[root@idmipa04 ~]#

Lastly, check for port 636 and 389 through netstat:

[root@idmipa04 pki-tomcat]# netstat -pnltu
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name
tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN      1069/sshd
tcp        0      0 0.0.0.0:88              0.0.0.0:*               LISTEN      1089/krb5kdc
tcp        0      0 127.0.0.1:25            0.0.0.0:*               LISTEN      1537/master
tcp6       0      0 :::22                   :::*                    LISTEN      1069/sshd
tcp6       0      0 :::88                   :::*                    LISTEN      1089/krb5kdc
tcp6       0      0 ::1:25                  :::*                    LISTEN      1537/master
tcp6       0      0 :::8443                 :::*                    LISTEN      16371/java
tcp6       0      0 :::443                  :::*                    LISTEN      15941/httpd
tcp6       0      0 127.0.0.1:8005          :::*                    LISTEN      16371/java
tcp6       0      0 127.0.0.1:8009          :::*                    LISTEN      16371/java
tcp6       0      0 :::8080                 :::*                    LISTEN      16371/java
tcp6       0      0 :::80                   :::*                    LISTEN      15941/httpd
udp        0      0 0.0.0.0:88              0.0.0.0:*                           1089/krb5kdc
udp6       0      0 :::88                   :::*                                1089/krb5kdc

If missing, start the directory server:

[root@idmipa04 pki-tomcat]# systemctl start dirsrv@MWS-MDS-XYZ.service

Check for the IP once started:

[root@idmipa04 pki-tomcat]# systemctl status dirsrv@MWS-MDS-XYZ.service
? dirsrv@MWS-MDS-XYZ.service – 389 Directory Server MWS-MDS-XYZ.
   Loaded: loaded (/usr/lib/systemd/system/dirsrv@.service; enabled; vendor preset: disabled)
   Active: active (running) since Sun 2020-05-24 01:44:55 EDT; 10s ago
  Process: 18618 ExecStartPre=/usr/sbin/ds_systemd_ask_password_acl /etc/dirsrv/slapd-%i/dse.ldif (code=exited, status=0/SUCCESS)
 Main PID: 18625 (ns-slapd)
   Status: "slapd started: Ready to process requests"
   CGroup: /system.slice/system-dirsrv.slice/dirsrv@MWS-MDS-XYZ.service
           ??18625 /usr/sbin/ns-slapd -D /etc/dirsrv/slapd-MWS-MDS-XYZ -i /var/run/dirsrv/slapd-MWS-…

May 24 01:44:55 idmipa04.mws.mds.xyz ns-slapd[18625]: GSSAPI client step 1
May 24 01:44:56 idmipa04.mws.mds.xyz ns-slapd[18625]: GSSAPI client step 1
May 24 01:44:56 idmipa04.mws.mds.xyz ns-slapd[18625]: GSSAPI client step 1
May 24 01:44:56 idmipa04.mws.mds.xyz ns-slapd[18625]: GSSAPI client step 2
May 24 01:44:57 idmipa04.mws.mds.xyz ns-slapd[18625]: [24/May/2020:01:44:57.329920836 -0400] – ERR…d.
May 24 01:44:57 idmipa04.mws.mds.xyz ns-slapd[18625]: [24/May/2020:01:44:57.331112434 -0400] – ERR…d.
May 24 01:45:00 idmipa04.mws.mds.xyz ns-slapd[18625]: [24/May/2020:01:45:00.339593970 -0400] – ERR…d.
May 24 01:45:00 idmipa04.mws.mds.xyz ns-slapd[18625]: [24/May/2020:01:45:00.340490104 -0400] – ERR…d.
May 24 01:45:03 idmipa04.mws.mds.xyz ns-slapd[18625]: [24/May/2020:01:45:03.348216609 -0400] – ERR…d.
May 24 01:45:03 idmipa04.mws.mds.xyz ns-slapd[18625]: [24/May/2020:01:45:03.354849567 -0400] – ERR…d.
Hint: Some lines were ellipsized, use -l to show in full.
[root@idmipa04 pki-tomcat]#

Check the ports are listening:

[root@idmipa04 pki-tomcat]# netstat -pnltu|grep 18625
tcp6       0      0 :::636                  :::*                    LISTEN      18625/ns-slapd
tcp6       0      0 :::389                  :::*                    LISTEN      18625/ns-slapd

Check the error logs for the service:

[root@idmipa04 pki-tomcat]# systemctl status dirsrv@MWS-MDS-XYZ.service -l
? dirsrv@MWS-MDS-XYZ.service – 389 Directory Server MWS-MDS-XYZ.
   Loaded: loaded (/usr/lib/systemd/system/dirsrv@.service; enabled; vendor preset: disabled)
   Active: active (running) since Sun 2020-05-24 01:44:55 EDT; 28s ago
  Process: 18618 ExecStartPre=/usr/sbin/ds_systemd_ask_password_acl /etc/dirsrv/slapd-%i/dse.ldif (code=exited, status=0/SUCCESS)
 Main PID: 18625 (ns-slapd)
   Status: "slapd started: Ready to process requests"
   CGroup: /system.slice/system-dirsrv.slice/dirsrv@MWS-MDS-XYZ.service
           ??18625 /usr/sbin/ns-slapd -D /etc/dirsrv/slapd-MWS-MDS-XYZ -i /var/run/dirsrv/slapd-MWS-MDS-XYZ.pid

May 24 01:45:09 idmipa04.mws.mds.xyz ns-slapd[18625]: [24/May/2020:01:45:09.372741696 -0400] – ERR – agmt="cn=caToidmipa03.mws.mds.xyz" (idmipa03:389) – clcache_load_buffer – Can't locate CSN 5c7bc2730000ffffffff in the changelog (DB rc=-30988). If replication stops, the consumer may need to be reinitialized.
May 24 01:45:09 idmipa04.mws.mds.xyz ns-slapd[18625]: [24/May/2020:01:45:09.373677051 -0400] – ERR – NSMMReplicationPlugin – send_updates – agmt="cn=caToidmipa03.mws.mds.xyz" (idmipa03:389): Missing data encountered. If the error persists the replica must be reinitialized.
[root@idmipa04 pki-tomcat]#

If you see the above, reinitialize the system:

[root@idmipa04 pki-tomcat]# ipa-csreplica-manage re-initialize –from idmipa03.mws.mds.xyz
Directory Manager password:

Update in progress, 3 seconds elapsed
Update succeeded

[root@idmipa04 pki-tomcat]# systemctl status dirsrv@MWS-MDS-XYZ.service -l
? dirsrv@MWS-MDS-XYZ.service – 389 Directory Server MWS-MDS-XYZ.
   Loaded: loaded (/usr/lib/systemd/system/dirsrv@.service; enabled; vendor preset: disabled)
   Active: active (running) since Sun 2020-05-24 01:44:55 EDT; 4min 29s ago
  Process: 18618 ExecStartPre=/usr/sbin/ds_systemd_ask_password_acl /etc/dirsrv/slapd-%i/dse.ldif (code=exited, status=0/SUCCESS)
 Main PID: 18625 (ns-slapd)
   Status: "slapd started: Ready to process requests"
   CGroup: /system.slice/system-dirsrv.slice/dirsrv@MWS-MDS-XYZ.service
           ??18625 /usr/sbin/ns-slapd -D /etc/dirsrv/slapd-MWS-MDS-XYZ -i /var/run/dirsrv/slapd-MWS-MDS-XYZ.pid

May 24 01:49:15 idmipa04.mws.mds.xyz ns-slapd[18625]: [24/May/2020:01:49:15.687759236 -0400] – WARN – NSMMReplicationPlugin – replica_reload_ruv – New data for replica o=ipaca does not match the data in the changelog.
May 24 01:49:15 idmipa04.mws.mds.xyz ns-slapd[18625]: Recreating the changelog file. This could affect replication with replica's  consumers in which case the consumers should be reinitialized.
May 24 01:49:15 idmipa04.mws.mds.xyz ns-slapd[18625]: [24/May/2020:01:49:15.721328728 -0400] – ERR – cos-plugin – cos_dn_defs_cb – Skipping CoS Definition cn=Password Policy,cn=accounts,dc=mws,dc=mds,dc=xyz–no CoS Templates found, which should be added before the CoS Definition.
May 24 01:49:15 idmipa04.mws.mds.xyz ns-slapd[18625]: [24/May/2020:01:49:15.727578549 -0400] – NOTICE – NSMMReplicationPlugin – changelog program – _cl5ConstructRUV – Rebuilding the replication changelog RUV, this may take several minutes…
May 24 01:49:15 idmipa04.mws.mds.xyz ns-slapd[18625]: [24/May/2020:01:49:15.728113208 -0400] – NOTICE – NSMMReplicationPlugin – changelog program – _cl5ConstructRUV – Rebuilding replication changelog RUV complete.  Result 0 (Success)
May 24 01:49:15 idmipa04.mws.mds.xyz ns-slapd[18625]: [24/May/2020:01:49:15.728579987 -0400] – NOTICE – NSMMReplicationPlugin – changelog program – _cl5ConstructRUV – Rebuilding the replication changelog RUV, this may take several minutes…
May 24 01:49:15 idmipa04.mws.mds.xyz ns-slapd[18625]: [24/May/2020:01:49:15.728985312 -0400] – NOTICE – NSMMReplicationPlugin – changelog program – _cl5ConstructRUV – Rebuilding replication changelog RUV complete.  Result 0 (Success)
May 24 01:49:15 idmipa04.mws.mds.xyz ns-slapd[18625]: GSSAPI server step 1
May 24 01:49:15 idmipa04.mws.mds.xyz ns-slapd[18625]: GSSAPI server step 2
May 24 01:49:15 idmipa04.mws.mds.xyz ns-slapd[18625]: GSSAPI server step 3

[root@idmipa04 pki-tomcat]#

Your FreeIPA server should now be back up?  Let's try that and see what happens.

/var/log/ipaupgrade.log
2020-05-24T06:00:06Z DEBUG request POST http://idmipa04.mws.mds.xyz:8080/ca/admin/ca/getStatus
2020-05-24T06:00:06Z DEBUG request body ''
2020-05-24T06:00:06Z DEBUG response status 200
2020-05-24T06:00:06Z DEBUG response headers Server: Apache-Coyote/1.1
Content-Type: application/xml
Content-Length: 168
Date: Sun, 24 May 2020 06:00:06 GMT

2020-05-24T06:00:06Z DEBUG response body '<?xml version="1.0" encoding="UTF-8" standalone="no"?><XMLResponse><State>1</State><Type>CA</Type><Status>running</Status><Version>10.5.17-6.el7</Version></XMLResponse>'
2020-05-24T06:00:06Z INFO The IPA services were upgraded
2020-05-24T06:00:06Z DEBUG Loading StateFile from '/var/lib/ipa/sysupgrade/sysupgrade.state'
2020-05-24T06:00:06Z DEBUG Saving StateFile to '/var/lib/ipa/sysupgrade/sysupgrade.state'
2020-05-24T06:00:06Z DEBUG Loading StateFile from '/var/lib/ipa/sysupgrade/sysupgrade.state'
2020-05-24T06:00:06Z DEBUG Saving StateFile to '/var/lib/ipa/sysupgrade/sysupgrade.state'
2020-05-24T06:00:06Z INFO The ipa-server-upgrade command was successful

Confirming the command now succeeded as expected:

[root@idmipa04 pki-tomcat]# ipactl start
IPA version error: data needs to be upgraded (expected version '4.6.6-11.el7.centos', current version '4.6.4-10.el7.centos.2')
Automatically running upgrade, for details see /var/log/ipaupgrade.log
Be patient, this may take a few minutes.
Existing service file detected!
Assuming stale, cleaning and proceeding
Starting Directory Service
Starting krb5kdc Service
Starting kadmin Service
Starting named Service
Starting httpd Service
Starting ipa-custodia Service
Starting ntpd Service
Starting pki-tomcatd Service
Starting ipa-otpd Service
Starting ipa-dnskeysyncd Service
ipa: INFO: The ipactl command was successful
[root@idmipa04 pki-tomcat]#

RELATED ERRORS:

The following errors were seen alongside the above-mentioned entries.

/var/log/ipaupgrade.log
2020-05-02T12:50:40Z DEBUG The ipa-server-upgrade command failed, exception: CalledProcessError: Command '/bin/systemctl start dirsrv@MWS-MDS-XYZ.service' returned non-zero exit status 1

2020-05-23T21:07:50Z DEBUG The CA status is: check interrupted due to error: Retrieving CA status failed with status 500

/var/log/pki/pki-tomcat/localhost.2020-05-24.log
SEVERE: Exception Processing /ca/admin/ca/getStatus
javax.ws.rs.ServiceUnavailableException: Subsystem unavailable

SEVERE: Servlet.service() for servlet [Resteasy] in context with path [/ca] threw exception
org.jboss.resteasy.spi.UnhandledException: org.jboss.resteasy.core.NoMessageBodyWriterFoundFailure: Could not find MessageBodyWriter for response object of type: com.netscape.certsrv.base.PKIException$Data of media type: application/x-www-form-urlencoded

/var/log/pki/pki-tomcat/ca/debug
Could not connect to LDAP server host idmipa04.mws.mds.xyz port 636 Error netscape.ldap.LDAPException: Unable to create socket: java.net.ConnectException: Connection refused (Connection refused) (-1)

/var/log/dirsrv/slapd-MWS-MDS-XYZ/errors
[24/May/2020:01:02:41.912364232 -0400] – ERR – NSMMReplicationPlugin – send_updates – agmt="cn=caToidmipa03.mws.mds.xyz" (idmipa03:389): Missing data encountered. If the error persists the replica must be reinitialized.

[23/May/2020:00:40:23.025920441 -0400] – ERR – set_krb5_creds – Could not get initial credentials for principal [ldap/idmipa04.mws.mds.xyz@MWS.MDS.XYZ] in keytab [FILE:/etc/dirsrv/ds.keytab]: -1765328228 (Cannot contact any KDC for requested realm)

Cheers,
TK
 

Cloudera: ERROR Heartbeating to :7182 failed. SSLError: unexpected eof

Getting this?

[18/May/2020 18:04:46 +0000] 2849 MainThread agent        ERROR    Heartbeating to srv-c01.mws.mds.xyz:7182 failed.
Traceback (most recent call last):
  File "/opt/cloudera/cm-agent/lib/python2.7/site-packages/cmf/agent.py", line 1387, in _send_heartbeat
    self.cfg.max_cert_depth)
  File "/opt/cloudera/cm-agent/lib/python2.7/site-packages/cmf/https.py", line 139, in __init__
    self.conn.connect()
  File "/opt/cloudera/cm-agent/lib/python2.7/site-packages/M2Crypto/httpslib.py", line 69, in connect
    sock.connect((self.host, self.port))
  File "/opt/cloudera/cm-agent/lib/python2.7/site-packages/M2Crypto/SSL/Connection.py", line 309, in connect
    ret = self.connect_ssl()
  File "/opt/cloudera/cm-agent/lib/python2.7/site-packages/M2Crypto/SSL/Connection.py", line 295, in connect_ssl
    return m2.ssl_connect(self.ssl, self._timeout)
SSLError: unexpected eof

In this case it was because the Cloudera SCM Server was offline.  Started it back up to resolve the above.  Having said this, if your server is up but you still get this, then Cloudera SCM Server hasn't sent an SSL / TLS Certificate.  

Cheers,
BK

Cloudera: WrongHost: Peer certificate subjectAltName does not match host, expected HOST01, got HOST02

Getting the following while connecting TLS enabled Azure, AWS or GCP cloud hosts to Cloudera Manager?

[18/May/2020 13:12:09 +0000] 2413 Thread-13 downloader   INFO     Fetching torrent: https://cm-r01nn01.mws.mds.xyz:7183/cmf/parcel/download/CDH-6.3.0-1.cdh6.3.0.p0.1279813-el7.parcel.torrent
[18/May/2020 13:12:09 +0000] 2413 Thread-13 https        ERROR    Failed to retrieve/store URL: https://cm-r01nn01.mws.mds.xyz:7183/cmf/parcel/download/CDH-6.3.0-1.cdh6.3.0.p0.1279813-el7.parcel.torrent -> /opt/cloudera/parcel-cache/CDH-6.3.0-1.cdh6.3.0.p0.1279813-el7.parcel.torrent Peer certificate subjectAltName does not match host, expected cm-r01nn01.mws.mds.xyz, got DNS:cm-c01.mws.mds.xyz
Traceback (most recent call last):
  File "/opt/cloudera/cm-agent/lib/python2.7/site-packages/cmf/https.py", line 193, in fetch_to_file
    resp = self.open(req_url)
  File "/opt/cloudera/cm-agent/lib/python2.7/site-packages/cmf/https.py", line 188, in open
    return self.opener(url, *pargs, **kwargs)
  File "/usr/lib64/python2.7/urllib2.py", line 431, in open
    response = self._open(req, data)
  File "/usr/lib64/python2.7/urllib2.py", line 449, in _open
    '_open', req)
  File "/usr/lib64/python2.7/urllib2.py", line 409, in _call_chain
    result = func(*args)
  File "/opt/cloudera/cm-agent/lib/python2.7/site-packages/cmf/https.py", line 179, in https_open
    return self.do_open(opener, req)
  File "/usr/lib64/python2.7/urllib2.py", line 1211, in do_open
    h.request(req.get_method(), req.get_selector(), req.data, headers)
  File "/usr/lib64/python2.7/httplib.py", line 1041, in request
    self._send_request(method, url, body, headers)
  File "/usr/lib64/python2.7/httplib.py", line 1075, in _send_request
    self.endheaders(body)
  File "/usr/lib64/python2.7/httplib.py", line 1037, in endheaders
    self._send_output(message_body)
  File "/usr/lib64/python2.7/httplib.py", line 881, in _send_output
    self.send(msg)
  File "/usr/lib64/python2.7/httplib.py", line 843, in send
    self.connect()
  File "/opt/cloudera/cm-agent/lib/python2.7/site-packages/M2Crypto/httpslib.py", line 69, in connect
    sock.connect((self.host, self.port))
  File "/opt/cloudera/cm-agent/lib/python2.7/site-packages/M2Crypto/SSL/Connection.py", line 313, in connect
    if not check(self.get_peer_cert(), self.addr[0]):
  File "/opt/cloudera/cm-agent/lib/python2.7/site-packages/M2Crypto/SSL/Checker.py", line 125, in __call__
    fieldName='subjectAltName')
WrongHost: Peer certificate subjectAltName does not match host, expected cm-r01nn01.mws.mds.xyz, got DNS:cm-c01.mws.mds.xyz

Read the rest of this entry »

Cloudera: SSLError: certificate verify failed

Receiving the following when enabling SSL Certs on remote Cloudera Worker nodes from Azure, AWS or GCP?

[17/May/2020 13:07:32 +0000] 3332 MainThread agent ERROR    Heartbeating to 108.168.115.113:7182 failed.
Traceback (most recent call last):
  File "/opt/cloudera/cm-agent/lib/python2.7/site-packages/cmf/agent.py", line 1387, in _send_heartbeat
    self.cfg.max_cert_depth)
  File "/opt/cloudera/cm-agent/lib/python2.7/site-packages/cmf/https.py", line 139, in __init__
    self.conn.connect()
  File "/opt/cloudera/cm-agent/lib/python2.7/site-packages/M2Crypto/httpslib.py", line 69, in connect
    sock.connect((self.host, self.port))
  File "/opt/cloudera/cm-agent/lib/python2.7/site-packages/M2Crypto/SSL/Connection.py", line 309, in connect
    ret = self.connect_ssl()
  File "/opt/cloudera/cm-agent/lib/python2.7/site-packages/M2Crypto/SSL/Connection.py", line 295, in connect_ssl
    return m2.ssl_connect(self.ssl, self._timeout)
SSLError: certificate verify failed

Read the rest of this entry »

Cloudera: Azure: URLError:

Receiving the following when connecting external cloud servers from Azure, AWS, or GCP?

[17/May/2020 13:29:50 +0000] 4894 Thread-13 https        ERROR    Failed to retrieve/store URL: https://cm-r01nn01.mws.mds.xyz:7183/cmf/pa    rcel/download/CDH-6.3.0-1.cdh6.3.0.p0.1279813-el7.parcel.torrent -> /opt/cloudera/parcel-cache/CDH-6.3.0-1.cdh6.3.0.p0.1279813-el7.parcel.    torrent Traceback (most recent call last):
  File "/opt/cloudera/cm-agent/lib/python2.7/site-packages/cmf/https.py", line 193, in fetch_to_file
    resp = self.open(req_url)
  File "/opt/cloudera/cm-agent/lib/python2.7/site-packages/cmf/https.py", line 188, in open
    return self.opener(url, *pargs, **kwargs)
  File "/usr/lib64/python2.7/urllib2.py", line 431, in open
    response = self._open(req, data)
  File "/usr/lib64/python2.7/urllib2.py", line 449, in _open
    '_open', req)
  File "/usr/lib64/python2.7/urllib2.py", line 409, in _call_chain
    result = func(*args)
  File "/opt/cloudera/cm-agent/lib/python2.7/site-packages/cmf/https.py", line 179, in https_open
    return self.do_open(opener, req)
  File "/usr/lib64/python2.7/urllib2.py", line 1214, in do_open
    raise URLError(err)
URLError: <urlopen error [Errno -2] Name or service not known>

Solve it by adding entries into your /etc/hosts file like this:

[root@cm-awn01 ~]# cat /etc/hosts
127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
::1         localhost localhost.localdomain localhost6 localhost6.localdomain6

100.100.100.10   cm-awn01.nix.mds.xyz cm-awn01
10.0.0.6        cm-awn01.nix.mds.xyz cm-awn01

123.123.123.123 srv-c01.mws.mds.xyz
123.123.123.123 cm-r01nn01.mws.mds.xyz
123.123.123.123 cm-r01nn02.mws.mds.xyz
[root@cm-awn01 ~]#

Thanks
TK

Cloudera: WrongHost: Peer certificate subjectAltName does not match host

Getting the following when configuring remote workers on Azure:

[17/May/2020 13:09:38 +0000] 3529 MainThread agent        ERROR    Heartbeating to 123.123.123.123:7182 failed.
Traceback (most recent call last):
  File "/opt/cloudera/cm-agent/lib/python2.7/site-packages/cmf/agent.py", line 1387, in _send_heartbeat
    self.cfg.max_cert_depth)
  File "/opt/cloudera/cm-agent/lib/python2.7/site-packages/cmf/https.py", line 139, in _init_
    self.conn.connect()
  File "/opt/cloudera/cm-agent/lib/python2.7/site-packages/M2Crypto/httpslib.py", line 69, in connect
    sock.connect((self.host, self.port))
  File "/opt/cloudera/cm-agent/lib/python2.7/site-packages/M2Crypto/SSL/Connection.py", line 313, in connect
    if not check(self.get_peer_cert(), self.addr[0]):
  File "/opt/cloudera/cm-agent/lib/python2.7/site-packages/M2Crypto/SSL/Checker.py", line 125, in _call_
    fieldName='subjectAltName')
WrongHost: Peer certificate subjectAltName does not match host, expected 123.123.123.123, got DNS:srv-c01.mws.mds.xyz, DNS:cm-r01nn01.mws.mds.xyz, DNS:cm-r01nn02.mws.mds.xyz

Because locally visible hostnames aren't externally, /etc/hosts modifications are necessary in this case to make the self signed certificates happy:

[root@cm-awn01 ~]# cat /etc/hosts
127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
::1         localhost localhost.localdomain localhost6 localhost6.localdomain6

100.100.100.10   cm-awn01.nix.mds.xyz cm-awn01
10.0.0.6        cm-awn01.nix.mds.xyz cm-awn01

123.123.123.123 srv-c01.mws.mds.xyz
123.123.123.123 cm-r01nn01.mws.mds.xyz
123.123.123.123 cm-r01nn02.mws.mds.xyz
[root@cm-awn01 ~]#

Ensure your Cloudera Agent Config matches:

[root@cm-awn01 ~]# cat /etc/cloudera-scm-agent/config.ini|grep server
server_host=srv-c01.mws.mds.xyz

Hackish but works.

GL,
TK

Cloudera: Ncat: Connection timed out.

Getting this error when connecting Azure instances back to a Cloudera CM + CDH installation?

[root@cm-awn01 ~]# nc -vz srv-c01.mws.mds.xyz 7191
Ncat: Version 7.50 ( https://nmap.org/ncat )
Ncat: Connection timed out.
[root@cm-awn01 ~]#

Ensure ports are defined correctly in the local firewall configuration:

[root@cm-r01xs01 .ssh]# grep -Ei 7191 /etc/firewalld/zones/public.xml
  <port protocol="tcp" port="7191"/>
  <port protocol="udp" port="7191"/>

[root@cm-r01xs01 .ssh]#

Ensure NAT rules are also in place in case connectivity occurs through an external facing router.

iptables -t nat -I PREROUTING -s 123.123.123.123 -p tcp --dport 7191 -j DNAT --to 192.168.0.120:7191
iptables -I FORWARD -p tcp -d 192.168.0.120 --dport 7191 -j ACCEPT
iptables -t nat -I PREROUTING -s 123.123.123.123 -p udp --dport 7191 -j DNAT --to 192.168.0.120:7191
iptables -I FORWARD -p udp -d 192.168.0.120 --dport 7191 -j ACCEPT

Good Luck!
TK

Cloudera: Ncat: Connection refused.

Getting the following connecting from your Azure instance back to your Cloudera CM + CDH Cluster?

[root@cm-awn01 ~]# nc -v 123.123.123.123 7183
Ncat: Version 7.50 ( https://nmap.org/ncat )
Ncat: Connection refused.

The corresponding Cloudera Agent error looks like this:

[17/May/2020 13:34:53 +0000] 5306 Thread-13 https        ERROR    Failed to retrieve/store URL: https://cm-r01nn01.mws.mds.xyz:7183/cmf/par   cel/download/CDH-6.3.0-1.cdh6.3.0.p0.1279813-el7.parcel.torrent -> /opt/cloudera/parcel-cache/CDH-6.3.0-1.cdh6.3.0.p0.1279813-el7.parcel.to   rrent Traceback (most recent call last): 
  File "/opt/cloudera/cm-agent/lib/python2.7/site-packages/cmf/https.py", line 193, in fetch_to_file
    resp = self.open(req_url)
  File "/opt/cloudera/cm-agent/lib/python2.7/site-packages/cmf/https.py", line 188, in open
    return self.opener(url, *pargs, **kwargs)
  File "/usr/lib64/python2.7/urllib2.py", line 431, in open
    response = self._open(req, data)
  File "/usr/lib64/python2.7/urllib2.py", line 449, in _open
    '_open', req)
  File "/usr/lib64/python2.7/urllib2.py", line 409, in _call_chain
    result = func(*args)
  File "/opt/cloudera/cm-agent/lib/python2.7/site-packages/cmf/https.py", line 179, in https_open
    return self.do_open(opener, req)
  File "/usr/lib64/python2.7/urllib2.py", line 1214, in do_open
    raise URLError(err)
URLError: <urlopen error [Errno 111] Connection refused>

If you're using HAProxy for traffic routing to various instances, you'll need the following stanza in the HAProxy configuration file:

# CM GUI
listen cm
        bind cm-c01:80
        mode    http
        redirect scheme https if !{ ssl_fc }

frontend cmin
        bind    cm-c01:443 ssl crt /etc/haproxy/certs/cm-c01.mws.mds.xyz-haproxy.pem no-sslv3
        default_backend cmback

frontend cm7183in
        bind    cm-c01:7183 ssl crt /etc/haproxy/certs/cm-c01.mws.mds.xyz-haproxy.pem no-sslv3
        default_backend cmback

backend cmback
        mode http
        balance roundrobin

        server cm-r01nn01.mws.mds.xyz    cm-r01nn01.mws.mds.xyz:7183 ssl check verify none port 7183 inter 12000 rise 3 fall 3
        server cm-r01nn02.mws.mds.xyz    cm-r01nn02.mws.mds.xyz:7183 ssl check verify none port 7183 inter 12000 rise 3 fall 3

Otherwise the following port won't be open:

[root@cm-r01xs01 .ssh]# netstat -pnltu 
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name
tcp        0      0 192.168.0.120:7183       0.0.0.0:*               LISTEN      15001/haproxy
[root@cm-r01xs01 .ssh]# 

And you'll receive the subject error.  If you're routing traffic through your external facing router, ensure you have proper firewall rules configured for NAT traffic to and from the Azure instances:

iptables -t nat -I PREROUTING -s 103.192.131.145 -p tcp --dport 7183 -j DNAT --to 192.168.0.120:7183
iptables -I FORWARD -p tcp -d 192.168.0.53 --dport 7183 -j ACCEPT
iptables -t nat -I PREROUTING -s 103.192.131.145 -p udp --dport 7183 -j DNAT --to 192.168.0.120:7183
iptables -I FORWARD -p udp -d 192.168.0.53 --dport 7183 -j ACCEPT

Allowing traffic from said external IP 103.192.131.145.  Furthermore, your HAproxy or Cloudera servers also contain the valid port.  For instance:

[root@cm-r01xs01 .ssh]# cat /etc/firewalld/zones/public.xml |grep -Ei 7183
  <port protocol="tcp" port="7183"/>
  <port protocol="udp" port="7183"/>
[root@cm-r01xs01 .ssh]#

With that, you can expect the following result from your Azure instances.

[root@cm-awn01 ~]# nc -v 123.123.123.123 7183
Ncat: Version 7.50 ( https://nmap.org/ncat )
Ncat: Connected to 123.123.123.123:7183.

HTH,
BD

Importing Certificates Into the Truststore: Connection not protected

Getting this message?

Connection not protected

The security of your connection is reduced. Criminals can attempt to steal your data from the website. You are advised to leave this website.

And you know this is a trusted site, such as a local HTTPS web server?  Then export the certificate and add it to your trusted certificates stash by following these steps.

  1. Click on View Certificate
  2. From the panel that opens up, click on the Details tab.
  3. Next, click on Copy to File….
  4. Leave default settings.  In our case, it was DER encoded binary X.509 (. CER )
  5. Following that select the location and the name of the file to copy the certificate too.
  6. Next, start mmc.exe by searching for this program on your Windows 10 machine or executing it from the command line. 
  7. Click on File then  Add / Remove Snap In
  8. Click on Certificates followed by clicking the Add > button.
  9. Save or click Finish
  10. Next, navigate to Console Root -> Certificates (Local Computer) -> Trusted Root Certificatiom Authorities -> Certificates
  11. On the panel that opens up, right click Certificates and All Tasks -> Import
  12. Once imported, File -> Save and Exit.
  13. Reload your site. 

Steps 1,2,3,4,5:

https://i2.wp.com/www.microdevsys.com/WordPressImages/ImportTrustedCertificates-CertificateExportWizard.JPG?ssl=1

Steps 6, 7 and 8:

https://i2.wp.com/www.microdevsys.com/WordPressImages/ImportTrustedCertificates-MMC-Add-Certificates-Snap-In.JPG?ssl=1

Steps 10 and 11:

https://i2.wp.com/www.microdevsys.com/WordPressImages/ImportTrustedCertificates-MMC-Import-Certificate-Wizard.JPG?ssl=1

Regards,
AS

Connection not protected: Kaspersky Antivirus

Getting this?

Connection not protected

The security of your connection is reduced. Criminals can attempt to steal your data from the website. You are advised to leave this website.

For regular web traffic, this is normal. But if you want to avoid this for a local HTTPS server you're running, then these are the steps you want to take to disable this:

  1. In Kacpersky Total Security, click the Cog Wheel near the bottom left.
  2. Next, select Protection then Application Control from the right side. 
  3. Click on Manage Applications
  4. Type the browser you would like to modify in the top right search box.
  5. Double click the browser app and click the Exclusions tab
  6. Click on Do not scan all traffic 
  7. Select the checkbox Only for specific IP addresses
  8. Type in the addresses.
  9. Save the settings

Example of what this look like:

https://i2.wp.com/www.microdevsys.com/WordPressImages/KasperskyAntivitus-AllowSites.JPG?ssl=1

NOTE: Above screenshots shows a sample application.  It should really be Microsoft iExplorer, Firefox or Chrome if those browsers are being used. 

Thx,
EJ


     
  Copyright © 2003 - 2013 Tom Kacperski (microdevsys.com). All rights reserved.

Creative Commons License
This work is licensed under a Creative Commons Attribution 3.0 Unported License