Header Shadow Image


Cloudera: ERROR Heartbeating to :7182 failed. SSLError: unexpected eof

Getting this?

[18/May/2020 18:04:46 +0000] 2849 MainThread agent        ERROR    Heartbeating to srv-c01.mws.mds.xyz:7182 failed.
Traceback (most recent call last):
  File "/opt/cloudera/cm-agent/lib/python2.7/site-packages/cmf/agent.py", line 1387, in _send_heartbeat
    self.cfg.max_cert_depth)
  File "/opt/cloudera/cm-agent/lib/python2.7/site-packages/cmf/https.py", line 139, in __init__
    self.conn.connect()
  File "/opt/cloudera/cm-agent/lib/python2.7/site-packages/M2Crypto/httpslib.py", line 69, in connect
    sock.connect((self.host, self.port))
  File "/opt/cloudera/cm-agent/lib/python2.7/site-packages/M2Crypto/SSL/Connection.py", line 309, in connect
    ret = self.connect_ssl()
  File "/opt/cloudera/cm-agent/lib/python2.7/site-packages/M2Crypto/SSL/Connection.py", line 295, in connect_ssl
    return m2.ssl_connect(self.ssl, self._timeout)
SSLError: unexpected eof

In this case it was because the Cloudera SCM Server was offline.  Started it back up to resolve the above.  Having said this, if your server is up but you still get this, then Cloudera SCM Server hasn't sent an SSL / TLS Certificate.  

Cheers,
BK

Cloudera: WrongHost: Peer certificate subjectAltName does not match host, expected HOST01, got HOST02

Getting the following while connecting TLS enabled Azure, AWS or GCP cloud hosts to Cloudera Manager?

[18/May/2020 13:12:09 +0000] 2413 Thread-13 downloader   INFO     Fetching torrent: https://cm-r01nn01.mws.mds.xyz:7183/cmf/parcel/download/CDH-6.3.0-1.cdh6.3.0.p0.1279813-el7.parcel.torrent
[18/May/2020 13:12:09 +0000] 2413 Thread-13 https        ERROR    Failed to retrieve/store URL: https://cm-r01nn01.mws.mds.xyz:7183/cmf/parcel/download/CDH-6.3.0-1.cdh6.3.0.p0.1279813-el7.parcel.torrent -> /opt/cloudera/parcel-cache/CDH-6.3.0-1.cdh6.3.0.p0.1279813-el7.parcel.torrent Peer certificate subjectAltName does not match host, expected cm-r01nn01.mws.mds.xyz, got DNS:cm-c01.mws.mds.xyz
Traceback (most recent call last):
  File "/opt/cloudera/cm-agent/lib/python2.7/site-packages/cmf/https.py", line 193, in fetch_to_file
    resp = self.open(req_url)
  File "/opt/cloudera/cm-agent/lib/python2.7/site-packages/cmf/https.py", line 188, in open
    return self.opener(url, *pargs, **kwargs)
  File "/usr/lib64/python2.7/urllib2.py", line 431, in open
    response = self._open(req, data)
  File "/usr/lib64/python2.7/urllib2.py", line 449, in _open
    '_open', req)
  File "/usr/lib64/python2.7/urllib2.py", line 409, in _call_chain
    result = func(*args)
  File "/opt/cloudera/cm-agent/lib/python2.7/site-packages/cmf/https.py", line 179, in https_open
    return self.do_open(opener, req)
  File "/usr/lib64/python2.7/urllib2.py", line 1211, in do_open
    h.request(req.get_method(), req.get_selector(), req.data, headers)
  File "/usr/lib64/python2.7/httplib.py", line 1041, in request
    self._send_request(method, url, body, headers)
  File "/usr/lib64/python2.7/httplib.py", line 1075, in _send_request
    self.endheaders(body)
  File "/usr/lib64/python2.7/httplib.py", line 1037, in endheaders
    self._send_output(message_body)
  File "/usr/lib64/python2.7/httplib.py", line 881, in _send_output
    self.send(msg)
  File "/usr/lib64/python2.7/httplib.py", line 843, in send
    self.connect()
  File "/opt/cloudera/cm-agent/lib/python2.7/site-packages/M2Crypto/httpslib.py", line 69, in connect
    sock.connect((self.host, self.port))
  File "/opt/cloudera/cm-agent/lib/python2.7/site-packages/M2Crypto/SSL/Connection.py", line 313, in connect
    if not check(self.get_peer_cert(), self.addr[0]):
  File "/opt/cloudera/cm-agent/lib/python2.7/site-packages/M2Crypto/SSL/Checker.py", line 125, in __call__
    fieldName='subjectAltName')
WrongHost: Peer certificate subjectAltName does not match host, expected cm-r01nn01.mws.mds.xyz, got DNS:cm-c01.mws.mds.xyz

Read the rest of this entry »

Cloudera: SSLError: certificate verify failed

Receiving the following when enabling SSL Certs on remote Cloudera Worker nodes from Azure, AWS or GCP?

[17/May/2020 13:07:32 +0000] 3332 MainThread agent ERROR    Heartbeating to 108.168.115.113:7182 failed.
Traceback (most recent call last):
  File "/opt/cloudera/cm-agent/lib/python2.7/site-packages/cmf/agent.py", line 1387, in _send_heartbeat
    self.cfg.max_cert_depth)
  File "/opt/cloudera/cm-agent/lib/python2.7/site-packages/cmf/https.py", line 139, in __init__
    self.conn.connect()
  File "/opt/cloudera/cm-agent/lib/python2.7/site-packages/M2Crypto/httpslib.py", line 69, in connect
    sock.connect((self.host, self.port))
  File "/opt/cloudera/cm-agent/lib/python2.7/site-packages/M2Crypto/SSL/Connection.py", line 309, in connect
    ret = self.connect_ssl()
  File "/opt/cloudera/cm-agent/lib/python2.7/site-packages/M2Crypto/SSL/Connection.py", line 295, in connect_ssl
    return m2.ssl_connect(self.ssl, self._timeout)
SSLError: certificate verify failed

Read the rest of this entry »

Cloudera: Azure: URLError:

Receiving the following when connecting external cloud servers from Azure, AWS, or GCP?

[17/May/2020 13:29:50 +0000] 4894 Thread-13 https        ERROR    Failed to retrieve/store URL: https://cm-r01nn01.mws.mds.xyz:7183/cmf/pa    rcel/download/CDH-6.3.0-1.cdh6.3.0.p0.1279813-el7.parcel.torrent -> /opt/cloudera/parcel-cache/CDH-6.3.0-1.cdh6.3.0.p0.1279813-el7.parcel.    torrent Traceback (most recent call last):
  File "/opt/cloudera/cm-agent/lib/python2.7/site-packages/cmf/https.py", line 193, in fetch_to_file
    resp = self.open(req_url)
  File "/opt/cloudera/cm-agent/lib/python2.7/site-packages/cmf/https.py", line 188, in open
    return self.opener(url, *pargs, **kwargs)
  File "/usr/lib64/python2.7/urllib2.py", line 431, in open
    response = self._open(req, data)
  File "/usr/lib64/python2.7/urllib2.py", line 449, in _open
    '_open', req)
  File "/usr/lib64/python2.7/urllib2.py", line 409, in _call_chain
    result = func(*args)
  File "/opt/cloudera/cm-agent/lib/python2.7/site-packages/cmf/https.py", line 179, in https_open
    return self.do_open(opener, req)
  File "/usr/lib64/python2.7/urllib2.py", line 1214, in do_open
    raise URLError(err)
URLError: <urlopen error [Errno -2] Name or service not known>

Solve it by adding entries into your /etc/hosts file like this:

[root@cm-awn01 ~]# cat /etc/hosts
127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
::1         localhost localhost.localdomain localhost6 localhost6.localdomain6

100.100.100.10   cm-awn01.nix.mds.xyz cm-awn01
10.0.0.6        cm-awn01.nix.mds.xyz cm-awn01

123.123.123.123 srv-c01.mws.mds.xyz
123.123.123.123 cm-r01nn01.mws.mds.xyz
123.123.123.123 cm-r01nn02.mws.mds.xyz
[root@cm-awn01 ~]#

Thanks
TK

Cloudera: WrongHost: Peer certificate subjectAltName does not match host

Getting the following when configuring remote workers on Azure:

[17/May/2020 13:09:38 +0000] 3529 MainThread agent        ERROR    Heartbeating to 123.123.123.123:7182 failed.
Traceback (most recent call last):
  File "/opt/cloudera/cm-agent/lib/python2.7/site-packages/cmf/agent.py", line 1387, in _send_heartbeat
    self.cfg.max_cert_depth)
  File "/opt/cloudera/cm-agent/lib/python2.7/site-packages/cmf/https.py", line 139, in _init_
    self.conn.connect()
  File "/opt/cloudera/cm-agent/lib/python2.7/site-packages/M2Crypto/httpslib.py", line 69, in connect
    sock.connect((self.host, self.port))
  File "/opt/cloudera/cm-agent/lib/python2.7/site-packages/M2Crypto/SSL/Connection.py", line 313, in connect
    if not check(self.get_peer_cert(), self.addr[0]):
  File "/opt/cloudera/cm-agent/lib/python2.7/site-packages/M2Crypto/SSL/Checker.py", line 125, in _call_
    fieldName='subjectAltName')
WrongHost: Peer certificate subjectAltName does not match host, expected 123.123.123.123, got DNS:srv-c01.mws.mds.xyz, DNS:cm-r01nn01.mws.mds.xyz, DNS:cm-r01nn02.mws.mds.xyz

Because locally visible hostnames aren't externally, /etc/hosts modifications are necessary in this case to make the self signed certificates happy:

[root@cm-awn01 ~]# cat /etc/hosts
127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
::1         localhost localhost.localdomain localhost6 localhost6.localdomain6

100.100.100.10   cm-awn01.nix.mds.xyz cm-awn01
10.0.0.6        cm-awn01.nix.mds.xyz cm-awn01

123.123.123.123 srv-c01.mws.mds.xyz
123.123.123.123 cm-r01nn01.mws.mds.xyz
123.123.123.123 cm-r01nn02.mws.mds.xyz
[root@cm-awn01 ~]#

Ensure your Cloudera Agent Config matches:

[root@cm-awn01 ~]# cat /etc/cloudera-scm-agent/config.ini|grep server
server_host=srv-c01.mws.mds.xyz

Hackish but works.

GL,
TK

Cloudera: Ncat: Connection timed out.

Getting this error when connecting Azure instances back to a Cloudera CM + CDH installation?

[root@cm-awn01 ~]# nc -vz srv-c01.mws.mds.xyz 7191
Ncat: Version 7.50 ( https://nmap.org/ncat )
Ncat: Connection timed out.
[root@cm-awn01 ~]#

Ensure ports are defined correctly in the local firewall configuration:

[root@cm-r01xs01 .ssh]# grep -Ei 7191 /etc/firewalld/zones/public.xml
  <port protocol="tcp" port="7191"/>
  <port protocol="udp" port="7191"/>

[root@cm-r01xs01 .ssh]#

Ensure NAT rules are also in place in case connectivity occurs through an external facing router.

iptables -t nat -I PREROUTING -s 123.123.123.123 -p tcp --dport 7191 -j DNAT --to 192.168.0.120:7191
iptables -I FORWARD -p tcp -d 192.168.0.120 --dport 7191 -j ACCEPT
iptables -t nat -I PREROUTING -s 123.123.123.123 -p udp --dport 7191 -j DNAT --to 192.168.0.120:7191
iptables -I FORWARD -p udp -d 192.168.0.120 --dport 7191 -j ACCEPT

Good Luck!
TK

Cloudera: Ncat: Connection refused.

Getting the following connecting from your Azure instance back to your Cloudera CM + CDH Cluster?

[root@cm-awn01 ~]# nc -v 123.123.123.123 7183
Ncat: Version 7.50 ( https://nmap.org/ncat )
Ncat: Connection refused.

The corresponding Cloudera Agent error looks like this:

[17/May/2020 13:34:53 +0000] 5306 Thread-13 https        ERROR    Failed to retrieve/store URL: https://cm-r01nn01.mws.mds.xyz:7183/cmf/par   cel/download/CDH-6.3.0-1.cdh6.3.0.p0.1279813-el7.parcel.torrent -> /opt/cloudera/parcel-cache/CDH-6.3.0-1.cdh6.3.0.p0.1279813-el7.parcel.to   rrent Traceback (most recent call last): 
  File "/opt/cloudera/cm-agent/lib/python2.7/site-packages/cmf/https.py", line 193, in fetch_to_file
    resp = self.open(req_url)
  File "/opt/cloudera/cm-agent/lib/python2.7/site-packages/cmf/https.py", line 188, in open
    return self.opener(url, *pargs, **kwargs)
  File "/usr/lib64/python2.7/urllib2.py", line 431, in open
    response = self._open(req, data)
  File "/usr/lib64/python2.7/urllib2.py", line 449, in _open
    '_open', req)
  File "/usr/lib64/python2.7/urllib2.py", line 409, in _call_chain
    result = func(*args)
  File "/opt/cloudera/cm-agent/lib/python2.7/site-packages/cmf/https.py", line 179, in https_open
    return self.do_open(opener, req)
  File "/usr/lib64/python2.7/urllib2.py", line 1214, in do_open
    raise URLError(err)
URLError: <urlopen error [Errno 111] Connection refused>

If you're using HAProxy for traffic routing to various instances, you'll need the following stanza in the HAProxy configuration file:

# CM GUI
listen cm
        bind cm-c01:80
        mode    http
        redirect scheme https if !{ ssl_fc }

frontend cmin
        bind    cm-c01:443 ssl crt /etc/haproxy/certs/cm-c01.mws.mds.xyz-haproxy.pem no-sslv3
        default_backend cmback

frontend cm7183in
        bind    cm-c01:7183 ssl crt /etc/haproxy/certs/cm-c01.mws.mds.xyz-haproxy.pem no-sslv3
        default_backend cmback

backend cmback
        mode http
        balance roundrobin

        server cm-r01nn01.mws.mds.xyz    cm-r01nn01.mws.mds.xyz:7183 ssl check verify none port 7183 inter 12000 rise 3 fall 3
        server cm-r01nn02.mws.mds.xyz    cm-r01nn02.mws.mds.xyz:7183 ssl check verify none port 7183 inter 12000 rise 3 fall 3

Otherwise the following port won't be open:

[root@cm-r01xs01 .ssh]# netstat -pnltu 
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name
tcp        0      0 192.168.0.120:7183       0.0.0.0:*               LISTEN      15001/haproxy
[root@cm-r01xs01 .ssh]# 

And you'll receive the subject error.  If you're routing traffic through your external facing router, ensure you have proper firewall rules configured for NAT traffic to and from the Azure instances:

iptables -t nat -I PREROUTING -s 103.192.131.145 -p tcp --dport 7183 -j DNAT --to 192.168.0.120:7183
iptables -I FORWARD -p tcp -d 192.168.0.53 --dport 7183 -j ACCEPT
iptables -t nat -I PREROUTING -s 103.192.131.145 -p udp --dport 7183 -j DNAT --to 192.168.0.120:7183
iptables -I FORWARD -p udp -d 192.168.0.53 --dport 7183 -j ACCEPT

Allowing traffic from said external IP 103.192.131.145.  Furthermore, your HAproxy or Cloudera servers also contain the valid port.  For instance:

[root@cm-r01xs01 .ssh]# cat /etc/firewalld/zones/public.xml |grep -Ei 7183
  <port protocol="tcp" port="7183"/>
  <port protocol="udp" port="7183"/>
[root@cm-r01xs01 .ssh]#

With that, you can expect the following result from your Azure instances.

[root@cm-awn01 ~]# nc -v 123.123.123.123 7183
Ncat: Version 7.50 ( https://nmap.org/ncat )
Ncat: Connected to 123.123.123.123:7183.

HTH,
BD

Importing Certificates Into the Truststore: Connection not protected

Getting this message?

Connection not protected

The security of your connection is reduced. Criminals can attempt to steal your data from the website. You are advised to leave this website.

And you know this is a trusted site, such as a local HTTPS web server?  Then export the certificate and add it to your trusted certificates stash by following these steps.

  1. Click on View Certificate
  2. From the panel that opens up, click on the Details tab.
  3. Next, click on Copy to File….
  4. Leave default settings.  In our case, it was DER encoded binary X.509 (. CER )
  5. Following that select the location and the name of the file to copy the certificate too.
  6. Next, start mmc.exe by searching for this program on your Windows 10 machine or executing it from the command line. 
  7. Click on File then  Add / Remove Snap In
  8. Click on Certificates followed by clicking the Add > button.
  9. Save or click Finish
  10. Next, navigate to Console Root -> Certificates (Local Computer) -> Trusted Root Certificatiom Authorities -> Certificates
  11. On the panel that opens up, right click Certificates and All Tasks -> Import
  12. Once imported, File -> Save and Exit.
  13. Reload your site. 

Steps 1,2,3,4,5:

https://i2.wp.com/www.microdevsys.com/WordPressImages/ImportTrustedCertificates-CertificateExportWizard.JPG?ssl=1

Steps 6, 7 and 8:

https://i2.wp.com/www.microdevsys.com/WordPressImages/ImportTrustedCertificates-MMC-Add-Certificates-Snap-In.JPG?ssl=1

Steps 10 and 11:

https://i2.wp.com/www.microdevsys.com/WordPressImages/ImportTrustedCertificates-MMC-Import-Certificate-Wizard.JPG?ssl=1

Regards,
AS

Connection not protected: Kaspersky Antivirus

Getting this?

Connection not protected

The security of your connection is reduced. Criminals can attempt to steal your data from the website. You are advised to leave this website.

For regular web traffic, this is normal. But if you want to avoid this for a local HTTPS server you're running, then these are the steps you want to take to disable this:

  1. In Kacpersky Total Security, click the Cog Wheel near the bottom left.
  2. Next, select Protection then Application Control from the right side. 
  3. Click on Manage Applications
  4. Type the browser you would like to modify in the top right search box.
  5. Double click the browser app and click the Exclusions tab
  6. Click on Do not scan all traffic 
  7. Select the checkbox Only for specific IP addresses
  8. Type in the addresses.
  9. Save the settings

Example of what this look like:

https://i2.wp.com/www.microdevsys.com/WordPressImages/KasperskyAntivitus-AllowSites.JPG?ssl=1

NOTE: Above screenshots shows a sample application.  It should really be Microsoft iExplorer, Firefox or Chrome if those browsers are being used. 

Thx,
EJ

This site is not secure

Getting this?

This site is not secure

This might mean that someone’s trying to fool you or steal any info you send to the server. You should close this site immediately

For regular web traffic, this is a valid warning message.  And you should be wary of sites that don't have valid certificates.  It's inadvisable to purchase anything from sites that don't conform to the expected security standards to keep your money safe.  Simply leave the site. 

However, when running a home-based HTTPS server, this can get annoying.  The site is known.  It's yours.  So why not just trust it and avoid the hassle. Despite IE giving the option to trust a set of sites by adding them to the trusted site's panel in internet options, that still will not work fully.  However, if you plan to use IE only for your local development work, then the steps below are fine. In this case, updates IE settings by:

  1. Click the IE Settings (Cog Wheel) at the top right corner of your IE browser.
  2. Select Internet Options
  3. Uncheck: Warn about certificate address mismatch
  4. Uncheck: Check for publisher's certificate revocation
  5. Uncheck: Check for server certificate revocation*
  6. Uncheck: Check for signatures on downloaded programs

Apply and save the settings.  Restart Internet Explorer.  The warning should now go away. 

Cheers,
TK


     
  Copyright © 2003 - 2013 Tom Kacperski (microdevsys.com). All rights reserved.

Creative Commons License
This work is licensed under a Creative Commons Attribution 3.0 Unported License