Header Shadow Image

Cloudera: Ncat: Connection refused.

Getting the following connecting from your Azure instance back to your Cloudera CM + CDH Cluster?

[root@cm-awn01 ~]# nc -v 7183
Ncat: Version 7.50 ( https://nmap.org/ncat )
Ncat: Connection refused.

The corresponding Cloudera Agent error looks like this:

[17/May/2020 13:34:53 +0000] 5306 Thread-13 https        ERROR    Failed to retrieve/store URL: https://cm-r01nn01.mws.mds.xyz:7183/cmf/par   cel/download/CDH-6.3.0-1.cdh6.3.0.p0.1279813-el7.parcel.torrent -> /opt/cloudera/parcel-cache/CDH-6.3.0-1.cdh6.3.0.p0.1279813-el7.parcel.to   rrent Traceback (most recent call last): 
  File "/opt/cloudera/cm-agent/lib/python2.7/site-packages/cmf/https.py", line 193, in fetch_to_file
    resp = self.open(req_url)
  File "/opt/cloudera/cm-agent/lib/python2.7/site-packages/cmf/https.py", line 188, in open
    return self.opener(url, *pargs, **kwargs)
  File "/usr/lib64/python2.7/urllib2.py", line 431, in open
    response = self._open(req, data)
  File "/usr/lib64/python2.7/urllib2.py", line 449, in _open
    '_open', req)
  File "/usr/lib64/python2.7/urllib2.py", line 409, in _call_chain
    result = func(*args)
  File "/opt/cloudera/cm-agent/lib/python2.7/site-packages/cmf/https.py", line 179, in https_open
    return self.do_open(opener, req)
  File "/usr/lib64/python2.7/urllib2.py", line 1214, in do_open
    raise URLError(err)
URLError: <urlopen error [Errno 111] Connection refused>

If you're using HAProxy for traffic routing to various instances, you'll need the following stanza in the HAProxy configuration file:

listen cm
        bind cm-c01:80
        mode    http
        redirect scheme https if !{ ssl_fc }

frontend cmin
        bind    cm-c01:443 ssl crt /etc/haproxy/certs/cm-c01.mws.mds.xyz-haproxy.pem no-sslv3
        default_backend cmback

frontend cm7183in
        bind    cm-c01:7183 ssl crt /etc/haproxy/certs/cm-c01.mws.mds.xyz-haproxy.pem no-sslv3
        default_backend cmback

backend cmback
        mode http
        balance roundrobin

        server cm-r01nn01.mws.mds.xyz    cm-r01nn01.mws.mds.xyz:7183 ssl check verify none port 7183 inter 12000 rise 3 fall 3
        server cm-r01nn02.mws.mds.xyz    cm-r01nn02.mws.mds.xyz:7183 ssl check verify none port 7183 inter 12000 rise 3 fall 3

Otherwise the following port won't be open:

[root@cm-r01xs01 .ssh]# netstat -pnltu 
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name
tcp        0      0*               LISTEN      15001/haproxy
[root@cm-r01xs01 .ssh]# 

And you'll receive the subject error.  If you're routing traffic through your external facing router, ensure you have proper firewall rules configured for NAT traffic to and from the Azure instances:

iptables -t nat -I PREROUTING -s -p tcp --dport 7183 -j DNAT --to
iptables -I FORWARD -p tcp -d --dport 7183 -j ACCEPT
iptables -t nat -I PREROUTING -s -p udp --dport 7183 -j DNAT --to
iptables -I FORWARD -p udp -d --dport 7183 -j ACCEPT

Allowing traffic from said external IP  Furthermore, your HAproxy or Cloudera servers also contain the valid port.  For instance:

[root@cm-r01xs01 .ssh]# cat /etc/firewalld/zones/public.xml |grep -Ei 7183
  <port protocol="tcp" port="7183"/>
  <port protocol="udp" port="7183"/>
[root@cm-r01xs01 .ssh]#

With that, you can expect the following result from your Azure instances.

[root@cm-awn01 ~]# nc -v 7183
Ncat: Version 7.50 ( https://nmap.org/ncat )
Ncat: Connected to


  Copyright © 2003 - 2013 Tom Kacperski (microdevsys.com). All rights reserved.

Creative Commons License
This work is licensed under a Creative Commons Attribution 3.0 Unported License