Header Shadow Image


java.lang.ClassNotFoundException: org.cloudera.log4j.redactor.RedactorAppender

Running into this?

java.lang.ClassNotFoundException: org.cloudera.log4j.redactor.RedactorAppender

Solve it with this:

[root@awx01 ansible]# ansible 'cm*' -m shell -a 'cd /opt/cloudera/parcels/CDH/lib/oozie/libtools; ln -s ../../../jars/log4j-core-2.8.2.jar log4j-core-2.8.2.jar; ln -s ../../../jars/logredactor-2.0.7.jar  logredactor-2.0.7.jar'

Cheers,
TK

ERROR StatusLogger Log4j2 could not find a logging implementation. Please add log4j-core to the classpath. Using SimpleLogger to log to the console…

Running into this?

ERROR StatusLogger Log4j2 could not find a logging implementation. Please add log4j-core to the classpath. Using SimpleLogger to log to the console...

Solve it with this:

[root@awx01 ansible]# ansible 'cm*' -m shell -a 'cd /opt/cloudera/parcels/CDH/lib/oozie/libtools; ln -s ../../../jars/log4j-core-2.8.2.jar log4j-core-2.8.2.jar; ln -s ../../../jars/logredactor-2.0.7.jar  logredactor-2.0.7.jar'

Cheers,
TK

Allow multicast traffic with firewalld

The commands to allow multicast traffic with firewalld are:

firewall-cmd –permanent –direct –add-rule ipv4 filter INPUT 0 -m pkttype –pkt-type multicast -j ACCEPT

firewall-cmd –permanent –direct –add-rule ipv6 filter INPUT 0 -m pkttype –pkt-type multicast -j ACCEPT

Cheers,
TK

 

cloudera-scm-agent: SSLError: sslv3 alert bad certificate

Getting this?

[06/Jul/2019 19:49:28 +0000] 2245 MainThread agent        ERROR    Heartbeating to cm-r01nn01.mws.mds.xyz:7182 failed.
Traceback (most recent call last):
  File "/opt/cloudera/cm-agent/lib/python2.7/site-packages/cmf/agent.py", line 1387, in _send_heartbeat
    self.cfg.max_cert_depth)
  File "/opt/cloudera/cm-agent/lib/python2.7/site-packages/cmf/https.py", line 139, in __init__
    self.conn.connect()
  File "/opt/cloudera/cm-agent/lib/python2.7/site-packages/M2Crypto/httpslib.py", line 69, in connect
    sock.connect((self.host, self.port))
  File "/opt/cloudera/cm-agent/lib/python2.7/site-packages/M2Crypto/SSL/Connection.py", line 309, in connect
    ret = self.connect_ssl()
  File "/opt/cloudera/cm-agent/lib/python2.7/site-packages/M2Crypto/SSL/Connection.py", line 295, in connect_ssl
    return m2.ssl_connect(self.ssl, self._timeout)
SSLError: sslv3 alert bad certificate

Then set the proper certificate path in the /etc/cloudera-scm-agent/config.ini file:

[Security]
use_tls=1
max_cert_depth=9
verify_cert_dir=/opt/cloudera/security/pki/

At this point we got:

SSLError: certificate verify failed

which we'll write about in the next post.

Cheers,
TK

ERROR    Heartbeating to failed.

Getting this error when connecting cloudera-scm-agents to the server?

[06/Jul/2019 16:16:55 +0000] 23442 MainThread agent        ERROR    Heartbeating to srv-c01.mws.mds.xyz:17182 failed.
Traceback (most recent call last):
  File "/opt/cloudera/cm-agent/lib/python2.7/site-packages/cmf/agent.py", line 1387, in _send_heartbeat
    self.cfg.max_cert_depth)
  File "/opt/cloudera/cm-agent/lib/python2.7/site-packages/cmf/https.py", line 139, in __init__
    self.conn.connect()
  File "/opt/cloudera/cm-agent/lib/python2.7/site-packages/M2Crypto/httpslib.py", line 69, in connect
    sock.connect((self.host, self.port))
  File "/opt/cloudera/cm-agent/lib/python2.7/site-packages/M2Crypto/SSL/Connection.py", line 309, in connect
    ret = self.connect_ssl()
  File "/opt/cloudera/cm-agent/lib/python2.7/site-packages/M2Crypto/SSL/Connection.py", line 295, in connect_ssl
    return m2.ssl_connect(self.ssl, self._timeout)
SSLError: unknown protocol

Enable TLS encryption for agents:

Administration -> Settings -> Use TLS Authentication of Agents to Server

Cheers,
TK

Setting tune.ssl.default-dh-param to 1024 by default, if your workload permits it you should set it to at least 2048. Please set a value >= 1024 to make this warning disappear.

Getting this?

haproxy-systemd-wrapper[17526]: [WARNING] 186/151058 (17527) : Setting tune.ssl.default-dh-param to 1024 by default, if your workload permits it you should set it to at least 2048. Please set a value >= 1024 to make this warning disappear.

add this to your haproxy config:

global
        log                     127.0.0.1       local3  warning
        pidfile                 /var/run/haproxy.pid
        maxconn                 4000
        user                    haproxy
        group                   haproxy
        daemon                  haproxy

        stats socket            /etc/haproxy/stats
        tune.ssl.default-dh-param 2048

Cheers,
TK

Unexpected error. Unable to verify database connection. / Cannot find CDH’\”s bigtop-detect-javahome.

If you get these errors:

Unexpected error. Unable to verify database connection. 

Checking the logs, we see the following:


2019-06-24 23:42:18,402 INFO scm-web-118667:com.cloudera.enterprise.JavaMelodyFacade: Entering HTTP Operation: Method:POST, Path:/clusters/2/healthStatusBar
2019-06-24 23:42:18,404 INFO scm-web-119745:com.cloudera.enterprise.JavaMelodyFacade: Entering HTTP Operation: Method:POST, Path:/clusters/2/statusContent
2019-06-24 23:42:18,412 INFO scm-web-118667:com.cloudera.enterprise.JavaMelodyFacade: Exiting HTTP Operation: Method:POST, Path:/clusters/2/healthStatusBar, Status:200
2019-06-24 23:42:18,420 INFO scm-web-119745:com.cloudera.enterprise.JavaMelodyFacade: Exiting HTTP Operation: Method:POST, Path:/clusters/2/statusContent, Status:200
2019-06-24 23:42:19,481 INFO scm-web-119111:com.cloudera.enterprise.JavaMelodyFacade: Entering HTTP Operation: Method:POST, Path:/dbTestConn/testConnection
2019-06-24 23:42:19,489 INFO scm-web-119111:com.cloudera.cmf.service.ServiceHandlerRegistry: Executing host command HiveTestDatabaseConnection with sensitive arguments. Host: DbHost{id=28, hostId=a787204f-931d-47e0-8aef-534ab6355a7a, hostName=cm-r01en01.mws.mds.xyz}
2019-06-24 23:42:19,500 INFO scm-web-119111:com.cloudera.cmf.service.ServiceHandlerRegistry: Executing host command OozieTestDatabaseConnection with sensitive arguments. Host: DbHost{id=28, hostId=a787204f-931d-47e0-8aef-534ab6355a7a, hostName=cm-r01en01.mws.mds.xyz}
2019-06-24 23:42:19,506 INFO scm-web-119111:com.cloudera.cmf.service.ServiceHandlerRegistry: Executing host command HueTestDatabaseConnection with sensitive arguments. Host: DbHost{id=28, hostId=a787204f-931d-47e0-8aef-534ab6355a7a, hostName=cm-r01en01.mws.mds.xyz}
2019-06-24 23:42:19,809 INFO scm-web-119111:com.cloudera.enterprise.JavaMelodyFacade: Exiting HTTP Operation: Method:POST, Path:/dbTestConn/testConnection, Status:200
2019-06-24 23:42:19,826 INFO avro-servlet-hb-processor-1:com.cloudera.server.common.AgentAvroServlet: (45 skipped) AgentAvroServlet: heartbeat processing stats: average=12ms, min=8ms, max=79ms.
2019-06-24 23:42:21,847 INFO scm-web-120167:com.cloudera.enterprise.JavaMelodyFacade: Entering HTTP Operation: Method:POST, Path:/dbTestConn/checkConnectionResult
2019-06-24 23:42:21,854 INFO scm-web-120167:com.cloudera.enterprise.JavaMelodyFacade: Exiting HTTP Operation: Method:POST, Path:/dbTestConn/checkConnectionResult, Status:200
2019-06-24 23:42:23,890 INFO scm-web-120477:com.cloudera.enterprise.JavaMelodyFacade: Entering HTTP Operation: Method:POST, Path:/dbTestConn/checkConnectionResult
2019-06-24 23:42:23,896 INFO scm-web-120477:com.cloudera.enterprise.JavaMelodyFacade: Exiting HTTP Operation: Method:POST, Path:/dbTestConn/checkConnectionResult, Status:200
2019-06-24 23:42:24,373 WARN avro-servlet-hb-processor-0:com.cloudera.server.cmf.AgentProtocolImpl: Received Process Heartbeat for unknown (or duplicate) process. Ignoring. This is expected to happen once after old process eviction or process deletion (as happens in restarts). id=242 name=null host=a787204f-931d-47e0-8aef-534ab6355a7a/cm-r01en01.mws.mds.xyz
2019-06-24 23:42:24,374 WARN avro-servlet-hb-processor-0:com.cloudera.server.cmf.AgentProtocolImpl: Received Process Heartbeat for unknown (or duplicate) process. Ignoring. This is expected to happen once after old process eviction or process deletion (as happens in restarts). id=243 name=null host=a787204f-931d-47e0-8aef-534ab6355a7a/cm-r01en01.mws.mds.xyz
2019-06-24 23:42:24,374 WARN avro-servlet-hb-processor-0:com.cloudera.server.cmf.AgentProtocolImpl: Received Process Heartbeat for unknown (or duplicate) process. Ignoring. This is expected to happen once after old process eviction or process deletion (as happens in restarts). id=244 name=null host=a787204f-931d-47e0-8aef-534ab6355a7a/cm-r01en01.mws.mds.xyz
2019-06-24 23:42:25,935 INFO scm-web-119807:com.cloudera.enterprise.JavaMelodyFacade: Entering HTTP Operation: Method:POST, Path:/dbTestConn/checkConnectionResult
2019-06-24 23:42:25,942 INFO scm-web-119807:com.cloudera.enterprise.JavaMelodyFacade: Exiting HTTP Operation: Method:POST, Path:/dbTestConn/checkConnectionResult, Status:200
2019-06-24 23:42:28,049 INFO scm-web-119778:com.cloudera.enterprise.JavaMelodyFacade: Entering HTTP Operation: Method:POST, Path:/dbTestConn/checkConnectionResult
2019-06-24 23:42:28,056 INFO scm-web-119778:com.cloudera.enterprise.JavaMelodyFacade: Exiting HTTP Operation: Method:POST, Path:/dbTestConn/checkConnectionResult, Status:200
2019-06-24 23:42:29,383 INFO scm-web-120247:com.cloudera.enterprise.JavaMelodyFacade: Entering HTTP Operation: Method:POST, Path:/clusters/2/healthStatusBar
2019-06-24 23:42:29,394 INFO scm-web-120247:com.cloudera.enterprise.JavaMelodyFacade: Exiting HTTP Operation: Method:POST, Path:/clusters/2/healthStatusBar, Status:200
2019-06-24 23:42:29,395 INFO scm-web-120167:com.cloudera.enterprise.JavaMelodyFacade: Entering HTTP Operation: Method:POST, Path:/clusters/2/statusContent
2019-06-24 23:42:29,422 INFO scm-web-120167:com.cloudera.enterprise.JavaMelodyFacade: Exiting HTTP Operation: Method:POST, Path:/clusters/2/statusContent, Status:200
2019-06-24 23:42:30,084 INFO scm-web-118667:com.cloudera.enterprise.JavaMelodyFacade: Entering HTTP Operation: Method:POST, Path:/dbTestConn/checkConnectionResult
2019-06-24 23:42:30,091 INFO scm-web-118667:com.cloudera.enterprise.JavaMelodyFacade: Exiting HTTP Operation: Method:POST, Path:/dbTestConn/checkConnectionResult, Status:200
2019-06-24 23:42:32,124 INFO scm-web-120167:com.cloudera.enterprise.JavaMelodyFacade: Entering HTTP Operation: Method:POST, Path:/dbTestConn/checkConnectionResult
2019-06-24 23:42:32,131 INFO scm-web-120167:com.cloudera.enterprise.JavaMelodyFacade: Exiting HTTP Operation: Method:POST, Path:/dbTestConn/checkConnectionResult, Status:200
2019-06-24 23:42:33,205 INFO CommandPusher:com.cloudera.cmf.service.AbstractOneOffHostCommand: Unsuccessful 'HueTestDatabaseConnection'
2019-06-24 23:42:33,207 INFO CommandPusher:com.cloudera.cmf.service.AbstractDbConnectionTestCommand: Command exited with code: 1
2019-06-24 23:42:33,207 INFO CommandPusher:com.cloudera.cmf.service.AbstractDbConnectionTestCommand: [24/Jun/2019 23:42:28 +0000] 30846 MainThread redactor     INFO     Started launcher: /opt/cloudera/cm-agent/service/hue/hue.sh is_db_alive
[24/Jun/2019 23:42:28 +0000] 30846 MainThread redactor     ERROR    Redaction rules file doesn't exist, not redacting logs. file: redaction-rules.json, directory: /run/cloudera-scm-agent/process/247-HUE-test-db-connection
[24/Jun/2019 23:42:28 +0000] 30846 MainThread redactor     INFO     Re-exec watcher: /opt/cloudera/cm-agent/bin/cm proc_watcher 30867
+ date
+ date
Mon Jun 24 23:42:30 EDT 2019
++ dirname /opt/cloudera/cm-agent/service/hue/hue.sh
+ cloudera_config=/opt/cloudera/cm-agent/service/hue
++ cd /opt/cloudera/cm-agent/service/hue/../common
++ pwd
+ cloudera_config=/opt/cloudera/cm-agent/service/common
+ . /opt/cloudera/cm-agent/service/common/cloudera-config.sh
++ : /opt/cloudera/cm
++ export CLOUDERA_DIR
++ set -x
+ source_parcel_environment
+ '[‘ ‘!’ -z ” ‘]'
+ locate_cdh_java_home
+ '[‘ -z ” ‘]'
+ '[‘ -z /usr/libexec/bigtop-utils ‘]'
+ local BIGTOP_DETECT_JAVAHOME=
+ for candidate in '"${JSVC_HOME}"' '"${JSVC_HOME}/.."' '"/usr/lib/bigtop-utils"' '"/usr/libexec"'
+ '[‘ -e /usr/libexec/bigtop-utils/bigtop-detect-javahome ‘]'
+ for candidate in '"${JSVC_HOME}"' '"${JSVC_HOME}/.."' '"/usr/lib/bigtop-utils"' '"/usr/libexec"'
+ '[‘ -e /usr/libexec/bigtop-utils/../bigtop-detect-javahome ‘]'
+ for candidate in '"${JSVC_HOME}"' '"${JSVC_HOME}/.."' '"/usr/lib/bigtop-utils"' '"/usr/libexec"'
+ '[‘ -e /usr/lib/bigtop-utils/bigtop-detect-javahome ‘]'
+ for candidate in '"${JSVC_HOME}"' '"${JSVC_HOME}/.."' '"/usr/lib/bigtop-utils"' '"/usr/libexec"'
+ '[‘ -e /usr/libexec/bigtop-detect-javahome ‘]'
+ '[‘ -z ” ‘]'
+ echo 'Cannot find CDH'\''s bigtop-detect-javahome.'
+ exit 1

2019-06-24 23:42:33,207 ERROR CommandPusher:com.cloudera.cmf.model.DbCommand: Command 742(HueTestDatabaseConnection) has completed. finalstate:FINISHED, success:false, msg:Unexpected error. Unable to verify database connection.
2019-06-24 23:42:34,168 INFO scm-web-119807:com.cloudera.enterprise.JavaMelodyFacade: Entering HTTP Operation: Method:POST, Path:/dbTestConn/checkConnectionResult
2019-06-24 23:42:34,175 INFO scm-web-119807:com.cloudera.enterprise.JavaMelodyFacade: Exiting HTTP Operation: Method:POST, Path:/dbTestConn/checkConnectionResult, Status:200
2019-06-24 23:42:36,228 INFO scm-web-120167:com.cloudera.enterprise.JavaMelodyFacade: Entering HTTP Operation: Method:POST, Path:/dbTestConn/checkConnectionResult
2019-06-24 23:42:36,234 INFO scm-web-120167:com.cloudera.enterprise.JavaMelodyFacade: Exiting HTTP Operation: Method:POST, Path:/dbTestConn/checkConnectionResult, Status:200
2019-06-24 23:42:37,481 INFO CommandPusher:com.cloudera.cmf.service.AbstractOneOffHostCommand: Successful 'OozieTestDatabaseConnection'
2019-06-24 23:42:37,483 INFO CommandPusher:com.cloudera.cmf.service.AbstractDbConnectionTestCommand: Command exited with code: 0
2019-06-24 23:42:37,483 INFO CommandPusher:com.cloudera.cmf.service.AbstractDbConnectionTestCommand: + local MISCJAVA_HOME_CANDIDATES
+ case ${BIGTOP_JAVA_MAJOR} in
+ JAVA_HOME_CANDIDATES=(${JAVA8_HOME_CANDIDATES[@]} ${MISCJAVA_HOME_CANDIDATES[@]} ${OPENJAVA8_HOME_CANDIDATES[@]} ${JAVA11_HOME_CANDIDATES[@]} ${OPENJAVA11_HOME_CANDIDATES[@]})
+ '[‘ -z ” ‘]'
+ for candidate_regex in '${JAVA_HOME_CANDIDATES[@]}'
++ ls -rvd /usr/java/jdk1.8.0_181-cloudera
+ for candidate in '`ls -rvd ${candidate_regex}* 2>/dev/null`'
+ '[‘ -e /usr/java/jdk1.8.0_181-cloudera/bin/java ‘]'
+ export JAVA_HOME=/usr/java/jdk1.8.0_181-cloudera
+ JAVA_HOME=/usr/java/jdk1.8.0_181-cloudera
+ break 2
+ verify_java_home
+ '[‘ -z /usr/java/jdk1.8.0_181-cloudera ‘]'
+ echo JAVA_HOME=/usr/java/jdk1.8.0_181-cloudera
+ source_parcel_environment
+ '[‘ ‘!’ -z ” ‘]'
+ JAVA=/usr/java/jdk1.8.0_181-cloudera/bin/java
+ [[ -z /opt/cloudera/cm ]]
+ MGMT_CLASSPATH='/opt/cloudera/cm/lib/*'
+ JDBC_JARS=/usr/share/java/mysql-connector-java.jar:/opt/cloudera/cm/lib/postgresql-42.1.4.jre7.jar:/usr/share/java/oracle-connector-java.jar
+ MGMT_CLASSPATH='/usr/share/java/mysql-connector-java.jar:/opt/cloudera/cm/lib/postgresql-42.1.4.jre7.jar:/usr/share/java/oracle-connector-java.jar:/opt/cloudera/cm/lib/*'
++ pwd
+ MGMT_CLASSPATH='/run/cloudera-scm-agent/process/246-OOZIE.OOZIE_SERVER-test-db-connection:/usr/share/java/mysql-connector-java.jar:/opt/cloudera/cm/lib/postgresql-42.1.4.jre7.jar:/usr/share/java/oracle-connector-java.jar:/opt/cloudera/cm/lib/*'
+ MGMT_JAVA_OPTS='-Djava.net.preferIPv4Stack=true '
+ exec /usr/java/jdk1.8.0_181-cloudera/bin/java -Djava.net.preferIPv4Stack=true -Djava.security.egd=file:///dev/urandom -cp '/run/cloudera-scm-agent/process/246-OOZIE.OOZIE_SERVER-test-db-connection:/usr/share/java/mysql-connector-java.jar:/opt/cloudera/cm/lib/postgresql-42.1.4.jre7.jar:/usr/share/java/oracle-connector-java.jar:/opt/cloudera/cm/lib/*' com.cloudera.enterprise.dbutil.DbCommandExecutor db.properties

2019-06-24 23:42:37,483 INFO CommandPusher:com.cloudera.cmf.model.DbCommand: Command 741(OozieTestDatabaseConnection) has completed. finalstate:FINISHED, success:true, msg:Successful
2019-06-24 23:42:37,497 INFO CommandPusher:com.cloudera.cmf.service.AbstractOneOffHostCommand: Successful 'HiveTestDatabaseConnection'
2019-06-24 23:42:37,498 INFO CommandPusher:com.cloudera.cmf.service.AbstractDbConnectionTestCommand: Command exited with code: 0
2019-06-24 23:42:37,498 INFO CommandPusher:com.cloudera.cmf.service.AbstractDbConnectionTestCommand: + MISCJAVA_HOME_CANDIDATES=('/Library/Java/Home' '/usr/java/default' '/usr/lib/jvm/default-java' '/usr/lib/jvm/java-openjdk' '/usr/lib/jvm/jre-openjdk')
+ local MISCJAVA_HOME_CANDIDATES
+ case ${BIGTOP_JAVA_MAJOR} in
+ JAVA_HOME_CANDIDATES=(${JAVA8_HOME_CANDIDATES[@]} ${MISCJAVA_HOME_CANDIDATES[@]} ${OPENJAVA8_HOME_CANDIDATES[@]} ${JAVA11_HOME_CANDIDATES[@]} ${OPENJAVA11_HOME_CANDIDATES[@]})
+ '[‘ -z ” ‘]'
+ for candidate_regex in '${JAVA_HOME_CANDIDATES[@]}'
++ ls -rvd /usr/java/jdk1.8.0_181-cloudera
+ for candidate in '`ls -rvd ${candidate_regex}* 2>/dev/null`'
+ '[‘ -e /usr/java/jdk1.8.0_181-cloudera/bin/java ‘]'
+ export JAVA_HOME=/usr/java/jdk1.8.0_181-cloudera
+ JAVA_HOME=/usr/java/jdk1.8.0_181-cloudera
+ break 2
+ verify_java_home
+ '[‘ -z /usr/java/jdk1.8.0_181-cloudera ‘]'
+ echo JAVA_HOME=/usr/java/jdk1.8.0_181-cloudera
+ source_parcel_environment
+ '[‘ ‘!’ -z ” ‘]'
+ JAVA=/usr/java/jdk1.8.0_181-cloudera/bin/java
+ [[ -z /opt/cloudera/cm ]]
+ MGMT_CLASSPATH='/opt/cloudera/cm/lib/*'
+ JDBC_JARS=/usr/share/java/mysql-connector-java.jar:/opt/cloudera/cm/lib/postgresql-42.1.4.jre7.jar:/usr/share/java/oracle-connector-java.jar
+ MGMT_CLASSPATH='/usr/share/java/mysql-connector-java.jar:/opt/cloudera/cm/lib/postgresql-42.1.4.jre7.jar:/usr/share/java/oracle-connector-java.jar:/opt/cloudera/cm/lib/*'
++ pwd
+ MGMT_CLASSPATH='/run/cloudera-scm-agent/process/245-HIVE-test-db-connection:/usr/share/java/mysql-connector-java.jar:/opt/cloudera/cm/lib/postgresql-42.1.4.jre7.jar:/usr/share/java/oracle-connector-java.jar:/opt/cloudera/cm/lib/*'
+ MGMT_JAVA_OPTS='-Djava.net.preferIPv4Stack=true '
+ exec /usr/java/jdk1.8.0_181-cloudera/bin/java -Djava.net.preferIPv4Stack=true -Djava.security.egd=file:///dev/urandom -cp '/run/cloudera-scm-agent/process/245-HIVE-test-db-connection:/usr/share/java/mysql-connector-java.jar:/opt/cloudera/cm/lib/postgresql-42.1.4.jre7.jar:/usr/share/java/oracle-connector-java.jar:/opt/cloudera/cm/lib/*' com.cloudera.enterprise.dbutil.DbCommandExecutor db.properties

2019-06-24 23:42:37,498 INFO CommandPusher:com.cloudera.cmf.model.DbCommand: Command 740(HiveTestDatabaseConnection) has completed. finalstate:FINISHED, success:true, msg:Successful
2019-06-24 23:42:38,336 INFO scm-web-118667:com.cloudera.enterprise.JavaMelodyFacade: Entering HTTP Operation: Method:POST, Path:/dbTestConn/checkConnectionResult
2019-06-24 23:42:38,351 INFO scm-web-118667:com.cloudera.enterprise.JavaMelodyFacade: Exiting HTTP Operation: Method:POST, Path:/dbTestConn/checkConnectionResult, Status:200
2019-06-24 23:42:40,393 INFO scm-web-120237:com.cloudera.enterprise.JavaMelodyFacade: Entering HTTP Operation: Method:POST, Path:/clusters/2/healthStatusBar
2019-06-24 23:42:40,418 INFO scm-web-119778:com.cloudera.enterprise.JavaMelodyFacade: Entering HTTP Operation: Method:POST, Path:/clusters/2/statusContent
2019-06-24 23:42:40,421 INFO scm-web-120237:com.cloudera.enterprise.JavaMelodyFacade: Exiting HTTP Operation: Method:POST, Path:/clusters/2/healthStatusBar, Status:200
2019-06-24 23:42:40,435 INFO scm-web-119778:com.cloudera.enterprise.JavaMelodyFacade: Exiting HTTP Operation: Method:POST, Path:/clusters/2/statusContent, Status:200
^C
[root@cm-r01nn01 ~]#

Verify you have the right bigtop packages installed:

[root@cm-r01nn01 cloudera-scm-agent]#
[root@cm-r01nn01 cloudera-scm-agent]#
[root@cm-r01nn01 cloudera-scm-agent]# ls -altri /usr/libexec/bigtop-detect-javahome
ls: cannot access /usr/libexec/bigtop-detect-javahome: No such file or directory
[root@cm-r01nn01 cloudera-scm-agent]# find / -iname bigtop-detect-javahome*
[root@cm-r01nn01 cloudera-scm-agent]# ssh cm-r01nn02
Password:
Last login: Tue Jun 25 23:21:59 2019 from cm-r01nn01.mws.mds.xyz
[root@cm-r01nn02 ~]#
[root@cm-r01nn02 ~]#
[root@cm-r01nn02 ~]#
[root@cm-r01nn02 ~]# find / -iname bigtop-detect-javahome*
/usr/bin/bigtop-detect-javahome
/usr/lib/bigtop-utils/bigtop-detect-javahome
[root@cm-r01nn02 ~]# rpm -qf /usr/bin/bigtop-detect-javahome
rpm -qf bigtop-utils-0.7.0+cdh6.2.0-967373.el7.noarch
[root@cm-r01nn02 ~]# rpm -qf /usr/lib/bigtop-utils/bigtop-detect-javahome
bigtop-utils-0.7.0+cdh6.2.0-967373.el7.noarch
[root@cm-r01nn02 ~]#

 

Secondly, ensure that you have set the JAVA_HOME in the Hosts Configuration of CM to point to the correct JDK home variable.  This ansible code can help create the link to the latest copy of the JDK:

[root@awx01 ansible]# cd /ansible && ansible 'cm*' -m shell -a 'ls -altrid /usr/java/jdk1.8.0_181-cloudera/; ln -s /usr/java/jdk1.8.0_181-cloudera/ /usr/java/latest; ls -altrid /usr/java/latest'

Then use /usr/java/latest in the Hosts Configuration (Look for JAVA_HOME and set it).

Thx,
TK

ERROR: cannot execute CREATE TABLE in a read-only transaction

If you're getting the following on your patroni cluster:

Jun 24 23:30:50 psql02 etcd: failed to send out heartbeat on time (exceeded the 250ms timeout for 1.229927109s)
Jun 24 23:30:50 psql02 etcd: server is likely overloaded
Jun 24 23:30:50 psql02 etcd: failed to send out heartbeat on time (exceeded the 250ms timeout for 1.22987947s)
Jun 24 23:30:50 psql02 etcd: server is likely overloaded

ERROR:  cannot execute CREATE TABLE in a read-only transaction

Then you likely commented out these:

[root@psql03 ~]# cat /etc/haproxy/haproxy.cfg
global
    log         127.0.0.1 local0
    stats       socket /var/run/haproxy.sock mode 0600 level admin
    user        haproxy
    group       haproxy
    daemon
    debug
    maxconn 1024

defaults
    mode tcp
    log global
    option                  dontlognull
    option                  redispatch
    retries 2
    timeout client 30m
    timeout connect 4s
    timeout server 30m
    timeout check 5s

frontend psql-in
    mode tcp
    bind psql-c01:5432
    option tcplog
    default_backend             psql-back


backend psql-back
    mode        tcp
    option      tcplog
#    option     httpchk
#    http-check         expect status 200

    default-server inter 3s fall 3 rise 2 on-marked-down shutdown-se
    server      psql01.nix.mds.xyz    psql01.nix.mds.xyz:5432 maxconn 1024 check port 8008
    server      psql02.nix.mds.xyz    psql02.nix.mds.xyz:5432 maxconn 1024 check port 8008
    server      psql03.nix.mds.xyz    psql03.nix.mds.xyz:5432 maxconn 1024 check port 8008

listen stats
    bind :9000
    mode http
    stats enable
    stats hide-version
    stats realm Haproxy\ Statistics
    stats uri /haproxy-stats
    stats auth admin:nopassword
[root@psql03 ~]#

Uncomment the two lines above to resolve the issue.

Thx,
TK

 

User cannot run DDL statements on the specified database. Attempt to create and drop a table failed.

In case you receive the following:


cloudera-scm-server.log:2019-06-21 01:16:57,763 ERROR CommandPusher:com.cloudera.cmf.model.DbCommand: Command 609(HiveTestDatabaseConnection) has completedFINISHED, success:false, msg:User cannot run DDL statements on the specified database. Attempt to create and drop a table failed.

check the logs or test creating a temporary table:

cloudera-scm-server.log:Caused by: org.postgresql.util.PSQLException: ERROR: cannot execute UPDATE in a read-only transaction
cloudera-scm-server.log:Caused by: java.sql.BatchUpdateException: Batch entry 0 update COMMANDS_DETAIL set OPTIMISTIC_LOCK_VERSION=4, INTERNAL_STATE=? where COMMAND_ID=618 and OPTIMISTIC_LOCK_VERSION=3 was aborted: ERROR: cannot execute UPDATE in a read-only transaction  Call getNextException to see other errors in the batch.
cloudera-scm-server.log:Caused by: org.postgresql.util.PSQLException: ERROR: cannot execute UPDATE in a read-only transaction
cloudera-scm-server.log:Caused by: java.sql.BatchUpdateException: Batch entry 0 update COMMANDS_DETAIL set OPTIMISTIC_LOCK_VERSION=6, INTERNAL_STATE=? where COMMAND_ID=618 and OPTIMISTIC_LOCK_VERSION=5 was aborted: ERROR: cannot execute UPDATE in a read-only transaction  Call getNextException to see other errors in the batch.
cloudera-scm-server.log:Caused by: org.postgresql.util.PSQLException: ERROR: cannot execute UPDATE in a read-only transaction
[root@cm-r01nn01 cloudera-scm-server]#

Try to test create a table:

hue_mws01=>
hue_mws01=> CREATE TABLE account(
   user_id serial PRIMARY KEY,
   username VARCHAR (50) UNIQUE NOT NULL,
   password VARCHAR (50) NOT NULL,
   email VARCHAR (355) UNIQUE NOT NULL,
   created_on TIMESTAMP NOT NULL,
   last_login TIMESTAMP
);
ERROR:  cannot execute CREATE TABLE in a read-only transaction
hue_mws01=>

In our case the above was resolved by restarting haproxy, keepalived and Patroni on psql02 (Node 2) of our PostgreSQL cluster.  We then tested and received:

[root@psql01 ~]# sudo su – postgres
Last login: Sat Jun 22 00:16:39 EDT 2019 on pts/0
-bash-4.2$ psql -h psql-c01.nix.mds.xyz -p 5432 -W -U hue_mws01 -d hue_mws01
Password for user hue_mws01:
psql (10.5)
Type "help" for help.

hue_mws01=> CREATE TABLE account(
   user_id serial PRIMARY KEY,
   username VARCHAR (50) UNIQUE NOT NULL,
   password VARCHAR (50) NOT NULL,
   email VARCHAR (355) UNIQUE NOT NULL,
   created_on TIMESTAMP NOT NULL,
   last_login TIMESTAMP
);
CREATE TABLE
hue_mws01=>

Tell us if this solved your issue.

Cheers,
TK

 

Cannot read XML: (41) Specification mandate value for attribute data-pjax-transient [Line: 38 | Column: 40].

When you get this:

Cannot read XML: (41) Specification mandate value for attribute data-pjax-transient [Line: 38 | Column: 40].

ensure you're download the xml or any file from github using the clone or download a zip option, instead of Save As / Save Link As ….

Thx,
TK


     
  Copyright © 2003 - 2013 Tom Kacperski (microdevsys.com). All rights reserved.

Creative Commons License
This work is licensed under a Creative Commons Attribution 3.0 Unported License