Getting this error from Ansible?

ERROR! Extraneous options or arguments

Simply drop down one folder where the infra file resides and reexecute:

[root@awx01 adhoc]# cd ..
[root@awx01 ansible]# ansible cm* -i infra -m copy -a 'src=adhoc/public.xml dest=/etc/firewalld/zones/public.xml'

This was just a fancy way of ansible telling you it can't find the infra file containing the hosts for your deployment.

Cheers,
TK

Before enabling kerberos, the HDFS user is the only one that can list out files owned by itself:

[root@cm-r01en01 run]# hdfs dfs -ls /tmp/.cloudera_health_monitoring_canary_files/.canary_file_2019_08_20-19_28_02.abab681cca56293als: Permission denied: user=root, access=EXECUTE, inode="/tmp/.cloudera_health_monitoring_canary_files":hdfs:supergroup:d———

You'll need to run the command via SUDO like this:

[root@cm-r01en01 run]# sudo su -c "hdfs dfs -ls /tmp/.cloudera_health_monitoring_canary_files/.canary_file_2019_08_20-19_28_02.abab681cca56293a" -s /bin/bash hdfs
-rw-rw-rw-   3 hdfs supergroup          0 2019-08-20 20:28 /tmp/.cloudera_health_monitoring_canary_files/.canary_file_2019_08_20-19_28_02.abab681cca56293a

Cheers,
TK

Seeing slow SSD performance through SATA 3.0?  Tired of transfer rates of 30MB/s when you should be seeing at least 150MB/s? 

Fix this by adding the following kernel parameters:

libata.noacpi=1

Cheers,
TK

Getting this?

FATAL: Module scsi_wait_scan not found.

This is likely followed by a kernel panic.  Look for a non existing mount point being defined, such as a LVM_swap partition that doesn't exist. Remove it.  Also remove any non existent partitions in /etc/fstab through the recovery console.

Cheers,
TK

Getting this?

SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
ERROR StatusLogger No log4j2 configuration file found. Using default configuration: logging only errors to the console. Set system property 'org.apache.logging.log4j.simplelog.StatusLogger.level' to TRACE to show Log4j2 internal initialization logging.

Investigate by first running the command manually:

[oozie@cm-r01en01 ~]$ /usr/java/latest/bin/java -Xms52428800 -Xmx52428800 -XX:+UseG1GC -XX:MaxGCPauseMillis=20 -XX:InitiatingHeapOccupancyPercent=35 -XX:G1HeapRegionSize=16M -XX:MinMetaspaceFreeRatio=50 -XX:MaxMetaspaceFreeRatio=80 -XX:+DisableExplicitGC -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/tmp/oozie_oozie-OOZIE_SERVER-901d5713a53510380392378fa81b483d_pid1397.hprof -XX:OnOutOfMemoryError=/opt/cloudera/cm-agent/service/common/killparent.sh -Doozie.home.dir=/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie -Doozie.config.dir=/run/cloudera-scm-agent/process/216-oozie-OOZIE-SERVER-upload-sharelib -Doozie.log.dir=/var/log/oozie -Doozie.log.file=oozie-cmf-oozie-OOZIE_SERVER-cm-r01en01.mws.mds.xyz.log.out -Doozie.config.file=oozie-site.xml -Doozie.log4j.file=log4j.properties -Doozie.log4j.reload=10 -Doozie.http.hostname=cm-r01en01.mws.mds.xyz -Doozie.http.port=11000 -Djava.net.preferIPv4Stack=true -Doozie.admin.port= -Dderby.stream.error.file=/var/log/oozie/derby.log -Doozie.instance.id=cm-r01en01.mws.mds.xyz -Djava.library.path=/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/hadoop/lib/native -cp ':/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/accessors-smart-1.2.jar: < VERY VERY LONG COMMAND OF JAR FILES> :/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libext/*.jar' org.apache.oozie.tools.OozieSharelibCLI create -fs hdfs://cm-r01nn02.mws.mds.xyz:8020 -locallib /opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/oozie-sharelib-yarn -concurrency 8
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/jars/log4j-slf4j-impl-2.8.2.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/jars/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/jars/slf4j-simple-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
ERROR StatusLogger No log4j2 configuration file found. Using default configuration: logging only errors to the console. Set system property 'org.apache.logging.log4j.simplelog.StatusLogger.level' to TRACE to show Log4j2 internal initialization logging.
Found Hadoop that supports Erasure Coding. Trying to disable Erasure Coding for path: /user/oozie/share/lib
Done
the destination path for sharelib is: /user/oozie/share/lib/lib_20190817204923
Running 1738 copy tasks on 8 threads
Copy tasks are done
[oozie@cm-r01en01 ~]$
 

Notice how the command completes successfully above but doesn't through the CM UI.

Next execute the same task from the CM UI and observe the space on HDFS:

[root@cm-r01en02 CDH]# hdfs dfs -du -s -h /user/oozie/share/lib/*
1.4 G    4.1 G  /user/oozie/share/lib/lib_20190813004211
1.4 G    4.1 G  /user/oozie/share/lib/lib_20190813011235
1.4 G    4.1 G  /user/oozie/share/lib/lib_20190813074412
1.4 G    4.1 G  /user/oozie/share/lib/lib_20190814001323
1.4 G    4.1 G  /user/oozie/share/lib/lib_20190814003111
1.4 G    4.1 G  /user/oozie/share/lib/lib_20190814222128
1.4 G    4.1 G  /user/oozie/share/lib/lib_20190814223531
1.4 G    4.1 G  /user/oozie/share/lib/lib_20190814224153
1.4 G    4.1 G  /user/oozie/share/lib/lib_20190814230015
1.4 G    4.1 G  /user/oozie/share/lib/lib_20190814231607
1.4 G    4.1 G  /user/oozie/share/lib/lib_20190814232614
1.4 G    4.1 G  /user/oozie/share/lib/lib_20190814233305
1.4 G    4.1 G  /user/oozie/share/lib/lib_20190815234730
1.4 G    4.1 G  /user/oozie/share/lib/lib_20190816173425
1.4 G    4.1 G  /user/oozie/share/lib/lib_20190816230157
1.4 G    4.1 G  /user/oozie/share/lib/lib_20190816232108
1.4 G    4.1 G  /user/oozie/share/lib/lib_20190816232800
1.4 G    4.1 G  /user/oozie/share/lib/lib_20190817092413
1.4 G    4.1 G  /user/oozie/share/lib/lib_20190817140917
1.4 G    4.1 G  /user/oozie/share/lib/lib_20190817192300
1.4 G    4.1 G  /user/oozie/share/lib/lib_20190817193419
1.4 G    4.1 G  /user/oozie/share/lib/lib_20190817202812
1.4 G    4.1 G  /user/oozie/share/lib/lib_20190817204018
1.4 G    4.1 G  /user/oozie/share/lib/lib_20190817204923
998.0 M  3.0 G  /user/oozie/share/lib/lib_20190817205610
[root@cm-r01en02 CDH]#

Notice the task in the CM UI ends but the command continues to copy into the folder.  This indicates that timeouts have been exhausted leading up to your failure.  This also indicates the storage you are using for the installation is rather slow.  Cloudera doesn't appear to indicate this explicitly in the log files. 

We need to set the timeout.  Browse to Oozie -> Configuration and set a greater timeout then the default of 270.

Oozie Upload ShareLib Command Timeout

Oozie (Service-Wide) 

The timeout in seconds used for the Oozie Upload ShareLib command. When the value is zero, there is no timeout for the command.

Change 270 to a higher value and retry the operation.

Cheers,
TK

Running into this?

java.lang.ClassNotFoundException: org.cloudera.log4j.redactor.RedactorAppender

Solve it with this:

[root@awx01 ansible]# ansible 'cm*' -m shell -a 'cd /opt/cloudera/parcels/CDH/lib/oozie/libtools; ln -s ../../../jars/log4j-core-2.8.2.jar log4j-core-2.8.2.jar; ln -s ../../../jars/logredactor-2.0.7.jar  logredactor-2.0.7.jar'

Cheers,
TK

Running into this?

ERROR StatusLogger Log4j2 could not find a logging implementation. Please add log4j-core to the classpath. Using SimpleLogger to log to the console...

Solve it with this:

[root@awx01 ansible]# ansible 'cm*' -m shell -a 'cd /opt/cloudera/parcels/CDH/lib/oozie/libtools; ln -s ../../../jars/log4j-core-2.8.2.jar log4j-core-2.8.2.jar; ln -s ../../../jars/logredactor-2.0.7.jar  logredactor-2.0.7.jar'

Cheers,
TK

The commands to allow multicast traffic with firewalld are:

firewall-cmd –permanent –direct –add-rule ipv4 filter INPUT 0 -m pkttype –pkt-type multicast -j ACCEPT

firewall-cmd –permanent –direct –add-rule ipv6 filter INPUT 0 -m pkttype –pkt-type multicast -j ACCEPT

Cheers,
TK

Getting this?

[06/Jul/2019 19:49:28 +0000] 2245 MainThread agent        ERROR    Heartbeating to cm-r01nn01.mws.mds.xyz:7182 failed.
Traceback (most recent call last):
  File "/opt/cloudera/cm-agent/lib/python2.7/site-packages/cmf/agent.py", line 1387, in _send_heartbeat
    self.cfg.max_cert_depth)
  File "/opt/cloudera/cm-agent/lib/python2.7/site-packages/cmf/https.py", line 139, in __init__
    self.conn.connect()
  File "/opt/cloudera/cm-agent/lib/python2.7/site-packages/M2Crypto/httpslib.py", line 69, in connect
    sock.connect((self.host, self.port))
  File "/opt/cloudera/cm-agent/lib/python2.7/site-packages/M2Crypto/SSL/Connection.py", line 309, in connect
    ret = self.connect_ssl()
  File "/opt/cloudera/cm-agent/lib/python2.7/site-packages/M2Crypto/SSL/Connection.py", line 295, in connect_ssl
    return m2.ssl_connect(self.ssl, self._timeout)
SSLError: sslv3 alert bad certificate

Then set the proper certificate path in the /etc/cloudera-scm-agent/config.ini file:

[Security]
use_tls=1
max_cert_depth=9
verify_cert_dir=/opt/cloudera/security/pki/

At this point we got:

SSLError: certificate verify failed

which we'll write about in the next post.

Cheers,
TK

Getting this error when connecting cloudera-scm-agents to the server?

[06/Jul/2019 16:16:55 +0000] 23442 MainThread agent        ERROR    Heartbeating to srv-c01.mws.mds.xyz:17182 failed.
Traceback (most recent call last):
  File "/opt/cloudera/cm-agent/lib/python2.7/site-packages/cmf/agent.py", line 1387, in _send_heartbeat
    self.cfg.max_cert_depth)
  File "/opt/cloudera/cm-agent/lib/python2.7/site-packages/cmf/https.py", line 139, in __init__
    self.conn.connect()
  File "/opt/cloudera/cm-agent/lib/python2.7/site-packages/M2Crypto/httpslib.py", line 69, in connect
    sock.connect((self.host, self.port))
  File "/opt/cloudera/cm-agent/lib/python2.7/site-packages/M2Crypto/SSL/Connection.py", line 309, in connect
    ret = self.connect_ssl()
  File "/opt/cloudera/cm-agent/lib/python2.7/site-packages/M2Crypto/SSL/Connection.py", line 295, in connect_ssl
    return m2.ssl_connect(self.ssl, self._timeout)
SSLError: unknown protocol

Enable TLS encryption for agents:

Administration -> Settings -> Use TLS Authentication of Agents to Server

Cheers,
TK