Header Shadow Image


Archive for August, 2019

Required executor memory (1024), overhead (384 MB), and PySpark memory (0 MB) is above the max threshold (1024 MB) of this cluster!

Getting this? java.lang.IllegalArgumentException: Required executor memory (1024), overhead (384 MB), and PySpark memory (0 MB) is above the max threshold (1024 MB) of this cluster! Please check the values of 'yarn.scheduler.maximum-allocation-mb' and/or 'yarn.nodemanager.resource.memory-mb'. Either increase the Container Memory: yarn.nodemanager.resource.memory-mb to 2GB from 1GB Or reduce the maximum container memory: yarn.scheduler.maximum-allocation-mb from 1GB to 512MB in the […]

Cloudera Clusters: Running kinit on a kerberos principal returns with a 1969 date.

Are you getting this result? [root@cm-r01en01 process]# systemctl restart cloudera-scm-agent [root@cm-r01en01 process]# [root@cm-r01en01 process]# [root@cm-r01en01 process]# [root@cm-r01en01 process]# kinit -kt ./1401-hdfs-NFSGATEWAY/hdfs.keytab hdfs/cm-r01en01.mws.mds.xyz@MWS.MDS.XYZ [root@cm-r01en01 process]# klist Ticket cache: FILE:/tmp/krb5cc_0 Default principal: host/cm-r01en01.mws.mds.xyz@MWS.MDS.XYZ Valid starting       Expires              Service principal 12/31/1969 19:00:00  12/31/1969 19:00:00  Encrypted/Credentials/v1@X-GSSPROXY: [root@cm-r01en01 process]#   Solve it […]

org.apache.hadoop.security.AccessControlException: Client cannot authenticate via:[TOKEN, KERBEROS]

Getting this? [root@cm-r01en01 ~]# hdfs dfs -ls / 19/08/25 22:43:19 WARN ipc.Client: Exception encountered while connecting to the server : org.apache.hadoop.security.AccessControlException: Client cannot authenticate via:[TOKEN, KERBEROS] ls: Failed on local exception: java.io.IOException: org.apache.hadoop.security.AccessControlException: Client cannot authenticate via:[TOKEN, KERBEROS]; Host Details : local host is: "cm-r01en01.mws.mds.xyz/192.168.0.140"; destination host is: "cm-r01nn02.mws.mds.xyz":8020; [root@cm-r01en01 ~]# Fix it by doing […]

The readiness of the Impala Daemon to process queries is not known.

Getting this from the Impala Daemon? The readiness of the Impala Daemon to process queries is not known. Investigation reveals: Aug 25 09:16:39 cm-r01wn01 kernel: FINAL_REJECT: IN=eth0 OUT= MAC=00:50:56:86:7e:b7:00:50:56:86:79:2a:08:00 SRC=192.168.0.132 DST=192.168.0.160 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=6227 DF PROTO=TCP SPT=59318 DPT=23000 WINDOW=29200 RES=0x00 SYN URGP=0 Solve it by adding the port to the firewall configuration and […]

preauth (encrypted_timestamp) verify failure: Preauthentication failed

Getting this? Aug 25 08:18:51 idmipa03.mws.mds.xyz krb5kdc[6704](info): preauth (encrypted_timestamp) verify failure: Preauthentication failed Aug 25 08:18:51 idmipa03.mws.mds.xyz krb5kdc[6704](info): AS_REQ (4 etypes {18 17 16 23}) 192.168.0.140: PREAUTH_FAILED: hue/cm-r01en01.mws.mds.xyz@MWS.MDS.XYZ for krbtgt/MWS.MDS.XYZ@MWS.MDS.XYZ, Preauthentication failed Check the principal vs the keytab you're using for the same service: kadmin.local:  getprinc hue/cm-r01en01.mws.mds.xyz@MWS.MDS.XYZ Principal: hue/cm-r01en01.mws.mds.xyz@MWS.MDS.XYZ Expiration date: [never] Last password change: […]

ERROR    Timed out waiting for worker process collecting filesystem usage to complete.

Getting this error? ==> /var/log/cloudera-scm-agent/cloudera-scm-agent.log <== [24/Aug/2019 22:00:08 +0000] 3697 Monitor-HostMonitor throttling_logger ERROR    Timed out waiting for worker process collecting filesystem usage to complete. This may occur if the host has an NFS or other remote filesystem that is not responding to requests in a timely fashion. Current nodev filesystems: /dev/shm,/run,/sys/fs/cgroup,/run/cloudera-scm-agent/process,/run/cloudera-scm-agent/process,/run/user/155601104,/n/mds.xyz,/run/user/0 [24/Aug/2019 22:00:08 +0000] […]

ERROR:desktop.kt_renewer:Couldn’t renew kerberos ticket in order to work around Kerberos 1.8.1 issue. Please check that the ticket for ‘‘ is still renewable:

Getting this? INFO:desktop.kt_renewer:Renewing kerberos ticket to work around kerberos 1.8.1: /usr/bin/kinit -R -c /var/run/hue/hue_krb5_ccache kinit: KDC can't fulfill requested option while renewing credentials ERROR:desktop.kt_renewer:Couldn't renew kerberos ticket in order to work around Kerberos 1.8.1 issue. Please check that the ticket for 'hue/cm-r01en01.mws.mds.xyz@MWS.MDS.XYZ' is still renewable: $ klist -f -c /var/run/hue/hue_krb5_ccache If the 'renew until' date […]

kadmin: Communication failure with server while initializing kadmin interface

Getting this? -bash-4.2$ kadmin -k -t /tmp/kadmin/cmf8392129202434993503.keytab -p cmadmin-530029b6@MWS.MDS.XYZ -r MWS.MDS.XYZ -q 'listprincs'         Authenticating as principal cmadmin-530029b6@MWS.MDS.XYZ with keytab /tmp/kadmin/cmf8392129202434993503.keytab. kadmin: Communication failure with server while initializing kadmin interface Ensure you have port 749 defined in your firewall config: [root@idmipa03 ~]# cat /etc/firewalld/zones/public.xml|grep -Ei 749   <port protocol="tcp" port="749"/> [root@idmipa03 ~]# […]

ERROR! Extraneous options or arguments

Getting this error from Ansible? ERROR! Extraneous options or arguments Simply drop down one folder where the infra file resides and reexecute: [root@awx01 adhoc]# cd .. [root@awx01 ansible]# ansible cm* -i infra -m copy -a 'src=adhoc/public.xml dest=/etc/firewalld/zones/public.xml' This was just a fancy way of ansible telling you it can't find the infra file containing the […]

ls: Permission denied: user=root, access=EXECUTE, inode=”/tmp/.cloudera_health_monitoring_canary_files”:hdfs:supergroup:d———

Before enabling kerberos, the HDFS user is the only one that can list out files owned by itself: [root@cm-r01en01 run]# hdfs dfs -ls /tmp/.cloudera_health_monitoring_canary_files/.canary_file_2019_08_20-19_28_02.abab681cca56293als: Permission denied: user=root, access=EXECUTE, inode="/tmp/.cloudera_health_monitoring_canary_files":hdfs:supergroup:d——— You'll need to run the command via SUDO like this: [root@cm-r01en01 run]# sudo su -c "hdfs dfs -ls /tmp/.cloudera_health_monitoring_canary_files/.canary_file_2019_08_20-19_28_02.abab681cca56293a" -s /bin/bash hdfs -rw-rw-rw-   3 hdfs […]


     
  Copyright © 2003 - 2013 Tom Kacperski (microdevsys.com). All rights reserved.

Creative Commons License
This work is licensed under a Creative Commons Attribution 3.0 Unported License