ERROR Timed out waiting for worker process collecting filesystem usage to complete.
Getting this error?
==> /var/log/cloudera-scm-agent/cloudera-scm-agent.log <==
[24/Aug/2019 22:00:08 +0000] 3697 Monitor-HostMonitor throttling_logger ERROR Timed out waiting for worker process collecting filesystem usage to complete. This may occur if the host has an NFS or other remote filesystem that is not responding to requests in a timely fashion. Current nodev filesystems: /dev/shm,/run,/sys/fs/cgroup,/run/cloudera-scm-agent/process,/run/cloudera-scm-agent/process,/run/user/155601104,/n/mds.xyz,/run/user/0
[24/Aug/2019 22:00:08 +0000] 3697 MainThread agent ERROR Heartbeating to srv-c01.mws.mds.xyz:7182 failed.
Verify your NFS storage. In our case one of the gluster brick's was out of space under root:
[root@nfs03 ~]# systemctl restart glusterd haproxy keepalived nfs-ganesha
Job for glusterd.service failed because the control process exited with error code. See "systemctl status glusterd.service" and "journalctl -xe" for details.
[root@nfs03 ~]# df -h
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/centos-root 41G 41G 20K 100% /
Free the space and restart services.
Cheers,
TK