Header Shadow Image


ERROR    Timed out waiting for worker process collecting filesystem usage to complete.

Getting this error?

==> /var/log/cloudera-scm-agent/cloudera-scm-agent.log <==
[24/Aug/2019 22:00:08 +0000] 3697 Monitor-HostMonitor throttling_logger ERROR    Timed out waiting for worker process collecting filesystem usage to complete. This may occur if the host has an NFS or other remote filesystem that is not responding to requests in a timely fashion. Current nodev filesystems: /dev/shm,/run,/sys/fs/cgroup,/run/cloudera-scm-agent/process,/run/cloudera-scm-agent/process,/run/user/155601104,/n/mds.xyz,/run/user/0
[24/Aug/2019 22:00:08 +0000] 3697 MainThread agent        ERROR    Heartbeating to srv-c01.mws.mds.xyz:7182 failed.

Verify your NFS storage.  In our case one of the gluster brick's was out of space under root:

[root@nfs03 ~]# systemctl restart glusterd haproxy keepalived nfs-ganesha
Job for glusterd.service failed because the control process exited with error code. See "systemctl status glusterd.service" and "journalctl -xe" for details.
[root@nfs03 ~]# df -h
Filesystem               Size  Used Avail Use% Mounted on
/dev/mapper/centos-root   41G   41G   20K 100% /

Free the space and restart services.

Cheers,
TK

Leave a Reply

You must be logged in to post a comment.


     
  Copyright © 2003 - 2013 Tom Kacperski (microdevsys.com). All rights reserved.

Creative Commons License
This work is licensed under a Creative Commons Attribution 3.0 Unported License