Header Shadow Image


ERROR    Timed out waiting for worker process collecting filesystem usage to complete.

Getting this error?

==> /var/log/cloudera-scm-agent/cloudera-scm-agent.log <==
[24/Aug/2019 22:00:08 +0000] 3697 Monitor-HostMonitor throttling_logger ERROR    Timed out waiting for worker process collecting filesystem usage to complete. This may occur if the host has an NFS or other remote filesystem that is not responding to requests in a timely fashion. Current nodev filesystems: /dev/shm,/run,/sys/fs/cgroup,/run/cloudera-scm-agent/process,/run/cloudera-scm-agent/process,/run/user/155601104,/n/mds.xyz,/run/user/0
[24/Aug/2019 22:00:08 +0000] 3697 MainThread agent        ERROR    Heartbeating to srv-c01.mws.mds.xyz:7182 failed.

Verify your NFS storage.  In our case one of the gluster brick's was out of space under root:

[root@nfs03 ~]# systemctl restart glusterd haproxy keepalived nfs-ganesha
Job for glusterd.service failed because the control process exited with error code. See "systemctl status glusterd.service" and "journalctl -xe" for details.
[root@nfs03 ~]# df -h
Filesystem               Size  Used Avail Use% Mounted on
/dev/mapper/centos-root   41G   41G   20K 100% /

Free the space and restart services.

Cheers,
TK

Subscribe
Notify of
0 Comments
Oldest
Newest Most Voted
Inline Feedbacks
View all comments

     
  Copyright © 2003 - 2025 Tom Kacperski (microdevsys.com). All rights reserved.

Creative Commons License
This work is licensed under a Creative Commons Attribution 3.0 Unported License

 

0
Would love your thoughts, please comment.x
()
x
The IT Development and Technology Mini Vault | MicroDevSys.com
Privacy Overview

This website uses cookies so that we can provide you with the best user experience possible. Cookie information is stored in your browser and performs functions such as recognising you when you return to our website and helping our team to understand which sections of the website you find most interesting and useful.