You get the following error message:

15:27:35.430633 IP 192.168.0.80.nfs > 192.168.0.145.843: Flags [P.], seq 29:53, ack 417, win 235, options [nop,nop,TS val 6635947 ecr 5159126], length 24: NFS reply xid 2938911306 reply ERR 20: Auth Invalid failure code 13

Checking the nfs01 / nfs02 server we see alot of the following:

[2018-04-08 17:19:39.014330] E [MSGID: 108006] [afr-common.c:5006:__afr_handle_child_down_event] 0-gv01-replicate-0: All subvolumes are down. Going offline until atleast one of them comes back up.

[2018-04-08 17:32:21.714643] E [MSGID: 114058] [client-handshake.c:1571:client_query_portmap_cbk] 0-gv01-client-0: failed to get the port number for remote subvolume. Please run 'gluster volume status' on server to see if brick process is running.

[2018-04-08 17:32:21.734187] E [MSGID: 101046] [dht-common.c:1501:dht_lookup_dir_cbk] 0-gv01-dht: dict is null

[2018-04-08 17:32:21.734437] I [MSGID: 104039] [glfs-resolve.c:935:__glfs_active_subvol] 0-gv01: first lookup on graph 6e667330-322e-6e69-782e-6d64732e7879 (0) failed (Transport endpoint is not connected) [Transport endpoint is not connected]

Looks like the backend store for our NFS mount is not looking too hot right now.  So we go into troubleshooting Gluster.

Cheers,
TK