GlusterFS: Configuration and Setup w/ NFS-Ganesha for an HA NFS Cluster
In this post we will go over how to setup a highly available NFS Cluster using:
- GlusterFS
- NFS Ganesha
- CentOS 7
- HAPROXY
- keepalived
- firewalld
- selinux
This post is very lengthy and goes over quite a few details on the way to configuring this setup. We document virtually every step including how to build out a GlusterFS filesystem on both physical or virtual environments. For those interested in a quick setup, please skip to the SUMMARY or TESTING sections at the bottom for a summary of commands and configuration files used. If you run into problems, just search the page for the issue you have, as it's likely listed, and read the solution attempted.
DETAILED CONFIGURATION ATTEMPT ( ATTEMPT LOG )
GlusterFS: Configuration and Setup
In this post we'll go over configuring and setting up GlusterFS gradually working towards an HA NFS server configuration. Depending on what you read, Gluster is supposed to be faster then Ceph but harder to troubleshoot and recover from dataloss or failure. We'll see for ourselves and configure it and give all that a test. We will use two main sources for going about this: Getting Started Install and Getting Started Configure.
We'll reference a number of online pages to accomplish setup. CentOS 7 documentation on installing the latest Gluster Packages and Gluster Configuration. We'll go with this one instead as it support NFSv4 . However the Gluster Quickstart will give us a primer on starting off.
To start, install the RPM's as per the two links above. You may need to install the following epel repo before proceeding. The page above didn't cover at this time. Install and enable gluster on both nodes ( PHYSICAL: m-ph01, m-ph02, VIRTUAL: nfs01 / nfs02 ) :
yum install epel-release
yum -y install glusterfs glusterfs-fuse glusterfs-server glusterfs-api glusterfs-cli
Once done, we need to mount and add this volume as a duplicate of the previous node we did, m-ph01 then start the service:
[root@m-ph02 yum.repos.d]# systemctl enable gluster
Failed to execute operation: Access denied
[root@m-ph02 yum.repos.d]#
If it fails as above, it's probably the service name that was used. Do the following instead:
[root@m-ph02 yum.repos.d]# systemctl enable glusterd.service
Created symlink from /etc/systemd/system/multi-user.target.wants/glusterd.service to /usr/lib/systemd/system/glusterd.service.
[root@m-ph02 yum.repos.d]#
[root@m-ph02 yum.repos.d]# systemctl start glusterd
[root@m-ph02 yum.repos.d]# systemctl list-unit-files|grep -i glu
glusterd.service enabled
glusterfsd.service disabled
[root@m-ph02 yum.repos.d]#
Open firewall rules on all participating nodes:
firewall-cmd –zone=public –add-port=24007-24008/tcp –permanent
firewall-cmd –zone=public –add-port=49152/tcp –permanent
firewall-cmd –zone=public –add-port=38465-38469/tcp –permanent
firewall-cmd –zone=public –add-port=111/tcp –permanent
firewall-cmd –zone=public –add-port=111/udp –permanent
firewall-cmd –zone=public –add-port=2049/tcp –permanent
firewall-cmd –reload
Yes, you need the ports (Virtual Host example provided below):
[root@nfs02 yum.repos.d]# gluster peer probe nfs01
peer probe: failed: Probe returned with Transport endpoint is not connected
[root@nfs02 yum.repos.d]# firewall-cmd –zone=public –add-port=24007-24008/tcp –permanent
success
[root@nfs02 yum.repos.d]# firewall-cmd –zone=public –add-port=49152/tcp –permanent
success
[root@nfs02 yum.repos.d]# firewall-cmd –zone=public –add-port=38465-38469/tcp –permanent
success
[root@nfs02 yum.repos.d]# firewall-cmd –zone=public –add-port=111/tcp –permanent
success
[root@nfs02 yum.repos.d]# firewall-cmd –zone=public –add-port=111/udp –permanent
success
[root@nfs02 yum.repos.d]# firewall-cmd –zone=public –add-port=2049/tcp –permanent
success
[root@nfs02 yum.repos.d]# firewall-cmd –reload
success
[root@nfs02 yum.repos.d]# gluster peer probe nfs01
peer probe: success.
[root@nfs02 yum.repos.d]#
Results:
[root@nfs01 ~]# gluster peer status
Number of Peers: 1
Hostname: nfs02.nix.mine.dom
Uuid: 395276d9-17c6-4284-964c-9c93c0a30551
State: Peer in Cluster (Connected)
Other names:
nfs02
[root@nfs01 ~]#
[root@nfs02 yum.repos.d]# gluster peer status
Number of Peers: 1
Hostname: nfs01
Uuid: 4067fbf3-6118-426c-a899-601afe3ff1d7
State: Peer in Cluster (Connected)
Other names:
nfs01.nix.mine.dom
[root@nfs02 yum.repos.d]#
Once the service is started, issue a probe command. If it doesn't work from this node, as you can see below, issue it from a working node towards the second node you are building:
[root@m-ph02 yum.repos.d]# gluster peer probe m-ph01
peer probe: failed: m-ph01 is either already part of another cluster or having volumes configured
Since we got the above error, we issue the following from the working node:
[root@m-ph01 yum.repos.d]# gluster peer probe m-ph02
peer probe: success.
[root@m-ph01 yum.repos.d]#
Then the following again from the new node we are building back to the working node to ensure all bits are communicating well:
[root@m-ph02 yum.repos.d]# ping m-ph01
PING m-ph01.mine.dom (192.168.0.60) 56(84) bytes of data.
64 bytes from 192.168.0.60: icmp_seq=1 ttl=64 time=0.269 ms
^C
— m-ph01.mine.dom ping statistics —
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 0.269/0.269/0.269/0.000 ms
[root@m-ph02 yum.repos.d]# gluster peer probe m-ph01
peer probe: success.
[root@m-ph02 yum.repos.d]#
Once the probe completes successfully, let's move on to the next steps of creating the VG's and LV's on each physical LUN we'll be using for the volume (This is identical for virtual hosts):
[root@m-ph02 yum.repos.d]# pvcreate /dev/sdb
[root@m-ph02 yum.repos.d]# vgcreate mdskvmsanvg /dev/sdb
[root@m-ph02 yum.repos.d]# lvcreate -l524286 -n mdskvmsanlv mdskvmsanvg
Logical volume "mdskvmsanlv" created.
[root@m-ph02 yum.repos.d]# lvs
LV VG Attr LSize Pool Origin Data% Meta% Move Log Cpy%Sync Convert
home centos -wi-ao—- 19.99g
root centos -wi-ao—- 40.95g
swap centos -wi-ao—- 6.84g
mdskvmsanlv mdskvmsanvg -wi-a—– 2.00t
[root@m-ph02 yum.repos.d]#
[root@m-ph02 yum.repos.d]# mkfs.xfs -i size=512 /dev/mapper/mdskvmsanvg-mdskvmsanlv
meta-data=/dev/mapper/mdskvmsanvg-mdskvmsanlv isize=512 agcount=4, agsize=134217216 blks
= sectsz=512 attr=2, projid32bit=1
= crc=0 finobt=0
data = bsize=4096 blocks=536868864, imaxpct=5
= sunit=0 swidth=0 blks
naming =version 2 bsize=4096 ascii-ci=0 ftype=0
log =internal log bsize=4096 blocks=262143, version=2
= sectsz=512 sunit=0 blks, lazy-count=1
realtime =none extsz=4096 blocks=0, rtextents=0
[root@m-ph02 yum.repos.d]#
Then create the /etc/fstab mount point (Do this on both nodes):
[root@m-ph02 yum.repos.d]# grep -Ei "xfs" /etc/fstab|grep mds
/dev/mapper/mdskvmsanvg-mdskvmsanlv /mnt/p02-d01 xfs logbufs=8,noatime,nodiratime,allocsize=512m 0 0
[root@m-ph02 yum.repos.d]#
Change the ownership of the mounts depending on what Cloud Manager you use:
[root@m-ph02 yum.repos.d]# mkdir /mnt/p02-d01
[root@m-ph02 yum.repos.d]# chown oneadmin.oneadmin /mnt/p02-d01
[root@m-ph02 yum.repos.d]# id oneadmin
uid=9869(oneadmin) gid=9869(oneadmin) groups=9869(oneadmin)
[root@m-ph02 yum.repos.d]#
Then try to add the brick. Two examples are provided. First a virtual example:
[root@nfs02 ~]# gluster volume create gv01 replica 2 nfs01:/bricks/0/gv0 nfs02:/bricks/0/gv0
Replica 2 volumes are prone to split-brain. Use Arbiter or Replica 3 to avoid this. See: http://docs.gluster.org/en/latest/Administrator%20Guide/Split%20brain%20and%20ways%20to%20deal%20with%20it/.
Do you still want to continue?
(y/n) y
volume create: gv01: success: please start the volume to access data
[root@nfs02 ~]# gluster volume start gv01
volume start: gv01: success
[root@nfs02 ~]# gluster volume info
Volume Name: gv01
Type: Replicate
Volume ID: e1485a5a-0edf-446d-9ae9-2705e6b57994
Status: Started
Snapshot Count: 0
Number of Bricks: 1 x 2 = 2
Transport-type: tcp
Bricks:
Brick1: nfs01:/bricks/0/gv0
Brick2: nfs02:/bricks/0/gv0
Options Reconfigured:
transport.address-family: inet
nfs.disable: on
performance.client-io-threads: off
[root@nfs02 ~]#
And a physical example:
[root@m-ph02 mnt]# gluster volume create gv0 replica 2 m-ph01.mine.dom:/mnt/p01-d01 m-ph02.mine.dom:/mnt/p02-d01
volume create: gv0: failed: The brick m-ph02.mine.dom:/mnt/p02-d01 is a mount point. Please create a sub-directory under the mount point and use that as the brick directory. Or use 'force' at the end of the command if you want to override this behavior.
[root@m-ph02 mnt]# mkdir /mnt/p02-d01/glusterv01
[root@m-ph02 mnt]# gluster volume create gv0 replica 2 m-ph01.mine.dom:/mnt/p01-d01 m-ph02.mine.dom:/mnt/p02-d01
volume create: gv0: failed: The brick m-ph02.mine.dom:/mnt/p02-d01 is a mount point. Please create a sub-directory under the mount point and use that as the brick directory. Or use 'force' at the end of the command if you want to override this behavior.
[root@m-ph02 mnt]# gluster volume create gv0 replica 2 m-ph01.mine.dom:/mnt/p01-d01/glusterv01 m-ph02.mine.dom:/mnt/p02-d01/glusterv01
volume create: gv0: failed: Staging failed on 192.168.0.60. Error: Brick: m-ph01.mine.dom:/mnt/p01-d01/glusterv01 not available. Brick may be containing or be contained by an existing brick
[root@m-ph02 mnt]#
If that fails, due to already being in a volume, simply add this one as a replicated brick of the first. First we need to remove the old brick and the whole cluster as we created a non replicated cluster. We do this by:
[root@m-ph02 mnt]# gluster volume delete mdsglusterv01
Deleting volume will erase all information about the volume. Do you want to continue? (y/n) y
volume delete: mdsglusterv01: success
[root@m-ph02 mnt]# gluster volume info
No volumes present
[root@m-ph02 mnt]#
Now add the bricks, indicating replication in the same command and test by copying some ISO's to the volume:
[root@m-ph01 p01-d01]# gluster volume create mdsgv01 replica 2 m-ph01.mine.dom:/mnt/p01-d01/glusterv01 m-ph02.mine.dom:/mnt/p02-d01/glusterv02
volume create: mdsgv01: success: please start the volume to access data
[root@m-ph01 p01-d01]# gluster volume start mdsgv01
volume start: mdsgv01: success
[root@m-ph01 p01-d01]# mv *.iso glusterv01/
[root@m-ph01 p01-d01]# cd glusterv01
[root@m-ph01 glusterv01]# ls -altri
total 6792216
146 -rwxr–r–. 1 oneadmin oneadmin 632291328 May 19 01:19 loadit-10.iso
145 -rwxr–r–. 1 oneadmin oneadmin 632291328 May 19 01:19 loadit-09.iso
144 -rwxr–r–. 1 oneadmin oneadmin 632291328 May 19 01:19 loadit-08.iso
143 -rwxr–r–. 1 oneadmin oneadmin 632291328 May 19 01:19 loadit-07.iso
142 -rwxr–r–. 1 oneadmin oneadmin 632291328 May 19 01:19 loadit-06.iso
141 -rwxr–r–. 1 oneadmin oneadmin 632291328 May 19 01:19 loadit-05.iso
140 -rwxr–r–. 1 oneadmin oneadmin 632291328 May 19 01:19 loadit-04.iso
136 -rwxr–r–. 1 oneadmin oneadmin 632291328 May 19 01:19 CentOS-7-x86_64-Minimal-1511.iso
137 -rwxr–r–. 1 oneadmin oneadmin 632291328 May 19 01:20 loadit-01.iso
139 -rwxr–r–. 1 oneadmin oneadmin 632291328 May 19 01:21 loadit-03.iso
138 -rwxr–r–. 1 oneadmin oneadmin 632291328 May 19 01:21 loadit-02.iso
2147483776 drwxr-xr-x. 3 root root 24 Jun 9 01:29 .trashcan
536871104 drw——-. 8 root root 4096 Jun 9 01:29 .glusterfs
128 drwxr-xr-x. 3 oneadmin oneadmin 23 Jun 9 01:30 ..
402653376 drwxr-xr-x. 4 oneadmin oneadmin 4096 Jun 9 01:30 .
[root@m-ph01 glusterv01]#
[root@m-ph01 glusterv01]#
[root@m-ph01 glusterv01]#
[root@m-ph01 glusterv01]# gluster volume info
Volume Name: mdsgv01
Type: Replicate
Volume ID: f5b57076-dbd4-4d77-ae13-c1f3ee3adbe0
Status: Started
Number of Bricks: 1 x 2 = 2
Transport-type: tcp
Bricks:
Brick1: m-ph01.mine.dom:/mnt/p01-d01/glusterv01
Brick2: m-ph02.mine.dom:/mnt/p02-d01/glusterv02
Options Reconfigured:
performance.readdir-ahead: on
[root@m-ph01 glusterv01]# ls -altri
total 6792216
146 -rwxr–r–. 1 oneadmin oneadmin 632291328 May 19 01:19 loadit-10.iso
145 -rwxr–r–. 1 oneadmin oneadmin 632291328 May 19 01:19 loadit-09.iso
144 -rwxr–r–. 1 oneadmin oneadmin 632291328 May 19 01:19 loadit-08.iso
143 -rwxr–r–. 1 oneadmin oneadmin 632291328 May 19 01:19 loadit-07.iso
142 -rwxr–r–. 1 oneadmin oneadmin 632291328 May 19 01:19 loadit-06.iso
141 -rwxr–r–. 1 oneadmin oneadmin 632291328 May 19 01:19 loadit-05.iso
140 -rwxr–r–. 1 oneadmin oneadmin 632291328 May 19 01:19 loadit-04.iso
136 -rwxr–r–. 1 oneadmin oneadmin 632291328 May 19 01:19 CentOS-7-x86_64-Minimal-1511.iso
137 -rwxr–r–. 1 oneadmin oneadmin 632291328 May 19 01:20 loadit-01.iso
139 -rwxr–r–. 1 oneadmin oneadmin 632291328 May 19 01:21 loadit-03.iso
138 -rwxr–r–. 1 oneadmin oneadmin 632291328 May 19 01:21 loadit-02.iso
2147483776 drwxr-xr-x. 3 root root 24 Jun 9 01:29 .trashcan
536871104 drw——-. 8 root root 4096 Jun 9 01:29 .glusterfs
128 drwxr-xr-x. 3 oneadmin oneadmin 23 Jun 9 01:30 ..
402653376 drwxr-xr-x. 4 oneadmin oneadmin 4096 Jun 9 01:30 .
[root@m-ph01 glusterv01]#
If the above commands failed, remove the volume then bricks and start fresh. Deleting a volume is easy:
[root@nfs02 0]# gluster volume remove gv01 replica 2 nfs01:/bricks/0/gv0 nfs02:/bricks/0/gv0 stop
volume remove-brick stop: failed: Volume gv01 is not a distribute volume or contains only 1 brick.
Not performing rebalance
[root@nfs02 0]# gluster volume stop gv01
Stopping volume will make its data inaccessible. Do you want to continue? (y/n) y
volume stop: gv01: success
[root@nfs02 0]#
[root@nfs02 0]#
[root@nfs02 0]#
[root@nfs02 0]# gluster volume delete gv01
Deleting volume will erase all information about the volume. Do you want to continue? (y/n) y
volume delete: gv01: success
[root@nfs02 0]#
Now that you have your GlusterFS setup, we want to share it using NFSv4. For that, we need NFS-Ganesha. It's the only application that will support NFSv4 plus a number of other protocols. We will follow the github project to set it up.
On both volumes type:
[root@nfs01 gv01]# systemctl status nfs
â nfs-server.service – NFS server and services
Loaded: loaded (/usr/lib/systemd/system/nfs-server.service; disabled; vendor preset: disabled)
Active: inactive (dead)
[root@nfs01 gv01]# systemctl disable nfs
[root@nfs01 gv01]# gluster volume set gv01 nfs.disable on
volume set: success
[root@nfs01 gv01]#
Start the ganesha nfsd daemon on both nodes using. We just want to ensure the service works and can start up. (You can stop this right after):
ganesha.nfsd -f nfs-ganesha.conf -L nfs-ganesha.log -N NIV_DEBUG
Then, still on both nodes, create a path inside each gluster server and mount it using the glusterfs protocol:
ON NODE 1:
[root@nfs01 gv01]# mkdir /n 2>/dev/null; mount -t glusterfs nfs02:/gv01 /n
[root@nfs01 gv01]# mount|grep -i gluste
nfs02:/gv01 on /n type fuse.glusterfs (rw,relatime,user_id=0,group_id=0,default_permissions,allow_other,max_read=131072)
[root@nfs01 gv01]# cd /n
[root@nfs01 n]# ls -altri
total 8
1 drwxr-xr-x. 3 root root 4096 Feb 18 14:17 .
128 dr-xr-xr-x. 19 root root 4096 Feb 18 14:22 ..
[root@nfs01 n]#
ON NODE 2:
[root@nfs02 gv01]# umount /n
[root@nfs02 gv01]# mkdir /n 2>/dev/null; mount -t glusterfs nfs02:/gv01 /n
[root@nfs02 gv01]# mount|grep -i gluster
nfs02:/gv01 on /n type fuse.glusterfs (rw,relatime,user_id=0,group_id=0,default_permissions,allow_other,max_read=131072)
[root@nfs02 gv01]#
Define your export:
ON NODE 1:
[root@nfs01 ganesha]# pwd
/etc/ganesha
[root@nfs01 ganesha]# cat export.conf
EXPORT{
Export_Id = 1 ; # Export ID unique to each export
Path = "/n"; # Path of the volume to be exported. Eg: "/test_volume"
FSAL {
name = GLUSTER;
hostname = "nfs01.nix.mine.dom"; # IP of one of the nodes in the trusted pool
volume = "/n"; # Volume name. Eg: "test_volume"
}
Access_type = RW; # Access permissions
Squash = root_squash; # To enable/disable root squashing
Disable_ACL = FALSE; # To enable/disable ACL
Pseudo = "pseudo_path"; # NFSv4 pseudo path for this export. Eg: "/test_volume_pseudo"
Protocols = "4" ; # NFS protocols supported
Transports = "TCP" ; # Transport protocols supported
SecType = "sys"; # Security flavors supported
}
[root@nfs01 ganesha]#
ON NODE 2:
[root@nfs02 ganesha]# pwd
/etc/ganesha
[root@nfs02 ganesha]# cat export.conf
EXPORT{
Export_Id = 1 ; # Export ID unique to each export
Path = "/n"; # Path of the volume to be exported. Eg: "/test_volume"
FSAL {
name = GLUSTER;
hostname = "nfs02.nix.mine.dom"; # IP of one of the nodes in the trusted pool
volume = "/n"; # Volume name. Eg: "test_volume"
}
Access_type = RW; # Access permissions
Squash = root_squash; # To enable/disable root squashing
Disable_ACL = FALSE; # To enable/disable ACL
Pseudo = "pseudo_path"; # NFSv4 pseudo path for this export. Eg: "/test_volume_pseudo"
Protocols = "4" ; # NFS protocols supported
Transports = "TCP" ; # Transport protocols supported
SecType = "sys"; # Security flavors supported
}
[root@nfs02 ganesha]#
Include the export.conf into ganesha.conf:
ON NODE 1:
[root@nfs02 ganesha]# grep -Ei include /etc/ganesha/ganesha.conf
%include "/etc/ganesha/export.conf"
[root@nfs02 ganesha]#
ON NODE 2:
[root@nfs01 ganesha]# grep -Ei include /etc/ganesha/ganesha.conf
%include "/etc/ganesha/export.conf"
[root@nfs01 ganesha]#
Check the services on both nodes:
[root@nfs02 ganesha]# systemctl list-unit-files –type=service|grep -Ei ganesh
nfs-ganesha-config.service static
nfs-ganesha-lock.service static
nfs-ganesha.service disabled
[root@nfs02 ganesha]# systemctl status nfs-ganesha.service
â nfs-ganesha.service – NFS-Ganesha file server
Loaded: loaded (/usr/lib/systemd/system/nfs-ganesha.service; disabled; vendor preset: disabled)
Active: active (running) since Sun 2018-02-18 01:14:09 EST; 14h ago
Docs: http://github.com/nfs-ganesha/nfs-ganesha/wiki
Main PID: 3180 (ganesha.nfsd)
CGroup: /system.slice/nfs-ganesha.service
ââ3180 /usr/bin/ganesha.nfsd -L /var/log/ganesha.log -f /etc/ganesha/ganesha.conf -N NIV_EVENT
Feb 18 01:14:09 nfs02.nix.mine.dom systemd[1]: Starting NFS-Ganesha file server…
Feb 18 01:14:09 nfs02.nix.mine.dom systemd[1]: Started NFS-Ganesha file server.
[root@nfs02 ganesha]#
[root@nfs02 ganesha]#
[root@nfs02 ganesha]#
[root@nfs02 ganesha]# ps -ef|grep -Ei ganesha
root 3180 1 0 01:14 ? 00:00:21 /usr/bin/ganesha.nfsd -L /var/log/ganesha.log -f /etc/ganesha/ganesha.conf -N NIV_EVENT
root 10721 4201 0 15:21 pts/0 00:00:00 grep –color=auto -Ei ganesha
[root@nfs02 ganesha]#
Should be the same on both nodes. This time we'll start ganesha.nfsd using systemd:
[root@nfs01 ganesha]# systemctl restart nfs-ganesha.service
[root@nfs01 ganesha]# ps -ef|grep -Ei [Gg]anesha
root 4825 1 0 15:25 ? 00:00:00 /usr/bin/ganesha.nfsd -L /var/log/ganesha.log -f /etc/ganesha/ganesha.conf -N NIV_EVENT
[root@nfs01 ganesha]#
Let's test the exports and see if anything is listed:
showmount -e localhost
But first ensure ganesha nfsd damon is enabled:
[root@nfs01 log]# systemctl enable nfs-ganesha.service
Created symlink from /etc/systemd/system/multi-user.target.wants/nfs-ganesha.service to /usr/lib/systemd/system/nfs-ganesha.service.
[root@nfs01 log]#
And we have problems (/var/log/ganesha.log):
NFS STARTUP :CRIT :Could not dlopen module:/usr/lib64/ganesha/libfsalvfs.so Error:/usr/lib64/ganesha/libfsalvfs.so: cannot open shared object file: No such file or directory
This is because we are missing the nfs-ganesha-vfs plugin while we are referencing VFS in one of the configs. Either remove reference or install the plugin. We choose the latter because more is more:
[root@nfs02 log]# rpm -aq|grep -Ei nfs-ganesha
nfs-ganesha-xfs-2.3.2-1.el7.x86_64
nfs-ganesha-2.3.2-1.el7.x86_64
[root@nfs02 log]# grep -Ei VFS /etc/ganesha/*
/etc/ganesha/ganesha.conf: Name = VFS;
[root@nfs02 log]#
Unfortunately when we try to install the right package we get another error:
[root@nfs02 log]# yum install nfs-ganesha-vfs.x86_64
Error: Package: nfs-ganesha-2.5.5-1.el7.x86_64 (centos-gluster312)
Requires: libntirpc.so.1.5(NTIRPC_1.5.4)(64bit)
Let's check where the packages are coming from so perhaps we can disable one of them to get the right version because it's trying to pull an updated nfs-ganesha version as well but gives us:
Error: Package: nfs-ganesha-2.5.5-1.el7.x86_64 (centos-gluster312)
Requires: libntirpc.so.1.5(NTIRPC_1.5.4)(64bit)
which wasn't uploaded to the CentOS 7 repos at the time of this writing. So we'll use the older package (We don't need to be recent up-to-a-day ATM):
[root@nfs02 log]# yum info nfs-ganesha
Loaded plugins: fastestmirror, priorities
Loading mirror speeds from cached hostfile
* base: centos.mirror.rafal.ca
* epel: mirror.math.princeton.edu
* extras: centos.mirror.rafal.ca
* updates: centos.mirror.rafal.ca
Installed Packages
Name : nfs-ganesha
Arch : x86_64
Version : 2.3.2
Release : 1.el7
Size : 1.6 M
Repo : installed
From repo : epel
Summary : NFS Server running in user space
URL : https://github.com/nfs-ganesha/nfs-ganesha/wiki
License : LGPLv3+
Description : nfs-ganesha : NFS-GANESHA is a NFS Server running in user space.
: It comes with various back-end modules (called FSALs) provided as
: shared objects to support different file systems and name-spaces.
Available Packages
Name : nfs-ganesha
Arch : x86_64
Version : 2.5.5
Release : 1.el7
Size : 650 k
Repo : centos-gluster312/7/x86_64
Summary : NFS-Ganesha is a NFS Server running in user space
URL : https://github.com/nfs-ganesha/nfs-ganesha/wiki
License : LGPLv3+
Description : nfs-ganesha : NFS-GANESHA is a NFS Server running in user space.
: It comes with various back-end modules (called FSALs) provided as
: shared objects to support different file systems and name-spaces.
[root@nfs02 log]#
So we use the following that will install the correct version for our NFS Ganesha install:
[root@nfs02 ~]# yum –disablerepo="*" –enablerepo=epel install nfs-ganesha-vfs -y
[root@nfs02 log]# ls -altri /usr/lib64/ganesha/libfsalvfs.so
102513 lrwxrwxrwx. 1 root root 15 Feb 18 16:31 /usr/lib64/ganesha/libfsalvfs.so -> libfsalvfs.so.4
[root@nfs02 log]#
Next we restart NFS Ganesha again and check. Still errors:
18/02/2018 16:37:55 : epoch 5a89f232 : nfs01.nix.mine.dom : ganesha.nfsd-5497[main] config_errs_to_log :CONFIG :CRIT :Config File (/etc/ganesha/export.conf:5): Failed to load FSAL (GLUSTER) because: Can not access a needed shared library
18/02/2018 16:37:55 : epoch 5a89f232 : nfs01.nix.mine.dom : ganesha.nfsd-5497[main] config_errs_to_log :CONFIG :CRIT :Config File (/etc/ganesha/export.conf:5): 1 validation errors in block FSAL
18/02/2018 16:37:55 : epoch 5a89f232 : nfs01.nix.mine.dom : ganesha.nfsd-5497[main] config_errs_to_log :CONFIG :CRIT :Config File (/etc/ganesha/export.conf:5): Errors processing block (FSAL)
18/02/2018 16:37:55 : epoch 5a89f232 : nfs01.nix.mine.dom : ganesha.nfsd-5497[main] config_errs_to_log :CONFIG :CRIT :Config File (/etc/ganesha/export.conf:1): 1 validation errors in block EXPORT
18/02/2018 16:37:55 : epoch 5a89f232 : nfs01.nix.mine.dom : ganesha.nfsd-5497[main] config_errs_to_log :CONFIG :CRIT :Config File (/etc/ganesha/export.conf:1): Errors processing block (EXPORT)
Problem with this block:
FSAL {
name = GLUSTER;
hostname = "nfs01.nix.mine.dom"; # IP of one of the nodes in the trusted pool
volume = "/n"; # Volume name. Eg: "test_volume"
}
Unfortunately, the error doesn't tell you WHICH library is missing but fortunately the page above indicated what could be wrong so we need to look for libgfapi.so.0()(64bit) .
yum install glusterfs-api
But that wasn't it. Looking a bit more carefuly, we notice the following:
18/02/2018 16:37:49 : epoch 5a89f22d : nfs02.nix.mine.dom : ganesha.nfsd-12399[main] load_fsal :NFS STARTUP :CRIT :Could not dlopen module:/usr/lib64/ganesha/libfsalgluster.so Error:/usr/lib64/ganesha/libfsalgluster.so: cannot open shared object file: No such file or directory
We need the following package then:
yum –disablerepo="*" –enablerepo=epel install nfs-ganesha-gluster
But this is only available from the new repository. 🙁 Fortunately, there is hope. So we'll use that rpm.
yum install http://cbs.centos.org/kojifiles/packages/nfs-ganesha/2.3.2/1.el7/x86_64/nfs-ganesha-gluster-2.3.2-1.el7.x86_64.rpm
Let's restart and try this thing again. We move forward but get this error:
18/02/2018 18:04:53 : epoch 5a8a0694 : nfs02.nix.mine.dom : ganesha.nfsd-13043[main] config_errs_to_log :CONFIG :CRIT :Config File (/etc/ganesha/export.conf:5): 1 validation errors in block FSAL
18/02/2018 18:04:53 : epoch 5a8a0694 : nfs02.nix.mine.dom : ganesha.nfsd-13043[main] config_errs_to_log :CONFIG :CRIT :Config File (/etc/ganesha/export.conf:5): Errors processing block (FSAL)
18/02/2018 18:04:53 : epoch 5a8a0694 : nfs02.nix.mine.dom : ganesha.nfsd-13043[main] config_errs_to_log :CONFIG :CRIT :Config File (/etc/ganesha/export.conf:1): 1 validation errors in block EXPORT
18/02/2018 18:04:53 : epoch 5a8a0694 : nfs02.nix.mine.dom : ganesha.nfsd-13043[main] config_errs_to_log :CONFIG :CRIT :Config File (/etc/ganesha/export.conf:1): Errors processing block (EXPORT)
Reverting the configuration to some basic settings, we finally export a mount using the following settings:
[root@nfs02 ganesha]# systemctl stop nfs-ganesha.service; >/var/log/ganesha.log; systemctl start nfs-ganesha.service
[root@nfs02 ganesha]# showmount -e localhost
Export list for localhost:
/bricks/0/gv01/n (everyone)
[root@nfs02 ganesha]#
[root@nfs02 ganesha]#
[root@nfs02 ganesha]#
[root@nfs02 ganesha]# cat export.conf
EXPORT{
Export_Id = 1 ; # Export ID unique to each export
Path = "/bricks/0/gv01/n"; # Path of the volume to be exported. Eg: "/test_volume"
FSAL {
name = GLUSTER;
hostname = "nfs02.nix.mine.dom"; # IP of one of the nodes in the trusted pool
volume = "gv01"; # Volume name. Eg: "test_volume"
}
Access_type = RW; # Access permissions
Squash = No_root_squash; # To enable/disable root squashing
Disable_ACL = TRUE; # To enable/disable ACL
Pseudo = "/n"; # NFSv4 pseudo path for this export. Eg: "/test_volume_pseudo"
Protocols = "3","4" ; # NFS protocols supported
Transports = "UDP","TCP" ; # Transport protocols supported
SecType = "sys"; # Security flavors supported
}
[root@nfs02 ganesha]# cat /var/log/ganesha.log
18/02/2018 19:32:34 : epoch 5a8a1b22 : nfs02.nix.mine.dom : ganesha.nfsd-14166[main] main :MAIN :EVENT :ganesha.nfsd Starting: Ganesha Version /builddir/build/BUILD/nfs-ganesha-2.3.2/src, built at May 4 2016 05:09:13 on
18/02/2018 19:32:34 : epoch 5a8a1b22 : nfs02.nix.mine.dom : ganesha.nfsd-14167[main] nfs_set_param_from_conf :NFS STARTUP :EVENT :Configuration file successfully parsed
18/02/2018 19:32:34 : epoch 5a8a1b22 : nfs02.nix.mine.dom : ganesha.nfsd-14167[main] init_server_pkgs :NFS STARTUP :EVENT :Initializing ID Mapper.
18/02/2018 19:32:34 : epoch 5a8a1b22 : nfs02.nix.mine.dom : ganesha.nfsd-14167[main] init_server_pkgs :NFS STARTUP :EVENT :ID Mapper successfully initialized.
18/02/2018 19:32:34 : epoch 5a8a1b22 : nfs02.nix.mine.dom : ganesha.nfsd-14167[main] glusterfs_create_export :FSAL :EVENT :Volume gv01 exported at : '/'
18/02/2018 19:32:35 : epoch 5a8a1b22 : nfs02.nix.mine.dom : ganesha.nfsd-14167[main] lower_my_caps :NFS STARTUP :EVENT :CAP_SYS_RESOURCE was successfully removed for proper quota management in FSAL
18/02/2018 19:32:35 : epoch 5a8a1b22 : nfs02.nix.mine.dom : ganesha.nfsd-14167[main] lower_my_caps :NFS STARTUP :EVENT :currenty set capabilities are: = cap_chown,cap_dac_override,cap_dac_read_search,cap_fowner,cap_fsetid,cap_kill,cap_setgid,cap_setuid,cap_setpcap,cap_linux_immutable,cap_net_bind_service,cap_net_broadcast,cap_net_admin,cap_net_raw,cap_ipc_lock,cap_ipc_owner,cap_sys_module,cap_sys_rawio,cap_sys_chroot,cap_sys_ptrace,cap_sys_pacct,cap_sys_admin,cap_sys_boot,cap_sys_nice,cap_sys_time,cap_sys_tty_config,cap_mknod,cap_lease,cap_audit_write,cap_audit_control,cap_setfcap+ep
18/02/2018 19:32:35 : epoch 5a8a1b22 : nfs02.nix.mine.dom : ganesha.nfsd-14167[main] nfs_Init_admin_thread :NFS CB :EVENT :Admin thread initialized
18/02/2018 19:32:35 : epoch 5a8a1b22 : nfs02.nix.mine.dom : ganesha.nfsd-14167[main] nfs4_start_grace :STATE :EVENT :NFS Server Now IN GRACE, duration 60
18/02/2018 19:32:35 : epoch 5a8a1b22 : nfs02.nix.mine.dom : ganesha.nfsd-14167[main] nfs_rpc_cb_init_ccache :NFS STARTUP :EVENT :Callback creds directory (/var/run/ganesha) already exists
18/02/2018 19:32:35 : epoch 5a8a1b22 : nfs02.nix.mine.dom : ganesha.nfsd-14167[main] nfs_rpc_cb_init_ccache :NFS STARTUP :WARN :gssd_refresh_krb5_machine_credential failed (-1765328203:0)
18/02/2018 19:32:35 : epoch 5a8a1b22 : nfs02.nix.mine.dom : ganesha.nfsd-14167[main] nfs_Start_threads :THREAD :EVENT :Starting delayed executor.
18/02/2018 19:32:35 : epoch 5a8a1b22 : nfs02.nix.mine.dom : ganesha.nfsd-14167[main] nfs_Start_threads :THREAD :EVENT :9P/TCP dispatcher thread was started successfully
18/02/2018 19:32:35 : epoch 5a8a1b22 : nfs02.nix.mine.dom : ganesha.nfsd-14167[_9p_disp] _9p_dispatcher_thread :9P DISP :EVENT :9P dispatcher started
18/02/2018 19:32:35 : epoch 5a8a1b22 : nfs02.nix.mine.dom : ganesha.nfsd-14167[main] nfs_Start_threads :THREAD :EVENT :gsh_dbusthread was started successfully
18/02/2018 19:32:35 : epoch 5a8a1b22 : nfs02.nix.mine.dom : ganesha.nfsd-14167[main] nfs_Start_threads :THREAD :EVENT :admin thread was started successfully
18/02/2018 19:32:35 : epoch 5a8a1b22 : nfs02.nix.mine.dom : ganesha.nfsd-14167[main] nfs_Start_threads :THREAD :EVENT :reaper thread was started successfully
18/02/2018 19:32:35 : epoch 5a8a1b22 : nfs02.nix.mine.dom : ganesha.nfsd-14167[reaper] nfs_in_grace :STATE :EVENT :NFS Server Now IN GRACE
18/02/2018 19:32:35 : epoch 5a8a1b22 : nfs02.nix.mine.dom : ganesha.nfsd-14167[main] nfs_Start_threads :THREAD :EVENT :General fridge was started successfully
18/02/2018 19:32:35 : epoch 5a8a1b22 : nfs02.nix.mine.dom : ganesha.nfsd-14167[main] nfs_start :NFS STARTUP :EVENT :————————————————-
18/02/2018 19:32:35 : epoch 5a8a1b22 : nfs02.nix.mine.dom : ganesha.nfsd-14167[main] nfs_start :NFS STARTUP :EVENT : NFS SERVER INITIALIZED
18/02/2018 19:32:35 : epoch 5a8a1b22 : nfs02.nix.mine.dom : ganesha.nfsd-14167[main] nfs_start :NFS STARTUP :EVENT :————————————————-
[root@nfs02 ganesha]#
[root@nfs02 ganesha]# showmount -e localhost
Export list for localhost:
/bricks/0/gv01/n (everyone)
[root@nfs02 ganesha]#
But we only want NFSv4 available. Let's experiment a bit more:
[root@nfs02 ganesha]# mount -t glusterfs nfs01:/gv01 /n
[root@nfs02 ganesha]# vi export.conf
[root@nfs02 ganesha]# systemctl stop nfs-ganesha.service; >/var/log/ganesha.log; systemctl start nfs-ganesha.service
[root@nfs02 ganesha]# showmount -e localhost
Export list for localhost:
/n (everyone)
[root@nfs02 ganesha]# cat export.conf |grep Path
Path = "/n"; # Path of the volume to be exported. Eg: "/test_volume"
[root@nfs02 ganesha]#
Let's try to change the NFS options to allow only for NFSv4:
Protocols = "3","4" ;
But removing "3" results in no mounts being shown. This is because showmount -e doesn't list NFSv4 mounts. It uses the NFSv3 protocol to list NFSv3 mounts so if we are not exporting using NFSv4 , we won't see the mounts. The way we can ensure it's working is if we actually mount the NFSv4 storage or use this alternate to list the mounts but at this time I still don't have this one liner working properly. Here it is anyway:
dbus-send –type=method_call –print-reply –system –dest=org.ganesha.nfsd /org/ganesha/nfsd/ExportMgr org.ganesha.nfsd.exportmgr.ShowExports
Add the following section to the /etc/ganesha/ganesha.conf file:
[root@nfs02 ganesha]# cat ganesha.conf
###################################################
#
# EXPORT
#
# To function, all that is required is an EXPORT
#
# Define the absolute minimal export
#
###################################################
NFS_Core_Param {
NFS_Port = 2049;
MNT_Port = 20048;
NLM_Port = 38468;
Rquota_Port = 4501;
}
%include "/etc/ganesha/export.conf"
[root@nfs02 ganesha]#
Follow this by allowing the last port otherwise you get this error below:
ganesha.nfsd-15334[main] Bind_sockets_V6 :DISP :WARN :Cannot bind RQUOTA udp6 socket, error 13 (Permission denied)
ganesha.nfsd-15334[main] Bind_sockets :DISP :FATAL :Error binding to V6 interface. Cannot continue.
[root@nfs02 audit]# tail -f audit.log|grep -Ei src
type=AVC msg=audit(1519003656.646:7601): avc: denied { name_bind } for pid=15452 comm="ganesha.nfsd" src=4501 scontext=system_u:system_r:ganesha_t:s0 tcontext=system_u:object_r:unreserved_port_t:s0 tclass=tcp_socket
If you get the above, use the following commands to allow the port:
audit2allow -a
audit2allow -a -M ganesha_4501_port
semodule -i ganesha_4501_port.pp
Alternate way that can work as well (In our test case here both methods were needed):
[root@nfs02 audit]# grep AVC /var/log/audit/audit.log | tail -n1 | audit2allow -M systemd-allow
******************** IMPORTANT ***********************
To make this policy package active, execute:
semodule -i systemd-allow.pp
[root@nfs02 audit]# grep AVC /var/log/audit/audit.log | tail -n1
type=AVC msg=audit(1519003656.646:7601): avc: denied { name_bind } for pid=15452 comm="ganesha.nfsd" src=4501 scontext=system_u:system_r:ganesha_t:s0 tcontext=system_u:object_r:unreserved_port_t:s0 tclass=tcp_socket
[root@nfs02 audit]# semodule -i systemd-allow.pp
libsemanage.add_user: user ipauser not in password file
[root@nfs02 audit]#
Restart and check:
[root@nfs02 audit]# systemctl stop nfs-ganesha.service; >/var/log/ganesha.log; systemctl start nfs-ganesha.service
[root@nfs02 audit]# firewall-cmd –zone=public –list-all
public
target: default
icmp-block-inversion: no
interfaces:
sources:
services: ssh dhcpv6-client
ports: 24007-24008/tcp 49152/tcp 38465-38469/tcp 111/tcp 111/udp 2049/tcp
protocols:
masquerade: no
forward-ports:
source-ports:
icmp-blocks:
rich rules:
[root@nfs02 audit]#
Do the same for DMZ rules as well. ( Test using nmap -oO -sS -sV nfs02 from the remote host when both dmz and public rules are added. )
firewall-cmd –zone=dmz –permanent –add-port=49152/tcp
firewall-cmd –zone=dmz –permanent –add-port=38465-38469/tcp
firewall-cmd –reload
firewall-cmd –zone=dmz –list-all
Let's enable logging of all firewall activity while we can't list mounts remotely:
[root@nfs02 ganesha]# firewall-cmd –set-log-denied=all
success
[root@nfs02 ganesha]#
[root@ipaclient01 ~]# showmount -e nfs02.nix.mine.dom
rpc mount export: RPC: Unable to receive; errno = No route to host
[root@ipaclient01 ~]#
Now check /var/log/messages for the real blocked port:
Feb 18 21:02:58 nfs02 kernel: FINAL_REJECT: IN=eth0 OUT= MAC=00:50:56:86:2d:21:00:50:56:86:d7:4c:08:00 SRC=192.168.0.236 DST=192.168.0.119 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=13717 DF PROTO=TCP SPT=921 DPT=20048 WINDOW=29200 RES=0x00 SYN URGP=0
Feb 18 21:02:58 nfs02 kernel: FINAL_REJECT: IN=eth0 OUT= MAC=00:50:56:86:2d:21:00:50:56:86:d7:4c:08:00 SRC=192.168.0.236 DST=192.168.0.119 LEN=116 TOS=0x00 PREC=0x00 TTL=64 ID=25816 DF PROTO=UDP SPT=921 DPT=20048 LEN=96
Feb 18 21:03:08 nfs02 kernel: FINAL_REJECT: IN=eth0 OUT= MAC=ff:ff:ff:ff:ff:ff:48:5d:60:cb:44:1c:08:00 SRC=192.168.0.102 DST=255.255.255.255 LEN=328 TOS=0x00 PREC=0x00 TTL=128 ID=2731 PROTO=UDP SPT=68 DPT=67 LEN=308
[root@nfs02 ganesha]# netstat -pnlt|grep -Ei 20048
tcp6 0 0 :::20048 :::* LISTEN 15674/ganesha.nfsd
[root@nfs02 ganesha]#
Sure enough, we need to set the F/W rule for this:
[root@nfs02 ganesha]# firewall-cmd –zone=public –permanent –add-port=20048/udp
success
[root@nfs02 ganesha]# firewall-cmd –reload
success
[root@nfs02 ganesha]#
But get this error instead:
[root@ipaclient01 ~]# showmount -e nfs02.nix.mine.dom
clnt_create: RPC: Port mapper failure – Unable to receive: errno 113 (No route to host)
[root@ipaclient01 ~]#
Feb 18 21:12:00 nfs02 kernel: FINAL_REJECT: IN=eth0 OUT= MAC=00:50:56:86:2d:21:00:50:56:86:d7:4c:08:00 SRC=192.168.0.236 DST=192.168.0.119 LEN=84 TOS=0x00 PREC=0x00 TTL=64 ID=25657 DF PROTO=UDP SPT=927 DPT=111 LEN=64
Feb 18 21:12:00 nfs02 kernel: FINAL_REJECT: IN=eth0 OUT= MAC=00:50:56:86:2d:21:00:50:56:86:d7:4c:08:00 SRC=192.168.0.236 DST=192.168.0.119 LEN=84 TOS=0x00 PREC=0x00 TTL=64 ID=25660 DF PROTO=UDP SPT=927 DPT=111 LEN=64
Add them all:
firewall-cmd –zone=public –permanent –add-port=2049/tcp
firewall-cmd –zone=public –permanent –add-port=111/tcp
firewall-cmd –zone=public –permanent –add-port=111/udp
firewall-cmd –zone=public –permanent –add-port=24007-24008/tcp
firewall-cmd –zone=public –permanent –add-port=49152/tcp
firewall-cmd –zone=public –permanent –add-port=38465-38469/tcp
firewall-cmd –zone=public –permanent –add-port=4501/tcp
firewall-cmd –zone=public –permanent –add-port=4501/udp
firewall-cmd –reload
Now try to connect from the client again:
[root@ipaclient01 ~]# showmount -e nfs02.nix.mine.dom
Export list for nfs02.nix.mine.dom:
/n (everyone)
[root@ipaclient01 ~]#
Works! Now let's try to mount as a true NFSv4 share:
[root@ipaclient01 ~]# df -h /n
Filesystem Size Used Avail Use% Mounted on
nfs02:/n 128G 43M 128G 1% /n
[root@ipaclient01 ~]# df -h /n|grep -Ei nfs4
[root@ipaclient01 ~]# mount|grep -Ei nfs0
nfs01:/gv01 on /mnt/gv01 type fuse.glusterfs (rw,relatime,user_id=0,group_id=0,default_permissions,allow_other,max_read=131072)
nfs02:/n on /n type nfs4 (rw,relatime,vers=4.0,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,port=0,timeo=600,retrans=2,sec=sys,clientaddr=192.168.0.236,local_lock=none,addr=192.168.0.119)
[root@ipaclient01 ~]#
Works like a charm! Now let's make this into a cluster by replicating it over to the nfs01.nix.mine.dom node. Let's run down the list very quickly.
The files:
[root@nfs01 gv01]# cat public.bash
firewall-cmd –zone=public –permanent –add-port=2049/tcp
firewall-cmd –zone=public –permanent –add-port=111/tcp
firewall-cmd –zone=public –permanent –add-port=111/udp
firewall-cmd –zone=public –permanent –add-port=24007-24008/tcp
firewall-cmd –zone=public –permanent –add-port=49152/tcp
firewall-cmd –zone=public –permanent –add-port=38465-38469/tcp
firewall-cmd –zone=public –permanent –add-port=4501/tcp
firewall-cmd –zone=public –permanent –add-port=4501/udp
firewall-cmd –zone=public –permanent –add-port=20048/udp
firewall-cmd –zone=public –permanent –add-port=20048/tcp
firewall-cmd –reload
[root@nfs01 gv01]# cat dmz.bash
firewall-cmd –zone=dmz –permanent –add-port=2049/tcp
firewall-cmd –zone=dmz –permanent –add-port=111/tcp
firewall-cmd –zone=dmz –permanent –add-port=111/udp
firewall-cmd –zone=dmz –permanent –add-port=24007-24008/tcp
firewall-cmd –zone=dmz –permanent –add-port=49152/tcp
firewall-cmd –zone=dmz –permanent –add-port=38465-38469/tcp
firewall-cmd –zone=dmz –permanent –add-port=4501/tcp
firewall-cmd –zone=dmz –permanent –add-port=4501/udp
firewall-cmd –zone=dmz –permanent –add-port=20048/tcp
firewall-cmd –zone=dmz –permanent –add-port=20048/udp
firewall-cmd –reload
[root@nfs01 gv01]# cat /etc/ganesha/ganesha.conf
###################################################
#
# EXPORT
#
# To function, all that is required is an EXPORT
#
# Define the absolute minimal export
#
###################################################
NFS_Core_Param {
NFS_Port = 2049;
MNT_Port = 20048;
NLM_Port = 38468;
Rquota_Port = 4501;
}
%include "/etc/ganesha/export.conf"
[root@nfs01 gv01]# cat /etc/ganesha/export.conf
EXPORT{
Export_Id = 1 ; # Export ID unique to each export
Path = "/n"; # Path of the volume to be exported. Eg: "/test_volume"
FSAL {
name = GLUSTER;
hostname = "nfs01.nix.mine.dom"; # IP of one of the nodes in the trusted pool
volume = "gv01"; # Volume name. Eg: "test_volume"
}
Access_type = RW; # Access permissions
Squash = No_root_squash; # To enable/disable root squashing
Disable_ACL = FALSE; # To enable/disable ACL
Pseudo = "/n"; # NFSv4 pseudo path for this export. Eg: "/test_volume_pseudo"
Protocols = "3","4"; # NFS protocols supported
Transports = "UDP","TCP" ; # Transport protocols supported
SecType = "sys"; # Security flavors supported
}
[root@nfs01 gv01]#
The commands for selinux in case things don't work:
grep AVC /var/log/audit/audit.log | tail -n1 | audit2allow -M systemd-allow
semodule -i systemd-allow.pp
systemctl stop nfs-ganesha.service; >/var/log/ganesha.log; systemctl start nfs-ganesha.service
audit2allow -a
audit2allow -a -M ganesha_4501_port
semodule -i ganesha_4501_port.pp
systemctl stop nfs-ganesha.service; >/var/log/ganesha.log; systemctl start nfs-ganesha.service
firewall-cmd –set-log-denied=all
firewall-cmd –reload
firewall-cmd –zone=dmz –list-all
firewall-cmd –zone=public –list-all
The result:
[root@ipaclient01 n]# showmount -e nfs02
Export list for nfs02:
/n (everyone)
[root@ipaclient01 n]# showmount -e nfs01
Export list for nfs01:
/n (everyone)
[root@ipaclient01 n]#
Perfect! Now we have to consider NFS Ganesha HA. There's a few solutions including having two of the same NFS servers both pointing to a common GlusterFS, then technically we only need a VIP and we're good. But we need to consider two options: HAPROXY / keepalived or NFS Ganesha HA. Alas, HAPROXY and keepalived was not to be on first attempt. We have a third party:
[root@nfs02 n]# yum install storhaug.noarch storhaug-nfs.noarch
But that brings in about 100 dependencies including pacemaker, corosync etc. The point of the gluster setup we want here is to avoid this mess. Plus there isn't much update to that storhaug project anyway. So let's skip that and try with HAPROXY and keepalived . On both nodes, install haproxy and keepalived using yum then configure each file with below contents:
[root@nfs01 ~]# cat /etc/haproxy/haproxy.cfg|grep -Evi "#"
global
log 127.0.0.1 local2
chroot /var/lib/haproxy
pidfile /var/run/haproxy.pid
maxconn 4000
user haproxy
group haproxy
daemon
stats socket /var/lib/haproxy/stats
defaults
mode http
log global
option httplog
option dontlognull
option http-server-close
option forwardfor except 127.0.0.0/8
option redispatch
retries 3
timeout http-request 10s
timeout queue 1m
timeout connect 10s
timeout client 1m
timeout server 1m
timeout http-keep-alive 10s
timeout check 10s
maxconn 3000
frontend main *:80
acl url_static path_beg -i /static /images /javascript /stylesheets
acl url_static path_end -i .jpg .gif .png .css .js
default_backend app
backend static
balance roundrobin
server static 127.0.0.1:4331 check
backend app
balance roundrobin
server nfs01.nix.mine.dom 192.168.0.131:80 check
server nfs02.nix.mine.dom 192.168.0.119:80 check
[root@nfs01 ~]#
Before we setup the below keepalived configuration, we need to set two binding options in the kernel (More on this.) :
[root@nfs02 keepalived]# echo "net.ipv4.ip_nonlocal_bind = 1" >> /etc/sysctl.conf
[root@nfs02 keepalived]# echo "net.ipv4.ip_forward = 1" >> /etc/sysctl.conf
[root@nfs02 keepalived]# sysctl -p
net.ipv4.ip_nonlocal_bind = 1
net.ipv4.ip_forward = 1
[root@nfs02 keepalived]#
Then define the keepalived.conf as follows:
[root@nfs01 ~]# cat /etc/keepalived/keepalived.conf
vrrp_script chk_haproxy {
script "killall -0 haproxy" # check the haproxy process
interval 2 # every 2 seconds
weight 2 # add 2 points if OK
}
vrrp_instance VI_1 {
interface eth0 # interface to monitor
state MASTER # MASTER on haproxy1, BACKUP on haproxy2
virtual_router_id 51
priority 101 # 101 on haproxy1, 100 on haproxy2
virtual_ipaddress {
192.168.0.80 # virtual ip address
}
track_script {
chk_haproxy
}
}
[root@nfs01 ~]#
Source document on which above was modeled can be found here. And create the DNS entries in the idmipa01 / idmipa02 FreeIPA servers we installed earlier. Despite this, the failover did not work and even after the host comes back, the NFSv4 fails to recognize when mounted over nfs-c01 VIP between nfs01 and nfs02: ( Later on, folks on the HAPROXY mailing list quickly corrected me that I'm using http instead of tcp. My bad! TY Guy's! )
[root@ipaclient01 ~]# cd /n
-bash: cd: /n: Stale file handle
[root@ipaclient01 ~]#
This is because NFSv4 is stateful and HAPROXY / keepalived might just be too simple for this but we'll try this later on. So we return to the NFS Ganesha method and earlier repos. We'll create a new repo with older GlusterFS binaries and install that. But that means we have to tear everything to bare bones again. Here are our options:
-
Download the older rpm's: wget –recursive –no-parent https://buildlogs.centos.org/centos/7/storage/x86_64/gluster-3.10/
-
(Optional) Create the repo. One might be already created after the download is finished.
-
Create the yum repo file:
[glusterfs-3.10]
name = GlusterFS 3.10 Packages
baseurl = file:///root/gluster-repo/buildlogs.centos.org/centos/7/storage/x86_64/gluster-3.10/
enabled = 1
gpgcheck = 1
-
Alternately, you can just point to the online repo location. Either way will do.
- Check that the older version is available using: yum –showduplicates list glusterfs
Trying the new configs while saving current and reinstalling nfs-ganesha to fix the earlier package problem. We will force the package because we notice that if you try to upgrade nfs-ganesha or update libntirpc, it results in the exact same message either way. Guessing they were made interdependent on each other and forcing libntirpc might allow us to user nfs-ganesha 2.5.3:
Error: Package: nfs-ganesha-2.5.5-1.el7.x86_64 (centos-gluster312)
Requires: libntirpc.so.1.5(NTIRPC_1.5.4)(64bit)
Rats! No luck. Ok, let's start from a clean slate again:
[root@nfs02 ganesha]# rpm -e $(rpm -aq|grep -Ei "ganesha|storhaug")
[root@nfs02 ganesha]# cat export.conf
EXPORT{
Export_Id = 1 ; # Export ID unique to each export
Path = "/n"; # Path of the volume to be exported. Eg: "/test_volume"
FSAL {
name = GLUSTER;
hostname = "nfs02.nix.mine.dom"; # IP of one of the nodes in the trusted pool
volume = "gv01"; # Volume name. Eg: "test_volume"
}
Access_type = RW; # Access permissions
Squash = No_root_squash; # To enable/disable root squashing
Disable_ACL = FALSE; # To enable/disable ACL
Pseudo = "/n"; # NFSv4 pseudo path for this export. Eg: "/test_volume_pseudo"
Protocols = "3","4"; # NFS protocols supported
Transports = "UDP","TCP" ; # Transport protocols supported
SecType = "sys"; # Security flavors supported
}
[root@nfs02 ganesha]# cat ganesha.conf
###################################################
#
# EXPORT
#
# To function, all that is required is an EXPORT
#
# Define the absolute minimal export
#
###################################################
NFS_Core_Param {
Bind_addr=192.168.0.119;
NFS_Port=2049;
MNT_Port=20048;
NLM_Port=38468;
Rquota_Port=4501;
}
%include "/etc/ganesha/export.conf"
[root@nfs02 ganesha]#
Now we install the trouble package only:
yum install libntirpc.x86_64
This worked without problems as long as we pick the N-1 version. Now we'll try to install nfs-ganesha again:
Success! However on an unexpected reboot, we run into this issue:
Feb 19 22:37:39 nfs01.nix.mine.dom haproxy-systemd-wrapper[2086]: haproxy-systemd-wrapper: executing /usr/sbin/haproxy -f /etc/haproxy/haproxy.cfg -p /run/haproxy.pid -Ds
Feb 19 22:37:39 nfs01.nix.mine.dom haproxy-systemd-wrapper[2086]: [ALERT] 049/223739 (2087) : Starting frontend nfs-in: cannot bind socket [192.168.0.80:2049]
And quickly find out it's selinux:
type=SERVICE_START msg=audit(1519098125.807:194): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=haproxy comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
type=AVC msg=audit(1519098125.826:195): avc: denied { name_bind } for pid=2389 comm="haproxy" src=2049 scontext=system_u:system_r:haproxy_t:s0 tcontext=system_u:object_r:nfs_port_t:s0 tclass=tcp_socket
type=SYSCALL msg=audit(1519098125.826:195): arch=c000003e syscall=49 success=no exit=-13 a0=5 a1=564f91774ed0 a2=10 a3=564f90eaf074 items=0 ppid=2388 pid=2389 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=4294967295 comm="haproxy" exe="/usr/sbin/haproxy" subj=system_u:system_r:haproxy_t:s0 key=(null)
type=PROCTITLE msg=audit(1519098125.826:195): proctitle=2F7573722F7362696E2F686170726F7879002D66002F6574632F686170726F78792F686170726F78792E636667002D70002F72756E2F686170726F78792E706964002D4473
type=SERVICE_STOP msg=audit(1519098125.842:196): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=haproxy comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=failed'
So we fix it in the same way we fixed the above issue:
grep AVC /var/log/audit/audit.log | tail -n1 | audit2allow -M systemd-allow
semodule -i systemd-allow.pp
And move on to testing further. We will try to start haproxy first then start nfs-ganesha which binds to all the ports when using the RPM version. We had to do this twice to get all the rules in. We do the same on nfs02 and noticed a similar error when trying to get HAPROXY going:
Feb 19 23:02:45 nfs02.nix.mine.dom haproxy-systemd-wrapper[12156]: [ALERT] 049/230245 (12157) : Starting frontend nfs-in: cannot bind socket [192.168.0.80:2049]
We try the selinux route but still get the same error. So we check using ip a and notice that the VIP from keepalived isn't up. So we start keepalived. Still doesn't work. So we give up on the RPM method.
We are now trying to compile from source. Links and packages you may need are below:
https://github.com/nfs-ganesha/nfs-ganesha/wiki/Compiling
https://github.com/nfs-ganesha/nfs-ganesha/wiki/GLUSTER
https://github.com/nfs-ganesha/nfs-ganesha/wiki/XFSLUSTRE
yum install glusterfs-api-devel.x86_64
yum install xfsprogs-devel.x86_64
yum install xfsprogs.x86_64
xfsdump-3.1.4-1.el7.x86_64
libguestfs-xfs-1.36.3-6.el7_4.3.x86_64
libntirpc-devel-1.5.4-1.el7.x86_64
libntirpc-1.5.4-1.el7.x86_64
COMMANDS
git clone https://github.com/nfs-ganesha/nfs-ganesha.git
cd nfs-ganesha;
git checkout V2.6-stable
git submodule update –init –recursive
ccmake /root/ganesha/nfs-ganesha/src/
# Press the c, e, c, g keys to create and generate the config and make files.
make
make install
Ensure you also enable IDMAPD support in NFS Ganesha or your UID / GID off the NFS mount will be nobody / nobody or nfsnobody / nfsnobody:
libnfsidmap-devel-0.25-17.el7.x86_64
jemalloc-devel-3.6.0-1.el7.x86_64
The options we want selected are as follows:
ALLOCATOR jemalloc
USE_FSAL_GLUSTER ON
USE_FSAL_XFS OFF
USE_NFSIDMAP ON
If you don't have above packages installed, you'll get an error message when selecting them via ccmake:
CMake Warning at CMakeLists.txt:698 (message):
jemalloc not found, falling back to libc
CMake Warning at CMakeLists.txt:745 (message):
libnfsidmap not found, disabling USE_NFSIDMAP
/root/nfs-ganesha/src/include/gsh_rpc.h:17:28: fatal error: rpc/xdr_inline.h: No such file or directory
#include <rpc/xdr_inline.h>
If we also don't get the nfsidmapd configured, we'll get error messages such as this in the messages and secure log files:
Mar 15 23:13:06 ipaclient01 nfsidmap[4999]: key: 0x3b3559c4 type: uid value: tom@mds.xyz@localdomain timeout 600
Mar 15 23:13:06 ipaclient01 nfsidmap[4999]: nfs4_name_to_uid: calling umich_ldap->name_to_uid
Mar 15 23:13:06 ipaclient01 nfsidmap[4999]: ldap_init_and_bind: version mismatch between API information and protocol version. Setting protocol version to 3
Mar 15 23:13:06 ipaclient01 nfsidmap[4999]: nfs4_name_to_uid: umich_ldap->name_to_uid returned -2
Mar 15 23:13:06 ipaclient01 nfsidmap[4999]: nfs4_name_to_uid: calling nsswitch->name_to_uid
Mar 15 23:13:06 ipaclient01 nfsidmap[4999]: nss_getpwnam: name 'tom@mds.xyz@localdomain' domain 'nix.my.dom': resulting localname '(null)'
Mar 15 23:13:06 ipaclient01 nfsidmap[4999]: nss_getpwnam: name 'tom@mds.xyz@localdomain' does not map into domain 'nix.my.dom'
Mar 15 23:13:06 ipaclient01 nfsidmap[4999]: nfs4_name_to_uid: nsswitch->name_to_uid returned -22
Mar 15 23:13:06 ipaclient01 nfsidmap[4999]: nfs4_name_to_uid: final return value is -22
Mar 15 23:13:06 ipaclient01 nfsidmap[4999]: nfs4_name_to_uid: calling umich_ldap->name_to_uid
Mar 15 23:13:06 ipaclient01 nfsidmap[4999]: ldap_init_and_bind: version mismatch between API information and protocol version. Setting protocol version to 3
Mar 15 23:13:06 ipaclient01 nfsidmap[4999]: nfs4_name_to_uid: umich_ldap->name_to_uid returned -2
Mar 15 23:13:06 ipaclient01 nfsidmap[4999]: nfs4_name_to_uid: calling nsswitch->name_to_uid
Mar 15 23:13:06 ipaclient01 nfsidmap[4999]: nss_getpwnam: name 'nobody@nix.my.dom' domain 'nix.my.dom': resulting localname 'nobody'
Mar 15 23:13:06 ipaclient01 nfsidmap[4999]: nfs4_name_to_uid: nsswitch->name_to_uid returned 0
Mar 15 23:13:06 ipaclient01 nfsidmap[4999]: nfs4_name_to_uid: final return value is 0
Mar 15 23:13:06 ipaclient01 nfsidmap[5001]: key: 0x3140cc17 type: gid value: tom@mds.xyz@localdomain timeout 600
Mar 15 23:13:06 ipaclient01 nfsidmap[5001]: nfs4_name_to_gid: calling umich_ldap->name_to_gid
Mar 15 23:13:06 ipaclient01 nfsidmap[5001]: ldap_init_and_bind: version mismatch between API information and protocol version. Setting protocol version to 3
Mar 15 23:13:06 ipaclient01 nfsidmap[5001]: nfs4_name_to_gid: umich_ldap->name_to_gid returned -2
Mar 15 23:13:06 ipaclient01 nfsidmap[5001]: nfs4_name_to_gid: calling nsswitch->name_to_gid
Mar 15 23:13:06 ipaclient01 nfsidmap[5001]: nfs4_name_to_gid: nsswitch->name_to_gid returned -22
Mar 15 23:13:06 ipaclient01 nfsidmap[5001]: nfs4_name_to_gid: final return value is -22
Mar 15 23:13:06 ipaclient01 nfsidmap[5001]: nfs4_name_to_gid: calling umich_ldap->name_to_gid
Mar 15 23:13:06 ipaclient01 nfsidmap[5001]: ldap_init_and_bind: version mismatch between API information and protocol version. Setting protocol version to 3
Mar 15 23:13:06 ipaclient01 nfsidmap[5001]: nfs4_name_to_gid: umich_ldap->name_to_gid returned -2
Mar 15 23:13:06 ipaclient01 nfsidmap[5001]: nfs4_name_to_gid: calling nsswitch->name_to_gid
Mar 15 23:13:06 ipaclient01 nfsidmap[5001]: nfs4_name_to_gid: nsswitch->name_to_gid returned 0
Mar 15 23:13:06 ipaclient01 nfsidmap[5001]: nfs4_name_to_gid: final return value is 0
A successful resolution looks like this:
Mar 18 10:28:28 ipaclient01 nfsidmap[19982]: key: 0x23cecc27 type: uid value: root@nix.mds.xyz timeout 600
Mar 18 10:28:28 ipaclient01 nfsidmap[19982]: nfs4_name_to_uid: calling nsswitch->name_to_uid
Mar 18 10:28:28 ipaclient01 nfsidmap[19982]: nss_getpwnam: name 'root@nix.mds.xyz' domain 'nix.mds.xyz': resulting localname 'root'
Mar 18 10:28:28 ipaclient01 nfsidmap[19982]: nss_name_to_uid: name 'root@nix.mds.xyz' uid 0
Mar 18 10:28:28 ipaclient01 nfsidmap[19982]: nfs4_name_to_uid: nsswitch->name_to_uid returned 0
Mar 18 10:28:28 ipaclient01 nfsidmap[19982]: nfs4_name_to_uid: final return value is 0
Mar 18 10:28:28 ipaclient01 nfsidmap[19984]: key: 0x3b0d8068 type: gid value: root@nix.mds.xyz timeout 600
Mar 18 10:28:28 ipaclient01 nfsidmap[19984]: nfs4_name_to_gid: calling nsswitch->name_to_gid
Mar 18 10:28:28 ipaclient01 nfsidmap[19984]: nss_name_to_gid: name 'root@nix.mds.xyz' domain 'nix.mds.xyz': resulting localname 'root'
Mar 18 10:28:28 ipaclient01 nfsidmap[19984]: nss_name_to_gid: name 'root@nix.mds.xyz' gid 0
Mar 18 10:28:28 ipaclient01 nfsidmap[19984]: nfs4_name_to_gid: nsswitch->name_to_gid returned 0
Mar 18 10:28:28 ipaclient01 nfsidmap[19984]: nfs4_name_to_gid: final return value is 0
Mar 18 10:28:28 ipaclient01 nfsidmap[19988]: key: 0x2adbb426 type: uid value: tom@mds.xyz@nix.mds.xyz timeout 600
Mar 18 10:28:28 ipaclient01 nfsidmap[19988]: nfs4_name_to_uid: calling nsswitch->name_to_uid
Mar 18 10:28:28 ipaclient01 nfsidmap[19988]: nss_getpwnam: name 'tom@mds.xyz@nix.mds.xyz' domain 'nix.mds.xyz': resulting localname 'tom@mds.xyz'
Mar 18 10:28:28 ipaclient01 nfsidmap[19988]: nss_name_to_uid: name 'tom@mds.xyz@nix.mds.xyz' uid 155601104
Mar 18 10:28:28 ipaclient01 nfsidmap[19988]: nfs4_name_to_uid: nsswitch->name_to_uid returned 0
Mar 18 10:28:28 ipaclient01 nfsidmap[19988]: nfs4_name_to_uid: final return value is 0
Mar 18 10:28:28 ipaclient01 nfsidmap[19990]: key: 0x6ef7be8 type: gid value: tom@mds.xyz@nix.mds.xyz timeout 600
Mar 18 10:28:28 ipaclient01 nfsidmap[19990]: nfs4_name_to_gid: calling nsswitch->name_to_gid
Mar 18 10:28:28 ipaclient01 nfsidmap[19990]: nss_name_to_gid: name 'tom@mds.xyz@nix.mds.xyz' domain 'nix.mds.xyz': resulting localname 'tom@mds.xyz'
Mar 18 10:28:28 ipaclient01 nfsidmap[19990]: nss_name_to_gid: name 'tom@mds.xyz@nix.mds.xyz' gid 155601104
Mar 18 10:28:28 ipaclient01 nfsidmap[19990]: nfs4_name_to_gid: nsswitch->name_to_gid returned 0
Mar 18 10:28:28 ipaclient01 nfsidmap[19990]: nfs4_name_to_gid: final return value is 0
When creating the config file, ensure to select GLUSTER FS:
[root@nfs02 nfs-ganesha]# ccmake /root/ganesha/nfs-ganesha/src
[root@nfs02 nfs-ganesha]# pwd
/root/ganesha/nfs-ganesha
[root@nfs02 nfs-ganesha]#
We successfully compiled from source the latest available version. But now we get:
20/02/2018 08:56:51 : epoch 5a8c2923 : nfs01.nix.mine.dom : ganesha.nfsd-25728[main] load_fsal :NFS STARTUP :CRIT :Could not dlopen module:/usr/lib64/ganesha/libfsalgluster.so Error:/usr/lib64/ganesha/libfsalgluster.so:
So we need to copy the folling binaries to the appropriate folder and link the versions accordingly (Wonder why it didn't do it on it's own.):
[root@nfs01 src]# find / -iname libfsalgluster*
/root/ganesha/nfs-ganesha/src/FSAL/FSAL_GLUSTER/libfsalgluster.so.4.2.0
/root/ganesha/nfs-ganesha/src/FSAL/FSAL_GLUSTER/libfsalgluster.so.4
/root/ganesha/nfs-ganesha/src/FSAL/FSAL_GLUSTER/libfsalgluster.so
[root@nfs01 src]#
End state is as follows with proper symlinks created (on both nodes):
537739 -rwxr-xr-x. 1 root root 507824 Feb 20 08:55 libfsalgluster.so.4.2.0
1186043 lrwxrwxrwx. 1 root root 23 Feb 20 21:16 libfsalgluster.so.4 -> libfsalgluster.so.4.2.0
1186045 lrwxrwxrwx. 1 root root 23 Feb 20 21:16 libfsalgluster.so -> libfsalgluster.so.4.2.0
544176 drwxr-xr-x. 2 root root 4096 Feb 20 21:16 .
[root@nfs01 ganesha]# pwd
/usr/lib64/ganesha
[root@nfs01 ganesha]#
Then start up ganesha.nfsd again. This time the logs are clear:
Let's also add XFS support since we use it very often. We try but get the following despite these installed packages:
CMake Warning at CMakeLists.txt:671 (message):
Cannot find XFS runtime. Disabling XFS build
Let's investigate what library we need to enable XFS support:
/root/ganesha/nfs-ganesha/src/CMakeLists.txt
[root@nfs01 nfs-ganesha]# grep -EiR "Cannot find XFS runtime. Disabling XFS build" *
src/CMakeLists.txt: message(FATAL_ERROR "STRICT_PACKAGE: Cannot find XFS runtime. Disabling XFS build")
src/CMakeLists.txt: message(WARNING "Cannot find XFS runtime. Disabling XFS build")
[root@nfs01 nfs-ganesha]#
if(USE_FSAL_XFS)
if(EXISTS /lib/libhandle.so)
check_library_exists(handle "open_by_handle" "/./lib" HAVE_XFS_LIB)
if(HAVE_XFS_LIB)
set(PATH_LIBHANDLE "/lib/libhandle.so" CACHE INTERNAL "debian stretch and ubuntu xenial hack")
endif(HAVE_XFS_LIB)
else(EXISTS /lib/libhandle.so)
check_library_exists(handle "open_by_handle" "" HAVE_XFS_LIB)
endif(EXISTS /lib/libhandle.so)
check_include_files("xfs/xfs.h" HAVE_XFS_H)
if((NOT HAVE_XFS_LIB) OR (NOT HAVE_XFS_H))
if(STRICT_PACKAGE)
message(FATAL_ERROR "STRICT_PACKAGE: Cannot find XFS runtime. Disabling XFS build")
else(STRICT_PACKAGE)
message(WARNING "Cannot find XFS runtime. Disabling XFS build")
set(USE_FSAL_XFS OFF)
endif(STRICT_PACKAGE)
endif((NOT HAVE_XFS_LIB) OR (NOT HAVE_XFS_H))
endif(USE_FSAL_XFS)
So we need to ensure /lib/libhandle.so is installed. Let's check that then:
[root@nfs01 src]# ls -altri /lib/libhandle.so
ls: cannot access /lib/libhandle.so: No such file or directory
[root@nfs01 src]# find / -iname libhandle.so
/usr/lib64/libhandle.so
[root@nfs01 src]# ls -altri /usr/lib64/libhandle.so
203702437 lrwxrwxrwx. 1 root root 14 Feb 20 02:40 /usr/lib64/libhandle.so -> libhandle.so.1
[root@nfs01 src]# cd /usr/lib64/
[root@nfs01 lib64]# rpm -aq|grep -Ei ^C
[root@nfs01 lib64]# rpm -qf /usr/lib64/libhandle.so
xfsprogs-devel-4.5.0-12.el7.x86_64
[root@nfs01 lib64]# rpm -aq|grep -Ei xfs
xfsprogs-4.5.0-12.el7.x86_64
xfsprogs-devel-4.5.0-12.el7.x86_64
xfsdump-3.1.4-1.el7.x86_64
libguestfs-xfs-1.36.3-6.el7_4.3.x86_64
[root@nfs01 lib64]#
Apparently there's a slight typo in the config file? Let's check further with a find on xfs.h using find / -iname xfs.h:
[root@nfs01 src]# find / -iname xfs.h
/usr/include/xfs/xfs.h
[root@nfs01 src]#
Appears our /usr/include isn't right somewhere? Until we can get this from an RPM package, we'll have to go without NFS Ganesha. No idea, so let's just start ganesha.nfsd without XFS for now pending an email to the nfs-ganesha mailing lists. Sure enough, in the latest version 2.60, when compiled from source, binding to a single IP address has been permanently fixed:
[root@nfs01 ganesha]# netstat -pnlt|grep -Ei ganesha
tcp6 0 0 192.168.0.131:20048 :::* LISTEN 13714/ganesha.nfsd
tcp6 0 0 :::564 :::* LISTEN 13714/ganesha.nfsd
tcp6 0 0 192.168.0.131:4501 :::* LISTEN 13714/ganesha.nfsd
tcp6 0 0 192.168.0.131:2049 :::* LISTEN 13714/ganesha.nfsd
tcp6 0 0 192.168.0.131:38468 :::* LISTEN 13714/ganesha.nfsd
[root@nfs01 ganesha]#
Check also the second node:
[root@nfs02 ganesha]# netstat -pnlt|grep -Ei ganesha
tcp6 0 0 192.168.0.119:20048 :::* LISTEN 12409/ganesha.nfsd
tcp6 0 0 :::564 :::* LISTEN 12409/ganesha.nfsd
tcp6 0 0 192.168.0.119:4501 :::* LISTEN 12409/ganesha.nfsd
tcp6 0 0 192.168.0.119:2049 :::* LISTEN 12409/ganesha.nfsd
tcp6 0 0 192.168.0.119:38468 :::* LISTEN 12409/ganesha.nfsd
[root@nfs02 ganesha]#
Let's start up haproxy configuration we used above and see if we can now bind appropriately:
[root@nfs02 haproxy]# netstat -pnlt|grep -Ei haproxy
tcp 0 0 192.168.0.80:2049 0.0.0.0:* LISTEN 13652/haproxy
[root@nfs02 haproxy]#
Bingo! Works, check further:
[root@nfs01 ganesha]# netstat -pnlt|grep -Ei haproxy
tcp 0 0 192.168.0.80:2049 0.0.0.0:* LISTEN 16014/haproxy
[root@nfs01 ganesha]# netstat -pnlt|grep -Ei ganesha
tcp6 0 0 192.168.0.131:20048 :::* LISTEN 13714/ganesha.nfsd
tcp6 0 0 :::564 :::* LISTEN 13714/ganesha.nfsd
tcp6 0 0 192.168.0.131:4501 :::* LISTEN 13714/ganesha.nfsd
tcp6 0 0 192.168.0.131:2049 :::* LISTEN 13714/ganesha.nfsd
tcp6 0 0 192.168.0.131:38468 :::* LISTEN 13714/ganesha.nfsd
[root@nfs01 ganesha]#
Perfect! Now we have the two products working properly. Time to test mounting on clients and failover testing. In our case we see an error with gluster on the second cluster node so we need to fix this first:
[root@nfs02 n]# gluster volume status
Status of volume: gv01
Gluster process TCP Port RDMA Port Online Pid
——————————————————————————
Brick nfs01:/bricks/0/gv01 49152 0 Y 1377
Brick nfs02:/bricks/0/gv01 N/A N/A N N/A
Self-heal Daemon on localhost N/A N/A Y 1193
Self-heal Daemon on nfs01 N/A N/A Y 1151
Task Status of Volume gv01
——————————————————————————
There are no active volume tasks
[root@nfs02 n]#
Run a restart then check gluster status again:
[root@nfs02 n]# systemctl restart glusterd
[root@nfs02 n]# gluster volume status
Status of volume: gv01
Gluster process TCP Port RDMA Port Online Pid
——————————————————————————
Brick nfs01:/bricks/0/gv01 49152 0 Y 1377
Brick nfs02:/bricks/0/gv01 49152 0 Y 16103
Self-heal Daemon on localhost N/A N/A Y 16094
Self-heal Daemon on nfs01 N/A N/A Y 1151
Task Status of Volume gv01
——————————————————————————
There are no active volume tasks
[root@nfs02 n]#
March 23 2018
So we encounter this issue and our nfs-c01 VIP can't be used to mount from clients:
[2018-03-23 05:01:56.944175] E [MSGID: 114058] [client-handshake.c:1565:client_query_portmap_cbk] 0-gv01-client-0: failed to get the port number for remote subvolume. Please run 'gluster volume status' on server to see if brick process is running.
[2018-03-23 05:01:56.944921] I [MSGID: 114018] [client.c:2285:client_rpc_notify] 0-gv01-client-0: disconnected from gv01-client-0. Client process will keep trying to connect to glusterd until brick's port is available
[2018-03-23 05:01:56.944962] E [MSGID: 108006] [afr-common.c:5006:__afr_handle_child_down_event] 0-gv01-replicate-0: All subvolumes are down. Going offline until atleast one of them comes back up.
Final graph:
+——————————————————————————+
1: volume gv01-client-0
2: type protocol/client
3: option ping-timeout 42
4: option remote-host nfs01
5: option remote-subvolume /bricks/0/gv01
6: option transport-type socket
7: option transport.address-family inet
8: option username 916ccf06-dc1d-467f-bc3d-f00a7449618f
9: option password a44739e0-9587-411f-8e6a-9a6a4e46156c
10: option event-threads 8
11: option transport.tcp-user-timeout 0
12: option transport.socket.keepalive-time 20
13: option transport.socket.keepalive-interval 2
14: option transport.socket.keepalive-count 9
15: option send-gids true
16: end-volume
17:
18: volume gv01-client-1
19: type protocol/client
20: option ping-timeout 42
21: option remote-host nfs02
22: option remote-subvolume /bricks/0/gv01
23: option transport-type socket
24: option transport.address-family inet
25: option username 916ccf06-dc1d-467f-bc3d-f00a7449618f
26: option password a44739e0-9587-411f-8e6a-9a6a4e46156c
27: option event-threads 8
28: option transport.tcp-user-timeout 0
29: option transport.socket.keepalive-time 20
30: option transport.socket.keepalive-interval 2
31: option transport.socket.keepalive-count 9
32: option send-gids true
33: end-volume
34:
35: volume gv01-replicate-0
36: type cluster/replicate
37: option afr-pending-xattr gv01-client-0,gv01-client-1
38: option quorum-type auto
39: option use-compound-fops off
40: subvolumes gv01-client-0 gv01-client-1
41: end-volume
42:
43: volume gv01-dht
44: type cluster/distribute
45: option lock-migration off
46: subvolumes gv01-replicate-0
47: end-volume
48:
49: volume gv01-write-behind
50: type performance/write-behind
51: option cache-size 8MB
52: subvolumes gv01-dht
53: end-volume
54:
55: volume gv01-read-ahead
[2018-03-23 05:01:56.950848] E [MSGID: 114058] [client-handshake.c:1565:client_query_portmap_cbk] 0-gv01-client-1: failed to get the port number for remote subvolume. Please run 'gluster volume status' on server to see if brick process is running.
56: type performance/read-ahead
57: subvolumes gv01-write-behind
58: end-volume
59:
60: volume gv01-readdir-ahead
61: type performance/readdir-ahead
62: option parallel-readdir off
63: option rda-request-size 131072
64: option rda-cache-limit 10MB
65: subvolumes gv01-read-ahead
66: end-volume
[2018-03-23 05:01:56.950954] I [MSGID: 114018] [client.c:2285:client_rpc_notify] 0-gv01-client-1: disconnected from gv01-client-1. Client process will keep trying to connect to glusterd until brick's port is available
67:
[2018-03-23 05:01:56.951029] E [MSGID: 108006] [afr-common.c:5006:__afr_handle_child_down_event] 0-gv01-replicate-0: All subvolumes are down. Going offline until atleast one of them comes back up.
68: volume gv01-io-cache
69: type performance/io-cache
70: option cache-size 1GB
71: subvolumes gv01-readdir-ahead
72: end-volume
73:
74: volume gv01-quick-read
75: type performance/quick-read
76: option cache-size 1GB
77: subvolumes gv01-io-cache
78: end-volume
79:
80: volume gv01-open-behind
81: type performance/open-behind
82: subvolumes gv01-quick-read
83: end-volume
84:
85: volume gv01-md-cache
86: type performance/md-cache
87: subvolumes gv01-open-behind
88: end-volume
89:
90: volume gv01
91: type debug/io-stats
92: option log-level INFO
93: option latency-measurement off
94: option count-fop-hits off
95: subvolumes gv01-md-cache
96: end-volume
97:
98: volume meta-autoload
99: type meta
100: subvolumes gv01
101: end-volume
102:
+——————————————————————————+
[2018-03-23 05:01:56.952617] I [io-stats.c:4076:fini] 0-gv01: io-stats translator unloaded
[2018-03-23 05:01:56.953591] I [MSGID: 101191] [event-epoll.c:644:event_dispatch_epoll_worker] 0-epoll: Exited thread with index 1
[2018-03-23 05:01:56.954172] I [MSGID: 101191] [event-epoll.c:644:event_dispatch_epoll_worker] 0-epoll: Exited thread with index 8
^C
[root@nfs02 ganesha]#
We simply run:
mount -a
then check if gluster is mounted on both nfs01 and nfs02:
[root@nfs01 ~]# mount|grep glusterfs
nfs01:/gv01 on /n type fuse.glusterfs (rw,relatime,user_id=0,group_id=0,default_permissions,allow_other,max_read=131072)
[root@nfs01 ~]#
Potentially restart NFS-Ganesha on both clients one by one to allow the mounts to work. If it still doesn't work, it could be because of this:
[root@nfs02 glusterfs]# nc -v 192.168.0.131 49154
Ncat: Version 6.40 ( http://nmap.org/ncat )
Ncat: No route to host.
[root@nfs02 glusterfs]#
[root@nfs02 glusterfs]#
[root@nfs02 glusterfs]#
[root@nfs02 glusterfs]# systemctl stop firewalld
[root@nfs02 glusterfs]# nc -v 192.168.0.131 49154
Ncat: Version 6.40 ( http://nmap.org/ncat )
Ncat: Connected to 192.168.0.131:49154.
^C
[root@nfs02 glusterfs]#
And this in the gluster logs:
[2018-03-23 06:00:54.425845] I [rpc-clnt.c:1986:rpc_clnt_reconfig] 0-gv01-client-0: changing port to 49154 (from 0)
[2018-03-23 06:00:54.433214] E [socket.c:2369:socket_connect_finish] 0-gv01-client-0: connection to 192.168.0.131:49154 failed (No route to host); disconnecting socket
Looks like our port may be too restrictive. Adjust it.
Here's a summary configuration for this whole work:
| HOST | SETTING | DESCRIPTION |
|---|---|---|
| nfs01 / nfs02 |
Create and reserve some IP's for your hosts. We are using the FreeIPA project to provide DNS and Kerberos functionality here:
192.168.0.80 nfs-c01 (nfs01, nfs02) VIP DNS Entry |
Add the hosts to your DNS server for a clean setup. Alternately add them to /etc/hosts (ugly) |
| nfs01 / nfs02 |
wget https://github.com/nfs-ganesha/nfs-ganesha/archive/V2.6-.0.tar.gz
[root@nfs01 ~]# ganesha.nfsd -v DETAILED INSTRUCTIONS: https://github.com/nfs-ganesha/nfs-ganesha/wiki/Compiling
https://github.com/nfs-ganesha/nfs-ganesha/wiki/GLUSTER PACKAGES:
yum install glusterfs-api-devel.x86_64 jemalloc-devel-3.6.0-1.el7.x86_64 COMMANDS
git clone https://github.com/nfs-ganesha/nfs-ganesha.git
|
Compile and build nfsganesha 2.60+ from source. (At this time RPM packages did not work) Install the listed packages before compiling as well.
|
| nfs01 / nfs02 | Add a disk to the VM such as /dev/sdb . |
Add secondary disk for the shared GlusterFS |
| nfs01 / nfs02 |
yum install centos-release-gluster
gluster volume info |
Configure the GlusterFS filesystem using |
| nfs01 / nfs02 |
PACKAGES: /etc/haproxy/haproxy.cfg
global
defaults
frontend nfs-in
|
Install and Configure HAPROXY. A great source that
helped with this part. |
| nfs01 / nfs02 |
# echo "net.ipv4.ip_nonlocal_bind = 1" >> /etc/sysctl.conf # echo "net.ipv4.ip_forward = 1" >> /etc/sysctl.conf # sysctl -p net.ipv4.ip_nonlocal_bind = 1 net.ipv4.ip_forward = 1 # |
Turn on kernel parameters. These allow keepalived below to function properly. |
| nfs01 / nfs02 |
PACKAGES: yum install keepalived # ( Used 1.3.5-1.el7.x86_64 in this case ) NFS01:
vrrp_script chk_haproxy {
vrrp_instance VI_1 { NFS02:
vrrp_script chk_haproxy {
vrrp_instance VI_1 { |
Configure keepalived. A great source that helped with this as well. |
| nfs01 / nfs02 |
# cat public.bash firewall-cmd –zone=public –permanent –add-port=2049/tcp firewall-cmd –zone=public –permanent –add-port=111/tcp firewall-cmd –zone=public –permanent –add-port=111/udp firewall-cmd –zone=public –permanent –add-port=24007-24008/tcp firewall-cmd –zone=public –permanent –add-port=49152/tcp firewall-cmd –zone=public –permanent –add-port=38465-38469/tcp firewall-cmd –zone=public –permanent –add-port=4501/tcp firewall-cmd –zone=public –permanent –add-port=4501/udp firewall-cmd –zone=public –permanent –add-port=20048/udp
firewall-cmd –zone=public –permanent –add-port=20048/tcp
firewall-cmd –zone=public –permanent –add-port=49000-59999/udp
# cat dmz.bash firewall-cmd –zone=dmz –permanent –add-port=2049/tcp firewall-cmd –zone=dmz –permanent –add-port=111/tcp firewall-cmd –zone=dmz –permanent –add-port=111/udp firewall-cmd –zone=dmz –permanent –add-port=24007-24008/tcp firewall-cmd –zone=dmz –permanent –add-port=49152/tcp firewall-cmd –zone=dmz –permanent –add-port=38465-38469/tcp firewall-cmd –zone=dmz –permanent –add-port=4501/tcp firewall-cmd –zone=dmz –permanent –add-port=4501/udp firewall-cmd –zone=dmz –permanent –add-port=20048/tcp
firewall-cmd –zone=dmz –permanent –add-port=20048/udp
firewall-cmd –zone=dmz –permanent –add-port=49000-59999/tcp # # On Both
firewall-cmd –permanent –direct –add-rule ipv4 filter INPUT 0 -m pkttype –pkt-type multicast -j ACCEPT HANDY STUFF:
firewall-cmd –zone=dmz –list-all |
Configure firewalld. DO NOT disable firewalld . |
| nfs01 / nfs02 |
Run any of the following command, or a combination of, on deny entries in /var/log/audit/audit.log that may appear as you stop, start or install above services:
METHOD 1:
METHOD 2: |
Configure selinux. Don't disable it. This actually makes your host safer and is actually easy to work with using just these commands. |
| nfs01 / nfs02 |
NODE 1:
[root@nfs01 ~]# cat /etc/ganesha/ganesha.conf
%include "/etc/ganesha/export.conf"
FSAL {
Access_type = RW; # Access permissions NODE 2:
[root@nfs02 ~]# cd /etc/ganesha/
%include "/etc/ganesha/export.conf"
FSAL {
Access_type = RW; # Access permissions STARTUP:
/usr/bin/ganesha.nfsd -L /var/log/ganesha/ganesha.log -f /etc/ganesha/ganesha.conf -N NIV_EVENT |
Configure NFS Ganesha |
| nfs01 / nfs02 |
[root@nfs01 ~]# cat /etc/fstab|grep -Ei "brick|gv01"
[root@nfs01 ~]# mount|grep -Ei "brick|gv01"
[root@nfs01 ~]# ps -ef|grep -Ei "haproxy|keepalived|ganesha"; netstat -pnlt|grep -Ei "haproxy|ganesha|keepalived"
|
Ensure mounts are done and everything is started up. |
| nfs01 / nfs02 |
yumdownloader nfs-ganesha.x86_64 Copy above to the same folders under / instead of ./ :
systemctl enable nfs-ganesha.service |
Since you compiled from source you don't have nice startup scripts. To get your nice startup scripts from an existing ganesha RPM do the following. Then use systemctl to stop and start nfs-ganesha as you would any other service. |
Now let's do some checks on our NFS HA. Mount the share using the VIP from a client then create a test file:
[root@ipaclient01 /]# mount -t nfs4 nfs-c01:/n /n
[root@ipaclient01 n]# echo -ne "Hacked It. Gluster, NFS Ganesha, HAPROXY, keepalived scalable NFS server." > some-people-find-this-awesome.txt
[root@ipaclient01 n]# mount|grep nfs4
nfs-c01:/n on /n type nfs4 (rw,relatime,vers=4.0,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,port=0,timeo=600,retrans=2,sec=sys,clientaddr=192.168.0.236,local_lock=none,addr=192.168.0.80)
[root@ipaclient01 n]#
Then check each brick to see if the file was replicated:
[root@nfs01 n]# cat /bricks/0/gv01/some-people-find-this-awesome.txt
Hacked It. Gluster, NFS Ganesha, HAPROXY, keepalived scalable NFS server.
[root@nfs01 n]# mount|grep -Ei gv01
nfs01:/gv01 on /n type fuse.glusterfs (rw,relatime,user_id=0,group_id=0,default_permissions,allow_other,max_read=131072)
[root@nfs01 n]#
[root@nfs02 n]# cat /bricks/0/gv01/some-people-find-this-awesome.txt
Hacked It. Gluster, NFS Ganesha, HAPROXY, keepalived scalable NFS server.
[root@nfs02 n]# mount|grep -Ei gv01
nfs02:/gv01 on /n type fuse.glusterfs (rw,relatime,user_id=0,group_id=0,default_permissions,allow_other,max_read=131072)
[root@nfs02 n]#
Good! Now let's hard shutdown one node, nfs01, the primary node. Expected behaviour is that we need to see failover to nfs02 and then when we bring back the nfs01 server, we need to see the file is replicated. While we do this, the client ipaclient01 is not supposed to loose any connection to the NFS mount via the VIP. Here are the results:
[root@nfs02 n]# ps -ef|grep -Ei "haproxy|ganesha|keepalived"
root 12245 1 0 Feb19 ? 00:00:03 /usr/sbin/keepalived -D
root 12246 12245 0 Feb19 ? 00:00:03 /usr/sbin/keepalived -D
root 12247 12245 0 Feb19 ? 00:00:41 /usr/sbin/keepalived -D
root 12409 1 16 Feb20 ? 00:13:05 /usr/bin/ganesha.nfsd -L /var/log/ganesha/ganesha.log -f /etc/ganesha/ganesha.conf -N NIV_EVENT
root 17892 1 0 00:37 ? 00:00:00 /usr/sbin/haproxy-systemd-wrapper -f /etc/haproxy/haproxy.cfg -p /run/haproxy.pid
haproxy 17893 17892 0 00:37 ? 00:00:00 /usr/sbin/haproxy -f /etc/haproxy/haproxy.cfg -p /run/haproxy.pid -Ds
haproxy 17894 17893 0 00:37 ? 00:00:00 /usr/sbin/haproxy -f /etc/haproxy/haproxy.cfg -p /run/haproxy.pid -Ds
root 17918 21084 0 00:38 pts/0 00:00:00 grep –color=auto -Ei haproxy|ganesha|keepalived
[root@nfs02 n]# ps -ef|grep -Ei "haproxy|ganesha|keepalived"; netstat -pnlt|grep -Ei ganesha; netstat -pnlt|grep -Ei haproxy; netstat -pnlt|grep -Ei keepalived
root 12245 1 0 Feb19 ? 00:00:03 /usr/sbin/keepalived -D
root 12246 12245 0 Feb19 ? 00:00:03 /usr/sbin/keepalived -D
root 12247 12245 0 Feb19 ? 00:00:41 /usr/sbin/keepalived -D
root 12409 1 16 Feb20 ? 00:13:09 /usr/bin/ganesha.nfsd -L /var/log/ganesha/ganesha.log -f /etc/ganesha/ganesha.conf -N NIV_EVENT
root 17892 1 0 00:37 ? 00:00:00 /usr/sbin/haproxy-systemd-wrapper -f /etc/haproxy/haproxy.cfg -p /run/haproxy.pid
haproxy 17893 17892 0 00:37 ? 00:00:00 /usr/sbin/haproxy -f /etc/haproxy/haproxy.cfg -p /run/haproxy.pid -Ds
haproxy 17894 17893 0 00:37 ? 00:00:00 /usr/sbin/haproxy -f /etc/haproxy/haproxy.cfg -p /run/haproxy.pid -Ds
root 17947 21084 0 00:38 pts/0 00:00:00 grep –color=auto -Ei haproxy|ganesha|keepalived
tcp6 0 0 192.168.0.119:20048 :::* LISTEN 12409/ganesha.nfsd
tcp6 0 0 :::564 :::* LISTEN 12409/ganesha.nfsd
tcp6 0 0 192.168.0.119:4501 :::* LISTEN 12409/ganesha.nfsd
tcp6 0 0 192.168.0.119:2049 :::* LISTEN 12409/ganesha.nfsd
tcp6 0 0 192.168.0.119:38468 :::* LISTEN 12409/ganesha.nfsd
tcp 0 0 192.168.0.80:2049 0.0.0.0:* LISTEN 17894/haproxy
[root@nfs02 n]#
[root@nfs02 n]#
[root@nfs02 n]#
[root@nfs02 n]# ssh nfs-c01
Password:
Last login: Wed Feb 21 00:37:28 2018 from nfs-c01.nix.mine.dom
[root@nfs02 ~]# logout
Connection to nfs-c01 closed.
[root@nfs02 n]#
From client we can still see all the files (seemless with no interruption to the NFS service). As a bonus, while we started this first test, we noticed that HAPROXY was offline on nfs02. While trying to list the client files, it appeared hung but still responded then listed files right after we started HAPROXY on nfs02:
[root@ipaclient01 n]# ls -altri some-people-find-this-awesome.txt
11782527620043058273 -rw-r–r–. 1 nobody nobody 74 Feb 21 00:26 some-people-find-this-awesome.txt
[root@ipaclient01 n]# df -h .
Filesystem Size Used Avail Use% Mounted on
nfs-c01:/n 128G 43M 128G 1% /n
[root@ipaclient01 n]# ssh nfs-c01
Password:
Last login: Wed Feb 21 00:41:06 2018 from nfs-c01.nix.mine.dom
[root@nfs02 ~]#
Checking the gluster volume on nfs02:
[root@nfs02 n]# gluster volume status
Status of volume: gv01
Gluster process TCP Port RDMA Port Online Pid
——————————————————————————
Brick nfs02:/bricks/0/gv01 49152 0 Y 16103
Self-heal Daemon on localhost N/A N/A Y 16094
Task Status of Volume gv01
——————————————————————————
There are no active volume tasks
[root@nfs02 n]#
Now let's bring back the first node and fail the second after nfs01 is up again. As soon as we bring nfs01 back up, the VIP fails over to nfs01 without any hickup or manual invervention on the client end:
[root@ipaclient01 n]# ls -altri
total 11
128 dr-xr-xr-x. 21 root root 4096 Feb 18 22:24 ..
11782527620043058273 -rw-r–r–. 1 nobody nobody 74 Feb 21 00:26 some-people-find-this-awesome.txt
1 drwxr-xr-x. 3 nobody nobody 4096 Feb 21 00:26 .
[root@ipaclient01 n]#
[root@ipaclient01 n]#
[root@ipaclient01 n]#
[root@ipaclient01 n]# ssh nfs-c01
Password:
Last login: Wed Feb 21 00:59:56 2018
[root@nfs01 ~]#
So now let's fail the second node. NFS still works:
[root@ipaclient01 ~]# ssh nfs-c01
Password:
Last login: Wed Feb 21 01:31:50 2018
[root@nfs01 ~]# logout
Connection to nfs-c01 closed.
[root@ipaclient01 ~]# cd /n
[root@ipaclient01 n]# ls -altri some-people-find-this-awesome.txt
11782527620043058273 -rw-r–r–. 1 nobody nobody 74 Feb 21 00:26 some-people-find-this-awesome.txt
[root@ipaclient01 n]# df -h .
Filesystem Size Used Avail Use% Mounted on
nfs-c01:/n 128G 43M 128G 1% /n
[root@ipaclient01 n]#
So we bring the second node back up.
Ran into a problem with directories showing up mounted with the nobody / nobody user. Issue the following to resolve that:
systemctl restart rpcidmapd
And that concludes the configuration! All works like a charm!
Good Luck!
Cheers,
Tom K.
[…] the bin and install it on the first node you want to use. We are using Gluster to install confluence on. See the Gluster specific bits on the preceding […]