Header Shadow Image


GlusterFS: Configuration and Setup w/ NFS-Ganesha for an HA NFS Cluster

In this post we will go over how to setup a highly available NFS Cluster using:

  • GlusterFS
  • NFS Ganesha
  • CentOS 7 
  • HAPROXY
  • keepalived
  • firewalld
  • selinux

This post is very lengthy and goes over quite a few details on the way to configuring this setup.  We document virtually every step including how to build out a GlusterFS filesystem on both physical or virtual environments.  For those interested in a quick setup, please skip to the SUMMARY or TESTING sections at the bottom for a summary of commands and configuration files used.  If you run into problems, just search the page for the issue you have, as it's likely listed, and read the solution attempted.

DETAILED CONFIGURATION ATTEMPT  ( ATTEMPT LOG )

GlusterFS: Configuration and Setup

In this post we'll go over configuring and setting up GlusterFS gradually working towards an HA NFS server configuration.  Depending on what you read, Gluster is supposed to be faster then Ceph but harder to troubleshoot and recover from dataloss or failure.   We'll see for ourselves and configure it and give all that a test. We will use two main sources for going about this: Getting Started Install and Getting Started Configure.  

We'll reference a number of online pages to accomplish setup. CentOS 7 documentation on installing the latest Gluster Packages and Gluster Configuration.  We'll go with this one instead as it support NFSv4 . However the Gluster Quickstart will give us a primer on starting off.

To start, install the RPM's as per the two links above.  You may need to install the following epel repo before proceeding.  The page above didn't cover at this time.  Install and enable gluster on both nodes ( PHYSICAL: m-ph01, m-ph02, VIRTUAL: nfs01 / nfs02 ) :

yum install epel-release
yum -y install glusterfs glusterfs-fuse glusterfs-server glusterfs-api glusterfs-cli

Once done, we need to mount and add this volume as a duplicate of the previous node we did, m-ph01 then start the service:

[root@m-ph02 yum.repos.d]# systemctl enable gluster
Failed to execute operation: Access denied
[root@m-ph02 yum.repos.d]#

If it fails as above, it's probably the service name that was used.  Do the following instead:  

[root@m-ph02 yum.repos.d]# systemctl enable glusterd.service
Created symlink from /etc/systemd/system/multi-user.target.wants/glusterd.service to /usr/lib/systemd/system/glusterd.service.
[root@m-ph02 yum.repos.d]#
[root@m-ph02 yum.repos.d]# systemctl start glusterd
[root@m-ph02 yum.repos.d]# systemctl list-unit-files|grep -i glu
glusterd.service                            enabled
glusterfsd.service                          disabled
[root@m-ph02 yum.repos.d]#

Open firewall rules on all participating nodes:

firewall-cmd –zone=public –add-port=24007-24008/tcp –permanent
firewall-cmd –zone=public –add-port=49152/tcp –permanent
firewall-cmd –zone=public –add-port=38465-38469/tcp –permanent
firewall-cmd –zone=public –add-port=111/tcp –permanent
firewall-cmd –zone=public –add-port=111/udp –permanent
firewall-cmd –zone=public –add-port=2049/tcp –permanent
firewall-cmd –reload

Yes, you need the ports (Virtual Host example provided below):

[root@nfs02 yum.repos.d]# gluster peer probe nfs01
peer probe: failed: Probe returned with Transport endpoint is not connected
[root@nfs02 yum.repos.d]# firewall-cmd –zone=public –add-port=24007-24008/tcp –permanent
success
[root@nfs02 yum.repos.d]# firewall-cmd –zone=public –add-port=49152/tcp –permanent
success
[root@nfs02 yum.repos.d]# firewall-cmd –zone=public –add-port=38465-38469/tcp –permanent
success
[root@nfs02 yum.repos.d]# firewall-cmd –zone=public –add-port=111/tcp –permanent
success
[root@nfs02 yum.repos.d]# firewall-cmd –zone=public –add-port=111/udp –permanent
success
[root@nfs02 yum.repos.d]# firewall-cmd –zone=public –add-port=2049/tcp –permanent
success
[root@nfs02 yum.repos.d]# firewall-cmd –reload
success
[root@nfs02 yum.repos.d]# gluster peer probe nfs01
peer probe: success.
[root@nfs02 yum.repos.d]#

Results:

[root@nfs01 ~]# gluster peer status
Number of Peers: 1

Hostname: nfs02.nix.mine.dom
Uuid: 395276d9-17c6-4284-964c-9c93c0a30551
State: Peer in Cluster (Connected)
Other names:
nfs02
[root@nfs01 ~]#

[root@nfs02 yum.repos.d]# gluster peer status
Number of Peers: 1

Hostname: nfs01
Uuid: 4067fbf3-6118-426c-a899-601afe3ff1d7
State: Peer in Cluster (Connected)
Other names:
nfs01.nix.mine.dom
[root@nfs02 yum.repos.d]#

Once the service is started, issue a probe command.  If it doesn't work from this node, as you can see below, issue it from a working node towards the second node you are building:

[root@m-ph02 yum.repos.d]# gluster peer probe m-ph01
peer probe: failed: m-ph01 is either already part of another cluster or having volumes configured

Since we got the above error, we issue the following from the working node:

[root@m-ph01 yum.repos.d]# gluster peer probe m-ph02
peer probe: success.
[root@m-ph01 yum.repos.d]#

Then the following again from the new node we are building back to the working node to ensure all bits are communicating well:

[root@m-ph02 yum.repos.d]# ping m-ph01
PING m-ph01.mine.dom (192.168.0.60) 56(84) bytes of data.
64 bytes from 192.168.0.60: icmp_seq=1 ttl=64 time=0.269 ms
^C
— m-ph01.mine.dom ping statistics —
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 0.269/0.269/0.269/0.000 ms
[root@m-ph02 yum.repos.d]# gluster peer probe m-ph01
peer probe: success.
[root@m-ph02 yum.repos.d]#

Once the probe completes successfully, let's move on to the next steps of creating the VG's and LV's on each physical LUN we'll be using for the volume (This is identical for virtual hosts):

[root@m-ph02 yum.repos.d]# pvcreate /dev/sdb
[root@m-ph02 yum.repos.d]# vgcreate mdskvmsanvg /dev/sdb

[root@m-ph02 yum.repos.d]# lvcreate -l524286 -n mdskvmsanlv mdskvmsanvg
  Logical volume "mdskvmsanlv" created.
[root@m-ph02 yum.repos.d]# lvs
  LV          VG          Attr       LSize  Pool Origin Data%  Meta%  Move Log Cpy%Sync Convert
  home        centos      -wi-ao—- 19.99g
  root        centos      -wi-ao—- 40.95g
  swap        centos      -wi-ao—-  6.84g
  mdskvmsanlv mdskvmsanvg -wi-a—–  2.00t
[root@m-ph02 yum.repos.d]#
[root@m-ph02 yum.repos.d]# mkfs.xfs -i size=512 /dev/mapper/mdskvmsanvg-mdskvmsanlv
meta-data=/dev/mapper/mdskvmsanvg-mdskvmsanlv isize=512    agcount=4, agsize=134217216 blks
         =                       sectsz=512   attr=2, projid32bit=1
         =                       crc=0        finobt=0
data     =                       bsize=4096   blocks=536868864, imaxpct=5
         =                       sunit=0      swidth=0 blks
naming   =version 2              bsize=4096   ascii-ci=0 ftype=0
log      =internal log           bsize=4096   blocks=262143, version=2
         =                       sectsz=512   sunit=0 blks, lazy-count=1
realtime =none                   extsz=4096   blocks=0, rtextents=0
[root@m-ph02 yum.repos.d]#

Then create the /etc/fstab mount point (Do this on both nodes):

[root@m-ph02 yum.repos.d]# grep -Ei "xfs" /etc/fstab|grep mds
/dev/mapper/mdskvmsanvg-mdskvmsanlv /mnt/p02-d01 xfs logbufs=8,noatime,nodiratime,allocsize=512m 0 0
[root@m-ph02 yum.repos.d]#

Change the ownership of the mounts depending on what Cloud Manager you use:

[root@m-ph02 yum.repos.d]# mkdir /mnt/p02-d01
[root@m-ph02 yum.repos.d]# chown oneadmin.oneadmin /mnt/p02-d01
[root@m-ph02 yum.repos.d]# id oneadmin
uid=9869(oneadmin) gid=9869(oneadmin) groups=9869(oneadmin)
[root@m-ph02 yum.repos.d]#

Then try to add the brick. Two examples are provided.   First a virtual example:

[root@nfs02 ~]# gluster volume create gv01 replica 2 nfs01:/bricks/0/gv0 nfs02:/bricks/0/gv0
Replica 2 volumes are prone to split-brain. Use Arbiter or Replica 3 to avoid this. See: http://docs.gluster.org/en/latest/Administrator%20Guide/Split%20brain%20and%20ways%20to%20deal%20with%20it/.
Do you still want to continue?
 (y/n) y
volume create: gv01: success: please start the volume to access data
[root@nfs02 ~]# gluster volume start gv01
volume start: gv01: success
[root@nfs02 ~]# gluster volume info

Volume Name: gv01
Type: Replicate
Volume ID: e1485a5a-0edf-446d-9ae9-2705e6b57994
Status: Started
Snapshot Count: 0
Number of Bricks: 1 x 2 = 2
Transport-type: tcp
Bricks:
Brick1: nfs01:/bricks/0/gv0
Brick2: nfs02:/bricks/0/gv0
Options Reconfigured:
transport.address-family: inet
nfs.disable: on
performance.client-io-threads: off
[root@nfs02 ~]#

And a physical example:

[root@m-ph02 mnt]# gluster volume create gv0 replica 2 m-ph01.mine.dom:/mnt/p01-d01 m-ph02.mine.dom:/mnt/p02-d01
volume create: gv0: failed: The brick m-ph02.mine.dom:/mnt/p02-d01 is a mount point. Please create a sub-directory under the mount point and use that as the brick directory. Or use 'force' at the end of the command if you want to override this behavior.
[root@m-ph02 mnt]# mkdir /mnt/p02-d01/glusterv01
[root@m-ph02 mnt]# gluster volume create gv0 replica 2 m-ph01.mine.dom:/mnt/p01-d01 m-ph02.mine.dom:/mnt/p02-d01
volume create: gv0: failed: The brick m-ph02.mine.dom:/mnt/p02-d01 is a mount point. Please create a sub-directory under the mount point and use that as the brick directory. Or use 'force' at the end of the command if you want to override this behavior.
[root@m-ph02 mnt]# gluster volume create gv0 replica 2 m-ph01.mine.dom:/mnt/p01-d01/glusterv01 m-ph02.mine.dom:/mnt/p02-d01/glusterv01
volume create: gv0: failed: Staging failed on 192.168.0.60. Error: Brick: m-ph01.mine.dom:/mnt/p01-d01/glusterv01 not available. Brick may be containing or be contained by an existing brick
[root@m-ph02 mnt]#

If that fails, due to already being in a volume, simply add this one as a replicated brick of the first.  First we need to remove the old brick and the whole cluster as we created a non replicated cluster.  We do this by:

[root@m-ph02 mnt]# gluster volume delete mdsglusterv01
Deleting volume will erase all information about the volume. Do you want to continue? (y/n) y
volume delete: mdsglusterv01: success
[root@m-ph02 mnt]# gluster volume info
No volumes present
[root@m-ph02 mnt]#

Now add the bricks, indicating replication in the same command and test by copying some ISO's to the volume:

[root@m-ph01 p01-d01]# gluster volume create mdsgv01 replica 2  m-ph01.mine.dom:/mnt/p01-d01/glusterv01 m-ph02.mine.dom:/mnt/p02-d01/glusterv02
volume create: mdsgv01: success: please start the volume to access data
[root@m-ph01 p01-d01]# gluster volume start mdsgv01
volume start: mdsgv01: success
[root@m-ph01 p01-d01]# mv *.iso glusterv01/
[root@m-ph01 p01-d01]# cd glusterv01
[root@m-ph01 glusterv01]# ls -altri
total 6792216
       146 -rwxr–r–. 1 oneadmin oneadmin 632291328 May 19 01:19 loadit-10.iso
       145 -rwxr–r–. 1 oneadmin oneadmin 632291328 May 19 01:19 loadit-09.iso
       144 -rwxr–r–. 1 oneadmin oneadmin 632291328 May 19 01:19 loadit-08.iso
       143 -rwxr–r–. 1 oneadmin oneadmin 632291328 May 19 01:19 loadit-07.iso
       142 -rwxr–r–. 1 oneadmin oneadmin 632291328 May 19 01:19 loadit-06.iso
       141 -rwxr–r–. 1 oneadmin oneadmin 632291328 May 19 01:19 loadit-05.iso
       140 -rwxr–r–. 1 oneadmin oneadmin 632291328 May 19 01:19 loadit-04.iso
       136 -rwxr–r–. 1 oneadmin oneadmin 632291328 May 19 01:19 CentOS-7-x86_64-Minimal-1511.iso
       137 -rwxr–r–. 1 oneadmin oneadmin 632291328 May 19 01:20 loadit-01.iso
       139 -rwxr–r–. 1 oneadmin oneadmin 632291328 May 19 01:21 loadit-03.iso
       138 -rwxr–r–. 1 oneadmin oneadmin 632291328 May 19 01:21 loadit-02.iso
2147483776 drwxr-xr-x. 3 root     root            24 Jun  9 01:29 .trashcan
 536871104 drw——-. 8 root     root          4096 Jun  9 01:29 .glusterfs
       128 drwxr-xr-x. 3 oneadmin oneadmin        23 Jun  9 01:30 ..
 402653376 drwxr-xr-x. 4 oneadmin oneadmin      4096 Jun  9 01:30 .
[root@m-ph01 glusterv01]#
[root@m-ph01 glusterv01]#
[root@m-ph01 glusterv01]#
[root@m-ph01 glusterv01]# gluster volume info

Volume Name: mdsgv01
Type: Replicate
Volume ID: f5b57076-dbd4-4d77-ae13-c1f3ee3adbe0
Status: Started
Number of Bricks: 1 x 2 = 2
Transport-type: tcp
Bricks:
Brick1: m-ph01.mine.dom:/mnt/p01-d01/glusterv01
Brick2: m-ph02.mine.dom:/mnt/p02-d01/glusterv02
Options Reconfigured:
performance.readdir-ahead: on
[root@m-ph01 glusterv01]# ls -altri
total 6792216
       146 -rwxr–r–. 1 oneadmin oneadmin 632291328 May 19 01:19 loadit-10.iso
       145 -rwxr–r–. 1 oneadmin oneadmin 632291328 May 19 01:19 loadit-09.iso
       144 -rwxr–r–. 1 oneadmin oneadmin 632291328 May 19 01:19 loadit-08.iso
       143 -rwxr–r–. 1 oneadmin oneadmin 632291328 May 19 01:19 loadit-07.iso
       142 -rwxr–r–. 1 oneadmin oneadmin 632291328 May 19 01:19 loadit-06.iso
       141 -rwxr–r–. 1 oneadmin oneadmin 632291328 May 19 01:19 loadit-05.iso
       140 -rwxr–r–. 1 oneadmin oneadmin 632291328 May 19 01:19 loadit-04.iso
       136 -rwxr–r–. 1 oneadmin oneadmin 632291328 May 19 01:19 CentOS-7-x86_64-Minimal-1511.iso
       137 -rwxr–r–. 1 oneadmin oneadmin 632291328 May 19 01:20 loadit-01.iso
       139 -rwxr–r–. 1 oneadmin oneadmin 632291328 May 19 01:21 loadit-03.iso
       138 -rwxr–r–. 1 oneadmin oneadmin 632291328 May 19 01:21 loadit-02.iso
2147483776 drwxr-xr-x. 3 root     root            24 Jun  9 01:29 .trashcan
 536871104 drw——-. 8 root     root          4096 Jun  9 01:29 .glusterfs
       128 drwxr-xr-x. 3 oneadmin oneadmin        23 Jun  9 01:30 ..
 402653376 drwxr-xr-x. 4 oneadmin oneadmin      4096 Jun  9 01:30 .
[root@m-ph01 glusterv01]#

If the above commands failed, remove the volume then bricks and start fresh.  Deleting a volume is easy:

[root@nfs02 0]# gluster volume remove gv01 replica 2 nfs01:/bricks/0/gv0 nfs02:/bricks/0/gv0 stop
volume remove-brick stop: failed: Volume gv01 is not a distribute volume or contains only 1 brick.
Not performing rebalance
[root@nfs02 0]# gluster volume stop gv01
Stopping volume will make its data inaccessible. Do you want to continue? (y/n) y
volume stop: gv01: success
[root@nfs02 0]#
[root@nfs02 0]#
[root@nfs02 0]#
[root@nfs02 0]# gluster volume delete gv01
Deleting volume will erase all information about the volume. Do you want to continue? (y/n) y
volume delete: gv01: success
[root@nfs02 0]#

Now that you have your GlusterFS setup, we want to share it using NFSv4.  For that, we need NFS-Ganesha.  It's the only application that will support NFSv4 plus a number of other protocols.  We will follow the github project to set it up.

On both volumes type:

[root@nfs01 gv01]# systemctl status nfs
â nfs-server.service – NFS server and services
   Loaded: loaded (/usr/lib/systemd/system/nfs-server.service; disabled; vendor preset: disabled)
   Active: inactive (dead)
[root@nfs01 gv01]# systemctl disable nfs
[root@nfs01 gv01]# gluster volume set gv01 nfs.disable on
volume set: success
[root@nfs01 gv01]#

Start the ganesha nfsd daemon on both nodes using.  We just want to ensure the service works and can start up. (You can stop this right after):

ganesha.nfsd -f nfs-ganesha.conf -L nfs-ganesha.log -N NIV_DEBUG

Then, still on both nodes, create a path inside each gluster server and mount it using the glusterfs protocol:

ON NODE 1:

[root@nfs01 gv01]# mkdir /n 2>/dev/null; mount -t glusterfs nfs02:/gv01 /n
[root@nfs01 gv01]# mount|grep -i gluste

nfs02:/gv01 on /n type fuse.glusterfs (rw,relatime,user_id=0,group_id=0,default_permissions,allow_other,max_read=131072)
[root@nfs01 gv01]# cd /n
[root@nfs01 n]# ls -altri

total 8
  1 drwxr-xr-x.  3 root root 4096 Feb 18 14:17 .
128 dr-xr-xr-x. 19 root root 4096 Feb 18 14:22 ..

[root@nfs01 n]#

 

ON NODE 2:

[root@nfs02 gv01]# umount /n
[root@nfs02 gv01]# mkdir /n 2>/dev/null; mount -t glusterfs nfs02:/gv01 /n
[root@nfs02 gv01]# mount|grep -i gluster

nfs02:/gv01 on /n type fuse.glusterfs (rw,relatime,user_id=0,group_id=0,default_permissions,allow_other,max_read=131072)
[root@nfs02 gv01]#

Define your export:

ON NODE 1:

[root@nfs01 ganesha]# pwd
/etc/ganesha
[root@nfs01 ganesha]# cat export.conf
EXPORT{
        Export_Id = 1 ;   # Export ID unique to each export
        Path = "/n";  # Path of the volume to be exported. Eg: "/test_volume"

        FSAL {
                name = GLUSTER;
                hostname = "nfs01.nix.mine.dom";  # IP of one of the nodes in the trusted pool
                volume = "/n";         # Volume name. Eg: "test_volume"
        }

        Access_type = RW;        # Access permissions
        Squash = root_squash;    # To enable/disable root squashing
        Disable_ACL = FALSE;     # To enable/disable ACL
        Pseudo = "pseudo_path";  # NFSv4 pseudo path for this export. Eg: "/test_volume_pseudo"
        Protocols = "4" ;        # NFS protocols supported
        Transports = "TCP" ;     # Transport protocols supported
        SecType = "sys";         # Security flavors supported
}
[root@nfs01 ganesha]#

ON NODE 2:

[root@nfs02 ganesha]# pwd
/etc/ganesha
[root@nfs02 ganesha]# cat export.conf
EXPORT{
        Export_Id = 1 ;   # Export ID unique to each export
        Path = "/n";  # Path of the volume to be exported. Eg: "/test_volume"

        FSAL {
                name = GLUSTER;
                hostname = "nfs02.nix.mine.dom";  # IP of one of the nodes in the trusted pool
                volume = "/n";         # Volume name. Eg: "test_volume"
        }

        Access_type = RW;        # Access permissions
        Squash = root_squash;    # To enable/disable root squashing
        Disable_ACL = FALSE;     # To enable/disable ACL
        Pseudo = "pseudo_path";  # NFSv4 pseudo path for this export. Eg: "/test_volume_pseudo"
        Protocols = "4" ;        # NFS protocols supported
        Transports = "TCP" ;     # Transport protocols supported
        SecType = "sys";         # Security flavors supported
}
[root@nfs02 ganesha]#

Include the export.conf into ganesha.conf:

ON NODE 1:

[root@nfs02 ganesha]# grep -Ei include /etc/ganesha/ganesha.conf
%include "/etc/ganesha/export.conf"
[root@nfs02 ganesha]#

ON NODE 2:

[root@nfs01 ganesha]# grep -Ei include /etc/ganesha/ganesha.conf
%include "/etc/ganesha/export.conf"
[root@nfs01 ganesha]#

Check the services on both nodes:

[root@nfs02 ganesha]# systemctl list-unit-files –type=service|grep -Ei ganesh
nfs-ganesha-config.service                    static
nfs-ganesha-lock.service                      static
nfs-ganesha.service                           disabled
[root@nfs02 ganesha]# systemctl status nfs-ganesha.service
â nfs-ganesha.service – NFS-Ganesha file server
   Loaded: loaded (/usr/lib/systemd/system/nfs-ganesha.service; disabled; vendor preset: disabled)
   Active: active (running) since Sun 2018-02-18 01:14:09 EST; 14h ago
     Docs: http://github.com/nfs-ganesha/nfs-ganesha/wiki
 Main PID: 3180 (ganesha.nfsd)
   CGroup: /system.slice/nfs-ganesha.service
           ââ3180 /usr/bin/ganesha.nfsd -L /var/log/ganesha.log -f /etc/ganesha/ganesha.conf -N NIV_EVENT

Feb 18 01:14:09 nfs02.nix.mine.dom systemd[1]: Starting NFS-Ganesha file server…
Feb 18 01:14:09 nfs02.nix.mine.dom systemd[1]: Started NFS-Ganesha file server.
[root@nfs02 ganesha]#
[root@nfs02 ganesha]#
[root@nfs02 ganesha]#
[root@nfs02 ganesha]# ps -ef|grep -Ei ganesha
root      3180     1  0 01:14 ?        00:00:21 /usr/bin/ganesha.nfsd -L /var/log/ganesha.log -f /etc/ganesha/ganesha.conf -N NIV_EVENT
root     10721  4201  0 15:21 pts/0    00:00:00 grep –color=auto -Ei ganesha
[root@nfs02 ganesha]#

Should be the same on both nodes. This time we'll start ganesha.nfsd using systemd:

[root@nfs01 ganesha]# systemctl restart nfs-ganesha.service
[root@nfs01 ganesha]# ps -ef|grep -Ei [Gg]anesha

root      4825     1  0 15:25 ?        00:00:00 /usr/bin/ganesha.nfsd -L /var/log/ganesha.log -f /etc/ganesha/ganesha.conf -N NIV_EVENT
[root@nfs01 ganesha]#

Let's test the exports and see if anything is listed:

showmount -e localhost

But first ensure ganesha nfsd damon is enabled:

[root@nfs01 log]# systemctl enable nfs-ganesha.service
Created symlink from /etc/systemd/system/multi-user.target.wants/nfs-ganesha.service to /usr/lib/systemd/system/nfs-ganesha.service.
[root@nfs01 log]# 

And we have problems (/var/log/ganesha.log):

NFS STARTUP :CRIT :Could not dlopen module:/usr/lib64/ganesha/libfsalvfs.so Error:/usr/lib64/ganesha/libfsalvfs.so: cannot open shared object file: No such file or directory

This is because we are missing the nfs-ganesha-vfs plugin while we are referencing VFS in one of the configs.  Either remove reference or install the plugin.  We choose the latter because more is more:

[root@nfs02 log]# rpm -aq|grep -Ei nfs-ganesha
nfs-ganesha-xfs-2.3.2-1.el7.x86_64
nfs-ganesha-2.3.2-1.el7.x86_64
[root@nfs02 log]# grep -Ei VFS /etc/ganesha/*
/etc/ganesha/ganesha.conf:              Name = VFS;
[root@nfs02 log]#

Unfortunately when we try to install the right package we get another error:

[root@nfs02 log]# yum install nfs-ganesha-vfs.x86_64
Error: Package: nfs-ganesha-2.5.5-1.el7.x86_64 (centos-gluster312)
           Requires: libntirpc.so.1.5(NTIRPC_1.5.4)(64bit)

Let's check where the packages are coming from so perhaps we can disable one of them to get the right version because it's trying to pull an updated nfs-ganesha version as well but gives us:

Error: Package: nfs-ganesha-2.5.5-1.el7.x86_64 (centos-gluster312)
           Requires: libntirpc.so.1.5(NTIRPC_1.5.4)(64bit)

which wasn't uploaded to the CentOS 7 repos at the time of this writing.  So we'll use the older package (We don't need to be recent up-to-a-day ATM):

[root@nfs02 log]# yum info nfs-ganesha
Loaded plugins: fastestmirror, priorities
Loading mirror speeds from cached hostfile
 * base: centos.mirror.rafal.ca
 * epel: mirror.math.princeton.edu
 * extras: centos.mirror.rafal.ca
 * updates: centos.mirror.rafal.ca
Installed Packages
Name        : nfs-ganesha
Arch        : x86_64
Version     : 2.3.2
Release     : 1.el7
Size        : 1.6 M
Repo        : installed
From repo   : epel
Summary     : NFS Server running in user space
URL         : https://github.com/nfs-ganesha/nfs-ganesha/wiki
License     : LGPLv3+
Description : nfs-ganesha : NFS-GANESHA is a NFS Server running in user space.
            : It comes with various back-end modules (called FSALs) provided as
            : shared objects to support different file systems and name-spaces.

Available Packages
Name        : nfs-ganesha
Arch        : x86_64
Version     : 2.5.5
Release     : 1.el7
Size        : 650 k
Repo        : centos-gluster312/7/x86_64
Summary     : NFS-Ganesha is a NFS Server running in user space
URL         : https://github.com/nfs-ganesha/nfs-ganesha/wiki
License     : LGPLv3+
Description : nfs-ganesha : NFS-GANESHA is a NFS Server running in user space.
            : It comes with various back-end modules (called FSALs) provided as
            : shared objects to support different file systems and name-spaces.

[root@nfs02 log]#

So we use the following that will install the correct version for our NFS Ganesha install:

[root@nfs02 ~]# yum –disablerepo="*" –enablerepo=epel install nfs-ganesha-vfs -y

[root@nfs02 log]# ls -altri /usr/lib64/ganesha/libfsalvfs.so
102513 lrwxrwxrwx. 1 root root 15 Feb 18 16:31 /usr/lib64/ganesha/libfsalvfs.so -> libfsalvfs.so.4
[root@nfs02 log]#

Next we restart NFS Ganesha again and check.  Still errors:

18/02/2018 16:37:55 : epoch 5a89f232 : nfs01.nix.mine.dom : ganesha.nfsd-5497[main] config_errs_to_log :CONFIG :CRIT :Config File (/etc/ganesha/export.conf:5): Failed to load FSAL (GLUSTER) because: Can not access a needed shared library
18/02/2018 16:37:55 : epoch 5a89f232 : nfs01.nix.mine.dom : ganesha.nfsd-5497[main] config_errs_to_log :CONFIG :CRIT :Config File (/etc/ganesha/export.conf:5): 1 validation errors in block FSAL
18/02/2018 16:37:55 : epoch 5a89f232 : nfs01.nix.mine.dom : ganesha.nfsd-5497[main] config_errs_to_log :CONFIG :CRIT :Config File (/etc/ganesha/export.conf:5): Errors processing block (FSAL)
18/02/2018 16:37:55 : epoch 5a89f232 : nfs01.nix.mine.dom : ganesha.nfsd-5497[main] config_errs_to_log :CONFIG :CRIT :Config File (/etc/ganesha/export.conf:1): 1 validation errors in block EXPORT
18/02/2018 16:37:55 : epoch 5a89f232 : nfs01.nix.mine.dom : ganesha.nfsd-5497[main] config_errs_to_log :CONFIG :CRIT :Config File (/etc/ganesha/export.conf:1): Errors processing block (EXPORT)

Problem with this block:

        FSAL {
                name = GLUSTER;
                hostname = "nfs01.nix.mine.dom";         # IP of one of the nodes in the trusted pool
                volume = "/n";                          # Volume name. Eg: "test_volume"
        }

Unfortunately, the error doesn't tell you WHICH library is missing but fortunately the page above indicated what could be wrong so we need to look for libgfapi.so.0()(64bit) .

yum install glusterfs-api

But that wasn't it.  Looking a bit more carefuly, we notice the following:

18/02/2018 16:37:49 : epoch 5a89f22d : nfs02.nix.mine.dom : ganesha.nfsd-12399[main] load_fsal :NFS STARTUP :CRIT :Could not dlopen module:/usr/lib64/ganesha/libfsalgluster.so Error:/usr/lib64/ganesha/libfsalgluster.so: cannot open shared object file: No such file or directory

 

We need the following package then:

yum –disablerepo="*" –enablerepo=epel install nfs-ganesha-gluster

But this is only available from the new repository.  :(   Fortunately, there is hope.  So we'll use that rpm.

yum install http://cbs.centos.org/kojifiles/packages/nfs-ganesha/2.3.2/1.el7/x86_64/nfs-ganesha-gluster-2.3.2-1.el7.x86_64.rpm

Let's restart and try this thing again.  We move forward but get this error:

18/02/2018 18:04:53 : epoch 5a8a0694 : nfs02.nix.mine.dom : ganesha.nfsd-13043[main] config_errs_to_log :CONFIG :CRIT :Config File (/etc/ganesha/export.conf:5): 1 validation errors in block FSAL
18/02/2018 18:04:53 : epoch 5a8a0694 : nfs02.nix.mine.dom : ganesha.nfsd-13043[main] config_errs_to_log :CONFIG :CRIT :Config File (/etc/ganesha/export.conf:5): Errors processing block (FSAL)
18/02/2018 18:04:53 : epoch 5a8a0694 : nfs02.nix.mine.dom : ganesha.nfsd-13043[main] config_errs_to_log :CONFIG :CRIT :Config File (/etc/ganesha/export.conf:1): 1 validation errors in block EXPORT
18/02/2018 18:04:53 : epoch 5a8a0694 : nfs02.nix.mine.dom : ganesha.nfsd-13043[main] config_errs_to_log :CONFIG :CRIT :Config File (/etc/ganesha/export.conf:1): Errors processing block (EXPORT)

Reverting the configuration to some basic settings, we finally export a mount using the following settings:

[root@nfs02 ganesha]# systemctl stop nfs-ganesha.service; >/var/log/ganesha.log; systemctl start nfs-ganesha.service
[root@nfs02 ganesha]# showmount -e localhost

Export list for localhost:
/bricks/0/gv01/n (everyone)
[root@nfs02 ganesha]#
[root@nfs02 ganesha]#
[root@nfs02 ganesha]#
[root@nfs02 ganesha]# cat export.conf
EXPORT{
    Export_Id = 1 ;                             # Export ID unique to each export
    Path = "/bricks/0/gv01/n";                  # Path of the volume to be exported. Eg: "/test_volume"

    FSAL {
        name = GLUSTER;
        hostname = "nfs02.nix.mine.dom";         # IP of one of the nodes in the trusted pool
        volume = "gv01";                        # Volume name. Eg: "test_volume"
    }

    Access_type = RW;                           # Access permissions
    Squash = No_root_squash;                    # To enable/disable root squashing
    Disable_ACL = TRUE;                         # To enable/disable ACL
    Pseudo = "/n";                              # NFSv4 pseudo path for this export. Eg: "/test_volume_pseudo"
    Protocols = "3","4" ;                       # NFS protocols supported
    Transports = "UDP","TCP" ;                  # Transport protocols supported
    SecType = "sys";                            # Security flavors supported
}
[root@nfs02 ganesha]# cat /var/log/ganesha.log
18/02/2018 19:32:34 : epoch 5a8a1b22 : nfs02.nix.mine.dom : ganesha.nfsd-14166[main] main :MAIN :EVENT :ganesha.nfsd Starting: Ganesha Version /builddir/build/BUILD/nfs-ganesha-2.3.2/src, built at May  4 2016 05:09:13 on
18/02/2018 19:32:34 : epoch 5a8a1b22 : nfs02.nix.mine.dom : ganesha.nfsd-14167[main] nfs_set_param_from_conf :NFS STARTUP :EVENT :Configuration file successfully parsed
18/02/2018 19:32:34 : epoch 5a8a1b22 : nfs02.nix.mine.dom : ganesha.nfsd-14167[main] init_server_pkgs :NFS STARTUP :EVENT :Initializing ID Mapper.
18/02/2018 19:32:34 : epoch 5a8a1b22 : nfs02.nix.mine.dom : ganesha.nfsd-14167[main] init_server_pkgs :NFS STARTUP :EVENT :ID Mapper successfully initialized.
18/02/2018 19:32:34 : epoch 5a8a1b22 : nfs02.nix.mine.dom : ganesha.nfsd-14167[main] glusterfs_create_export :FSAL :EVENT :Volume gv01 exported at : '/'
18/02/2018 19:32:35 : epoch 5a8a1b22 : nfs02.nix.mine.dom : ganesha.nfsd-14167[main] lower_my_caps :NFS STARTUP :EVENT :CAP_SYS_RESOURCE was successfully removed for proper quota management in FSAL
18/02/2018 19:32:35 : epoch 5a8a1b22 : nfs02.nix.mine.dom : ganesha.nfsd-14167[main] lower_my_caps :NFS STARTUP :EVENT :currenty set capabilities are: = cap_chown,cap_dac_override,cap_dac_read_search,cap_fowner,cap_fsetid,cap_kill,cap_setgid,cap_setuid,cap_setpcap,cap_linux_immutable,cap_net_bind_service,cap_net_broadcast,cap_net_admin,cap_net_raw,cap_ipc_lock,cap_ipc_owner,cap_sys_module,cap_sys_rawio,cap_sys_chroot,cap_sys_ptrace,cap_sys_pacct,cap_sys_admin,cap_sys_boot,cap_sys_nice,cap_sys_time,cap_sys_tty_config,cap_mknod,cap_lease,cap_audit_write,cap_audit_control,cap_setfcap+ep
18/02/2018 19:32:35 : epoch 5a8a1b22 : nfs02.nix.mine.dom : ganesha.nfsd-14167[main] nfs_Init_admin_thread :NFS CB :EVENT :Admin thread initialized
18/02/2018 19:32:35 : epoch 5a8a1b22 : nfs02.nix.mine.dom : ganesha.nfsd-14167[main] nfs4_start_grace :STATE :EVENT :NFS Server Now IN GRACE, duration 60
18/02/2018 19:32:35 : epoch 5a8a1b22 : nfs02.nix.mine.dom : ganesha.nfsd-14167[main] nfs_rpc_cb_init_ccache :NFS STARTUP :EVENT :Callback creds directory (/var/run/ganesha) already exists
18/02/2018 19:32:35 : epoch 5a8a1b22 : nfs02.nix.mine.dom : ganesha.nfsd-14167[main] nfs_rpc_cb_init_ccache :NFS STARTUP :WARN :gssd_refresh_krb5_machine_credential failed (-1765328203:0)
18/02/2018 19:32:35 : epoch 5a8a1b22 : nfs02.nix.mine.dom : ganesha.nfsd-14167[main] nfs_Start_threads :THREAD :EVENT :Starting delayed executor.
18/02/2018 19:32:35 : epoch 5a8a1b22 : nfs02.nix.mine.dom : ganesha.nfsd-14167[main] nfs_Start_threads :THREAD :EVENT :9P/TCP dispatcher thread was started successfully
18/02/2018 19:32:35 : epoch 5a8a1b22 : nfs02.nix.mine.dom : ganesha.nfsd-14167[_9p_disp] _9p_dispatcher_thread :9P DISP :EVENT :9P dispatcher started
18/02/2018 19:32:35 : epoch 5a8a1b22 : nfs02.nix.mine.dom : ganesha.nfsd-14167[main] nfs_Start_threads :THREAD :EVENT :gsh_dbusthread was started successfully
18/02/2018 19:32:35 : epoch 5a8a1b22 : nfs02.nix.mine.dom : ganesha.nfsd-14167[main] nfs_Start_threads :THREAD :EVENT :admin thread was started successfully
18/02/2018 19:32:35 : epoch 5a8a1b22 : nfs02.nix.mine.dom : ganesha.nfsd-14167[main] nfs_Start_threads :THREAD :EVENT :reaper thread was started successfully
18/02/2018 19:32:35 : epoch 5a8a1b22 : nfs02.nix.mine.dom : ganesha.nfsd-14167[reaper] nfs_in_grace :STATE :EVENT :NFS Server Now IN GRACE
18/02/2018 19:32:35 : epoch 5a8a1b22 : nfs02.nix.mine.dom : ganesha.nfsd-14167[main] nfs_Start_threads :THREAD :EVENT :General fridge was started successfully
18/02/2018 19:32:35 : epoch 5a8a1b22 : nfs02.nix.mine.dom : ganesha.nfsd-14167[main] nfs_start :NFS STARTUP :EVENT :————————————————-
18/02/2018 19:32:35 : epoch 5a8a1b22 : nfs02.nix.mine.dom : ganesha.nfsd-14167[main] nfs_start :NFS STARTUP :EVENT :             NFS SERVER INITIALIZED
18/02/2018 19:32:35 : epoch 5a8a1b22 : nfs02.nix.mine.dom : ganesha.nfsd-14167[main] nfs_start :NFS STARTUP :EVENT :————————————————-
[root@nfs02 ganesha]#
[root@nfs02 ganesha]# showmount -e localhost

Export list for localhost:
/bricks/0/gv01/n (everyone)
[root@nfs02 ganesha]#

But we only want NFSv4 available.  Let's experiment a bit more:

[root@nfs02 ganesha]# mount -t glusterfs nfs01:/gv01 /n
[root@nfs02 ganesha]# vi export.conf
[root@nfs02 ganesha]# systemctl stop nfs-ganesha.service; >/var/log/ganesha.log; systemctl start nfs-ganesha.service
[root@nfs02 ganesha]# showmount -e localhost

Export list for localhost:
/n (everyone)
[root@nfs02 ganesha]# cat export.conf |grep Path
    Path = "/n";                                # Path of the volume to be exported. Eg: "/test_volume"
[root@nfs02 ganesha]#

Let's try to change the NFS options to allow only for NFSv4:

Protocols = "3","4" ; 

But removing "3" results in no mounts being shown.  This is because showmount -e doesn't list NFSv4 mounts.  It uses the NFSv3 protocol to list NFSv3 mounts so if we are not exporting using NFSv4 , we won't see the mounts.  The way we can ensure it's working is if we actually mount the NFSv4 storage or use this alternate to list the mounts but at this time I still don't have this one liner working properly.  Here it is anyway:

dbus-send –type=method_call –print-reply –system  –dest=org.ganesha.nfsd /org/ganesha/nfsd/ExportMgr  org.ganesha.nfsd.exportmgr.ShowExports

Add the following section to the /etc/ganesha/ganesha.conf file:

[root@nfs02 ganesha]# cat ganesha.conf
###################################################
#
# EXPORT
#
# To function, all that is required is an EXPORT
#
# Define the absolute minimal export
#
###################################################


NFS_Core_Param {
        NFS_Port = 2049;
        MNT_Port = 20048;
        NLM_Port = 38468;
        Rquota_Port = 4501;
}

%include "/etc/ganesha/export.conf"
[root@nfs02 ganesha]#

Follow this by allowing the last port otherwise you get this error below:

ganesha.nfsd-15334[main] Bind_sockets_V6 :D ISP :WARN :Cannot bind RQUOTA udp6 socket, error 13 (Permission denied)
ganesha.nfsd-15334[main] Bind_sockets :D ISP :FATAL :Error binding to V6 interface. Cannot continue.

[root@nfs02 audit]# tail -f audit.log|grep -Ei src
type=AVC msg=audit(1519003656.646:7601): avc:  denied  { name_bind } for  pid=15452 comm="ganesha.nfsd" src=4501 scontext=system_u:system_r:ganesha_t:s0 tcontext=system_u:object_r:unreserved_port_t:s0 tclass=tcp_socket

If you get the above, use the following commands to allow the port:

audit2allow -a
audit2allow -a -M ganesha_4501_port
semodule -i ganesha_4501_port.pp

Alternate way that can work as well (In our test case here both methods were needed):

[root@nfs02 audit]# grep AVC /var/log/audit/audit.log | tail -n1 | audit2allow -M systemd-allow
******************** IMPORTANT ***********************
To make this policy package active, execute:

semodule -i systemd-allow.pp

[root@nfs02 audit]# grep AVC /var/log/audit/audit.log | tail -n1
type=AVC msg=audit(1519003656.646:7601): avc:  denied  { name_bind } for  pid=15452 comm="ganesha.nfsd" src=4501 scontext=system_u:system_r:ganesha_t:s0 tcontext=system_u:object_r:unreserved_port_t:s0 tclass=tcp_socket
[root@nfs02 audit]# semodule -i systemd-allow.pp
libsemanage.add_user: user ipauser not in password file
[root@nfs02 audit]#

Restart and check:

[root@nfs02 audit]# systemctl stop nfs-ganesha.service; >/var/log/ganesha.log; systemctl start nfs-ganesha.service
[root@nfs02 audit]# firewall-cmd –zone=public –list-all

public
  target: default
  icmp-block-inversion: no
  interfaces:
  sources:
  services: ssh dhcpv6-client
  ports: 24007-24008/tcp 49152/tcp 38465-38469/tcp 111/tcp 111/udp 2049/tcp
  protocols:
  masquerade: no
  forward-ports:
  source-ports:
  icmp-blocks:
  rich rules:

[root@nfs02 audit]#

Do the same for DMZ rules as well. ( Test using nmap -oO -sS -sV nfs02 from the remote host when both dmz and public rules are added. )

firewall-cmd –zone=dmz –permanent –add-port=49152/tcp
firewall-cmd –zone=dmz –permanent –add-port=38465-38469/tcp
firewall-cmd –reload
firewall-cmd –zone=dmz –list-all

Let's enable logging of all firewall activity while we can't list mounts remotely:

[root@nfs02 ganesha]# firewall-cmd –set-log-denied=all
success
[root@nfs02 ganesha]#

[root@ipaclient01 ~]# showmount -e nfs02.nix.mine.dom
rpc mount export: RPC: Unable to receive; errno = No route to host
[root@ipaclient01 ~]#

Now check /var/log/messages for the real blocked port:

Feb 18 21:02:58 nfs02 kernel: FINAL_REJECT: IN=eth0 OUT= MAC=00:50:56:86:2d:21:00:50:56:86:d7:4c:08:00 SRC=192.168.0.236 DST=192.168.0.119 LEN=60 TOS=0×00 PREC=0×00 TTL=64 ID=13717 DF PROTO=TCP SPT=921 DPT=20048 WINDOW=29200 RES=0×00 SYN URGP=0
Feb 18 21:02:58 nfs02 kernel: FINAL_REJECT: IN=eth0 OUT= MAC=00:50:56:86:2d:21:00:50:56:86:d7:4c:08:00 SRC=192.168.0.236 DST=192.168.0.119 LEN=116 TOS=0×00 PREC=0×00 TTL=64 ID=25816 DF PROTO=UDP SPT=921 DPT=20048 LEN=96
Feb 18 21:03:08 nfs02 kernel: FINAL_REJECT: IN=eth0 OUT= MAC=ff:ff:ff:ff:ff:ff:48:5d:60:cb:44:1c:08:00 SRC=192.168.0.102 DST=255.255.255.255 LEN=328 TOS=0×00 PREC=0×00 TTL=128 ID=2731 PROTO=UDP SPT=68 DPT=67 LEN=308

[root@nfs02 ganesha]# netstat -pnlt|grep -Ei 20048
tcp6       0      0 :::20048                :::*                    LISTEN      15674/ganesha.nfsd
[root@nfs02 ganesha]#

Sure enough, we need to set the F/W rule for this:

[root@nfs02 ganesha]# firewall-cmd –zone=public –permanent –add-port=20048/udp
success
[root@nfs02 ganesha]# firewall-cmd –reload
success
[root@nfs02 ganesha]#

But get this error instead:

[root@ipaclient01 ~]# showmount -e nfs02.nix.mine.dom
clnt_create: RPC: Port mapper failure – Unable to receive: errno 113 (No route to host)
[root@ipaclient01 ~]#

 

Feb 18 21:12:00 nfs02 kernel: FINAL_REJECT: IN=eth0 OUT= MAC=00:50:56:86:2d:21:00:50:56:86:d7:4c:08:00 SRC=192.168.0.236 DST=192.168.0.119 LEN=84 TOS=0×00 PREC=0×00 TTL=64 ID=25657 DF PROTO=UDP SPT=927 DPT=111 LEN=64
Feb 18 21:12:00 nfs02 kernel: FINAL_REJECT: IN=eth0 OUT= MAC=00:50:56:86:2d:21:00:50:56:86:d7:4c:08:00 SRC=192.168.0.236 DST=192.168.0.119 LEN=84 TOS=0×00 PREC=0×00 TTL=64 ID=25660 DF PROTO=UDP SPT=927 DPT=111 LEN=64

Add them all:

firewall-cmd –zone=public –permanent –add-port=2049/tcp
firewall-cmd –zone=public –permanent –add-port=111/tcp
firewall-cmd –zone=public –permanent –add-port=111/udp
firewall-cmd –zone=public –permanent –add-port=24007-24008/tcp
firewall-cmd –zone=public –permanent –add-port=49152/tcp
firewall-cmd –zone=public –permanent –add-port=38465-38469/tcp
firewall-cmd –zone=public –permanent –add-port=4501/tcp

firewall-cmd –zone=public –permanent –add-port=4501/udp
firewall-cmd –reload

 

Now try to connect from the client again:

[root@ipaclient01 ~]# showmount -e nfs02.nix.mine.dom
Export list for nfs02.nix.mine.dom:
/n (everyone)
[root@ipaclient01 ~]#

Works!  Now let's try to mount as a true NFSv4 share:

[root@ipaclient01 ~]# df -h /n
Filesystem      Size  Used Avail Use% Mounted on
nfs02:/n        128G   43M  128G   1% /n
[root@ipaclient01 ~]# df -h /n|grep -Ei nfs4
[root@ipaclient01 ~]# mount|grep -Ei nfs0

nfs01:/gv01 on /mnt/gv01 type fuse.glusterfs (rw,relatime,user_id=0,group_id=0,default_permissions,allow_other,max_read=131072)
nfs02:/n on /n type nfs4 (rw,relatime,vers=4.0,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,port=0,timeo=600,retrans=2,sec=sys,clientaddr=192.168.0.236,local_lock=none,addr=192.168.0.119)
[root@ipaclient01 ~]#

Works like a charm!  Now let's make this into a cluster by replicating it over to the nfs01.nix.mine.dom node.  Let's run down the list very quickly.

 The files:

[root@nfs01 gv01]# cat public.bash
firewall-cmd –zone=public –permanent –add-port=2049/tcp
firewall-cmd –zone=public –permanent –add-port=111/tcp
firewall-cmd –zone=public –permanent –add-port=111/udp
firewall-cmd –zone=public –permanent –add-port=24007-24008/tcp
firewall-cmd –zone=public –permanent –add-port=49152/tcp
firewall-cmd –zone=public –permanent –add-port=38465-38469/tcp
firewall-cmd –zone=public –permanent –add-port=4501/tcp
firewall-cmd –zone=public –permanent –add-port=4501/udp
firewall-cmd –zone=public –permanent –add-port=20048/udp
firewall-cmd –zone=public –permanent –add-port=20048/tcp
firewall-cmd –reload
[root@nfs01 gv01]# cat dmz.bash
firewall-cmd –zone=dmz –permanent –add-port=2049/tcp
firewall-cmd –zone=dmz –permanent –add-port=111/tcp
firewall-cmd –zone=dmz –permanent –add-port=111/udp
firewall-cmd –zone=dmz –permanent –add-port=24007-24008/tcp
firewall-cmd –zone=dmz –permanent –add-port=49152/tcp
firewall-cmd –zone=dmz –permanent –add-port=38465-38469/tcp
firewall-cmd –zone=dmz –permanent –add-port=4501/tcp
firewall-cmd –zone=dmz –permanent –add-port=4501/udp
firewall-cmd –zone=dmz –permanent –add-port=20048/tcp
firewall-cmd –zone=dmz –permanent –add-port=20048/udp
firewall-cmd –reload
[root@nfs01 gv01]# cat /etc/ganesha/ganesha.conf
###################################################
#
# EXPORT
#
# To function, all that is required is an EXPORT
#
# Define the absolute minimal export
#
###################################################


NFS_Core_Param {
        NFS_Port = 2049;
        MNT_Port = 20048;
        NLM_Port = 38468;
        Rquota_Port = 4501;
}

%include "/etc/ganesha/export.conf"
[root@nfs01 gv01]# cat /etc/ganesha/export.conf
EXPORT{
    Export_Id = 1 ;                             # Export ID unique to each export
    Path = "/n";                                # Path of the volume to be exported. Eg: "/test_volume"

    FSAL {
        name = GLUSTER;
        hostname = "nfs01.nix.mine.dom";         # IP of one of the nodes in the trusted pool
        volume = "gv01";                        # Volume name. Eg: "test_volume"
    }

    Access_type = RW;                           # Access permissions
    Squash = No_root_squash;                    # To enable/disable root squashing
    Disable_ACL = FALSE;                        # To enable/disable ACL
    Pseudo = "/n";                              # NFSv4 pseudo path for this export. Eg: "/test_volume_pseudo"
    Protocols = "3","4";                        # NFS protocols supported
    Transports = "UDP","TCP" ;                  # Transport protocols supported
    SecType = "sys";                            # Security flavors supported
}
[root@nfs01 gv01]#

The commands for selinux in case things don't work:

grep AVC /var/log/audit/audit.log | tail -n1 | audit2allow -M systemd-allow
semodule -i systemd-allow.pp
systemctl stop nfs-ganesha.service; >/var/log/ganesha.log; systemctl start nfs-ganesha.service

audit2allow -a
audit2allow -a -M ganesha_4501_port
semodule -i ganesha_4501_port.pp

systemctl stop nfs-ganesha.service; >/var/log/ganesha.log; systemctl start nfs-ganesha.service

firewall-cmd –set-log-denied=all
firewall-cmd –reload
firewall-cmd –zone=dmz –list-all
firewall-cmd –zone=public –list-all

The result:

[root@ipaclient01 n]# showmount -e nfs02
Export list for nfs02:
/n (everyone)
[root@ipaclient01 n]# showmount -e nfs01
Export list for nfs01:
/n (everyone)
[root@ipaclient01 n]#

Perfect!  Now we have to consider NFS Ganesha HA.  There's a few solutions including having two of the same NFS servers both pointing to a common GlusterFS, then technically we only need a VIP and we're good.  But we need to consider two options: HAPROXY / keepalived or NFS Ganesha HA.  Alas,  HAPROXY and keepalived was not to be on first attempt.  We have a third party:

[root@nfs02 n]# yum install storhaug.noarch storhaug-nfs.noarch

But that brings in about 100 dependencies including pacemaker, corosync etc.  The point of the gluster setup we want here is to avoid this mess.  Plus there isn't much update to that storhaug project anyway.  So let's skip that and try with HAPROXY and keepalived . On both nodes, install haproxy and keepalived using yum then configure each file with below contents:

[root@nfs01 ~]# cat /etc/haproxy/haproxy.cfg|grep -Evi "#"

global
    log         127.0.0.1 local2

    chroot      /var/lib/haproxy
    pidfile     /var/run/haproxy.pid
    maxconn     4000
    user        haproxy
    group       haproxy
    daemon

    stats socket /var/lib/haproxy/stats

defaults
    mode                    http
    log                     global
    option                  httplog
    option                  dontlognull
    option http-server-close
    option forwardfor       except 127.0.0.0/8
    option                  redispatch
    retries                 3
    timeout http-request    10s
    timeout queue           1m
    timeout connect         10s
    timeout client          1m
    timeout server          1m
    timeout http-keep-alive 10s
    timeout check           10s
    maxconn                 3000

frontend  main *:80
    acl url_static       path_beg       -i /static /images /javascript /stylesheets
    acl url_static       path_end       -i .jpg .gif .png .css .js

    default_backend             app

backend static
    balance     roundrobin
    server      static 127.0.0.1:4331 check

backend app
    balance     roundrobin
    server nfs01.nix.mine.dom    192.168.0.131:80 check
    server nfs02.nix.mine.dom    192.168.0.119:80 check
[root@nfs01 ~]#

Before we setup the below keepalived configuration, we need to set two binding options in the kernel (More on this.) :

[root@nfs02 keepalived]# echo "net.ipv4.ip_nonlocal_bind = 1" >> /etc/sysctl.conf
[root@nfs02 keepalived]# echo "net.ipv4.ip_forward = 1" >> /etc/sysctl.conf
[root@nfs02 keepalived]# sysctl -p
net.ipv4.ip_nonlocal_bind = 1
net.ipv4.ip_forward = 1
[root@nfs02 keepalived]# 

Then define the keepalived.conf as follows:

[root@nfs01 ~]# cat /etc/keepalived/keepalived.conf
vrrp_script chk_haproxy {
  script "killall -0 haproxy"           # check the haproxy process
  interval 2                            # every 2 seconds
  weight 2                              # add 2 points if OK
}

vrrp_instance VI_1 {
  interface eth0                        # interface to monitor
  state MASTER                          # MASTER on haproxy1, BACKUP on haproxy2
  virtual_router_id 51
  priority 101                          # 101 on haproxy1, 100 on haproxy2
  virtual_ipaddress {
    192.168.0.80                        # virtual ip address
  }
  track_script {
    chk_haproxy
  }
}
[root@nfs01 ~]#

Source document on which above was modeled can be found here.  And create the DNS entries in the idmipa01 / idmipa02 FreeIPA servers we installed earlier.  Despite this, the failover did not work and even after the host comes back, the NFSv4 fails to recognize when mounted over nfs-c01 VIP between nfs01 and nfs02:  ( Later on, folks on the HAPROXY mailing list quickly corrected me that I'm using http instead of tcp.  My bad!  TY Guy's! )

[root@ipaclient01 ~]# cd /n
-bash: cd: /n: Stale file handle
[root@ipaclient01 ~]#

This is because NFSv4 is stateful and HAPROXY / keepalived might just be too simple for this but we'll try this later on.  So we return to the NFS Ganesha method and earlier repos.  We'll create a new repo with older GlusterFS binaries and install that.  But that means we have to tear everything to bare bones again. Here are our options:

  1. Download the older rpm's: wget –recursive –no-parent https://buildlogs.centos.org/centos/7/storage/x86_64/gluster-3.10/
     
  2. (Optional) Create the repo.  One might be already created after the download is finished.
     
  3. Create the yum repo file:
    [glusterfs-3.10]
    name = GlusterFS 3.10 Packages
    baseurl = file:///root/gluster-repo/buildlogs.centos.org/centos/7/storage/x86_64/gluster-3.10/
    enabled = 1
    gpgcheck = 1

     
  4. Alternately, you can just point to the online repo location.  Either way will do.
     
  5. Check that the older version is available using: yum –showduplicates list glusterfs

Trying the new configs while saving current and reinstalling nfs-ganesha to fix the earlier package problem.  We will force the package because we notice that if you try to upgrade nfs-ganesha or update libntirpc, it results in the exact same message either way.  Guessing they were made interdependent on each other and forcing libntirpc might allow us to user nfs-ganesha 2.5.3:

Error: Package: nfs-ganesha-2.5.5-1.el7.x86_64 (centos-gluster312)
           Requires: libntirpc.so.1.5(NTIRPC_1.5.4)(64bit)

Rats!  No luck.  Ok, let's start from a clean slate again:

[root@nfs02 ganesha]# rpm -e $(rpm -aq|grep -Ei "ganesha|storhaug")

[root@nfs02 ganesha]# cat export.conf
EXPORT{
    Export_Id = 1 ;                             # Export ID unique to each export
    Path = "/n";                                # Path of the volume to be exported. Eg: "/test_volume"

    FSAL {
        name = GLUSTER;
        hostname = "nfs02.nix.mine.dom";         # IP of one of the nodes in the trusted pool
        volume = "gv01";                        # Volume name. Eg: "test_volume"
    }

    Access_type = RW;                           # Access permissions
    Squash = No_root_squash;                    # To enable/disable root squashing
    Disable_ACL = FALSE;                        # To enable/disable ACL
    Pseudo = "/n";                              # NFSv4 pseudo path for this export. Eg: "/test_volume_pseudo"
    Protocols = "3","4";                        # NFS protocols supported
    Transports = "UDP","TCP" ;                  # Transport protocols supported
    SecType = "sys";                            # Security flavors supported
}
[root@nfs02 ganesha]# cat ganesha.conf
###################################################
#
# EXPORT
#
# To function, all that is required is an EXPORT
#
# Define the absolute minimal export
#
###################################################


NFS_Core_Param {
        Bind_addr=192.168.0.119;
        NFS_Port=2049;
        MNT_Port=20048;
        NLM_Port=38468;
        Rquota_Port=4501;
}

%include "/etc/ganesha/export.conf"
[root@nfs02 ganesha]#

Now we install the trouble package only:

yum install libntirpc.x86_64

This worked without problems as long as we pick the N-1 version.  Now we'll try to install nfs-ganesha again:

Success!  However on an unexpected reboot, we run into this issue: 

Feb 19 22:37:39 nfs01.nix.mine.dom haproxy-systemd-wrapper[2086]: haproxy-systemd-wrapper: executing /usr/sbin/haproxy -f /etc/haproxy/haproxy.cfg -p /run/haproxy.pid -Ds
Feb 19 22:37:39 nfs01.nix.mine.dom haproxy-systemd-wrapper[2086]: [ALERT] 049/223739 (2087) : Starting frontend nfs-in: cannot bind socket [192.168.0.80:2049]

And quickly find out it's selinux:  

type=SERVICE_START msg=audit(1519098125.807:194): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=haproxy comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
type=AVC msg=audit(1519098125.826:195): avc:  denied  { name_bind } for  pid=2389 comm="haproxy" src=2049 scontext=system_u:system_r:haproxy_t:s0 tcontext=system_u:object_r:nfs_port_t:s0 tclass=tcp_socket
type=SYSCALL msg=audit(1519098125.826:195): arch=c000003e syscall=49 success=no exit=-13 a0=5 a1=564f91774ed0 a2=10 a3=564f90eaf074 items=0 ppid=2388 pid=2389 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=4294967295 comm="haproxy" exe="/usr/sbin/haproxy" subj=system_u:system_r:haproxy_t:s0 key=(null)
type=PROCTITLE msg=audit(1519098125.826:195): proctitle=2F7573722F7362696E2F686170726F7879002D66002F6574632F686170726F78792F686170726F78792E636667002D70002F72756E2F686170726F78792E706964002D4473
type=SERVICE_STOP msg=audit(1519098125.842:196): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=haproxy comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=failed'

So we fix it in the same way we fixed the above issue:

grep AVC /var/log/audit/audit.log | tail -n1 | audit2allow -M systemd-allow
semodule -i systemd-allow.pp

And move on to testing further.  We will try to start haproxy first then start nfs-ganesha which binds to all the ports when using the RPM version.  We had to do this twice to get all the rules in.  We do the same on nfs02 and noticed a similar error when trying to get HAPROXY going:

Feb 19 23:02:45 nfs02.nix.mine.dom haproxy-systemd-wrapper[12156]: [ALERT] 049/230245 (12157) : Starting frontend nfs-in: cannot bind socket [192.168.0.80:2049]

We try the selinux route but still get the same error.  So we check using ip a and notice that the VIP from keepalived isn't up.  So we start keepalived.  Still doesn't work.  So we give up on the RPM method.

We are now trying to compile from source.  Links and packages you may need are below:

https://github.com/nfs-ganesha/nfs-ganesha/wiki/Compiling

https://github.com/nfs-ganesha/nfs-ganesha/wiki/GLUSTER

https://github.com/nfs-ganesha/nfs-ganesha/wiki/XFSLUSTRE

yum install glusterfs-api-devel.x86_64
yum install xfsprogs-devel.x86_64
yum install xfsprogs.x86_64
xfsdump-3.1.4-1.el7.x86_64
libguestfs-xfs-1.36.3-6.el7_4.3.x86_64
libntirpc-devel-1.5.4-1.el7.x86_64
libntirpc-1.5.4-1.el7.x86_64

 

COMMANDS

git clone https://github.com/nfs-ganesha/nfs-ganesha.git
cd nfs-ganesha;
git checkout V2.6-stable

git submodule update –init –recursive
ccmake /root/ganesha/nfs-ganesha/src/
# Press the c, e, c, g keys to create and generate the config and make files.
make
make install

 

Ensure you also enable IDMAPD support in NFS Ganesha or your UID / GID off the NFS mount will be nobody / nobody or nfsnobody / nfsnobody:

libnfsidmap-devel-0.25-17.el7.x86_64
jemalloc-devel-3.6.0-1.el7.x86_64

The options we want selected are as follows:

ALLOCATOR                            jemalloc
USE_FSAL_GLUSTER                     ON
USE_FSAL_XFS                         OFF
USE_NFSIDMAP                         ON

If you don't have above packages installed, you'll get an error message when selecting them via ccmake:

 CMake Warning at CMakeLists.txt:698 (message):
   jemalloc not found, falling back to libc

 CMake Warning at CMakeLists.txt:745 (message):
   libnfsidmap not found, disabling USE_NFSIDMAP

/root/nfs-ganesha/src/include/gsh_rpc.h:17:28: fatal error: rpc/xdr_inline.h: No such file or directory
 #include <rpc/xdr_inline.h>

If we also don't get the nfsidmapd configured, we'll get error messages such as this in the messages and secure log files:

Mar 15 23:13:06 ipaclient01 nfsidmap[4999]: key: 0x3b3559c4 type: uid value: tom@mds.xyz@localdomain timeout 600
Mar 15 23:13:06 ipaclient01 nfsidmap[4999]: nfs4_name_to_uid: calling umich_ldap->name_to_uid
Mar 15 23:13:06 ipaclient01 nfsidmap[4999]: ldap_init_and_bind: version mismatch between API information and protocol version. Setting protocol version to 3
Mar 15 23:13:06 ipaclient01 nfsidmap[4999]: nfs4_name_to_uid: umich_ldap->name_to_uid returned -2
Mar 15 23:13:06 ipaclient01 nfsidmap[4999]: nfs4_name_to_uid: calling nsswitch->name_to_uid
Mar 15 23:13:06 ipaclient01 nfsidmap[4999]: nss_getpwnam: name 'tom@mds.xyz@localdomain' domain 'nix.my.dom': resulting localname '(null)'
Mar 15 23:13:06 ipaclient01 nfsidmap[4999]: nss_getpwnam: name 'tom@mds.xyz@localdomain' does not map into domain 'nix.my.dom'
Mar 15 23:13:06 ipaclient01 nfsidmap[4999]: nfs4_name_to_uid: nsswitch->name_to_uid returned -22
Mar 15 23:13:06 ipaclient01 nfsidmap[4999]: nfs4_name_to_uid: final return value is -22
Mar 15 23:13:06 ipaclient01 nfsidmap[4999]: nfs4_name_to_uid: calling umich_ldap->name_to_uid
Mar 15 23:13:06 ipaclient01 nfsidmap[4999]: ldap_init_and_bind: version mismatch between API information and protocol version. Setting protocol version to 3
Mar 15 23:13:06 ipaclient01 nfsidmap[4999]: nfs4_name_to_uid: umich_ldap->name_to_uid returned -2
Mar 15 23:13:06 ipaclient01 nfsidmap[4999]: nfs4_name_to_uid: calling nsswitch->name_to_uid
Mar 15 23:13:06 ipaclient01 nfsidmap[4999]: nss_getpwnam: name 'nobody@nix.my.dom' domain 'nix.my.dom': resulting localname 'nobody'
Mar 15 23:13:06 ipaclient01 nfsidmap[4999]: nfs4_name_to_uid: nsswitch->name_to_uid returned 0
Mar 15 23:13:06 ipaclient01 nfsidmap[4999]: nfs4_name_to_uid: final return value is 0
Mar 15 23:13:06 ipaclient01 nfsidmap[5001]: key: 0x3140cc17 type: gid value: tom@mds.xyz@localdomain timeout 600
Mar 15 23:13:06 ipaclient01 nfsidmap[5001]: nfs4_name_to_gid: calling umich_ldap->name_to_gid
Mar 15 23:13:06 ipaclient01 nfsidmap[5001]: ldap_init_and_bind: version mismatch between API information and protocol version. Setting protocol version to 3
Mar 15 23:13:06 ipaclient01 nfsidmap[5001]: nfs4_name_to_gid: umich_ldap->name_to_gid returned -2
Mar 15 23:13:06 ipaclient01 nfsidmap[5001]: nfs4_name_to_gid: calling nsswitch->name_to_gid
Mar 15 23:13:06 ipaclient01 nfsidmap[5001]: nfs4_name_to_gid: nsswitch->name_to_gid returned -22
Mar 15 23:13:06 ipaclient01 nfsidmap[5001]: nfs4_name_to_gid: final return value is -22
Mar 15 23:13:06 ipaclient01 nfsidmap[5001]: nfs4_name_to_gid: calling umich_ldap->name_to_gid
Mar 15 23:13:06 ipaclient01 nfsidmap[5001]: ldap_init_and_bind: version mismatch between API information and protocol version. Setting protocol version to 3
Mar 15 23:13:06 ipaclient01 nfsidmap[5001]: nfs4_name_to_gid: umich_ldap->name_to_gid returned -2
Mar 15 23:13:06 ipaclient01 nfsidmap[5001]: nfs4_name_to_gid: calling nsswitch->name_to_gid
Mar 15 23:13:06 ipaclient01 nfsidmap[5001]: nfs4_name_to_gid: nsswitch->name_to_gid returned 0
Mar 15 23:13:06 ipaclient01 nfsidmap[5001]: nfs4_name_to_gid: final return value is 0 

A successful resolution looks like this:

Mar 18 10:28:28 ipaclient01 nfsidmap[19982]: key: 0x23cecc27 type: uid value: root@nix.mds.xyz timeout 600
Mar 18 10:28:28 ipaclient01 nfsidmap[19982]: nfs4_name_to_uid: calling nsswitch->name_to_uid
Mar 18 10:28:28 ipaclient01 nfsidmap[19982]: nss_getpwnam: name 'root@nix.mds.xyz' domain 'nix.mds.xyz': resulting localname 'root'
Mar 18 10:28:28 ipaclient01 nfsidmap[19982]: nss_name_to_uid: name 'root@nix.mds.xyz' uid 0
Mar 18 10:28:28 ipaclient01 nfsidmap[19982]: nfs4_name_to_uid: nsswitch->name_to_uid returned 0
Mar 18 10:28:28 ipaclient01 nfsidmap[19982]: nfs4_name_to_uid: final return value is 0
Mar 18 10:28:28 ipaclient01 nfsidmap[19984]: key: 0x3b0d8068 type: gid value: root@nix.mds.xyz timeout 600
Mar 18 10:28:28 ipaclient01 nfsidmap[19984]: nfs4_name_to_gid: calling nsswitch->name_to_gid
Mar 18 10:28:28 ipaclient01 nfsidmap[19984]: nss_name_to_gid: name 'root@nix.mds.xyz' domain 'nix.mds.xyz': resulting localname 'root'
Mar 18 10:28:28 ipaclient01 nfsidmap[19984]: nss_name_to_gid: name 'root@nix.mds.xyz' gid 0
Mar 18 10:28:28 ipaclient01 nfsidmap[19984]: nfs4_name_to_gid: nsswitch->name_to_gid returned 0
Mar 18 10:28:28 ipaclient01 nfsidmap[19984]: nfs4_name_to_gid: final return value is 0
Mar 18 10:28:28 ipaclient01 nfsidmap[19988]: key: 0x2adbb426 type: uid value: tom@mds.xyz@nix.mds.xyz timeout 600
Mar 18 10:28:28 ipaclient01 nfsidmap[19988]: nfs4_name_to_uid: calling nsswitch->name_to_uid
Mar 18 10:28:28 ipaclient01 nfsidmap[19988]: nss_getpwnam: name 'tom@mds.xyz@nix.mds.xyz' domain 'nix.mds.xyz': resulting localname 'tom@mds.xyz'
Mar 18 10:28:28 ipaclient01 nfsidmap[19988]: nss_name_to_uid: name 'tom@mds.xyz@nix.mds.xyz' uid 155601104
Mar 18 10:28:28 ipaclient01 nfsidmap[19988]: nfs4_name_to_uid: nsswitch->name_to_uid returned 0
Mar 18 10:28:28 ipaclient01 nfsidmap[19988]: nfs4_name_to_uid: final return value is 0
Mar 18 10:28:28 ipaclient01 nfsidmap[19990]: key: 0x6ef7be8 type: gid value: tom@mds.xyz@nix.mds.xyz timeout 600
Mar 18 10:28:28 ipaclient01 nfsidmap[19990]: nfs4_name_to_gid: calling nsswitch->name_to_gid
Mar 18 10:28:28 ipaclient01 nfsidmap[19990]: nss_name_to_gid: name 'tom@mds.xyz@nix.mds.xyz' domain 'nix.mds.xyz': resulting localname 'tom@mds.xyz'
Mar 18 10:28:28 ipaclient01 nfsidmap[19990]: nss_name_to_gid: name 'tom@mds.xyz@nix.mds.xyz' gid 155601104
Mar 18 10:28:28 ipaclient01 nfsidmap[19990]: nfs4_name_to_gid: nsswitch->name_to_gid returned 0
Mar 18 10:28:28 ipaclient01 nfsidmap[19990]: nfs4_name_to_gid: final return value is 0

When creating the config file, ensure to select GLUSTER FS:

[root@nfs02 nfs-ganesha]# ccmake /root/ganesha/nfs-ganesha/src

[root@nfs02 nfs-ganesha]# pwd
/root/ganesha/nfs-ganesha
[root@nfs02 nfs-ganesha]#

We successfully compiled from source the latest available version.  But now we get:

20/02/2018 08:56:51 : epoch 5a8c2923 : nfs01.nix.mine.dom : ganesha.nfsd-25728[main] load_fsal :NFS STARTUP :CRIT :Could not dlopen module:/usr/lib64/ganesha/libfsalgluster.so Error:/usr/lib64/ganesha/libfsalgluster.so:

So we need to copy the folling binaries to the appropriate folder and link the versions accordingly (Wonder why it didn't do it on it's own.):

[root@nfs01 src]# find / -iname libfsalgluster*
/root/ganesha/nfs-ganesha/src/FSAL/FSAL_GLUSTER/libfsalgluster.so.4.2.0
/root/ganesha/nfs-ganesha/src/FSAL/FSAL_GLUSTER/libfsalgluster.so.4
/root/ganesha/nfs-ganesha/src/FSAL/FSAL_GLUSTER/libfsalgluster.so
[root@nfs01 src]#

End state is as follows with proper symlinks created (on both nodes):

   537739 -rwxr-xr-x.  1 root root  507824 Feb 20 08:55 libfsalgluster.so.4.2.0
  1186043 lrwxrwxrwx.  1 root root      23 Feb 20 21:16 libfsalgluster.so.4 -> libfsalgluster.so.4.2.0
  1186045 lrwxrwxrwx.  1 root root      23 Feb 20 21:16 libfsalgluster.so -> libfsalgluster.so.4.2.0
   544176 drwxr-xr-x.  2 root root    4096 Feb 20 21:16 .
[root@nfs01 ganesha]# pwd
/usr/lib64/ganesha
[root@nfs01 ganesha]#

Then start up ganesha.nfsd again.  This time the logs are clear:

Let's also add XFS support since we use it very often.  We try but get the following despite these installed packages:

 CMake Warning at CMakeLists.txt:671 (message):
   Cannot find XFS runtime.  Disabling XFS build 

Let's investigate what library we need to enable XFS support:

/root/ganesha/nfs-ganesha/src/CMakeLists.txt

[root@nfs01 nfs-ganesha]# grep -EiR "Cannot find XFS runtime. Disabling XFS build" *
src/CMakeLists.txt:      message(FATAL_ERROR "STRICT_PACKAGE: Cannot find XFS runtime. Disabling XFS build")
src/CMakeLists.txt:      message(WARNING "Cannot find XFS runtime. Disabling XFS build")
[root@nfs01 nfs-ganesha]#

if(USE_FSAL_XFS)

  if(EXISTS /lib/libhandle.so)
    check_library_exists(handle "open_by_handle" "/./lib" HAVE_XFS_LIB)
    if(HAVE_XFS_LIB)
      set(PATH_LIBHANDLE "/lib/libhandle.so" CACHE INTERNAL "debian stretch and ubuntu xenial hack")
    endif(HAVE_XFS_LIB)
  else(EXISTS /lib/libhandle.so)
    check_library_exists(handle "open_by_handle" "" HAVE_XFS_LIB)
  endif(EXISTS /lib/libhandle.so)
  check_include_files("xfs/xfs.h" HAVE_XFS_H)
  if((NOT HAVE_XFS_LIB) OR (NOT HAVE_XFS_H))
   if(STRICT_PACKAGE)
      message(FATAL_ERROR "STRICT_PACKAGE: Cannot find XFS runtime. Disabling XFS build")
    else(STRICT_PACKAGE)
      message(WARNING "Cannot find XFS runtime. Disabling XFS build")
      set(USE_FSAL_XFS OFF)
    endif(STRICT_PACKAGE)
  endif((NOT HAVE_XFS_LIB) OR (NOT HAVE_XFS_H))
endif(USE_FSAL_XFS)

So we need to ensure /lib/libhandle.so is installed.  Let's check that then:

[root@nfs01 src]# ls -altri /lib/libhandle.so
ls: cannot access /lib/libhandle.so: No such file or directory
[root@nfs01 src]# find / -iname libhandle.so
/usr/lib64/libhandle.so
[root@nfs01 src]# ls -altri /usr/lib64/libhandle.so
203702437 lrwxrwxrwx. 1 root root 14 Feb 20 02:40 /usr/lib64/libhandle.so -> libhandle.so.1
[root@nfs01 src]# cd /usr/lib64/
[root@nfs01 lib64]# rpm -aq|grep -Ei ^C
[root@nfs01 lib64]# rpm -qf /usr/lib64/libhandle.so
xfsprogs-devel-4.5.0-12.el7.x86_64
[root@nfs01 lib64]# rpm -aq|grep -Ei xfs
xfsprogs-4.5.0-12.el7.x86_64
xfsprogs-devel-4.5.0-12.el7.x86_64
xfsdump-3.1.4-1.el7.x86_64
libguestfs-xfs-1.36.3-6.el7_4.3.x86_64
[root@nfs01 lib64]#

Apparently there's a slight typo in the config file?  Let's check further with a find on xfs.h using find / -iname xfs.h:

[root@nfs01 src]# find / -iname xfs.h
/usr/include/xfs/xfs.h
[root@nfs01 src]# 

Appears our /usr/include isn't right somewhere?  Until we can get this from an RPM package, we'll have to go without NFS Ganesha.  No idea, so let's just start ganesha.nfsd without XFS for now pending an email to the nfs-ganesha mailing lists.  Sure enough, in the latest version 2.60, when compiled from source, binding to a single IP address has been permanently fixed:

[root@nfs01 ganesha]# netstat -pnlt|grep -Ei ganesha
tcp6       0      0 192.168.0.131:20048     :::*                    LISTEN      13714/ganesha.nfsd
tcp6       0      0 :::564                  :::*                    LISTEN      13714/ganesha.nfsd
tcp6       0      0 192.168.0.131:4501      :::*                    LISTEN      13714/ganesha.nfsd
tcp6       0      0 192.168.0.131:2049      :::*                    LISTEN      13714/ganesha.nfsd
tcp6       0      0 192.168.0.131:38468     :::*                    LISTEN      13714/ganesha.nfsd
[root@nfs01 ganesha]#

 

Check also the second node:  

[root@nfs02 ganesha]# netstat -pnlt|grep -Ei ganesha
tcp6       0      0 192.168.0.119:20048     :::*                    LISTEN      12409/ganesha.nfsd
tcp6       0      0 :::564                  :::*                    LISTEN      12409/ganesha.nfsd
tcp6       0      0 192.168.0.119:4501      :::*                    LISTEN      12409/ganesha.nfsd
tcp6       0      0 192.168.0.119:2049      :::*                    LISTEN      12409/ganesha.nfsd
tcp6       0      0 192.168.0.119:38468     :::*                    LISTEN      12409/ganesha.nfsd
[root@nfs02 ganesha]#

Let's start up haproxy configuration we used above and see if we can now bind appropriately:

[root@nfs02 haproxy]# netstat -pnlt|grep -Ei haproxy
tcp        0      0 192.168.0.80:2049       0.0.0.0:*               LISTEN      13652/haproxy
[root@nfs02 haproxy]#

Bingo!  Works, check further:

[root@nfs01 ganesha]# netstat -pnlt|grep -Ei haproxy
tcp        0      0 192.168.0.80:2049       0.0.0.0:*               LISTEN      16014/haproxy
[root@nfs01 ganesha]# netstat -pnlt|grep -Ei ganesha
tcp6       0      0 192.168.0.131:20048     :::*                    LISTEN      13714/ganesha.nfsd
tcp6       0      0 :::564                  :::*                    LISTEN      13714/ganesha.nfsd
tcp6       0      0 192.168.0.131:4501      :::*                    LISTEN      13714/ganesha.nfsd
tcp6       0      0 192.168.0.131:2049      :::*                    LISTEN      13714/ganesha.nfsd
tcp6       0      0 192.168.0.131:38468     :::*                    LISTEN      13714/ganesha.nfsd
[root@nfs01 ganesha]#

Perfect!  Now we have the two products working properly.  Time to test mounting on clients and failover testing.  In our case we see an error with gluster on the second cluster node so we need to fix this first:

[root@nfs02 n]# gluster volume status
Status of volume: gv01
Gluster process                             TCP Port  RDMA Port  Online  Pid
——————————————————————————
Brick nfs01:/bricks/0/gv01                  49152     0          Y       1377
Brick nfs02:/bricks/0/gv01                  N/A       N/A        N       N/A
Self-heal Daemon on localhost               N/A       N/A        Y       1193
Self-heal Daemon on nfs01                   N/A       N/A        Y       1151

Task Status of Volume gv01
——————————————————————————
There are no active volume tasks

[root@nfs02 n]#

Run a restart then check gluster status again:

[root@nfs02 n]# systemctl restart glusterd
[root@nfs02 n]# gluster volume status
Status of volume: gv01
Gluster process                             TCP Port  RDMA Port  Online  Pid
——————————————————————————
Brick nfs01:/bricks/0/gv01                  49152     0          Y       1377
Brick nfs02:/bricks/0/gv01                  49152     0          Y       16103
Self-heal Daemon on localhost               N/A       N/A        Y       16094
Self-heal Daemon on nfs01                   N/A       N/A        Y       1151

Task Status of Volume gv01
——————————————————————————
There are no active volume tasks

[root@nfs02 n]#

March 23 2018

So we encounter this issue and our nfs-c01 VIP can't be used to mount from clients:

[2018-03-23 05:01:56.944175] E [MSGID: 114058] [client-handshake.c:1565:client_query_portmap_cbk] 0-gv01-client-0: failed to get the port number for remote subvolume. Please run 'gluster volume status' on server to see if brick process is running.
[2018-03-23 05:01:56.944921] I [MSGID: 114018] [client.c:2285:client_rpc_notify] 0-gv01-client-0: disconnected from gv01-client-0. Client process will keep trying to connect to glusterd until brick's port is available
[2018-03-23 05:01:56.944962] E [MSGID: 108006] [afr-common.c:5006:__afr_handle_child_down_event] 0-gv01-replicate-0: All subvolumes are down. Going offline until atleast one of them comes back up.
Final graph:
+——————————————————————————+
  1: volume gv01-client-0
  2:     type protocol/client
  3:     option ping-timeout 42
  4:     option remote-host nfs01
  5:     option remote-subvolume /bricks/0/gv01
  6:     option transport-type socket
  7:     option transport.address-family inet
  8:     option username 916ccf06-dc1d-467f-bc3d-f00a7449618f
  9:     option password a44739e0-9587-411f-8e6a-9a6a4e46156c
 10:     option event-threads 8
 11:     option transport.tcp-user-timeout 0
 12:     option transport.socket.keepalive-time 20
 13:     option transport.socket.keepalive-interval 2
 14:     option transport.socket.keepalive-count 9
 15:     option send-gids true
 16: end-volume
 17:
 18: volume gv01-client-1
 19:     type protocol/client
 20:     option ping-timeout 42
 21:     option remote-host nfs02
 22:     option remote-subvolume /bricks/0/gv01
 23:     option transport-type socket
 24:     option transport.address-family inet
 25:     option username 916ccf06-dc1d-467f-bc3d-f00a7449618f
 26:     option password a44739e0-9587-411f-8e6a-9a6a4e46156c
 27:     option event-threads 8
 28:     option transport.tcp-user-timeout 0
 29:     option transport.socket.keepalive-time 20
 30:     option transport.socket.keepalive-interval 2
 31:     option transport.socket.keepalive-count 9
 32:     option send-gids true
 33: end-volume
 34:
 35: volume gv01-replicate-0
 36:     type cluster/replicate
 37:     option afr-pending-xattr gv01-client-0,gv01-client-1
 38:     option quorum-type auto
 39:     option use-compound-fops off
 40:     subvolumes gv01-client-0 gv01-client-1
 41: end-volume
 42:
 43: volume gv01-dht
 44:     type cluster/distribute
 45:     option lock-migration off
 46:     subvolumes gv01-replicate-0
 47: end-volume
 48:
 49: volume gv01-write-behind
 50:     type performance/write-behind
 51:     option cache-size 8MB
 52:     subvolumes gv01-dht
 53: end-volume
 54:
 55: volume gv01-read-ahead
[2018-03-23 05:01:56.950848] E [MSGID: 114058] [client-handshake.c:1565:client_query_portmap_cbk] 0-gv01-client-1: failed to get the port number for remote subvolume. Please run 'gluster volume status' on server to see if brick process is running.
 56:     type performance/read-ahead
 57:     subvolumes gv01-write-behind
 58: end-volume
 59:
 60: volume gv01-readdir-ahead
 61:     type performance/readdir-ahead
 62:     option parallel-readdir off
 63:     option rda-request-size 131072
 64:     option rda-cache-limit 10MB
 65:     subvolumes gv01-read-ahead
 66: end-volume
[2018-03-23 05:01:56.950954] I [MSGID: 114018] [client.c:2285:client_rpc_notify] 0-gv01-client-1: disconnected from gv01-client-1. Client process will keep trying to connect to glusterd until brick's port is available
 67:
[2018-03-23 05:01:56.951029] E [MSGID: 108006] [afr-common.c:5006:__afr_handle_child_down_event] 0-gv01-replicate-0: All subvolumes are down. Going offline until atleast one of them comes back up.
 68: volume gv01-io-cache
 69:     type performance/io-cache
 70:     option cache-size 1GB
 71:     subvolumes gv01-readdir-ahead
 72: end-volume
 73:
 74: volume gv01-quick-read
 75:     type performance/quick-read
 76:     option cache-size 1GB
 77:     subvolumes gv01-io-cache
 78: end-volume
 79:
 80: volume gv01-open-behind
 81:     type performance/open-behind
 82:     subvolumes gv01-quick-read
 83: end-volume
 84:
 85: volume gv01-md-cache
 86:     type performance/md-cache
 87:     subvolumes gv01-open-behind
 88: end-volume
 89:
 90: volume gv01
 91:     type debug/io-stats
 92:     option log-level INFO
 93:     option latency-measurement off
 94:     option count-fop-hits off
 95:     subvolumes gv01-md-cache
 96: end-volume
 97:
 98: volume meta-autoload
 99:     type meta
100:     subvolumes gv01
101: end-volume
102:
+——————————————————————————+
[2018-03-23 05:01:56.952617] I [io-stats.c:4076:fini] 0-gv01: io-stats translator unloaded
[2018-03-23 05:01:56.953591] I [MSGID: 101191] [event-epoll.c:644:event_dispatch_epoll_worker] 0-epoll: Exited thread with index 1
[2018-03-23 05:01:56.954172] I [MSGID: 101191] [event-epoll.c:644:event_dispatch_epoll_worker] 0-epoll: Exited thread with index 8
^C
[root@nfs02 ganesha]#

We simply run:

mount -a

then check if gluster is mounted on both nfs01 and nfs02:

[root@nfs01 ~]# mount|grep glusterfs
nfs01:/gv01 on /n type fuse.glusterfs (rw,relatime,user_id=0,group_id=0,default_permissions,allow_other,max_read=131072)
[root@nfs01 ~]#

Potentially restart NFS-Ganesha on both clients one by one to allow the mounts to work.  If it still doesn't work, it could be because of this:

[root@nfs02 glusterfs]# nc -v 192.168.0.131 49154
Ncat: Version 6.40 ( http://nmap.org/ncat )
Ncat: No route to host.
[root@nfs02 glusterfs]#
[root@nfs02 glusterfs]#
[root@nfs02 glusterfs]#
[root@nfs02 glusterfs]# systemctl stop firewalld
[root@nfs02 glusterfs]# nc -v 192.168.0.131 49154
Ncat: Version 6.40 ( http://nmap.org/ncat )
Ncat: Connected to 192.168.0.131:49154.
^C
[root@nfs02 glusterfs]#

And this in the gluster logs:

[2018-03-23 06:00:54.425845] I [rpc-clnt.c:1986:rpc_clnt_reconfig] 0-gv01-client-0: changing port to 49154 (from 0)
[2018-03-23 06:00:54.433214] E [socket.c:2369:socket_connect_finish] 0-gv01-client-0: connection to 192.168.0.131:49154 failed (No route to host); disconnecting socket

Looks like our port may be too restrictive.  Adjust it.

SUMMARY

Here's a summary configuration for this whole work:

HOST SETTING DESCRIPTION
nfs01 / nfs02

Create and reserve some IP's for your hosts.  We are using the FreeIPA project to provide DNS and Kerberos functionality here:

192.168.0.80 nfs-c01 (nfs01, nfs02)  VIP DNS Entry
192.168.0.131 nfs01
192.168.0.119 nfs02

Add the hosts to your DNS server for a clean setup. Alternately  add them to /etc/hosts (ugly)
nfs01 / nfs02

wget https://github.com/nfs-ganesha/nfs-ganesha/archive/V2.6-.0.tar.gz

[root@nfs01 ~]# ganesha.nfsd -v
NFS-Ganesha Release = V2.6.0
nfs-ganesha compiled on Feb 20 2018 at 08:55:23
Release comment = GANESHA file server is 64 bits compliant and supports NFS v3,4.0,4.1 (pNFS) and 9P
Git HEAD = 97867975b2ee69d475876e222c439b1bc9764a78
Git Describe = V2.6-.0-0-g9786797
[root@nfs01 ~]#

DETAILED INSTRUCTIONS:

https://github.com/nfs-ganesha/nfs-ganesha/wiki/Compiling

https://github.com/nfs-ganesha/nfs-ganesha/wiki/GLUSTER
https://github.com/nfs-ganesha/nfs-ganesha/wiki/XFSLUSTRE

PACKAGES:

yum install glusterfs-api-devel.x86_64
yum install xfsprogs-devel.x86_64
yum install xfsprogs.x86_64
xfsdump-3.1.4-1.el7.x86_64
libguestfs-xfs-1.36.3-6.el7_4.3.x86_64
libntirpc-devel-1.5.4-1.el7.x86_64
libntirpc-1.5.4-1.el7.x86_64

libnfsidmap-devel-0.25-17.el7.x86_64

jemalloc-devel-3.6.0-1.el7.x86_64

COMMANDS

git clone https://github.com/nfs-ganesha/nfs-ganesha.git
cd nfs-ganesha;
git checkout V2.6-stable

git submodule update –init –recursive
ccmake /root/ganesha/nfs-ganesha/src/
# Press the c, e, c, g keys to create and generate the config and make files.
make
make install

 

Compile and build
nfsganesha 2.60+
from source.  (At
this time RPM
packages did not work) Install the listed packages before compiling as well.

 

 

 

 

 

 

 

 

 

 

 

 

 

 

nfs01 / nfs02 Add a disk to the VM such as /dev/sdb . Add secondary
disk for the
shared GlusterFS
nfs01 / nfs02 


mkfs.xfs /dev/sdb
mount /dev/sdb /bricks/0

yum install centos-release-gluster
systemctl enable glusterd.service
yum -y install glusterfs glusterfs-fuse glusterfs-server glusterfs-api glusterfs-cli
( node01 ONLY ) gluster volume create gv01 replica 2 nfs01:/bricks/0/gv01 nfs02:/bricks/0/gv01

gluster volume info
gluster volume status

Configure the
GlusterFS filesystem
using
nfs01 / nfs02

PACKAGES:
yum install haproxy     # ( 1.5.18-6.el7.x86_64 used in this case )

/etc/haproxy/haproxy.cfg

global
    log         127.0.0.1 local2
    stats       socket /var/run/haproxy.sock mode 0600 level admin
    # stats     socket /var/lib/haproxy/stats
    maxconn     4000
    user        haproxy
    group       haproxy
    daemon
    debug

defaults
    mode                    tcp
    log                     global
    option                  dontlognull
    option                  redispatch
    retries                 3
    timeout http-request    10s
    timeout queue           1m
    timeout connect         10s
    timeout client          1m
    timeout server          1m
    timeout http-keep-alive 10s
    timeout check           10s
    maxconn                 3000

frontend nfs-in
    bind nfs-c01:2049
    mode tcp
    option tcplog
    default_backend             nfs-back


backend nfs-back
    balance     roundrobin
    server      nfs01.nix.mine.dom    nfs01.nix.mine.dom:2049 check
    server      nfs02.nix.mine.dom    nfs02.nix.mine.dom:2049 check

Install and
Configure HAPROXY. great source that

helped with this part.

nfs01 / nfs02 # echo "net.ipv4.ip_nonlocal_bind = 1" >> /etc/sysctl.conf


# echo "net.ipv4.ip_forward = 1" >> /etc/sysctl.conf


# sysctl -p


net.ipv4.ip_nonlocal_bind = 1


net.ipv4.ip_forward = 1


Turn on kernel parameters.  These allow keepalived below to function properly.
nfs01 / nfs02 

PACKAGES:

yum install keepalived    # ( Used 1.3.5-1.el7.x86_64 in this case )

NFS01:

vrrp_script chk_haproxy {
  script "killall -0 haproxy"           # check the haproxy process
  interval 2                            # every 2 seconds
  weight 2                              # add 2 points if OK
}

vrrp_instance VI_1 {
  interface eth0                        # interface to monitor
  state MASTER                          # MASTER on haproxy1, BACKUP on haproxy2
  virtual_router_id 51
  priority 101                          # 101 on haproxy1, 100 on haproxy2
  virtual_ipaddress {
       192.168.0.80                        # virtual ip address
  }
  track_script {
       chk_haproxy
  }
}

NFS02:

vrrp_script chk_haproxy {
  script "killall -0 haproxy"           # check the haproxy process
  interval 2                            # every 2 seconds
  weight 2                              # add 2 points if OK
}

vrrp_instance VI_1 {
  interface eth0                        # interface to monitor
  state BACKUP                          # MASTER on haproxy1, BACKUP on haproxy2
  virtual_router_id 51
  priority 102                          # 101 on haproxy1, 100 on haproxy2
  virtual_ipaddress {
    192.168.0.80                        # virtual ip address
  }
  track_script {
    chk_haproxy
  }
}

Configure keepalived. A great source that helped with this as well.
 

nfs01 / nfs02

# cat public.bash

firewall-cmd –zone=public –permanent –add-port=2049/tcp

firewall-cmd –zone=public –permanent –add-port=111/tcp

firewall-cmd –zone=public –permanent –add-port=111/udp

firewall-cmd –zone=public –permanent –add-port=24007-24008/tcp

firewall-cmd –zone=public –permanent –add-port=49152/tcp

firewall-cmd –zone=public –permanent –add-port=38465-38469/tcp

firewall-cmd –zone=public –permanent –add-port=4501/tcp

firewall-cmd –zone=public –permanent –add-port=4501/udp

firewall-cmd –zone=public –permanent –add-port=20048/udp

firewall-cmd –zone=public –permanent –add-port=20048/tcp
firewall-cmd –zone=public –permanent –add-port=49000-59999/tcp

firewall-cmd –zone=public –permanent –add-port=49000-59999/udp
firewall-cmd –reload
 

 

# cat dmz.bash

firewall-cmd –zone=dmz –permanent –add-port=2049/tcp

firewall-cmd –zone=dmz –permanent –add-port=111/tcp

firewall-cmd –zone=dmz –permanent –add-port=111/udp

firewall-cmd –zone=dmz –permanent –add-port=24007-24008/tcp

firewall-cmd –zone=dmz –permanent –add-port=49152/tcp

firewall-cmd –zone=dmz –permanent –add-port=38465-38469/tcp

firewall-cmd –zone=dmz –permanent –add-port=4501/tcp

firewall-cmd –zone=dmz –permanent –add-port=4501/udp

firewall-cmd –zone=dmz –permanent –add-port=20048/tcp

firewall-cmd –zone=dmz –permanent –add-port=20048/udp
firewall-cmd –zone=dmz –permanent –add-port=49000-59999/udp

firewall-cmd –zone=dmz –permanent –add-port=49000-59999/tcp
firewall-cmd –reload

#

# On Both

firewall-cmd –permanent –direct –add-rule ipv4 filter INPUT 0 -m pkttype –pkt-type multicast -j ACCEPT
firewall-cmd –reload

 

HANDY STUFF:

firewall-cmd –zone=dmz –list-all
firewall-cmd –zone=public –list-all
firewall-cmd –set-log-denied=all
firewall-cmd –permanent –add-service=haproxy
firewall-cmd –list-all
firewall-cmd –runtime-to-permanent

Configure firewalld.
DO NOT
disable
firewalld .
nfs01 / nfs02

Run any of the following command, or a combination of, on deny entries in /var/log/audit/audit.log that may appear as you stop, start or install above services:

METHOD 1:
grep AVC /var/log/audit/audit.log | tail -n1 | audit2allow -M systemd-allow
semodule -i systemd-allow.pp

METHOD 2:
audit2allow -a
audit2allow -a -M ganesha_2049_port
semodule -i ganesha_2049_port.pp

Configure selinux. 
Don't disable it.
This actually  makes your host safer and is actually
easy to work with using just these commands.
nfs01 / nfs02

NODE 1:

[root@nfs01 ~]# cat /etc/ganesha/ganesha.conf
###################################################
#
# EXPORT
#
# To function, all that is required is an EXPORT
#
# Define the absolute minimal export
#
###################################################


NFS_Core_Param {
        Bind_addr = 192.168.0.131;
        NFS_Port = 2049;
        MNT_Port = 20048;
        NLM_Port = 38468;
        Rquota_Port = 4501;
}

%include "/etc/ganesha/export.conf"
[root@nfs01 ~]# cat /etc/ganesha/export.conf
EXPORT{
    Export_Id = 1 ;                             # Export ID unique to each export
    Path = "/n";                                # Path of the volume to be exported. Eg: "/test_volume"

    FSAL {
        name = GLUSTER;
        hostname = "nfs01.nix.mine.dom";         # IP of one of the nodes in the trusted pool
        volume = "gv01";                        # Volume name. Eg: "test_volume"
    }

    Access_type = RW;                           # Access permissions
    Squash = No_root_squash;                    # To enable/disable root squashing
    Disable_ACL = FALSE;                        # To enable/disable ACL
    Pseudo = "/n";                              # NFSv4 pseudo path for this export. Eg: "/test_volume_pseudo"
    Protocols = "3","4";                        # NFS protocols supported
    Transports = "UDP","TCP" ;                  # Transport protocols supported
    SecType = "sys";                            # Security flavors supported
}
[root@nfs01 ~]#

NODE 2:

[root@nfs02 ~]# cd /etc/ganesha/
[root@nfs02 ganesha]# cat ganesha.conf
###################################################
#
# EXPORT
#
# To function, all that is required is an EXPORT
#
# Define the absolute minimal export
#
###################################################


NFS_Core_Param {
        Bind_addr=192.168.0.119;
        NFS_Port=2049;
        MNT_Port=20048;
        NLM_Port=38468;
        Rquota_Port=4501;
}

%include "/etc/ganesha/export.conf"
[root@nfs02 ganesha]# cat export.conf
EXPORT{
    Export_Id = 1 ;                             # Export ID unique to each export
    Path = "/n";                                # Path of the volume to be exported. Eg: "/test_volume"

    FSAL {
        name = GLUSTER;
        hostname = "nfs02.nix.mine.dom";         # IP of one of the nodes in the trusted pool
        volume = "gv01";                        # Volume name. Eg: "test_volume"
    }

    Access_type = RW;                           # Access permissions
    Squash = No_root_squash;                    # To enable/disable root squashing
    Disable_ACL = FALSE;                        # To enable/disable ACL
    Pseudo = "/n";                              # NFSv4 pseudo path for this export. Eg: "/test_volume_pseudo"
    Protocols = "3","4";                        # NFS protocols supported
    Transports = "UDP","TCP" ;                  # Transport protocols supported
    SecType = "sys";                            # Security flavors supported
}
[root@nfs02 ganesha]#

STARTUP:

/usr/bin/ganesha.nfsd -L /var/log/ganesha/ganesha.log -f /etc/ganesha/ganesha.conf -N NIV_EVENT
 

Configure NFS Ganesha
nfs01 / nfs02

 

[root@nfs01 ~]# cat /etc/fstab|grep -Ei "brick|gv01"
/dev/sdb /bricks/0                              xfs     defaults        0 0
nfs01:/gv01 /n                                  glusterfs defaults      0 0
[root@nfs01 ~]#

[root@nfs01 ~]# mount|grep -Ei "brick|gv01"
/dev/sdb on /bricks/0 type xfs (rw,relatime,seclabel,attr2,inode64,noquota)
nfs01:/gv01 on /n type fuse.glusterfs (rw,relatime,user_id=0,group_id=0,default_permissions,allow_other,max_read=131072)
[root@nfs01 ~]#

 

[root@nfs01 ~]# ps -ef|grep -Ei "haproxy|keepalived|ganesha"; netstat -pnlt|grep -Ei "haproxy|ganesha|keepalived"
root      1402     1  0 00:59 ?        00:00:00 /usr/sbin/keepalived -D
root      1403  1402  0 00:59 ?        00:00:00 /usr/sbin/keepalived -D
root      1404  1402  0 00:59 ?        00:00:02 /usr/sbin/keepalived -D
root     13087     1  0 01:02 ?        00:00:00 /usr/sbin/haproxy-systemd-wrapper -f /etc/haproxy/haproxy.cfg -p /run/haproxy.pid
haproxy  13088 13087  0 01:02 ?        00:00:00 /usr/sbin/haproxy -f /etc/haproxy/haproxy.cfg -p /run/haproxy.pid -Ds
haproxy  13089 13088  0 01:02 ?        00:00:01 /usr/sbin/haproxy -f /etc/haproxy/haproxy.cfg -p /run/haproxy.pid -Ds
root     13129     1 15 01:02 ?        00:13:11 /usr/bin/ganesha.nfsd -L /var/log/ganesha/ganesha.log -f /etc/ganesha/ganesha.conf -N NIV_EVENT
root     19742 15633  0 02:30 pts/2    00:00:00 grep –color=auto -Ei haproxy|keepalived|ganesha
tcp        0      0 192.168.0.80:2049       0.0.0.0:*               LISTEN      13089/haproxy
tcp6       0      0 192.168.0.131:20048     :::*                    LISTEN      13129/ganesha.nfsd
tcp6       0      0 :::564                  :::*                    LISTEN      13129/ganesha.nfsd
tcp6       0      0 192.168.0.131:4501      :::*                    LISTEN      13129/ganesha.nfsd
tcp6       0      0 192.168.0.131:2049      :::*                    LISTEN      13129/ganesha.nfsd
tcp6       0      0 192.168.0.131:38468     :::*                    LISTEN      13129/ganesha.nfsd
[root@nfs01 ~]#

 

Ensure mounts are
done and everything
is started up.
nfs01 / nfs02

yumdownloader nfs-ganesha.x86_64
rpm2cpio nfs-ganesha-2.5.5-1.el7.x86_64.rpm | cpio -idmv ./usr/lib/systemd/system/nfs-ganesha-lock.service
rpm2cpio nfs-ganesha-2.5.5-1.el7.x86_64.rpm | cpio -idmv ./usr/lib/systemd/system/nfs-ganesha.service
rpm2cpio nfs-ganesha-2.5.5-1.el7.x86_64.rpm | cpio -idmv ./usr/lib/systemd/system/nfs-ganesha-config.service
rpm2cpio nfs-ganesha-2.5.5-1.el7.x86_64.rpm | cpio -idmv ./usr/libexec/ganesha/nfs-ganesha-config.sh

Copy above to the same folders under / instead of ./ :

systemctl enable nfs-ganesha.service
systemctl status nfs-ganesha.service

Since you compiled from source you don't have nice startup scripts.  To get your nice startup scripts from an existing ganesha RPM do the following.  Then use systemctl to stop and start nfs-ganesha as you would any other service.
 

TESTING

Now let's do some checks on our NFS HA.  Mount the share using the VIP from a client then create a test file:

[root@ipaclient01 /]# mount -t nfs4 nfs-c01:/n /n
[root@ipaclient01 n]# echo -ne "Hacked It.  Gluster, NFS Ganesha, HAPROXY, keepalived scalable NFS server." > some-people-find-this-awesome.txt

[root@ipaclient01 n]# mount|grep nfs4
nfs-c01:/n on /n type nfs4 (rw,relatime,vers=4.0,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,port=0,timeo=600,retrans=2,sec=sys,clientaddr=192.168.0.236,local_lock=none,addr=192.168.0.80)
[root@ipaclient01 n]#

 

Then check each brick to see if the file was replicated:

[root@nfs01 n]# cat /bricks/0/gv01/some-people-find-this-awesome.txt
Hacked It.  Gluster, NFS Ganesha, HAPROXY, keepalived scalable NFS server.
[root@nfs01 n]# mount|grep -Ei gv01
nfs01:/gv01 on /n type fuse.glusterfs (rw,relatime,user_id=0,group_id=0,default_permissions,allow_other,max_read=131072)
[root@nfs01 n]#

[root@nfs02 n]# cat /bricks/0/gv01/some-people-find-this-awesome.txt
Hacked It.  Gluster, NFS Ganesha, HAPROXY, keepalived scalable NFS server.
[root@nfs02 n]# mount|grep -Ei gv01
nfs02:/gv01 on /n type fuse.glusterfs (rw,relatime,user_id=0,group_id=0,default_permissions,allow_other,max_read=131072)
[root@nfs02 n]#

Good!  Now let's hard shutdown one node, nfs01, the primary node.  Expected behaviour is that we need to see failover to nfs02 and then when we bring back the nfs01 server, we need to see the file is replicated.  While we do this, the client ipaclient01 is not supposed to loose any connection to the NFS mount via the VIP.  Here are the results:

[root@nfs02 n]# ps -ef|grep -Ei "haproxy|ganesha|keepalived"
root     12245     1  0 Feb19 ?        00:00:03 /usr/sbin/keepalived -D
root     12246 12245  0 Feb19 ?        00:00:03 /usr/sbin/keepalived -D
root     12247 12245  0 Feb19 ?        00:00:41 /usr/sbin/keepalived -D
root     12409     1 16 Feb20 ?        00:13:05 /usr/bin/ganesha.nfsd -L /var/log/ganesha/ganesha.log -f /etc/ganesha/ganesha.conf -N NIV_EVENT
root     17892     1  0 00:37 ?        00:00:00 /usr/sbin/haproxy-systemd-wrapper -f /etc/haproxy/haproxy.cfg -p /run/haproxy.pid
haproxy  17893 17892  0 00:37 ?        00:00:00 /usr/sbin/haproxy -f /etc/haproxy/haproxy.cfg -p /run/haproxy.pid -Ds
haproxy  17894 17893  0 00:37 ?        00:00:00 /usr/sbin/haproxy -f /etc/haproxy/haproxy.cfg -p /run/haproxy.pid -Ds
root     17918 21084  0 00:38 pts/0    00:00:00 grep –color=auto -Ei haproxy|ganesha|keepalived
[root@nfs02 n]# ps -ef|grep -Ei "haproxy|ganesha|keepalived"; netstat -pnlt|grep -Ei ganesha; netstat -pnlt|grep -Ei haproxy; netstat -pnlt|grep -Ei keepalived
root     12245     1  0 Feb19 ?        00:00:03 /usr/sbin/keepalived -D
root     12246 12245  0 Feb19 ?        00:00:03 /usr/sbin/keepalived -D
root     12247 12245  0 Feb19 ?        00:00:41 /usr/sbin/keepalived -D
root     12409     1 16 Feb20 ?        00:13:09 /usr/bin/ganesha.nfsd -L /var/log/ganesha/ganesha.log -f /etc/ganesha/ganesha.conf -N NIV_EVENT
root     17892     1  0 00:37 ?        00:00:00 /usr/sbin/haproxy-systemd-wrapper -f /etc/haproxy/haproxy.cfg -p /run/haproxy.pid
haproxy  17893 17892  0 00:37 ?        00:00:00 /usr/sbin/haproxy -f /etc/haproxy/haproxy.cfg -p /run/haproxy.pid -Ds
haproxy  17894 17893  0 00:37 ?        00:00:00 /usr/sbin/haproxy -f /etc/haproxy/haproxy.cfg -p /run/haproxy.pid -Ds
root     17947 21084  0 00:38 pts/0    00:00:00 grep –color=auto -Ei haproxy|ganesha|keepalived
tcp6       0      0 192.168.0.119:20048     :::*                    LISTEN      12409/ganesha.nfsd
tcp6       0      0 :::564                  :::*                    LISTEN      12409/ganesha.nfsd
tcp6       0      0 192.168.0.119:4501      :::*                    LISTEN      12409/ganesha.nfsd
tcp6       0      0 192.168.0.119:2049      :::*                    LISTEN      12409/ganesha.nfsd
tcp6       0      0 192.168.0.119:38468     :::*                    LISTEN      12409/ganesha.nfsd
tcp        0      0 192.168.0.80:2049       0.0.0.0:*               LISTEN      17894/haproxy
[root@nfs02 n]#
[root@nfs02 n]#
[root@nfs02 n]#
[root@nfs02 n]# ssh nfs-c01
Password:
Last login: Wed Feb 21 00:37:28 2018 from nfs-c01.nix.mine.dom
[root@nfs02 ~]# logout
Connection to nfs-c01 closed.
[root@nfs02 n]#

From client we can still see all the files (seemless with no interruption to the NFS service).  As a bonus, while we started this first test, we noticed that HAPROXY was offline on nfs02.  While trying to list the client files, it appeared hung but still responded then listed files right after we started HAPROXY on nfs02

[root@ipaclient01 n]# ls -altri some-people-find-this-awesome.txt
11782527620043058273 -rw-r–r–. 1 nobody nobody 74 Feb 21 00:26 some-people-find-this-awesome.txt
[root@ipaclient01 n]# df -h .
Filesystem      Size  Used Avail Use% Mounted on
nfs-c01:/n      128G   43M  128G   1% /n
[root@ipaclient01 n]# ssh nfs-c01
Password:
Last login: Wed Feb 21 00:41:06 2018 from nfs-c01.nix.mine.dom
[root@nfs02 ~]#

Checking the gluster volume on nfs02:

[root@nfs02 n]# gluster volume status
Status of volume: gv01
Gluster process                             TCP Port  RDMA Port  Online  Pid
——————————————————————————
Brick nfs02:/bricks/0/gv01                  49152     0          Y       16103
Self-heal Daemon on localhost               N/A       N/A        Y       16094

Task Status of Volume gv01
——————————————————————————
There are no active volume tasks

[root@nfs02 n]#

Now let's bring back the first node and fail the second after nfs01 is up again.  As soon as we bring nfs01 back up, the VIP fails over to nfs01 without any hickup or manual invervention on the client end:

[root@ipaclient01 n]# ls -altri
total 11
                 128 dr-xr-xr-x. 21 root   root   4096 Feb 18 22:24 ..
11782527620043058273 -rw-r–r–.  1 nobody nobody   74 Feb 21 00:26 some-people-find-this-awesome.txt
                   1 drwxr-xr-x.  3 nobody nobody 4096 Feb 21 00:26 .
[root@ipaclient01 n]#
[root@ipaclient01 n]#
[root@ipaclient01 n]#
[root@ipaclient01 n]# ssh nfs-c01
Password:
Last login: Wed Feb 21 00:59:56 2018
[root@nfs01 ~]#

So now let's fail the second node.  NFS still works:

[root@ipaclient01 ~]# ssh nfs-c01
Password:
Last login: Wed Feb 21 01:31:50 2018
[root@nfs01 ~]# logout
Connection to nfs-c01 closed.
[root@ipaclient01 ~]# cd /n
[root@ipaclient01 n]# ls -altri some-people-find-this-awesome.txt
11782527620043058273 -rw-r–r–. 1 nobody nobody 74 Feb 21 00:26 some-people-find-this-awesome.txt
[root@ipaclient01 n]# df -h .
Filesystem      Size  Used Avail Use% Mounted on
nfs-c01:/n      128G   43M  128G   1% /n
[root@ipaclient01 n]#

So we bring the second node back up.  And that concludes the configuration!  All works like a charm!

Good Luck!

Cheers,
Tom K.

 

 

Leave a Reply

You must be logged in to post a comment.


     
  Copyright © 2003 - 2013 Tom Kacperski (microdevsys.com). All rights reserved.

Creative Commons License
This work is licensed under a Creative Commons Attribution 3.0 Unported License