Header Shadow Image

Passless keys ssh does not work when home folder is an NFS mount.

On this CentOS 7 (both worker and controller), we cannot ssh using pass less keys from the controller opennebula01 to the worker mdskvm-p01 when the NFS share ( /var/lib/one) is mounted on the worker. But I can as soon as I unmount the opennebula01 NFS share off of the worker node mdskvm-p01. When the NFS is mounted, both worker and controller share a common /var/lib/one/.ssh/authorized_keys file, which seems to be the intent of the setup:

[oneadmin@mdskvm-p01 .ssh]$ mount|tail -n 1 on /var/lib/one type nfs4 (rw,relatime,vers=4.0,rsize=8192,wsize=8192,namlen=255,soft,proto=tcp,port=0,timeo=600,retrans=2,sec=sys,clientaddr=,local_lock=none,addr=
[oneadmin@mdskvm-p01 .ssh]$ pwd
[oneadmin@mdskvm-p01 .ssh]$

Now when I run SSHD in debug mode using port 2222, the passless key works fine with or without NFS mounted on the worker node. Why?

Is there a specific sshd config file entry that prevents passless key login if sshd is not running in debug mode or when the OpenNebula NFS share is mounted? Again, in debug, SSHD works fine with or without the NFS mount on the worker mdskvm-p01.  To answer these questions we will run through some checks:

First ensure that the known_hosts file and authorized_keys file is set to 600. Second, we can run ssh in debug mode, both on when we ssh to the target and on the target we can set the ssh daemon into debug mode like this:

(server) /usr/sbin/sshd -p 2222 -D -dddd -e 
(client) ssh mdskvm-p01 -p 2222

This can point to an easily identifyable issue that is easy to fix, and usualy does if it's something configuration related.  For further digging we can also run tail -f /var/log/secure to see additional details.  However, in case this doesn't help, the other area to look at is the SELinux policy.  One thing we did NOT do is disable SELinux on the OpenNebula nodes like the OpenNebula Document pages suggest.  However we will try to work around this issue by working with SELinux rather then disabling it.  This is a safer approach however you may find yourself in a position to tweak SELinux rules for every small action OpenNebula takes.  Sure enough, checking the /var/log/audit/audit.log file yields these messages:

type=AVC msg=audit(1459215027.430:35013): avc:  denied  { read } for  pid=4345 comm="sshd" name="authorized_keys" dev="0:41" ino=68066706 scontext=system_u:system_r:sshd_t:s0-s0:c0.c1023 tcontext=system_u:object_r:nfs_t:s0 tclass=file
type=SYSCALL msg=audit(1459215027.430:35013): arch=c000003e syscall=2 success=no exit=-13 a0=7f9946d498a0 a1=800 a2=1 a3=7f9941c912e0 items=0 ppid=24665 pid=4345 auid=4294967295 uid=0 gid=0 euid=9869 suid=0 fsuid=9869 egid=9869 sgid=0 fsgid=9869 tty=(none) ses=4294967295 comm="sshd" exe="/usr/sbin/sshd" subj=system_u:system_r:sshd_t:s0-s0:c0.c1023 key=(null)
type=USER_AUTH msg=audit(1459215027.433:35014): pid=4345 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:sshd_t:s0-s0:c0.c1023 msg='op=pubkey acct="oneadmin" exe="/usr/sbin/sshd" hostname=? addr= terminal=ssh res=failed'

What we need is the item in green above.  The reason why this happens is because we have NFS mounted /var/lib/one folder meaning NFS SELinux policies need to be adjusted.  We can also use aureport -a and ausearch -a <NUM> or audit2why or audit2allow etc from first command, to get similar details.  This doesn't happen when NFS home folders are not mounted.  Now borrowing slightly off of our earlier post on a similar NFS issue, we check the permissions on the folder and set the context accordingly:

[root@mdskvm-p01 ~]#  ls -lda –author -Z /var/lib/one
drwxr-x—. oneadmin oneadmin system_u:object_r:nfs_t:s0       /var/lib/one
[root@mdskvm-p01 ~]#

Now to install audit2why and friends, type this:

yum install policycoreutils-python

And this gets you:

[root@mdskvm-p01 ~]# rpm -ql policycoreutils-python|grep -i bin
[root@mdskvm-p01 ~]#

Now executing the audit2why yeilds the answer we were looking for:

grep ssh /var/log/audit/audit.log | audit2why -w
type=AVC msg=audit(1459235024.202:33774): avc:  denied  { read open } for  pid=26163 comm="sshd" path="/var/lib/one/.ssh/authorized_keys" dev="0:41" ino=68066706 scontext=system_u:system_r:sshd_t:s0-s0:c0.c1023 tcontext=system_u:object_r:nfs_t:s0 tclass=file

        Was caused by:
        The boolean use_nfs_home_dirs was set incorrectly.
        Allow use to nfs home dirs

        Allow access by executing:
        # setsebool -P use_nfs_home_dirs 1

So in this case, we will try the first option and mount the NFS mount differently, by adding in a context to it:

[root@mdskvm-p01 ~]# grep context /etc/fstab      /var/lib/one/  nfs   context=system_u:object_r:nfs_t:s0,soft,intr,rsize=8192,wsize=8192,noauto
[root@mdskvm-p01 ~]#

And sure enough, we get a mount:

[root@mdskvm-p01 ~]# mount /var/lib/one
[root@mdskvm-p01 ~]#

But that didn't work either so we run the recommended command above:

[root@mdskvm-p01 ~]# setsebool -P use_nfs_home_dirs 1
[root@mdskvm-p01 ~]#

And try to mount:

[oneadmin@opennebula01 .ssh]$ ssh mdskvm-p01
Warning: Permanently added 'mdskvm-p01,' (ECDSA) to the list of known hosts.
Last failed login: Tue Mar 29 03:03:45 EDT 2016 from opennebula01 on ssh:notty
There were 2 failed login attempts since the last successful login.
Last login: Tue Mar 29 02:54:54 2016 from opennebula01
[oneadmin@mdskvm-p01 ~]$

SUCCESS!!  Now if you add the wrong context, this can happen:

[root@mdskvm-p01 ~]# mount /var/lib/one
mount.nfs: access denied by server while mounting
[root@mdskvm-p01 ~]#
[root@mdskvm-p01 ~]#
[root@mdskvm-p01 ~]#
[root@mdskvm-p01 ~]# grep /etc/fstab      /var/lib/one/  nfs   context=system_u:system_r:sshd_t:s0,soft,intr,rsize=8192,wsize=8192,noauto
[root@mdskvm-p01 ~]#

However, we get this problem now:

type=AVC msg=audit(1459218411.696:37452): avc:  denied  { relabelto } for  pid=25792 comm="mount.nfs" scontext=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023 tcontext=system_u:system_r:sshd_t:s0 tclass=filesystem
type=SYSCALL msg=audit(1459218411.696:37452): arch=c000003e syscall=165 success=no exit=-13 a0=7f9b20242ff0 a1=7f9b20243e70 a2=7f9b20243e90 a3=0 items=0 ppid=25791 pid=25792 auid=0 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=pts2 ses=95 comm="mount.nfs" exe="/usr/sbin/mount.nfs" subj=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023 key=(null)

And we'll leave that for another article.  It was the first context that we needed.  wink


One Response to “Passless keys ssh does not work when home folder is an NFS mount.”

  1. […] see the following post for details on how to tailor SELinux rules for similar […]

Leave a Reply

You must be logged in to post a comment.

  Copyright © 2003 - 2013 Tom Kacperski (microdevsys.com). All rights reserved.

Creative Commons License
This work is licensed under a Creative Commons Attribution 3.0 Unported License