{"id":2833,"date":"2016-03-28T21:43:06","date_gmt":"2016-03-29T01:43:06","guid":{"rendered":"http:\/\/microdevsys.com\/wp\/?p=2833"},"modified":"2016-05-29T15:06:04","modified_gmt":"2016-05-29T19:06:04","slug":"passless-keys-ssh-does-not-work-when-home-folder-is-an-nfs-mount","status":"publish","type":"post","link":"https:\/\/microdevsys.com\/wp\/passless-keys-ssh-does-not-work-when-home-folder-is-an-nfs-mount\/","title":{"rendered":"Passless keys ssh does not work when home folder is an NFS mount."},"content":{"rendered":"

\n\tOn this CentOS 7 (both worker and controller), we cannot ssh using pass less keys from the controller opennebula01<\/strong> to the worker mdskvm-p01<\/strong> when the NFS share ( 192.168.0.70:\/var\/lib\/one \/var\/lib\/one<\/strong>) is mounted on the worker. But I can as soon as I unmount the opennebula01 <\/strong>NFS share off of the worker node mdskvm-p01<\/strong>. When the NFS<\/strong> is mounted, both worker and controller share a common \/var\/lib\/one\/.ssh\/authorized_keys<\/strong> file, which seems to be the intent of the setup:\n<\/p>\n

\n\t[oneadmin@mdskvm-p01 .ssh]$ mount|tail -n 1
\n\t192.168.0.70:\/var\/lib\/one on \/var\/lib\/one type nfs4 (rw,relatime,vers=4.0,rsize=8192,wsize=8192,namlen=255,soft,proto=tcp,port=0,timeo=600,retrans=2,sec=sys,clientaddr=192.168.0.60,local_lock=none,addr=192.168.0.70)
\n\t[oneadmin@mdskvm-p01 .ssh]$ pwd
\n\t\/var\/lib\/one\/.ssh
\n\t[oneadmin@mdskvm-p01 .ssh]$<\/span><\/span>\n<\/p>\n

\n\tNow when I run SSHD in debug mode using port 2222, the passless key works fine with or without NFS mounted on the worker node. Why?\n<\/p>\n

\n\tIs there a specific sshd config file entry that prevents passless key login if sshd is not running in debug mode or when the OpenNebula NFS share is mounted? Again, in debug, SSHD works fine with or without the NFS mount on the worker mdskvm-p01.  To answer these questions we will run through some checks:\n<\/p>\n

\n\t\n<\/p>\n

\n\tFirst ensure that the known_hosts<\/strong> file and authorized_keys <\/strong>file is set to 600. Second, we can run ssh in debug mode, both on when we ssh to the target and on the target we can set the ssh daemon into debug mode like this:\n<\/p>\n

\n\t(server) \/usr\/sbin\/sshd -p 2222 -D -dddd -e 
\n\t(client) ssh mdskvm-p01 -p 2222<\/span><\/span>\n<\/p>\n

\n\tThis can point to an easily identifyable issue that is easy to fix, and usualy does if it's something configuration related.  For further digging we can also run tail -f \/var\/log\/secure to see additional details.  However, in case this doesn't help, the other area to look at is the SELinux policy.  One thing we did NOT do is disable SELinux on the OpenNebula nodes like the OpenNebula Document pages suggest.  However we will try to work around this issue by working with SELinux rather then disabling it.  This is a safer approach however you may find yourself in a position to tweak SELinux rules for every small action OpenNebula takes.  Sure enough, checking the \/var\/log\/audit\/audit.log file yields these messages:\n<\/p>\n

\n\ttype=AVC msg=audit(1459215027.430:35013): avc:  denied  { read } for  pid=4345 comm="sshd" name="authorized_keys" dev="0:41" ino=68066706 scontext=system_u:system_r:sshd_t:s0-s0:c0.c1023 tcontext=system_u:object_r:nfs_t:s0<\/span><\/strong> tclass=file
\n\ttype=SYSCALL msg=audit(1459215027.430:35013): arch=c000003e syscall=2 success=no exit=-13 a0=7f9946d498a0 a1=800 a2=1 a3=7f9941c912e0 items=0 ppid=24665 pid=4345 auid=4294967295 uid=0 gid=0 euid=9869 suid=0 fsuid=9869 egid=9869 sgid=0 fsgid=9869 tty=(none) ses=4294967295 comm="sshd" exe="\/usr\/sbin\/sshd" subj=system_u:system_r:sshd_t:s0-s0:c0.c1023 key=(null)
\n\ttype=USER_AUTH msg=audit(1459215027.433:35014): pid=4345 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:sshd_t:s0-s0:c0.c1023 msg='op=pubkey acct="oneadmin" exe="\/usr\/sbin\/sshd" hostname=? addr=192.168.0.70 terminal=ssh res=failed'<\/span><\/span>\n<\/p>\n

\n\tWhat we need is the item in green above.  The reason why this happens is because we have NFS mounted \/var\/lib\/one<\/strong> folder meaning NFS SELinux<\/strong> policies need to be adjusted.  We can also use aureport -a<\/strong> and ausearch -a <NUM><\/strong> or audit2why<\/strong> or audit2allow <\/strong>etc from first command, to get similar details.  This doesn't happen when NFS home folders are not mounted.  Now borrowing slightly off of our earlier post on a similar NFS<\/a> issue, we check the permissions on the folder and set the context accordingly:\n<\/p>\n

\n\t[root@mdskvm-p01 ~]#  ls -lda –author -Z \/var\/lib\/one
\n\tdrwxr-x—. oneadmin oneadmin system_u:object_r:nfs_t:s0       \/var\/lib\/one
\n\t[root@mdskvm-p01 ~]#<\/span><\/span>\n<\/p>\n

\n\tNow to install audit2why<\/strong> and friends, type this:\n<\/p>\n

\n\tyum install policycoreutils-python<\/span><\/span>\n<\/p>\n

\n\tAnd this gets you:\n<\/p>\n

\n\t[root@mdskvm-p01 ~]# rpm -ql policycoreutils-python|grep -i bin
\n\t\/usr\/bin\/audit2allow
\n\t\/usr\/bin\/audit2why
\n\t\/usr\/bin\/chcat
\n\t\/usr\/bin\/sandbox
\n\t\/usr\/bin\/semodule_package
\n\t\/usr\/sbin\/semanage
\n\t[root@mdskvm-p01 ~]#<\/span>\n<\/p>\n

\n\tNow executing the audit2why yeilds the answer we were looking for:\n<\/p>\n

\n\tgrep ssh \/var\/log\/audit\/audit.log | audit2why -w<\/strong>
\n\ttype=AVC msg=audit(1459235024.202:33774): avc:  denied  { read open } for  pid=26163 comm="sshd" path="\/var\/lib\/one\/.ssh\/authorized_keys" dev="0:41" ino=68066706 scontext=system_u:system_r:sshd_t:s0-s0:c0.c1023 tcontext=system_u:object_r:nfs_t:s0 tclass=file<\/span><\/span><\/span>\n<\/p>\n

\n\t        Was caused by:
\n\t        The boolean use_nfs_home_dirs was set incorrectly.
\n\t        Description:
\n\t        Allow use to nfs home dirs<\/span><\/span>\n<\/p>\n

\n\t        Allow access by executing:
\n\t        # setsebool -P use_nfs_home_dirs 1<\/span><\/span>\n<\/p>\n

\n\tSo in this case, we will try the first option and mount the NFS mount differently, by adding in a context to it:\n<\/p>\n

\n\t[root@mdskvm-p01 ~]# grep context \/etc\/fstab<\/strong>
\n\t192.168.0.70:\/var\/lib\/one\/      \/var\/lib\/one\/  nfs   context=system_u:object_r:nfs_t:s0,soft,intr,rsize=8192,wsize=8192,noauto
\n\t[root@mdskvm-p01 ~]#<\/span>\n<\/p>\n

\n\tAnd sure enough, we get a mount:<\/span>\n<\/p>\n

\n\t[root@mdskvm-p01 ~]# mount \/var\/lib\/one
\n\t[root@mdskvm-p01 ~]#<\/span>\n<\/p>\n

\n\tBut that didn't work either so we run the recommended command above:\n<\/p>\n

\n\t[root@mdskvm-p01 ~]# setsebool -P use_nfs_home_dirs 1<\/strong>
\n\t[root@mdskvm-p01 ~]#<\/span>\n<\/p>\n

\n\tAnd try to mount:\n<\/p>\n

\n\t[oneadmin@opennebula01 .ssh]$ ssh mdskvm-p01<\/strong>
\n\tWarning: Permanently added 'mdskvm-p01,192.168.0.60' (ECDSA) to the list of known hosts.
\n\tLast failed login: Tue Mar 29 03:03:45 EDT 2016 from opennebula01 on ssh:notty
\n\tThere were 2 failed login attempts since the last successful login.
\n\tLast login: Tue Mar 29 02:54:54 2016 from opennebula01
\n\t[oneadmin@mdskvm-p01 ~]$<\/span>\n<\/p>\n

\n\tSUCCESS!! <\/strong> Now if you add the wrong context, this can happen:\n<\/p>\n

\n\t[root@mdskvm-p01 ~]# mount \/var\/lib\/one
\n\tmount.nfs: access denied by server while mounting 192.168.0.70:\/var\/lib\/one\/
\n\t[root@mdskvm-p01 ~]#
\n\t[root@mdskvm-p01 ~]#
\n\t[root@mdskvm-p01 ~]#
\n\t[root@mdskvm-p01 ~]# grep 192.168.0.70 \/etc\/fstab<\/strong>
\n\t192.168.0.70:\/var\/lib\/one\/      \/var\/lib\/one\/  nfs   context=system_u:system_r:sshd_t:s0,soft,intr,rsize=8192,wsize=8192,noauto
\n\t[root@mdskvm-p01 ~]#<\/span><\/span>\n<\/p>\n

\n\tHowever, we get this problem now:\n<\/p>\n

\n\ttype=AVC msg=audit(1459218411.696:37452): avc:  denied  { relabelto } for  pid=25792 comm="mount.nfs" scontext=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023 tcontext=system_u:system_r:sshd_t:s0 tclass=filesystem
\n\ttype=SYSCALL msg=audit(1459218411.696:37452): arch=c000003e syscall=165 success=no exit=-13 a0=7f9b20242ff0 a1=7f9b20243e70 a2=7f9b20243e90 a3=0 items=0 ppid=25791 pid=25792 auid=0 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=pts2 ses=95 comm="mount.nfs" exe="\/usr\/sbin\/mount.nfs" subj=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023 key=(null)<\/span><\/span>\n<\/p>\n

\n\tAnd we'll leave that for another article.  It was the first context that we needed.  \"wink\"\n<\/p>\n

\n\tCheers,
\n\tTK<\/p>\n\n

\n\n\t\t\n
    \n\t\t\t <\/ul>\n <\/div> \n","protected":false},"excerpt":{"rendered":"

    On this CentOS 7 (both worker and controller), we cannot ssh using pass less keys from the controller opennebula01 to the worker mdskvm-p01 when the NFS share ( 192.168.0.70:\/var\/lib\/one \/var\/lib\/one) is mounted on the worker. But I can as soon as I unmount the opennebula01 NFS share off of the worker node mdskvm-p01. When the NFS […]<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_monsterinsights_skip_tracking":false,"_monsterinsights_sitenote_active":false,"_monsterinsights_sitenote_note":"","_monsterinsights_sitenote_category":0,"_uf_show_specific_survey":0,"_uf_disable_surveys":false,"footnotes":""},"categories":[3],"tags":[],"class_list":["post-2833","post","type-post","status-publish","format-standard","hentry","category-unix-linux-admin-stuff"],"aioseo_notices":[],"_links":{"self":[{"href":"https:\/\/microdevsys.com\/wp\/wp-json\/wp\/v2\/posts\/2833","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/microdevsys.com\/wp\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/microdevsys.com\/wp\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/microdevsys.com\/wp\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/microdevsys.com\/wp\/wp-json\/wp\/v2\/comments?post=2833"}],"version-history":[{"count":5,"href":"https:\/\/microdevsys.com\/wp\/wp-json\/wp\/v2\/posts\/2833\/revisions"}],"predecessor-version":[{"id":3116,"href":"https:\/\/microdevsys.com\/wp\/wp-json\/wp\/v2\/posts\/2833\/revisions\/3116"}],"wp:attachment":[{"href":"https:\/\/microdevsys.com\/wp\/wp-json\/wp\/v2\/media?parent=2833"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/microdevsys.com\/wp\/wp-json\/wp\/v2\/categories?post=2833"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/microdevsys.com\/wp\/wp-json\/wp\/v2\/tags?post=2833"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}