Header Shadow Image


error: can’t canonicalize path /var/lib/one/datastores/ Permission denied

Appears the error can't canonicalize path /var/lib/one/datastores/ Permission denied has to do with passwords not being in sync with what they were before.  Changing oneadmin password.  Password is hashed with sha1 in the database and the database is /var/lib/one/one.db.  Passwords in the ~/.one/one_auth and other *_auth as well as one_key files is plaintext:

  • $ one start
  • $ oneuser passwd –sha1 oneadmin 'newpassword'
  • $ echo -n 'newpassword' | openssl sha1
  • Open the opennebula database file with sqlite3:

    [oneadmin@opennebula01 .one]$ sqlite3 /var/lib/one/one.db
    SQLite version 3.7.17 2013-05-20 00:56:22
    Enter ".help" for instructions
    Enter SQL statements terminated with a ";"
    sqlite> select body from user_pool where name='oneadmin';
    sqlite>

    XML text will appear above with the password listed. Update it with the new password:

  • sqlite> update user_pool set body='<ENTIRE XML FROM ABOVE + YOUR EDITS SUCH AS SHA1 OF YOUR PASS>' where name='oneadmin';

  • sqlite> .quit
     

  • $ echo "oneadmin:newpassword" > ~/.one/one_auth                # Pass to use here is the generated sha1 hash.
  • $ one start

And you are done.  Coincidently this also resolved some of the below error messages as well:

VirtualMachinePoolInfo result FAILURE [VirtualMachinePoolInfo] User couldn't be authenticated, aborting call.

Wed Mar 30 02:32:11 2016 [Z0][TM][D]: Message received: TRANSFER SUCCESS 11 –
Wed Mar 30 02:32:11 2016 [Z0][VMM][D]: Message received: LOG I 11 ExitCode: 0
Wed Mar 30 02:32:11 2016 [Z0][VMM][D]: Message received: LOG I 11 Successfully execute network driver operation: pre.
Wed Mar 30 02:32:11 2016 [Z0][VMM][D]: Message received: LOG I 11 Command execution fail: cat << EOT | /var/tmp/one/vmm/kvm/deploy '/var/lib/one//datastores/0/11/deployment.0' 'mdskvm-p01' 11 mdskvm-p01
Wed Mar 30 02:32:11 2016 [Z0][VMM][D]: Message received: LOG I 11 error: Failed to create domain from /var/lib/one//datastores/0/11/deployment.0
Wed Mar 30 02:32:11 2016 [Z0][VMM][D]: Message received: LOG I 11 error: can't canonicalize path '/var/lib/one//datastores/0/11/disk.1': Permission denied
Wed Mar 30 02:32:11 2016 [Z0][VMM][D]: Message received: LOG E 11 Could not create domain from /var/lib/one//datastores/0/11/deployment.0
Wed Mar 30 02:32:11 2016 [Z0][VMM][D]: Message received: LOG I 11 ExitCode: 255
Wed Mar 30 02:32:11 2016 [Z0][VMM][D]: Message received: LOG I 11 Failed to execute virtualization driver operation: deploy.
Wed Mar 30 02:32:11 2016 [Z0][VMM][D]: Message received: DEPLOY FAILURE 11 Could not create domain from /var/lib/one//datastores/0/11/deployment.0

Another source of the above problem is when the OpenNebula VM instantiation tries to create a VM within a folder that's NFS mounted off of the controller (opennebula01 in this case):

[root@mdskvm-p01 audit]# ls -altri /var/tmp/one/vmm/kvm/deploy
669864 -rwxr-xr-x. 1 oneadmin oneadmin 1545 Apr  7  2015 /var/tmp/one/vmm/kvm/deploy
[root@mdskvm-p01 audit]#

[root@mdskvm-p01 audit]# /var/tmp/one/vmm/kvm/deploy '/var/lib/one//datastores/0/33/deployment.0' 'mdskvm-p01' 33 mdskvm-p01
mkdir: cannot create directory '/var/lib/one': Permission denied
/var/tmp/one/vmm/kvm/deploy: line 25: /var/lib/one//datastores/0/33/deployment.0: Permission denied
error: Failed to open file '/var/lib/one//datastores/0/33/deployment.0': Permission denied
ERROR MESSAGE –8<——
Could not create domain from /var/lib/one//datastores/0/33/deployment.0
ERROR MESSAGE ——>8–
[root@mdskvm-p01 audit]#

And running as the oneadmin user, the process get's stuck:

[oneadmin@mdskvm-p01 ~]$ /var/tmp/one/vmm/kvm/deploy '/var/lib/one//datastores/0/33/deployment.0' 'mdskvm-p01' 33 mdskvm-p01

Could it be that the deployment on the NODE for libvirt / kvm was done with root previlidges preventing VM deployment in OpenNebula?  After all, the oneadmin owned NFS off of the controller cannot be accessed from the node by the root user: permission denied.  This is expected.  Let's investigate.  We notice that the following executable is invoked above, /var/tmp/one/vmm/kvm/deploy, so let's modify that script slightly for additional debug information:

[oneadmin@mdskvm-p01 datastores]$ head /var/tmp/one/vmm/kvm/deploy -n 2
#!/bin/bash -x

[oneadmin@mdskvm-p01 datastores]$

And then let's issue another build request using onetemplate instantiate "CentOS-7" to get these more detailed messages:

Sun Apr  3 22:08:09 2016 [Z0][VMM][D]: Message received: LOG I 34 ++ SED='sed -r'
Sun Apr  3 22:08:09 2016 [Z0][VMM][D]: Message received: LOG I 34 +++ basename /var/tmp/one/vmm/kvm/deploy
Sun Apr  3 22:08:09 2016 [Z0][VMM][D]: Message received: LOG I 34 ++ SCRIPT_NAME=deploy
Sun Apr  3 22:08:09 2016 [Z0][VMM][D]: Message received: LOG I 34 + domain=/var/lib/one//datastores/0/34/deployment.0
Sun Apr  3 22:08:09 2016 [Z0][VMM][D]: Message received: LOG I 34 ++ dirname /var/lib/one//datastores/0/34/deployment.0
Sun Apr  3 22:08:09 2016 [Z0][VMM][D]: Message received: LOG I 34 + mkdir -p /var/lib/one//datastores/0/34
Sun Apr  3 22:08:09 2016 [Z0][VMM][D]: Message received: LOG I 34 + cat
Sun Apr  3 22:08:09 2016 [Z0][VMM][D]: Message received: LOG I 34 ++ virsh –connect qemu:///system create /var/lib/one//datastores/0/34/deployment.0
Sun Apr  3 22:08:09 2016 [Z0][VMM][D]: Message received: LOG I 34 error: Failed to create domain from /var/lib/one//datastores/0/34/deployment.0

Sun Apr  3 22:08:09 2016 [Z0][VMM][D]: Message received: LOG I 34 error: can't canonicalize path '/var/lib/one//datastores/0/34/disk.1': Permission denied
Sun Apr  3 22:08:09 2016 [Z0][VMM][D]: Message received: LOG I 34 + data=
Sun Apr  3 22:08:09 2016 [Z0][VMM][D]: Message received: LOG I 34 + ‘[‘ x1 = x0 ‘]'
Sun Apr  3 22:08:09 2016 [Z0][VMM][D]: Message received: LOG I 34 + error_message 'Could not create domain from /var/lib/one//datastores/0/34/deployment.0'
Sun Apr  3 22:08:09 2016 [Z0][VMM][D]: Message received: LOG I 34 + echo 'ERROR MESSAGE –8<——'
Sun Apr  3 22:08:09 2016 [Z0][VMM][D]: Message received: LOG E 34 + echo 'Could not create domain from /var/lib/one//datastores/0/34/deployment.0'
Sun Apr  3 22:08:09 2016 [Z0][VMM][D]: Message received: LOG E 34 Could not create domain from /var/lib/one//datastores/0/34/deployment.0
Sun Apr  3 22:08:09 2016 [Z0][VMM][D]: Message received: LOG E 34 + echo 'ERROR MESSAGE ——>8–'
Sun Apr  3 22:08:09 2016 [Z0][VMM][D]: Message received: LOG I 34 + exit -1
Sun Apr  3 22:08:09 2016 [Z0][VMM][D]: Message received: LOG I 34 ExitCode: 255
Sun Apr  3 22:08:09 2016 [Z0][VMM][D]: Message received: LOG I 34 Failed to execute virtualization driver operation: deploy.
Sun Apr  3 22:08:09 2016 [Z0][VMM][D]: Message received: DEPLOY FAILURE 34 + echo 'Could not create domain from /var/lib/one//datastores/0/34/deployment.0'
Sun Apr  3 22:08:09 2016 [Z0][VMM][D]: Message received: Could not create domain from /var/lib/one//datastores/0/34/deployment.0
?
Sun Apr  3 22:08:09 2016 [Z0][VMM][D]: Message received: + echo 'ERROR MESSAGE ——>8–'

As we can see the command in red is being ran.  So let's run it as the oneadmin user off of the node mdskvm-p01:

[oneadmin@mdskvm-p01 ~]$ virsh –connect qemu:///system create /var/lib/one//datastores/0/34/deployment.0
error: Failed to create domain from /var/lib/one//datastores/0/34/deployment.0
error: can't canonicalize path '/var/lib/one//datastores/0/34/disk.1': Permission denied

[oneadmin@mdskvm-p01 ~]$

And we get the same error.  So we isolated it to virsh and it seems something the way virsh is running is triggering that message.  Let's check the file type:

[oneadmin@mdskvm-p01 ~]$ file /usr/bin/virsh
/usr/bin/virsh: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked (uses shared libs), for GNU/Linux 2.6.32, BuildID[sha1]=78ccd8aec639cc2ae448b2ea31140ac6fa185f1f, stripped
[oneadmin@mdskvm-p01 ~]$

Ok it's a binary.  Let's run the above entire virsh line through a trace to find out more details:

write(2, "error: ", 7error: )                  = 7
write(2, "Failed to create domain from /va"…, 72Failed to create domain from /var/lib/one//datastores/0/34/deployment.0
) = 72

but it didn't give us much more details.  mount has a canonicalize options:

[oneadmin@mdskvm-p01 ~]$ mount –help|grep -i canoni
 -c, –no-canonicalize   don't canonicalize paths
[oneadmin@mdskvm-p01 ~]$

So let's check that out and how it affects NFS mounts. (VM creation works without an NFS mount, why not with the mount?)  Trying the option using:

mount –no-canonicalize /var/lib/one

Produces no difference.  Let's compare the mount point permissions before and after mounting from our controller OpenNebula01, possibly it has something to do with that:

[root@mdskvm-p01 ~]# ls -altrid /var/lib/one
1405 drwxr-x— 12 oneadmin oneadmin 4096 Apr  3 23:09 /var/lib/one
[root@mdskvm-p01 ~]# ls -altrid /var/lib/one/datastore
ls: cannot access /var/lib/one/datastore: Permission denied
[root@mdskvm-p01 ~]# mount|tail -n 1
192.168.0.70:/var/lib/one on /var/lib/one type nfs4 (rw,relatime,vers=4.0,rsize=8192,wsize=8192,namlen=255,soft,proto=tcp,port=0,timeo=600,retrans=2,sec=sys,clientaddr=192.168.0.60,local_lock=none,addr=192.168.0.70)
[root@mdskvm-p01 ~]#
[root@mdskvm-p01 ~]#
[root@mdskvm-p01 ~]# umount /var/lib/one
[root@mdskvm-p01 ~]# ls -altrid /var/lib/one
68718089 drwxr-xr-x. 2 oneadmin oneadmin 6 Apr  3 21:20 /var/lib/one
[root@mdskvm-p01 ~]#

And our controller export:

[root@opennebula01 one]# ls -altrid /var/lib/one/
1405 drwxr-x—. 12 oneadmin oneadmin 4096 Apr  4 01:15 /var/lib/one/
[root@opennebula01 one]#

Then we check libvirtd to see for any potential issues and sure enough:

[root@mdskvm-p01 ~]# systemctl status libvirtd
â libvirtd.service – Virtualization daemon
   Loaded: loaded (/usr/lib/systemd/system/libvirtd.service; enabled; vendor preset: enabled)
   Active: active (running) since Sun 2016-04-03 23:28:18 EDT; 1h 45min ago
     Docs: man:libvirtd(8)
           http://libvirt.org
 Main PID: 1461 (libvirtd)
   CGroup: /system.slice/libvirtd.service
           ââ1461 /usr/sbin/libvirtd
           ââ2422 /sbin/dnsmasq –conf-file=/var/lib/libvirt/dnsmasq/default.conf –leasefile-ro –dhcp-script=/usr/libexec/libvirt_leaseshe…
           ââ2424 /sbin/dnsmasq –conf-file=/var/lib/libvirt/dnsmasq/default.conf –leasefile-ro –dhcp-script=/usr/libexec/libvirt_leaseshe…

Apr 04 00:09:29 mdskvm-p01 libvirtd[1461]: can't canonicalize path '/var/lib/one//datastores/0/34/disk.1': Permission denied
Apr 04 00:09:51 mdskvm-p01 libvirtd[1461]: can't canonicalize path '/var/lib/one//datastores/0/34/disk.1': Permission denied
Apr 04 00:21:57 mdskvm-p01 libvirtd[1461]: can't canonicalize path '/var/lib/one//datastores/0/34/disk.1': Permission denied
Apr 04 00:22:31 mdskvm-p01 libvirtd[1461]: can't canonicalize path '/var/lib/one//datastores/0/34/disk.1': Permission denied
Apr 04 00:22:33 mdskvm-p01 libvirtd[1461]: can't canonicalize path '/var/lib/one//datastores/0/34/disk.1': Permission denied
Apr 04 00:25:20 mdskvm-p01 libvirtd[1461]: can't canonicalize path '/var/lib/one//datastores/0/34/disk.1': Permission denied
Apr 04 00:29:30 mdskvm-p01 libvirtd[1461]: can't canonicalize path '/var/lib/one//datastores/0/34/disk.1': Permission denied
Apr 04 00:43:55 mdskvm-p01 libvirtd[1461]: can't canonicalize path '/var/lib/one//datastores/0/34/disk.1': Permission denied
Apr 04 00:45:01 mdskvm-p01 libvirtd[1461]: can't canonicalize path '/var/lib/one//datastores/0/35/disk.1': Permission denied
Apr 04 00:47:31 mdskvm-p01 libvirtd[1461]: can't canonicalize path '/var/lib/one//datastores/0/36/disk.1': Permission denied

[root@mdskvm-p01 ~]#

So it appears if we have libvird running as root, could be the issue.  As root it will not be able to access the opennebula01:/var/lib/one NFS mount from within mdskvm-p01 node.  That makes sense.  So have to try and configure libvirtd to run as oneadmin instead.  Appears we may need to add the oneadmin user to the libvirtd group so let's try that:

[root@mdskvm-p01 ~]# grep -i libvir /etc/group /etc/passwd
/etc/group:libvirt:x:992:
[root@mdskvm-p01 ~]#
[root@mdskvm-p01 ~]#
[root@mdskvm-p01 ~]# grep libvirt /etc/group /etc/passwd
/etc/group:libvirt:x:992:
[root@mdskvm-p01 ~]# grep -i 992 /etc/group /etc/passwd
/etc/group:libvirt:x:992:
[root@mdskvm-p01 ~]# id oneadmin
uid=9869(oneadmin) gid=9869(oneadmin) groups=9869(oneadmin)
[root@mdskvm-p01 ~]#
[root@mdskvm-p01 ~]#
[root@mdskvm-p01 ~]# usermod -a -G libvirt oneadmin
[root@mdskvm-p01 ~]# id oneadmin
uid=9869(oneadmin) gid=9869(oneadmin) groups=9869(oneadmin),992(libvirt)
[root@mdskvm-p01 ~]#

And the result is the same.  Noticed also that the SELinux ACL's are set (The dot after the file permissions like drwxr-x—.  ):

[root@mdskvm-p01 ~]# getenforce
Disabled
[root@mdskvm-p01 ~]#

[root@mdskvm-p01 ~]# ls -altrid /var/lib/one
68718089 drwxr-x—. 5 oneadmin oneadmin 70 Apr  5 21:37 /var/lib/one
[root@mdskvm-p01 ~]# mount /var/lib/one
[root@mdskvm-p01 ~]# ls -altrid /var/lib/one
1405 drwxr-x— 12 oneadmin oneadmin 4096 Apr  6 20:20 /var/lib/one
[root@mdskvm-p01 ~]# mount | tail -n 1
tmpfs on /run/user/9869 type tmpfs (rw,nosuid,nodev,relatime,size=7405336k,mode=700,uid=9869,gid=9869)
[root@mdskvm-p01 ~]# mount | grep "\.70"
192.168.0.70:/var/lib/one on /var/lib/one type nfs4 (rw,relatime,vers=4.0,rsize=8192,wsize=8192,namlen=255,soft,proto=tcp,port=0,timeo=600,retrans=2,sec=sys,clientaddr=192.168.0.60,local_lock=none,addr=192.168.0.70)
[root@mdskvm-p01 ~]#

On the controller:

[root@opennebula01 ~]# getenforce
Disabled
[root@opennebula01 ~]#

So I notice above that the folder has SELinux ACL's on it due to adding in context rules:

# 192.168.0.70:/var/lib/one/    /var/lib/one/  nfs   context=system_u:object_r:nfs_t:s0,soft,intr,rsize=8192,wsize=8192,noauto
192.168.0.70:/var/lib/one/      /var/lib/one/  nfs   soft,intr,rsize=8192,wsize=8192,noauto

so I remove it from the /etc/fstab mount line and retry the virsh command:

[oneadmin@mdskvm-p01 ~]$ virsh -d 1 –connect qemu:///system create /var/lib/one//datastores/0/38/deployment.0
create: file(optdata): /var/lib/one//datastores/0/38/deployment.0
error: Failed to create domain from /var/lib/one//datastores/0/38/deployment.0
error: can't canonicalize path '/var/lib/one//datastores/0/38/disk.1': Permission denied

Same thing.  So even though SELinux is disabled and with or without the context modifier in /etc/fstab, same issue happens: it exits on "can't canonicalize path".  A suggestion was given by the developers to add oneadmin to a few other groups, including the sudo group but the sudo group didn't exist on RHEL 7 clones:

[oneadmin@mdskvm-p01 ~]$ id oneadmin
uid=9869(oneadmin) gid=9869(oneadmin) groups=9869(oneadmin),992(libvirt),36(kvm)
[oneadmin@mdskvm-p01 ~]$

So we'll also set some debug options in alongside the above to get more details:

[root@mdskvm-p01 qemu]# grep -v "#" /etc/sysconfig/libvirtd
LIBVIRTD_ARGS="–listen –config /etc/libvirt/libvirtd.conf"
LIBVIRT_DEBUG=1
[root@mdskvm-p01 qemu]#

and in /etc/libvirt/libvirtd.conf add the log level variable:

[root@mdskvm-p01 qemu]# grep -v "#" /etc/libvirt/libvirtd.conf|sed '/^\s*$/d'
log_level = 1
unix_sock_group = "oneadmin"
unix_sock_ro_perms = "0777"
unix_sock_rw_perms = "0770"
auth_unix_ro = "none"
auth_unix_rw = "none"
listen_tls = 0
listen_tcp = 1
auth_tcp   = "none"
[root@mdskvm-p01 qemu]#

Then we will use 

journalctl -u libvirtd

To look at the debug output.  But that didn't show much more.  Now we run it via strace like this:

strace /usr/sbin/libvirtd –listen –config /etc/libvirt/libvirtd.conf

Then in second window:

virsh -d 1 –connect qemu:///system create /var/lib/one//datastores/0/38/deployment.0

to get a few more details:

2016-04-06 04:15:35.945+0000: 14072: debug : qemuProcessStart:4477 : Determining emulator version
2016-04-06 04:15:35.945+0000: 14072: debug : virQEMUCapsCacheLookup:3765 : Returning caps 0x7f6a6c25e1c0 for /usr/libexec/qemu-kvm
2016-04-06 04:15:35.945+0000: 14072: info : virObjectRef:296 : OBJECT_REF: obj=0x7f6a6c25e1c0
2016-04-06 04:15:35.945+0000: 14072: info : virObjectNew:202 : OBJECT_NEW: obj=0x7f6aa4002f50 classname=virQEMUCaps
2016-04-06 04:15:35.945+0000: 14072: info : virObjectUnref:259 : OBJECT_UNREF: obj=0x7f6a6c25e1c0
2016-04-06 04:15:35.945+0000: 14072: debug : qemuProcessStart:4488 : Preparing network devices
2016-04-06 04:15:35.945+0000: 14072: debug : qemuProcessStart:4493 : Preparing host devices
2016-04-06 04:15:35.945+0000: 14072: debug : qemuProcessStart:4502 : Preparing chr devices
2016-04-06 04:15:35.945+0000: 14072: debug : qemuProcessStart:4509 : Checking domain and device security labels
2016-04-06 04:15:35.945+0000: 14072: debug : qemuProcessStart:4515 : Generating domain security label (if required)
2016-04-06 04:15:35.945+0000: 14072: debug : qemuProcessStart:4542 : Ensuring no historical cgroup is lying around
2016-04-06 04:15:35.945+0000: 14072: debug : virFileMakePathHelper:2660 : path=/var/log/libvirt/qemu mode=0777
2016-04-06 04:15:35.945+0000: 14072: debug : qemuProcessStart:4619 : Creating domain log file
2016-04-06 04:15:35.945+0000: 14072: info : virObjectRef:296 : OBJECT_REF: obj=0x7f6a6c0ee900
2016-04-06 04:15:35.945+0000: 14072: info : virObjectUnref:259 : OBJECT_UNREF: obj=0x7f6a6c0ee900
2016-04-06 04:15:35.945+0000: 14072: debug : qemuProcessStart:4624 : Checking for KVM availability
2016-04-06 04:15:35.945+0000: 14072: debug : qemuDomainCheckDiskPresence:2761 : Checking for disk presence
2016-04-06 04:15:35.945+0000: 14072: info : virObjectRef:296 : OBJECT_REF: obj=0x7f6a6c0ee900
2016-04-06 04:15:35.945+0000: 14072: debug : virStorageFileGetMetadata:3100 : path=/var/lib/one//datastores/0/38/disk.1 format=1 uid=9869 gid=9869 probe=0, report_broken=1
2016-04-06 04:15:35.945+0000: 14072: debug : virStorageFileGetMetadataRecurse:2983 : path=/var/lib/one//datastores/0/38/disk.1 format=1 uid=9869 gid=9869 probe=0
2016-04-06 04:15:35.945+0000: 14072: debug : virStorageFileBackendFileInit:1441 : initializing FS storage file 0x7f6aa4009000 (file:/var/lib/one//datastores/0/38/disk.1)[9869:9869]
2016-04-06 04:15:35.954+0000: 14072: error : virStorageFileBackendFileGetUniqueIdentifier:1523 : can't canonicalize path '/var/lib/one//datastores/0/38/disk.1': Permission denied
2016-04-06 04:15:35.954+0000: 14072: debug : virStorageFileBackendFileDeinit:1424 : deinitializing FS storage file 0x7f6aa4009000 (file:/var/lib/one//datastores/0/38/disk.1)

2016-04-06 04:15:35.954+0000: 14072: info : virObjectUnref:259 : OBJECT_UNREF: obj=0x7f6a6c0ee900
2016-04-06 04:15:35.954+0000: 14072: debug : virFileClose:102 : Closed fd 23
2016-04-06 04:15:35.954+0000: 14072: info : virObjectRef:296 : OBJECT_REF: obj=0x7f6a6c0ee900
2016-04-06 04:15:35.954+0000: 14072: debug : qemuProcessStop:5189 : Shutting down vm=0x7f6aa4002b80 name=one-38 id=3 pid=0 flags=2
2016-04-06 04:15:35.954+0000: 14072: debug : virNetDaemonRemoveShutdownInhibition:460 : dmn=0x7f6ac8e89d90 inhibitions=0
2016-04-06 04:15:35.954+0000: 14072: info : virObjectRef:296 : OBJECT_REF: obj=0x7f6a6c0ee900
2016-04-06 04:15:35.954+0000: 14072: info : virObjectUnref:259 : OBJECT_UNREF: obj=0x7f6a6c0ee900
2016-04-06 04:15:35.954+0000: 14072: debug : virFileClose:102 : Closed fd 23
2016-04-06 04:15:35.955+0000: 14072: debug : qemuProcessKill:5146 : vm=0x7f6aa4002b80 name=one-38 pid=0 flags=5
2016-04-06 04:15:35.955+0000: 14072: debug : virProcessKillPainfully:352 : vpid=0 force=1
2016-04-06 04:15:35.955+0000: 14072: debug : qemuDomainCleanupRun:2852 : driver=0x7f6a6c0ebf60, vm=one-38

But not much more.  Have to dig into the C code.  After some reading I find this however it's not clear how much of this applies to the issue here:  

https://www.redhat.com/archives/libvir-list/2014-May/msg00194.html

The current implementation works for local
storage only and returns the canonical path of the volume.

But it seems the logic is applied to NFS mounts. Perhaps it shouldn't be?

After posting on the libvirtd mailing lists, it was suggested to modify the permissions of the /var/lib/one folder to chmod o+rx or just chmod o+x.  This worked to resolve the issue but we will instead keep the no permission setting on other and add the oneadmin user to the nfsnobody group like this:

# id oneadmin
uid=9869(oneadmin) gid=9869(oneadmin) groups=9869(oneadmin),992(libvirt),36(kvm)
#
# id nfsnobody
uid=65534(nfsnobody) gid=65534(nfsnobody) groups=65534(nfsnobody)

# usermod -a -G nfsnobody oneadmin
# id oneadmin
uid=9869(oneadmin) gid=9869(oneadmin) groups=9869(oneadmin),992(libvirt),36(kvm),65534(nfsnobody)
# id nfsnobody
uid=65534(nfsnobody) gid=65534(nfsnobody) groups=65534(nfsnobody)
#

( NOTE: To reverse you can run  gpasswd -d nfsnobody oneadmin and gpasswd -d oneadmin nfsnobody )

We even tried nobody user and that didn't work either.  Seems the system does this via the root account.

Cheers,
TK

2 Responses to “error: can’t canonicalize path /var/lib/one/datastores/ Permission denied”

  1. […] « error: can’t canonicalize path /var/lib/one/datastores/ Permission denied […]

  2. […] file we want to modify is the ~/.one/one_key with the same password as the *_auth files have after modifying the OpenNebula database as well as other authentication errors in the log file /var/log/one/oned.log […]

Leave a Reply

You must be logged in to post a comment.


     
  Copyright © 2003 - 2013 Tom Kacperski (microdevsys.com). All rights reserved.

Creative Commons License
This work is licensed under a Creative Commons Attribution 3.0 Unported License