When you receive the following errors:

[oneadmin@opennebula01 .one]$ /var/lib/one/remotes/tm/shared/clone opennebula01:/var/lib/one//datastores/1/370d6f8205e2d85d328e922d1f356ab7 mdskvm-p01:/var/lib/one//datastores/0/16/disk.0 16 1
INFO: clone: Cloning /var/lib/one/datastores/1/370d6f8205e2d85d328e922d1f356ab7 in mdskvm-p01:/var/lib/one//datastores/0/16/disk.0
ERROR: clone: Command "cd /var/lib/one/datastores/0/16; cp /var/lib/one/datastores/1/370d6f8205e2d85d328e922d1f356ab7 /var/lib/one/datastores/0/16/disk.0" failed: Warning: Permanently added 'mdskvm-p01,192.168.0.60' (ECDSA) to the list of known hosts.
cp: cannot stat '/var/lib/one/datastores/1/370d6f8205e2d85d328e922d1f356ab7': No such file or directory
ERROR MESSAGE –8<——
Error copying opennebula01:/var/lib/one//datastores/1/370d6f8205e2d85d328e922d1f356ab7 to mdskvm-p01:/var/lib/one//datastores/0/16/disk.0
ERROR MESSAGE ——>8–
[oneadmin@opennebula01 .one]$

Wed Mar 30 18:43:59 2016 [Z0][TM][D]: Message received: LOG I 16 Command execution fail: /var/lib/one/remotes/tm/shared/clone opennebula01:/var/lib/one//datastores/1/370d6f8205e2d85d328e922d1f356ab7 mdskvm-p01:/var/lib/one//datastores/0/16/disk.0 16 1

Wed Mar 30 18:43:59 2016 [Z0][TM][D]: Message received: LOG I 16 clone: Cloning /var/lib/one/datastores/1/370d6f8205e2d85d328e922d1f356ab7 in mdskvm-p01:/var/lib/one//datastores/0/16/disk.0

Wed Mar 30 18:43:59 2016 [Z0][TM][D]: Message received: LOG E 16 clone: Command "cd /var/lib/one/datastores/0/16; cp /var/lib/one/datastores/1/370d6f8205e2d85d328e922d1f356ab7 /var/lib/one/datastores/0/16/disk.0" failed: Warning: Permanently added 'mdskvm-p01,192.168.0.60' (ECDSA) to the list of known hosts.

Wed Mar 30 18:43:59 2016 [Z0][TM][D]: Message received: LOG I 16 cp: cannot stat '/var/lib/one/datastores/1/370d6f8205e2d85d328e922d1f356ab7': No such file or directory

Wed Mar 30 18:43:59 2016 [Z0][TM][D]: Message received: LOG E 16 Error copying opennebula01:/var/lib/one//datastores/1/370d6f8205e2d85d328e922d1f356ab7 to mdskvm-p01:/var/lib/one//datastores/0/16/disk.0

Wed Mar 30 18:43:59 2016 [Z0][TM][D]: Message received: LOG I 16 ExitCode: 1

Wed Mar 30 18:43:59 2016 [Z0][TM][D]: Message received: TRANSFER FAILURE 16 Error copying opennebula01:/var/lib/one//datastores/1/370d6f8205e2d85d328e922d1f356ab7 to mdskvm-p01:/var/lib/one//datastores/0/16/disk.0

here's what you can do about them.  After further inspection by jumping in on the target node, we see that the NFS filesystem is NOT mounted.  Also ensure that the NFS service on sunstone (OpenNebula collector) is running:

[root@opennebula01 one]# service nfs restart
Redirecting to /bin/systemctl restart  nfs.service
[root@opennebula01 one]#
[root@mdskvm-p01 ~]# grep one /etc/fstab
192.168.0.70:/var/lib/one/      /var/lib/one/  nfs   context=system_u:object_r:nfs_t:s0,soft,intr,rsize=8192,wsize=8192,noauto
[root@mdskvm-p01 ~]#

Cheers,
TK

When you get this message, it means the passwords between oneadmin and serveradmin are not in sync or you have the wrong hash in ~/.one/ files and the oned (pgrep -lf oned) service isn't really connecting to anything to get the username it needs.  

"OpenNebula USER name not found, use the ID instead"

We will set about modifying the one file that has the old key.  

Read the rest of this entry »

For the error VirtualMachinePoolInfo result FAILURE [VirtualMachinePoolInfo] User couldn't be authenticated, aborting call. 7 this is likely due to wrong passwords for the oneadmin user.  You can reset it here.

However, this can also be due to the oned service not recycling using one stop then one start. When we try this:

[oneadmin@opennebula01 ~]$ one stop
oned and scheduler stopped
[oneadmin@opennebula01 ~]$ one start
oned failed to start
[oneadmin@opennebula01 ~]$ one stop
oned and scheduler stopped
[oneadmin@opennebula01 ~]$ pgrep -lf oned
1024 oned
41244 tail
[oneadmin@opennebula01 ~]$

To kill it, simply use the kill command and recycle.  The running daemon was still using the old password, hence the authentication failures.  Issuing a basic kill without a -9 should gracefully stop the agent:

Wed Mar 30 12:21:46 2016 [Z0][ACL][I]: ACL Manager stopped.
Wed Mar 30 12:21:46 2016 [Z0][LCM][I]: Stopping Life-cycle Manager…
Wed Mar 30 12:21:46 2016 [Z0][ImM][I]: Stopping Image Manager…
Wed Mar 30 12:21:46 2016 [Z0][TrM][I]: Stopping Transfer Manager…
Wed Mar 30 12:21:46 2016 [Z0][DiM][I]: Stopping Dispatch Manager…
Wed Mar 30 12:21:46 2016 [Z0][DiM][I]: Dispatch Manager stopped.
Wed Mar 30 12:21:46 2016 [Z0][VMM][I]: Stopping Virtual Machine Manager…
Wed Mar 30 12:21:46 2016 [Z0][ReM][I]: Stopping Request Manager…
Wed Mar 30 12:21:46 2016 [Z0][HKM][I]: Stopping Hook Manager…
Wed Mar 30 12:21:46 2016 [Z0][InM][I]: Stopping Information Manager…
Wed Mar 30 12:21:46 2016 [Z0][LCM][I]: Life-cycle Manager stopped.
Wed Mar 30 12:21:46 2016 [Z0][ReM][I]: XML-RPC server stopped.
Wed Mar 30 12:21:46 2016 [Z0][ReM][I]: Request Manager stopped.
Wed Mar 30 12:21:47 2016 [Z0][TrM][I]: Transfer Manager stopped.
Wed Mar 30 12:21:47 2016 [Z0][HKM][I]: Hook Manager stopped.
Wed Mar 30 12:21:47 2016 [Z0][ImM][I]: Image Manager stopped.
Wed Mar 30 12:21:48 2016 [Z0][VMM][I]: Virtual Machine Manager stopped.
Wed Mar 30 12:21:49 2016 [Z0][InM][I]: Information Manager stopped.
Wed Mar 30 12:21:49 2016 [Z0][ONE][I]: All modules finalized, exiting.

If that doesn't work we will check the hash passwords stored in the DB vs the files:

[oneadmin@opennebula01 one]$ sqlite3 /var/lib/one/one.db
SQLite version 3.7.17 2013-05-20 00:56:22
Enter ".help" for instructions
Enter SQL statements terminated with a ";"
sqlite> select body from user_pool where name='serveradmin';
<USER><ID>1</ID><GID>0</GID><GROUPS><ID>0</ID></GROUPS><GNAME>oneadmin</GNAME><NAME>serveradmin</NAME><PASSWORD>4987bfaa9b688a59ac4c3ed4c1d9b02276831dbe</PASSWORD><AUTH_DRIVER>server_cipher</AUTH_DRIVER><ENABLED>1</ENABLED><LOGIN_TOKEN/><TEMPLATE><TOKEN_PASSWORD><![CDATA[1ca6ef8b8530f50001301c69dac1da095b40075e]]></TOKEN_PASSWORD></TEMPLATE></USER>
sqlite>

[oneadmin@opennebula01 .one]$ for service in occi sunstone ec2 oneflow onegate; do cat ${service}_auth; done
serveradmin:a29082b94205a6f1bb8f17538c563acdf7340095
serveradmin:a29082b94205a6f1bb8f17538c563acdf7340095
serveradmin:a29082b94205a6f1bb8f17538c563acdf7340095
serveradmin:a29082b94205a6f1bb8f17538c563acdf7340095
serveradmin:a29082b94205a6f1bb8f17538c563acdf7340095
[oneadmin@opennebula01 .one]$ pwd
/var/lib/one/.one
[oneadmin@opennebula01 .one]$

If it is not the same, copy the DB password too the files or set the serveradmin pass in the DB to the one saved in the configuration files.  If that doesn't work, copy the oneadmin password to be used as the serveradmin password:

[oneadmin@opennebula01 .one]$ sqlite3 /var/lib/one/one.db
SQLite version 3.7.17 2013-05-20 00:56:22
Enter ".help" for instructions
Enter SQL statements terminated with a ";"
sqlite> select body from user_pool where name='oneadmin';
<USER><ID>0</ID><GID>0</GID><GROUPS><ID>0</ID></GROUPS><GNAME>oneadmin</GNAME><NAME>oneadmin</NAME><PASSWORD>ac1bf93cba27488935f522a78b764f5f65ae208a</PASSWORD><AUTH_DRIVER>core</AUTH_DRIVER><ENABLED>1</ENABLED><LOGIN_TOKEN/><TEMPLATE><SSH_PUBLIC_KEY><![CDATA[ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC+s5dZEwpCXE0OM9ZX+v6KvIS+o+NIyFzg1OYOgIemb60eu4E5T1z918vadr6LD15o1CC2TQRB8LqUAmrMOHXa2DusdLzDrVYOS60Z56Sc0e23ZbO/KnubgFRzeiTngnJDnUzuxfIrlaGqUmNvOUP02+aJP3T9+AzMgKw/lQvUp7MCX2ZAZKNuEvMlVluCcEzbB9XHG4puB6AEU7505nfSgavDxCEVFmEtmJHvv8a6LwPk/3AaPM5souLQL6yjD8/Y7gQ8qz1BHlaVTr9UwTGM0Ccecr8QQtzGxPKNE3no2ASdbTuQh0f1/UsOgm1hHGuHQnzvD9gpw+RXorU/UJHJ oneadmin@opennebula01]]> sqlite>

sqlite> select body from user_pool where name='serveradmin';
<USER><ID>1</ID><GID>0</GID><GROUPS><ID>0</ID></GROUPS><GNAME>oneadmin</GNAME><NAME>serveradmin</NAME><PASSWORD>4987bfaa9b688a59ac4c3ed4c1d9b02276831dbe</PASSWORD><AUTH_DRIVER>server_cipher</AUTH_DRIVER><ENABLED>1</ENABLED><LOGIN_TOKEN/><TEMPLATE><TOKEN_PASSWORD><![CDATA[1ca6ef8b8530f50001301c69dac1da095b40075e]]></TOKEN_PASSWORD></TEMPLATE></USER>
sqlite> .quit
 

And then update the ~/.one/*_auth files mentioned above.  

If that doesn't work to resolve the above, also change the hash inside the ~/.one/one_key file to match the hash above.

Restart using one stop and one start and enjoy!

Cheers,
TK

Appears the error can't canonicalize path /var/lib/one/datastores/ Permission denied has to do with passwords not being in sync with what they were before.  Changing oneadmin password.  Password is hashed with sha1 in the database and the database is /var/lib/one/one.db.  Passwords in the ~/.one/one_auth and other *_auth as well as one_key files is plaintext:

Read the rest of this entry »

On this CentOS 7 (both worker and controller), we cannot ssh using pass less keys from the controller opennebula01 to the worker mdskvm-p01 when the NFS share ( 192.168.0.70:/var/lib/one /var/lib/one) is mounted on the worker. But I can as soon as I unmount the opennebula01 NFS share off of the worker node mdskvm-p01. When the NFS is mounted, both worker and controller share a common /var/lib/one/.ssh/authorized_keys file, which seems to be the intent of the setup:

[oneadmin@mdskvm-p01 .ssh]$ mount|tail -n 1
192.168.0.70:/var/lib/one on /var/lib/one type nfs4 (rw,relatime,vers=4.0,rsize=8192,wsize=8192,namlen=255,soft,proto=tcp,port=0,timeo=600,retrans=2,sec=sys,clientaddr=192.168.0.60,local_lock=none,addr=192.168.0.70)
[oneadmin@mdskvm-p01 .ssh]$ pwd
/var/lib/one/.ssh
[oneadmin@mdskvm-p01 .ssh]$

Now when I run SSHD in debug mode using port 2222, the passless key works fine with or without NFS mounted on the worker node. Why?

Is there a specific sshd config file entry that prevents passless key login if sshd is not running in debug mode or when the OpenNebula NFS share is mounted? Again, in debug, SSHD works fine with or without the NFS mount on the worker mdskvm-p01.  To answer these questions we will run through some checks:

Read the rest of this entry »

This is a quick way to enable system services using systemctl instead of using chkconfig:

[root@mdskvm-p01 audit]# systemctl list-unit-files|grep -i ntp
ntpd.service                              disabled
ntpdate.service                           disabled
[root@mdskvm-p01 audit]# systemctl enable ntp
Failed to execute operation: Access denied
[root@mdskvm-p01 audit]# systemctl enable ntpd
Created symlink from /etc/systemd/system/multi-user.target.wants/ntpd.service to /usr/lib/systemd/system/ntpd.service.
[root@mdskvm-p01 audit]# systemctl list-unit-files|grep -i ntp
ntpd.service                              enabled
ntpdate.service                           disabled
[root@mdskvm-p01 audit]# systemctl enable ntpdate
Created symlink from /etc/systemd/system/multi-user.target.wants/ntpdate.service to /usr/lib/systemd/system/ntpdate.service.
[root@mdskvm-p01 audit]# systemctl list-unit-files|grep -i ntp
ntpd.service                              enabled
ntpdate.service                           enabled
[root@mdskvm-p01 audit]#

Cheers,
TK

Right here:

/usr/lib/systemd/system

RHEL 7 / Cent OS 7 / Scientific Linux 7 are quite different beasts from previous RHEL environments.  The systemctl restart <SERVICE> calls scripts from the above aformentioned folder.

Cheers,
TK

Small problem encountered when instantiating a virtual machine on OpenNebula ().  

PROBLEM

[oneadmin@opennebula01 .ssh]$ tail -f /var/log/one/1.log
Mon Mar 28 00:15:43 2016 [Z0][DiM][I]: New VM state is ACTIVE.
Mon Mar 28 00:15:43 2016 [Z0][LCM][I]: New VM state is PROLOG.
Mon Mar 28 00:15:43 2016 [Z0][TM][I]: Command execution fail: /var/lib/one/remotes/tm/shared/clone opennebula01:/var/lib/one//datastores/1/ac152a0c5c61978e8448f68ade249a98 mdskvm-p01:/var/lib/one//datastores/0/1/disk.0 1 1
Mon Mar 28 00:15:43 2016 [Z0][TM][I]: clone: Cloning /var/lib/one/datastores/1/ac152a0c5c61978e8448f68ade249a98 in mdskvm-p01:/var/lib/one//datastores/0/1/disk.0
Mon Mar 28 00:15:43 2016 [Z0][TM][E]: clone: Command "cd /var/lib/one/datastores/0/1; cp /var/lib/one/datastores/1/ac152a0c5c61978e8448f68ade249a98 /var/lib/one/datastores/0/1/disk.0" failed: Warning: Permanently added 'mdskvm-p01,192.168.0.60' (ECDSA) to the list of known hosts.
Mon Mar 28 00:15:43 2016 [Z0][TM][I]: cp: cannot stat '/var/lib/one/datastores/1/ac152a0c5c61978e8448f68ade249a98': No such file or directory
Mon Mar 28 00:15:43 2016 [Z0][TM][E]: Error copying opennebula01:/var/lib/one//datastores/1/ac152a0c5c61978e8448f68ade249a98 to mdskvm-p01:/var/lib/one//datastores/0/1/disk.0
Mon Mar 28 00:15:43 2016 [Z0][TM][I]: ExitCode: 1
Mon Mar 28 00:15:43 2016 [Z0][TM][E]: Error executing image transfer script: Error copying opennebula01:/var/lib/one//datastores/1/ac152a0c5c61978e8448f68ade249a98 to mdskvm-p01:/var/lib/one//datastores/0/1/disk.0
Mon Mar 28 00:15:43 2016 [Z0][DiM][I]: New VM state is FAILED

[oneadmin@opennebula01 .ssh]$

SOLUTION

Ensure that the mount is possible and that the NFS daemon is running on OpenNebula:

/etc/fstab
192.168.0.70:/var/lib/one/  /var/lib/one/  nfs   soft,intr,rsize=8192,wsize=8192,noauto

[oneadmin@opennebula01 .ssh]$ cat /etc/exports
/var/lib/one/ *(rw,sync,no_subtree_check,root_squash)
[oneadmin@opennebula01 .ssh]$ ps -ef|grep -i nfs
oneadmin 12481  9559  0 00:22 pts/0    00:00:00 grep –color=auto -i nfs
[oneadmin@opennebula01 .ssh]$
[oneadmin@opennebula01 .ssh]$

[root@opennebula01 ~]# systemctl start nfs
[root@opennebula01 ~]#

Alos if you note above, I had used noauto in /etc/fstab.  Change to auto if you trust the NFS won't lock up your boot order getting stuck on NFS.

Cheers,
TK

This is likely due to registry changes via installed software.  To change, revert this value to 2:

HKEY_LOCAL_MACHINE/System/CurrentControlSet/Control/Windows/ErrorMode Set to 2 from 0.

Cheers,
TK

There are two way's to setup virtual machine connectivity to the outside using libvirt.  The preferred is to create a bridged network (shared physical device) and the alternate is to create NAT forwarding (virtual network).  We will look at the first, then the second. The virbr0 interface may come installed as part of the KVM virtualization package on your environment.  To change it change to the following path to edit it's settings:

Read the rest of this entry »