Command execution fail: /var/lib/one/remotes/tm/shared/clone
When receiving these errors in the log files:
[oneadmin@opennebula01 one]$ cat 19.log
Wed Mar 30 23:59:30 2016 [Z0][DiM][I]: New VM state is ACTIVE.
Wed Mar 30 23:59:30 2016 [Z0][LCM][I]: New VM state is PROLOG.
Wed Mar 30 23:59:30 2016 [Z0][TM][I]: Command execution fail: /var/lib/one/remotes/tm/shared/clone opennebula01:/var/lib/one//datastores/1/370d6f8205e2d85d328e922d1f356ab7 mdskvm-p01:/var/lib/one//datastores/0/19/disk.0 19 1
Wed Mar 30 23:59:30 2016 [Z0][TM][I]: clone: Cloning /var/lib/one/datastores/1/370d6f8205e2d85d328e922d1f356ab7 in mdskvm-p01:/var/lib/one//datastores/0/19/disk.0
Wed Mar 30 23:59:30 2016 [Z0][TM][E]: clone: Command "cd /var/lib/one/datastores/0/19; cp /var/lib/one/datastores/1/370d6f8205e2d85d328e922d1f356ab7 /var/lib/one/datastores/0/19/disk.0" failed: Warning: Permanently added 'mdskvm-p01,192.168.0.60' (ECDSA) to the list of known hosts.
Wed Mar 30 23:59:30 2016 [Z0][TM][I]: cp: cannot stat '/var/lib/one/datastores/1/370d6f8205e2d85d328e922d1f356ab7': No such file or directory
Wed Mar 30 23:59:30 2016 [Z0][TM][E]: Error copying opennebula01:/var/lib/one//datastores/1/370d6f8205e2d85d328e922d1f356ab7 to mdskvm-p01:/var/lib/one//datastores/0/19/disk.0
Wed Mar 30 23:59:30 2016 [Z0][TM][I]: ExitCode: 1
Wed Mar 30 23:59:30 2016 [Z0][TM][E]: Error executing image transfer script: Error copying opennebula01:/var/lib/one//datastores/1/370d6f8205e2d85d328e922d1f356ab7 to mdskvm-p01:/var/lib/one//datastores/0/19/disk.0
Wed Mar 30 23:59:30 2016 [Z0][DiM][I]: New VM state is FAILED
[oneadmin@opennebula01 one]$
[oneadmin@opennebula01 ~]$ /var/lib/one/remotes/tm/shared/clone opennebula01:/var/lib/one//datastores/1/370d6f8205e2d85d328e922d1f356ab7 mdskvm-p01:/var/lib/one//datastores/0/19/disk.0 19 1
INFO: clone: Cloning /var/lib/one/datastores/1/370d6f8205e2d85d328e922d1f356ab7 in mdskvm-p01:/var/lib/one//datastores/0/19/disk.0
ERROR: clone: Command "cd /var/lib/one/datastores/0/19; cp /var/lib/one/datastores/1/370d6f8205e2d85d328e922d1f356ab7 /var/lib/one/datastores/0/19/disk.0" failed: Warning: Permanently added 'mdskvm-p01,192.168.0.60' (ECDSA) to the list of known hosts.
cp: cannot stat '/var/lib/one/datastores/1/370d6f8205e2d85d328e922d1f356ab7': No such file or directory
ERROR MESSAGE –8<——
Error copying opennebula01:/var/lib/one//datastores/1/370d6f8205e2d85d328e922d1f356ab7 to mdskvm-p01:/var/lib/one//datastores/0/19/disk.0
ERROR MESSAGE ——>8–
[oneadmin@opennebula01 ~]$
We will follow these few steps to resolve it. First we will check the ~/.ssh/config file that is created:
[oneadmin@opennebula01 .ssh]$ cat config
Host *
StrictHostKeyChecking no
UserKnownHostsFile /dev/null
[oneadmin@opennebula01 .ssh]$
Because we note the problem is with the red comment above. Next we will uncomment the UserKnownHostsFile /dev/null despite it's recommendation by OpenNebula (It is less secure anyway):
[oneadmin@mdskvm-p01 .ssh]$ ssh mdskvm-p01
Warning: Permanently added 'mdskvm-p01,192.168.0.60' (ECDSA) to the list of known hosts.
Last login: Thu Mar 31 00:21:12 2016 from opennebula01
[oneadmin@mdskvm-p01 ~]$ logout
Connection to mdskvm-p01 closed.
[oneadmin@mdskvm-p01 .ssh]$ ssh mdskvm-p01
Last login: Thu Mar 31 00:32:16 2016 from mdskvm-p01
[oneadmin@mdskvm-p01 ~]$
The second way to resolve this is by explicitly generating the known_hosts entries using the following but you will still have to remove the above UserKnownHostsFile /dev/null entry:
ssh-keyscan <ip-address> >> ~/.ssh/known_hosts
ssh-keyscan <hostname> >> ~/.ssh/known_hosts
For example:
ssh-keyscan 192.168.0.60 >> ~/.ssh/known_hosts
ssh-keyscan mdskvm-p01 >> ~/.ssh/known_hosts
A further solution, if above doesnt' work is to edit the datastore and change setting from shared to ssh. The commands needed are:
onedatastore list -x
onehost list -x
onedatastore update <datastore id>
Resulting in:
[oneadmin@opennebula01 datastores]$ onedatastore list -x|grep -E "ID|BASE|TM_MAD"
<ID>0</ID>
<UID>0</UID>
<GID>0</GID>
<TM_MAD><![CDATA[ssh]]></TM_MAD>
<BASE_PATH><![CDATA[/var/lib/one//datastores/0]]></BASE_PATH>
<CLUSTER_ID>-1</CLUSTER_ID>
<BASE_PATH><![CDATA[/var/lib/one//datastores/]]></BASE_PATH>
<TM_MAD><![CDATA[ssh]]></TM_MAD>
<ID>1</ID>
<UID>0</UID>
<GID>0</GID>
<TM_MAD><![CDATA[ssh]]></TM_MAD>
<BASE_PATH><![CDATA[/var/lib/one//datastores/1]]></BASE_PATH>
<CLUSTER_ID>-1</CLUSTER_ID>
<ID>1</ID>
<BASE_PATH><![CDATA[/var/lib/one//datastores/]]></BASE_PATH>
<TM_MAD><![CDATA[ssh]]></TM_MAD>
<ID>2</ID>
<UID>0</UID>
<GID>0</GID>
<TM_MAD><![CDATA[ssh]]></TM_MAD>
<BASE_PATH><![CDATA[/var/lib/one//datastores/2]]></BASE_PATH>
<CLUSTER_ID>-1</CLUSTER_ID>
<BASE_PATH><![CDATA[/var/lib/one//datastores/]]></BASE_PATH>
<TM_MAD><![CDATA[ssh]]></TM_MAD>
[oneadmin@opennebula01 datastores]$
Another reason is that I had accidentally deleted earlier so have to recreate the template and image again. The file below was missing:
/var/lib/one/datastores/1/370d6f8205e2d85d328e922d1f356ab7
And try again!
Cheers,
TK