Header Shadow Image


Cloudera Manager Installation Issues

When getting the following errors below on the Cloudera Manager Installation on RHEL 7.2+, try both a Date and Time Configuration and stopping then starting the agent service with these commands:

systemctl stop cloudera-scm-agent
systemctl start cloudera-scm-agent

If you get the following messages.  In the absence of a DNS server, also check and configure the /etc/hosts for the hostname and IP similar to this: 

123.123.123.123        host.domain.xyz host

Or if you are using DD-WRT use records such as this:

address=/host1/192.168.0.165
address=/host1.mds.xyz/192.168.0.165
ptr-record=165.0.168.192.in-addr.arpa,"mds-host1.mds.xyz"
ptr-record=165.0.168.192.in-addr.arpa,"mds-host1"

Exact error message received:

Installation failed. Failed to receive heartbeat from agent.

  • Ensure that the host's hostname is configured properly.
  • Ensure that port 7182 is accessible on the Cloudera Manager Server (check firewall rules).
  • Ensure that ports 9000 and 9001 are not in use on the host being added.
  • Check agent logs in /var/log/cloudera-scm-agent/ on the host being added. (Some of the logs can be found in the installation details).
  • If Use TLS Encryption for Agents is enabled in Cloudera Manager (Administration -> Settings -> Security), ensure that/etc/cloudera-scm-agent/config.ini has use_tls=1 on the host being added. Restart the corresponding agent and click the Retry link here.


>>[14/Sep/2016 15:24:13 +0000] 16330 Dummy-14 agent ERROR Failed to kill process with pid 16358
OSError: [Errno 3] No such process
>>[14/Sep/2016 15:24:13 +0000] 16330 Dummy-14 agent ERROR Shutdown callback failed.
>>OSError: [Errno 9] Bad file descriptor
>>[14/Sep/2016 15:24:13 +0000] 16330 Dummy-14 agent ERROR Shutdown callback failed.
KeyError: 15

The proper response given is:

# nslookup 192.168.0.165
Server:         192.168.0.1
Address:        192.168.0.1#53

165.0.168.192.in-addr.arpa      name = mds-host1.
165.0.168.192.in-addr.arpa      name = mds-host1.mds.xyz.

#

Once installation progressed, we got these messages:

Transparent Huge Page Compaction is enabled and can cause significant performance problems. Run "echo never > /sys/kernel/mm/transparent_hugepage/defrag" to disable this, then add the same command to an init script such as /etc/rc.local so it will be set upon system reboot. The following hosts are affected: 
mds-host05; mds-host[01-04]


Cloudera recommends setting /proc/sys/vm/swappiness to a maximum of 10. Current setting is 30. Use the sysctl command to change this setting at run time and edit /etc/sysctl.conf for this setting to be saved after a reboot. You can continue with installation, but Cloudera Manager might report that your hosts are unhealthy because they are swapping. The following hosts are affected: 
mds-host05; mds-host[01-04]


The following failures were observed in checking hostnames… 
Host mds-host01 expected to have name mds-host01 but resolved (InetAddress.getLocalHost().getHostName()) itself to mds-host01.mds.xyz.

 

Resolve these accordingly to above instructions and continue.  To set the swappiness run the following:

sysctl -w vm.swappiness=10

And continue with the installation.  If you get this error:

/usr/lib64/cmf/service/zookeeper/zkserver.sh: line 41: /var/lib/zookeeper/myid: Permission denied
Supervisor returned FATAL. Please check the role log file, stderr, or stdout.
Completed only 0/1 steps. First failure: Command (85) has failed
Failed to start role.
Completed only 1/2 steps. First failure: Failed to execute command Start on service ZooKeeper

Simply change permissions like this:

101947599 d———.  2 root root    6 Sep 14 23:52 /var/lib/zookeeper

chmod 755 /var/lib/zookeeper

101947599 drwxr-xr-x. 2 root root 6 Sep 14 23:52 /var/lib/zookeeper

And continue the install.  Possibly /usr/lib64/cmf/service/zookeeper/zkserver.sh might not be setting permissions correctly.  Also set the folder to zookeeper.zookeeper as well:

chmod 755 /var/lib/zookeeper; chown zookeeper.zookeeper /var/lib/zookeeper; ls -altrid /var/lib/zookeeper

And continue.  Also set the ACL's on the folder:

# setfacl -m "u:zookeeper:rwx,g:zookeeper:rwx" /var/lib/zookeeper/
# getfacl zookeeper
# file: zookeeper
# owner: zookeeper
# group: zookeeper
user::rwx
user:zookeeper:rwx
group::rwx
group:zookeeper:rwx
mask::rwx
other::rwx

#

Yet still did not work.  Digging deeper we see:

[root@mds-host01 zookeeper]# pwd
/var/log/zookeeper
[root@mds-host01 zookeeper]# tail -f zookeeper-cmf-zookeeper-SERVER-mds-host01.log -n 10
2016-09-15 01:48:59,395 INFO org.apache.zookeeper.server.ZooKeeperServer: Server environment:user.home=/var/lib/zookeeper
2016-09-15 01:48:59,396 INFO org.apache.zookeeper.server.ZooKeeperServer: Server environment:user.dir=/run/cloudera-scm-agent/process/31-zookeeper-server
2016-09-15 01:48:59,396 ERROR org.apache.zookeeper.server.ZooKeeperServerMain: Unable to access datadir, exiting abnormally
org.apache.zookeeper.server.persistence.FileTxnSnapLog$DatadirException: Missing data directory /var/lib/zookeeper/version-2, automatic data directory creation is disabled (zookeeper.datadir.autocreate is false). Please create this directory manually.
        at org.apache.zookeeper.server.persistence.FileTxnSnapLog.<init>(FileTxnSnapLog.java:102)
        at org.apache.zookeeper.server.ZooKeeperServerMain.runFromConfig(ZooKeeperServerMain.java:109)
        at org.apache.zookeeper.server.ZooKeeperServerMain.initializeAndRun(ZooKeeperServerMain.java:91)
        at org.apache.zookeeper.server.ZooKeeperServerMain.main(ZooKeeperServerMain.java:53)
        at org.apache.zookeeper.server.quorum.QuorumPeerMain.initializeAndRun(QuorumPeerMain.java:121)
        at org.apache.zookeeper.server.quorum.QuorumPeerMain.main(QuorumPeerMain.java:79)

 

So let's create it manually and retry the installation.  So command was:

chmod 755 /var/lib/zookeeper; chown zookeeper.zookeeper /var/lib/zookeeper; ls -altrid /var/lib/zookeeper; setfacl -m "u:zookeeper:rwx,g:zookeeper:rwx" /var/lib/zookeeper/; getfacl /var/lib/zookeeper/; mkdir /var/lib/zookeeper/version-2; chown zookeeper.zookeeper /var/lib/zookeeper/version-2;

This time above fixed the issue.  Retry and continue with installation. 

Cheers,
TK

Date and Time Configuration on RHEL 7, CentOS 7, Scientific Linux 7 or Fedora

yum install ntp -y; timedatectl set-ntp yes; timedatectl set-timezone America/Toronto

Use timedatectl list-timezones to get your timezone.

Cheers,
TK

Wrap lines in journalctl output

journalctl -b –no-pager | less

Easy peasy!

Cheers, TK

Creating a Minecraft PE Server

Basically it boils down to this running on Ubuntu:

minecraft@minecraftpe01:~$ wget -q -O – https://raw.githubusercontent.com/PocketMine/php-build-scripts/master/installer.sh | bash -s -

If the above doesn't work, try this:

wget -q -O compile.sh https://raw.githubusercontent.com/PocketMine/php-build-scripts/master/compile.sh

then install any dependencies it complains about.

To ensure you are compatible with the latest protocols, ensure your .phar file is updated. Direct link is here:

wget -O PocketMine-MP_1.6dev-27_ef8227a0_API-2.0.0.phar https://bintray.com/pocketmine/PocketMine/download_file?file_path=PocketMine-MP_1.6dev-27_ef8227a0_API-2.0.0.phar

minecraft@minecraftpe01:~$ cp -ip PocketMine-MP_1.6dev-27_ef8227a0_API-2.0.0.phar PocketMine-MP.phar
cp: overwrite 'PocketMine-MP.phar'? y
minecraft@minecraftpe01:~$ ls -altri *phar
694443 -rw-rw-r– 1 minecraft minecraft 3966174 Jun 15 18:50 PocketMine-MP.phar
701448 -rw-rw-r– 1 minecraft minecraft 3966174 Jun 15 18:50 PocketMine-MP_1.6dev-27_ef8227a0_API-2.0.0.phar
minecraft@minecraftpe01:~$ ls -altri *phar*
700731 -rw-rw-r– 1 minecraft minecraft     287 May  2 00:34 PocketMine-MP.phar.sig
700960 -rw-rw-r– 1 minecraft minecraft 3961710 May  2 00:38 PocketMine-MP.phar-old
694443 -rw-rw-r– 1 minecraft minecraft 3966174 Jun 15 18:50 PocketMine-MP.phar
701448 -rw-rw-r– 1 minecraft minecraft 3966174 Jun 15 18:50 PocketMine-MP_1.6dev-27_ef8227a0_API-2.0.0.phar
minecraft@minecraftpe01:~$
minecraft@minecraftpe01:~$

You can also download the latest .phar file to allow your latest client to work with this server.  The .phar file can be found here: https://ci.itxtech.org/job/Genisys/lastSuccessfulBuild/

 Source documentation.  Further sources available here and here.

Cheers,
TK

Failed to execute operation: Access denied

When running you get this message:

# systemctl enable nfs-server.service
Failed to execute operation: Access denied

Solution is to disable selinux or to allow for proper SELinux rules to allow for the above action:

# vi /etc/sysconfig/selinux

SELINUX=disabled

# setenforce 0
# getenforce
Permissive

Alternately see the following post for details on how to tailor SELinux rules for similar cases.

Cheers,
TK

Error: A specified parameter was not correct vim.host.diskpartitioninfo.spec

SSH to esxi (enable ssh via console if not enabled) then:

esxcfg-scsidevs -l

then 

partedUtil delete <DISK> <PARTITION>

example:

partedUtil delete (Gave invalid partition number) :

partedUtil delete /vmfs/devices/disks/naa.600508b1001c7ef7f736cd3dbeb5915d vml.0200010000600508b1001c7ef7f736cd3dbeb5915d4c4f47494341

Cheers,
TK

OpenNebula VM Password Prompt

When instantiating VM's through OpenNebula, when logging into those VM's, the VM's may ask you for a password.  The solution here is to edit the template in the GUI and add a context to it including the public SSH key.  Click on the template then Update from the top right hand side buttom row then enter the key in:

 

SSH Password Prompt - Missing SSH Contextualization

If the above field is empty, ensure it has the proper passless SSH key.  Further reading on contextualization.

The other reason for this issue is that you are trying to get into your machine as opennebula.  Try as root instead:

[oneadmin@opennebula01 .ssh]$ ssh 192.168.0.100
oneadmin@192.168.0.100's password:

[oneadmin@opennebula01 .ssh]$ ssh root@192.168.0.100
[root@localhost ~]#

Cheers,
TK

 

How to tail two files simultaneously

[oneadmin@opennebula01 one]$ tail -f oned.log sched.log

==> sched.log <==

Tue Apr 12 21:09:36 2016 [Z0][SCHED][I]: Getting scheduled actions information. Total time: 0.00560014s

==> oned.log <==
Tue Apr 12 21:09:36 2016 [Z0][ReM][D]: Req:8160 UID:0 VirtualMachinePoolInfo invoked , -2, -1, -1, -1
Tue Apr 12 21:09:36 2016 [Z0][ReM][D]: Req:8160 UID:0 VirtualMachinePoolInfo result SUCCESS, "<VM_POOL><VM><ID>40<…"

==> sched.log <==
Tue Apr 12 21:09:36 2016 [Z0][SCHED][I]: Getting VM and Host information. Total time: 0.00360117s

==> oned.log <==
Tue Apr 12 21:09:52 2016 [Z0][InM][D]: Host mdskvm-p01 (2) successfully monitored.
Tue Apr 12 21:09:52 2016 [Z0][VMM][D]: VM 40 successfully monitored: STATE=a USEDCPU=0.0 USEDMEMORY=524288 NETRX=344326 NETTX=1360

 

Cheers,
TK

VNC Failed to connect to server (code: 1006)

When receiving the following on OpenStack or OpenNebula:

VNC Failed to connect to server (code: 1006)

try the below steps to resolve the issue on either hypervisor manager:

Read the rest of this entry »

[TemplateInstantiate] Error allocating a new virtual machine. DISK 0: Cannot acquire image 3, it is in state: LOCKED

Give it some time for the image to download and create first.  When ready, instantiate again:

[oneadmin@opennebula01 1]$ ls -altri
total 647896
 68474907 drwxrwxr-x 5 oneadmin oneadmin        30 Mar 30 18:58 ..
101870759 -rw-r–r– 1 oneadmin oneadmin 395074048 Mar 31 02:41 2a3d337ded54dcbcc35865daf54e3c9a
101870807 drwxrwxr-x 2 oneadmin oneadmin        84 Mar 31 02:41 .
101870760 -rw-r–r– 1 oneadmin oneadmin 174383104 Mar 31 02:42 bf2a2c5916e78efcd0223b9c98971278
[oneadmin@opennebula01 1]$
[oneadmin@opennebula01 1]$
[oneadmin@opennebula01 1]$ onevm list
    ID USER     GROUP    NAME            STAT UCPU    UMEM HOST             TIME

[oneadmin@opennebula01 1]$
[oneadmin@opennebula01 1]$
[oneadmin@opennebula01 1]$
[oneadmin@opennebula01 1]$ onetemplate instantiate "CentOS-7"
[TemplateInstantiate] Error allocating a new virtual machine. DISK 0: Cannot acquire image 3, it is in state: LOCKED
[oneadmin@opennebula01 1]$ ls -altri
total 771632
 68474907 drwxrwxr-x 5 oneadmin oneadmin        30 Mar 30 18:58 ..
101870759 -rw-r–r– 1 oneadmin oneadmin 395074048 Mar 31 02:41 2a3d337ded54dcbcc35865daf54e3c9a
101870807 drwxrwxr-x 2 oneadmin oneadmin        84 Mar 31 02:41 .
101870760 -rw-r–r– 1 oneadmin oneadmin 395074048 Mar 31 02:43 bf2a2c5916e78efcd0223b9c98971278
[oneadmin@opennebula01 1]$
[oneadmin@opennebula01 1]$
[oneadmin@opennebula01 1]$ onetemplate instantiate "CentOS-7"
VM ID: 24
[oneadmin@opennebula01 1]$ onevm list
    ID USER     GROUP    NAME            STAT UCPU    UMEM HOST             TIME
    24 oneadmin oneadmin CentOS-7-24     pend    0      0K              0d 00h00
[oneadmin@opennebula01 1]$


     
  Copyright © 2003 - 2013 Tom Kacperski (microdevsys.com). All rights reserved.

Creative Commons License
This work is licensed under a Creative Commons Attribution 3.0 Unported License