Header Shadow Image


SAN: High Level Overview

A picture says a thousand words:

 

SAN: High Level Diagram

Cannot add dependency job for unit firewalld.service, ignoring: Unit firewalld.service is masked.

Restore firewalld.service file contents.

[root@idmipa01 ~]# systemctl enable firewalld
Failed to execute operation: Operation not supported
[root@idmipa01 ~]# 

[root@idmipa01 ~]# systemctl enable firewall
Failed to execute operation: Access denied
[root@idmipa01 ~]# 

[root@idmipa01 system]# vi firewalld.service
[root@idmipa01 system]# systemctl enable firewalld
Created symlink from /etc/systemd/system/dbus-org.fedoraproject.FirewallD1.service to /usr/lib/systemd/system/firewalld.service.
Created symlink from /etc/systemd/system/basic.target.wants/firewalld.service to /usr/lib/systemd/system/firewalld.service.
[root@idmipa01 system]# systemctl start firewalld
[root@idmipa01 system]#
[root@idmipa01 system]#
[root@idmipa01 system]#
[root@idmipa01 system]# cat firewalld.service
[Unit]
Description=firewalld – dynamic firewall daemon
Before=network.target
Before=libvirtd.service
Before=NetworkManager.service
Conflicts=iptables.service ip6tables.service ebtables.service

[Service]
EnvironmentFile=-/etc/sysconfig/firewalld
ExecStart=/usr/sbin/firewalld –nofork –nopid $FIREWALLD_ARGS
ExecReload=/bin/kill -HUP $MAINPID
# supress to log debug and error output also to /var/log/messages
StandardOutput=null
StandardError=null
Type=dbus
BusName=org.fedoraproject.FirewallD1

[Install]
WantedBy=basic.target
Alias=dbus-org.fedoraproject.FirewallD1.service
[root@idmipa01 system]#

 

 

/etc/resolv.conf is getting overwritten

So /etc/resolv.conf was getting consistently overwritten even though NetworkManager was removed using rpm -e $(rpm -aq|grep Network Manager).  To find the culprit, we checked what is running on this CentOS 7 system using:

Read the rest of this entry »

Setting Temporary Network Addresses

Setting Temporary Network Addresses

ip a add 192.168.0.200/255.255.255.0 dev eth0
ip route add default 192.168.0.1/255.255.255.0 dev eth0

Cheers,
TK

INFO task kworker blocked for more than 120 seconds.

Well one day we get this issue (INFO task kworker blocked for more than 120 seconds.):

Read the rest of this entry »

VMware : Unable to connect to the MKS: Login (username / password) incorrect

For this error make sure that the Windows Firewall with Advanced Security has the following two rules enabled on both the Domain and Private profiles:

VMware Authd Service
VMware Authd Service (private)

Cheers,
TK

David Korn, where are you?

Folks are asking where you are?

I've wondered the same and hope to see David Korn back in action again.  Thread's been quiet.  Did some reasearch earlier and David Korn was hired by Google after departing from AT&T (http://www.unix.com/what-is-on-your-mind-/237119-david-korn-glenn-fowler-laid-off.html). No idea what's he's working on but his profile is: 

https://www.linkedin.com/in/david-korn-b23185

And some chat about this is at:
http://unix.stackexchange.com/questions/246338/is-the-shell-ksh93-dead
https://github.com/dgkorn

There was some GitHub work on it:
https://gist.github.com/qbit/5483415
https://github.com/att/ksh

Read the rest of this entry »

Cloudera Manager Installation Issues

When getting the following errors below on the Cloudera Manager Installation on RHEL 7.2+, try both a Date and Time Configuration and stopping then starting the agent service with these commands:

systemctl stop cloudera-scm-agent
systemctl start cloudera-scm-agent

If you get the following messages.  In the absence of a DNS server, also check and configure the /etc/hosts for the hostname and IP similar to this: 

123.123.123.123        host.domain.xyz host

Or if you are using DD-WRT use records such as this:

address=/host1/192.168.0.165
address=/host1.mds.xyz/192.168.0.165
ptr-record=165.0.168.192.in-addr.arpa,"mds-host1.mds.xyz"
ptr-record=165.0.168.192.in-addr.arpa,"mds-host1"

Exact error message received:

Installation failed. Failed to receive heartbeat from agent.

  • Ensure that the host's hostname is configured properly.
  • Ensure that port 7182 is accessible on the Cloudera Manager Server (check firewall rules).
  • Ensure that ports 9000 and 9001 are not in use on the host being added.
  • Check agent logs in /var/log/cloudera-scm-agent/ on the host being added. (Some of the logs can be found in the installation details).
  • If Use TLS Encryption for Agents is enabled in Cloudera Manager (Administration -> Settings -> Security), ensure that/etc/cloudera-scm-agent/config.ini has use_tls=1 on the host being added. Restart the corresponding agent and click the Retry link here.


>>[14/Sep/2016 15:24:13 +0000] 16330 Dummy-14 agent ERROR Failed to kill process with pid 16358
OSError: [Errno 3] No such process
>>[14/Sep/2016 15:24:13 +0000] 16330 Dummy-14 agent ERROR Shutdown callback failed.
>>OSError: [Errno 9] Bad file descriptor
>>[14/Sep/2016 15:24:13 +0000] 16330 Dummy-14 agent ERROR Shutdown callback failed.
KeyError: 15

The proper response given is:

# nslookup 192.168.0.165
Server:         192.168.0.1
Address:        192.168.0.1#53

165.0.168.192.in-addr.arpa      name = mds-host1.
165.0.168.192.in-addr.arpa      name = mds-host1.mds.xyz.

#

Once installation progressed, we got these messages:

Transparent Huge Page Compaction is enabled and can cause significant performance problems. Run "echo never > /sys/kernel/mm/transparent_hugepage/defrag" to disable this, then add the same command to an init script such as /etc/rc.local so it will be set upon system reboot. The following hosts are affected: 
mds-host05; mds-host[01-04]


Cloudera recommends setting /proc/sys/vm/swappiness to a maximum of 10. Current setting is 30. Use the sysctl command to change this setting at run time and edit /etc/sysctl.conf for this setting to be saved after a reboot. You can continue with installation, but Cloudera Manager might report that your hosts are unhealthy because they are swapping. The following hosts are affected: 
mds-host05; mds-host[01-04]


The following failures were observed in checking hostnames… 
Host mds-host01 expected to have name mds-host01 but resolved (InetAddress.getLocalHost().getHostName()) itself to mds-host01.mds.xyz.

 

Resolve these accordingly to above instructions and continue.  To set the swappiness run the following:

sysctl -w vm.swappiness=10

And continue with the installation.  If you get this error:

/usr/lib64/cmf/service/zookeeper/zkserver.sh: line 41: /var/lib/zookeeper/myid: Permission denied
Supervisor returned FATAL. Please check the role log file, stderr, or stdout.
Completed only 0/1 steps. First failure: Command (85) has failed
Failed to start role.
Completed only 1/2 steps. First failure: Failed to execute command Start on service ZooKeeper

Simply change permissions like this:

101947599 d———.  2 root root    6 Sep 14 23:52 /var/lib/zookeeper

chmod 755 /var/lib/zookeeper

101947599 drwxr-xr-x. 2 root root 6 Sep 14 23:52 /var/lib/zookeeper

And continue the install.  Possibly /usr/lib64/cmf/service/zookeeper/zkserver.sh might not be setting permissions correctly.  Also set the folder to zookeeper.zookeeper as well:

chmod 755 /var/lib/zookeeper; chown zookeeper.zookeeper /var/lib/zookeeper; ls -altrid /var/lib/zookeeper

And continue.  Also set the ACL's on the folder:

# setfacl -m "u:zookeeper:rwx,g:zookeeper:rwx" /var/lib/zookeeper/
# getfacl zookeeper
# file: zookeeper
# owner: zookeeper
# group: zookeeper
user::rwx
user:zookeeper:rwx
group::rwx
group:zookeeper:rwx
mask::rwx
other::rwx

#

Yet still did not work.  Digging deeper we see:

[root@mds-host01 zookeeper]# pwd
/var/log/zookeeper
[root@mds-host01 zookeeper]# tail -f zookeeper-cmf-zookeeper-SERVER-mds-host01.log -n 10
2016-09-15 01:48:59,395 INFO org.apache.zookeeper.server.ZooKeeperServer: Server environment:user.home=/var/lib/zookeeper
2016-09-15 01:48:59,396 INFO org.apache.zookeeper.server.ZooKeeperServer: Server environment:user.dir=/run/cloudera-scm-agent/process/31-zookeeper-server
2016-09-15 01:48:59,396 ERROR org.apache.zookeeper.server.ZooKeeperServerMain: Unable to access datadir, exiting abnormally
org.apache.zookeeper.server.persistence.FileTxnSnapLog$DatadirException: Missing data directory /var/lib/zookeeper/version-2, automatic data directory creation is disabled (zookeeper.datadir.autocreate is false). Please create this directory manually.
        at org.apache.zookeeper.server.persistence.FileTxnSnapLog.<init>(FileTxnSnapLog.java:102)
        at org.apache.zookeeper.server.ZooKeeperServerMain.runFromConfig(ZooKeeperServerMain.java:109)
        at org.apache.zookeeper.server.ZooKeeperServerMain.initializeAndRun(ZooKeeperServerMain.java:91)
        at org.apache.zookeeper.server.ZooKeeperServerMain.main(ZooKeeperServerMain.java:53)
        at org.apache.zookeeper.server.quorum.QuorumPeerMain.initializeAndRun(QuorumPeerMain.java:121)
        at org.apache.zookeeper.server.quorum.QuorumPeerMain.main(QuorumPeerMain.java:79)

 

So let's create it manually and retry the installation.  So command was:

chmod 755 /var/lib/zookeeper; chown zookeeper.zookeeper /var/lib/zookeeper; ls -altrid /var/lib/zookeeper; setfacl -m "u:zookeeper:rwx,g:zookeeper:rwx" /var/lib/zookeeper/; getfacl /var/lib/zookeeper/; mkdir /var/lib/zookeeper/version-2; chown zookeeper.zookeeper /var/lib/zookeeper/version-2;

This time above fixed the issue.  Retry and continue with installation. 

Cheers,
TK

Date and Time Configuration on RHEL 7, CentOS 7, Scientific Linux 7 or Fedora

yum install ntp -y; timedatectl set-ntp yes; timedatectl set-timezone America/Toronto

Use timedatectl list-timezones to get your timezone.

Cheers,
TK

Wrap lines in journalctl output

journalctl -b –no-pager | less

Easy peasy!

Cheers, TK


     
  Copyright © 2003 - 2013 Tom Kacperski (microdevsys.com). All rights reserved.

Creative Commons License
This work is licensed under a Creative Commons Attribution 3.0 Unported License