Header Shadow Image

Atlassian Confluence: Reducing GlusterFS IO, Disk Log Usage, and DEBUG Logging.

So it became apparent that, while sitting on a GlusterFS on two of my nodes, Confluence was dumping 18GB of logs to catalina.out. Unfortunately, there isn't a good way to rotate that file:


All the while writing to the GlusterFS, which by itself network copies this to the secondary host, atlas01:

atlas02 # du -sh logs
18G     logs

To fix this, created a folder on the host OS called /confluence-logs/logs, then copied the /atlas/atlassian/confluence/logs folder to the OS folder.  Then linked them up and changed permissions:

# atlas02 # /atlas/atlassian/confluence # ln -s /confluence-logs/ logs
# chown -h confluence.confluence logs
# ls -atlrid logs
11784356602385662682 lrwxrwxrwx. 1 confluence confluence 17 Nov  4 20:14 logs -> /confluence-logs/

Then started up confluence once more.  Don't forget to do this on the second note too!  The drawback to this, is that when nodes failover, the secondary host doesn't really have a copy of the logs anymore, since it won't be shared at that point.

Now you can enjoy Documentation Engineering in it's full glory!  


OpenVPN: Cannot ping or access internal VLAN’s

Seeing timed out accessing external and internal VLAN's after connecting to the OpenVPN server?

Reply from bytes=32 time=673ms TTL=47
Request timed out.
Request timed out.

Reply from bytes=32 time=673ms TTL=47
Request timed out.
Request timed out.

Moreover, also seeing timeout on accessing local VLAN's?

root@DD-WRT-INTERNET-ASUS:~# tail -f /var/log/messages|grep -Ei "DROP"|grep -Ei ""
Nov  4 00:06:16 DD-WRT-INTERNET-ASUS kern.warn kernel: DROP IN=tun2 OUT=br0 MAC= SRC= DST= LEN=60 TOS=0x00 PREC=0x00 TTL=127 ID=54730 PROTO=ICMP TYPE=8 CODE=0 ID=1 SEQ=6098
Nov  4 00:06:30 DD-WRT-INTERNET-ASUS kern.warn kernel: DROP IN=tun2 OUT=br0 MAC= SRC= DST= LEN=52 TOS=0x00 PREC=0x00 TTL=127 ID=54733 DF PROTO=TCP SPT=56718 DPT=22 SEQ=2130463582 ACK=0 WINDOW=64240 RES=0x00 SYN URGP=0 OPT (0204054B0103030801010402)

Chances are you're missing the following rules:

# VPN: Required to be able to ping local on-prem or Azure VLAN's
iptables -I FORWARD -i br0 -o tun2 -j ACCEPT
iptables -I FORWARD -i tun2 -o br0 -j ACCEPT
iptables -I INPUT -i tun2 -j LOGREJECT
iptables -t nat -A POSTROUTING -o tun2 -j MASQUERADE

iptables -I FORWARD -i br0 -o tun1 -j ACCEPT
iptables -I FORWARD -i tun1 -o br0 -j ACCEPT
iptables -I INPUT -i tun1 -j LOGREJECT
iptables -t nat -A POSTROUTING -o tun1 -j MASQUERADE

iptables -I FORWARD -i br0 -o tun0 -j ACCEPT
iptables -I FORWARD -i tun0 -o br0 -j ACCEPT
iptables -I INPUT -i tun0 -j LOGREJECT
iptables -t nat -A POSTROUTING -o tun0 -j MASQUERADE

Rules such as these, do not work:

# Allow TCP from tun2 (VPN)
iptables -A OUTPUT -s -p tcp -j ACCEPT
iptables -A INPUT  -s -p tcp -j ACCEPT

# Allow UDP from tun2 (VPN)
iptables -A OUTPUT -s -p udp -j ACCEPT
iptables -A INPUT  -s -p udp -j ACCEPT

# Allow ICMP from tun2 (VPN)
iptables -A OUTPUT -s -p icmp -j ACCEPT
iptables -A INPUT  -s -p icmp -j ACCEPT

Enjoy your new, shiny reponsive network!  🙂


OpenWRT: Disable invalid default gateway selection

It indeed happened that the default GW provided on various network interfaces was the router that we do not want to be the GW.  In our case the OpenWRT Raspberry Pi 2 became the GW for any hosts dynamically getting an IP.  So all requests, were sent via the Raspberry Pi 2, which is not what we want.  To fix this, check your device if it is running a DHCP server:

root@OWRT01:~# ps | grep -Ei dhcp
  547 root      1240 S    /usr/sbin/odhcpd
 1556 root      1072 S    grep -Ei dhcp

To disable this, you can do so in the Luci interfaces panel ( Luci -> Network -> Interfaces -> LAN ) then Edit then disable the DHCP server in the right most tab:


Save and restart the device.  


OpenWRT: Resolving the /etc/resolv.conf lack of proper DNS resolution.

OpenWRT links /etc/resolv.conf to /tmp/resolv.conf and only adjust entries in /tmp/resov.conf if $localuse is enabled in the UI:


The above entered as text, is:


Your connection is not private: Trusting your own LAB SelfSigned Certificates in Kaspersky, Windows and Chrome

This use case scenario is aimed at those folks who are developing on their local environment and need to trust a set of certificates. This is so they are not always prompted for verification to a domain they know is already trusted and safe.  Despite that site having self signed certificates as is the case in many labs. Here's how to suppress these for specific sites.

Your connection is not private

The steps below assume you are running on Windows 10 and using a non previlieged account.  As of this writing, Chrome appears to make use of it's own Trust Root Certificate Authorities which could not be updated using import in that category.

1) Chrome  ( First 4 steps may not work )

  • Export the certificate to a file by clicking the Lock or Not Secure text that may appear to the left of your URL. 
  • Select View Certificate -> Details tab then Copy to File... Then save the certificate.  Name the file something easily descernable to prevent confusion later on. https://i0.wp.com/www.microdevsys.com/WordPressImages/Trusting-Self-Signed-Certificates-Allow-In-Chrome.PNG?ssl=1
  • in chrome://settings, or using the three dot menu from the top right, search for SSL in the search field then select Security -> Manage Certificates .
  • Import your certificate under the 
  • If the above doesn't work, on the error page type thisisunsafe .to bypass the prompt in the future.  The site will still be marked as insecure however it will no longer prompt for a pass.

2) Kaspersky Total Security

  • Add the sites to the list of Trusted Addresses to bypass the above Kaspersky warning.

3) Enjoy!




Adjusting Memory in Atlassian Confluence

Adjusted the bolded lines for optimum startup and performance:

# pwd
# cat setenv.sh
# Set the JVM arguments used to start Confluence.
# For a description of the vm options of jdk 8, see:
# http://www.oracle.com/technetwork/java/javase/tech/vmoptions-jsp-140102.html
# For a description of the vm options of jdk 11, see:
# https://docs.oracle.com/en/java/javase/11/tools/java.html
CATALINA_OPTS="-XX:+IgnoreUnrecognizedVMOptions ${CATALINA_OPTS}"
CATALINA_OPTS="-XX:-PrintGCDetails -XX:+PrintGCDateStamps -XX:-PrintTenuringDistribution ${CATALINA_OPTS}"
CATALINA_OPTS="-Xlog:gc+age=debug:file=$LOGBASEABS/logs/gc-`date +%F_%H-%M-%S`.log::filecount=5,filesize=2M ${CATALINA_OPTS}"
CATALINA_OPTS="-Xloggc:$LOGBASEABS/logs/gc-`date +%F_%H-%M-%S`.log -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=5 -XX:GCLogFileSize=2M ${CATALINA_OPTS}"
CATALINA_OPTS="-Djava.awt.headless=true ${CATALINA_OPTS}"
CATALINA_OPTS="-Datlassian.plugins.enable.wait=300 ${CATALINA_OPTS}"
CATALINA_OPTS="-Xms1024m -Xmx6144m -XX:+UseG1GC ${CATALINA_OPTS}"
CATALINA_OPTS="-Dsynchrony.enable.xhr.fallback=true ${CATALINA_OPTS}"
CATALINA_OPTS="-Dorg.apache.tomcat.websocket.DEFAULT_BUFFER_SIZE=32768 ${CATALINA_OPTS}"
CATALINA_OPTS="-Djdk.tls.server.protocols=TLSv1.1,TLSv1.2 -Djdk.tls.client.protocols=TLSv1.1,TLSv1.2 ${CATALINA_OPTS}"
CATALINA_OPTS="-XX:ReservedCodeCacheSize=256m -XX:+UseCodeCacheFlushing ${CATALINA_OPTS}"


# ps -ef|grep -Ei confluence
conflue+ 22167     1 99 22:18 ?        00:00:35 /atlas/atlassian/confluence/jre//bin/java -Djava.util.logging.config.file=/atlas/atlassian/confluence/conf/logging.properties -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager -Djdk.tls.ephemeralDHKeySize=2048 -Djava.protocol.handler.pkgs=org.apache.catalina.webresources -Dorg.apache.catalina.security.SecurityListener.UMASK=0027 -XX:ReservedCodeCacheSize=256m -XX:+UseCodeCacheFlushing -Djdk.tls.server.protocols=TLSv1.1,TLSv1.2 -Djdk.tls.client.protocols=TLSv1.1,TLSv1.2 -Dconfluence.context.path= -Datlassian.plugins.startup.options= -Djava.locale.providers=JRE,SPI,CLDR -Dorg.apache.tomcat.websocket.DEFAULT_BUFFER_SIZE=32768 -Dsynchrony.enable.xhr.fallback=true -Xms1024m -Xmx6144m -XX:+UseG1GC -Datlassian.plugins.enable.wait=300 -Djava.awt.headless=true -XX:G1ReservePercent=20 -Xloggc:/atlas/atlassian/confluence/logs/gc-2021-07-13_22-18-48.log -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=5 -XX:GCLogFileSize=2M -Xlog:gc+age=debug:file=/atlas/atlassian/confluence/logs/gc-2021-07-13_22-18-48.log::filecount=5,filesize=2M -XX:-PrintGCDetails -XX:+PrintGCDateStamps -XX:-PrintTenuringDistribution -XX:+IgnoreUnrecognizedVMOptions -Dignore.endorsed.dirs= -classpath /atlas/atlassian/confluence/bin/bootstrap.jar:/atlas/atlassian/confluence/bin/tomcat-juli.jar -Dcatalina.base=/atlas/atlassian/confluence -Dcatalina.home=/atlas/atlassian/confluence -Djava.io.tmpdir=/atlas/atlassian/confluence/temp org.apache.catalina.startup.Bootstrap start
root     22391 12708  0 22:19 pts/0    00:00:00 grep –color=auto -Ei confluence
# pwd
# cat setenv.sh




ERR – dse_check_file – The backup file /etc/dirsrv/slapd-NIX-MDS-XYZ/dse.ldif.bak has zero length, refusing to restore it.

Recover the backup from the OK copy, literally:

/etc/dirsrv/slapd-NIX-MDS-XYZ# ls -altri
total 1904
     2076 -rw——-. 1 dirsrv root   197845 May 24  2020 dse.ldif.ipa.b22658eb606be0d2
   249372 -rw-r–r–. 1 dirsrv root   197954 May 24  2020 dse.ldif.modified.out
   130281 -rw——-. 1 dirsrv dirsrv 197835 Mar  7 15:50 dse.ldif.startOK
   456855 -rw——-. 1 dirsrv dirsrv      0 May 17 03:12 dse.ldif.bak-backup
/etc/dirsrv/slapd-NIX-MDS-XYZ# cp -ip dse.ldif.startOK dse.ldif.bak


/etc/dirsrv/slapd-NIX-MDS-XYZ#ipactl start
Starting Directory Service
Starting krb5kdc Service
Starting kadmin Service
Starting named Service
Starting httpd Service
Starting ipa-custodia Service
Starting ntpd Service
Starting pki-tomcatd Service
Starting smb Service
Starting winbind Service
Starting ipa-otpd Service
Starting ipa-dnskeysyncd Service
ipa: INFO: The ipactl command was successful


OpenVpn: Can’t browse web when connected and VPN is active.

Configured your OpenVPN, however now you can't browse the web when connected?  Turns out the following NAT rule was missing from the F/W configuration on the OpenVPN router:

iptables -t nat -I POSTROUTING -s -j SNAT --to $(nvram get wan_ipaddr)

Our VPN subnet is but no rule existed to route traffic to the WAN interface.  Meaning no external traffic was passed to the external IP address resulting in no web connectivity with the outside world.  The rule above fixes this.  In case you're wondering what nvram get wan_ipaddress does.  It is busy box ( DD-WRT ) custom firmware command used on various routers.  It is a replacement for the standard Web UI on routers.  If DD-WRT is not used, the default external IP of the router will do.


OpenShift w/ Kubernetes Setup: Installing using the UPI Method

Building an OpenShift Kubernetes Cluster. Method used here will be the UPI installation method.  Start off by loading the official page from RedHat:


Before you begin, ensure the following files are downloaded off the RedHat OpenShift pages (see links in the above document):

/root/openshift # ls -altri
total 439680
201572861 -rw-r–r–.  1 root        root              706 Apr 25 04:15 README.md
201572704 -rwxr-xr-x.  1 root        root        360710144 Apr 25 04:15 openshift-install
201572859 -rw-rw-r–.  1 tom@mds.xyz tom@mds.xyz      2775 May  8 22:53 pull-secret.txt
201572858 -rw-rw-r–.  1 tom@mds.xyz tom@mds.xyz  89491042 May  8 22:55 openshift-install-linux.tar.gz
201572850 drwxr-xr-x.  3 root        root             4096 May  8 23:58 .
201326721 dr-xr-x—. 12 root        root             4096 May  9 08:43 ..

Extract the .tar.gz using:

tar -zxf openshift-install-linux.tar.gz

Read the rest of this entry »

Firewalld. Add VLAN’s to allowed trusted / public zone rules.

Short list of commands for adding VLAN's to trusted zones:

firewall-cmd –zone=trusted –add-source=
firewall-cmd –zone=trusted –add-source=
firewall-cmd –zone=trusted –add-source=
firewall-cmd –zone=trusted –add-source=
firewall-cmd –zone=trusted –add-source=
cat /etc/firewalld/zones/public.xml
firewall-cmd –runtime-to-permanent
cat /etc/firewalld/zones/public.xml

Result of this is:

cat /etc/firewalld/zones/trusted.xml
<?xml version="1.0" encoding="utf-8"?>
<zone target="ACCEPT">
  <description>All network connections are accepted.</description>
  <source address=""/>
  <source address=""/>
  <source address=""/>
  <source address=""/>
  <source address=""/>



  Copyright © 2003 - 2013 Tom Kacperski (microdevsys.com). All rights reserved.

Creative Commons License
This work is licensed under a Creative Commons Attribution 3.0 Unported License