So you want to replace a GlustetFS brick? In other words, how do I rename a brick path in glusterfs?

NOTICE: Anytime there is any data manipulation on any environment, backups should always be taken.  So please make a copy or a backup of your files prior to executing any of the commands.  The steps expect the user has a backup.

Here's a few ways to do so:

gluster volume replace-brick <vol> <host>:/bricks/0/gv01     <host>:/bricks/0/abc-gv01  commit force

A detailed sequence of another way to rename the brick storage folder:

TKS

REF: https://docs.gluster.org/en/latest/Administrator-Guide/Managing-Volumes/#replace-brick

Getting this?

Got permission denied while trying to connect to the Docker daemon socket

Solve it with this:

sudo chmod 666 /var/run/docker.sock

Cheers,
Tom

To add a cname record: 

ipa dnsrecord-add nix.mds.xyz portal –cname-rec='long-host01.nix.mds.xyz.'

to remove a cname record: 

ipa dnsrecord-del nix.mds.xyz portal –cname-rec='long-host01.nix.mds.xyz.'

Cheers,
TK

Seeing the following alerts?

[2020-04-17 21:53:30,207] 2020-04-18T01:53:30.200Z Component 'Operating System' alerted 'Low free memory' (details: {"freeInMegabytes":62,"totalInMegabytes":5805,"minimumInMegabytes":256}, trigger: {"pluginKey": "not-detected"})
[2021-12-13 06:44:52,432] 2021-12-13T11:44:48.953Z Component ‘Java’ alerted ‘Garbage collection exceeded time limit’ (details: {“durationInMillis”:2181,”windowInMillis”:20000,”limitPercent”:10,”threadMemoryAllocations”:””,”threadDump”:[]}, trigger: {"pluginKey": "not-detected"})

Time to up the memory on the host and adjust the environment properties:

/atlas/atlassian/confluence/ cat bin/setenv.sh
# See the CATALINA_OPTS below for tuning the JVM arguments used to start Confluence.

echo "If you encounter issues starting up Confluence, please see the Installation guide at http://confluence.atlassian.com/display/DOC/Confluence+Installation+Guide"

……..

# Set the JVM arguments used to start Confluence.
# For a description of the vm options of jdk 8, see:
# http://www.oracle.com/technetwork/java/javase/tech/vmoptions-jsp-140102.html
# For a description of the vm options of jdk 11, see:
# https://docs.oracle.com/en/java/javase/11/tools/java.html
CATALINA_OPTS="-XX:+IgnoreUnrecognizedVMOptions ${CATALINA_OPTS}"
CATALINA_OPTS="-XX:-PrintGCDetails -XX:+PrintGCDateStamps -XX:-PrintTenuringDistribution ${CATALINA_OPTS}"
CATALINA_OPTS="-Xlog:gc+age=debug:file=$LOGBASEABS/logs/gc-`date +%F_%H-%M-%S`.log::filecount=5,filesize=2M ${CATALINA_OPTS}"
CATALINA_OPTS="-Xloggc:$LOGBASEABS/logs/gc-`date +%F_%H-%M-%S`.log -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=5 -XX:GCLogFileSize=2M ${CATALINA_OPTS}"
CATALINA_OPTS="-XX:G1ReservePercent=20 ${CATALINA_OPTS}"
CATALINA_OPTS="-Djava.awt.headless=true ${CATALINA_OPTS}"
CATALINA_OPTS="-Datlassian.plugins.enable.wait=300 ${CATALINA_OPTS}"
CATALINA_OPTS="-Xms1024m -Xmx6144m -XX:+UseG1GC ${CATALINA_OPTS}"
CATALINA_OPTS="-Dsynchrony.enable.xhr.fallback=true ${CATALINA_OPTS}"
CATALINA_OPTS="-Dorg.apache.tomcat.websocket.DEFAULT_BUFFER_SIZE=32768 ${CATALINA_OPTS}"
CATALINA_OPTS="-Djava.locale.providers=JRE,SPI,CLDR ${CATALINA_OPTS}"
CATALINA_OPTS="${START_CONFLUENCE_JAVA_OPTS} ${CATALINA_OPTS}"
CATALINA_OPTS="-Dconfluence.context.path=${CONFLUENCE_CONTEXT_PATH} ${CATALINA_OPTS}"
CATALINA_OPTS="-Djdk.tls.server.protocols=TLSv1.1,TLSv1.2 -Djdk.tls.client.protocols=TLSv1.1,TLSv1.2 ${CATALINA_OPTS}"
CATALINA_OPTS="-XX:ReservedCodeCacheSize=256m -XX:+UseCodeCacheFlushing ${CATALINA_OPTS}"

export CATALINA_OPTS

# /atlas/atlassian/confluence/

Upped the memory to double from 8GB to 16GB and min Xms settings to 4096GB for faster startup and adjusted the following line accordingly:

CATALINA_OPTS="-Xms4g -Xmx14g -XX:+UseG1GC ${CATALINA_OPTS} -XX:MaxGCPauseMillis=100 -XX:+AlwaysPreTouch -XX:ParallelGCThreads=4 -XX:+UnlockExperimen talVMOptions -XX:ActiveProcessorCount=8 -XX:+DisableExplicitGC -XX:TargetSurvivorRatio=90 -XX:G1NewSizePercent=50 -XX:G1MaxNewSizePercent=80 -XX:G1 MixedGCLiveThresholdPercent=50"

Also added the following from the Minecraft Servers:

Cheers,
TK

MySQL Galera while using HAProxy gives the following:

#
Dec 11 22:45:43 localhost haproxy[24265]: Proxy mysql-back started.
Dec 11 22:45:43 localhost haproxy[24265]: Server mysql-back/mysql01.nix.mds.xyz is DOWN, reason: Layer4 connection problem, info: "Connection refused", check duration: 0ms. 2 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
Dec 11 22:45:43 localhost haproxy[24265]: Server mysql-back/mysql01.nix.mds.xyz is DOWN, reason: Layer4 connection problem, info: "Connection refused", check duration: 0ms. 2 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
Dec 11 22:45:44 localhost haproxy[24271]: Server mysql-back/mysql02.nix.mds.xyz is DOWN, reason: Layer4 connection problem, info: "Connection refused", check duration: 0ms. 1 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
Dec 11 22:45:44 localhost haproxy[24271]: Server mysql-back/mysql02.nix.mds.xyz is DOWN, reason: Layer4 connection problem, info: "Connection refused", check duration: 0ms. 1 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
Dec 11 22:45:45 localhost haproxy[24271]: Server mysql-back/mysql03.nix.mds.xyz is DOWN, reason: Layer4 connection problem, info: "Connection refused", check duration: 0ms. 0 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
Dec 11 22:45:45 localhost haproxy[24271]: backend mysql-back has no server available!
Dec 11 22:45:45 localhost haproxy[24271]: Server mysql-back/mysql03.nix.mds.xyz is DOWN, reason: Layer4 connection problem, info: "Connection refused", check duration: 0ms. 0 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
Dec 11 22:45:45 localhost haproxy[24271]: backend mysql-back has no server available!
#
 

With the following configuration:

# cat /etc/haproxy/haproxy.cfg
global
    log         127.0.0.1 local0 debug
    stats       socket /var/run/haproxy.sock mode 0600 level admin
    # stats     socket /var/lib/haproxy/stats
    maxconn     4000
    user        haproxy
    group       haproxy
    daemon
    debug

defaults
    mode                    tcp
    log                     global
    option                  dontlognull
    option                  redispatch
    retries                 3
    timeout http-request    10s
    timeout queue           1m
    timeout connect         10s
    timeout client          1m
    timeout server          1m
    timeout http-keep-alive 10s
    timeout check           10s
    maxconn                 3000

listen stats
    bind :9000
    mode http
    stats enable
    stats hide-version
    stats realm Haproxy\ Statistics
    stats uri /haproxy-stats
    stats auth admin:somepass

frontend mysql-in
    mode tcp
    bind mysql-c01:3306
    option tcplog
    default_backend             mysql-back

backend mysql-back
    mode        tcp
    option      tcplog
    option      mysql-check user haproxy
    balance     roundrobin
    default-server inter 3s fall 3 rise 2 on-marked-down shutdown-sessions
    server      mysql01.nix.mds.xyz    mysql01.nix.mds.xyz:3306 maxconn 1024 check port 3306
    server      mysql02.nix.mds.xyz    mysql02.nix.mds.xyz:3306 maxconn 1024 check port 3306
    server      mysql03.nix.mds.xyz    mysql03.nix.mds.xyz:3306 maxconn 1024 check port 3306

HAProxy stats are red:

Checking using tcpdump;

tcpdump -w trace.dat -s 0 port not 22
tcpdump -r trace.dat -nnvvveXXS > trace.dat.txt

reveals these messages:

22:54:53.222232 00:50:56:86:da:36 > 00:50:56:86:b0:e5, ethertype IPv6 (0x86dd), length 94: (hlim 64, next-header TCP (6) payload length: 40) fdc8:29db:a9ed:0:250:56ff:fe86:da36.43910 > fdc8:29db:a9ed:0:250:56ff:fe86:b0e5.3306: Flags [S], cksum 0xde1a (incorrect -> 0x68ac), seq 2474721840, win 28800, options [mss 1440,sackOK,TS val 2107016514 ecr 0,nop,wscale 7], length 0
        0x0000:  0050 5686 b0e5 0050 5686 da36 86dd 6000  .PV….PV..6..`.
        0x0010:  0000 0028 0640 fdc8 29db a9ed 0000 0250  …(.@..)……P
        0x0020:  56ff fe86 da36 fdc8 29db a9ed 0000 0250  V….6..)……P
        0x0030:  56ff fe86 b0e5 ab86 0cea 9381 4230 0000  V………..B0..
        0x0040:  0000 a002 7080 de1a 0000 0204 05a0 0402  ….p………..
        0x0050:  080a 7d96 8542 0000 0000 0103 0307       ..}..B……..

Which indicates that HAProxy is attempting to make the connections using IPv6.  Further confirmed through:

# nc -vz4 mysql03 3306
Ncat: Version 7.50 ( https://nmap.org/ncat )
Ncat: Connected to 192.168.0.114:3306.
Ncat: 0 bytes sent, 0 bytes received in 0.02 seconds.
# nc -vz6 mysql03 3306
Ncat: Version 7.50 ( https://nmap.org/ncat )
Ncat: Connection refused.

In this case, turning off IPv6 resolved the issue, since IPv6 is not configured on MySQL nor through Galera either.

cat /etc/default/grub
GRUB_TIMEOUT=5
GRUB_DISTRIBUTOR="$(sed 's, release .*$,,g' /etc/system-release)"
GRUB_DEFAULT=saved
GRUB_DISABLE_SUBMENU=true
GRUB_TERMINAL_OUTPUT="console"
GRUB_CMDLINE_LINUX="ipv6.disable=1 crashkernel=auto rd.lvm.lv=centos/root rd.lvm.lv=centos/swap rhgb quiet"
GRUB_DISABLE_RECOVERY="true"

Then running:

grub2-mkconfig -o /boot/grub2/grub.cfg

To recompile the kernel.  Likewise, disable using /etc/sysctl.conf:

net.ipv6.conf.all.disable_ipv6 = 1
net.ipv6.conf.default.disable_ipv6 = 1

And run:

sysctl -p

This resulted in a few additional messages:

Dec 11 23:57:00 localhost haproxy[6635]: Server mysql-back/mysql01.nix.mds.xyz is DOWN, reason: Layer7 wrong status, code: 0, info: "Access denied for user 'haproxy'@'mysql01.nix.mds.xyz' (using password: NO)", check duration: 1ms. 2 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
Dec 11 23:57:00 localhost haproxy[6635]: Server mysql-back/mysql01.nix.mds.xyz is DOWN, reason: Layer7 wrong status, code: 0, info: "Access denied for user 'haproxy'@'mysql01.nix.mds.xyz' (using password: NO)", check duration: 1ms. 2 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
Dec 11 23:57:01 localhost haproxy[6640]: Server mysql-back/mysql02.nix.mds.xyz is DOWN, reason: Layer7 wrong status, code: 0, info: "Access denied for user 'haproxy'@'mysql01.nix.mds.xyz' (using password: NO)", check duration: 4ms. 1 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
Dec 11 23:57:01 localhost haproxy[6640]: Server mysql-back/mysql02.nix.mds.xyz is DOWN, reason: Layer7 wrong status, code: 0, info: "Access denied for user 'haproxy'@'mysql01.nix.mds.xyz' (using password: NO)", check duration: 4ms. 1 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
Dec 11 23:57:02 localhost haproxy[6640]: Server mysql-back/mysql03.nix.mds.xyz is DOWN, reason: Layer7 wrong status, code: 0, info: "Access denied for user 'haproxy'@'mysql01.nix.mds.xyz' (using password: NO)", check duration: 4ms. 0 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
Dec 11 23:57:02 localhost haproxy[6640]: backend mysql-back has no server available!
Dec 11 23:57:02 localhost haproxy[6640]: Server mysql-back/mysql03.nix.mds.xyz is DOWN, reason: Layer7 wrong status, code: 0, info: "Access denied for user 'haproxy'@'mysql01.nix.mds.xyz' (using password: NO)", check duration: 4ms. 0 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
Dec 11 23:57:02 localhost haproxy[6640]: backend mysql-back has no server available!

Which was temporary:

Dec 11 23:58:01 localhost haproxy[9156]: 192.168.0.104:59906 [11/Dec/2021:23:57:01.888] mysql-in mysql-back/mysql01.nix.mds.xyz 1/0/60009 126 cD 22/22/22/7/0 0/0
Dec 11 23:58:30 localhost haproxy[9156]: 192.168.0.104:59930 [11/Dec/2021:23:57:02.161] mysql-in mysql-back/mysql02.nix.mds.xyz 1/1/88075 274 cD 20/20/20/7/0 0/0
Dec 11 23:58:33 localhost haproxy[9156]: 192.168.0.104:59928 [11/Dec/2021:23:57:02.161] mysql-in mysql-back/mysql03.nix.mds.xyz 1/1/91445 200 cD 20/20/20/6/0 0/0

Status is now green:

Enjoy the Fix!

So it became apparent that, while sitting on a GlusterFS on two of my nodes, Confluence was dumping 18GB of logs to catalina.out. Unfortunately, there isn't a good way to rotate that file:

https://confluence.atlassian.com/confkb/catalina-logs-are-not-rotated-or-removed-289276264.html

All the while writing to the GlusterFS, which by itself network copies this to the secondary host, atlas01:

atlas02 # du -sh logs
18G     logs

To fix this, created a folder on the host OS called /confluence-logs/logs, then copied the /atlas/atlassian/confluence/logs folder to the OS folder.  Then linked them up and changed permissions:

# atlas02 # /atlas/atlassian/confluence # ln -s /confluence-logs/ logs
# chown -h confluence.confluence logs
# ls -atlrid logs
11784356602385662682 lrwxrwxrwx. 1 confluence confluence 17 Nov  4 20:14 logs -> /confluence-logs/
#

Then started up confluence once more.  Don't forget to do this on the second note too!  The drawback to this, is that when nodes failover, the secondary host doesn't really have a copy of the logs anymore, since it won't be shared at that point.

NOTE:

Additional SQL Logging can be enabled via the UI.

Sum It Up!

Enjoy faster performance!  

Cheers,
 

Seeing timed out accessing external and internal VLAN's after connecting to the OpenVPN server?

Reply from 98.136.103.23: bytes=32 time=673ms TTL=47
Request timed out.
Request timed out.

Reply from 10.3.0.100: bytes=32 time=673ms TTL=47
Request timed out.
Request timed out.

Moreover, also seeing timeout on accessing local VLAN's?

root@DD-WRT-INTERNET-ASUS:~# tail -f /var/log/messages|grep -Ei "DROP"|grep -Ei "10.3.0.100"
Nov  4 00:06:16 DD-WRT-INTERNET-ASUS kern.warn kernel: DROP IN=tun2 OUT=br0 MAC= SRC=10.1.1.2 DST=10.3.0.101 LEN=60 TOS=0x00 PREC=0x00 TTL=127 ID=54730 PROTO=ICMP TYPE=8 CODE=0 ID=1 SEQ=6098
Nov  4 00:06:30 DD-WRT-INTERNET-ASUS kern.warn kernel: DROP IN=tun2 OUT=br0 MAC= SRC=10.1.1.2 DST=10.3.0.1001 LEN=52 TOS=0x00 PREC=0x00 TTL=127 ID=54733 DF PROTO=TCP SPT=56718 DPT=22 SEQ=2130463582 ACK=0 WINDOW=64240 RES=0x00 SYN URGP=0 OPT (0204054B0103030801010402)

Chances are you're missing the following rules:

# VPN: Required to be able to ping local on-prem or Azure VLAN's
iptables -I FORWARD -i br0 -o tun2 -j ACCEPT
iptables -I FORWARD -i tun2 -o br0 -j ACCEPT
iptables -I INPUT -i tun2 -j LOGREJECT
iptables -t nat -A POSTROUTING -o tun2 -j MASQUERADE

iptables -I FORWARD -i br0 -o tun1 -j ACCEPT
iptables -I FORWARD -i tun1 -o br0 -j ACCEPT
iptables -I INPUT -i tun1 -j LOGREJECT
iptables -t nat -A POSTROUTING -o tun1 -j MASQUERADE

iptables -I FORWARD -i br0 -o tun0 -j ACCEPT
iptables -I FORWARD -i tun0 -o br0 -j ACCEPT
iptables -I INPUT -i tun0 -j LOGREJECT
iptables -t nat -A POSTROUTING -o tun0 -j MASQUERADE

Rules such as these, do not work:

# Allow TCP from tun2 (VPN)
iptables -A OUTPUT -s 10.1.1.0/24 -p tcp -j ACCEPT
iptables -A INPUT  -s 10.1.1.0/24 -p tcp -j ACCEPT

# Allow UDP from tun2 (VPN)
iptables -A OUTPUT -s 10.1.1.0/24 -p udp -j ACCEPT
iptables -A INPUT  -s 10.1.1.0/24 -p udp -j ACCEPT

# Allow ICMP from tun2 (VPN)
iptables -A OUTPUT -s 10.1.1.0/24 -p icmp -j ACCEPT
iptables -A INPUT  -s 10.1.1.0/24 -p icmp -j ACCEPT

Enjoy your new, shiny reponsive network!  🙂

Cheers,
Admin

It indeed happened that the default GW provided on various network interfaces was the router that we do not want to be the GW.  In our case the OpenWRT Raspberry Pi 2 became the GW for any hosts dynamically getting an IP.  So all requests, were sent via the Raspberry Pi 2, which is not what we want.  To fix this, check your device if it is running a DHCP server:

root@OWRT01:~# ps | grep -Ei dhcp
  547 root      1240 S    /usr/sbin/odhcpd
 1556 root      1072 S    grep -Ei dhcp
root@OWRT01:~#

To disable this, you can do so in the Luci interfaces panel ( Luci -> Network -> Interfaces -> LAN ) then Edit then disable the DHCP server in the right most tab:

Save and restart the device.  

Cheers,
Tom

OpenWRT links /etc/resolv.conf to /tmp/resolv.conf and only adjust entries in /tmp/resov.conf if $localuse is enabled in the UI:

The above entered as text, is:

DHCP and DNS -> [General Settings] -> [Local domain] : openwrt.mds.xyz mds.xyz nix.mds.xyz mws.mds.xyz

DHCP and DNS -> [General Settings] -> Allow localhost : Yes

DHCP and DNS -> [General Settings] -> Local Service Only : Yes
 

Cheers,
AD

This use case scenario is aimed at those folks who are developing on their local environment and need to trust a set of certificates. This is so they are not always prompted for verification to a domain they know is already trusted and safe.  Despite that site having self signed certificates as is the case in many labs. Here's how to suppress these for specific sites.

Your connection is not private

The steps below assume you are running on Windows 10 and using a non previlieged account.  As of this writing, Chrome appears to make use of it's own Trust Root Certificate Authorities which could not be updated using import in that category.

1) Chrome  ( First 4 steps may not work )

• Export the certificate to a file by clicking the Lock or Not Secure text that may appear to the left of your URL. 
• Select View Certificate -> Details tab then Copy to File... Then save the certificate.  Name the file something easily descernable to prevent confusion later on. 
• in chrome://settings, or using the three dot menu from the top right, search for SSL in the search field then select Security -> Manage Certificates .
• Import your certificate under the 
• If the above doesn't work, on the error page type thisisunsafe .to bypass the prompt in the future.  The site will still be marked as insecure however it will no longer prompt for a pass.

2) Kaspersky Total Security

• Add the sites to the list of Trusted Addresses to bypass the above Kaspersky warning.
  

3) Enjoy!

TK