Header Shadow Image


DD-WRT WiFi: Slow connection or no connectivity at all.

When running DD-WRT firmware, slow speeds are seen or there is no connectivity?  Try to disable the Frame Burst rate in Wireless -> wl1-Advanced .  According to the definition, this can help with multiple devices.

CTS Protection Mode
The default value is Disabled. When set to Auto, a protection mechanism will ensure that your Wireless-B devices will connect to the Wireless-G router when many Wireless-G devices are present. However, performance of your Wireless-G devices may be decreased.
 
The default value is Disabled. Frame burst allows packet bursting which will increase overall network speed though this is only recommended for approx 1-3 wireless clients, Anymore clients and there can be a negative result and throughput will be affected.

 HS

journalctl: wrap lines and tail effect

Trying to use journalctl to view logs but the output get's truncated or doesn't wrap around.  Using the following for me had best results in my case:

journalctl -xe -u kubelet.service -f

Other variations with slightly different results:

journalctl -xe -u kubelet.service -fn100

Another variant that can be usefull includes:

journalctl -xe -u kubelet.service | more

Likewise, the following:

journalctl -xe -u kubelet.service | less

Cheers,

GlusterFS: Renaming a Brick

So you want to replace a GlustetFS brick? In other words, how do I rename a brick path in glusterfs?

NOTICE: Anytime there is any data manipulation on any environment, backups should always be taken.  So please make a copy or a backup of your files prior to executing any of the commands.  The steps expect the user has a backup.

Here's a few ways to do so:

gluster volume replace-brick <vol> <host>:/bricks/0/gv01     <host>:/bricks/0/abc-gv01  commit force

A detailed sequence of another way to rename the brick storage folder:

Command Description
gluster v reset-brick gvol peer01:/previous/path start Stop glusterfs on node.
mv  /previous/path /latest/path      Move the old path to the new path.
gluster v add-brick gvol peer02:/new/path force Add new path as new brick.
gluster v remove-brick gvol peer01:/previous/path force Maintain sequence. Data duplication will occur if order is not maintained.
gluster v rebalance gvol fix-layout  start Optional Command.  Might not be needed.

 

TKS

REF: https://docs.gluster.org/en/latest/Administrator-Guide/Managing-Volumes/#replace-brick

 

Got permission denied while trying to connect to the Docker daemon socket

Getting this?

Got permission denied while trying to connect to the Docker daemon socket

Solve it with this:

sudo chmod 666 /var/run/docker.sock

Cheers,
Tom

FreeIPA: Adding and removing CNAME records.

To add a cname record: 

ipa dnsrecord-add nix.mds.xyz portal –cname-rec='long-host01.nix.mds.xyz.'

to remove a cname record: 

ipa dnsrecord-del nix.mds.xyz portal –cname-rec='long-host01.nix.mds.xyz.'

Cheers,
TK

Atlassian Confluence:Component ‘Operating System’ alerted ‘Low free memory’ and Component ‘Java’ alerted ‘Garbage collection exceeded time limit’

Seeing the following alerts?

[2020-04-17 21:53:30,207] 2020-04-18T01:53:30.200Z Component 'Operating System' alerted 'Low free memory' (details: {"freeInMegabytes":62,"totalInMegabytes":5805,"minimumInMegabytes":256}, trigger: {"pluginKey": "not-detected"})
[2021-12-13 06:44:52,432] 2021-12-13T11:44:48.953Z Component ‘Java’ alerted ‘Garbage collection exceeded time limit’ (details: {“durationInMillis”:2181,”windowInMillis”:20000,”limitPercent”:10,”threadMemoryAllocations”:””,”threadDump”:[]}, trigger: {"pluginKey": "not-detected"})

Time to up the memory on the host and adjust the environment properties:

/atlas/atlassian/confluence/ cat bin/setenv.sh
# See the CATALINA_OPTS below for tuning the JVM arguments used to start Confluence.

echo "If you encounter issues starting up Confluence, please see the Installation guide at http://confluence.atlassian.com/display/DOC/Confluence+Installation+Guide"

……..

# Set the JVM arguments used to start Confluence.
# For a description of the vm options of jdk 8, see:
# http://www.oracle.com/technetwork/java/javase/tech/vmoptions-jsp-140102.html
# For a description of the vm options of jdk 11, see:
# https://docs.oracle.com/en/java/javase/11/tools/java.html
CATALINA_OPTS="-XX:+IgnoreUnrecognizedVMOptions ${CATALINA_OPTS}"
CATALINA_OPTS="-XX:-PrintGCDetails -XX:+PrintGCDateStamps -XX:-PrintTenuringDistribution ${CATALINA_OPTS}"
CATALINA_OPTS="-Xlog:gc+age=debug:file=$LOGBASEABS/logs/gc-`date +%F_%H-%M-%S`.log::filecount=5,filesize=2M ${CATALINA_OPTS}"
CATALINA_OPTS="-Xloggc:$LOGBASEABS/logs/gc-`date +%F_%H-%M-%S`.log -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=5 -XX:GCLogFileSize=2M ${CATALINA_OPTS}"
CATALINA_OPTS="-XX:G1ReservePercent=20 ${CATALINA_OPTS}"
CATALINA_OPTS="-Djava.awt.headless=true ${CATALINA_OPTS}"
CATALINA_OPTS="-Datlassian.plugins.enable.wait=300 ${CATALINA_OPTS}"
CATALINA_OPTS="-Xms1024m -Xmx6144m -XX:+UseG1GC ${CATALINA_OPTS}"
CATALINA_OPTS="-Dsynchrony.enable.xhr.fallback=true ${CATALINA_OPTS}"
CATALINA_OPTS="-Dorg.apache.tomcat.websocket.DEFAULT_BUFFER_SIZE=32768 ${CATALINA_OPTS}"
CATALINA_OPTS="-Djava.locale.providers=JRE,SPI,CLDR ${CATALINA_OPTS}"
CATALINA_OPTS="${START_CONFLUENCE_JAVA_OPTS} ${CATALINA_OPTS}"
CATALINA_OPTS="-Dconfluence.context.path=${CONFLUENCE_CONTEXT_PATH} ${CATALINA_OPTS}"
CATALINA_OPTS="-Djdk.tls.server.protocols=TLSv1.1,TLSv1.2 -Djdk.tls.client.protocols=TLSv1.1,TLSv1.2 ${CATALINA_OPTS}"
CATALINA_OPTS="-XX:ReservedCodeCacheSize=256m -XX:+UseCodeCacheFlushing ${CATALINA_OPTS}"

export CATALINA_OPTS

# /atlas/atlassian/confluence/

Upped the memory to double from 8GB to 16GB and min Xms settings to 4096GB for faster startup and adjusted the following line accordingly:

CATALINA_OPTS="-Xms4g -Xmx14g -XX:+UseG1GC ${CATALINA_OPTS} -XX:MaxGCPauseMillis=100 -XX:+AlwaysPreTouch -XX:ParallelGCThreads=4 -XX:+UnlockExperimen talVMOptions -XX:ActiveProcessorCount=8 -XX:+DisableExplicitGC -XX:TargetSurvivorRatio=90 -XX:G1NewSizePercent=50 -XX:G1MaxNewSizePercent=80 -XX:G1 MixedGCLiveThresholdPercent=50"

Also added the following from the Minecraft Servers:

 

Cheers,
TK

MySQL Galera while using HAProxy prints following error: DOWN, reason: Layer4 connection problem, info: Connection refused

MySQL Galera while using HAProxy gives the following:

#
Dec 11 22:45:43 localhost haproxy[24265]: Proxy mysql-back started.
Dec 11 22:45:43 localhost haproxy[24265]: Server mysql-back/mysql01.nix.mds.xyz is DOWN, reason: Layer4 connection problem, info: "Connection refused", check duration: 0ms. 2 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
Dec 11 22:45:43 localhost haproxy[24265]: Server mysql-back/mysql01.nix.mds.xyz is DOWN, reason: Layer4 connection problem, info: "Connection refused", check duration: 0ms. 2 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
Dec 11 22:45:44 localhost haproxy[24271]: Server mysql-back/mysql02.nix.mds.xyz is DOWN, reason: Layer4 connection problem, info: "Connection refused", check duration: 0ms. 1 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
Dec 11 22:45:44 localhost haproxy[24271]: Server mysql-back/mysql02.nix.mds.xyz is DOWN, reason: Layer4 connection problem, info: "Connection refused", check duration: 0ms. 1 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
Dec 11 22:45:45 localhost haproxy[24271]: Server mysql-back/mysql03.nix.mds.xyz is DOWN, reason: Layer4 connection problem, info: "Connection refused", check duration: 0ms. 0 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
Dec 11 22:45:45 localhost haproxy[24271]: backend mysql-back has no server available!
Dec 11 22:45:45 localhost haproxy[24271]: Server mysql-back/mysql03.nix.mds.xyz is DOWN, reason: Layer4 connection problem, info: "Connection refused", check duration: 0ms. 0 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
Dec 11 22:45:45 localhost haproxy[24271]: backend mysql-back has no server available!
#

 

With the following configuration:

# cat /etc/haproxy/haproxy.cfg
global
    log         127.0.0.1 local0 debug
    stats       socket /var/run/haproxy.sock mode 0600 level admin
    # stats     socket /var/lib/haproxy/stats
    maxconn     4000
    user        haproxy
    group       haproxy
    daemon
    debug

defaults
    mode                    tcp
    log                     global
    option                  dontlognull
    option                  redispatch
    retries                 3
    timeout http-request    10s
    timeout queue           1m
    timeout connect         10s
    timeout client          1m
    timeout server          1m
    timeout http-keep-alive 10s
    timeout check           10s
    maxconn                 3000

listen stats
    bind :9000
    mode http
    stats enable
    stats hide-version
    stats realm Haproxy\ Statistics
    stats uri /haproxy-stats
    stats auth admin:somepass

frontend mysql-in
    mode tcp
    bind mysql-c01:3306
    option tcplog
    default_backend             mysql-back


backend mysql-back
    mode        tcp
    option      tcplog
    option      mysql-check user haproxy
    balance     roundrobin
    default-server inter 3s fall 3 rise 2 on-marked-down shutdown-sessions
    server      mysql01.nix.mds.xyz    mysql01.nix.mds.xyz:3306 maxconn 1024 check port 3306
    server      mysql02.nix.mds.xyz    mysql02.nix.mds.xyz:3306 maxconn 1024 check port 3306
    server      mysql03.nix.mds.xyz    mysql03.nix.mds.xyz:3306 maxconn 1024 check port 3306

HAProxy stats are red:

https://i1.wp.com/www.microdevsys.com/WordPressImages/MySQL-Galera-Connection-Refused-Haproxy-Red-Stats.PNG?ssl=1

Checking using tcpdump;

tcpdump -w trace.dat -s 0 port not 22
tcpdump -r trace.dat -nnvvveXXS > trace.dat.txt

reveals these messages:

22:54:53.222232 00:50:56:86:da:36 > 00:50:56:86:b0:e5, ethertype IPv6 (0x86dd), length 94: (hlim 64, next-header TCP (6) payload length: 40) fdc8:29db:a9ed:0:250:56ff:fe86:da36.43910 > fdc8:29db:a9ed:0:250:56ff:fe86:b0e5.3306: Flags [S], cksum 0xde1a (incorrect -> 0x68ac), seq 2474721840, win 28800, options [mss 1440,sackOK,TS val 2107016514 ecr 0,nop,wscale 7], length 0
        0x0000:  0050 5686 b0e5 0050 5686 da36 86dd 6000  .PV….PV..6..`.
        0x0010:  0000 0028 0640 fdc8 29db a9ed 0000 0250  …(.@..)……P
        0x0020:  56ff fe86 da36 fdc8 29db a9ed 0000 0250  V….6..)……P
        0x0030:  56ff fe86 b0e5 ab86 0cea 9381 4230 0000  V………..B0..
        0x0040:  0000 a002 7080 de1a 0000 0204 05a0 0402  ….p………..
        0x0050:  080a 7d96 8542 0000 0000 0103 0307       ..}..B……..

Which indicates that HAProxy is attempting to make the connections using IPv6.  Further confirmed through:

# nc -vz4 mysql03 3306
Ncat: Version 7.50 ( https://nmap.org/ncat )
Ncat: Connected to 192.168.0.114:3306.
Ncat: 0 bytes sent, 0 bytes received in 0.02 seconds.
# nc -vz6 mysql03 3306
Ncat: Version 7.50 ( https://nmap.org/ncat )
Ncat: Connection refused.

In this case, turning off IPv6 resolved the issue, since IPv6 is not configured on MySQL nor through Galera either.

cat /etc/default/grub
GRUB_TIMEOUT=5
GRUB_DISTRIBUTOR="$(sed 's, release .*$,,g' /etc/system-release)"
GRUB_DEFAULT=saved
GRUB_DISABLE_SUBMENU=true
GRUB_TERMINAL_OUTPUT="console"
GRUB_CMDLINE_LINUX="ipv6.disable=1 crashkernel=auto rd.lvm.lv=centos/root rd.lvm.lv=centos/swap rhgb quiet"
GRUB_DISABLE_RECOVERY="true"

Then running:

grub2-mkconfig -o /boot/grub2/grub.cfg

To recompile the kernel.  Likewise, disable using /etc/sysctl.conf:

net.ipv6.conf.all.disable_ipv6 = 1
net.ipv6.conf.default.disable_ipv6 = 1

And run:

sysctl -p

This resulted in a few additional messages:

Dec 11 23:57:00 localhost haproxy[6635]: Server mysql-back/mysql01.nix.mds.xyz is DOWN, reason: Layer7 wrong status, code: 0, info: "Access denied for user 'haproxy'@'mysql01.nix.mds.xyz' (using password: NO)", check duration: 1ms. 2 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
Dec 11 23:57:00 localhost haproxy[6635]: Server mysql-back/mysql01.nix.mds.xyz is DOWN, reason: Layer7 wrong status, code: 0, info: "Access denied for user 'haproxy'@'mysql01.nix.mds.xyz' (using password: NO)", check duration: 1ms. 2 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
Dec 11 23:57:01 localhost haproxy[6640]: Server mysql-back/mysql02.nix.mds.xyz is DOWN, reason: Layer7 wrong status, code: 0, info: "Access denied for user 'haproxy'@'mysql01.nix.mds.xyz' (using password: NO)", check duration: 4ms. 1 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
Dec 11 23:57:01 localhost haproxy[6640]: Server mysql-back/mysql02.nix.mds.xyz is DOWN, reason: Layer7 wrong status, code: 0, info: "Access denied for user 'haproxy'@'mysql01.nix.mds.xyz' (using password: NO)", check duration: 4ms. 1 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
Dec 11 23:57:02 localhost haproxy[6640]: Server mysql-back/mysql03.nix.mds.xyz is DOWN, reason: Layer7 wrong status, code: 0, info: "Access denied for user 'haproxy'@'mysql01.nix.mds.xyz' (using password: NO)", check duration: 4ms. 0 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
Dec 11 23:57:02 localhost haproxy[6640]: backend mysql-back has no server available!
Dec 11 23:57:02 localhost haproxy[6640]: Server mysql-back/mysql03.nix.mds.xyz is DOWN, reason: Layer7 wrong status, code: 0, info: "Access denied for user 'haproxy'@'mysql01.nix.mds.xyz' (using password: NO)", check duration: 4ms. 0 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
Dec 11 23:57:02 localhost haproxy[6640]: backend mysql-back has no server available!

Which was temporary:

Dec 11 23:58:01 localhost haproxy[9156]: 192.168.0.104:59906 [11/Dec/2021:23:57:01.888] mysql-in mysql-back/mysql01.nix.mds.xyz 1/0/60009 126 cD 22/22/22/7/0 0/0
Dec 11 23:58:30 localhost haproxy[9156]: 192.168.0.104:59930 [11/Dec/2021:23:57:02.161] mysql-in mysql-back/mysql02.nix.mds.xyz 1/1/88075 274 cD 20/20/20/7/0 0/0
Dec 11 23:58:33 localhost haproxy[9156]: 192.168.0.104:59928 [11/Dec/2021:23:57:02.161] mysql-in mysql-back/mysql03.nix.mds.xyz 1/1/91445 200 cD 20/20/20/6/0 0/0

Status is now green:

https://i2.wp.com/www.microdevsys.com/WordPressImages/MySQL-Galera-Connection-Refused-Haproxy-Green-Stats.PNG?ssl=1

Enjoy the Fix!

Atlassian Confluence: Reducing GlusterFS IO, Disk Log Usage, and DEBUG Logging.

So it became apparent that, while sitting on a GlusterFS on two of my nodes, Confluence was dumping 18GB of logs to catalina.out. Unfortunately, there isn't a good way to rotate that file:

https://confluence.atlassian.com/confkb/catalina-logs-are-not-rotated-or-removed-289276264.html

All the while writing to the GlusterFS, which by itself network copies this to the secondary host, atlas01:

atlas02 # du -sh logs
18G     logs

To fix this, created a folder on the host OS called /confluence-logs/logs, then copied the /atlas/atlassian/confluence/logs folder to the OS folder.  Then linked them up and changed permissions:

# atlas02 # /atlas/atlassian/confluence # ln -s /confluence-logs/ logs
# chown -h confluence.confluence logs
# ls -atlrid logs
11784356602385662682 lrwxrwxrwx. 1 confluence confluence 17 Nov  4 20:14 logs -> /confluence-logs/
#

Then started up confluence once more.  Don't forget to do this on the second note too!  The drawback to this, is that when nodes failover, the secondary host doesn't really have a copy of the logs anymore, since it won't be shared at that point.

NOTE:

Additional SQL Logging can be enabled via the UI.

Sum It Up!

Enjoy faster performance!  

Cheers,
 

OpenVPN: Cannot ping or access internal VLAN’s

Seeing timed out accessing external and internal VLAN's after connecting to the OpenVPN server?

Reply from 98.136.103.23: bytes=32 time=673ms TTL=47
Request timed out.
Request timed out.

Reply from 10.3.0.100: bytes=32 time=673ms TTL=47
Request timed out.
Request timed out.

Moreover, also seeing timeout on accessing local VLAN's?

root@DD-WRT-INTERNET-ASUS:~# tail -f /var/log/messages|grep -Ei "DROP"|grep -Ei "10.3.0.100"
Nov  4 00:06:16 DD-WRT-INTERNET-ASUS kern.warn kernel: DROP IN=tun2 OUT=br0 MAC= SRC=10.1.1.2 DST=10.3.0.101 LEN=60 TOS=0x00 PREC=0x00 TTL=127 ID=54730 PROTO=ICMP TYPE=8 CODE=0 ID=1 SEQ=6098
Nov  4 00:06:30 DD-WRT-INTERNET-ASUS kern.warn kernel: DROP IN=tun2 OUT=br0 MAC= SRC=10.1.1.2 DST=10.3.0.1001 LEN=52 TOS=0x00 PREC=0x00 TTL=127 ID=54733 DF PROTO=TCP SPT=56718 DPT=22 SEQ=2130463582 ACK=0 WINDOW=64240 RES=0x00 SYN URGP=0 OPT (0204054B0103030801010402)

Chances are you're missing the following rules:

# VPN: Required to be able to ping local on-prem or Azure VLAN's
iptables -I FORWARD -i br0 -o tun2 -j ACCEPT
iptables -I FORWARD -i tun2 -o br0 -j ACCEPT
iptables -I INPUT -i tun2 -j LOGREJECT
iptables -t nat -A POSTROUTING -o tun2 -j MASQUERADE

iptables -I FORWARD -i br0 -o tun1 -j ACCEPT
iptables -I FORWARD -i tun1 -o br0 -j ACCEPT
iptables -I INPUT -i tun1 -j LOGREJECT
iptables -t nat -A POSTROUTING -o tun1 -j MASQUERADE

iptables -I FORWARD -i br0 -o tun0 -j ACCEPT
iptables -I FORWARD -i tun0 -o br0 -j ACCEPT
iptables -I INPUT -i tun0 -j LOGREJECT
iptables -t nat -A POSTROUTING -o tun0 -j MASQUERADE

Rules such as these, do not work:

# Allow TCP from tun2 (VPN)
iptables -A OUTPUT -s 10.1.1.0/24 -p tcp -j ACCEPT
iptables -A INPUT  -s 10.1.1.0/24 -p tcp -j ACCEPT

# Allow UDP from tun2 (VPN)
iptables -A OUTPUT -s 10.1.1.0/24 -p udp -j ACCEPT
iptables -A INPUT  -s 10.1.1.0/24 -p udp -j ACCEPT

# Allow ICMP from tun2 (VPN)
iptables -A OUTPUT -s 10.1.1.0/24 -p icmp -j ACCEPT
iptables -A INPUT  -s 10.1.1.0/24 -p icmp -j ACCEPT

Enjoy your new, shiny reponsive network!  🙂

Cheers,
Admin

OpenWRT: Disable invalid default gateway selection

It indeed happened that the default GW provided on various network interfaces was the router that we do not want to be the GW.  In our case the OpenWRT Raspberry Pi 2 became the GW for any hosts dynamically getting an IP.  So all requests, were sent via the Raspberry Pi 2, which is not what we want.  To fix this, check your device if it is running a DHCP server:

root@OWRT01:~# ps | grep -Ei dhcp
  547 root      1240 S    /usr/sbin/odhcpd
 1556 root      1072 S    grep -Ei dhcp
root@OWRT01:~#

To disable this, you can do so in the Luci interfaces panel ( Luci -> Network -> Interfaces -> LAN ) then Edit then disable the DHCP server in the right most tab:

https://i0.wp.com/www.microdevsys.com/WordPressImages/OpenWRT01-Configure-Disable-DHCP-Server-Default-Gateway.png?ssl=1

Save and restart the device.  

Cheers,
Tom


     
  Copyright © 2003 - 2013 Tom Kacperski (microdevsys.com). All rights reserved.

Creative Commons License
This work is licensed under a Creative Commons Attribution 3.0 Unported License