Another issue we can run into is the following set of messages:

2019-06-08T04:56:24.518538Z 0 [ERROR] WSREP: failed to open gcomm backend connection: 110: failed to reach primary view: 110 (Connection timed out)
at gcomm/src/pc.cpp:connect():158
2019-06-08T04:56:24.518591Z 0 [ERROR] WSREP: gcs/src/gcs_core.cpp:gcs_core_open():209: Failed to open backend connection: -110 (Connection timed out)
2019-06-08T04:56:24.518764Z 0 [ERROR] WSREP: gcs/src/gcs.cpp:gcs_open():1458: Failed to open channel ‘galera_cluster1’ at ‘gcomm://192.168.0.126,192.168.0.107,192.168.0.114’: -110 (Connection timed out)
2019-06-08T04:56:24.518793Z 0 [ERROR] WSREP: gcs connect failed: Connection timed out
2019-06-08T04:56:24.518812Z 0 [ERROR] WSREP: wsrep::connect(gcomm://192.168.0.126,192.168.0.107,192.168.0.114) failed: 7
2019-06-08T04:56:24.518835Z 0 [ERROR] Aborting

To solve this, take a look at your /etc/my.cnf file. The following three fields have to match the server that you are on:

wsrep_node_address=”192.168.0.126″
wsrep_node_name=”mysql01″
server_id=1
bind-address=192.168.0.126

If they don’t, the above error is thrown.  server_id must be unique for each node.  This can happen when you’re restoring your database from one node to another or trying recovery steps where you copy the /etc/my.cnf over to another host.

Another solution involves copying the data directory from the master or most current host to the target host we want as a new master.  We do this because we wanted to test when the master is offline while working only with 2/3 nodes.  Here is our situation:

mysql01 GOOD
mysql02 BAD
mysql03 BAD

Copy the data dir from mysql01 to mysql02:

mysql02 # cd /var/lib/mysql; scp -rp mysql01:/var/lib/mysql/  .

Set the safe_to_bootstrap flag to 1:

cat grastate.dat
# GALERA saved state
version: 2.1
uuid:    f25fc12b-8a0b-11e9-b58d-bfb801e3b36d
seqno:   -1
safe_to_bootstrap: 1

(IF Grant Tables is an Issue) Start mysqld in safe mode to reset the pass, if need be:

[root@mysql03 mysql]# systemctl set-environment MYSQLD_OPTS=”–wsrep-new-cluster –skip-grant-tables”

(Recommended) OR just set the following variable indicating this is a new bootstrapped cluster:

[root@mysql03 mysql]# systemctl set-environment MYSQLD_OPTS=”–wsrep-new-cluster”

Bootstrap this node, in other words, make it primary:

mysql02 # /usr/bin/mysqld_bootstrap

Then check:

mysql02 # mysql -e “SHOW STATUS LIKE ‘wsrep_cluster_status’;”
+———————-+————-+
| Variable_name | Value |
+———————-+————-+
| wsrep_cluster_status | non-Primary |
+———————-+————-+

On the third node, mysql03, remove all files from the /var/lib/mysql folder because we’ll let it sync up from mysql02:

mysql03 # cd /var/lib; mv mysql mysql-bk01; mkdir mysql; chown mysql:mysql mysql; cd mysql;
OR
mysql03 # cd /var/lib/mysql; rm -rf *

Start mysql on mysql03 so it sync’s from mysql02:

mysql03 # systemctl start mysqld

Let it sync.  You should have an accessible 2/3 node cluster at this point.  If the original node you bootstrapped from exhibits the same issue, also clear it’s directory and restart it to allow data to be copied from the two good nodes.

Restart the original bootstrapped node without the flag set before, rebooting to test after the next two steps:

[root@mysql01 mysql]# systemctl set-environment MYSQLD_OPTS=””;
[root@mysql01 mysql]# systemctl restart mysqld;

Restart the cluster one node at a time at first, then all together to verify cluster comes back up properly.

2025 Dec 31st

Another way to recover is to use the last known commit:

mysql02 # /usr/sbin/mysqld –wsrep-recover

And look for a line like this:

mysql02 # 2025-12-31T19:43:57.479659Z 0 [Note] WSREP: Recovered position: 751c8bd5-224f-11ea-b912-c7063c41e2a2:127588425

That recovery number is your recovery commit.  Add it to the following file:

mysql02 # cat /var/lib/mysql/grastate.dat
# GALERA saved state
version: 2.1
uuid: 751c8bd5-224f-11ea-b912-c7063c41e2a2
seqno: 127588425
safe_to_bootstrap: 1

Then when you start, you should see the following:

mysql02 # /usr/sbin/mysqld –user=mysql –wsrep-new-cluster –daemonize  –pid-file=/var/run/mysqld/mysqld.pid –log-error=/var/log/mysqld.log
mysql02 # mysql -e “SHOW STATUS LIKE ‘wsrep_cluster_status’;”
+———————-+———+
| Variable_name | Value |
+———————-+———+
| wsrep_cluster_status | Primary |
+———————-+———+

If you get this error:

Dec 31 14:50:21 mysql03.nix.mds.xyz mysqld[30802]: 2025-12-31T19:50:21.276605Z 1 [ERROR] WSREP: Requesting state transfer failed: -77(File descriptor in bad state)
Dec 31 14:50:21 mysql03.nix.mds.xyz mysqld[30802]: 2025-12-31T19:50:21.276730Z 1 [ERROR] WSREP: State transfer request failed unrecoverably: 77 (File descriptor in bad state). Most likely it is due to inability to communicate with the cluster primary component. Restart required.

It is almost certainly selinux related:

type=AVC msg=audit(1767210964.307:165110): avc: denied { write } for pid=19521 comm=”mysql” name=”mysql.sock” dev=”dm-0″ ino=201702945 scontext=system_u:system_r:zabbix_agent_t:s0 tcontext=unconfined_u:object_r:mysqld_db_t:s0 tclass=sock_file permissive=0
type=AVC msg=audit(1767210965.452:165111): avc: denied { write } for pid=19525 comm=”mysql” name=”mysql.sock” dev=”dm-0″ ino=201702945 scontext=system_u:system_r:zabbix_agent_t:s0 tcontext=unconfined_u:object_r:mysqld_db_t:s0 tclass=sock_file permissive=0

on the node, node 2 in this case, that is being used as a bootstrap node.  Use this command to fix that up:

mysql02 # grep AVC /var/log/audit/audit.log* >> /var/log/audit/audit-denied-all.log; cat /var/log/audit/audit-denied-all.log | audit2allow -M systemd-allow; semodule -i systemd-allow.pp

Successful messages, after clearing mysql03 /var/lib/mysql/ folder of any files, and restarting it ( see earlier ).  Successful messages were:

Dec 31 15:20:07 mysql03.nix.mds.xyz rsyncd[6921]: connect from mysql02.nix.mds.xyz (192.168.0.107)
Dec 31 15:20:07 mysql03.nix.mds.xyz rsyncd[6921]: rsync to rsync_sst/ from mysql02.nix.mds.xyz (192.168.0.107)
Dec 31 15:20:07 mysql03.nix.mds.xyz rsyncd[6921]: receiving file list
Dec 31 15:20:07 mysql03.nix.mds.xyz rsyncd[6921]: sent 48 bytes received 190 bytes total size 47
Dec 31 15:20:07 mysql03.nix.mds.xyz mysqld[4008]: 2025-12-31T20:20:07.250428Z 0 [Note] WSREP: 1.0 (mysql02): State transfer to 0.0 (mysql03) complete.
Dec 31 15:20:07 mysql03.nix.mds.xyz mysqld[4008]: 2025-12-31T20:20:07.251352Z 0 [Note] WSREP: Member 1.0 (mysql02) synced with group.
Dec 31 15:20:07 mysql03.nix.mds.xyz mysqld[4008]: WSREP_SST: [INFO] Joiner cleanup. rsync PID: 4068 (20251231 15:20:07.727)
Dec 31 15:20:08 mysql03.nix.mds.xyz mysqld[4008]: WSREP_SST: [INFO] Joiner cleanup done. (20251231 15:20:08.244)
Dec 31 15:20:08 mysql03.nix.mds.xyz mysqld[4008]: 2025-12-31T20:20:08.249592Z 0 [Note] WSREP: SST complete, seqno: 127589412
Dec 31 15:20:08 mysql03.nix.mds.xyz mysqld[4008]: 2025-12-31T20:20:08.301565Z 0 [Note] InnoDB: PUNCH HOLE support available
Dec 31 15:20:08 mysql03.nix.mds.xyz mysqld[4008]: 2025-12-31T20:20:08.301659Z 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
Dec 31 15:20:08 mysql03.nix.mds.xyz mysqld[4008]: 2025-12-31T20:20:08.301683Z 0 [Note] InnoDB: Uses event mutexes
Dec 31 15:20:08 mysql03.nix.mds.xyz mysqld[4008]: 2025-12-31T20:20:08.301696Z 0 [Note] InnoDB: GCC builtin __atomic_thread_fence() is used for memory barrier
Dec 31 15:20:08 mysql03.nix.mds.xyz mysqld[4008]: 2025-12-31T20:20:08.301709Z 0 [Note] InnoDB: Compressed tables use zlib 1.2.11
Dec 31 15:20:08 mysql03.nix.mds.xyz mysqld[4008]: 2025-12-31T20:20:08.301727Z 0 [Note] InnoDB: Using Linux native AIO
Dec 31 15:20:08 mysql03.nix.mds.xyz mysqld[4008]: 2025-12-31T20:20:08.302643Z 0 [Note] InnoDB: Number of pools: 1
Dec 31 15:20:08 mysql03.nix.mds.xyz mysqld[4008]: 2025-12-31T20:20:08.303038Z 0 [Note] InnoDB: Using CPU crc32 instructions
Dec 31 15:20:08 mysql03.nix.mds.xyz mysqld[4008]: 2025-12-31T20:20:08.311476Z 0 [Note] InnoDB: Initializing buffer pool, total size = 128M, instances = 1, chunk size = 128M
Dec 31 15:20:08 mysql03.nix.mds.xyz mysqld[4008]: 2025-12-31T20:20:08.334984Z 0 [Note] InnoDB: Completed initialization of buffer pool
Dec 31 15:20:08 mysql03.nix.mds.xyz mysqld[4008]: 2025-12-31T20:20:08.341182Z 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority().
Dec 31 15:20:08 mysql03.nix.mds.xyz mysqld[4008]: 2025-12-31T20:20:08.426660Z 0 [Note] InnoDB: Highest supported file format is Barracuda.
Dec 31 15:20:08 mysql03.nix.mds.xyz mysqld[4008]: 2025-12-31T20:20:08.442088Z 0 [Note] InnoDB: Log scan progressed past the checkpoint lsn 434182622771
Dec 31 15:20:09 mysql03.nix.mds.xyz mysqld[4008]: 2025-12-31T20:20:09.210975Z 0 [Note] InnoDB: Doing recovery: scanned up to log sequence number 434187865600
Dec 31 15:20:10 mysql03.nix.mds.xyz mysqld[4008]: 2025-12-31T20:20:10.093181Z 0 [Note] InnoDB: Doing recovery: scanned up to log sequence number 434193108480
Dec 31 15:20:10 mysql03.nix.mds.xyz mysqld[4008]: 2025-12-31T20:20:10.904396Z 0 [Note] InnoDB: Doing recovery: scanned up to log sequence number 434198351360
Dec 31 15:20:11 mysql03.nix.mds.xyz mysqld[4008]: 2025-12-31T20:20:11.628316Z 0 [Note] InnoDB: Doing recovery: scanned up to log sequence number 434203594240
Dec 31 15:20:12 mysql03.nix.mds.xyz mysqld[4008]: 2025-12-31T20:20:12.576014Z 0 [Note] InnoDB: Doing recovery: scanned up to log sequence number 434208837120
Dec 31 15:20:13 mysql03.nix.mds.xyz mysqld[4008]: 2025-12-31T20:20:13.030583Z 0 [Note] InnoDB: Doing recovery: scanned up to log sequence number 434211868844
Dec 31 15:20:13 mysql03.nix.mds.xyz mysqld[4008]: 2025-12-31T20:20:13.036111Z 0 [Note] InnoDB: Database was not shutdown normally!
Dec 31 15:20:13 mysql03.nix.mds.xyz mysqld[4008]: 2025-12-31T20:20:13.036199Z 0 [Note] InnoDB: Starting crash recovery.
Dec 31 15:20:13 mysql03.nix.mds.xyz mysqld[4008]: 2025-12-31T20:20:13.665787Z 0 [Note] InnoDB: 4 transaction(s) which must be rolled back or cleaned up in total 41 row operations to undo
Dec 31 15:20:13 mysql03.nix.mds.xyz mysqld[4008]: 2025-12-31T20:20:13.665893Z 0 [Note] InnoDB: Trx id counter is 403032320
Dec 31 15:20:13 mysql03.nix.mds.xyz mysqld[4008]: 2025-12-31T20:20:13.665951Z 0 [Note] InnoDB: Starting an apply batch of log records to the database…
Dec 31 15:20:16 mysql03.nix.mds.xyz mysqld[4008]: InnoDB: Progress in percent: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99
Dec 31 15:20:16 mysql03.nix.mds.xyz mysqld[4008]: 2025-12-31T20:20:16.631310Z 0 [Note] InnoDB: Apply batch completed
Dec 31 15:20:16 mysql03.nix.mds.xyz mysqld[4008]: 2025-12-31T20:20:16.632266Z 0 [Note] InnoDB: Last MySQL binlog file position 0 11806, file name mysql_binary_log.000432
Dec 31 15:20:19 mysql03.nix.mds.xyz mysqld[4008]: 2025-12-31T20:20:19.498303Z 0 [Note] InnoDB: Creating shared tablespace for temporary tables
Dec 31 15:20:19 mysql03.nix.mds.xyz mysqld[4008]: 2025-12-31T20:20:19.498340Z 0 [Note] InnoDB: Starting in background the rollback of uncommitted transactions
Dec 31 15:20:19 mysql03.nix.mds.xyz mysqld[4008]: 2025-12-31T20:20:19.498675Z 0 [Note] InnoDB: Setting file ‘./ibtmp1’ size to 12 MB. Physically writing the file full; Please wait …
Dec 31 15:20:19 mysql03.nix.mds.xyz mysqld[4008]: 2025-12-31T20:20:19.498740Z 0 [Note] InnoDB: Rolling back trx with id 403031914, 9 rows to undo
Dec 31 15:20:19 mysql03.nix.mds.xyz mysqld[4008]: 2025-12-31T20:20:19.558833Z 0 [Note] InnoDB: File ‘./ibtmp1’ size is now 12 MB.
Dec 31 15:20:19 mysql03.nix.mds.xyz mysqld[4008]: 2025-12-31T20:20:19.567082Z 0 [Note] InnoDB: 96 redo rollback segment(s) found. 96 redo rollback segment(s) are active.
Dec 31 15:20:19 mysql03.nix.mds.xyz mysqld[4008]: 2025-12-31T20:20:19.567148Z 0 [Note] InnoDB: 32 non-redo rollback segment(s) are active.
Dec 31 15:20:19 mysql03.nix.mds.xyz mysqld[4008]: 2025-12-31T20:20:19.569051Z 0 [Note] InnoDB: Waiting for purge to start
Dec 31 15:20:19 mysql03.nix.mds.xyz mysqld[4008]: 2025-12-31T20:20:19.619491Z 0 [Note] InnoDB: page_cleaner: 1000ms intended loop took 11278ms. The settings might not be optimal. (flushed=0 and evicted=0, during the time.)
Dec 31 15:20:19 mysql03.nix.mds.xyz mysqld[4008]: 2025-12-31T20:20:19.622393Z 0 [Note] InnoDB: 5.7.25 started; log sequence number 434211868844
Dec 31 15:20:19 mysql03.nix.mds.xyz mysqld[4008]: 2025-12-31T20:20:19.624115Z 0 [Note] InnoDB: Loading buffer pool(s) from /var/lib/mysql/ib_buffer_pool
Dec 31 15:20:19 mysql03.nix.mds.xyz mysqld[4008]: 2025-12-31T20:20:19.722659Z 0 [Note] Plugin ‘FEDERATED’ is disabled.
Dec 31 15:20:19 mysql03.nix.mds.xyz mysqld[4008]: 2025-12-31T20:20:19.743210Z 0 [Note] InnoDB: Buffer pool(s) load completed at 251231 15:20:19
Dec 31 15:20:20 mysql03.nix.mds.xyz mysqld[4008]: 2025-12-31T20:20:20.131948Z 0 [Note] InnoDB: Rollback of trx with id 403031914 completed
Dec 31 15:20:20 mysql03.nix.mds.xyz mysqld[4008]: 2025-12-31T20:20:20.132056Z 0 [Note] InnoDB: Rolling back trx with id 403031913, 8 rows to undo
Dec 31 15:20:20 mysql03.nix.mds.xyz mysqld[4008]: 2025-12-31T20:20:20.293691Z 0 [Note] InnoDB: Rollback of trx with id 403031913 completed
Dec 31 15:20:20 mysql03.nix.mds.xyz mysqld[4008]: 2025-12-31T20:20:20.293810Z 0 [Note] InnoDB: Rolling back trx with id 403031908, 12 rows to undo
Dec 31 15:20:20 mysql03.nix.mds.xyz mysqld[4008]: 2025-12-31T20:20:20.431091Z 0 [Note] Found ca.pem, server-cert.pem and server-key.pem in data directory. Trying to enable SSL support using them.
Dec 31 15:20:20 mysql03.nix.mds.xyz mysqld[4008]: 2025-12-31T20:20:20.456280Z 0 [Warning] CA certificate ca.pem is self signed.
Dec 31 15:20:20 mysql03.nix.mds.xyz mysqld[4008]: 2025-12-31T20:20:20.459943Z 0 [Note] Server hostname (bind-address): ‘192.168.0.114’; port: 3306
Dec 31 15:20:20 mysql03.nix.mds.xyz mysqld[4008]: 2025-12-31T20:20:20.460097Z 0 [Note] – ‘192.168.0.114’ resolves to ‘192.168.0.114’;
Dec 31 15:20:20 mysql03.nix.mds.xyz mysqld[4008]: 2025-12-31T20:20:20.460310Z 0 [Note] Server socket created on IP: ‘192.168.0.114’.
Dec 31 15:20:20 mysql03.nix.mds.xyz mysqld[4008]: 2025-12-31T20:20:20.467839Z 0 [Note] InnoDB: Rollback of trx with id 403031908 completed
Dec 31 15:20:20 mysql03.nix.mds.xyz mysqld[4008]: 2025-12-31T20:20:20.467987Z 0 [Note] InnoDB: Rolling back trx with id 403031888, 12 rows to undo
Dec 31 15:20:20 mysql03.nix.mds.xyz mysqld[4008]: 2025-12-31T20:20:20.637832Z 0 [Note] InnoDB: Rollback of trx with id 403031888 completed
Dec 31 15:20:20 mysql03.nix.mds.xyz mysqld[4008]: 2025-12-31T20:20:20.637915Z 0 [Note] InnoDB: Rollback of non-prepared transactions completed
Dec 31 15:20:20 mysql03.nix.mds.xyz mysqld[4008]: 2025-12-31T20:20:20.811238Z 0 [Note] Failed to start slave threads for channel ”
Dec 31 15:20:21 mysql03.nix.mds.xyz mysqld[4008]: 2025-12-31T20:20:21.227860Z 0 [Note] Event Scheduler: Loaded 0 events
Dec 31 15:20:21 mysql03.nix.mds.xyz mysqld[4008]: 2025-12-31T20:20:21.228014Z 0 [Note] WSREP: Signalling provider to continue.
Dec 31 15:20:21 mysql03.nix.mds.xyz systemd[1]: Started MySQL Server.
— Subject: Unit mysqld.service has finished start-up
— Defined-By: systemd
— Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
—
— Unit mysqld.service has finished starting up.
—
— The start-up result is done.

Now that mysql03 and mysql01 are synced, and only then, stop mysql02 and bring it back to a standard state:

mysql02 # kill $(cat /var/run/mysqld/mysqld.pid)

Start mysqld using systemd:

mysql02 # systemctl start mysqld

Verify:

mysql02 # mysql -e “SHOW STATUS LIKE ‘wsrep_cluster_size’;”
+——————–+——-+
| Variable_name | Value |
+——————–+——-+
| wsrep_cluster_size | 3 |
+——————–+——-+

Check sync status:

mysql02 # mysql -e “SHOW STATUS LIKE ‘wsrep_local_state_comment’;”
+—————————+——–+
| Variable_name | Value |
+—————————+——–+
| wsrep_local_state_comment | Synced |
+—————————+——–+

Verify by creating a test database on one node:

mysql02 # mysql -e “CREATE DATABASE cluster_test;”

mysql03 # mysql -e “show databases;”
+——————–+
| Database |
+——————–+
| information_schema |
| cluster_test |
| galera_test |
| haproxy |
| mysql |
| opennebula |
| performance_schema |
| sys |
| zabbix |
+——————–+

Then drop the database on mysql03 and verify off mysql02:

mysql03 # mysql -e “drop database cluster_test;”

mysql02 # mysql -e “show databases;”
+——————–+
| Database |
+——————–+
| information_schema |
| galera_test |
| haproxy |
| mysql |
| opennebula |
| performance_schema |
| sys |
| zabbix |
+——————–+

It should be showing as gone on all the nodes.

Thx,
TK