Header Shadow Image


Patroni: FATAL:  could not connect to the primary server: server closed the connection unexpectedly

Getting this?

[root@psql04 ~]# tail -f /data/patroni/log/postgresql-Wed.log
2022-03-09 20:07:40.890 EST [27627] FATAL:  could not connect to the primary server: server closed the connection unexpectedly
                This probably means the server terminated abnormally
                before or while processing the request.
^C
[root@psql04 ~]#

Check on the targe primary cluster that it is not getting blocked via Haproxy:

[root@psql07 patroni]# tail -f /var/log/audit/audit.log|grep -Ei denied
type=AVC msg=audit(1646874430.882:1393): avc:  denied  { name_connect } for  pid=1045 comm="haproxy" dest=5432 scontext=system_u:system_r:haproxy_t:s0 tcontext=system_u:object_r:postgresql_port_t:s0 tclass=tcp_socket permissive=0

Both logs should scroll as the Standby Cluster tries to make a connection to the Primary Cluster.  Resolve using:

grep AVC /var/log/audit/audit.log* |grep -Ei denied >/var/log/audit/audit.previous; cat /var/log/audit/audit.previous |  audit2allow -M systemd-allow; semodule -i systemd-allow.pp

Better yet, allow all haproxy traffic:

semanage permissive -a haproxy_t

As soon as this is set, Haproxy connections work perfectly and all of a sudden, the Patroni cluster is able to replicate just fine:

[root@psql04 ~]# tail -f /data/patroni/log/postgresql-Wed.log
                before or while processing the request.
2022-03-09 20:28:01.807 EST [30690] FATAL:  could not connect to the primary server: server closed the connection unexpectedly
                This probably means the server terminated abnormally
                before or while processing the request.
2022-03-09 20:28:06.816 EST [30703] LOG:  fetching timeline history file for timeline 1226 from primary server
2022-03-09 20:28:06.838 EST [30703] LOG:  started streaming WAL from primary at 32/55000000 on timeline 1225
2022-03-09 20:28:06.854 EST [30703] LOG:  replication terminated by primary server
2022-03-09 20:28:06.854 EST [30703] DETAIL:  End of WAL reached on timeline 1225 at 32/5508DE08.
2022-03-09 20:28:06.857 EST [27428] LOG:  new target timeline is 1226
2022-03-09 20:28:06.859 EST [30703] LOG:  restarted WAL streaming at 32/55000000 on timeline 1226
^C
[root@psql04 ~]#  patronictl –config-file=/etc/patroni.yml list
+————-+——————–+—————-+———+——+———–+—————–+
| Member      | Host               | Role           | State   |   TL | Lag in MB | Pending restart |
+ Cluster: postgres (6617627977882355208) ———-+———+——+———–+—————–+
| postgresql0 | psql04.nix.mds.xyz | Standby Leader | running | 1226 |           | *               |
| postgresql1 | psql05.nix.mds.xyz | Replica        | running | 1226 |         0 | *               |
| postgresql2 | psql06.nix.mds.xyz | Replica        | running | 1226 |         0 | *               |
+————-+——————–+—————-+———+——+———–+—————–+
[root@psql04 ~]#

Cheers,
 

Leave a Reply

You must be logged in to post a comment.


     
  Copyright © 2003 - 2013 Tom Kacperski (microdevsys.com). All rights reserved.

Creative Commons License
This work is licensed under a Creative Commons Attribution 3.0 Unported License