Patroni: FATAL: could not connect to the primary server: server closed the connection unexpectedly
Getting this?
[root@psql04 ~]# tail -f /data/patroni/log/postgresql-Wed.log
2022-03-09 20:07:40.890 EST [27627] FATAL: could not connect to the primary server: server closed the connection unexpectedly
This probably means the server terminated abnormally
before or while processing the request.
^C
[root@psql04 ~]#
Check on the targe primary cluster that it is not getting blocked via Haproxy:
[root@psql07 patroni]# tail -f /var/log/audit/audit.log|grep -Ei denied
type=AVC msg=audit(1646874430.882:1393): avc: denied { name_connect } for pid=1045 comm="haproxy" dest=5432 scontext=system_u:system_r:haproxy_t:s0 tcontext=system_u:object_r:postgresql_port_t:s0 tclass=tcp_socket permissive=0
Both logs should scroll as the Standby Cluster tries to make a connection to the Primary Cluster. Resolve using:
grep AVC /var/log/audit/audit.log* |grep -Ei denied >/var/log/audit/audit.previous; cat /var/log/audit/audit.previous | audit2allow -M systemd-allow; semodule -i systemd-allow.pp
Better yet, allow all haproxy traffic:
semanage permissive -a haproxy_t
As soon as this is set, Haproxy connections work perfectly and all of a sudden, the Patroni cluster is able to replicate just fine:
[root@psql04 ~]# tail -f /data/patroni/log/postgresql-Wed.log
before or while processing the request.
2022-03-09 20:28:01.807 EST [30690] FATAL: could not connect to the primary server: server closed the connection unexpectedly
This probably means the server terminated abnormally
before or while processing the request.
2022-03-09 20:28:06.816 EST [30703] LOG: fetching timeline history file for timeline 1226 from primary server
2022-03-09 20:28:06.838 EST [30703] LOG: started streaming WAL from primary at 32/55000000 on timeline 1225
2022-03-09 20:28:06.854 EST [30703] LOG: replication terminated by primary server
2022-03-09 20:28:06.854 EST [30703] DETAIL: End of WAL reached on timeline 1225 at 32/5508DE08.
2022-03-09 20:28:06.857 EST [27428] LOG: new target timeline is 1226
2022-03-09 20:28:06.859 EST [30703] LOG: restarted WAL streaming at 32/55000000 on timeline 1226
^C
[root@psql04 ~]# patronictl –config-file=/etc/patroni.yml list
+————-+——————–+—————-+———+——+———–+—————–+
| Member | Host | Role | State | TL | Lag in MB | Pending restart |
+ Cluster: postgres (6617627977882355208) ———-+———+——+———–+—————–+
| postgresql0 | psql04.nix.mds.xyz | Standby Leader | running | 1226 | | * |
| postgresql1 | psql05.nix.mds.xyz | Replica | running | 1226 | 0 | * |
| postgresql2 | psql06.nix.mds.xyz | Replica | running | 1226 | 0 | * |
+————-+——————–+—————-+———+——+———–+—————–+
[root@psql04 ~]#
Cheers,