Now-Next

SaaS architecture

server closed the connection unexpectedly in PostgreSQL

PostgreSQL's server closed the connection unexpectedly, measured in six clients: what the server log says per cause, where the reason goes, and lost COMMITs.

Chris van Eijk, Now-Next · · 14 min read

What does “server closed the connection unexpectedly” mean?

server closed the connection unexpectedly means the client library libpq found the connection gone. PostgreSQL does not send this text, it has no SQLSTATE, and it does not say why: we got the same sentence after a kill -9, an OOM kill, a shutdown, a timeout, a pooler that died and a network reset. The reason is usually in the server log. If the log has nothing at that moment, look outside PostgreSQL, or at whether PostgreSQL itself was killed.

CauseWhat the server log saysFix
pg_terminate_backend()FATAL: terminating connection due to administrator command (57P01)find who ran it; reconnect
fast shutdown, docker stop, docker restartreceived fast shutdown request, then the same FATAL per sessionreconnect after the restart; a pool with a pre-ping hides it once the server is back
idle_session_timeoutFATAL: terminating connection due to idle-session timeout (57P05)keep pooled connections shorter than the timeout, or raise it
idle_in_transaction_session_timeoutFATAL: terminating connection due to idle-in-transaction timeout (25P03)do not leave transactions open
transaction_timeout (17 and 18)FATAL: terminating connection due to transaction timeout (25P04)shorten the transaction, or raise it
a backend killed or crashed, here or in another sessionserver process (PID 240) was terminated by signal 9: Killed, and on 18 client backend (PID 213) was terminated…read the DETAIL line if there is one, for an idle backend it shows the last query; find what sent the signal
OOM killerwhen it picks a backend, exactly the same signal 9 lines; the word OOM does not appearcheck the kernel or container, see below
immediate shutdownreceived immediate shutdown requestuse fast shutdown; immediate replays WAL at the next start
whole server killed (docker kill -s KILL)nothing at that moment; after the restart database system was not properly shut downlook at the host or orchestrator
PgBouncer closed the clientnothing, the backend stays; PgBouncer log: closing because: client_idle_timeout (age=2s)raise the pooler timeout or reconnect
reset from the networknothing when only the client received the reset; a reset that reaches the server logs could not receive data from client: Connection reset by peerlook at the proxy, load balancer or firewall
client keepalive gave up on a dead pathnot read in this runsee below

The causes and log lines are measured; the fixes are our advice unless a section below says we tested them. At connection time the same sentence has a different cause, covered in Connection refused: is the server running?: something accepted the connection, in our case Docker’s port mapping, and no server was behind it.

We measured on the official postgres Docker images 13.23, 15.19, 17.11 and 18.6, in the night of 30 to 31 October 2026, and repeated most cases later that night with the scripts below. The clients were psql 17.11, psycopg2 2.9.13, psycopg 3.3.6, pgJDBC 42.7.13, node-postgres 8.23.1 and pgx 5.11.0, plus SQLAlchemy 2.1.6 and PgBouncer 1.26.0. Without a version in the text, the result is from 17.11.

The sentence comes from five places in libpq, none in the server. At one of them, a read that returned 0 bytes, the source says “the connection has been closed”, and then libpq prints “This probably means the server terminated abnormally”. That guess can be wrong, as the reset case below shows. In this message “server” just means the other end of the socket.

How do I find the real cause?

Start with what your client printed above the sentence. A FATAL means the server or a pooler said why. A WARNING means the session was ended because of something else: another backend crashed, an immediate shutdown ran, or a wait for a synchronous standby was cancelled. Nothing above it proves little, because psycopg2 and the other libraries we tested did not show these messages. Then read the server log around that time. Search for was terminated by signal, not server process: PostgreSQL 18 writes client backend (PID …) where 13, 15 and 17 write server process (PID …).

If you are the bystander, the client shows a warning that psql prints and the libraries we tested did not. A kill -9 on one backend makes the server end every session:

WARNING:  terminating connection because of crash of another server process
DETAIL:  The postmaster has commanded this server process to roll back the current transaction and exit, because another server process exited abnormally and possibly corrupted shared memory.
HINT:  In a moment you should be able to reconnect to the database and repeat your command.
server closed the connection unexpectedly

The log then shows the recovery. In an empty database, it took about 150 ms from the kill to database system is ready to accept connections; that says nothing about a server with real WAL to replay. With WAL to replay you see redo starts and redo done instead of redo is not required.

LOG:  server process (PID 240) was terminated by signal 9: Killed
DETAIL:  Failed process was running: set application_name=victim; select pg_sleep(300)
LOG:  terminating any other active server processes
LOG:  all server processes terminated; reinitializing
LOG:  database system was interrupted; last known up at 2026-10-30 22:54:49 UTC
LOG:  database system was not properly shut down; automatic recovery in progress
LOG:  redo is not required
LOG:  database system is ready to accept connections

That warning has SQLSTATE 57P02, but you cannot catch it. It reaches the client as a notice, followed by the end of the connection. In psycopg 3 a notice handler saw 57P02, while the exception itself had no SQLSTATE. On 15, 17 and 18 the warning goes only to the client; 13 also logged it per session. Immediate shutdown has its own wording from 15 on (terminating connection due to immediate shutdown command), while 13 reports a crash of another process that never happened.

Why does psycopg2 show less than psql?

Because the same event looks different per client. With current psycopg2 the server’s reason does not reach your code:

Clientpg_terminate_backend()kill -9 on the backendNext query
psqlFATAL: terminating connection…, then server closed…server closed…stops, exit code 2 when not interactive
psycopg2 2.9.13server closed…, pgcode is Noneserver closed…, pgcode is Noneconnection already closed
psycopg 3.3.6AdminShutdown, 57P01consuming input failed: server closed…the connection is closed
pgJDBC 42.7.13FATAL: terminating connection…, 57P0108006, An I/O error occurred while sending to the backend.08003, This connection has been closed.
node-postgres 8.23.1idle: 'error' event with 57P01; running: the query rejects with 57P01'error' event, Connection terminated unexpectedlyClient has encountered a connection error and is not queryable
pgx 5.11.0FATAL: terminating connection… (SQLSTATE 57P01)conn closed, or unexpected EOF during a queryconn closed

The pg_terminate_backend() column was the same on 13, 15, 17 and 18. psycopg2 gave its one sentence for an idle session and during a running query. So with psycopg2 a 57P01, a timeout and a crash look the same, and the server log is the only place that separates them.

This depends on the psycopg2 wheel, and on the libpq inside it. Older answers online show the server’s reason in the text, and that was true. We terminated the same sessions on 17.11 with four wheels:

psycopg2-binaryBundled libpqExceptionpgcodeServer’s reason in the message
2.8.611.5AdminShutdown57P01yes
2.9.314.1OperationalErrorNoneyes
2.9.916.0OperationalErrorNoneno
2.9.1317.11OperationalErrorNoneno

Code that catches psycopg2.errors.AdminShutdown therefore stopped firing on a version upgrade, without an error. Idle and running sessions behaved the same in each row. If your log only has a message from the last column of the table above, such as connection already closed, that is the follow-up error; look for the one before it.

Why does the error only appear on my next query?

Because the client is not reading the socket while it is idle. With idle_session_timeout = 2s the server closed the session after 2 s, and psql, psycopg2, psycopg 3, pgJDBC and pgx noticed nothing until the next query. So an application that waits ten minutes will see the failure on a harmless SELECT, with the timeout long past. idle_in_transaction_session_timeout and transaction_timeout behaved the same. idle_session_timeout did not exist on 13 (unrecognized configuration parameter); it worked on 15, 17 and 18.

node-postgres is the exception: it reads all the time, so it gets an 'error' event with no query running. Without a listener that event ends the process:

node:events:502
      throw er; // Unhandled 'error' event
error: terminating connection due to administrator command

This held for Client and for Pool. With pool.on('error', …) the pool cleaned up and the next pool.query worked.

The documentation warns about this for idle_session_timeout: “Be wary of enforcing this timeout on connections made through connection-pooling software or other middleware, as such a layer may not react well to unexpected connection closure.”

Was it the OOM killer?

PostgreSQL will not tell you. We ran a real one: a container with --memory=300m --memory-swap=300m, work_mem = 2GB and a sort of 30 million rows. The client got the plain sentence. The log was identical to a manual kill -9, and DETAIL named the sort:

LOG:  server process (PID 66) was terminated by signal 9: Killed
DETAIL:  Failed process was running: select count(*) from (select g, md5(g::text) m from generate_series(1,30000000) g order by m) s
LOG:  terminating any other active server processes

The proof was outside PostgreSQL: docker inspect gave OOMKilled=true Running=true. The container was still running, because only a backend died and the postmaster restarted the rest. Monitoring that counts container restarts sees nothing here. Other places to look are dmesg and kubectl describe, which we did not test. The documentation’s advice on the OOM killer is vm.overcommit_memory=2, which “will lower the chances significantly”. That is a host-level setting for the kernel’s global OOM killer; it does not protect against a container memory limit like the one in our test, where the fix is a lower work_mem or a higher limit.

Why does it happen when the server did nothing?

Two cases. In the reset run the server log stayed empty, because only the client received the reset.

A reset from the network. We made the server’s own network stack answer with a TCP reset, while PostgreSQL kept running. libpq turns ECONNRESET into the same sentence:

[psql]     server closed the connection unexpectedly
[psycopg3] consuming input failed: server closed the connection unexpectedly
[pgjdbc]   08006 An I/O error occurred while sending to the backend. cause=java.net.SocketException: Connection reset
[node-pg]  read ECONNRESET
[pgx]      conn closed

“This probably means the server terminated abnormally” is simply wrong here. Only pgJDBC and node-postgres name the reset.

A dead path and client keepalive. We dropped all packets, as a NAT table does when it has forgotten a connection, with keepalive set to 3 s idle, 1 s interval, 3 probes. If the query came 10 s later, it failed at once with server closed the connection unexpectedly, while the server was fine. Our reading is that keepalive had already closed the socket. So if you enable keepalive and this message becomes more common, you may be seeing the old hang as a fast failure.

Not every network problem gives an error. With the server disconnected from the Docker network for 35 s, the client just waited, and its query succeeded after 52.75 s, once TCP’s retransmits got through.

What if the connection hangs instead of failing?

A silent path, without a reset, gives no error for a long time. We dropped packets in both directions with psycopg2 against a live server:

State when the path diedClient settingResult
idle, query 1 s laterkeepalive 3/1/3hangs (we stopped at 45 s)
idle, query 10 s laterkeepalive 3/1/3at once: server closed the connection unexpectedly
idle, query 1 s latertcp_user_timeout=5000+5.44 s: could not receive data from server: Connection timed out
idle, query 10 s latertcp_user_timeout=5000+5.47 s: same
query runningnonehangs (we stopped at 45 s)
query runningkeepalive 3/1/3+6.13 s: could not receive data…
query runningtcp_user_timeout=5000hangs (we stopped at 45 s)
query runningboth+5.11 s: could not receive data…

You need both settings. Keepalive did not fire while a query that had just been sent was unacknowledged. tcp_user_timeout did not fire when the query had been acknowledged and the client was only waiting for the answer. That explanation is ours; the measured part is the table.

Without any setting it takes a long time. All six clients, with an idle connection and a query sent 1 s after the path died, failed after about 951 s, nearly 16 minutes, on Linux with tcp_retries2=15. None of them printed our sentence: libpq said could not receive data from server: Connection timed out, node-postgres read ETIMEDOUT, pgJDBC 08006, pgx unexpected EOF. The libpq documentation says keepalive defaults to on, but with the kernel default of 7200 s idle that means two hours before the first probe.

The server has the same problem from its side. When the client became a black hole during a running query, client_connection_check_interval = 1s alone did nothing: the backend was still there when we stopped looking after 25 s. With server keepalive 3/1/3 added it was gone after about 6 s, with FATAL: connection to client lost. An idle backend on a dead connection stayed for as long as we looked (45 s) by default, and it takes up a connection slot, which adds up to remaining connection slots are reserved.

Does pool_pre_ping fix it?

Mostly, with two gaps. We took a SQLAlchemy pool of 3, killed all its connections with pg_terminate_backend() (psycopg2 and psycopg 3) or docker restart (psycopg2 only), and ran five requests one after another. With pool_pre_ping all five worked. Without it the first request failed with OperationalError … server closed… (psycopg 3: AdminShutdown) and the next four worked.

So exactly one request failed, not three, which fits SQLAlchemy’s rule of invalidating the whole pool after a disconnect. That was single-threaded; we did not test parallel load.

The first gap: pre-ping checks one moment. A connection that dies after checkout still gave OperationalError … server closed the connection unexpectedly. The second gap: with a black hole the ping itself hung, and no request came back in the 30 s we waited. With connect_args=dict(tcp_user_timeout=3000) the first request failed after 6.49 s and the second after 3.01 s. So pre-ping needs a timeout under it.

With PgBouncer 1.26.0 in between, the “server” in the message is the pooler. client_idle_timeout = 2 gave psql FATAL: client_idle_timeout above the sentence, psycopg2 only the sentence, and psycopg 3 and pgx 08P01, the code for protocol_violation, though nothing in the protocol went wrong; node-postgres got the same code as an event, and pgJDBC started last and stayed inside the timeout. server_lifetime gave no message in our test, with idle clients and no query running.

Did my COMMIT go through?

From the error alone, the client cannot know. We held a COMMIT in place and killed the backend, either before the commit record was written or after it was flushed:

One COMMIT, one kill -9: the only difference is when it lands

Killed before the commit record

  1. ClientSends COMMIT and waits.
    Transaction open
  2. BackendKilled before it writes the commit record.
    No commit record in WAL
  3. ClientGets server closed…
    After recovery: 0 rows

Killed after the flush

  1. ClientSends COMMIT and waits.
    Transaction open
  2. BackendFlushes the commit record, then is killed before it answers.
    Commit record on disk
  3. ClientGets the same server closed…
    After recovery: 1 row

Row counts and client messages are measured on 17 and 18; the other clients are in the table below.

kill -9 before the commit recordkill -9 after the flush
row after recovery01
psql, psycopg2server closed…server closed…
psycopg 3consuming input failed: server closed…same
pgJDBC08006, An I/O error…same
node-postgresConnection terminated unexpectedlysame
pgxconn closedsame

Same output, opposite outcome, on 17 and 18. With pg_terminate_backend() the clients differed, except psycopg2, which again printed the same sentence for both outcomes. With a broken network it was no better: with tcp_user_timeout=3000, losing the answer and losing the COMMIT itself both gave could not receive data from server: Connection timed out after 3.42 s, with the row present in the first case and absent in the second. In the second case the backend also stayed behind, idle in transaction.

In a commit-heavy loop it is common. We ran an autocommit INSERT loop with psycopg2 and sent kill -9 to the backend at random moments, about every 0.9 s, until 100 statements had failed:

synchronous_commitFailed statementsFailed, but the row was thereAcknowledged, but the row was gone
on100770 of 214,597
off1000197,385 of 1,824,920

Blindly retrying those 77 would have written duplicates or hit a unique violation. A second run gave 81 of 100, and 0 of 210,747 lost. The number belongs to this loop of single-row commits; it shows that it happens, not how often it happens to you. The off row is the other side: there the error tells the truth and the success does not.

The state says as much: after the failure info.transaction_status was UNKNOWN in psycopg 3 and 4 in psycopg2, as for any dead connection. SQLSTATE 08007 transaction_resolution_unknown exists, but no client gave it to us. pgx 5.11.0 reported SafeToRetry as true for conn closed, on a COMMIT whose row was in the table, so do not decide on a COMMIT retry from that flag.

Can pg_xact_status tell me?

Not reliably after a crash. The documentation offers it for this case: “to determine whether their transaction committed or aborted after the application and database server become disconnected while a COMMIT is in progress.” It worked after pg_terminate_backend(), and after a kill once the commit was on disk.

After kill -9 during COMMIT, before the commit record, we asked for the status of our own xid, let another session run one transaction, and asked again. A round on 17.11:

r1 xid=763 right-after='ERROR:  transaction ID 763 is in the future' other-xid=763 then='committed' own-rows=0

The other session received the same xid, and pg_xact_status said committed while our row was gone. That happened in 6 of 8 rounds in the first run and 8 of 8 in the second. The other two rounds said aborted, which was right. Our explanation is that the xid’s WAL had not left PostgreSQL’s buffers, so recovery did not know it and handed it out again. On a rechecked reuse the WAL held only the other session’s records. When the WAL had been written first, for example by another commit, the status was correct and the next xid moved on. We saw the same with an open transaction on 18 (2 of 2) and 13 (1 of 2). Under load it was no better: with four sessions committing and 15 crashes per run, 18 of 33 and 9 of 39 interrupted transactions with a known xid ended up reported as committed while their row was gone. Right after recovery those same transactions gave transaction ID … is in the future, until another session received the xid. The function’s documentation does not mention this. It may be known behaviour, and we have not asked the PostgreSQL developers.

What did work is a key of your own: write a unique value in the same transaction, and after a failure, reconnect and look for it; if the old backend may still be alive, retry with insert … on conflict do nothing instead. That is how the loop above counted its 77.

Why did half my pasted script run in psql?

Because interactive psql reconnects and carries on. We terminated the session in the middle of a transaction:

postgres=*# insert into y values(2);
FATAL:  terminating connection due to administrator command
server closed the connection unexpectedly
The connection to the server was lost. Attempting reset: Succeeded.
postgres=# insert into y values(3);
INSERT 0 1
postgres=# commit;
WARNING:  there is no transaction in progress
COMMIT

The table then held only 3: row 1 was rolled back, row 2 never ran, and row 3 went in outside any transaction. The prompt shows it, =*# became =#. Non-interactive psql stops with exit code 2, which the documentation describes as “the connection to the server went bad and the session was not interactive”.

What are the other spellings of this error?

With SSL a closed connection usually gets other wording, and that wording depends on the client’s libpq, not on the server. A TCP reset still gave the plain sentence with sslmode=require (psql and psycopg2, libpq 17.11, rechecked 2 November):

Event, sslmode=requirelibpq 17.11 (psql, psycopg2)libpq 18.6 (psycopg 3, psql in the images)
kill -9SSL SYSCALL error: EOF detectedSSL error: unexpected eof while reading
pg_terminate_backend()SSL connection has been closed unexpectedly (psql adds the FATAL)psycopg 3: AdminShutdown, 57P01

The rest: consuming input failed: server closed… is psycopg 3. pg_dump during COPY prints Dumping the contents of table "big" failed: PQgetCopyData() failed. with the sentence under Error message from server:; the Command was: line shows where it stopped. connection to server at …, port 5432 failed: server closed… is the connection phase, as in no pg_hba.conf entry for host, where listen_addresses behind Docker’s port produced it. The codes the server does send are in our measured SQLSTATE overview.

Reproduce it

Three scripts: setup.sh starts the servers and one container per client stack on a private Docker network, with the test password test, and run.sh prints the main client messages and log lines above. Run PREFIX=pgclosed ./setup.sh, then PREFIX=pgclosed ./run.sh 2>&1 | tee results.txt; ./run.sh long adds the 16-minute black hole, and ./setup.sh clean removes everything. xid-load.sh runs the pg_xact_status test under load on its own, with PREFIX=pgclosed ./xid-load.sh. The race and timing numbers will be your own. The scripts cover the main cases, not all of them. Not included: SSL, pg_dump, interactive psql, docker network disconnect, docker restart, the COMMIT runs on 18 and with psql, psycopg 3, pgJDBC and node-postgres, the network COMMIT case, the open-transaction pg_xact_status runs on 13 and 18, the pre-ping gap after checkout, PgBouncer being killed and server_lifetime, node-postgres Pool, and the 57P02 notice handler.

What we did not measure

PostgreSQL 14 and 16, and 12 and older. Failover with Patroni, RDS or Render, and whether an acknowledged COMMIT survives one. Real proxies with idle timeouts such as AWS NLB, RDS Proxy, Azure load balancers and Kubernetes conntrack: our black hole is iptables. Keepalive with a default configuration for the full two hours. pg_xact_status with a replica. pgJDBC socketTimeout, HikariCP, node-postgres keepAlive, pgxpool. Django, Rails, PHP PDO (General error: 7), asyncpg, pgAdmin and Supabase. psql with ON_ERROR_STOP. SQLAlchemy under parallel load, and pool_recycle. pg_ctl stop -m smart. Which libpq version made psycopg2 drop the FATAL: we tested wheels, not libpq separately.

Sources: PostgreSQL 18 documentation on the protocol message flow, system information functions, shutting down the server, connection settings, client connection defaults, libpq connection strings, error codes, psql, asynchronous commit and kernel resources, read on 30 October 2026; the libpq source (fe-misc.c, fe-secure.c, fe-secure-openssl.c) and postgres.c on branch REL_18_STABLE, read the same night. All measurements are our own, on PostgreSQL 13.23, 15.19, 17.11 and 18.6 in containers, in the night of 30 to 31 October 2026; the psycopg2 wheel comparison and the repeat run later that night; the SSL reset, PgBouncer log and pg_xact_status rechecks on 2 November 2026.

Let's talk

What are we building?

One conversation is enough to know whether we fit. Tell us what you have in mind — we will tell you how fast it can happen, and whether we are the right people for it.

Start the conversation