Anyone who has run PostgreSQL for long enough has read this, usually at a bad moment:
1 WARNING: terminating connection because of crash of another server process 2 DETAIL: The postmaster has commanded this server process to roll back the current transaction and exit, because another server process exited abnormally and possibly corrupted shared memory. 3 HINT: In a moment you should be able to reconnect to the database and repeat your command.
restart_after_crash decides whether the HINT is telling the truth.
It is a boolean, default on, context sighup, so a reload changes it and no restart is needed. It arrived in PostgreSQL 9.1, and it is the reason the manual has an “Error Handling” section at all: Robert Haas’s July 2010 commit created that group to hold it, and documented exit_on_error for the first time while he was in there.
What the manual calls “a backend crash” is a backend ending with any exit status other than 0 or 1, or ending by signal. Status 1 is how a FATAL ends. pg_terminate_backend() sends SIGTERM, the backend logs “terminating connection due to administrator command,” exits with 1, and nothing else happens. Any other exit code, and any death by signal, is a crash: kill -9, the OOM killer, a segfault, and a PANIC from anywhere in the server, since PANIC ends in abort() and abort() is SIGABRT.
In production the usual crash is the OOM killer, and with the postmaster at the recommended oom_score_adj of -1000 its victim is a backend, which makes an out-of-memory event this entire cycle rather than one lost session. The core auxiliary processes are held to a stricter standard (the checkpointer, background writer, WAL writer and autovacuum launcher are not allowed even a FATAL), and kill -9 on the checkpointer produces exactly the same cycle as kill -9 on a backend. The postmaster logs who died and how, and for a backend adds a DETAIL line naming the query it was running, which is the most useful line in any crash investigation and the one to read first:
1 LOG: client backend (PID 4687) was terminated by signal 9: Killed 2 DETAIL: Failed process was running: select pg_sleep(600) 3 LOG: terminating any other active server processes
The third line is the part that surprises people. The dead process may have been holding a lock, or halfway through an update to shared memory; the postmaster cannot know, so it assumes the worst and sends SIGQUIT to every other child (the logger excepted, so that the messages have somewhere to go). They exit through quickdie() without cleaning up, which is the reason remove_temp_files_after_crash exists, and they get five seconds to go, after which the postmaster logs “issuing SIGKILL to recalcitrant children” and stops asking. Every connection is gone, including the ones that were doing nothing.
What happens next is this parameter. With on, the postmaster logs “all server processes terminated; reinitializing,” throws away shared memory and the semaphores and rebuilds them, re-reads pg_control, and launches a startup process, which runs crash recovery exactly as if the machine had been power-cycled: the data directory sync from recovery_init_sync_method, replay from the last checkpoint, an end-of-recovery checkpoint, and then “database system is ready to accept connections.” The postmaster itself never went anywhere. Its PID is unchanged, postmaster.pid was never removed, and pg_postmaster_start_time() still reports the original start, so a health check that asks “is the postmaster up?” is told yes throughout. On an idle test cluster the whole cycle took about 120 milliseconds on 18.6 and on the 19 release candidate. On the same cluster after writing 7 GB of WAL since its last checkpoint, the replay alone took 49 seconds, because the cost of a crash restart is the cost of crash recovery, and that is governed by checkpoint spacing and max_wal_size, not by anything in this section of the manual.
With off, the postmaster does everything up to the reinitialization, then logs “shutting down because “restart_after_crash” is off” and exits with status 1. postmaster.pid is gone, pg_controldata reports a cluster state of “in production” with no server running, and the HINT every client just received about reconnecting in a moment was a lie: nothing comes back until something starts it, and crash recovery runs as part of that start instead. The init system will not be the something. Debian’s postgresql@.service ships with Restart= commented out, with a note giving two reasons: it would break pg_ctlcluster stop, and the postmaster restarts itself on most problems anyway. The Red Hat unit has no Restart= line at all. Neither packager will start the postmaster for you.
The documented reason to turn it off is clusterware: let the cluster manager see the postmaster disappear and take whatever action it deems appropriate, which in practice means choosing between recovering in place and failing over. That rationale came from Fujii Masao during review. The patch itself began as something else. The original thread was about using PostgreSQL as a disposable in-memory cache, and Tom Lane’s proposal was a startup script that would rm -rf $PGDATA, copy in a prepared image, and loop; all it needed from the server was “a tweak to make the postmaster exit(1) after a crash.” Haas wrote the patch, Simon Riggs talked him out of calling it automatic_restart, and a parameter invented for a database that had given up on durability shipped as a parameter for high availability.
Sixteen years later the clusterware does not want it. Patroni, PAF, the Pacemaker pgsql agent and pg_auto_failover all leave restart_after_crash alone and watch the postmaster from outside. Patroni’s primary_start_timeout (default 300 seconds) is precisely the policy “how long do we let a crashed primary recover in place before we fail over,” and it sits on top of the postmaster’s own restart rather than replacing it. The one place the PostgreSQL project turns the parameter off is its own test suite: PostgreSQL::Test::Cluster writes restart_after_crash = off into every instance it creates, which means a crash fails the test instead of healing quietly behind it, and the few tests that are about the restart turn it back on by hand. That is the honest use case. A development or CI server where a crash should stop the world and leave the wreckage for inspection: turn it off there.
There is one thing on will not do, which is loop. If the startup process itself dies during crash recovery, the postmaster logs “shutting down due to startup process failure” and exits regardless of this setting, on the theory that it would most likely just fail the same way again. I tested that by SIGKILLing the startup process mid-replay. The next manual start opened with “database system was interrupted while in recovery” and a HINT about corruption and restoring from the last backup, then replayed to the end without incident, because nothing was corrupted; recovery had merely been killed. Know that HINT before you meet it in an incident.
One note for the 19 release candidate. The five seconds of patience before the SIGKILL is currently up to a minute. Commit d1c01b79, the automatic I/O worker pool, reworked the postmaster’s sleep calculation so that the countdown is honored only during an immediate shutdown, and left itself a comment asking whether the crash case should count too. It should. On 19rc1, a crash with one child that could not respond to SIGQUIT took 60 seconds to reach the SIGKILL on an idle cluster, against five on 18.6; any connection attempt wakes the postmaster and it catches up, so a busy server would rarely show it, and an idle one with a stuck backend would be down for a minute.
What on costs you is visibility, and that is the thing to fix. A crash restart leaves surprisingly little trace outside the log. pg_postmaster_start_time() does not move. Crash recovery discards the cumulative statistics: the cluster-wide views (pg_stat_bgwriter, pg_stat_checkpointer, pg_stat_wal, pg_stat_io) come back with a fresh stats_reset stamped at the moment of the crash, which is the one fingerprint you can query for, while every counter in pg_stat_database and pg_stat_user_tables simply drops to zero with stats_reset still NULL. Autovacuum’s dead-tuple and insert counts go with them, so the tables that were due for a vacuum before the crash are, as far as the threshold logic can tell, now clean (wraparound vacuums are driven by the catalogs and are unaffected). If your monitoring does not read the server log, start with “terminating any other active server processes.” It is the line that opens a crash cycle, and it is logged for no other reason.
Leave it on. The restart it performs is the one you would perform by hand, done before you could type the command, and the only thing it takes from you is the chance to be the one who noticed. Get that from the log.