remove_temp_files_after_crash is a disk-space protection filed under debugging. The manual lists it in Developer Options, the section whose preamble tells you never to use any of this on a production database, and for once the warning is exactly right: the default, on, is the production setting, and the only thing the parameter lets you do is turn the protection off. It is a boolean, context sighup, flagged GUC_NOT_IN_SAMPLE, so there is no commented-out line for it in postgresql.conf. It arrived in 14 and has not changed since.
What it protects you from is what a backend crash does to temporary files. A sort, hash, or tuplestore that outgrows work_mem goes to disk under pgsql_tmp (the log_temp_files post has the longer version), and the backend that created the file is the one that deletes it, either when the query is done with it or, failing that, from an on-exit callback when the process ends. (For the shared filesets of a parallel query that job falls to the last participant to finish with them; same idea.) A process that dies from a signal runs no callbacks. And in PostgreSQL the crash of one backend is the crash of all of them: the postmaster sends SIGQUIT to every other child, and the handler, quickdie(), leaves via _exit(2) without running any cleanup, deliberately, because shared memory may be corrupt and the exit path can’t be trusted. So the moment one backend is killed, every temporary file on the server is orphaned: every session’s spill files, and the storage for every session’s temporary tables as well. I checked this on 18.6 and the 19 release candidate with a bystander session holding a cursor over a spilled sort and a temp table; its files survived the other backend’s SIGKILL exactly as the victim’s did.
Before 14, those orphans stayed on disk until the next time the postmaster started. Postmaster startup runs RemovePgTempFiles() unconditionally, and has since long before 14, on the grounds that nothing else can be running in the data directory. But with restart_after_crash at its default of on, a crash doesn’t go through startup; the postmaster reinitializes in place, and the files were left where they lay. The stated reason was that someone might want them for debugging. The actual effect was that a server which crashed under a heavy analytic load, which is to say precisely when the largest temporary files exist, kept all of them, and a second crash doubled the pile, until something hit ENOSPC. The commit message says as much, and adds that nobody had ever been seen doing the debugging. [credential aside, confirm or cut: I have cleaned up after exactly this on a pre-14 server: a reporting job the OOM killer took out twice in one week, and tens of gigabytes of sort files that nobody could account for.]
Nobody could account for them because the orphans are invisible to the instrumentation. pg_stat_database.temp_files and temp_bytes, and the log_temp_files line, are all recorded by the owning backend at the moment it deletes the file; a backend that never got to delete it never reports it. pg_ls_tmpdir() shows the plain spill files in the default tablespace (pass a tablespace OID for the others), but it skips directories, so the .fileset trees that parallel queries leave behind are invisible to it, and temporary-table storage lives in the database directories rather than under pgsql_tmp at all. For the whole picture you need a shell. Either way, a pgsql_tmp<PID>.<N> file, or a pgsql_tmp<PID>.<N>.fileset directory, whose PID matches nothing in pg_stat_activity is an orphan.
With the parameter on, the postmaster deletes all of that during the reinitialization, right after the “all server processes terminated; reinitializing” line and before crash recovery begins. Two things about that matter before you ever consider off.
The first is that off buys less than it appears to. It retains the files only until the next postmaster start, because the startup path doesn’t consult this parameter and never has. If you turn it off to capture a crashed backend’s temporary files, copy them somewhere else before anyone restarts the service, including the clusterware. And if restart_after_crash is off, the postmaster exits on a crash and this parameter is never read at all; the files sit there until the next start deletes them.
The second is that the deletion runs single-threaded in the postmaster while the server is down. A report from AWS on pgsql-hackers in September 2021, while 14 was still in beta, described cases in its fleet of hundreds of thousands of temporary files, and one customer with a little over 2.2 million, where cleanup at startup had already delayed recovery badly enough that RDS’s workaround was to move pgsql_tmp aside before starting the server; the worry was that 14’s new default would impose the same stall on every crash reinitialization. In my sandbox, on ext4, 300,000 planted leftovers cost the postmaster 2.6 seconds, and those were empty files on local disk. Turning the parameter off does make the reinitialization fast again, but only by leaving the files for the next postmaster start to delete, unconditionally, with the same stall. It moves the stall; it doesn’t remove it. Millions of temporary files is a hash join whose batch count has exploded, or a work_mem small enough that every connection is spilling, and the parameters for that are hash_mem_multiplier and work_mem. If the server has to come up before you can fix that, do what RDS did: move pgsql_tmp aside and delete it at your leisure.
Leave it on. If you have a specific crash to dissect, turn it off, reload, reproduce the crash, copy the files out, and turn it back on. If you are turning it off because the cleanup is slow, the file count is the problem, and this parameter only decides which restart pays for it.