ClickHouse released WalShadow last week, and it does something very… brave: it replicates PostgreSQL into ClickHouse without using logical decoding at all. It reads the physical WAL stream, the same bytes a streaming replica gets, decodes heap records itself, and writes ClickHouse-native blocks. The latency and throughput numbers they published are good, and I believe with them.
The interesting part is not the benchmark. It is the bill of materials.
Logical decoding is not a thin convenience layer over the WAL. It is a substantial pile of machinery that exists to answer questions the WAL does not answer on its own, and if you decline to use it, you inherit every one of those questions. WalShadow’s repository is honest about this, which is why it makes such a good teaching example. Read their plans/ directory and docs/limitations.md and you are essentially reading a from-scratch reimplementation of snapbuild.c, reorderbuffer.c, and the parts of heapam.c that nobody thinks about until they are wrong.
So let’s go through what you would be signing up for. Then let’s talk about why ClickHouse can sign up for it and you can’t.
What the WAL does not tell you
A WAL record says “this tuple, on this page, in this relfilenode.” It does not say what the columns are called, what types they have, or whether the transaction that wrote it ever committed. Four separate mechanisms in PostgreSQL turn that into a change stream you can act on.
The catalog, as of then
To interpret a heap tuple you need the tuple descriptor for the relation as it existed when the record was written, not as it exists now. A transaction that does ALTER TABLE and then INSERT writes tuples that only make sense against an intermediate version of the catalog which was never visible to anyone else and does not exist anymore.
PostgreSQL solves this with historic snapshots. From snapbuild.c:
We build snapshots which can only be used to read catalog contents and we do so by reading and interpreting the WAL stream. The aim is to build a snapshot that behaves the same as a freshly taken MVCC snapshot would have at the time the XLogRecord was generated.
The hard part is not the snapshot, it is command IDs. cmin and cmax are not written to WAL, they are reset on crash recovery, and combo CIDs live only in the memory of the backend that created them. So at wal_level = logical, heapam emits an extra record, XLOG_HEAP2_NEW_CID, every time a catalog row is modified, carrying the cmin and cmax. Decoding stuffs the ctid to (cmin, cmax) mappings into the reorder buffer and uses those at visibility check time instead of what is on the tuple. There is a function called ResolveCminCmaxDuringDecoding() and it exists because there was no cheaper way.
This is the single most underappreciated thing logical decoding does for you. Mixed DDL/DML transactions are not an edge case; every migration tool generates them.
Transactions, reassembled
The WAL interleaves. Records from a dozen concurrent transactions land in LSN order, subtransactions are not linked to their parents until commit or an explicit assignment record, and a transaction can be several times larger than your available memory. reorderbuffer.c:
This module gets handed individual pieces of transactions in the order they are written to the WAL and is responsible to reassemble them into toplevel transaction sized pieces. When a transaction is completely reassembled - signaled by reading the transaction commit record - it will then call the output plugin (cf.
ReorderBufferCommit()) with the individual changes.
It splices subtransactions together with a binary heap keyed on each substream’s smallest current LSN, spills the transaction consuming the most memory to disk when logical_decoding_work_mem is exceeded, and hands the output plugin a complete transaction at the commit boundary. By default you get all of a transaction or none of it, in commit order. (Streaming of in-progress transactions is opt-in, and you have to ask for it.)
TOAST
Out-of-line values arrive as chunks in the TOAST relation, separately from the heap tuple that points at them. reorderbuffer.c again:
When a new (or initial) version of a tuple is stored in WAL it will always be preceded by the toast chunks emitted for the columns stored out of line. Within a single toplevel transaction there will be no other data carrying records between a row’s toast chunks and the row data itself.
So newly-written values are reconstructible from the WAL, and ReorderBufferToastAppendChunk() and ReorderBufferToastReplace() do the reconstruction. The case that bites is an UPDATE that does not touch a TOASTed column. Nothing is rewritten, so the chunks are not in the stream, and the pointer in the new tuple refers to data that was written at some arbitrary point in the past.
Logical replication declines to solve this, and it is right to. In the TupleData message part, one of the per-column submessages is the single byte 'u', documented as “identifies unchanged TOASTed value (the actual value is not sent).” The subscriber is a row-per-key replica, so it already holds the old value and simply leaves that column alone.
Hold onto that. It matters in a minute.
The start point
None of the above gets you an initial copy. PostgreSQL’s answer is the exported snapshot: create a logical slot on the replication protocol and the server hands back a snapshot identifier which, per the docs, “will show exactly the state of the database after which all changes will be included in the change stream.” You SET TRANSACTION SNAPSHOT to it, dump at your leisure, and the slot picks up with no gap and no overlap. Underneath, snapbuild walks a state machine from START through BUILDING_SNAPSHOT and FULL_SNAPSHOT to CONSISTENT, driven by xl_running_xacts records, and only replays transactions that commit after CONSISTENT is reached. That is what makes the seam exact.
The slot itself is the part everyone complains about, with reason. It pins WAL, it pins catalog_xmin so VACUUM cannot remove catalog rows the decoder still needs, and the documentation’s own warning is that in extreme cases a forgotten slot can cause the database to shut down to prevent transaction ID wraparound. That cost is what WalShadow is trying to get away from.
What WalShadow does instead
Roughly, and I am compressing a lot of careful work into a paragraph: it runs a second PostgreSQL of the same major version as the source, bootstrapped from a base backup and fed a filtered WAL stream by WalShadow’s own walsender, with a PGXS module loaded through shared_preload_libraries. That shadow instance is the catalog authority. At schema boundaries the pipeline holds publication until the shadow has replayed to an exact position, reads descriptors out of it, and persists them; the parallel Rust decoders then work from that versioned descriptor history rather than touching the shadow for every record. An explicit buffer with a disk spill directory reassembles transactions, an acknowledgement collector holds the durable cursor back until every earlier commit has landed, and an optional TOAST mirror persists chunks so older values can be served later. Auto-created destination tables get _lsn, _xid, _commit_ts, and _is_deleted appended and are keyed on the source replica identity, so ClickHouse converges by keeping the highest _lsn per key.
That is a coherent design. The repository also opens with a warning block labelling itself Experimental, project status Development Preview, under active development, interfaces and behavior may evolve. It is also, item for item, the list from the previous section.
The announcement says “WalShadow doesn’t use Postgres logical replication” and a reader can take that further than it goes. From docs/limitations.md, WalShadow requires:
wal_level = logical- every replicated table needs usable replica identity
Those are the same two things logical replication requires, and for the same reason. wal_level = logical is what makes XLogLogicalInfoActive() true, which is one of the conditions in RelationIsLogicallyLogged(), the test heapam uses to decide whether to write, in the words of the comment in rel.h, “enough information to extract the data from the WAL stream.” That extra information includes the old tuple’s replica-identity columns on UPDATE and DELETE, which is how any key-addressed consumer knows which row a DELETE removed. WalShadow is not escaping logical decoding’s WAL format. It is escaping logical decoding’s decoder while still consuming logical decoding’s WAL. Those are different claims, and only the second one is true.
The WAL retention problem does not entirely go away either. docs/operations.md recommends a physical replication slot for routine deployments, and if you do without one, you are sizing wal_keep_size or keeping a continuous archive against worst-case backlog, because missing source WAL stops replication rather than skipping data. A physical slot pins WAL the same way a logical one does. What you save is the catalog_xmin half of the problem, which is worth having and is not the same as saving the whole problem.
And there is a type-system dependency that clarifies the whole exercise. WalShadow has its own codecs for the common types, but jsonb, arrays, hstore, enums, ranges, domains, and extension types are converted by the shadow PostgreSQL, one request per insert batch. During greenfield bootstrap there is no shadow yet, so it starts a throwaway PostgreSQL from the source schema to do the conversions, which requires pg_dump and the ability to install the source’s extensions on the daemon host. If you cannot install your source’s extensions on your replication daemon’s host, bootstrap stops.
A WAL decoder that can stand alone does not exist here, and I do not think one can. Half the type system is defined by C functions in the server.
The failure mode is silent divergence
Here is where I want to be careful, because the temptation is to read the following as a hit list. It isn’t. Almost all of it comes from ClickHouse’s own documentation, written down voluntarily, at a level of specificity most vendors never approach. The first line under “Production readiness” in plans/INDEX.md is “Start with small guards against silent divergence and tests of restart behavior,” which tells you they know exactly where they are.
On initial load, from plans/bootstrap.md:
Backup bootstrap can finish while transactions affecting walked tuples remain open. Replaying from oldest buffered WAL record covers changes inside backup window, but earlier inserts can remain absent and earlier deletes can remain visible. Increasing handoff wait reduces exposure without proving completeness
Read that twice (I had to). The initial copy can be missing rows that exist, and can contain rows that were deleted, and the current mitigation is a timeout (--bootstrap-wind-down-secs, default 5) which they correctly decline to describe as a proof. This is precisely the problem the exported snapshot and the SNAPBUILD_CONSISTENT state machine solve, and solving it is most of what snapbuild.c is.
On large values, from docs/limitations.md:
Reused TOAST value IDs can leave ambiguous generations in backup chunk mirrors when hint bits do not prove older chunks dead. CTID repair fixes baseline rows, but later unchanged-pointer updates still depend on those mirrors. Chunk lookup orders by
(ver, blkno, offnum)for determinism, not generation correctness: a newer generation at a lower TID can lose when versions tie.
That is the unchanged-TOAST case coming due. Logical replication does not have this bug because it never has to resolve a stale pointer; the subscriber holds the row. Once your destination is an append-only versioned table instead of a row-per-key replica, you own the pointer, and TOAST value IDs get reused.
The docs are honest about how bad this is. The default [toast] mode = "disabled" fills values it cannot recover with null or the column default and bumps a counter, and a type conversion the shadow cannot perform stops the batch and names the column and row. Those are loud. The quiet failure is the one above: turn on the ClickHouse TOAST mirror, reuse a value ID, tie the versions, and you get a column holding a plausible older value with nothing to tell you.
Also documented: ordering is per-table, so “updates from different tables inside one PostgreSQL transaction may become visible in ClickHouse at different moments.” No cross-table transaction boundary at the destination, which is the guarantee reorderbuffer.c exists to provide. Sequence state is not replicated, though values already sitting in table rows are. Prepared transactions are not production-ready. Non-default tablespaces are unsafe for bootstrap. Unplanned primary promotion is not supported, and continuing across an unexpected timeline switch is listed as future work.
And two sentences that should govern how anyone evaluates this, both theirs. From docs/limitations.md: “Unsupported behavior is not uniformly rejected at startup.” From plans/INDEX.md: “A documented limitation does not imply code rejects it.”
Who can afford an unstable interface
Everything above is the mechanics argument, and mechanics can be finished. Given enough engineering the carry file gets written, the visibility gate gets its proof, the TOAST generations get resolved. I would not bet against them.
The structural argument does not get finished, and it is the one that should decide this for you.
The logical replication wire format is a supported interface. It has a chapter in the documentation, down to the byte. It has version numbers, and the documentation says things like “this field is available since protocol version 2” and “available since protocol version 4,” which is what a contract looks like when someone means it. The output plugin API on the server side is documented alongside it, and the server refuses to load a plugin built against the wrong major version rather than reading it wrong.
The physical WAL record layout and the on-disk heap tuple format are not interfaces at all. They are implementation. XLOG_PAGE_MAGIC in PostgreSQL 18 is 0xD118, and the comment next to it in xlog_internal.h says “can be used as WAL version indicator,” which is the polite way of saying that a reader’s job is to notice it does not understand the bytes and stop.
It is not a hypothetical risk. Look at what has happened to TOAST in the last nine months. In January, Michael Paquier committed oid8, a 64-bit unsigned identifier type, which came out of the thread to add support for 64-bit TOAST values. Then on September 7, on the development branch, after PostgreSQL 19 had already branched off, he renamed varatt_external to varatt_external_oid, taking VARTAG_ONDISK to VARTAG_ONDISK_OID, TOAST_POINTER_SIZE to TOAST_OID_POINTER_SIZE, and TOAST_MAX_CHUNK_SIZE to TOAST_OID_MAX_CHUNK_SIZE with it. The commit message says why: it is “part of a conversation to add support for more types of external TOAST pointers, not only OID.”
The structure a physical WAL decoder has to parse to find a TOAST value is being reshaped to hold more than one kind of pointer. In public, with review, correctly, and with no deprecation period, because there is nothing to deprecate. Nobody was ever promised anything.
If you want to know how much history is compressed into these bytes, look at the tag. In varatt.h, VARTAG_ONDISK is defined as 18, and the comment explains that “the peculiar value for VARTAG_ONDISK comes from a requirement for on-disk compatibility with a previous notion that the tag field was the pointer datum’s length.” The constant is the number 18 because an on-disk TOAST pointer happens to be eighteen bytes long and once upon a time something depended on that. Every physical decoder you write has to know this. PostgreSQL’s own decoder gets it for free, because it is the same code that wrote the bytes.
There is a second-order version of this that is worse. xlog_internal.h exports RegisterCustomRmgr(). Any extension your source database installs can define its own resource manager and write its own record types into your WAL. PostgreSQL’s decoder dispatches those through rm_decode and knows what to do. A third-party decoder meets bytes from a resource manager that did not exist when it was compiled.
ClickHouse knows this, and their gate says so better than I can. From docs/limitations.md: PostgreSQL 16, 17, and 18 are supported, and the daemon rejects unaudited majors. The shadow’s major version must equal the source’s. Preflight checks server_version_num at boot and refuses to start.
In real-life operations? Your ability to upgrade PostgreSQL is now downstream of somebody else’s audit schedule. PostgreSQL 19 Beta 4 is due on September 24 and the release team’s stated goal is GA by the end of October. If you are running this against a 19 source in November, you are either waiting for an audit or running with --skip-preflight, whose own help text reserves it for recovery drills. You cannot upgrade a major version until the decoder has been taught the new format, and you do not control when that happens.
You can see the shape of that audit in the repository. docs/development.md has a section on WAL fixtures, and the instruction is to regenerate them “using PostgreSQL major being tested,” with a capture script per fixture class. That is the right way to build this. It is also a per-major-version engineering commitment that somebody has to keep making, forever, for every version you might want to run.
This is, in fact, one of the reasons that we still have a major song-and-dance for each major version upgrade: maintaining stability and backwards compatibility is a huge amount of work, and someone has to do that, and a project like PostgreSQL lacks that “someone.”
A vendor who ships both the source server and the consumer can carry it. They pin the major version, they recapture the fixtures and audit the WAL format before shipping a new one, and they decide when their customers upgrade. The audit gate is not a limitation for them. It is a normal release step.
You do not have that. Your Postgres major version is driven by EOL dates, your cloud provider, a security advisory, or a CVE that lands on a Tuesday. Coupling your analytics pipeline’s correctness to a byte format that the PostgreSQL project has never promised you anything about, and that you will be forced to change on someone else’s schedule, is a bad trade at any latency.
What to do
If you need Postgres in ClickHouse, use logical decoding. The slot management is annoying and the throughput ceiling is lower and you will occasionally get paged because somebody left a slot behind on a decommissioned consumer. Pay it. What you are buying is a supported interface, a correct initial snapshot, transaction boundaries, and TOAST reconstruction that cannot silently hand you last month’s value.
If you are evaluating WalShadow specifically, the question to ask is not about the benchmark. It is: who is going to own the physical WAL decoder when you upgrade? If the answer is a vendor who also controls your server version, that is a real answer. If the answer is you, read docs/limitations.md end to end first, and then read plans/coordination.md, and notice that both documents are better written than most of the code you are comparing them against. That is not an accident. People write documents like that when they have found out the hard way how many invariants are in play.
And if you are thinking about writing your own physical WAL reader because logical decoding is too slow for your workload: snapbuild.c is about 2,000 lines, reorderbuffer.c is about 5,600, and neither of those numbers includes the part where you are wrong about TOAST for eighteen months and nobody notices.