Change streams

Every table carries a change log, consumed transactionally

A consumer reads the log and advances its position inside its own transaction, so the position moves only when the consumer commits. APPLY CHANGES lands a stream into a target as SCD type 1 or type 2, pipelines trigger on change data, and on a consensus group the feed is written at the raft index so a consumed position names the same record on every member.

CompleteCREATE CHANGE STREAMChange streams ship in the engine today.
CHANGE FEED · ordersordersINSERT#1041UPDATE#1042UPDATE#1043DELETE#1044INSERT#1045Consumer positionCommit, position advancesorders_history · SCD type 2idamountvalid_fromvalid_toAPPLY CHANGES … AS SCD TYPE 2The position moves only when the consumer commitsOn a Raft group, written at the raft index

Consumption

How a stream is consumed

Every table carries a change log. A consumer opens a transaction, reads a window of entries past its consumed position, does its work, and advances the position in that same transaction. COMMIT lands the work and the new position together. ROLLBACK lands neither, and the next read starts from the same position.

Source tableordersheap or lake, on a branchwrites appendChange log41INSERT42UPDATE43INSERT44UPDATE45DELETE46INSERT47UPDATE48INSERTthe window a consumer readsconsumed positionread inside the transactionConsumer transactionreads and advances in its own transactionBEGINRead a windowAdvance the positionCOMMIT or ROLLBACKCOMMITROLLBACKCOMMITthe position advances with the commit4142434445464748position advancesROLLBACKthe position stays where it was4142434445464748nothing moved, the window is read again

The same transaction that read the window moves the consumed position. A commit lands the consumer's own writes and the new position together, a rollback lands neither, so the window is read again on the next attempt.

Reading and advancing happen inside an ordinary transaction on the engine, with the MVCC and durability story on the database page.

Landing changes

Apply a stream as SCD type 1 or type 2

APPLY CHANGES lands a stream into a target table as a slowly changing dimension. Type 1 keeps the current state of each key, type 2 keeps every version. A pipeline can run the same landing on a change data trigger.

SCD type 1

overwrite

Each change updates the current row for its key, so the target holds the latest state of every key and nothing older.

SCD type 2

versioned rows

Each change becomes a new row for its key instead of replacing the old one, so the target keeps every version of a key alongside its current state.

Pipelines triggered on change data

A declarative pipeline can be triggered ON CHANGE DATA, with CONSUME CHANGES and APPLY CHANGES available as its stages, so landing a stream can be a pipeline definition instead of a consumer written by hand.

ON CHANGE DATACONSUME CHANGESAPPLY CHANGES

The statement surface

CREATE CHANGE STREAMAPPLY CHANGESCONSUME CHANGESON CHANGE DATA

The full syntax for each lives in the SQL reference, which is generated from the grammar registry so an undocumented statement fails the build.

Change streams, pipelines, and the transactions that consume them all run inside the engine, covered on the database page.

Sources and scope

Per branch, from heap or lake, the same position everywhere

A stream is scoped to what it watches. Feeds are kept per branch, a lake table can be the source, and on a group the feed is written at the same index on every member.

Per-branch feeds

Feeds are kept per branch. A stream created on a branch carries that branch's changes, so change data stays with the copy-on-write branch it was written on.

Lake tables as derived sources

A ZyronLake table can be the source of a stream. Change feed between two versions is part of the table format itself, and cross-format federation keeps heap and lake tables first-class in the same SQL.

Written at the raft index

On a consensus group every member runs the same applier over each committed entry and writes the change feed at the raft index, so a consumed position names the same record on every member of the group.

CDC

Change data for everything outside the engine

Change streams are consumed inside the engine. The CDC layer carries change data out to systems that are not Zyron, in the formats their tooling already reads, and to the people who need to hear about it.

Change feeds and replication slots

Change feeds carry the ordered record of change for a table, and replication slots give an external reader a named place in that feed. Both sit behind the same PostgreSQL wire protocol that existing drivers and tooling already speak.

Logical decoders

Change data is decoded into the shape the downstream system reads, with Debezium, Avro, Wal2Json, and a native decoder available.

DebeziumAvroWal2JsonNative

Publications and snapshots

Publications and subscriptions move change data between Zyron instances with exactly-once wire-level push, and snapshots are part of the same CDC surface. Publications are covered in the federation story on the mesh page.

Contact channels

Delivery to people as well as to systems, with contact channels for Slack, Discord, email, and webhook delivery.

SlackDiscordEmailWebhook

Durability

The feed rides the write-ahead log

A change log that could lag the table it describes would be a second source of truth. The feed is made durable by the same log as the write, and comes back with it.

No segment sync per commit

Durability

Change feed durability is carried by the write-ahead log rather than by a segment sync per commit. The commit that produced a change is what makes the change durable, and the WAL already fsyncs through group commit before the client is acknowledged.

Recovery

Recovery replays the write-ahead log with physical page redo, so the feed comes back with the table it belongs to.

The write-ahead log, group commit, and crash recovery are covered on the database page. Change streams are marked complete on the roadmap.

Running in the next five minutes

Prebuilt binaries for Linux and Windows, no external dependencies, one command to a running server.

Download ZyronView on GitHub