This link got posted slightly earlier to HN as well. Is there a particular trigger that made this one year old document so relevant that it needs to be brought to people’s attention on both platforms independently? Or is this a cheap bot resubmitting links that work on HN?
Another great writeup from Jack V. Would love to see this rendered more nicely as a gitbook, mdbook, asciidoc or something similar.
Just a random aside related to metadata replication amongst the kraft controllers…notice how they consolidate the changes into snapshots every so often to shorten the time it takes for controllers to rehydrate when they come back online. I think there is something fundamental here we can learn about stream processing and stream / table duality.
Today, we have change stream like from debezium CDC, where every record is a change and you have to read the initial snapshot (which is really a set of changes) + all subsequent changes to get the current state. The initial snapshot happens once, and from there, you just have changes. There is no ongoing maintenance of the snapshot over time.
And then we have compact topics, where we don’t model changes but rather do updates and deletes. For example, Kafka streams has compact changelog topics. Every new change is really thought of as an update or delete, and old records are garbage collected (compacted) away. This is like incremental maintenance of a snapshot.
But the kraft controllers do something a little more sophisticated than either of these. They are modeled as snapshot + recent changes. The snapshot gets updated periodically. If the architects of kraft thought this was a good design decision, wouldn’t it be a good approach for other event-driven applications too?
I know Materialize does something like this in its storage layer. For the direct Postgres source, there’s an initial snapshot and then a sequence of “diffs”. Then there’s an ongoing compaction mechanism to consolidate older diffs.
I wonder if Fluss is built like this as well, with these kind of periodically / continuously maintained snapshots. I don’t know Fluss very well yet.
I’d love to hear others’ experience and insights on this idea
pillenpopper@reddit
This link got posted slightly earlier to HN as well. Is there a particular trigger that made this one year old document so relevant that it needs to be brought to people’s attention on both platforms independently? Or is this a cheap bot resubmitting links that work on HN?
kabooozie@reddit
Another great writeup from Jack V. Would love to see this rendered more nicely as a gitbook, mdbook, asciidoc or something similar.
Just a random aside related to metadata replication amongst the kraft controllers…notice how they consolidate the changes into snapshots every so often to shorten the time it takes for controllers to rehydrate when they come back online. I think there is something fundamental here we can learn about stream processing and stream / table duality.
Today, we have change stream like from debezium CDC, where every record is a change and you have to read the initial snapshot (which is really a set of changes) + all subsequent changes to get the current state. The initial snapshot happens once, and from there, you just have changes. There is no ongoing maintenance of the snapshot over time.
And then we have compact topics, where we don’t model changes but rather do updates and deletes. For example, Kafka streams has compact changelog topics. Every new change is really thought of as an update or delete, and old records are garbage collected (compacted) away. This is like incremental maintenance of a snapshot.
But the kraft controllers do something a little more sophisticated than either of these. They are modeled as snapshot + recent changes. The snapshot gets updated periodically. If the architects of kraft thought this was a good design decision, wouldn’t it be a good approach for other event-driven applications too?
I know Materialize does something like this in its storage layer. For the direct Postgres source, there’s an initial snapshot and then a sequence of “diffs”. Then there’s an ongoing compaction mechanism to consolidate older diffs.
I wonder if Fluss is built like this as well, with these kind of periodically / continuously maintained snapshots. I don’t know Fluss very well yet.
I’d love to hear others’ experience and insights on this idea