Full CDC semantics land in the Iceberg output for Redpanda Connect

Your operational database is the source of truth for the business, right up until you copy it into your lakehouse, where it quietly goes stale. The moment an order is updated or a customer record is deleted, your analytics table starts telling a story that’s no longer true. And that’s an expensive place to be for anything that depends on the current state, like fraud checks and inventory counts.

The usual way to keep that table honest is a pile of moving parts: an Apache Kafka^® Connect cluster, a Debezium source connector, a transform step to flatten Debezium’s envelope into something usable, and a separate Iceberg sink connector. Four services, four configs, four ways for 2 AM to go badly. All just to keep one table in sync with one database.

Back in March, we shipped the Iceberg output for Redpanda Connect, which is a native way to land any stream into an Iceberg table without sidecar services or a Flink job hiding in the corner. It was great at one thing: appending. Once a row landed, it was there for good. That was fine for pure event streams, but not for change data capture, where the whole point is that rows get updated and deleted, sometimes seconds after they're created. If your customer's database can change its mind, your lakehouse table should be allowed to do the same.

With Redpanda Connect v4.99.0, it can. The iceberg output now supports row-level operations, like insert, upsert, and delete, so a CDC stream can be applied to an Iceberg table the same way it would be applied to the database it came from.

The two fields that make it work

• row_operation: an interpolated field you set per message to insert, upsert, or delete. The default remains insert, so every pipeline you already have keeps humming along exactly as before.
• identifier_fields: the equality-delete key, i.e., how Connect knows which row you're talking about when you say "update this one" or "get rid of that one."

Because the mapping is source-agnostic, it doesn't care where the CDC events came from. Point postgres_cdc, mysql_cdc, sqlserver_cdc, oracledb_cdc, or aws_dynamodb_cdc straight at it, or consume CDC events that are already sitting in a Redpanda topic.

input:
  postgres_cdc:
    dsn: ${PG_DSN}
    schema: ${PG_SCHEMA}
    tables: ["orders"]
    slot_name: ${PG_SLOT_NAME}

output:
  iceberg:
    catalog:
      url: https://polaris.example.com/api/catalog
      warehouse: analytics
    namespace: raw
    table: orders
    identifier_fields: ["id"]
    row_operation: |
      root = if meta("operation") == "delete" { "delete" } else { "upsert" }
    max_in_flight: 1 # keyed writes need to stay in order, don't get greedy
    storage:
      aws_s3:
        bucket: my-iceberg-data
        region: us-west-2

Your lakehouse can now mirror the database

The boring engineering win is the collapse in complexity. Where keeping an Iceberg table in sync with a live database used to mean four services and four configs, it’s now a single pipeline definition.

The exciting win is what that does to data freshness. Full CDC semantics in Iceberg mean the lakehouse reflects what the source database looks like right now, not what it looked like during last night’s batch window. And nobody on the customer’s team has to babysit a brittle four-piece connector chain to make that true.

If an analogy helps, picture it this way:

The old append-only Iceberg output was like a checkbook register. Every insert, update, and delete was a fresh line, and to know an account’s real balance, you had to total up the whole log of recent transactions by hand. Row-level operations are like opening a spreadsheet and directly updating a cell's value. When an order ships or a row is deleted at the source, the table just reflects it—no replaying a log, no reconciling of values. Your lakehouse mirrors the database instantly.

Check our CDC catalog

The iceberg output's row-level operations shipped in v4.99.0. Check the component reference for the full field list and the CDC cookbooks for shaping change events before they ever reach the output.

Your database already knows how to update and delete rows. Now your data streaming platform does too.

Happy change-capturing!