Pinterest uses event time watermarks to finalize petabyte-scale data partitions

Pinterest introduced event time-based partition finalization in its database ingestion framework: Apache Flink tracks event time progress while Apache Iceberg metadata publishes a non-regressing finalization watermark. The design addresses late-arriving CDC records in streaming ingestion of petabyte-scale data from MySQL, TiDB and KVStore.

• Watermark is stored as an Iceberg table property, removing the need for a separate coordination service
• Default policy uses the minimum event time across recent commits, rounded down to the hour
• Online changes are typically available within 15 minutes to an hour across thousands of pipelines
• Percentile-based policies trade some completeness for fresher data