Idempotent processing of data streams
摘要
說(shuō)明書
Upon retrieving data records 112, data consumer 110 may update a checkpoint intention 114 that is maintained as part of partition checkpoint metadata 120. Partition checkpoint metadata 120 may be any information maintained to indicate a progress point or state of processing for the data stream partition. For example, in at least some embodiments, a checkpoint metadata may include a number of other indicator that identifies a location in the sequence of data records in the partition up to which data records have been processed (e.g., a sequence number 149,232 which indicates that data records with sequence numbers in the partition<=149,232 have been processed). The update to checkpoint metadata 114 may provide some indication of an intention to process data records 112 up to the location identified in the checkpoint metadata 120. For example, a particular flag, value, field, or other indicator may be set to identify the checkpoint value as intentional so that it may be determined based on the checkpoint metadata whether processing of the data records is pending or has been completed. In at least some embodiments, checkpoint metadata may indicate information for delivering the processed data records (e.g., a storage location in a data store).
