白丝美女被狂躁免费视频网站,500av导航大全精品,yw.193.cnc爆乳尤物未满,97se亚洲综合色区,аⅴ天堂中文在线网官网

Idempotent processing of data streams

專利號
US11178197B2
公開日期
2021-11-16
申請人
Amazon Technologies, Inc.(US WA Seattle)
發(fā)明人
Gaurav D. Ghare
IPC分類
H04L29/06; H04L29/08; G06F16/23
技術(shù)領(lǐng)域
data,records,stream,checkpoint,may,partition,processing,or,worker,in
地域: WA WA Seattle

摘要

Idempotent processing of data may be implemented for data records retrieved from a data stream. A data stream may receive data records as input and distribute the ingestion, storage, and processing of the data records amongst one or more partitions of the data stream. Partition metadata may be maintained which includes checkpoint metadata for retrieving, processing, and sending data records in the data stream to a specified destination. When assigned a partition for processing, checkpoint metadata for partition may be accessed to determine whether a pending checkpoint for the partition exists. If not pending checkpoint exists, new data records may be retrieved, processed, and sent from the partition of the data stream to a specified destination. If a checkpoint is pending, then the data records identified by the checkpoint metadata as pending may be retrieved, processed, and sent to the specified destination.

說明書

As indicated at 620, checkpoint metadata for the partition may be updated to indicate a pending checkpoint for the data records. Checkpoint metadata may be updated to indicate a pending checkpoint in many ways. For example, a checkpoint for the data records may be stored as a location or point in the sequence of data records (e.g., a sequence number) which indicates the point in the sequence of data records in the partition of the data stream at which log records have been retrieved and processed. A marker or other indication may also be included in the checkpoint metadata which may indicate that the checkpoint is pending or complete. In some embodiments, the checkpoint metadata may include data to ensure that any processing or emission of data records is overwritten or not repeated. For instance, the delivery destination may be a data store and the checkpoint metadata for the partition may include a location in the data store at which the data records are to be written. Using this location, partially emitted data records may be over written in the same location so that the result of processing the data records utilizing the checkpoint metadata is idempotent (because the state of the data records in the data store is the same based on the first attempt to perform or the later successful attempt to process the records).

權(quán)利要求

1
微信群二維碼
意見反饋