白丝美女被狂躁免费视频网站,500av导航大全精品,yw.193.cnc爆乳尤物未满,97se亚洲综合色区,аⅴ天堂中文在线网官网

Idempotent processing of data streams

專利號(hào)
US11178197B2
公開日期
2021-11-16
申請(qǐng)人
Amazon Technologies, Inc.(US WA Seattle)
發(fā)明人
Gaurav D. Ghare
IPC分類
H04L29/06; H04L29/08; G06F16/23
技術(shù)領(lǐng)域
data,records,stream,checkpoint,may,partition,processing,or,worker,in
地域: WA WA Seattle

摘要

Idempotent processing of data may be implemented for data records retrieved from a data stream. A data stream may receive data records as input and distribute the ingestion, storage, and processing of the data records amongst one or more partitions of the data stream. Partition metadata may be maintained which includes checkpoint metadata for retrieving, processing, and sending data records in the data stream to a specified destination. When assigned a partition for processing, checkpoint metadata for partition may be accessed to determine whether a pending checkpoint for the partition exists. If not pending checkpoint exists, new data records may be retrieved, processed, and sent from the partition of the data stream to a specified destination. If a checkpoint is pending, then the data records identified by the checkpoint metadata as pending may be retrieved, processed, and sent to the specified destination.

說明書

As illustrated in scene 106, a replacement data consumer, replacement data consumer 130, may take over processing of data records from data stream partition 100. Replacement data consumer 130 may first obtain 132 partition checkpoint metadata 120. Based on the partition checkpoint metadata 120, replacement data consumer 130 may determine whether or not the data records retrieved at 112 were emitted. In this way, data records may be processed idempotently so that the data records are only delivered to a destination once. For example, an intention marker, record, or other indicator included in the update 114 to checkpoint metadata may be evaluated to detect whether the data records were emitted. A key-value store, for instance, that maps a stored data value to a unique key, may be the destination for data records emitted from data consumer 130. The checkpoint metadata may include a key value that is intended to be paired with the data record(s) emitted. Replacement data consumer 130 may send a request to the key-value store to check to see if the key exists, and if so, then it may be determined that the data records were already delivered to the key-value store. If, however, the key does not exist, then it may be determined that the data records were not delivered. Note that such a determination may be performed differently according to the destination to which the data records are bound or the marker indicator, or record included in partition checkpoint metadata to indicate the intention to process records.

權(quán)利要求

1
微信群二維碼
意見反饋