白丝美女被狂躁免费视频网站,500av导航大全精品,yw.193.cnc爆乳尤物未满,97se亚洲综合色区,аⅴ天堂中文在线网官网

Idempotent processing of data streams

專利號
US11178197B2
公開日期
2021-11-16
申請人
Amazon Technologies, Inc.(US WA Seattle)
發(fā)明人
Gaurav D. Ghare
IPC分類
H04L29/06; H04L29/08; G06F16/23
技術(shù)領(lǐng)域
data,records,stream,checkpoint,may,partition,processing,or,worker,in
地域: WA WA Seattle

摘要

Idempotent processing of data may be implemented for data records retrieved from a data stream. A data stream may receive data records as input and distribute the ingestion, storage, and processing of the data records amongst one or more partitions of the data stream. Partition metadata may be maintained which includes checkpoint metadata for retrieving, processing, and sending data records in the data stream to a specified destination. When assigned a partition for processing, checkpoint metadata for partition may be accessed to determine whether a pending checkpoint for the partition exists. If not pending checkpoint exists, new data records may be retrieved, processed, and sent from the partition of the data stream to a specified destination. If a checkpoint is pending, then the data records identified by the checkpoint metadata as pending may be retrieved, processed, and sent to the specified destination.

說明書

As indicated at 730, the data record(s) in the partition of the data stream from the previous checkpoint to the current checkpoint may be retrieved, in some embodiments. One or more retrieval requests, for instance, may be sent to storage nodes or to a stream management system to retrieve the identified data records. As indicated at 740, a determination may be made as to whether the current checkpoint is pending. For example based on the checkpoint metadata. For example, the checkpoint metadata may include a key that indicates a location in a key value data store that serves as a destination for data records in the partition of the data stream. A request may be sent to the key-value data store to determine whether the key value exists. If the key exists, then the key value data store was already received the data records described in the checkpoint metadata (and therefore it may be inferred that reassignment of the partition occurred after completion but before the checkpoint metadata could be updated to indicate that the checkpoint is no longer pending). In some embodiments, other data, markers, indicators, or codes may be included in the checkpoint metadata for the partition which may be evaluated or used to determine whether the data records have been successfully processed and emitted. For example, a transaction id may be in the metadata may be compared with data values in a database table stored in a database that serves as the destination, and if it exists, then it may be determined that the current checkpoint is not pending. The transaction table may be stored in addition to a table that stores the data records retrieved from the data stream.

權(quán)利要求

1
微信群二維碼
意見反饋