白丝美女被狂躁免费视频网站,500av导航大全精品,yw.193.cnc爆乳尤物未满,97se亚洲综合色区,аⅴ天堂中文在线网官网

Idempotent processing of data streams

專利號(hào)
US11178197B2
公開(kāi)日期
2021-11-16
申請(qǐng)人
Amazon Technologies, Inc.(US WA Seattle)
發(fā)明人
Gaurav D. Ghare
IPC分類
H04L29/06; H04L29/08; G06F16/23
技術(shù)領(lǐng)域
data,records,stream,checkpoint,may,partition,processing,or,worker,in
地域: WA WA Seattle

摘要

Idempotent processing of data may be implemented for data records retrieved from a data stream. A data stream may receive data records as input and distribute the ingestion, storage, and processing of the data records amongst one or more partitions of the data stream. Partition metadata may be maintained which includes checkpoint metadata for retrieving, processing, and sending data records in the data stream to a specified destination. When assigned a partition for processing, checkpoint metadata for partition may be accessed to determine whether a pending checkpoint for the partition exists. If not pending checkpoint exists, new data records may be retrieved, processed, and sent from the partition of the data stream to a specified destination. If a checkpoint is pending, then the data records identified by the checkpoint metadata as pending may be retrieved, processed, and sent to the specified destination.

說(shuō)明書(shū)

In at least some embodiments, data processing system 330 may provide staged processing and delivery of data in a data stream. For instance, data processing 330 may maintain a workflow, graph, data structure or other representation of multiple stages of operations to be performed or delivered with respect to a data stream. For example, processing worker node(s) 334 may be directed to perform one or more processing operations for data, send the data to another service (e.g., a storage service, such as object store 240), then send a request to another service, such as distributed processing service 250, to access the stored data in the storage service and perform specified processing operations on the data in a distributed processing cluster. The worker processing node(s) 334 may then direct that the processed data from the distributed processing cluster may be stored in data warehouse service 360. State information concerning the state of staged processing for a data stream may be maintained so that the next step or stage of processing data in the data stream may be identified. The workflow, graph, or data structure may indicate the various accounts, objects, configurations, or other information necessary to perform the desired stages. Processing worker node(s) 334 may be configured to perform the various requests, formatted to respective interfaces, such as APIs, which may instigate processing of the stage. Thus, processing worker node(s) 334 may direct the processing of data in the data stream in addition to performing processing locally, in some embodiments.

權(quán)利要求

1
微信群二維碼
意見(jiàn)反饋