System and method for real-time processing of compressed videos
地域：
MA
MA Cambridge
摘要
A real-time system and method for displaying video on a display are disclosed. Received compressed video data is decoded to produce an uncompressed first video frame, a first frame syntax element, an uncompressed second video frame, and a second frame syntax element. A computationally intensive process is applied to the uncompressed first video frame to produce an enhanced first video frame. A block having a portion of the enhanced first video frame from the enhanced first video frame is adaptively transferred to the uncompressed second video frame to produce an enhanced second video frame without applying the computationally intensive process to the uncompressed second video frame. The transferring is guided by the first frame syntax element and the second frame syntax element. The enhanced first video frame and the enhanced second video frame are displayed.
說明書
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
This application is a continuation of U.S. patent application Ser. No. 15/456,788, filed Mar. 13, 2017, entitled System and Method for Real-Time Processing of Compressed Videos, which claims the benefit of U.S. Provisional Patent Application Ser. No. 62/340,203, filed May 23, 2016, entitled “FAST: Free Adaptive Super-Resolution via Transfer for Compressed Videos,” both of which are incorporated by reference herein in their entirety.
The present invention relates to video processing, and more particularly, is related to optimizing video processing by leveraging existing compression parameters.
High resolution displays are garnering increasing popularity. Televisions with 8K resolution (7680×4320 pixels) display are already available for sale, and even a cellphone can have 4K Ultra-HD (high definition) resolution (3840×2160 pixels). Unfortunately, the availability of high resolution media content has not kept up with the increase in display resolutions. For example, one popular video streaming service currently offers only 26 titles in 4K Ultra-HD resolution. In addition, transmitting 4K Ultra-HD video requires high communication bandwidth. Given the abundant amount of existing lower resolution videos, as well as limited communication bandwidth, it would be desirable to up-sample these videos to higher resolution at the display.
權(quán)利要求
What is claimed is:1. A method for displaying video on a display by a real-time video processing device comprising a processor and a memory containing non-transient instructions that, when executed by the processor, comprise the steps of:receiving compressed video data; decoding the compressed video data to produce an uncompressed first video frame, a first frame syntax element, an uncompressed second video frame adjacent to the first video frame in a video sequence, and a second frame syntax element; applying a first video enhancement process to the uncompressed first video frame to produce an enhanced first video frame having a higher resolution than the uncompressed first video frame; applying an adaptive deblocking filter to the enhanced first video frame; applying a second process to the uncompressed second video frame to produce an enhanced second video frame, further comprising the steps of:adaptively transferring a block comprising a portion of the enhanced first video frame from the enhanced first video frame to the uncompressed second video frame according to the first frame syntax element and the second frame syntax element to produce the enhanced second video frame having a higher resolution than the uncompressed second video frame; and displaying the enhanced first video frame and the enhanced second video frame with the display; wherein the first video enhancement process is more computationally intensive than the second video enhancement process, and the second video enhancement process leverages an inter-frame similarity between adjacent frames in the video sequence and transfers an enhancement result from the first enhanced video frame to the second enhanced video frame. 2. The method of claim 1 , wherein the first frame syntax element further comprises a size of the block and a source position of the block, and the second frame syntax element further comprises the size of the block, and a target position of the block. 3. The method of claim 1 , wherein the compressed video data comprises a compressed video bit stream. 4. The method of claim 2 , further comprising the step of determining a spatial relationship between pixels in the uncompressed first video frame and pixels in the uncompressed second video frame based upon the first frame syntax element and/or the second frame syntax element. 5. A method of claim 4 , further comprising the step of avoiding processing to speed up a rendering of the enhanced second video frame based at least in part on the spatial relationship between pixels in the uncompressed first video frame and pixels in the uncompressed second video frame. 6. The method of claim 4 , further comprising the step of combining a first pixel from the uncompressed first video frame with a related second pixel from the uncompressed second video frame for improving an image quality in the enhanced second video frame. 7. The method of claim 6 , wherein the improving the image quality in the enhanced second video frame comprises super resolution enhancement, dynamic range expansion, object recognition and/or detection, optical flow estimation, time-to-contact estimation, tracking in videos, video stabilization, video segmentation, frame interpolation, scene understanding, visual style editing, deblurring, high dynamic range enhancement, and reducing a level of noise in the enhanced second video frame. 8. A method of claim 1 , wherein the first frame syntax element and/or the second frame syntax element comprises one or more of the group consisting of a motion vector, a block structure, a residual and/or prediction error, a transform coefficient, an inter prediction mode, an intra prediction mode, an intra prediction mode direction, a skip flag, a merge flag, a coded block flag, a loop filter parameter, a quantization parameter, and a reference index. 9. The method of claim 1 , further comprising the steps of: decompressing the compressed video data to produce an uncompressed third video frame, and a third frame syntax element; andtransferring a block comprising a portion of the enhanced second video frame from the enhanced second video frame to the uncompressed third video frame according to the second frame syntax element and the third frame syntax element to produce an enhanced third video frame having a higher resolution than the uncompressed third video frame without applying the first video enhancement process to the uncompressed third video frame. 10. The method of claim 9 , wherein the third frame syntax element further comprises the size of the block, and a second target position of the block. 11. The method of claim 1 , wherein applying the first video enhancement process to the uncompressed first video frame to produce the enhanced first video frame further comprises dividing the enhanced first video frame into a plurality of non-overlapping blocks so that each pixel is included by exactly one block of the plurality of non-overlapping blocks. 12. The method of claim 11 , wherein a first block of the plurality of non-overlapping blocks comprises a larger number of pixels than a second block of the plurality of non-overlapping blocks, wherein the pixels of the first block correspond to a simpler structure and/or better-predicted content than the pixels of the second block. 13. The method of claim 1 , further comprising the step of for blocks with zero motion vector and/or zero residual, directly copying the same pixels from the enhanced first video frame to the uncompressed second video frame without interpolation. 14. A method for displaying video on a display by a real-time video processing device comprising a processor and a memory containing non-transient instructions that, when executed by the processor, comprise the steps of:receiving compressed video data; decoding the compressed video data to produce an uncompressed first video frame, a first frame syntax element, an uncompressed second video frame adjacent to the first video frame in a video sequence, and a second frame syntax element; applying a first video enhancement process to the uncompressed first video frame to produce an enhanced first video frame having a higher resolution than the uncompressed first video frame; applying a second process to the uncompressed second video frame to produce an enhanced second video frame, further comprising the steps of:adaptively transferring a block comprising a portion of the enhanced first video frame from the enhanced first video frame to the uncompressed second video frame according to the first frame syntax element and the second frame syntax element to produce the enhanced second video frame having a higher resolution than the uncompressed second video frame; and displaying the enhanced first video frame and the enhanced second video frame with the display; accumulating a metric for a parameter indicating a computation cost change and/or a quality change between the enhanced first video frame and the enhanced second video frame; determining an accumulated metric threshold value; and comparing the accumulated metric to the accumulated metric threshold, wherein the first video enhancement process is more computationally intensive than the second video enhancement process, and the second video enhancement process leverages an inter-frame similarity between adjacent frames in the video sequence and transfers an enhancement result from the first enhanced video frame to the second enhanced video frame. 15. The method of claim 14 , further comprising the steps of:for an enhanced first video frame and an enhanced second video frame where the accumulated metric is below the accumulated metric threshold value, adaptively transferring a block comprising a portion of the enhanced second video frame from the enhanced second video frame to a uncompressed third video frame to produce an enhanced third video frame without applying the first video enhancement process to the uncompressed third video frame for an enhanced first video frame and an enhanced second video frame where the accumulated metric meets or exceeds the accumulated metric threshold value, producing an enhanced third video frame using the computationally intensive process. 16. The method of claim 14 , wherein the accumulated metric comprises an indication of accumulated error, and the accumulated metric threshold comprises an accumulated error threshold.
意見反饋
使用該功能遇到問題