Framework for video conferencing based on face restoration
地域：
CA
CA Palo Alto
摘要
There is included a method and apparatus comprising computer code configured to cause a processor or processors to perform obtaining video data, detecting at least one face from at least one frame of the video data, determining a set of facial landmark features of the at least one face from the at least one frame of the video data, and coding the video data at least partly by a neural network based on the determined set of facial landmark features.
說明書
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
The present application claims priority to provisional application U.S. 63/134,522 filed on Jan. 6, 2021 which is hereby expressly incorporated by reference, in its entirety, into the present application.
This disclosure relates to video conferencing involving face restoration (or face hallucination) that may recover realistic details from a real low-quality (LQ) face to a high-quality (HW) one based on landmark features.
The international standardization organizations ISO/IEC/IEEE are actively searching for AI-based video coding technologies, especially focusing on technologies based on Deep Neural Networks (DNNs). Various AhGs have been formed to investigate Neural Network Compression (NNR), Video Coding for Machine (VCM), Neural Network-based Video Coding (NNVC), etc. The Chinese AITISA and AVS also established corresponding expert groups to study standardization of similar technologies.
權(quán)利要求
What is claimed is:1. A method for video coding performed by at least one processor, the method comprising:obtaining video data; detecting at least one face from at least one frame of the video data; determining a set of facial landmark features of the at least one face from the at least one frame of the video data; determining an extended face area (EFA) which comprises a boundary area extended from an area of the detected at least one face from the at least one frame of the video data; determining a set of EFA features from the EFA; and coding the video data at least partly by a neural network based on the determined set of facial landmark features and on aggregating the set of facial landmark features, reconstructed EFA features, and an up-sampled sequence that is up-sampled from at least one down-sampled sequence, wherein the video data comprises an encoded bitstream of the video data, wherein determining the set of facial landmark features comprises up-sampling the at least one down-sampled sequence obtained by decompressing the encoded bitstream, wherein determining the EFA and determining the set of EFA features comprise up-sampling the at least one down-sampled sequence obtained by decompressing the encoded bitstream, and wherein determining the EFA and determining the set of EFA features further comprise reconstructing the EFA features, into the reconstructed EFA features, each respective to ones of the facial landmark features of the set of facial landmark features by a generative adversarial network. 2. The method according to claim 1 ,wherein the at least one face from the at least one frame of the video data is determined to be a largest face among a plurality of faces in the at least one frame of the video data. 3. The method according claim 1 , further comprising:determining a plurality of sets of facial landmark features, other than the set of facial landmark features of the at least one face from the at least one frame of the video data, respect to each of the plurality of faces in the at least one frame of the video data; and coding the video data at least partly by the neural network based on the determined set of facial landmark features and the determined plurality of sets of facial landmark features. 4. The method according to claim 1 ,wherein the neural network comprises a deep neural network (DNN). 5. An apparatus for video coding, the apparatus comprising:at least one memory configured to store computer program code; at least one processor configured to execute the computer program code to implement:obtaining video data; detecting at least one face from at least one frame of the video data; determining a set of facial landmark features of the at least one face from the at least one frame of the video data; determining an extended face area (EFA) which comprises a boundary area extended from an area of the detected at least one face from the at least one frame of the video data; determining a set of EFA features from the EFA; and coding the video data at least partly by a neural network based on the determined set of facial landmark features and on aggregating the set of facial landmark features, reconstructed EFA features, and an up-sampled sequence that is up-sampled from at least one down-sampled sequence, wherein the video data comprises an encoded bitstream of the video data, wherein determining the set of facial landmark features comprises up-sampling the at least one down-sampled sequence obtained by decompressing the encoded bitstream, wherein determining the EFA and determining the set of EFA features comprise up-sampling the at least one down-sampled sequence obtained by decompressing the encoded bitstream, and wherein determining the EFA and determining the set of EFA features further comprise reconstructing the EFA features, into the reconstructed EFA features, each respective to ones of the facial landmark features of the set of facial landmark features by a generative adversarial network. 6. The apparatus according to claim 5 ,wherein the at least one face from the at least one frame of the video data is determined to be a largest face among a plurality of faces in the at least one frame of the video data. 7. The apparatus according to claim 5 ,wherein the at least one hardware processor is further configured to execute the computer program code to implement:determining a plurality of sets of facial landmark features, other than the set of facial landmark features of the at least one face from the at least one frame of the video data, respect to each of the plurality of faces in the at least one frame of the video data; and coding the video data at least partly by the neural network based on the determined set of facial landmark features and the determined plurality of sets of facial landmark features. 8. A non-transitory computer readable medium storing a program causing a computer to execute a process, the process comprising:obtaining video data; detecting at least one face from at least one frame of the video data; determining a set of facial landmark features of the at least one face from the at least one frame of the video data; determining an extended face area (EFA) which comprises a boundary area extended from an area of the detected at least one face from the at least one frame of the video data; determining a set of EFA features from the EFA; and coding the video data at least partly by a neural network based on the determined set of facial landmark features and on aggregating the set of facial landmarks, reconstructed EFA features, and an up-sampled sequence that is up-sampled from at least one down-sampled sequence, wherein the video data comprises an encoded bitstream of the video data, wherein determining the set of facial landmark features comprises up-sampling the at least one down-sampled sequence obtained by decompressing the encoded bitstream, wherein determining the EFA and determining the set of EFA features comprise up-sampling the at least one down-sampled sequence obtained by decompressing the encoded bitstream, and wherein determining the EFA and determining the set of EFA features further comprise reconstructing the EFA features, into the reconstructed EFA features, each respective to ones of the facial landmark features of the set of facial landmark features by a generative adversarial network.
意見反饋
使用該功能遇到問題