Reduced rendering of six-degree of freedom video
地域：
CA
CA Santa Clara
摘要
Embodiments described herein provide for techniques to reduce the complexity of rendering immersive 3D video content. One embodiment provides for an apparatus comprising one or more processors to receive a data set that represents a two-dimensional encoding of planar projections of a frame of a three-dimensional video, decode the two-dimensional encoding into texture data, geometry data, and metadata, determine, based on the metadata, a visibility status and an occupancy status for a sample position in the three-dimensional video, and render video data for the sample position when the sample position is visible and occupied.
說明書
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
Six degree of freedom (6DoF) video is an emerging immersive video use case, which provides a viewer an immersive media experience where the viewer controls the viewpoint of a scene. The simpler three degree of freedom (3DoF) video, (e.g. 360 degree or panoramic video) allows a viewer to change orientation around the X, Y, and Z axes, (described as yaw, pitch, and roll), from a fixed position. 6DoF video enables the viewer to change position through translational movements along the X, Y, and Z axes.
6DoF video can be represented using point clouds. However, the rendering of point cloud data is computationally expensive, making it difficult to render point cloud video containing large number of points at high frame rates. Furthermore, point cloud data rates are large, requiring a large capacity for storage or transmission.
So that the manner in which the above recited features of the present embodiments can be understood in detail, a more particular description of the embodiments, briefly summarized above, may be had by reference to embodiments, some of which are illustrated in the appended drawings. It is to be noted, however, that the appended drawings illustrate only typical embodiments and are therefore not to be considered limiting of its scope.
權(quán)利要求
What is claimed is:1. An apparatus comprising:one or more processors to:receive a data set that represents a two-dimensional encoding of planar projections of an input frame of a three-dimensional video, wherein the planar projections are two-dimensional projections of a point cloud representation of the three-dimensional video; decode the two-dimensional encoding into texture data, geometry data, and projection metadata, the geometry data to specify a set of sample positions in a set of geometry planes that are positioned relative to a set of projection planes of the input frame of the three-dimensional video and the texture data includes a set of texture planes to specify color data for the set of sample positions; determine, based on the projection metadata, a visibility status for one or more projection planes in the set of projection planes and an occupancy status for one or more sample positions in the set of sample positions, the visibility status determined from multiple view points for the input frame; and render video data for occupied sample positions that are associated with a visible projection plane, wherein to render the video data includes to color the occupied sample positions based on the texture data without reconstruction of the point cloud representation of the three-dimensional video. 2. The apparatus as in claim 1 , wherein the set of geometry planes include depth data for each of a plurality of samples in the set of sample positions, the geometry planes generated based on texture patches from separate projections of the three-dimensional video. 3. The apparatus as in claim 2 , wherein the projection metadata includes auxiliary patch information that indicates at least one projection plane for each of the plurality of samples in the set of sample positions and the auxiliary patch information is input to an occlusion filling unit to fill an occluded portion of an object in a first view based in part on the auxiliary patch information. 4. The apparatus as in claim 1 , wherein the data set is a bitstream to be received from a remote device. 5. The apparatus as in claim 1 , the one or more processors additionally to:determine the visibility status from a first unit of the projection metadata; and determine the occupancy status from a second unit of the projection metadata. 6. The apparatus as in claim 5 , wherein to determine the visibility status, the one or more processors are to:receive a position and orientation associated with a display device; determine a projection plane associated with the sample position; and determine the visibility status based on visibility of the projection plane from the position and orientation of the display device. 7. The apparatus as in claim 6 , wherein the position and orientation associated with the display device is the position and orientation in a view space of the three-dimensional video. 8. The apparatus as in claim 7 , wherein the geometry data and the texture data include two-dimensional patch image representations for projection angles determined based on a maximization of a dot product of a point normal and a plane normal. 9. The apparatus as in claim 7 , wherein the first unit of the projection metadata includes auxiliary patch information that indicates the projection plane of the sample position. 10. The apparatus as in claim 7 , wherein the second unit of the projection metadata includes an occupancy map that indicates whether a sample position has corresponding point cloud data. 11. A method comprising:receiving a data set that represents a two-dimensional encoding of planar projections of an input frame of a three-dimensional video, wherein the planar projections are two-dimensional projections of a point cloud representation of the three-dimensional video; decoding the two-dimensional encoding into texture data, geometry data, and projection metadata, wherein the geometry data specifies a set of sample positions in a set of geometry planes that are positioned relative to a set of projection planes of the input frame of the three-dimensional video and the texture data includes a set of texture planes that specify color data for the set of sample positions; determining, based on the projection metadata, a visibility status for one or more projection planes in the set of projection planes and an occupancy status for one or more sample positions in the set of sample positions, the visibility status determined from multiple view points for the input frame; and rendering video data for occupied sample positions that are associated with a visible projection plane, wherein to render the video data includes to color the occupied sample positions based on the texture data without reconstruction of the point cloud representation of the three-dimensional video. 12. The method as in claim 11 , wherein the set of geometry planes include depth data for each of a plurality of samples in the set of sample positions, the geometry planes are generated based on texture patches from separate projections of the three-dimensional video, the projection metadata includes auxiliary patch information that indicates at least one projection plane for each of the plurality of samples in the set of sample positions, and the auxiliary patch information is input to an occlusion filling unit to fill an occluded portion of an object in a first view based in part on the auxiliary patch information. 13. The method as in claim 11 , additionally comprising:determining the visibility status from a first unit of the projection metadata; and determining the occupancy status from a second unit of the projection metadata, wherein determining the visibility status includes receiving a position and orientation associated with a display device, determining a projection plane associated with the sample position, and determining the visibility status based on visibility of the projection plane from the position and orientation of the display device. 14. The method as in claim 13 , wherein the position and orientation associated with the display device is the position and orientation in a view space of the three-dimensional video and the geometry data and the texture data include two-dimensional patch image representations for projection angles determined based on a maximization of a dot product of a point normal and a plane normal. 15. A non-transitory machine-readable medium storing instructions to cause one or more processors to perform operations comprising:receiving a data set that represents a two-dimensional encoding of planar projections of an input frame of a three-dimensional video, wherein the planar projections are two-dimensional projections of a point cloud representation of the three-dimensional video; decoding the two-dimensional encoding into texture data, geometry data, and projection metadata, wherein the geometry data specifies a set of sample positions in a set of geometry planes that are positioned relative to a set of projection planes of the input frame of the three-dimensional video and the texture data includes a set of texture planes that specify color data for the set of sample positions; determining, based on the projection metadata, a visibility status for one or more projection planes in the set of projection planes and an occupancy status for one or more sample positions in the set of sample positions, the visibility status determined from multiple view points for the input frame; and rendering video data for occupied sample positions that are associated with a visible projection plane, wherein to render the video data includes to color the occupied sample positions based on the texture data without reconstruction of the point cloud representation of the three-dimensional video. 16. The non-transitory machine-readable medium as in claim 15 , wherein the set of geometry planes include depth data for each of a plurality of samples in the set of sample positions, the geometry planes are generated based on texture patches from separate projections of the three-dimensional video, the projection metadata includes auxiliary patch information that indicates at least one projection plane for each of the plurality of samples in the set of sample positions, and the auxiliary patch information is input to an occlusion filling unit to fill an occluded portion of an object in a first view based in part on the auxiliary patch information. 17. The non-transitory machine-readable medium as in claim 16 , wherein the data set is a bitstream received from a remote device. 18. The non-transitory machine-readable medium as in claim 16 , additionally comprising:determining the visibility status from a first unit of the projection metadata; and determining the occupancy status from a second unit of the projection metadata. 19. The non-transitory machine-readable medium as in claim 18 , wherein determining the visibility status includes:receiving a position and orientation associated with a display device; determining a projection plane associated with the sample position; and determining the visibility status based on visibility of the projection plane from the position and orientation of the display device. 20. The non-transitory machine-readable medium as in claim 19 , wherein the position and orientation associated with the display device is the position and orientation in a view space of the three-dimensional video and the geometry data and the texture data include two-dimensional patch image representations for projection angles determined based on a maximization of a dot product of a point normal and a plane normal.
意見反饋
使用該功能遇到問題