KR20250096744A - Context-aware voxel-based upsampling for point cloud processing - Google Patents

Abstract

Some embodiments of the method may include the steps of: upsampling a first point cloud using initial upsampling to obtain a second point cloud; associating features of the second point cloud with context information to obtain a third point cloud; predicting an occupancy state of at least one voxel of the third point cloud; and removing voxels of the third point cloud classified as empty based on the predicted occupancy state to generate a pruned point cloud.

Description

Context-aware voxel-based upsampling for point cloud processing

Cross-reference to related applications

This application claims the benefit of U.S. Provisional Patent Application Serial No. 63/438,212, filed Jan. 10, 2023, entitled “CONTEXT-AWARE VOXEL-BASED UPSAMPLING FOR POINT CLOUD PROCESSING” (the “'212 application”), and U.S. Provisional Patent Application Serial No. 63/417,284, filed Oct. 18, 2022, entitled “CONTEXT-AWARE VOXEL-BASED UPSAMPLING FOR POINT CLOUD PROCESSING” (the “'284 application”), which are hereby incorporated by reference in their entireties.

Inclusion by reference

This application claims the benefit of the following applications: U.S. Provisional Patent Application Serial No. 63/291,015, filed December 17, 2021, entitled “Hybrid Framework for Point Cloud Compression” (the “'015 application”); U.S. Provisional Patent Application Serial No. 63/297,869, filed January 10, 2022, entitled “A Scalable Framework for Point Cloud Compression” (the “'869 application”); U.S. Provisional Patent Application Serial No. 63/388,087, filed July 11, 2022, entitled “A Scalable Framework for Point Cloud Compression” (the “'087 application”); No. 63/252,482, filed Oct. 5, 2021, entitled “Method and Apparatus for Point Cloud Compression Using Hybrid Deep Entropy Coding” (the “'482 application”); U.S. Provisional Patent Application Serial No. 63/297,894, filed Jan. 10, 2022, entitled “Coordinate Refinement and Upsampling from Quantized Point Cloud Reconstruction” (the “'894 application”); and U.S. Provisional Patent Application Serial No. 63/388,600, filed July 12, 2022, entitled “Deep Distribution-Aware Point Feature Extractor for AI-Based Point Cloud Compression” (the “'600 application”), which are hereby incorporated by reference in their entirety.

Point cloud (PC) data format is a common data format across many business areas, such as autonomous driving, robotics, augmented reality/virtual reality (AR/VR), civil engineering, computer graphics, and animation/film industries. 3D LiDAR (Light Detection and Ranging) sensors have been deployed in self-driving cars, and inexpensive LiDAR sensors are available. With the advancement of sensing technology, 3D point cloud data has become more practical than ever.

Embodiments described herein include methods used for video encoding and decoding (collectively, “coding”).

A first exemplary method/device according to some embodiments may include: obtaining a second point cloud by upsampling a first point cloud using initial upsampling; obtaining a third point cloud by associating features of the second point cloud with context information; predicting an occupancy state of at least one voxel of the third point cloud; and generating a pruned point cloud by removing voxels of the third point cloud classified as empty based on the predicted occupancy state.

In some embodiments of the first exemplary method, the initial upsampling includes nearest neighbor upsampling.

In some embodiments of the first exemplary method, the step of associating features includes the step of concatenating features of the second point cloud with context information to obtain a third point cloud.

In some embodiments of the first exemplary method, the context information is voxel-wise context information.

In some embodiments of the first exemplary method, the context information includes a context point cloud.

In some embodiments of the first exemplary method, the context information includes information about the second point cloud.

In some embodiments of the first exemplary method, the context information includes information about voxel occupancy status of the second point cloud.

In some embodiments of the first exemplary method, the context information includes information about the location of a child voxel relative to the location of a parent voxel in the first point cloud.

In some embodiments of the first exemplary method, the context information includes coordinate information regarding the location of an occupied voxel of at least one of the first and second point clouds.

In some embodiments of the first exemplary method, the context information includes coordinate information, and the coordinate information is in the form of one of Euclidean coordinates, spherical coordinates, and cylindrical coordinates.

In some embodiments of the first exemplary method, the context information provides known information about the first point cloud in addition to information available for initial upsampling of the first point cloud.

In some embodiments of the first exemplary method, the context information includes the bit depth of the second point cloud.

Some embodiments of the first exemplary method may further include a step of performing feature decode on the input point cloud and the first bitstream to generate a first point cloud.

Some embodiments of the first exemplary method may further include: performing feature aggregation on the pruned point cloud to generate aggregated features; and performing a context-aware upsampling process on the aggregated features to generate a decoded point cloud.

Some embodiments of the first exemplary method may further include: performing a feature to residual conversion on the pruned point cloud to generate a residual output; and adding the pruned point cloud to the residual output to generate a decoded point cloud.

Some embodiments of the first exemplary method may further include a step of performing feature aggregation on the pruned point cloud to generate aggregated features, wherein feature-to-residual transformation is performed on the aggregated features.

In some embodiments of the first exemplary method, the step of predicting the occupancy state is performed using a first neural network.

In some embodiments of the first exemplary method, the step of predicting occupancy status predicts a ground-truth occupancy status of at least one voxel.

In some embodiments of the first exemplary method, the step of predicting occupancy predicts a likelihood that at least one voxel will be occupied.

In some embodiments of the first exemplary method, the step of removing voxels of the third point cloud removes voxels using a voxel pruning process.

Some embodiments of the first exemplary method may further include a step of aggregating at least one feature of the second point cloud.

In some embodiments of the first exemplary method, the step of predicting the occupancy state of at least one voxel comprises: aggregating at least one feature of the third point cloud; processing the aggregated feature with multilayer perceptron (MLP) layers to generate an MLP layer output; performing a softmax process on the MLP layer output to generate softmax output values; and performing thresholding on the softmax output values to generate a predicted occupancy state of the at least one voxel of the third point cloud.

In some embodiments of the first exemplary method, thresholding of the softmax output values converts softmax output values greater than 0.5 to an output value of 1, and converts softmax output values less than or equal to 0.5 to an output value of 0.

In some embodiments of the first exemplary method, the step of predicting the occupancy state of at least one voxel comprises: the step of aggregating at least one feature of the third point cloud; and the step of generating a predicted occupancy state of at least one voxel of the third point cloud based on the aggregated feature.

In some embodiments of the first exemplary method, the step of aggregating at least one feature comprises: repeating a cascading process one or more times, wherein the cascading process comprises: performing a sparse 3D convolution of an input point cloud to generate a convolution output point cloud; performing a nonlinear activation process on the convolution output point cloud to generate a nonlinear output point cloud; and preparing the nonlinear output point cloud as an input point cloud for a next cycle of the cascading process, wherein the third point cloud is an input point cloud for a first cycle of the cascading process, and wherein a last cycle of the cascading process generates aggregated features.

Some embodiments of the first exemplary method may further include a step of adding a third point cloud to the ReLU output point cloud of the last cycle of the cascading process.

In some embodiments of the first exemplary method, the step of aggregating at least one feature comprises: performing a sparse 3D convolution of an input point cloud to generate a convolution output point cloud; and performing a nonlinear activation process on the convolution output point cloud to generate aggregated features.

In some embodiments of the first exemplary method, the nonlinear activation process comprises a rectifier linear unit (ReLU) activation process, and the nonlinear output point cloud comprises a ReLU output point cloud.

In some embodiments of the first exemplary method, the step of aggregating at least one feature comprises: repeating a first cascading process one or more times, wherein the first cascading process comprises: performing a first sparse 3D convolution of a first input point cloud to generate a first convolution output point cloud; performing a first nonlinear activation process on the first convolution output point cloud to generate a first nonlinear output point cloud; and preparing the first nonlinear output point cloud as the first input point cloud for a next cycle of the first cascading process, wherein the third point cloud is the first input point cloud for the first cycle of the first cascading process, and wherein the last cycle of the first cascading process generates the first cascading process output; repeating a second cascading process one or more times, wherein the second cascading process comprises: performing a second sparse 3D convolution of the second input point cloud to generate a second convolution output point cloud; A method comprising: performing a second nonlinear activation process on a second convolution output point cloud to generate a second nonlinear output point cloud; and preparing the second nonlinear output point cloud as a second input point cloud when there is a next cycle of the second cascading process, wherein the third point cloud is a second input point cloud for a first cycle of the second cascading process, and the last cycle of the second cascading process generates a second cascading process output; concatenating the first cascading process output and the second cascading process output to generate a concatenation output; and adding the third point cloud to the concatenation output to generate an aggregated feature.

In some embodiments of the first exemplary method, the step of aggregating at least one feature comprises: repeating a first cascading process one or more times, wherein the first cascading process comprises: performing a first sparse 3D convolution of a first input point cloud to generate a first convolution output point cloud; performing a first rectifier linear unit (ReLU) activation process on the first convolution output point cloud to generate a first ReLU output point cloud; and preparing the first ReLU output point cloud as the first input point cloud when there is a next cycle of the first cascading process, wherein the third point cloud is the first input point cloud for the first cycle of the first cascading process, and wherein a last cycle of the first cascading process generates the first cascading process output; A step of repeating a second cascading process one or more times, wherein the second cascading process comprises: performing a second sparse 3D convolution of a second input point cloud to generate a second convolution output point cloud; performing a second ReLU (rectifier linear unit) activation process on the second convolution output point cloud to generate a second ReLU output point cloud; and preparing the second ReLU output point cloud as a second input point cloud when there is a next cycle of the second cascading process, wherein the third point cloud is the second input point cloud for a first cycle of the second cascading process, and the last cycle of the second cascading process generates a second cascading process output; connecting the first cascading process output and the second cascading process output to generate a connected output; and adding the third point cloud to the connected output to generate aggregated features.

In some embodiments of the first exemplary method, the step of aggregating at least one feature comprises: performing a self-attention process on the third point cloud; adding the third point cloud to an output of the self-attention process to generate an MLP process input; performing an MLP process on the MLP process input; and adding the MLP process input to the MLP process output to generate an aggregated feature.

In some embodiments of the first exemplary method, the self-attention process generates output features based on the k nearest neighbors of a voxel of the third point cloud.

In some embodiments of the first exemplary method, the step of aggregating at least one feature of the third point cloud comprises performing the feature aggregation process two or more times.

A first exemplary method/device according to some embodiments may include: a processor; and a non-transitory computer-readable medium storing instructions that, when executed by the processor, cause the device to: upsample a first point cloud using initial upsampling to obtain a second point cloud; associate features of the second point cloud with context information to obtain a third point cloud; predict an occupancy state of at least one voxel of the third point cloud; and remove voxels of the third point cloud classified as empty based on the predicted occupancy state to generate a pruned point cloud.

In some embodiments of the first exemplary device, the initial upsampling includes nearest neighbor upsampling.

In some embodiments of the first exemplary device, associating the features includes obtaining a third point cloud by connecting features of the second point cloud with context information.

An exemplary device according to some embodiments may include: a device according to any of the devices listed above; and at least one of (i) an antenna configured to receive a signal, the signal comprising data representing an image, (ii) a band limiter configured to limit the received signal to a frequency band comprising the data representing the image, or (iii) a display configured to display the image.

Some embodiments of the exemplary device may further include at least one of a TV, a cell phone, a tablet, and a set-top box (STB).

An exemplary computer-readable medium according to some embodiments may include instructions that cause one or more processors to: obtain a second point cloud by upsampling a first point cloud using initial upsampling; obtain a third point cloud by associating features of the second point cloud with context information; predict an occupancy state of at least one voxel of the third point cloud; and remove voxels of the third point cloud classified as empty based on the predicted occupancy state to generate a pruned point cloud.

An exemplary computer program product according to some embodiments may include instructions that, when the program is executed by one or more processors, cause the one or more processors to: obtain a second point cloud by upsampling a first point cloud using initial upsampling; obtain a third point cloud by associating features of the second point cloud with context information; predict an occupancy state of at least one voxel of the third point cloud; and remove voxels of the third point cloud classified as empty based on the predicted occupancy state to generate a pruned point cloud.

A second exemplary method according to some embodiments may include performing context-aware upsampling of the first point cloud to determine an upsampled second point cloud, wherein the context-aware upsampling comprises: associating features of a third point cloud with context information, the third point cloud being at least partially based on an initial upsampled version of the first point cloud; and removing voxels of a fourth point cloud that are predicted to be empty based at least partially on the context information from the third point cloud to generate the upscaled second point cloud.

A third exemplary method according to some embodiments comprises: obtaining a second point cloud by upsampling a first point cloud using an initial upsampling; obtaining a third point cloud by associating features of the second point cloud with context information; predicting an occupancy state of at least one voxel of the third point cloud, wherein the predicting the occupancy state of the at least one voxel comprises aggregating at least one feature of the third point cloud, wherein the aggregating the at least one feature of the third point cloud comprises using a first neural network, and wherein the aggregating the at least one feature of the third point cloud using the first neural network comprises using a first neural network parameter set together with the first neural network; generating a pruned point cloud by removing voxels of the third point cloud classified as empty according to the predicted occupancy state; and performing feature aggregation on the pruned point cloud to generate aggregated features, wherein the performing feature aggregation on the pruned point cloud comprises using a second neural network, and wherein the step of generating the aggregated features using the second neural network comprises using a second neural network parameter set together with the second neural network, wherein the first neural network parameter set is identical to the second neural network parameter set.

Some embodiments of the third exemplary method may further include a step of aggregating at least one feature of the second point cloud.

In some embodiments of the third exemplary method, wherein the step of aggregating at least one feature of the second point cloud comprises the step of using a third neural network, wherein the step of aggregating at least one feature of the second point cloud using the third neural network comprises the step of using a third neural network parameter set together with the third neural network, wherein the third neural network parameter set is identical to the first neural network parameter set.

In some embodiments of the third exemplary method, the initial upsampling includes nearest neighbor upsampling.

In some embodiments of the third exemplary method, the step of associating features includes the step of obtaining a third point cloud by associating features of the second point cloud with context information.

In some embodiments of the third exemplary method, the step of associating features includes the step of obtaining a third point cloud by associating features of the second point cloud with context information.

In some embodiments of the third exemplary method, the context information is voxel-wise context information.

Some embodiments of the third exemplary method may further include a step of performing feature decoding on the input point cloud and the first bitstream to generate a first point cloud.

Some embodiments of the third exemplary method may further include a step of performing a context-aware upsampling process on the aggregated features to generate a decoded point cloud.

Some embodiments of the third exemplary method may further include the steps of performing a feature-residual transformation on the pruned point cloud to generate a residual output; and the steps of adding the pruned point cloud to the residual output to generate a decoded point cloud.

In some embodiments of the third exemplary method, feature-residual transformation is performed on aggregated features.

In some embodiments of the third exemplary method, the step of predicting an occupancy state predicts a ground truth occupancy state of at least one voxel.

In some embodiments of the third exemplary method, the step of predicting occupancy predicts the likelihood that at least one voxel will be occupied.

In some embodiments of the third exemplary method, the step of removing voxels of the third point cloud removes voxels using a voxel pruning process.

In some embodiments of the third exemplary method, the step of predicting the occupancy state of at least one voxel further comprises: processing the aggregated features with multilayer perceptron (MLP) layers to generate MLP layer outputs; performing a softmax process on the MLP layer outputs to generate softmax output values; and performing thresholding on the softmax output values to generate a predicted occupancy state of at least one voxel of the third point cloud.

In some embodiments of the third exemplary method, thresholding of the softmax output values converts softmax output values greater than 0.5 to an output value of 1, and converts softmax output values less than or equal to 0.5 to an output value of 0.

In some embodiments of the third exemplary method, the step of predicting the occupancy state of at least one voxel comprises: the step of aggregating at least one feature of the third point cloud; and the step of generating a predicted occupancy state of at least one voxel of the third point cloud based on the aggregated feature.

In some embodiments of the third exemplary method, the step of aggregating at least one feature of the third point cloud comprises: repeating a cascading process one or more times, wherein the cascading process comprises: performing a sparse 3D convolution of an input point cloud to generate a convolution output point cloud; performing a nonlinear activation process on the convolution output point cloud to generate a nonlinear output point cloud; and preparing the nonlinear output point cloud as an input point cloud for a next cycle of the cascading process, wherein the third point cloud is an input point cloud for a first cycle of the cascading process, and wherein a last cycle of the cascading process generates aggregated features.

Some embodiments of the third exemplary method may further include a step of adding the third point cloud to the ReLU output point cloud of the last cycle of the cascading process.

In some embodiments of the third exemplary method, the step of aggregating at least one feature comprises: performing a sparse 3D convolution of an input point cloud to generate a convolution output point cloud; and performing a nonlinear activation process on the convolution output point cloud to generate aggregated features.

In some embodiments of the third exemplary method, the nonlinear activation process comprises a rectifier linear unit (ReLU) activation process, and the nonlinear output point cloud comprises a ReLU output point cloud.

In some embodiments of the third exemplary method, the step of aggregating at least one feature of the third point cloud comprises: repeating a first cascading process one or more times, wherein the first cascading process comprises: performing a first sparse 3D convolution of a first input point cloud to generate a first convolution output point cloud; performing a first nonlinear activation process on the first convolution output point cloud to generate a first nonlinear output point cloud; and preparing the first nonlinear output point cloud as the first input point cloud if there is a next cycle of the first cascading process, wherein the third point cloud is the first input point cloud for the first cycle of the first cascading process, and wherein a last cycle of the first cascading process generates the first cascading process output; A method for generating a second cascading process, the method comprising: performing a second sparse 3D convolution of a second input point cloud to generate a second convolution output point cloud; performing a second nonlinear activation process on the second convolution output point cloud to generate a second nonlinear output point cloud; and preparing the second nonlinear output point cloud as a second input point cloud when there is a next cycle of the second cascading process, wherein the third point cloud is the second input point cloud for a first cycle of the second cascading process, and the last cycle of the second cascading process generates a second cascading process output; concatenating the first cascading process output and the second cascading process output to generate a concatenated output; and adding the third point cloud to the concatenated output to generate aggregated features.

In some embodiments of the third exemplary method, the step of aggregating at least one feature comprises: repeating a first cascading process one or more times, wherein the first cascading process comprises: performing a first sparse 3D convolution of a first input point cloud to generate a first convolution output point cloud; performing a first ReLU (rectifier linear unit) activation process on the first convolution output point cloud to generate a first ReLU output point cloud; and preparing the first ReLU output point cloud as the first input point cloud when there is a next cycle of the first cascading process, wherein the third point cloud is the first input point cloud for the first cycle of the first cascading process, and wherein a last cycle of the first cascading process generates the first cascading process output; A step of repeating a second cascading process one or more times, wherein the second cascading process comprises: performing a second sparse 3D convolution of a second input point cloud to generate a second convolution output point cloud; performing a second ReLU (rectifier linear unit) activation process on the second convolution output point cloud to generate a second ReLU output point cloud; and preparing the second ReLU output point cloud as a second input point cloud when there is a next cycle of the second cascading process, wherein the third point cloud is the second input point cloud for a first cycle of the second cascading process, and the last cycle of the second cascading process generates a second cascading process output; connecting the first cascading process output and the second cascading process output to generate a connected output; and adding the third point cloud to the connected output to generate aggregated features.

In some embodiments of the third exemplary method, the step of aggregating at least one feature comprises: performing a self-attention process on the third point cloud; adding the third point cloud to an output of the self-attention process to generate an MLP process input; performing an MLP process on the MLP process input; and adding the MLP process input to the MLP process output to generate an aggregated feature.

In some embodiments of the third exemplary method, the self-attention process generates output features based on the k nearest neighbors of a voxel of the third point cloud.

In some embodiments of the third exemplary method, the step of aggregating at least one feature of the third point cloud comprises performing the feature aggregation process two or more times.

In some embodiments of the third exemplary method, the first neural network parameter set and the second neural network parameter set are the same neural network parameter set, and the same neural network parameter set is used by at least the first neural network and the second neural network.

In some embodiments of the third exemplary method, the first neural network parameter set and the second neural network parameter set are distinct but identical neural network parameter sets.

A fourth exemplary method according to some embodiments may include: obtaining a first point cloud; determining an occupancy state of at least one voxel of the first point cloud; generating a second point cloud by removing voxels of the first point cloud classified as empty according to the determined occupancy state; obtaining a third point cloud by associating features of the second point cloud with context information; downsampling the third point cloud using initial downsampling to obtain a fourth point cloud; and outputting the fourth point cloud as an encoded point cloud.

A fourth exemplary device according to some embodiments may include: a processor; and a non-transitory computer-readable medium storing instructions that, when executed by the processor, cause the device to obtain a first point cloud; determine an occupancy state of at least one voxel of the first point cloud; generate a second point cloud by removing voxels of the first point cloud classified as empty according to the determined occupancy state; obtain a third point cloud by associating features of the second point cloud with context information; obtain a fourth point cloud by downsampling the third point cloud using initial downsampling; and output the fourth point cloud as an encoded point cloud.

A fifth exemplary method/device according to some embodiments may include: accessing data including a first point cloud; and transmitting data including the first point cloud.

A fifth exemplary method/device according to some embodiments may include: an access unit configured to access data including a first point cloud; and a transmitter configured to transmit data including the first point cloud.

A sixth exemplary method/device according to some embodiments may include: a processor; and a non-transitory computer-readable medium storing instructions that, when executed by the processor, operate to cause the device to perform any one of the methods listed above.

A seventh exemplary method/device according to some embodiments may include at least one processor configured to perform any one of the methods listed above.

An eighth exemplary method/device according to some embodiments may include a computer-readable medium storing instructions for causing one or more processors to perform any one of the methods listed above.

A ninth exemplary method/device according to some embodiments may include at least one processor and at least one non-transitory computer-readable medium storing instructions for causing the at least one processor to perform any one of the methods listed above.

An exemplary signal according to some embodiments may include a bitstream generated according to any of the methods listed above.

In additional embodiments, an encoder and decoder device are provided for performing the methods described herein. The encoder or decoder device may include a processor configured to perform the methods described herein. The device may include a computer-readable medium (e.g., a non-transitory medium) storing instructions for performing the methods described herein. In some embodiments, the computer-readable medium (e.g., a non-transitory medium) stores video encoded using any one of the methods described herein.

One or more of the present embodiments also provide a computer-readable storage medium having stored thereon instructions for performing bidirectional optical flow, or encoding or decoding video data, according to any one of the methods described above. The present embodiments also provide a computer-readable storage medium having stored thereon a bitstream generated according to the methods described above. The present embodiments also provide a method and apparatus for transmitting a bitstream generated according to the methods described above. The present embodiments also provide a computer program product comprising instructions for performing any one of the methods described above.

FIG. 1A is a system diagram illustrating an exemplary communications system according to some embodiments.
FIG. 1b is a system diagram illustrating an exemplary wireless transmit/receive unit (WTRU) that may be used within the communications system illustrated in FIG. 1a according to some embodiments.
FIG. 1c is a system diagram illustrating an exemplary set of interfaces for a system according to some embodiments.
Figure 2a is a schematic example showing an exemplary voxel-based representation of a point cloud.
Figure 2b is a schematic example showing an exemplary sparse voxel-based representation of a point cloud.
Figure 3 is a schematic process diagram showing an exemplary nearest neighbor (NN) upsampling for a point cloud.
Figure 4 is a schematic process diagram showing an example voxel-based upsampling using pruning.
Figure 5 is a schematic process diagram showing an exemplary context-aware voxel-based upsampling using pruning according to some embodiments.
FIG. 6a is a table illustrating exemplary position values according to some embodiments.
FIG. 6b is a schematic perspective diagram illustrating exemplary child voxel locations as context information according to some embodiments.
FIG. 7 is a flowchart illustrating an exemplary process for cascading multiple context-aware upsampling according to some embodiments.
FIG. 8 is a schematic process diagram showing an exemplary context-aware voxel-based upsampling using initial feature aggregation according to some embodiments.
FIG. 9 is a flowchart illustrating an exemplary process for binary classification according to some embodiments.
FIG. 10 is a block diagram illustrating an exemplary process having cascaded sparse convolutional layers for feature aggregation according to some embodiments.
FIG. 11 is a block diagram illustrating an exemplary ResNet block for feature aggregation according to some embodiments.
FIG. 12 is a block diagram illustrating an exemplary Inception-ResNet block for feature aggregation according to some embodiments.
FIG. 13 is a block diagram illustrating an exemplary transformer block for feature aggregation according to some embodiments.
FIG. 14 is a block diagram illustrating an exemplary architecture of a self-attention block according to some embodiments.
FIG. 15 is a flowchart illustrating an exemplary process for cascading multiple feature aggregations according to some embodiments.
FIG. 16 is a block diagram illustrating an exemplary original decoder architecture according to some embodiments.
FIG. 17 is a block diagram illustrating an exemplary decoder architecture with voxel-based upsampling according to some embodiments.
FIG. 18 is a block diagram illustrating an exemplary decoder architecture with voxel-based upsampling and feature aggregation according to some embodiments.
FIG. 19 is a block diagram illustrating an exemplary decoder architecture without a feature-residual transformer according to some embodiments.
FIG. 20 is a block diagram illustrating an exemplary decoder architecture with a single pass through voxel-based upsampling and feature aggregation according to some embodiments.
FIG. 21 is a block diagram illustrating exemplary sparse tensor operations according to some embodiments.
FIG. 22 is a block diagram illustrating an exemplary decoder architecture according to some embodiments.
FIG. 23 is a block diagram illustrating an exemplary decoder architecture according to some embodiments.
FIG. 24 is a flowchart illustrating an exemplary process of context-aware voxel-based upsampling using pruning according to some embodiments.
FIG. 25 is a flowchart illustrating an exemplary process of context-aware voxel-based upsampling and feature aggregation according to some embodiments.
FIG. 26 is a flowchart illustrating an exemplary process for encoding a bitstream according to some embodiments.
The entities, connections, arrangements, and the like depicted in - and described in connection with - the various drawings are presented by way of example, and not limitation. Accordingly, any and all statements or other instructions as to what a particular drawing "depicts," what a particular element or entity "is" or "has," and any and all similar statements - which alone, taken out of context, could be read as absolute and therefore limiting - are properly read only as if they were structurally preceded by a phrase such as "In at least one embodiment, ..." In the interest of brevity and clarity of presentation, such implied preceding phrases are not tediously repeated in the Detailed Description.

FIG. 1A is a system diagram illustrating an exemplary communications system (100) in which one or more of the disclosed embodiments may be implemented. The communications system (100) may be a multiple access system that provides content, such as voice, data, video, messaging, broadcast, and the like, to multiple wireless users. The communications system (100) may enable multiple wireless users to access such content through sharing of system resources, including wireless bandwidth. For example, the communication system (100) may utilize one or more channel access methods, such as code division multiple access (CDMA), time division multiple access (TDMA), frequency division multiple access (FDMA), orthogonal FDMA (OFDMA), single-carrier FDMA (SC-FDMA), zero-tail unique-word DFT-Spread OFDM (ZT UW DTS-s OFDM), unique word OFDM (UW-OFDM), resource block-filtered OFDM, filter bank multicarrier (FBMC), etc.

As illustrated in FIG. 1a, the communications system (100) may include wireless transmit/receive units (WTRUs) (102a, 102b, 102c, 102d), a RAN (104/113), a CN (106), a public switched telephone network (PSTN) (108), the Internet (110), and other networks (112), although it will be appreciated that the disclosed embodiments contemplate any number of WTRUs, base stations, networks, and/or network elements. Each of the WTRUs (102a, 102b, 102c, 102d) may be any type of device configured to operate and/or communicate in a wireless environment. For example, the WTRUs (102a, 102b, 102c, 102d)—any of which may be referred to as a “station” and/or a “STA”—may be configured to transmit and/or receive wireless signals, and may include user equipment (UE), a mobile station, a fixed or mobile subscriber unit, a subscription-based unit, a pager, a cellular telephone, a personal digital assistant (PDA), a smartphone, a laptop, a netbook, a personal computer, a wireless sensor, a hotspot or Mi-Fi device, an Internet of Things (IoT) device, a watch or other wearable, a head-mounted display (HMD), a vehicle, a drone, medical devices and applications (e.g., remote surgery), industrial devices and applications (e.g., robots and/or other wireless devices operating in an industrial and/or automated process chain context), consumer electronics devices, devices operating on commercial and/or industrial wireless networks, and the like. Any of the WTRUs (102a, 102b, 102c, and 102d) may be interchangeably referred to as UEs.

The communication system (100) may also include a base station (114a) and/or a base station (114b). Each of the base stations (114a, 114b) may be any type of device configured to wirelessly interface with at least one of the WTRUs (102a, 102b, 102c, 102d) to facilitate access to one or more communication networks, such as the CN (106), the Internet (110), and/or other networks (112). By way of example, the base stations (114a, 114b) may be a base transceiver station (BTS), a Node-B, an eNode B, a Home Node B, a Home eNode B, a gNB, a NR NodeB, a site controller, an access point (AP), a wireless router, and/or the like. Although the base stations (114a, 114b) are each depicted as a single element, it will be appreciated that the base stations (114a, 114b) may include any number of interconnected base stations and/or network elements.

The base station (114a) may be part of a RAN (104/113) that may also include other base stations and/or network elements (not shown), such as a base station controller (BSC), a radio network controller (RNC), a relay node, etc. The base station (114a) and/or the base station (114b) may be configured to transmit and/or receive wireless signals on one or more carrier frequencies, which may be referred to as a cell (not shown). These frequencies may be in a licensed spectrum, an unlicensed spectrum, or a combination of licensed and unlicensed spectrum. A cell may provide coverage for wireless services for a particular geographic area, which may be relatively fixed or may vary over time. A cell may be further divided into cell sectors. For example, a cell associated with the base station (114a) may be divided into three sectors. Thus, in one embodiment, the base station (114a) may include three transceivers, one for each sector of the cell. In an embodiment, the base station (114a) may utilize multiple-input multiple-output (MIMO) technology and utilize multiple transceivers for each sector of the cell. For example, beamforming may be used to transmit and/or receive signals in desired spatial directions.

The base stations (114a, 114b) may communicate with one or more of the WTRUs (102a, 102b, 102c, 102d) via an air interface (116), which may be any suitable wireless communication link (e.g., radio frequency (RF), microwave, centimeter wave, micrometer wave, infrared (IR), ultraviolet (UV), visible light, etc.). The air interface (116) may be established using any suitable radio access technology (RAT).

More specifically, as noted above, the communication system (100) may be a multiple-access system and may utilize one or more channel access schemes, such as CDMA, TDMA, FDMA, OFDMA, SC-FDMA, etc. For example, the base station (114a) and the WTRUs (102a, 102b, 102c) within the RAN (104/113) may implement a radio technology, such as Universal Mobile Telecommunications System (UMTS) Terrestrial Radio Access (UTRA), which may establish the air interface (116) using wideband CDMA (WCDMA). WCDMA may include communication protocols such as High-Speed Packet Access (HSPA) and/or Evolved HSPA (HSPA+). HSPA may include High-Speed DL (Downlink) Packet Access (HSDPA) and/or High-Speed UL Packet Access (HSUPA).

In an embodiment, the base station (114a) and the WTRUs (102a, 102b, 102c) may implement a radio technology, such as Evolved UMTS Terrestrial Radio Access (E-UTRA), which may establish the air interface (116) using Long Term Evolution (LTE) and/or LTE-Advanced (LTE-A) and/or LTE-Advanced Pro (LTE-A Pro).

In an embodiment, the base station (114a) and the WTRUs (102a, 102b, 102c) may implement a radio technology, such as New Radio (NR) radio access, that may establish the air interface (116) using NR.

In an embodiment, the base station (114a) and the WTRUs (102a, 102b, 102c) may implement multiple radio access technologies. For example, the base station (114a) and the WTRUs (102a, 102b, 102c) may implement LTE radio access and NR radio access together, for example, using dual connectivity (DC) principles. Thus, the air interface utilized by the WTRUs (102a, 102b, 102c) may be characterized by multiple types of radio access technologies and/or transmissions to/from multiple types of base stations (e.g., eNBs and gNBs).

In other embodiments, the base station (114a) and the WTRUs (102a, 102b, 102c) may implement radio technologies such as IEEE 802.11 (i.e., Wireless Fidelity (WiFi)), IEEE 802.16 (i.e., Worldwide Interoperability for Microwave Access (WiMAX)), CDMA2000, CDMA2000 1X, CDMA2000 EV-DO, Interim Standard 2000 (IS-2000), Interim Standard 95 (IS-95), Interim Standard 856 (IS-856), Global System for Mobile communications (GSM), Enhanced Data rates for GSM Evolution (EDGE), GSM EDGE (GERAN), and the like.

The base station (114b) in FIG. 1a may be, for example, a wireless router, a Home Node B, a Home eNode B, or an access point, and may utilize any suitable RAT to facilitate wireless connectivity in a localized area, such as a business premises, a home, a vehicle, a campus, an industrial facility, an air corridor (e.g., for use by drones), a roadway, and the like. In one embodiment, the base station (114b) and the WTRUs (102c, 102d) may implement a radio technology, such as IEEE 802.11, to establish a wireless local area network (WLAN). In an embodiment, the base station (114b) and the WTRUs (102c, 102d) may implement a radio technology, such as IEEE 802.15, to establish a wireless personal area network (WPAN). In another embodiment, the base station (114b) and the WTRUs (102c, 102d) may utilize a cellular-based RAT (e.g., WCDMA, CDMA2000, GSM, LTE, LTE-A, LTE-A Pro, NR, etc.) to establish a picocell or femtocell. As illustrated in FIG. 1a, the base station (114b) may have a direct connection to the Internet (110). Thus, the base station (114b) may not need to access the Internet (110) via the CN (106).

The RAN (104/113) may be in communication with the CN (106), which may be any type of network configured to provide voice, data, application, and/or voice over internet protocol (VoIP) services to one or more of the WTRUs (102a, 102b, 102c, 102d). The data may have different quality of service (QoS) requirements, such as different throughput requirements, latency requirements, error tolerance requirements, reliability requirements, data throughput requirements, mobility requirements, and the like. The CN (106) may provide call control, billing services, mobile location-based services, pre-paid calling, Internet connectivity, video distribution, and/or may perform higher level security functions, such as user authentication. Although not shown in FIG. 1a, it will be appreciated that the RAN (104/113) and/or the CN (106) may communicate directly or indirectly with other RANs that utilize the same RAT as the RAN (104/113) or a different RAT. For example, in addition to being connected to the RAN (104/113) that may utilize NR radio technology, the CN (106) may also communicate with other RANs (not shown) that utilize GSM, UMTS, CDMA 2000, WiMAX, E-UTRA, or WiFi radio technologies.

The CN (106) may also act as a gateway for the WTRUs (102a, 102b, 102c, 102d) to access the PSTN (108), the Internet (110), and/or other networks (112). The PSTN (108) may include circuit-switched telephone networks that provide plain old telephone service (POTS). The Internet (110) may include a global system of interconnected computer networks and devices that use common communication protocols, such as transmission control protocol (TCP), user datagram protocol (UDP), and/or internet protocol (IP) within the TCP/IP suite of Internet protocols. The networks (112) may include wired and/or wireless communication networks owned and/or operated by other service providers. For example, the networks (112) may include other CNs connected to one or more RANs, which may utilize the same RAT as the RAN (104/113) or a different RAT.

Some or all of the WTRUs (102a, 102b, 102c, 102d) within the communication system (100) may include multi-mode capabilities (e.g., the WTRUs (102a, 102b, 102c, 102d) may include multiple transceivers for communicating with different wireless networks over different wireless links). For example, the WTRU (102c) illustrated in FIG. 1a may be configured to communicate with a base station (114a) that may utilize a cellular-based radio technology and to communicate with a base station (114b) that may utilize an IEEE 802 radio technology.

FIG. 1B is a system diagram illustrating an exemplary WTRU (102). As depicted in FIG. 1B, the WTRU (102) may include, among other things, a processor (118), a transceiver (120), a transmit/receive element (122), a speaker/microphone (124), a keypad (126), a display/touchpad (128), non-removable memory (130), removable memory (132), a power source (134), a global positioning system (GPS) chipset (136), and/or other peripherals (138). It will be appreciated that the WTRU (102) may include any sub-combination of the foregoing elements while remaining consistent with the embodiment.

The processor (118) may be a general purpose processor, a special purpose processor, a conventional processor, a digital signal processor (DSP), multiple microprocessors, one or more microprocessors in conjunction with a DSP core, a controller, a microcontroller, an Application Specific Integrated Circuit (ASIC), a Field Programmable Gate Array (FPGA) circuit, any other type of integrated circuit (IC), a state machine, or the like. The processor (118) may perform signal coding, data processing, power control, input/output processing, and/or any other functions that enable the WTRU (102) to operate in a wireless environment. The processor (118) may be coupled to the transceiver (120), which may be coupled to the transmit/receive element (122). Although FIG. 1b depicts the processor (118) and the transceiver (120) as separate components, it will be appreciated that the processor (118) and the transceiver (120) may be integrated together in an electronic package or chip.

The transmit/receive element (122) may be configured to transmit signals to or receive signals from a base station (e.g., base station 114a) over the air interface (116). For example, in one embodiment, the transmit/receive element (122) may be an antenna configured to transmit and/or receive RF signals. In an embodiment, the transmit/receive element (122) may be an emitter/detector configured to transmit and/or receive, for example, IR, UV, or visible light signals. In another embodiment, the transmit/receive element (122) may be configured to transmit and/or receive both RF signals and optical signals. It will be appreciated that the transmit/receive element (122) may be configured to transmit and/or receive any combination of wireless signals.

Although the transmit/receive element (122) is depicted as a single element in FIG. 1B, the WTRU (102) may include any number of transmit/receive elements (122). More specifically, the WTRU (102) may utilize MIMO technology. Thus, in one embodiment, the WTRU (102) may include two or more transmit/receive elements (122) (e.g., multiple antennas) for transmitting and receiving wireless signals over the air interface (116).

The transceiver (120) may be configured to modulate signals to be transmitted by the transmit/receive element (122) and to demodulate signals to be received by the transmit/receive element (122). As noted above, the WTRU (102) may have multi-mode capabilities. Accordingly, the transceiver (120) may include multiple transceivers to enable the WTRU (102) to communicate over multiple RATs, such as, for example, NR and IEEE 802.11.

The processor (118) of the WTRU (102) may be coupled to and receive user input data from a speaker/microphone (124), a keypad (126), and/or a display/touchpad (128) (e.g., a liquid crystal display (LCD) display unit or an organic light-emitting diode (OLED) display unit). The processor (118) may also output user data to the speaker/microphone (124), the keypad (126), and/or the display/touchpad (128). Additionally, the processor (118) may access information from and store data in any type of suitable memory, such as non-removable memory (130) and/or removable memory (132). The non-removable memory (130) may include random-access memory (RAM), read-only memory (ROM), a hard disk, or any other type of memory storage device. The removable memory (132) may include a subscriber identity module (SIM) card, a memory stick, a secure digital (SD) memory card, etc. In other embodiments, the processor (118) may access information from and store data in memory that is not physically located on the WTRU (102), such as on a server or a home computer (not shown).

The processor (118) may receive power from a power source (134) and may be configured to distribute power to and/or control power to other components within the WTRU (102). The power source (134) may be any suitable device for providing power to the WTRU (102). For example, the power source (134) may include one or more dry cell batteries (e.g., nickel cadmium (NiCd), nickel zinc (NiZn), nickel metal hydride (NiMH), lithium ion (Li ion), etc.), a solar cell, a fuel cell, etc.

The processor (118) may also be coupled to a GPS chipset (136) that may be configured to provide location information (e.g., longitude and latitude) regarding the current location of the WTRU (102). In addition to or instead of information from the GPS chipset (136), the WTRU (102) may receive location information from a base station (e.g., base stations 114a, 114b) over the air interface (116) and/or may determine its location based on the timing at which signals are received from two or more nearby base stations. It will be appreciated that the WTRU (102) may obtain location information by any suitable location determination method while remaining consistent with an embodiment.

The processor (118) may be further coupled to other peripherals (138), which may include one or more software and/or hardware modules that provide additional features, functionality, and/or wired or wireless connectivity. For example, the peripherals (138) may include an accelerometer, an e-compass, a satellite transceiver, a digital camera (for photography and/or video), a universal serial bus (USB) port, a vibration device, a television transceiver, a hands-free headset, a Bluetooth® module, a frequency modulated (FM) radio unit, a digital music player, a media player, a video game player module, an Internet browser, a virtual reality and/or augmented reality (VR/AR) device, an activity tracker, and the like. The peripherals (138) may include one or more sensors, including but not limited to a gyroscope, an accelerometer, a Hall effect sensor, a magnetometer, an orientation sensor, a proximity sensor, a temperature sensor, a time sensor; a geolocation sensor; It may be one or more of an altimeter, a light sensor, a touch sensor, a magnetometer, a barometer, a gesture sensor, a biometric sensor, and/or a humidity sensor.

The WTRU (102) may include a full duplex radio in which transmission and reception of some or all of the signals (e.g., associated with particular subframes for both the UL (e.g., for transmitting) and the downlink (e.g., for receiving)) may be concurrent and/or simultaneous. The full duplex radio may include an interference management unit to reduce and/or substantially eliminate self-interference, either via hardware (e.g., a choke) or via signal processing via a processor (e.g., a separate processor (not shown) or the processor 118 ). In an embodiment, the WTRU (102) may include a half-duplex radio in which transmission and reception of some or all of the signals associated with particular subframes for either the UL (e.g., for transmitting) or the downlink (e.g., for receiving).

Although the WTRU is illustrated in FIGS. 1A and 1B as a wireless terminal, it is contemplated that in certain representative embodiments such a terminal may utilize wired communication interfaces with a communications network (e.g., temporarily or permanently).

In representative embodiments, the other network (112) may be a WLAN.

In view of FIGS. 1A and 1B and the corresponding descriptions, one or more, or all, of the functions described herein may be performed by one or more emulation devices (not shown). The emulation devices may be one or more devices configured to emulate one or more, or all, of the functions described herein. For example, the emulation devices may be used to test other devices and/or to simulate network and/or WTRU functions.

The emulation device may be designed to implement one or more tests of other devices in a lab environment and/or in an operator network environment. For example, the one or more emulation devices may be implemented and/or deployed, in whole or in part, as part of a wired and/or wireless communications network to test other devices within the communications network while performing one or more or all of the functions. The one or more emulation devices may be temporarily implemented/deployed as part of a wired and/or wireless communications network while performing one or more or all of the functions. The emulation device may be directly coupled to the other device for testing purposes and/or may perform the tests using over-the-air wireless communications.

One or more emulation devices may perform one or more functions - including all functions - without being implemented/deployed as part of a wired and/or wireless communications network. For example, the emulation devices may be utilized in a test scenario in a test lab and/or in a non-deployed (e.g., test) wired and/or wireless communications network to implement tests for one or more components. The one or more emulation devices may be test equipment. Direct RF coupling and/or wireless communications via RF circuitry (which may include one or more antennas, for example) may be used by the emulation devices to transmit and/or receive data.

FIG. 1C is a system diagram illustrating an exemplary set of interfaces for a system according to some embodiments. An extended reality display device, along with its control electronics, may be implemented. The system (150) may be embodied as a device comprising various components described below and configured to perform one or more of the aspects described herein. Examples of such devices include, but are not limited to, various electronic devices such as personal computers, laptop computers, smart phones, tablet computers, digital multimedia set-top boxes, digital television receivers, personal video recording systems, connected home appliances, and servers. The elements of the system (150), alone or in combination, may be embodied as a single integrated circuit (IC), multiple ICs, and/or discrete components. For example, in at least one embodiment, the processing and encoder/decoder elements of the system (150) are distributed across multiple ICs and/or discrete components. In various embodiments, the system (150) is communicatively coupled to one or more other systems, or other electronic devices, for example, via a communications bus or via dedicated input and/or output ports. In various embodiments, the system (1000) is configured to implement one or more of the aspects described herein.

The system (150) includes at least one processor (152) configured to execute instructions loaded therein to implement, for example, various aspects described herein. The processor (152) may include embedded memory, input/output interfaces, and various other circuits as are known in the art. The system (150) includes at least one memory (154) (e.g., a volatile memory device and/or a nonvolatile memory device). The system (150) may include a storage device (158) which may include nonvolatile memory and/or volatile memory, including but not limited to Electrically Erasable Programmable Read-Only Memory (EEPROM), Read-Only Memory (ROM), Programmable Read-Only Memory (PROM), Random Access Memory (RAM), Dynamic Random Access Memory (DRAM), Static Random Access Memory (SRAM), flash, magnetic disk drives, and/or optical disk drives. The storage device (158) may include, but is not limited to, an internal storage device, an attachable storage device (including removable and non-removable storage devices), and/or a network accessible storage device.

The system (150) includes an encoder/decoder module (156) configured to process data to provide, for example, encoded video or decoded video, wherein the encoder/decoder module (156) may include its own processor and memory. The encoder/decoder module (156) represents a module(s) that may be included in a device to perform encoding and/or decoding functions. As will be appreciated, a device may include one or both of an encoding module and a decoding module. Additionally, the encoder/decoder module (156) may be implemented as a separate element of the system (150) or may be integrated into the processor (152) as a combination of hardware and software, as will be appreciated by those skilled in the art.

Program code to be loaded into the processor (152) or the encoder/decoder (156) to perform the various aspects described herein may be stored in the storage device (158) and subsequently loaded into the memory (154) for execution by the processor (152). According to various embodiments, one or more of the processor (152), the memory (154), the storage device (158), and the encoder/decoder module (156) may store one or more of various items during performance of the processes described herein. Such stored items may include, but are not limited to, input video, decoded video or portions of decoded video, a bitstream, matrices, variables, and intermediate or final results from processing equations, formulas, operations and computational logic.

In some embodiments, memory within the processor (152) and/or the encoder/decoder module (156) is used to store instructions and provide working memory for processing necessary during encoding or decoding. However, in other embodiments, memory external to the processing device (e.g., the processing device may be either the processor (152) or the encoder/decoder module (152)) is used for one or more of these functions. The external memory may be memory (154) and/or a storage device (158), such as dynamic volatile memory and/or nonvolatile flash memory. In several embodiments, the external nonvolatile flash memory is used to store, for example, an operating system of the television. In at least one embodiment, high-speed external dynamic volatile memory, such as RAM, is used as working memory for video coding and decoding operations, such as, for example, MPEG-2 (MPEG stands for Moving Picture Experts Group; MPEG-2 is also known as ISO/IEC 13818; 13818-1 is also known as H.222 and 13818-2 is also known as H.262), HEVC (HEVC stands for High Efficiency Video Coding; also known as H.265 and MPEG-H Part 2), or VVC (Versatile Video Coding, a new standard under development by the Joint Video Experts Team (JVET)).

Inputs to elements of the system (150) may be provided via a variety of input devices, as illustrated in block (172). Such input devices include, but are not limited to, (i) a radio frequency (RF) portion for receiving RF signals transmitted over the air, for example, by a broadcaster, (ii) a component (COMP) input terminal (or a set of COMP input terminals), (iii) a Universal Serial Bus (USB) input terminal, and/or (iv) a High Definition Multimedia Interface (HDMI) input terminal. Other examples not illustrated in FIG. 1c include composite video.

In various embodiments, the input devices of block (172) have their own associated input processing elements as is known in the art. For example, the RF portion may be associated with elements suitable for (i) selecting a desired frequency (also referred to as selecting a signal or band-limiting a signal to a frequency band), (ii) down-converting the selected signal, (iii) further band-limiting the signal to a narrower frequency band (for example) which may in certain embodiments be referred to as a channel, (iv) demodulating the down-converted and band-limited signal, (v) performing error correction, and (vi) demultiplexing to select a desired data packet stream. The RF portion of various embodiments may include one or more elements for performing these functions, such as a frequency selector, a signal selector, a band-limiter, a channel selector, a filter, a down-converter, a demodulator, an error corrector, and a demultiplexer. The RF portion may include a tuner that performs many of these functions, including, for example, downconverting a received signal to a lower frequency (e.g., an intermediate frequency or near-baseband frequency) or to baseband. In one set-top box embodiment, the RF portion and its associated input processing elements perform frequency selection by receiving an RF signal transmitted over a wired (e.g., cable) medium, filtering it to a desired frequency band, downconverting it, and filtering it again. Various embodiments rearrange the order of the elements described above (and others), removing some of these elements, and/or adding other elements that perform similar or different functions. Adding elements may include inserting elements between existing elements, such as inserting amplifiers and analog-to-digital converters. In various embodiments, the RF portion includes an antenna.

Additionally, the USB and/or HDMI terminals may include their own interface processors for connecting the system (150) to other electronic devices via the USB and/or HDMI connections. It should be appreciated that various aspects of input processing, such as Reed-Solomon error correction, may be implemented, as desired, within separate input processing ICs or within the processor (152). Similarly, aspects of USB or HDMI interface processing may be implemented, as desired, within separate interface ICs or within the processor (152). The demodulated, error corrected, and demultiplexed stream is provided to various processing elements, including, for example, the processor (152) and an encoder/decoder (156) that operate in conjunction with memory and storage elements to process the data stream as desired for presentation on an output device.

The various elements of the system (150) may be provided within an integrated housing. Within the integrated housing, the various elements may be interconnected and data may be transmitted therebetween using suitable connection arrangements (174), for example, an Inter-IC (I2C) bus, wiring, and printed circuit boards, as known in the art.

The system (150) includes a communication interface (160) that enables communication with other devices via a communication channel (162). The communication interface (160) may include, but is not limited to, a transceiver configured to transmit and receive data via the communication channel (162). The communication interface (160) may include, but is not limited to, a modem or a network card, and the communication channel (162) may be implemented within, for example, wired and/or wireless media.

Data is, in various embodiments, streamed or otherwise provided to the system (150) using a wireless network, such as a Wi-Fi network, for example, an IEEE 802.11 (wherein IEEE stands for Institute of Electrical and Electronics Engineers). The Wi-Fi signal of such embodiments is received via a communication channel (162) and a communication interface (160) suitable for Wi-Fi communications. The communication channel (162) of such embodiments is typically connected to an access point or router that provides access to external networks, including the Internet, to enable streaming applications and other over-the-top communications. Other embodiments provide the streamed data to the system (150) using a set-top box that passes the data via an HDMI connection of the input block (172). Still other embodiments provide the streamed data to the system (150) using an RF connection of the input block (172). As noted above, various embodiments provide the data in a non-streaming manner. Additionally, various embodiments use wireless networks other than Wi-Fi, such as a cellular network or a Bluetooth network.

The system (150) can provide output signals to various output devices, including a display (176), speakers (178), and other peripheral devices (180). The display (176) of various embodiments includes, for example, one or more of a touchscreen display, an organic light-emitting diode (OLED) display, a curved display, and/or a foldable display. The display (176) can be for a television, a tablet, a laptop, a cell phone (mobile phone), or other device. The display (176) can also be integrated with other components (e.g., in a smart phone) or separate (e.g., as an external monitor for a laptop). The other peripheral devices (180), in various examples of embodiments, include one or more of a standalone digital video disc (or digital versatile disc) (for both terms, a DVR), a disc player, a stereo system, and/or a lighting system. Various embodiments utilize one or more peripheral devices (180) that provide functionality based on the output of the system (150). For example, a disc player performs the function of reproducing the output of the system (150).

In various embodiments, control signals are communicated between the system (150) and the display (176), speakers (178), or other peripheral devices (180) using signaling such as AV.Link, Consumer Electronics Control (CEC), or other communication protocols that enable device-to-device control with or without user intervention. The output devices may be communicatively coupled to the system (1000) via dedicated connections through their respective interfaces (164, 166, and 168). Alternatively, the output devices may be communicatively connected to the system (150) using a communication channel (162) via the communication interface (160). The display (176) and speakers (178) may be integrated into a single unit with other components of the system (150), for example in an electronic device such as a television. In various embodiments, the display interface (164) includes a display driver, such as a timing controller (T Con) chip.

The display (176) and speakers (178) may alternatively be separate from one or more of the other components, for example, if the RF portion of the input (172) is part of a separate set-top box. In various embodiments where the display (176) and speakers (178) are external components, the output signal may be provided via dedicated output connections, including, for example, HDMI ports, USB ports, or COMP outputs.

The system (150) may include one or more sensor devices (168). Examples of sensor devices that may be used include one or more GPS sensors, gyroscopes, accelerometers, optical sensors, cameras, depth cameras, microphones, and/or magnetometers. Such sensors may be used to determine information such as the user's position and orientation. When the system (150) is used as a control module for an extended reality display (such as the control modules (124, 132)), the user's position and orientation may be used to determine how to render image data so that the user perceives the correct portion of a virtual object or virtual scene at the correct point in time. In the case of a head-mounted display device, the position and orientation of the device itself may be used to determine the user's position and orientation for the purpose of rendering virtual content. In the case of other display devices, such as a phone, tablet, computer monitor, or television, other inputs may be used to determine the user's position and orientation for the purpose of rendering content. For example, a user may select and/or adjust a desired viewpoint and/or orientation using a touchscreen, keypad or keyboard, trackball, joystick, or other input. If the display device has sensors such as an accelerometer and/or gyroscope, the viewpoint and orientation used for rendering content may be selected and/or adjusted based on movement of the display device.

The embodiments may be performed by computer software implemented by the processor (152), by hardware, or by a combination of hardware and software. As a non-limiting example, the embodiments may be implemented by one or more integrated circuits. The memory (154) may be of any type suitable to the technical environment, and may be implemented using any suitable data storage technology, such as, but not limited to, optical memory devices, magnetic memory devices, semiconductor-based memory devices, fixed memory, and removable memory. The processor (152) may be of any type suitable to the technical environment, and may include, but not limited to, one or more of a microprocessor, a general purpose computer, a special purpose computer, and a multi-core architecture-based processor.

The present application discusses point cloud processing and compression, including processing, compression, representation, analysis and understanding of point cloud signals. The present application also discusses a deep neural network-based adaptive voxel-based point cloud upsampling method, including some embodiments applied to point cloud processing and compression. The present application also discusses performing upsampling in the voxel domain.

Point cloud data can consume a large portion of network traffic, for example, between connected cars over 5G networks and in immersive (e.g., AR/VR/MR) communications. Efficient representation formats can be used for point cloud and communication. In particular, raw point cloud data can be organized and processed for modeling and sensing, such as the world, environment or scene. Compression of the raw point cloud can be used for storage and transmission of the data.

In addition, point clouds may represent sequential scans of the same scene, which may include multiple moving objects. Dynamic point clouds capture moving objects, whereas static point clouds capture static scenes and/or static objects. Dynamic point clouds may typically consist of frames, with different frames captured at different times. Processing and compression of dynamic point clouds may be performed in real time or with a small amount of delay.

For example, the automotive industry, including autonomous vehicles, may use point clouds. An autonomous vehicle "probes" its environment to make driving decisions based on its immediate surroundings. Typically, a LiDAR sensor produces a (dynamic) point cloud that is used by the perception engine. Furthermore, typically, such point clouds are dynamic, sparse, not necessarily colored, and invisible to the human eye, with a high capture rate. Such point clouds may include other properties, such as reflectance ratio, provided by the LiDAR, that may indicate the material of the detected object and may be used in decision making.

The automotive industry and autonomous vehicles are some of the areas where point clouds can be used. Autonomous vehicles “navigate” and sense their environment in order to make good driving decisions based on the reality of their immediate surroundings. Sensors such as LiDAR generate (dynamic) point clouds that are used by the perception engine. These point clouds are typically not intended to be visible to the human eye, they may or may not be colored, and they are typically sparse and dynamic with a high capture rate. These point clouds may have other properties such as reflectivity provided by the LiDAR because this property indicates the material of the detected object and can be helpful in decision making.

Virtual reality (VR) and immersive worlds have become hot topics and are seen by many as the future of 2D flat video. Unlike standard TV where the viewer only sees the virtual world in front of them, viewers can be immersed in an all-around environment. There are different levels of immersion depending on the viewer's degree of freedom in the environment. Point cloud formats can be used to distribute VR world and environment data. These point clouds can be static or dynamic and are typically of average size, such as less than a million points at a time.

Point clouds can also be used for a variety of other purposes, such as scanning cultural heritage objects and/or buildings, where objects such as statues or buildings are scanned in 3D. The spatial configuration data of the object can be shared without sending or visiting the actual object or building. Additionally, this data can be used to preserve knowledge about the object in case the object or building is destroyed, such as a temple being destroyed by an earthquake. These point clouds are typically static, colored, and huge in size.

Another use case is in topography and cartography, where the map is not limited to a flat surface and can include relief, using a 3D representation. For example, Google Maps can use meshes instead of point clouds for its 3D maps. Nevertheless, point clouds can be a suitable data format for 3D maps, and these point clouds are typically also static, colored, and huge in size.

World modeling and sensing via point clouds allows machines to record and use spatial configuration data about the 3D world around them, which can be used in the applications discussed above.

3D point cloud data contains discrete samples of the surface of an object or scene. To fully represent the real world with point samples, a huge number of points can be used. For example, a typical VR immersive scene contains millions of points, while a point cloud can typically contain hundreds of millions of points. Therefore, processing such large point clouds is computationally expensive, especially for consumer devices that may have limited computational capabilities, such as smartphones, tablets, and car navigation systems.

To store and process the input point cloud at low computational cost, the input point cloud can be downsampled, where the downsampled point cloud summarizes the geometry of the input point cloud while having much fewer points. The downsampled point cloud is input to a subsequent machine task for further processing. The downsampled point cloud can be processed by incrementally upsampling the point cloud. In particular, in some embodiments, a learning-based autoencoder architecture can use downsampling for feature extraction and upsampling for reconstruction. For example, such upsampling can be used for point cloud compression (e.g., in the decoder) and for point cloud super-resolution. Among others, the present application discusses an example of an adaptive point cloud upsampling method, according to some embodiments.

Fig. 2a is a schematic illustration showing an exemplary voxel-based representation of a point cloud. In a voxel-based representation of point cloud data, 3D point coordinates are uniformly quantized by a quantization step. As illustrated in Fig. 2a, each point in the representation (200) corresponds to an occupied voxel having the same size as the quantization step. “Simple (Na The “ve)” voxel representation may not be memory efficient since most of the voxels may be empty. Therefore, a sparse voxel representation is introduced where occupied voxels are arranged in a sparse tensor format for efficient storage and processing.

Figure 2b is a schematic illustration showing an exemplary sparse voxel-based representation of a point cloud. In the example sparse voxel representation (250) depicted in Figure 2b, empty voxels (252) (represented by a dashed line) do not necessarily consume as much memory or storage as occupied voxels (254) (represented by a solid diagonal line).

Note that the remaining drawings, including FIGS. 2A and 2B and FIGS. 3, 4, 5, 8, and 9, illustrate point clouds in 2D for illustrative and simplistic purposes only, and that the concepts are generally applicable and extendable to 3D.

By representing the point cloud as a 3D voxel, the point cloud can be processed (digested) by a 3D convolutional neural network. The application of a 2D convolutional neural network to a 2D image has been successful. In the case of a typical 3D convolution, a 3D kernel is overlaid at every location specified by the stride step, regardless of whether the voxel is occupied or empty. The stride represents the amount of movement or step size on the 3D voxel grid when applying the convolution. When the step size is set to 1, the 3D kernel typically slides over every voxel in the 3D space and computes the output. In this case, the dimensions (height, width, depth) of the output voxel space are the same as the dimensions of the input voxel space. When the stride step is set to 2, the 3D kernel slides over every other two voxels and computes the output. In this case, all dimensions of the output voxel space are half of the input voxel space. An empty voxel is a voxel that does not have a 3D point at its location. To avoid the computation and memory consumption caused by empty voxels, a sparse 3D convolution layer can be applied if the point cloud voxels are represented as sparse tensors.

Figure 3 is a schematic process diagram showing an example nearest neighbor (NN) upsampling for a point cloud. For example, to perform two-fold (2x) upsampling in the voxel domain, an occupied voxel is split into two voxels along both the x- , y- , and z- directions. Thus, one (occupied) parent voxel is split into eight (= ) occupied child voxels, and the resolution of the point cloud is upsampled by a factor of two in each dimension (x, y, and z). According to this exemplary approach, if there is any feature vector associated with a parent voxel, this feature vector is directly inherited by its eight child voxels. The mechanism of this upsampling approach is depicted in Fig. 3 and is referred to as nearest neighbor (NN) upsampling. In the example of Fig. 3, after the input point cloud (302) is NN upsampled (304), the output point cloud (306) and its features are input to a subsequent pipeline for further processing.

This NN upsampling approach has several drawbacks. First, after upsampling, the geometry of the point cloud is a simple extension of the original geometry - there is no refinement in the shape of the point cloud. Second, the number of occupied voxels after upsampling is always 8 times that of the original, which can lead to high computational costs in subsequent processing.

Figure 4 is a schematic process diagram showing an example voxel-based upsampling using pruning. In the paper Wang, Jianqiang, et al., Multiscale Point Cloud Geometry Compression , 2021 Data Compression Conference (DCC) 73-82, IEEE (2021) (“ Wang ”), the authors proposed a method to further refine the upsampled geometric structures by binary classification and voxel pruning. Figure 4 illustrates the approach taken in Wang .

According to an exemplary approach (400), an input point cloud PC ₀ (402) is first upsampled with a NN upsampling block (404) (as shown in FIG. 4 ), resulting in an initially upsampled point cloud PC ₁ (406). The PC ₁ is input to a neural network-based binary classifier (408), which determines an occupancy state (410) for each of the occupied voxels in PC ₁ . The initially upsampled point cloud PC ₁ is then pruned (412) by removing all voxels classified as unoccupied ("0" in FIG. 4 ). The refined and upsampled PC ₂ (414) is the output. For an example of pruning, see FIG. 21 .

This approach addresses the two aforementioned shortcomings of the NN upsampling method of Fig. 3. However, in many applications, the success of the binary classifier is critical for accurate geometric segmentation. In some embodiments, the present application improves the performance of the binary classification by introducing additional voxel context information.

The feature information according to the approach of Fig. 4 is a high-level, abstract descriptor of the geometric structure generated by the deep neural network. This feature information may, in some cases, be too abstract to be sufficient for accurate classification. In contrast, the context information introduced in the present application may include local, per voxel, knowledge that may be a clue helpful for binary classification.

FIG. 5 is a schematic process diagram illustrating an exemplary context-aware voxel-based upsampling using pruning according to some embodiments. The present application initially (“naively”) introduces a context point cloud that conveys additional known information about the upsampled point cloud PC ₁ . By associating (associating) (512) the context point cloud (510) with the upsampled point cloud PC ₁ (506), the subsequent binary classification (516) and voxel pruning (520) processes can better refine the initial upsampled point cloud PC ₁ (506), which can generate a more accurate upsampled point cloud PC ₂ (522). In some embodiments, the voxel pruning process (520) takes as input an upsampled point cloud PC ₁ (506) and a binary classified point cloud PC" ₁ (518) and generates an output point cloud PC ₂ (522). The block diagram of FIG. 5 illustrates a voxel-based upsampling method (500) for some embodiments. The voxel-based upsampling method performs classification (516) and pruning (520) to refine the upsampled geometry in addition to the "simple" upsampled point cloud PC ₁ (e.g., due to NN upsampling (504) of the input point cloud _PC 0 (502)). In FIG. 5, a context point cloud PC _CTX (510) that conveys context information is introduced. The context construction block (508) takes as input an initial upsampled point cloud PC ₁ (506) and generates a context point cloud PC Outputs _{a CTX} (510). The context point cloud PC _CTX (510) is "simply" concatenated with the upsampled point cloud PC ₁ (506) to generate an augmented point cloud (514) as input to a binary classification stage (516). In this example, the context point cloud PC _CTX (510) shares the same geometry as PC ₁ (506), while PC _CTX (510) is intended to include context information (e.g., voxel-wise discriminative information) for predicting ground truth occupancy state (in this case ground truth occupancy state for a voxel, whether its child voxels are "empty" or "occupied"). The augmented point cloud PC' ₁ (514) is generated by concatenating (512) features of PC _CTX (510) and PC ₁ (506).

Concatenation is a commonly used operator in deep neural networks. The concatenation operator concatenates (all) features in PC _CTX with their corresponding features in PC ₁ to generate the augmented point cloud PC' ₁ . If an occupied voxel ( x , y , z ) in PC _CTX has an associated context information vector c of length a , while the same location ( x , y , z ) in PC ₁ has an associated feature vector f ₁ of length b , the concatenation operator concatenates c and f ₁ together to generate another feature vector [ cf ₁ ] of length ( a + b ). This generated feature vector [ cf ₁ ] will be assigned to the voxel location ( x , y , z ) in the augmented point cloud PC' ₁ . This step can be performed for all occupied voxels in PC _CTX and PC ₁ to generate the augmented point cloud PC' ₁ . Augmented point cloud PC' ₁ replaces PC ₁ , which is the input to the binary classifier.

In some embodiments, the context information can be any known knowledge or known context about the voxel. For example, the context information can be the location of the voxel, such as [x, y, z] coordinates. The context information can include, for example, the bit depth of the input point cloud. The context information can be other information, such as, for example, the relative location of the voxel to its parent voxel. However, the context is not limited to location information, and in some embodiments can include other types of information in addition to or instead of location information. In some embodiments, the context information (e.g., voxel-wise discriminant information) can be used to predict the ground truth occupancy state of the voxel (e.g., whether the voxel is "empty" or "occupied"), because the context information can provide some information that is already known about the voxel. By incorporating such known context information, the deep neural network can better infer the occupancy state.

In some embodiments, the context information is obtained from processing the input point cloud. The context information can be any information about the voxel and can be determined even before encoding the voxel. For example, assume that the current bit depth of the voxel for the location [x, y, z] of the voxel and [x, y, z] information of the voxel are known. Therefore, the context information can be expressed in spherical coordinates. See Equations 1, 2, and 3 below.

The context information PC _CTX is obtained from the input point cloud PC _0. In some embodiments, there is no other external source of context information. For example, the context information may be (x, y, z) position coordinates. Context information vector can be directly assigned to the voxel location (x, y, z) in the PC _CTX . In some embodiments, this (x, y, z) coordinate location is preprocessed and transformed to another coordinate system, for example via Equations 1 to 3 as described below, and can be assigned to the voxel location (x, y, z) in the PC _CTX .

In some embodiments, the context information includes x , y , and z coordinates. For example, for an occupied voxel ( x , y , z ) in a PC _CTX , the context information is a vector can be. In some embodiments, normalized coordinates are used as context information. Assume that PC ₁ has a bit depth of N , which means that PC ₁ has dimension 2 ^N x 2 ^N x 2 ^N. Therefore, the context information vector associated with a voxel ( x , y , z ) in PC _CTX is It could be.

Moreover, instead of working in Euclidean coordinates, spherical coordinates can be used, which is particularly useful for handling LiDAR sweeps. To do this, the following mathematical equations 1, 2, and 3 are applied:

where N is the bit depth, r is the radial distance, φ is the elevation angle, and θ is the azimuth angle. The vector c is This is or if the distance is normalized This becomes. In some embodiments, the context information can also be the bit depth of PC ₁ , i.e., N. In this case, the feature is a constant scalar c = N. Furthermore, instead of working in Euclidean or spherical coordinates, cylindrical coordinates can be used because cylindrical coordinates can be used to process LiDAR sweeps. To do so, Equations 1 and 3 are, respectively, expressed in terms of the radial distance ( r ) and the azimuth ( ) is applied to calculate the Euclidean coordinates ( x , y , z ) of the voxel. When the Euclidean coordinates ( x , y , z ) of the voxel are converted to cylindrical coordinates, Therefore, the vector c containing the context information is This is or if the distance is normalized Providing context information in different ways can facilitate the task of the binary classification process. For example, in some cases, including height z can help classification if height z is particularly relevant to the occupancy state of a voxel. Another example is LiDAR point clouds, where height is in a reasonable range, for example, height is greater than 0 because LiDAR cannot detect something underground. And in some embodiments, azimuth Including azimuth may aid in classification if (equation 3) is highly relevant to occupancy.

In some embodiments, the encoder can operate in the opposite direction to that illustrated in FIG. 5. For example, the encoder can obtain a first point cloud, such as a pruned point cloud (522). A voxel occupancy state of the point cloud can be determined. A second point cloud can be generated by removing voxels determined to be empty from the first point cloud. Features of the second point cloud can be determined and associated with context information to generate a third point cloud. In some embodiments, features of the second point cloud can be associated with the context information. The third point cloud can be downsampled to obtain a fourth point cloud. The fourth point cloud can be output as an encoder output.

FIG. 6A is a table illustrating exemplary location values according to some embodiments. In some embodiments, context information may be the location of a child voxel relative to its parent voxel. For example, "front"/"back", "left"/"right", and "top"/"bottom" may be represented as "0" and "1", respectively, as shown in Tables 600, 602, and 604 of FIG. 6A . In other words, the leftmost value in the feature array represents the front/back state, the center value represents the left/right state, and the rightmost value represents the top/bottom state. A 0 for the front/back state represents front, while a 1 for the front/back state represents back. A 0 for the left/right state represents left, while a 1 for the left/right state represents right. A 0 for the top/bottom state represents top, while a 1 for the top/bottom state represents bottom.

Fig. 6b is a schematic perspective diagram illustrating exemplary child voxel locations as context information according to some embodiments. As shown in Fig. 6b, a child voxel of PC ₁ located in front, right and above its parent voxel (650) is provided with 3-bit context information On the other hand, the child voxel of PC ₁ (650), located posterior, right and above its parent voxel, has a 3-bit context feature (652). In some embodiments, the 3-bit context feature vector may be interpreted as binary and converted to decimal, for example, becomes the scalar c = 2 becomes a scalar c = 6. In some embodiments, the context information may be any portion, combination and/or permutation of the exemplary context features mentioned above.

Going back to Figure 5, Features in the upper left corner of the Consider the following example. This example shows an input point cloud before the binary classification block. Augmented point cloud in Of the Follows .

Features of and other features Each is a one-dimensional vector. In this non-limiting example, the vector length will be 5. Therefore, each feature vector will be a 1x5 vector with 5 numbers. This feature vector is contains information about the actual occupancy status of the object and is therefore called a geometric feature. For example, is in the upper left corner While it contains information about the actual occupancy status of; is in the upper right corner Contains information about the actual occupancy status.

In this example, are abstract, high-level features/descriptors generated by deep neural networks. In this example, The values of do not have a specific physical meaning and are therefore abstract and "high-level". However, these values provide meaning to the neural network itself for performing inference. These vector features, , which is chosen to show the movement of Assuming that it is a random number, such as

As part of, as shown in Fig. 5, is passed through the NN upsampling block, which is in the upper left corner of Creates four copies of . is passed to the context configuration block, which is in the upper left corner of Generate four context information vectors corresponding to each other. These four corresponding context information vectors are , and Assuming that the context configuration block constructs a context information vector using the coordinates ( x , y , z ) of the voxel (or just ( x , y ) in the 2D example of Fig. 5),

And,

And,

And,

am.

and is entered into the connection block, point cloud In some embodiments, the connection block may perform the following exemplary connections:

silver Connected with, a new vector:

The voxel in the upper left corner (first row, first column) of

Creates .

Is Connected with, a new vector:

The voxel in the first row and second column of

Creates.

silver Connected with, a new vector:

The voxel in the second row, first column of

Creates.

Is Connected with, a new vector:

The voxel in the second row, second column of

Creates.

Acquired point cloud silver The ground truth point cloud in which the voxels estimated to be occupied are determined It is input into a binary classification block to determine whether it is actually occupied or not.

FIG. 7 is a flowchart illustrating an exemplary process for cascading multiple context-aware upsamplings according to some embodiments. In some embodiments of the exemplary process (700), the context-aware voxel-based upsamplings (702, 706, 710) can be cascaded multiple times to achieve a higher upsampling ratio, as illustrated in FIG. 7. In this case, a feature aggregation block (704, 708, 712) can be inserted between two consecutive upsamplings (702, 706, 710) for segmentation and feature aggregation. For example, the feature aggregation block can take as input a sparse tensor with features having N channels. The feature aggregation block modifies the features to better suit the compression task. In particular, to obtain a high-quality reconstruction for point cloud decompression, the feature aggregation block generates descriptive or unique geometric features that can represent local geometric details. The output features still have N channels, which means that the feature aggregation block does not change the shape of the sparse tensor. In some embodiments, the feature aggregation can be, for example, a cascaded sparse convolutional layer architecture, a residual network (ResNet) architecture, an Inception-ResNet (IRN) architecture, or a transformer block. In some embodiments, the context-aware upsampling can be performed by the process illustrated in FIG. 5.

In some embodiments, the feature aggregation block illustrated in FIG. 7 may include a weight sharing mechanism. In FIG. 7, multiple context-aware blocks are cascaded (in series). In some embodiments, the feature aggregation blocks and the context-aware upsampling blocks of FIG. 7 may share the same set of neural network parameters.

FIG. 8 is a schematic process diagram showing an exemplary context-aware voxel-based upsampling using initial feature aggregation according to some embodiments. In some embodiments, a feature aggregation block (806) can be inserted immediately after the NN upsampling block (804) for initial feature segmentation, as illustrated in FIG. 8. The features of PC ₀ (802) are passed to the features of PC ₁ (808) based on the NN upsampling (804) assignment. Additionally, a concatenation block (814) is inserted before the binary classification block (818), similar to that illustrated in FIG. 5. In some embodiments, the blocks of FIG. 8 operate identically to the blocks described in FIG. 5, with the addition of a feature aggregation block.

As in FIG. 5, the context building block (810) takes as input an initial upsampled point cloud PC ₁ (808) and outputs a context point cloud PC _CTX (812). By concatenating (814) the context point cloud (812) with the upsampled point cloud PC ₁ (808), subsequent binary classification (818) and voxel pruning (822) processes can refine the initial upsampled point cloud PC ₁ (808), which can generate a more accurate upsampled point cloud PC ₂ (824). The augmented point cloud PC' ₁ (816) is generated by concatenating (814) features of the PC _CTX (812) and PC ₁ (808). In some embodiments of the exemplary process (800), the voxel pruning process (822) takes as input an upsampled point cloud PC ₁ (808) and a binary classified point cloud PC'' ₁ (820) and generates an output point cloud PC ₂ (824).

FIG. 9 is a flowchart illustrating an exemplary process for binary classification according to some embodiments. FIG. 9 may be considered a method for performing binary classification using a neural network. A binary classifier may be used to predict a ground truth occupancy state for each occupied voxel in an input point cloud (PC ₁ ). The binary classifier classifies each occupied voxel in PC ₁ as 1 (occupied) or 0 (empty) so that the geometry of PC ₁ can be refined. In some embodiments, the binary classification process (900) of FIG. 9 may be used in the binary classification blocks of FIGS. 5 and 8 .

In Fig. 9, the connected point cloud PC' ₁ is input to the feature aggregation block for feature segmentation and extraction, and the output channel size is D ₁ . The input point cloud goes through feature aggregation (902). The aggregated features are then input to multilayer perceptron (MLP) layers (904) having channel dimensions ( D ₁ , D ₂ , …, 1) for classification. The MLP layer is a neural network layer that applies a linear mapping to the input feature vector. For example, to map an input feature of length D ₁ to an output feature of length D ₂ , the MLP layer multiplies the input feature by a matrix of size D ₁ × D ₂ to generate an output feature of length D ₂ . When cascading two MLP layers, a nonlinear activation function (e.g., ReLU function) is inserted between the two MLP layers. The output is input to a softmax function (906), and the softmax function (906) converts the MLP output value into a range of 0 to 1. For values greater than 0.5 and less than or equal to 1, the threshold processing block (908) converts the value to 1 to indicate an occupied status. For values in the range of 0 to 0.5, the threshold processing block converts the value to 0 to indicate an empty status, as reflected in the binary classified output (910).

Figures 10-13 illustrate four different design choices for feature aggregation for some embodiments. For example, in some embodiments, the feature aggregation block can be a cascaded sparse convolutional layer architecture (1000) (e.g., FIG. 10 ), a residual network (ResNet) architecture (1100) (e.g., FIG. 11 ), an Inception-ResNet (IRN) architecture (1200) (e.g., FIG. 12 ), or a transformer block architecture (1300) (e.g., FIG. 13 ).

FIG. 10 is a block diagram illustrating an exemplary process having cascaded sparse convolutional layers for feature aggregation according to some embodiments. In some embodiments, such as the example illustrated in FIG. 10 , two blocks are repeated multiple times to form a series. The two blocks are a sparse 3D convolutional layer (1002, 1006, 1010) (“CONV D”) followed by a ReLU activation (1004, 1008, 1012) (“ReLU”). In the example illustrated in FIG. 10 , “CONV D ” represents a sparse 3D convolutional layer having D output channels. The “ReLU” activation refers to a rectifier linear unit activation function. For example, a ReLU activation block may output 0 for a negative input value and may output the input multiplied by a scalar value for a positive input value. In other embodiments, the ReLU activation function can be replaced with other activation functions, such as the tanh() activation function and/or the sigmoid() activation function. In some embodiments, the nonlinear activation process can include a rectifier linear unit (ReLU) activation process.

FIG. 11 is a block diagram illustrating an exemplary ResNet block for feature aggregation according to some embodiments. In some embodiments, as illustrated in FIG. 11 , the feature aggregation process may use a ResNet architecture. The paper He, Kaiming, et al., Deep Residual Learning for Image Recognition , Proceedings of the IEEE Conf. on Computer Vision and Pattern Recognition 770-778, IEEE (2016) (“ He ”) describes an exemplary ResNet architecture. See, for example, the rightmost process line of FIG. 3 on page 4 of He .

An example in Fig. 11 shows a ResNet block architecture that aggregates features using D channels. In some embodiments, such as the example illustrated in Fig. 11, two blocks are repeated multiple times to form a series. The two blocks are a sparse 3D convolutional layer (1102, 1106, 1110) (“CONV D”) followed by a ReLU activation (1104, 1108, 1112) (“ReLU”). Compared to Fig. 10, Fig. 11 introduces a residual connection (1114) to the input to add the input to the output of a series of convolutional layers.

Fig. 12 is a block diagram illustrating an exemplary Inception-ResNet block for feature aggregation according to some embodiments. In some embodiments, as illustrated in Fig. 12, feature aggregation can be constructed with an Inception-ResNet (IRN) architecture. Figure 1(b) of Wang also illustrates an IRN architecture. The example in Fig. 12 shows the architecture of an IRN block that aggregates features using D channels. The IRN block splits the feature aggregation process into three parallel paths.

A path with more convolutional layers (the left path in Figure 12) aggregates (more) global information into a larger receptive field. In some embodiments, the left path may include a convolutional layer (1202), followed by two sets of ReLU activations (1204, 1208) and a convolutional layer (1206, 1210).

A path with fewer convolutional layers (the middle path in Figure 12) aggregates local details into a smaller receptive field. In some embodiments, the middle path may include a convolutional layer (1212), followed by a ReLU activation (1214), and a convolutional layer (1216).

The last path (1220) (right path in Fig. 12) is a residual connection that passes the input directly to the output, similar to the residual connection in Fig. 11. In some embodiments, a ReLU block may be inserted after the CONV D/2 block and before the connection (1218) in each of the left path and the middle path in Fig. 12.

FIG. 13 is a block diagram illustrating an exemplary transformer block for feature aggregation according to some embodiments. Section 3.2 of the paper Mao, Jiageng, et al., Voxel Transformer for 3D Object Detection , Proceedings of the IEEE/CVF International Conference on Computer Vision 3164-3173, IEEE (2021) (“ Mao ”) discusses a voxel transformer. In some embodiments, the transformer architecture of the present application may be similar to the voxel transformer of Mao .

A diagram of the transformer block is illustrated in Fig. 13. The output of the self-attention block (1302) is added to the input of the self-attention block via a residual connection. An MLP block (1304) is connected in series with the result of the addition, and the output of the MLP block (1304) is added to the input of the MLP block (1304) via another residual connection. The MLP block includes a series of multilayer perceptron (MLP) layers.

FIG. 14 is a block diagram illustrating an exemplary architecture of a self-attention block according to some embodiments. In some embodiments, the self-attention block (1302) of FIG. 13 may be implemented using the architecture (1400) described in FIG. 14. Voxel location Current feature vector associated with and voxel locations Its k neighboring features are associated with - Here )for is in the input sparse tensor. Given the k nearest neighbors of - , the self-attention block (1400) selects all neighboring features Features based on Try to update the points Is It is obtained by k nearest neighbor (kNN) search (1402) based on the coordinates of . for The query (1404) embedding is calculated by mathematical expression 4:

Here (1404) represents the MLP layers for obtaining queries. are voxels and is the position encoding (1410) between, which is calculated by mathematical expression 5:

Here (1410) represents the MLP layers used to obtain positional encoding. and are, respectively, voxels and is the 3D coordinate for the center of . Key embeddings of all nearest neighbors of and value embedding is calculated using Equations 6 and 7:

Here (1406) and (1408) are MLP layers for obtaining keys and values, respectively. The self-attention block is located as given by Equation 8. Output characteristics of and prints:

Here is the softmax regularization function (1412), is a feature vector is the length of, is a predefined constant.

Mathematical expressions 4 to 8 are illustrated in Fig. 14. Mathematical expression 4 is illustrated in the upper left of Fig. 14, where Block (1404) is the current feature vector Take as input In the upper part of Fig. 14, the kNN block (1402) outputs the current feature vector Take as input, Based on the coordinates of the k nearest neighbor (kNN) search is performed. The output of the kNN block (1402) is for is the coordinate of. Mathematical expression 5 is shown in the upper right corner of Fig. 14, where Block (1410) is a voxel and Take the difference between the two as input . Mathematical expression 6 is shown in the left center part of Fig. 14, where the feature vector Is Input for block (1406), The output of block (1406) is Added to for . Mathematical expression 7 is shown in the right center part of Fig. 14, where the feature vector Is Input for block (1408), The output of block (1408) is Added to for When mathematical expression 8 is applied to Fig. 14, the input to the softmax regularization function (1412) is and is the dot product. This dot product is the feature vector. To normalize to the length of is divided into. The location shown in the lower part of Fig. 14 Output features for ( ) is the output of the softmax function and is the sum of the k nearest neighbors of the inner product.

In some embodiments, instead of using a kNN search to find the k points in the point cloud closest to point A , all points in the point cloud within a distance r from point A can be used. This operation is called a ball query. The value (or radius) r for the ball query can be determined by the quantization step size s of the quantizer. For example, given a larger s corresponding to a coarser point cloud, the value of r will be larger to cover more points from the original point cloud.

In some embodiments, a kNN search may be used to find the k closest points to a query point, e.g., A. However, only points within a distance r from A are then retained. The value of r may be determined in the same manner as for the ball query. The distance metric used by the kNN search may be any distance metric.

In some embodiments, a transformer block (e.g., the example illustrated in FIG. 13) updates features for (all) occupied positions in the sparse tensor in the same manner and outputs the updated sparse tensor. In some embodiments, , , , and can contain only one fully-connected layer corresponding to a linear projection.

FIG. 15 is a flowchart illustrating an exemplary process for cascading multiple feature aggregations according to some embodiments. In some embodiments, as illustrated in the exemplary process (1500) of FIG. 15, multiple feature aggregation blocks (1502, 1504, 1506, 1508) are cascaded together (such as the feature aggregation block examples illustrated in FIGS. 10-13) to further improve performance. The feature aggregation blocks can be of the same type, for example, they can all be transformer blocks.

In some embodiments where the feature aggregation blocks are of the same type, the parameters of their neural network layers are shared. By sharing the neural network parameters between the feature aggregation blocks, the total number of parameters of the neural network model can be reduced, which has two advantages (for example): First, the model size of the neural network can be reduced, which makes it easier to store or transmit the neural network model. Second, reducing the total number of parameters to be learned during the training stage can make the training converge faster. However, the result of sharing the neural network parameters between the feature aggregation blocks can be a reduction in the capacity of the neural network model, which can reduce the ability of the neural network to extract high-level features and can degrade the performance of the neural network. This potential consequence can prove detrimental to some applications.

In some embodiments, each aggregation block may use the same neural network with a distinct set of neural network parameters. In some embodiments, each aggregation block may use separate neural networks with distinct sets of neural network parameters. In some embodiments, the separate sets of neural network parameters may be the same. In some embodiments, the first neural network parameter set and the second neural network parameter set are the same set of (identical) neural network parameters, and the same set of neural network parameters may be used by at least the first neural network and the second neural network. In some embodiments, the first neural network parameter set and the second neural network parameter set are distinct but identical sets of neural network parameters.

Of course, in some embodiments, not all neural network parameter sets are identical across all neural networks, and not all neural networks are modeled identically across all functional blocks (e.g., feature aggregation blocks). In some embodiments using more than one feature aggregation block, two or more feature aggregation blocks comprising two or more respective neural networks may utilize two or more respective sets of neural network parameters.

In some embodiments, the feature aggregation blocks may be a mixture of different types of feature aggregation blocks, for example , a mixture of IRN blocks and Transformer blocks. In some embodiments, a single feature aggregation block may be replaced by two or more cascaded feature aggregation blocks to achieve better compression performance.

Context-aware voxel-based upsampling can be applied to point cloud decompression. In some embodiments, context-aware voxel-based upsampling can be applied to the decoder of the '087 application to generate point clouds that are closer to ground truth.

FIG. 16 is a block diagram illustrating an exemplary original decoder architecture according to some embodiments. The architecture (1600) of the decoder of the '087 application, including a base layer and an enhancement layer, is depicted in FIG. 16. The base layer receives a bitstream BS ₀ , which is used to perform a base decode (1602) and inverse quantization (1604) to generate a coarse/simplified point cloud PC ₀ . PC ₀ is a simplified or low-resolution version of the original input point cloud in a voxel-based representation. From the bitstream BS ₁ , a feature decoder block (1606) decodes voxel-wise features of PC ₀ , which equip every occupied voxel in PC ₀ with a vector feature, which is an abstraction of the local geometric structure. If BS ₁ is not available - referred to as "skip mode" in the '087 application - the feature decoder still synthesizes, for each voxel in PC ₀ , vector features based on the geometry of PC ₀ . The resulting features, which are appended to the point cloud, are denoted as PC' ₀ . In the '087 application, all features in PC' ₀ are input to the feature-to-residual transformer (1608) to decode the local 3D point set. Specifically, For voxel A at PC' ₀ located at , its features is input to a feature-residual transformer, which transforms a set of k 3D points outputs. The feature-residual transformer (1608) may be a series of MLP layers. Geometric summation (see “ ”) translates the decoded point set by translating it using (x, y, z) as shown in Equations 9 to 11:

A set of translationally translated points associated with all voxels in feature set PC' ₀ forms a decoded point cloud PC _DEC . If there are M voxels in PC' ₀ , the decoded point cloud PC _DEC contains Mk points.

In the base layer of Fig. 16, the point cloud is decoded from the bitstream BS ₀ using a basic decoder. A dequantizer is applied to the point cloud to obtain a coarser point cloud PC ₀ . In some embodiments, the dequantizer can use a step size of s.

In the enhancement layer, a feature decoder is applied to decode BS ₁ together with the already decoded coarser point cloud PC ₀ to output a pointwise feature set PC' ₀ . The feature set PC' ₀ contains pointwise features for each point in PC ₀ . For example, point A in PC ₀ has its own feature vector f' _A . The decoded feature vector f' _A may have different magnitude from its corresponding feature vector f _A at the encoder side. However, both f _A and f' _A are generated to describe the fine local geometric details of PC ₀ close to point A. The decoded feature set PC' ₀ is input to a feature-residual transformer, which generates a residual component of PC _DEC . The coarser point cloud PC ₀ and the residual are input to a geometric summation block. The summation block adds the residual components to the coarser point cloud PC ₀ to produce the final decoded point cloud PC _DEC .

In some embodiments, the base decoder can be any PCC codec. In some embodiments, the base decoder is selected to use a lossy PCC codec, such as Wang 's codec. In some embodiments, the base codec can be a lossless PCC codec, such as the MPEG G-PCC standard, or a deep entropy model using an octree representation.

The number of decoded points m can be a fixed constant, such as m = 5, or the number of decoded points can be chosen adaptively, for example, based on prior knowledge about the density level of the original point cloud. For example, if the original point cloud is very sparse, m can be set to a small number, such as m = 2.

The feature-residual transformer transforms the decoded feature set PC' ₀ back into the residual components of PC _DEC . In particular, in some embodiments, the feature-residual transformer applies a deep neural network to transform every feature vector f' _A in PC' ₀ (associated with a point A in PC ₀ ) back into the corresponding residual point set S' _A.

In some embodiments, the feature-to-residual transformer can be a sequence of MLP layers. In this case, the feature vector at PC' ₀ , e.g., f' _A , is input to a sequence of MLP layers. The MLP layers directly output a set of m 3D points C ₀ , C ₁ , ..., C _{m -1} , which provides a decoded residual set S' _A . Thus, for PC ₀ with n points A ₀ , A ₁ , ..., A _{n -1} , the feature-to-residual transformer produces its own decoded residual sets, denoted S' ₀ , S' ₁ , ... , S' _{n -1} . These residual sets together constitute a decoded residual component.

In some embodiments, corresponding 3D points in the residual component that are too far from the origin can be removed. Specifically, for a point C _i in the residual component, if its distance from the origin is greater than a threshold t , the point C is considered as an outlier and is removed from the residual component. The threshold t can be a predefined constant. The threshold can also be chosen depending on the quantization step size s of the quantizer on the encoder. For example, a larger s means a coarser PC ₀ , and the threshold can be set to a larger value to retain more nodes in the residual component.

The value of k can be chosen based on the density level of the input point cloud. For dense point clouds, the value of k can be larger, for example, k = 10. For sparse point clouds, such as LiDAR sweeps, the value of k can be very small, such as k = 1, which can mean that every point in PC ₀ in Fig. 16 can be associated with only one point in the original point cloud.

FIG. 17 is a block diagram illustrating an exemplary decoder architecture with voxel-based upsampling according to some embodiments. In some embodiments, as depicted here in FIGS. 16 and 17 , the base layer receives a bitstream BS ₀ , which is used to perform base decode (1702) and dequantization (1704). In some embodiments, a context-aware upsampling block (1708) is inserted between the feature decoder block (1706) and the feature-to-residual transformer (1710), as depicted in FIG. 17 . Furthermore, the geometric summation module now takes PC ₁ as input instead of PC ₀ as depicted in FIG. 16 . In some embodiments, instead of encoding/decoding the original PC ₀ in FIG. 16 , the encoded/decoded PC ₀ in FIG. 17 is now two-times smaller and thus has fewer bits. In some embodiments, the context-aware upsampling block (1708) of FIG. 17 may be performed by context-aware voxel-based upsampling using the exemplary pruning process illustrated in FIG. 5.

FIG. 18 is a block diagram illustrating an exemplary decoder architecture with voxel-based upsampling and feature aggregation according to some embodiments. In some embodiments of the exemplary process (1800), a feature aggregation block (1810) is inserted between the context-aware upsampling block (1808) and the feature-to-residual transformer (1812), as illustrated in FIG. 18. In this case, the context-aware upsampling block (1808) can also be cascaded multiple times, for example, in the manner presented in FIG. 7, to achieve higher upsampling ratio and additional bit savings of _PC 0 . In some embodiments, the base layer receives a bitstream BS ₀ , which is used to perform base decode (1802), dequantization (1804), and feature decode (1806).

FIG. 19 is a block diagram illustrating an exemplary decoder architecture without a feature-to-residual transformer according to some embodiments. In some embodiments, compared to FIG. 18, only the context-aware upsampling blocks (1908, 1912) are presented, and the feature-to-residual transformer is removed, as illustrated in FIG. 19. In this scenario, PC ₀ is progressively upsampled and refined to obtain a decoded point cloud PC _DEC . In the example of FIG. 18, two context-aware upsampling blocks (1908, 1912) are illustrated on either side of the feature aggregation block (1910). However, in some embodiments, a different number of context-aware upsampling blocks may be used to obtain the PC _DEC . In some embodiments, the base layer receives a bitstream BS ₀ , which is used to perform base decode (1902), dequantization (1904), and feature decode (1906).

In the '015 application, a hybrid coding framework for PCC is used to implement octree-based PCC, voxel-based PCC, and point-based PCC. In some embodiments, more than one of these types of PCCs may be used in the methods illustrated in FIGS. 17, 18 and 19. The '015 application proposes combining two of these types of PCCs. In particular, in one case, only (i) the octree-based PCC method and (ii) the voxel-based PCC method are used. This configuration corresponds to FIG. 19.

The processes described in the present application can be applied to point cloud super-resolution. In some embodiments, a context-aware upsampling process can be applied to a feature set associated with an input point cloud PC ₀ and each of its occupied voxels to achieve a 2x super-resolution. In some embodiments, multiple context-aware upsampling processes can be applied to a feature set associated with an input point cloud PC ₀ and each of its occupied voxels to achieve a greater than 2x super-resolution, as illustrated in FIG. 7.

In some embodiments, the features associated with voxels can be attributes such as color and intensity. In some embodiments, the features associated with voxels can be local geometric features of PC ₀ extracted by neural network layers, such as the feature aggregation processes illustrated in FIGS. 10-12. In some embodiments, the features associated with voxels can be a concatenation of both attributes and geometric features.

FIG. 20 is a block diagram illustrating an exemplary decoder architecture having a single pass through voxel-based upsampling and feature aggregation according to some embodiments. In some embodiments of the exemplary feature decoder process (2000), where context-aware voxel-based upsampling (2002) is applied only once, as illustrated in FIG. 20 , feature aggregation (2004) may be appended after the context-aware voxel-based upsampling for feature aggregation and segmentation.

When applying feature aggregation following context-aware voxel-based upsampling (as illustrated in FIG. 20), all other feature aggregations within the feature aggregation and context-aware upsampling blocks have the same neural network architecture and share the same neural network parameters. In some embodiments, by having all feature aggregation blocks share the same set of weights, the total number of neural network parameters can be reduced. By sharing neural network parameters between feature aggregation blocks, the total number of parameters of the neural network model can be reduced, which has two advantages (for example): First, the model size of the neural network can be reduced, which makes it easier to store or transmit the neural network model. Second, reducing the total number of parameters to be learned during the training stage can make the training converge faster. However, the result of sharing neural network parameters between feature aggregation blocks can be a reduction in the capacity of the neural network model, which can reduce the ability of the neural network to extract high-level features and degrade the performance of the neural network. These potential consequences may prove detrimental to some applications.

Of course, in some embodiments, not all neural network parameter sets are identical across all neural networks, and not all neural networks are modeled identically across all functional blocks (e.g., feature aggregation blocks). In some embodiments using more than one feature aggregation block, two or more feature aggregation blocks comprising two or more respective neural networks may utilize two or more respective sets of neural network parameters.

In some embodiments, when context-aware voxel-based upsampling with pruning is used as illustrated in FIG. 5, the following feature aggregation blocks can share the same set of neural network parameters: (i) the feature aggregation block within the binary classification block (illustrated in FIG. 9), and (ii) the feature aggregation block after the context-aware voxel-based upsampling block (illustrated in FIG. 20).

Similarly, in some embodiments, when context-aware voxel-based upsampling with initial feature aggregation is used as illustrated in FIG. 8 , the following feature aggregation blocks can share the same set of neural network parameters: (i) the feature aggregation block after the nearest neighbor (NN) upsampling block (illustrated in FIG. 8 ), (ii) the feature aggregation block within the binary classification block (illustrated in FIG. 9 ), and (iii) the feature aggregation block after the context-aware voxel-based upsampling block (illustrated in FIG. 20 ).

FIG. 21 is a block diagram illustrating exemplary sparse tensor operations according to some embodiments. FIG. 21 is used to illustrate an exemplary process (2100) for downsampling (2104), upsampling (2108), coordinate reading/splitting (2116), and coordinate pruning (2112) processes. For simplicity, the operations in FIG. 21 are described in 2D space, but the same rationale can be applied to 3D space. In this example, the input point cloud A0 (2102) occupies voxels at locations (0, 2), (0, 3), (0, 4), (0, 5), (1, 1), (1, 6), (2, 6), (3, 5), (4, 4), (5, 4), (6, 4), and (7, 4) (2118), where the origin is zero-based and in the upper left corner. Therefore, the coordinate reader/segmenter (2116) outputs the occupied coordinates as (0, 2), (0, 3), (0, 4), (0, 5), (1, 1), (1, 6), (2, 6), (3, 5), (4, 4), (5, 4), (6, 4), and (7, 4) (2118). By downsampling A0 (2102) by a ratio of 2, the number of voxels in each dimension in the downsampled point cloud A1 (2106) is reduced by half, and a voxel is considered occupied if any of its four corresponding points in A0 (2102) are occupied. After upsampling A1(2106), the number of voxels in A2(2110) is resumed, and a voxel in A2(2110) is considered occupied if its corresponding voxel in A1(2106) is occupied. A2(2110) is more dense than the input point cloud A0(2102). To remove/prune the unoccupied points(voxels) in A0(2102) from A2(2110), the occupied coordinate information is used by the coordinate pruning block. In the resulting point cloud A3(2114), only the voxels that are occupied in the original point cloud A0 are treated as occupied.

FIG. 22 is a block diagram illustrating an exemplary decoder architecture according to some embodiments. An exemplary feature decoder (2200) based on sparse 3D convolution, downsampling and upsampling in some embodiments is illustrated in FIG. 22. A bitstream BS ₁ (2202) is entropy decoded (2204) and feature dequantized (2206) to generate a downsampled feature set F' _down , followed by sequential upsampling using the geometry of PC ₀ to progressively enlarge and refine the features. In FIG. 22 , the bitstream BS ₁ is decoded by an entropy decoder, followed by a dequantizer to generate a downsampled feature set F' _down .

As illustrated in the upper right corner of Fig. 22, a 3D sparse tensor is constructed (2244) based (solely) on the geometry (coordinates) of PC ₀ . The tensor is sequentially downsampled (2240, 2236) to generate a tensor PC' _down . PC' _down and PC _down for feature encoder (not illustrated) in Fig. 22 may have the same geometry, but in some embodiments their features may be different. To upsample F' _down , F' _down is transformed into the geometry of PC' _down . To transform the geometry of F' _down into that of PC' _down , a feature replacement block (2208) replaces original features of PC' _down with F' _down to generate another sparse tensor PC" _down .

PC" _down is upsampled by two upsampling processing blocks, each block including one upsampling operator (2210, 2222) and two sparse 3D convolutional layers. In FIG. 22, "upsampling 2" is a sparse tensor upsampling operator (2210, 2222) with a ratio of 2. In some embodiments, the upsampling 2 block may include the sparse tensor upsampling operator (2210, 2222) and two sets of convolutional decoders (2212, 2216, 2224, 2228) and rectified linear units (ReLUs) (2214, 2218, 2226, 2230). The upsampling 2 block may include sparse tensor upsampling operators (2210, 2222) along each dimension, similar to the upsampling operator of a "regular" 2D image. Double the size of the tensor. For an illustrative example, see Fig. 21. After each upsampling processing block, the resulting tensor is refined using respective coordinate readers (2238, 2242) and coordinate pruning blocks (2220, 2232). Fig. 21 shows an illustrative example of coordinate pruning, which removes some of the occupied voxels of the input tensor and keeps the rest based on the input coordinate set that can be obtained from the coordinate readers. The coordinate pruning block (2220) removes some voxels (and associated features) from the upsampled version of PC" _down and keeps only those voxels that also appear in the downsampled version of PC ₀ . The output of the second coordinate pruning block (2232) is a tensor having the same geometry as PC ₀ . This tensor is input to a feature reader (2234) to obtain a decoded feature set PC' ₀ .

Fig. 23 is a block diagram illustrating an exemplary decoder architecture according to some embodiments. Compared to Fig. 22, the coordinate pruning blocks (2318, 2332) (and in some embodiments the feature aggregation blocks (2320, 2334)) are absorbed into the preceding upsampling processing block of Fig. 23 in some embodiments.

In some embodiments of the exemplary decoder (2300), the tensor is sequentially downsampled (2342, 2338) to generate a tensor PC' _down . In some embodiments, the bitstream is entropy decoded (2302) and feature dequantized (2304) to generate a downsampled feature set F' _down . To transform the geometry of F' _down into the geometry of PC' _down , the feature replacement block (2306) replaces the original features of PC' _down with F' _down to generate another sparse tensor PC" _down . In some embodiments, the upsampling 2 block may include a sparse tensor upsampling operator (2308, 2322) and two sets of convolutional decoders (2310, 2314, 2324, 2328) and rectified linear units (ReLUs) (2312, 2316, 2326, 2330). In some embodiments, after each upsampling processing block, the resulting tensor is refined using respective coordinate readers (2340, 2344) and coordinate pruning blocks (2318, 2332). The output of the second feature aggregation block (2334) is the same as PC ₀ . A tensor having a geometric structure. This tensor is input to a feature reader (2336) to obtain a decoded feature set PC' ₀ .

FIG. 24 is a flowchart illustrating an exemplary process of context-aware voxel-based upsampling using pruning according to some embodiments. In some embodiments, the exemplary process (2400) may include a step (2402) of upsampling a first point cloud using an initial upsampling to obtain a second point cloud. In some embodiments, the exemplary process (2400) may further include a step (2404) of associating features of the second point cloud with voxel-wise context information to obtain a third point cloud. In some embodiments, the exemplary process (2400) may further include a step (2406) of predicting an occupancy state of at least one voxel of the third point cloud. In some embodiments, the exemplary process (2400) may further include a step (2408) of removing voxels of the third point cloud classified as empty based on the predicted occupancy state to generate a pruned point cloud. In some embodiments, the initial upsampling may include nearest neighbor upsampling. In some embodiments, associating features may include concatenating features.

FIG. 25 is a flowchart illustrating an exemplary process of context-aware voxel-based upsampling and feature aggregation according to some embodiments. In some embodiments, the exemplary process (2500) may include a step (2502) of upsampling a first point cloud using an initial upsampling to obtain a second point cloud. In some embodiments, the exemplary process (2500) may further include a step (2504) of associating features of the second point cloud with context information to obtain a third point cloud. In some embodiments, the exemplary process (2500) may further include a step (2506) of predicting an occupancy state of at least one voxel of the third point cloud, wherein the step of predicting the occupancy state of the at least one voxel comprises aggregating at least one feature of the third point cloud, wherein the step of aggregating the at least one feature of the third point cloud comprises using a first neural network, and wherein the step of aggregating the at least one feature of the third point cloud using the first neural network comprises using a first neural network parameter set in conjunction with the first neural network. In some embodiments, the exemplary process (2500) may further include a step (2508) of removing voxels of the third point cloud classified as empty, based on the predicted occupancy state, to generate a pruned point cloud. In some embodiments, the exemplary process (2500) may further include a step (2510) of performing feature aggregation on the pruned point cloud to generate aggregated features, wherein the step of performing feature aggregation on the pruned point cloud comprises using a second neural network, and wherein the step of generating the aggregated features using the second neural network comprises using a second neural network parameter set in conjunction with the second neural network, wherein the first neural network parameter set is identical to the second neural network parameter set.

FIG. 26 is a flowchart illustrating an exemplary process for encoding a bitstream according to some embodiments. In some embodiments, the exemplary process (2600) may include a step of obtaining a first point cloud (2602). In some embodiments, the exemplary process (2600) may further include a step of determining an occupancy state of at least one voxel of the first point cloud (2604). In some embodiments, the exemplary process (2600) may further include a step of generating a second point cloud by removing voxels of the first point cloud classified as empty based on the determined occupancy state (2606). In some embodiments, the exemplary process (2600) may further include a step of obtaining a third point cloud by associating features of the second point cloud with context information (2608). In some embodiments, the exemplary process (2600) may further include a step (2610) of downsampling the third point cloud using initial downsampling to obtain a fourth point cloud. In some embodiments, the exemplary process (2600) may further include a step (2612) of outputting the fourth point cloud as an encoded point cloud.

In some embodiments, the device may include one or more processors configured to obtain a second point cloud by upsampling a first point cloud using nearest neighbor upsampling; obtain a third point cloud by concatenating features of the second point cloud with voxel-wise context information; predict an occupancy state of at least one voxel of the third point cloud; and remove voxels of the third point cloud classified as empty based on the predicted occupancy state.

While the methods and systems according to some embodiments are discussed in a virtual reality (VR) context, some embodiments may also be applied in a mixed reality (MR)/augmented reality (AR) context. Additionally, while the term "head-mounted display (HMD)" is used herein according to some embodiments, some embodiments may be applied to, for example, a wearable device (which may or may not be head-mounted) that is capable of VR, AR, and/or MR, for example, for some embodiments.

An exemplary method according to some embodiments may include: upsampling a first point cloud using initial upsampling to obtain a second point cloud; associating features of the second point cloud with context information to obtain a third point cloud; predicting an occupancy state of at least one voxel of the third point cloud; and removing voxels of the third point cloud classified as empty based on the predicted occupancy state to generate a pruned point cloud.

In some embodiments of the exemplary method, the initial upsampling may include nearest neighbor upsampling.

In some embodiments of the exemplary method, the step of associating features may include the step of obtaining a third point cloud by associating features of the second point cloud with context information.

In some embodiments of the exemplary method, the step of predicting the occupancy state may be performed using a first neural network.

In some embodiments of the exemplary method, the step of predicting an occupancy state may predict a ground truth occupancy state of at least one voxel.

In some embodiments of the exemplary method, the step of predicting occupancy may predict a likelihood that at least one voxel is occupied.

In some embodiments of the exemplary method, the step of removing voxels of the third point cloud may remove voxels using a voxel pruning process.

Some embodiments of the exemplary method may further include a step of aggregating at least one feature of the second point cloud.

In some embodiments of the exemplary method, the context information may be voxel-wise context information.

In some embodiments of the exemplary method, the step of predicting the occupancy state of at least one voxel may include: aggregating at least one feature of the third point cloud; processing the aggregated feature with multilayer perceptron (MLP) layers to generate an MLP layer output; performing a softmax process on the MLP layer output to generate softmax output values; and performing thresholding on the softmax output values to generate a predicted occupancy state of the at least one voxel of the third point cloud.

In some embodiments of the exemplary method, thresholding of the softmax output values converts softmax output values greater than 0.5 to an output value of 1, and converts softmax output values less than or equal to 0.5 to an output value of 0.

In some embodiments of the exemplary method, the step of predicting the occupancy state of at least one voxel may include: aggregating at least one feature of the third point cloud; and generating a predicted occupancy state of the at least one voxel of the third point cloud based on the aggregated feature.

In some embodiments of the exemplary method, the step of aggregating at least one feature may comprise: repeating a cascading process one or more times, wherein the cascading process may comprise: performing a sparse 3D convolution of an input point cloud to generate a convolution output point cloud; performing a nonlinear activation process on the convolution output point cloud to generate a nonlinear output point cloud; and preparing the nonlinear output point cloud as an input point cloud for a next cycle of the cascading process, wherein the third point cloud is an input point cloud for a first cycle of the cascading process, and wherein a last cycle of the cascading process generates aggregated features.

Some embodiments of the exemplary method may further include a step of adding a third point cloud to the ReLU output point cloud of the last cycle of the cascading process.

In some embodiments of the exemplary method, the step of aggregating at least one feature may include: performing a sparse 3D convolution of an input point cloud to generate a convolution output point cloud; and performing a nonlinear activation process on the convolution output point cloud to generate aggregated features.

In some embodiments of the exemplary method, the nonlinear activation process may be a rectifier linear unit (ReLU) activation process.

In some embodiments of the exemplary method, the step of aggregating at least one feature may include: repeating a first cascading process one or more times, wherein the first cascading process comprises: performing a first sparse 3D convolution of a first input point cloud to generate a first convolution output point cloud; performing a first nonlinear activation process on the first convolution output point cloud to generate a first nonlinear output point cloud; and preparing the first nonlinear output point cloud as the first input point cloud for a next cycle of the first cascading process, wherein the third point cloud is the first input point cloud for the first cycle of the first cascading process, and wherein a last cycle of the first cascading process generates the first cascading process output; repeating a second cascading process one or more times, wherein the second cascading process comprises: performing a second sparse 3D convolution of the second input point cloud to generate a second convolution output point cloud; The method may further include: performing a second nonlinear activation process on the second convolution output point cloud to generate a second nonlinear output point cloud; and preparing the second nonlinear output point cloud as a second input point cloud when there is a next cycle of the second cascading process, wherein the third point cloud is a second input point cloud for a first cycle of the second cascading process, and the last cycle of the second cascading process generates a second cascading process output; connecting the first cascading process output and the second cascading process output to generate a connected output; and adding the third point cloud to the connected output to generate aggregated features.

In some embodiments of the exemplary method, the step of aggregating at least one feature may include: repeating a first cascading process one or more times, wherein the first cascading process comprises: performing a first sparse 3D convolution of a first input point cloud to generate a first convolution output point cloud; performing a first rectifier linear unit (ReLU) activation process on the first convolution output point cloud to generate a first ReLU output point cloud; and preparing the first ReLU output point cloud as the first input point cloud for a next cycle of the first cascading process, wherein the third point cloud is the first input point cloud for the first cycle of the first cascading process, and wherein a last cycle of the first cascading process generates the first cascading process output; The second cascading process may include: performing a second sparse 3D convolution of a second input point cloud to generate a second convolution output point cloud; performing a second ReLU (rectifier linear unit) activation process on the second convolution output point cloud to generate a second ReLU output point cloud; and preparing the second ReLU output point cloud as a second input point cloud when there is a next cycle of the second cascading process, wherein the third point cloud is the second input point cloud for a first cycle of the second cascading process, and the last cycle of the second cascading process generates the second cascading process output; concatenating the first cascading process output and the second cascading process output to generate a concatenated output; and adding the third point cloud to the concatenated output to generate aggregated features.

In some embodiments of the exemplary method, the step of aggregating at least one feature may include: performing a self-attention process on the third point cloud; adding the third point cloud to the self-attention process output to generate an MLP process input; performing an MLP process on the MLP process input; and adding the MLP process input to the MLP process output to generate an aggregated feature.

In some embodiments of the exemplary method, the self-attention process generates output features based on the k nearest neighbors of a voxel in the third point cloud.

In some embodiments of the exemplary method, the step of aggregating at least one feature of the third point cloud may include performing the feature aggregation process more than once.

Some embodiments of the exemplary method may further include a step of performing feature decoding on the input point cloud and the first bitstream to generate a first point cloud.

Some embodiments of the exemplary method may further include performing a feature-residual transform on the pruned point cloud to generate a residual output; and adding the pruned point cloud to the residual output to generate a decoded point cloud.

Some embodiments of the exemplary method may further include a step of performing feature aggregation on the pruned point cloud to generate aggregated features, wherein feature-to-residual transformation is performed on the aggregated features.

Some embodiments of the exemplary method may further include the steps of performing feature aggregation on the pruned point cloud to generate aggregated features; and performing a context-aware upsampling process on the aggregated features to generate a decoded point cloud.

An exemplary device according to some embodiments may include a processor; and a non-transitory computer-readable medium storing instructions that, when executed by the processor, cause the device to: upsample a first point cloud using nearest neighbor upsampling to obtain a second point cloud; associate features of the second point cloud with context information to obtain a third point cloud; predict an occupancy state of at least one voxel of the third point cloud; and remove voxels of the third point cloud classified as empty based on the predicted occupancy state to generate a pruned point cloud.

An exemplary device according to some embodiments may include a device according to the exemplary device; and at least one of (i) an antenna configured to receive a signal, the signal comprising data representing an image, (ii) a band limiter configured to limit the received signal to a frequency band comprising the data representing the image, or (iii) a display configured to display the image.

Some embodiments of the exemplary method may further include at least one of a TV, a cell phone, a tablet, and a set-top box (STB).

An exemplary device according to some embodiments may include an access unit configured to access data including a first point cloud; and a transmitter configured to transmit data including the first point cloud.

An exemplary method according to some embodiments may include: accessing data comprising a first point cloud; and transmitting data comprising the first point cloud.

An exemplary computer-readable medium according to some embodiments may include instructions that cause one or more processors to: obtain a second point cloud by upsampling a first point cloud using nearest neighbor upsampling; obtain a third point cloud by concatenating features of the second point cloud with context information; predict an occupancy state of at least one voxel of the third point cloud; and remove voxels of the third point cloud classified as empty based on the predicted occupancy state to generate a pruned point cloud.

An exemplary computer program product according to some embodiments may include instructions that, when the program is executed by one or more processors, cause the one or more processors to: obtain a second point cloud by upsampling a first point cloud using nearest neighbor upsampling; obtain a third point cloud by concatenating features of the second point cloud with context information; predict an occupancy state of at least one voxel of the third point cloud; and remove voxels of the third point cloud classified as empty based on the predicted occupancy state to generate a pruned point cloud.

An exemplary method according to some embodiments may include performing context-aware upsampling of a first point cloud to determine an upsampled second point cloud, wherein the context-aware upsampling may include: associating features of a third point cloud with context information, the third point cloud being at least partially based on an initial upsampled version of the first point cloud; and removing voxels of a fourth point cloud that are predicted to be empty based at least partially on the context information from the third point cloud to generate the upscaled second point cloud.

An additional exemplary method according to some embodiments comprises: obtaining a second point cloud by upsampling a first point cloud using an initial upsampling; obtaining a third point cloud by associating features of the second point cloud with context information; predicting an occupancy state of at least one voxel of the third point cloud, wherein predicting the occupancy state of the at least one voxel comprises aggregating at least one feature of the third point cloud, wherein aggregating the at least one feature of the third point cloud comprises using a first neural network, and wherein aggregating the at least one feature of the third point cloud using the first neural network comprises using a first neural network parameter set together with the first neural network; generating a pruned point cloud by removing voxels of the third point cloud classified as empty according to the predicted occupancy state; and performing feature aggregation on the pruned point cloud to generate aggregated features, wherein the performing feature aggregation on the pruned point cloud comprises using a second neural network, and wherein the step of generating the aggregated features using the second neural network comprises using a second neural network parameter set together with the second neural network, wherein the first neural network parameter set is identical to the second neural network parameter set.

Some embodiments of the additional exemplary method may further include a step of aggregating at least one feature of the second point cloud.

In some embodiments of the additional exemplary method, the step of aggregating at least one feature of the second point cloud may comprise using a third neural network, and the step of aggregating at least one feature of the second point cloud using the third neural network may comprise using a third neural network parameter set together with the third neural network, wherein the third neural network parameter set may be identical to the first neural network parameter set.

Some embodiments of the additional exemplary method may further include a step of performing feature decoding on the input point cloud and the first bitstream to generate a first point cloud.

Some embodiments of the additional exemplary method may further include: performing a feature-residual transformation on the pruned point cloud to generate a residual output; and adding the pruned point cloud to the residual output to generate a decoded point cloud.

In some embodiments of the additional exemplary method, feature-residual transformation can be performed on aggregated features.

Some embodiments of the additional exemplary method may further include a step of performing a context-aware upsampling process on the aggregated features to generate a decoded point cloud.

In some embodiments of the additional exemplary method, the initial upsampling may include nearest neighbor upsampling.

In some embodiments of the additional exemplary method, the step of associating features may include the step of obtaining a third point cloud by associating features of the second point cloud with context information.

In some embodiments of the additional exemplary method, the step of predicting an occupancy state may predict a ground truth occupancy state of at least one voxel.

In some embodiments of the additional exemplary method, the step of predicting occupancy may predict a likelihood that at least one voxel is occupied.

In some embodiments of the additional exemplary method, the step of removing voxels of the third point cloud may remove voxels using a voxel pruning process.

In some embodiments of the additional exemplary method, the context information may be voxel-wise context information.

In some embodiments of the additional exemplary method, the step of predicting the occupancy state of at least one voxel may include: processing the aggregated features with multilayer perceptron (MLP) layers to generate MLP layer outputs; performing a softmax process on the MLP layer outputs to generate softmax output values; and performing thresholding on the softmax output values to generate a predicted occupancy state of at least one voxel of the third point cloud.

In some embodiments of the additional exemplary method, thresholding of the softmax output values can convert softmax output values greater than 0.5 to an output value of 1, and convert softmax output values less than or equal to 0.5 to an output value of 0.

In some embodiments of the additional exemplary method, the step of predicting the occupancy state of at least one voxel may include: aggregating at least one feature of the third point cloud; and generating a predicted occupancy state of the at least one voxel of the third point cloud based on the aggregated feature.

In some embodiments of the additional exemplary method, the step of aggregating at least one feature of the third point cloud may comprise: repeating a cascading process one or more times, wherein the cascading process may comprise: performing a sparse 3D convolution of an input point cloud to generate a convolution output point cloud; performing a nonlinear activation process on the convolution output point cloud to generate a nonlinear output point cloud; and preparing the nonlinear output point cloud as an input point cloud for a next cycle of the cascading process, wherein the third point cloud may be an input point cloud for a first cycle of the cascading process, and wherein a last cycle of the cascading process may generate the aggregated features.

Some embodiments of the additional exemplary method may further include a step of adding a third point cloud to the ReLU output point cloud of the last cycle of the cascading process.

In some embodiments of the additional exemplary method, the step of aggregating at least one feature may include: performing a sparse 3D convolution of an input point cloud to generate a convolution output point cloud; and performing a nonlinear activation process on the convolution output point cloud to generate aggregated features.

In some embodiments of the additional exemplary method, the nonlinear activation process may include a rectifier linear unit (ReLU) activation process, and the nonlinear output point cloud includes a ReLU output point cloud.

In some embodiments of the additional exemplary method, the step of aggregating at least one feature of the third point cloud may include: repeating a first cascading process one or more times, wherein the first cascading process comprises: performing a first sparse 3D convolution of a first input point cloud to generate a first convolution output point cloud; performing a first nonlinear activation process on the first convolution output point cloud to generate a first nonlinear output point cloud; and preparing the first nonlinear output point cloud as the first input point cloud if there is a next cycle of the first cascading process, wherein the third point cloud may be the first input point cloud for a first cycle of the first cascading process, and wherein a last cycle of the first cascading process may generate the first cascading process output; A step of repeating a second cascading process one or more times, wherein the second cascading process may include: performing a second sparse 3D convolution of a second input point cloud to generate a second convolution output point cloud; performing a second nonlinear activation process on the second convolution output point cloud to generate a second nonlinear output point cloud; and preparing the second nonlinear output point cloud as the second input point cloud if there is a next cycle of the second cascading process, wherein the third point cloud may be the second input point cloud for a first cycle of the second cascading process, and a last cycle of the second cascading process may generate the second cascading process output; concatenating the first cascading process output and the second cascading process output to generate a concatenated output; and adding the third point cloud to the concatenated output to generate aggregated features.

In some embodiments of the additional exemplary method, the step of aggregating at least one feature may include: repeating a first cascading process one or more times, wherein the first cascading process comprises: performing a first sparse 3D convolution of a first input point cloud to generate a first convolution output point cloud; performing a first rectifier linear unit (ReLU) activation process on the first convolution output point cloud to generate a first ReLU output point cloud; and preparing the first ReLU output point cloud as the first input point cloud if there is a next cycle of the first cascading process, wherein the third point cloud may be the first input point cloud for the first cycle of the first cascading process, and wherein a last cycle of the first cascading process may generate the first cascading process output; A step of repeating a second cascading process one or more times, wherein the second cascading process may include: performing a second sparse 3D convolution of a second input point cloud to generate a second convolution output point cloud; performing a second ReLU (rectifier linear unit) activation process on the second convolution output point cloud to generate a second ReLU output point cloud; and preparing the second ReLU output point cloud as a second input point cloud when there is a next cycle of the second cascading process, wherein the third point cloud may be the second input point cloud for a first cycle of the second cascading process, and a last cycle of the second cascading process may generate the second cascading process output; connecting the first cascading process output and the second cascading process output to generate a connected output; and adding the third point cloud to the connected output to generate an aggregated feature.

In some embodiments of the additional exemplary method, the step of aggregating at least one feature may include: performing a self-attention process on the third point cloud; adding the third point cloud to the self-attention process output to generate an MLP process input; performing an MLP process on the MLP process input; and adding the MLP process input to the MLP process output to generate an aggregated feature.

In some embodiments of the additional exemplary method, the self-attention process can generate output features based on the k nearest neighbors of a voxel of the third point cloud.

In some embodiments of the additional exemplary method, the step of aggregating at least one feature of the third point cloud may include performing the feature aggregation process more than once.

In some embodiments of the additional exemplary method, the first neural network parameter set and the second neural network parameter set can be the same neural network parameter set, and the same neural network parameter set is used by at least the first neural network and the second neural network.

In some embodiments of the additional exemplary method, the first neural network parameter set and the second neural network parameter set may be distinct but identical neural network parameter sets.

An additional exemplary device according to some embodiments may include: a processor; and a non-transitory computer-readable medium storing instructions that, when executed by the processor, cause the device to perform any one of the methods listed above.

A first exemplary method/device according to some embodiments may include: obtaining a second point cloud by upsampling a first point cloud using initial upsampling; obtaining a third point cloud by associating features of the second point cloud with context information; predicting an occupancy state of at least one voxel of the third point cloud; and generating a pruned point cloud by removing voxels of the third point cloud classified as empty based on the predicted occupancy state.

In some embodiments of the first exemplary method, the initial upsampling includes nearest neighbor upsampling.

In some embodiments of the first exemplary method, the step of associating features includes the step of obtaining a third point cloud by associating features of the second point cloud with context information.

In some embodiments of the first exemplary method, the context information is voxel-wise context information.

In some embodiments of the first exemplary method, the context information includes a context point cloud.

In some embodiments of the first exemplary method, the context information includes information about the second point cloud.

In some embodiments of the first exemplary method, the context information includes information about voxel occupancy status of the second point cloud.

In some embodiments of the first exemplary method, the context information includes information about the location of a child voxel relative to the location of a parent voxel in the first point cloud.

In some embodiments of the first exemplary method, the context information includes coordinate information regarding the location of an occupied voxel of at least one of the first and second point clouds.

In some embodiments of the first exemplary method, the context information includes coordinate information, and the coordinate information is in the form of one of Euclidean coordinates, spherical coordinates, and cylindrical coordinates.

In some embodiments of the first exemplary method, the context information provides known information about the first point cloud in addition to information available for initial upsampling of the first point cloud.

In some embodiments of the first exemplary method, the context information includes the bit depth of the second point cloud.

Some embodiments of the first exemplary method may further include a step of performing feature decoding on the input point cloud and the first bitstream to generate a first point cloud.

Some embodiments of the first exemplary method may further include: performing feature aggregation on the pruned point cloud to generate aggregated features; and performing a context-aware upsampling process on the aggregated features to generate a decoded point cloud.

Some embodiments of the first exemplary method may further include: performing a feature-residual transformation on the pruned point cloud to generate a residual output; and adding the pruned point cloud to the residual output to generate a decoded point cloud.

Some embodiments of the first exemplary method may further include a step of performing feature aggregation on the pruned point cloud to generate aggregated features, wherein feature-to-residual transformation is performed on the aggregated features.

In some embodiments of the first exemplary method, the step of predicting the occupancy state is performed using a first neural network.

In some embodiments of the first exemplary method, the step of predicting an occupancy state predicts a ground truth occupancy state of at least one voxel.

In some embodiments of the first exemplary method, the step of predicting occupancy predicts a likelihood that at least one voxel will be occupied.

In some embodiments of the first exemplary method, the step of removing voxels of the third point cloud removes voxels using a voxel pruning process.

Some embodiments of the first exemplary method may further include a step of aggregating at least one feature of the second point cloud.

In some embodiments of the first exemplary method, the step of predicting the occupancy state of at least one voxel comprises: aggregating at least one feature of the third point cloud; processing the aggregated feature with multilayer perceptron (MLP) layers to generate an MLP layer output; performing a softmax process on the MLP layer output to generate softmax output values; and performing thresholding on the softmax output values to generate a predicted occupancy state of the at least one voxel of the third point cloud.

In some embodiments of the first exemplary method, thresholding of the softmax output values converts softmax output values greater than 0.5 to an output value of 1, and converts softmax output values less than or equal to 0.5 to an output value of 0.

In some embodiments of the first exemplary method, the step of predicting an occupancy state of at least one voxel comprises: aggregating at least one feature of a third point cloud; and generating a predicted occupancy state of at least one voxel of the third point cloud based on the aggregated feature.

In some embodiments of the first exemplary method, the step of aggregating at least one feature comprises: repeating a cascading process one or more times, wherein the cascading process comprises: performing a sparse 3D convolution of an input point cloud to generate a convolution output point cloud; performing a nonlinear activation process on the convolution output point cloud to generate a nonlinear output point cloud; and preparing the nonlinear output point cloud as an input point cloud for a next cycle of the cascading process, wherein the third point cloud is an input point cloud for a first cycle of the cascading process, and wherein a last cycle of the cascading process generates aggregated features.

Some embodiments of the first exemplary method may further include a step of adding a third point cloud to the ReLU output point cloud of the last cycle of the cascading process.

In some embodiments of the first exemplary method, the step of aggregating at least one feature comprises: performing a sparse 3D convolution of an input point cloud to generate a convolution output point cloud; and performing a nonlinear activation process on the convolution output point cloud to generate aggregated features.

In some embodiments of the first exemplary method, the nonlinear activation process comprises a rectifier linear unit (ReLU) activation process, and the nonlinear output point cloud comprises a ReLU output point cloud.

In some embodiments of the first exemplary method, the step of aggregating at least one feature comprises: repeating a first cascading process one or more times, wherein the first cascading process comprises: performing a first sparse 3D convolution of a first input point cloud to generate a first convolution output point cloud; performing a first nonlinear activation process on the first convolution output point cloud to generate a first nonlinear output point cloud; and preparing the first nonlinear output point cloud as the first input point cloud for a next cycle of the first cascading process, wherein the third point cloud is the first input point cloud for the first cycle of the first cascading process, and wherein the last cycle of the first cascading process generates the first cascading process output; repeating a second cascading process one or more times, wherein the second cascading process comprises: performing a second sparse 3D convolution of the second input point cloud to generate a second convolution output point cloud; A method comprising: performing a second nonlinear activation process on a second convolution output point cloud to generate a second nonlinear output point cloud; and preparing the second nonlinear output point cloud as a second input point cloud when there is a next cycle of the second cascading process, wherein the third point cloud is a second input point cloud for a first cycle of the second cascading process, and the last cycle of the second cascading process generates a second cascading process output; connecting the first cascading process output and the second cascading process output to generate a connected output; and adding the third point cloud to the connected output to generate aggregated features.

In some embodiments of the first exemplary method, the step of aggregating at least one feature comprises: repeating a first cascading process one or more times, wherein the first cascading process comprises: performing a first sparse 3D convolution of a first input point cloud to generate a first convolution output point cloud; performing a first rectifier linear unit (ReLU) activation process on the first convolution output point cloud to generate a first ReLU output point cloud; and preparing the first ReLU output point cloud as the first input point cloud when there is a next cycle of the first cascading process, wherein the third point cloud is the first input point cloud for the first cycle of the first cascading process, and wherein a last cycle of the first cascading process generates the first cascading process output; A step of repeating a second cascading process one or more times, wherein the second cascading process comprises: performing a second sparse 3D convolution of a second input point cloud to generate a second convolution output point cloud; performing a second ReLU (rectifier linear unit) activation process on the second convolution output point cloud to generate a second ReLU output point cloud; and preparing the second ReLU output point cloud as a second input point cloud when there is a next cycle of the second cascading process, wherein the third point cloud is the second input point cloud for a first cycle of the second cascading process, and the last cycle of the second cascading process generates a second cascading process output; connecting the first cascading process output and the second cascading process output to generate a connected output; and adding the third point cloud to the connected output to generate aggregated features.

In some embodiments of the first exemplary method, the step of aggregating at least one feature comprises: performing a self-attention process on the third point cloud; adding the third point cloud to the self-attention process output to generate an MLP process input; performing an MLP process on the MLP process input; and adding the MLP process input to the MLP process output to generate an aggregated feature.

In some embodiments of the first exemplary method, the self-attention process generates output features based on the k nearest neighbors of a voxel of the third point cloud.

In some embodiments of the first exemplary method, the step of aggregating at least one feature of the third point cloud comprises performing the feature aggregation process two or more times.

A first exemplary method/device according to some embodiments may include: a processor; and a non-transitory computer-readable medium storing instructions that, when executed by the processor, cause the device to: upsample a first point cloud using initial upsampling to obtain a second point cloud; associate features of the second point cloud with context information to obtain a third point cloud; predict an occupancy state of at least one voxel of the third point cloud; and remove voxels of the third point cloud classified as empty based on the predicted occupancy state to generate a pruned point cloud.

In some embodiments of the first exemplary device, the initial upsampling includes nearest neighbor upsampling.

In some embodiments of the first exemplary device, associating the features includes obtaining a third point cloud by connecting features of the second point cloud with context information.

An exemplary device according to some embodiments may include: a device according to any of the devices listed above; and at least one of (i) an antenna configured to receive a signal, the signal comprising data representing an image, (ii) a band limiter configured to limit the received signal to a frequency band comprising the data representing the image, or (iii) a display configured to display the image.

Some embodiments of the exemplary device may further include at least one of a TV, a cell phone, a tablet, and a set-top box (STB).

An exemplary computer-readable medium according to some embodiments may include instructions that cause one or more processors to: obtain a second point cloud by upsampling a first point cloud using initial upsampling; obtain a third point cloud by associating features of the second point cloud with context information; predict an occupancy state of at least one voxel of the third point cloud; and remove voxels of the third point cloud classified as empty based on the predicted occupancy state to generate a pruned point cloud.

An exemplary computer program product according to some embodiments may include instructions that, when the program is executed by one or more processors, cause the one or more processors to: obtain a second point cloud by upsampling a first point cloud using initial upsampling; obtain a third point cloud by associating features of the second point cloud with context information; predict an occupancy state of at least one voxel of the third point cloud; and remove voxels of the third point cloud classified as empty based on the predicted occupancy state to generate a pruned point cloud.

A second exemplary method according to some embodiments may include performing context-aware upsampling of the first point cloud to determine an upsampled second point cloud, wherein the context-aware upsampling comprises: associating features of a third point cloud with context information, the third point cloud being at least partially based on an initial upsampled version of the first point cloud; and removing voxels of a fourth point cloud that are predicted to be empty based at least partially on the context information from the third point cloud to generate the upscaled second point cloud.

A third exemplary method according to some embodiments comprises: obtaining a second point cloud by upsampling a first point cloud using an initial upsampling; obtaining a third point cloud by associating features of the second point cloud with context information; predicting an occupancy state of at least one voxel of the third point cloud, wherein the predicting the occupancy state of the at least one voxel comprises aggregating at least one feature of the third point cloud, wherein the aggregating the at least one feature of the third point cloud comprises using a first neural network, and wherein the aggregating the at least one feature of the third point cloud using the first neural network comprises using a first neural network parameter set together with the first neural network; generating a pruned point cloud by removing voxels of the third point cloud classified as empty according to the predicted occupancy state; and performing feature aggregation on the pruned point cloud to generate aggregated features, wherein the performing feature aggregation on the pruned point cloud comprises using a second neural network, and wherein the step of generating the aggregated features using the second neural network comprises using a second neural network parameter set together with the second neural network, wherein the first neural network parameter set is identical to the second neural network parameter set.

Some embodiments of the third exemplary method may further include a step of aggregating at least one feature of the second point cloud.

In some embodiments of the third exemplary method, wherein the step of aggregating at least one feature of the second point cloud comprises the step of using a third neural network, wherein the step of aggregating at least one feature of the second point cloud using the third neural network comprises the step of using a third neural network parameter set together with the third neural network, wherein the third neural network parameter set is identical to the first neural network parameter set.

In some embodiments of the third exemplary method, the initial upsampling includes nearest neighbor upsampling.

In some embodiments of the third exemplary method, the step of associating features includes the step of obtaining a third point cloud by associating features of the second point cloud with context information.

In some embodiments of the third exemplary method, the step of associating features includes the step of obtaining a third point cloud by associating features of the second point cloud with context information.

In some embodiments of the third exemplary method, the context information is voxel-wise context information.

Some embodiments of the third exemplary method may further include a step of performing feature decoding on the input point cloud and the first bitstream to generate a first point cloud.

Some embodiments of the third exemplary method may further include a step of performing a context-aware upsampling process on the aggregated features to generate a decoded point cloud.

Some embodiments of the third exemplary method may further include the steps of performing a feature-residual transformation on the pruned point cloud to generate a residual output; and the steps of adding the pruned point cloud to the residual output to generate a decoded point cloud.

In some embodiments of the third exemplary method, feature-residual transformation is performed on aggregated features.

In some embodiments of the third exemplary method, the step of predicting an occupancy state predicts a ground truth occupancy state of at least one voxel.

In some embodiments of the third exemplary method, the step of predicting occupancy predicts the likelihood that at least one voxel will be occupied.

In some embodiments of the third exemplary method, the step of removing voxels of the third point cloud removes voxels using a voxel pruning process.

In some embodiments of the third exemplary method, the step of predicting the occupancy state of at least one voxel further comprises: processing the aggregated features with multilayer perceptron (MLP) layers to generate MLP layer outputs; performing a softmax process on the MLP layer outputs to generate softmax output values; and performing thresholding on the softmax output values to generate a predicted occupancy state of at least one voxel of the third point cloud.

In some embodiments of the third exemplary method, thresholding of the softmax output values converts softmax output values greater than 0.5 to an output value of 1, and converts softmax output values less than or equal to 0.5 to an output value of 0.

In some embodiments of the third exemplary method, the step of predicting the occupancy state of at least one voxel comprises: aggregating at least one feature of the third point cloud; and generating a predicted occupancy state of at least one voxel of the third point cloud based on the aggregated feature.

In some embodiments of the third exemplary method, the step of aggregating at least one feature of the third point cloud comprises: repeating a cascading process one or more times, wherein the cascading process comprises: performing a sparse 3D convolution of an input point cloud to generate a convolution output point cloud; performing a nonlinear activation process on the convolution output point cloud to generate a nonlinear output point cloud; and preparing the nonlinear output point cloud as an input point cloud for a next cycle of the cascading process, wherein the third point cloud is an input point cloud for a first cycle of the cascading process, and wherein a last cycle of the cascading process generates aggregated features.

Some embodiments of the third exemplary method may further include a step of adding the third point cloud to the ReLU output point cloud of the last cycle of the cascading process.

In some embodiments of the third exemplary method, the step of aggregating at least one feature comprises: performing a sparse 3D convolution of an input point cloud to generate a convolution output point cloud; and performing a nonlinear activation process on the convolution output point cloud to generate aggregated features.

In some embodiments of the third exemplary method, the nonlinear activation process comprises a rectifier linear unit (ReLU) activation process, and the nonlinear output point cloud comprises a ReLU output point cloud.

In some embodiments of the third exemplary method, the step of aggregating at least one feature of the third point cloud comprises: repeating a first cascading process one or more times, wherein the first cascading process comprises: performing a first sparse 3D convolution of a first input point cloud to generate a first convolution output point cloud; performing a first nonlinear activation process on the first convolution output point cloud to generate a first nonlinear output point cloud; and preparing the first nonlinear output point cloud as the first input point cloud if there is a next cycle of the first cascading process, wherein the third point cloud is the first input point cloud for the first cycle of the first cascading process, and wherein a last cycle of the first cascading process generates the first cascading process output; A method for generating a second cascading process, the method comprising: performing a second sparse 3D convolution of a second input point cloud to generate a second convolution output point cloud; performing a second nonlinear activation process on the second convolution output point cloud to generate a second nonlinear output point cloud; and preparing the second nonlinear output point cloud as a second input point cloud when there is a next cycle of the second cascading process, wherein the third point cloud is the second input point cloud for a first cycle of the second cascading process, and the last cycle of the second cascading process generates a second cascading process output; concatenating the first cascading process output and the second cascading process output to generate a concatenated output; and adding the third point cloud to the concatenated output to generate aggregated features.

In some embodiments of the third exemplary method, the step of aggregating at least one feature comprises: repeating a first cascading process one or more times, wherein the first cascading process comprises: performing a first sparse 3D convolution of a first input point cloud to generate a first convolution output point cloud; performing a first ReLU (rectifier linear unit) activation process on the first convolution output point cloud to generate a first ReLU output point cloud; and preparing the first ReLU output point cloud as the first input point cloud when there is a next cycle of the first cascading process, wherein the third point cloud is the first input point cloud for the first cycle of the first cascading process, and wherein a last cycle of the first cascading process generates the first cascading process output; A step of repeating a second cascading process one or more times, wherein the second cascading process comprises: performing a second sparse 3D convolution of a second input point cloud to generate a second convolution output point cloud; performing a second ReLU (rectifier linear unit) activation process on the second convolution output point cloud to generate a second ReLU output point cloud; and preparing the second ReLU output point cloud as a second input point cloud when there is a next cycle of the second cascading process, wherein the third point cloud is the second input point cloud for a first cycle of the second cascading process, and the last cycle of the second cascading process generates a second cascading process output; connecting the first cascading process output and the second cascading process output to generate a connected output; and adding the third point cloud to the connected output to generate aggregated features.

In some embodiments of the third exemplary method, the step of aggregating at least one feature comprises: performing a self-attention process on the third point cloud; adding the third point cloud to an output of the self-attention process to generate an MLP process input; performing an MLP process on the MLP process input; and adding the MLP process input to the MLP process output to generate an aggregated feature.

In some embodiments of the third exemplary method, the self-attention process generates output features based on the k nearest neighbors of a voxel of the third point cloud.

In some embodiments of the third exemplary method, the step of aggregating at least one feature of the third point cloud comprises performing the feature aggregation process two or more times.

In some embodiments of the third exemplary method, the first neural network parameter set and the second neural network parameter set are the same neural network parameter set, and the same neural network parameter set is used by at least the first neural network and the second neural network.

In some embodiments of the third exemplary method, the first neural network parameter set and the second neural network parameter set are distinct but identical neural network parameter sets.

A fourth exemplary method according to some embodiments may include: obtaining a first point cloud; determining an occupancy state of at least one voxel of the first point cloud; generating a second point cloud by removing voxels of the first point cloud classified as empty according to the determined occupancy state; obtaining a third point cloud by associating features of the second point cloud with context information; downsampling the third point cloud using initial downsampling to obtain a fourth point cloud; and outputting the fourth point cloud as an encoded point cloud.

A fourth exemplary device according to some embodiments may include: a processor; and a non-transitory computer-readable medium storing instructions that, when executed by the processor, cause the device to obtain a first point cloud; determine an occupancy state of at least one voxel of the first point cloud; generate a second point cloud by removing voxels of the first point cloud classified as empty according to the determined occupancy state; obtain a third point cloud by associating features of the second point cloud with context information; obtain a fourth point cloud by downsampling the third point cloud using initial downsampling; and output the fourth point cloud as an encoded point cloud.

A fifth exemplary method/device according to some embodiments may include: accessing data including a first point cloud; and transmitting data including the first point cloud.

A fifth exemplary method/device according to some embodiments may include: an access unit configured to access data including a first point cloud; and a transmitter configured to transmit data including the first point cloud.

A sixth exemplary method/device according to some embodiments may include: a processor; and a non-transitory computer-readable medium storing instructions that, when executed by the processor, operate to cause the device to perform any one of the methods listed above.

A seventh exemplary method/device according to some embodiments may include at least one processor configured to perform any one of the methods listed above.

An eighth exemplary method/device according to some embodiments may include a computer-readable medium storing instructions for causing one or more processors to perform any one of the methods listed above.

A ninth exemplary method/device according to some embodiments may include at least one processor and at least one non-transitory computer-readable medium storing instructions for causing the at least one processor to perform any one of the methods listed above.

An exemplary signal according to some embodiments may include a bitstream generated according to any of the methods listed above.

The present disclosure describes various aspects, including tools, features, embodiments, models, approaches, and the like. Many of these aspects are described in detail, and sometimes in a way that may sound limiting, at least to show individual characteristics. However, this is for clarity of description and does not limit the disclosure or scope of the aspects. In fact, all of the different aspects can be combined and interchanged to provide additional aspects. Moreover, these aspects can also be combined and interchanged with aspects described in the previous applications.

The aspects described and contemplated in this disclosure may be implemented in many different forms. While certain embodiments are specifically illustrated, other embodiments are contemplated, and the discussion of particular embodiments does not limit the scope of implementations. At least one of the aspects relates generally to video encoding and decoding, and at least one other aspect relates generally to transmitting a bitstream generated or encoded. These and other aspects may be implemented as a method, an apparatus, a computer-readable storage medium storing instructions for encoding or decoding video data according to any of the described methods, and/or a computer-readable storage medium storing a bitstream generated according to any of the described methods.

In this disclosure, the terms "reconstructed" and "decoded" may be used interchangeably, the terms "pixel" and "sample" may be used interchangeably, and the terms "image", "picture" and "frame" may be used interchangeably. Typically, but not necessarily, the term "reconstructed" is used on the encoder side, while the term "decoded" is used on the decoder side.

The terms HDR (high dynamic range) and SDR (standard dynamic range) often convey particular dynamic range values to one of ordinary skill in the art. However, additional embodiments are also intended in which reference to HDR is understood to mean "higher dynamic range" and reference to SDR is understood to mean "lower dynamic range." These additional embodiments are not limited by any particular dynamic range values that may often be associated with the terms "high dynamic range" and "standard dynamic range."

Various methods are described herein, each of which comprises one or more steps or acts for achieving the described method. The order and/or use of particular steps and/or acts may be modified or combined, unless a particular order of steps or acts is required for the proper operation of the method. Additionally, terms such as “first,” “second,” etc. may be used in various embodiments to describe elements, components, steps, operations, etc., such as, for example, “first decoding” and “second decoding.” The use of such terms does not imply an order for the described operations, unless specifically required. Thus, in this example, the first decoding need not be performed before the second decoding, and may occur, for example, before the second decoding, during the second decoding, or in a time period overlapping with the second decoding.

For example, various numerical values may be used in the present disclosure. Specific values are provided for illustrative purposes only, and the described embodiments are not limited to these specific values.

The embodiments described herein may be performed by computer software implemented by a processor or other hardware, or by a combination of hardware and software. As a non-limiting example, the embodiments may be implemented by one or more integrated circuits. The processor may be of any type suitable to the technical environment, including, but not limited to, one or more of a microprocessor, a general purpose computer, a special purpose computer, and a multi-core architecture-based processor.

Various implementations include decoding. "Decoding," as used herein, may encompass all or part of, for example, processes performed on a received encoded sequence to generate a final output suitable for display. In various embodiments, such processes include processes typically performed by a decoder, such as one or more of entropy decoding, inverse quantization, inverse transform, and differential decoding. In various embodiments, such processes also, or alternatively, include processes performed by a decoder of various implementations described herein, such as extracting a picture from a tiled (packed) picture, determining an upsampling filter to use, and then upsampling the picture, and flipping the picture back to its intended orientation.

As additional examples, in one embodiment, "decoding" refers only to entropy decoding, in another embodiment, "decoding" refers only to differential decoding, and in another embodiment, "decoding" refers to a combination of entropy decoding and differential decoding. Whether the phrase "decoding process" is intended to refer specifically to a subset of operations or to the broader decoding process in general will be clear based on the context of the specific description.

Various implementations include encoding. In a manner similar to the discussion above regarding "decoding," "encoding," as used herein, may encompass all or part of the processes performed on an input video sequence to generate, for example, an encoded bitstream. In various embodiments, such processes include one or more of the processes typically performed by an encoder, for example, segmentation, differential encoding, transform, quantization, and entropy encoding. In various embodiments, such processes also or alternatively include processes performed by an encoder of various implementations described herein.

As additional examples, in one embodiment "encoding" refers only to entropy encoding, in another embodiment "encoding" refers only to differential encoding, and in another embodiment "encoding" refers to a combination of differential encoding and entropy encoding. Whether the phrase "encoding process" is intended to refer specifically to a subset of operations or to the broader encoding process in general will be clear based on the context of the specific description.

When a drawing is presented as a flow diagram, it should be understood that the drawing also provides a block diagram of a corresponding device. Similarly, when a drawing is presented as a block diagram, it should be understood that the drawing also provides a flow diagram of a corresponding method/process.

The implementations and aspects described herein may be implemented, for example, as a method or process, an apparatus, a software program, a data stream, or a signal. Even if discussed in the context of only a single form of implementation (e.g., discussed only as a method), the implementation of the features discussed may also be implemented in other forms (e.g., as an apparatus or a program). An apparatus may be implemented, for example, in suitable hardware, software, and firmware. The methods may be implemented in, for example, a processor, which refers to general processing devices, including, for example, a computer, a microprocessor, an integrated circuit, or a programmable logic device. Processors also include communication devices, such as, for example, computers, cell phones, portable/personal digital assistants ("PDAs"), and other devices that facilitate communication of information between end users.

Reference to “one embodiment” or “an embodiment” or “an implementation” or “an implementation” as well as other variations thereof means that a particular feature, structure, characteristic, etc. described in connection with the embodiment is included in at least one embodiment. Thus, the appearances of the phrases “in one embodiment” or “in an embodiment” or “in an implementation” or “in an implementation” as well as any other variations thereof in various places are not necessarily all referring to the same embodiment.

Additionally, the present disclosure may refer to "determining" various pieces of information. Determining the information may include, for example, one or more of estimating the information, calculating the information, predicting the information, or retrieving the information from memory.

Additionally, the present disclosure may refer to "accessing" various pieces of information. Accessing the information may include, for example, one or more of receiving the information, retrieving the information (e.g., from memory), storing the information, moving the information, copying the information, calculating the information, determining the information, predicting the information, or estimating the information.

Additionally, the present disclosure may refer to "receiving" various pieces of information. Receiving is intended to be a broad term, as in "accessing." Receiving information may include, for example, one or more of accessing the information, or retrieving the information (e.g., from memory). Furthermore, "receiving" may typically involve, in some way, actions such as storing the information, processing the information, transmitting the information, moving the information, copying the information, erasing the information, generating the information, determining the information, predicting the information, or estimating the information.

For example, in the cases of "A/B", "A and/or B", and "at least one of A and B", it should be understood that the use of any of the following "/", "and/or", and "at least one of" is intended to encompass the selection of the first listed option (A) only, or the selection of the second listed option (B) only, or the selection of both options (A and B). As a further example, in the cases of "A, B, and/or C" and "at least one of A, B, and C", such phrases are intended to encompass the selection of the first listed option (A) only, or the selection of the second listed option (B) only, or the selection of the third listed option (C) only, or the selection of the first and second listed options (A and B) only, the selection of the first and third listed options (A and C) only, or the selection of the second and third listed options (B and C) only, or the selection of all three options (A and B and C). This can be extended by as many items as listed.

Also, as used herein, the word "signal" refers, inter alia, to informing a corresponding decoder of something. For example, in certain embodiments, an encoder signals a particular parameter among a plurality of parameters for region-based filter parameter selection for de-artifact filtering. In this way, in one embodiment, the same parameter is used on both the encoder side and the decoder side. Thus, for example, the encoder can transmit a particular parameter to the decoder so that the decoder can use the same particular parameter (explicit signaling). Conversely, if the decoder already has the particular parameter as well as other parameters, signaling can be used without transmission simply to enable the decoder to know and select the particular parameter (implicit signaling). By avoiding transmission of any actual functions, bit savings are realized in various embodiments. It should be appreciated that signaling can be accomplished in a variety of ways. For example, in various embodiments, one or more syntax elements, flags, etc. are used to signal information to a corresponding decoder. While the foregoing relates to the word "signaling" in verb form, the word "signal" may also be used herein as a noun.

Implementations may generate various signals formatted to convey information that may be stored or transmitted, for example. The information may include, for example, instructions for performing a method, or data generated by one of the described implementations. For example, the signal may be formatted to convey a bitstream of the described embodiment. Such a signal may be formatted, for example, as an electromagnetic wave (e.g., using a radio frequency portion of the spectrum) or as a baseband signal. Formatting may include, for example, encoding a data stream and modulating a carrier with the encoded data stream. The information conveyed by the signal may be, for example, analog or digital information. The signal may be transmitted over a variety of different wired or wireless links, as is known. The signal may be stored on a processor-readable medium.

Note that various hardware elements of one or more of the described embodiments are referred to as "modules" which carry out (i.e., perform, execute, etc.) various functions described herein in connection with their respective modules. As used herein, a module includes hardware (e.g., one or more processors, one or more microprocessors, one or more microcontrollers, one or more microchips, one or more application-specific integrated circuits (ASICs), one or more field programmable gate arrays (FPGAs), one or more memory devices) deemed suitable by one of ordinary skill in the art for a given implementation. It should be noted that each described module may also include instructions executable to perform one or more of the functions described as being performed by the respective module, which instructions may take the form of or include hardware (i.e., hardwired) instructions, firmware instructions, software instructions, and the like, and may be stored in any suitable non-transitory computer-readable medium or media, such as commonly referred to as RAM, ROM, etc.

Although the features and elements are described above in specific combinations, those skilled in the art will appreciate that each feature or element may be used alone or in any combination with other features and elements. Additionally, the methods described herein may be implemented as a computer program, software, or firmware contained in a computer-readable medium for execution by a computer or processor. Examples of computer-readable storage media include, but are not limited to, read only memory (ROM), random access memory (RAM), registers, cache memory, semiconductor memory devices, magnetic media such as internal hard disks and removable disks, magneto-optical media, and optical media such as CD-ROM disks and digital versatile disks (DVDs). The processor, in conjunction with software, may be used to implement a radio frequency transceiver for use in a WTRU, a UE, a terminal, a base station, an RNC, or any host computer.

Claims (78)

As a method,
A step of obtaining a second point cloud by upsampling the first point cloud using initial upsampling;
A step of obtaining a third point cloud by associating features of the second point cloud with context information;
a step of predicting the occupancy state of at least one voxel of the third point cloud; and
A step of generating a pruned point cloud by removing voxels of the third point cloud classified as empty based on the predicted occupancy status.
A method comprising: A method in claim 1, wherein the initial upsampling includes nearest neighbor upsampling. A method according to claim 1 or 2, wherein associating features comprises obtaining the third point cloud by connecting features of the second point cloud with the context information. A method according to any one of claims 1 to 3, wherein the context information is voxel-unit context information. A method according to any one of claims 1 to 4, wherein the context information includes a context point cloud. A method according to any one of claims 1 to 5, wherein the context information includes information about the second point cloud. A method according to any one of claims 1 to 6, wherein the context information includes information on voxel occupancy status of the second point cloud. A method according to any one of claims 1 to 7, wherein the context information includes information about the location of a child voxel with respect to the location of a parent voxel of the first point cloud. A method according to any one of claims 1 to 8, wherein the context information includes coordinate information regarding the location of an occupied voxel of at least one of the first and second point clouds. In any one of claims 1 to 9,
The above context information includes coordinate information,
A method wherein the above coordinate information is in one of the forms of Euclidean coordinates, spherical coordinates, and cylindrical coordinates. A method according to any one of claims 1 to 10, wherein the context information provides known information about the first point cloud in addition to information available for the initial upsampling of the first point cloud. A method according to any one of claims 1 to 11, wherein the context information includes a bit depth of the second point cloud. A method according to any one of claims 1 to 21, further comprising the step of generating the first point cloud by performing feature decode on the input point cloud and the first bitstream. In Article 13,
A step of performing feature aggregation on the above pruned point cloud to generate aggregated features; and
A method further comprising the step of performing a context-aware upsampling process on the above-mentioned aggregated features to generate a decoded point cloud. In Article 13,
A step of performing feature to residual conversion on the above pruned point cloud to generate residual output; and
A method further comprising the step of adding the pruned point cloud to the residual output to generate a decoded point cloud. In Article 15,
Further comprising a step of performing feature aggregation on the above pruned point cloud to generate aggregated features,
A method wherein the above feature-residual transformation is performed on the above aggregated features. A method according to any one of claims 1 to 16, wherein the step of predicting the occupancy state is performed using a first neural network. A method according to any one of claims 1 to 17, wherein the step of predicting the occupancy status predicts the ground-truth occupancy status of at least one voxel. A method according to any one of claims 1 to 18, wherein the step of predicting the occupancy state predicts a possibility that the at least one voxel is occupied. A method according to any one of claims 1 to 19, wherein removing voxels of the third point cloud removes voxels using a voxel pruning process. A method according to any one of claims 1 to 20, further comprising a step of aggregating at least one feature of the second point cloud. In any one of claims 1 to 21, the step of predicting the occupancy state of at least one voxel comprises:
a step of aggregating at least one feature of the third point cloud;
A step of processing the aggregated features with multilayer perceptron (MLP) layers to generate MLP layer output;
A step of performing a softmax process on the above MLP layer output to generate softmax output values; and
A method comprising the step of performing thresholding of the softmax output values to generate a predicted occupancy state of at least one voxel of the third point cloud. In the 22nd paragraph, the threshold processing of the softmax output values is a method in which softmax output values exceeding 0.5 are converted into an output value of 1, and softmax output values less than or equal to 0.5 are converted into an output value of 0. In any one of claims 1 to 21, the step of predicting the occupancy state of at least one voxel comprises:
a step of aggregating at least one feature of the third point cloud; and
A method comprising the step of generating a predicted occupancy state of at least one voxel of the third point cloud based on the aggregated features. In any one of claims 22 to 24, the step of aggregating at least one feature comprises:
A method comprising the steps of repeating a cascading process one or more times, said cascading process comprising:
A step of performing sparse 3D convolution of an input point cloud to generate a convolution output point cloud;
A step of generating a nonlinear output point cloud by performing a nonlinear activation process on the above convolution output point cloud; and
Including a step of preparing the nonlinear output point cloud as an input point cloud when there is a next cycle of the above cascading process,
The above third point cloud is the input point cloud for the first cycle of the above cascading process,
The final cycle of the above cascading process is a method for generating the aggregated features. A method in claim 25, further comprising the step of adding the third point cloud to the ReLU output point cloud of the last cycle of the cascading process. In any one of claims 22 to 24, the step of aggregating at least one feature comprises:
A step of performing sparse 3D convolution of an input point cloud to generate a convolution output point cloud; and
A method comprising the step of performing a nonlinear activation process on the convolution output point cloud to generate the aggregated features. A method according to any one of claims 25 to 27, wherein the nonlinear activation process comprises a rectifier linear unit (ReLU) activation process, and the nonlinear output point cloud comprises a ReLU output point cloud. In any one of claims 22 to 24, the step of aggregating at least one feature comprises:
A step of repeating a first cascading process one or more times, wherein said first cascading process:
A step of performing a first sparse 3D convolution of a first input point cloud to generate a first convolution output point cloud;
A step of generating a first nonlinear output point cloud by performing a first nonlinear activation process on the first convolution output point cloud; and
Including the step of preparing the first nonlinear output point cloud as the first input point cloud when there is a next cycle of the first cascading process,
The above third point cloud is the first input point cloud for the first cycle of the above first cascading process,
The last cycle of the first cascading process produces the first cascading process output;
A step of repeating the second cascading process one or more times, wherein the second cascading process comprises:
A step of performing a second sparse 3D convolution of the second input point cloud to generate a second convolution output point cloud;
A step of generating a second nonlinear output point cloud by performing a second nonlinear activation process on the second convolution output point cloud; and
Including the step of preparing the second nonlinear output point cloud as a second input point cloud when there is a next cycle of the second cascading process,
The above third point cloud is the second input point cloud for the first cycle of the second cascading process,
The last cycle of the second cascading process produces the second cascading process output;
A step of concatenating the output of the first cascading process and the output of the second cascading process to generate a concatenation output; and
A method comprising the step of adding the third point cloud to the connected output to generate the aggregated features. In any one of claims 22 to 24, the step of aggregating at least one feature comprises:
A step of repeating the first cascading process one or more times, wherein the first cascading process comprises:
A step of performing a first sparse 3D convolution of a first input point cloud to generate a first convolution output point cloud;
A step of generating a first ReLU output point cloud by performing a first ReLU (rectifier linear unit) activation process on the first convolution output point cloud; and
Including a step of preparing the first ReLU output point cloud as the first input point cloud when there is a next cycle of the first cascading process,
The above third point cloud is the first input point cloud for the first cycle of the above first cascading process,
The last cycle of the first cascading process produces the first cascading process output;
A step of repeating a second cascading process one or more times, wherein said second cascading process comprises:
A step of performing a second sparse 3D convolution of the second input point cloud to generate a second convolution output point cloud;
A step of generating a second ReLU output point cloud by performing a second ReLU (rectifier linear unit) activation process on the second convolution output point cloud; and
Including a step of preparing the second ReLU output point cloud as a second input point cloud when there is a next cycle of the second cascading process,
The above third point cloud is the second input point cloud for the first cycle of the second cascading process,
The last cycle of the second cascading process produces the second cascading process output;
A step of generating a connected output by connecting the output of the first cascading process and the output of the second cascading process; and
A method comprising the step of adding the third point cloud to the connected output to generate the aggregated features. In any one of claims 22 to 24, the step of aggregating at least one feature comprises:
A step of performing a self-attention process on the third point cloud;
A step of adding the third point cloud to the self-attention process output to generate an MLP process input;
A step of performing an MLP process on the above MLP process input; and
A method comprising the step of adding said MLP process input to said MLP process output to generate said aggregated features. In the 31st paragraph, the self-attention process generates output features based on k nearest neighbors of voxels of the third point cloud. A method according to any one of claims 22 to 24, wherein the step of aggregating at least one feature of the third point cloud comprises performing the feature aggregation process two or more times. As a device,
processor; and
When executed by said processor, said device causes
Obtain a second point cloud by upsampling the first point cloud using initial upsampling;
The features of the second point cloud are associated with context information to obtain a third point cloud;
Predict the occupancy state of at least one voxel of the third point cloud;
Based on the predicted occupancy status, the voxels of the third point cloud classified as empty are removed to generate a pruned point cloud.
A non-transitory computer-readable medium storing instructions for operation.
A device comprising: A device in claim 34, wherein the initial upsampling comprises nearest neighbor upsampling. A device according to claim 34 or 35, wherein associating features comprises obtaining the third point cloud by connecting features of the second point cloud with the context information. As a device,
Device according to Article 34; and
(i) an antenna configured to receive a signal, said signal comprising data representing an image; (ii) a band limiter configured to limit the received signal to a frequency band comprising said data representing said image; or (iii) at least one display configured to display said image.
A device comprising: A device according to claim 37, further comprising at least one of a TV, a cell phone, a tablet, and a set-top box (STB). A computer-readable medium containing instructions, said instructions causing one or more processors to:
Obtain a second point cloud by upsampling the first point cloud using initial upsampling;
The features of the second point cloud are associated with context information to obtain a third point cloud;
Predict the occupancy state of at least one voxel of the third point cloud;
Based on the predicted occupancy state, the voxels of the third point cloud classified as empty are removed to generate a pruned point cloud.
Computer readable medium. A computer program product comprising instructions, wherein the instructions, when the program is executed by one or more processors, cause the one or more processors to:
Obtain a second point cloud by upsampling the first point cloud using initial upsampling;
The features of the second point cloud are associated with context information to obtain a third point cloud;
Predict the occupancy state of at least one voxel of the third point cloud;
A computer program product that generates a pruned point cloud by removing voxels of the third point cloud classified as empty based on the predicted occupancy state. As a method,
A step of determining an upsampled second point cloud by performing context-aware upsampling of the first point cloud.
Including,
The above context-aware upsampling:
associating features of a third point cloud with contextual information, wherein the third point cloud is at least partially based on an initial upsampled version of the first point cloud; and
A method comprising generating the upscaled second point cloud by removing voxels of the fourth point cloud that are predicted to be empty at least partially based on the context information from the third point cloud. As a method,
A step of obtaining a second point cloud by upsampling the first point cloud using initial upsampling;
A step of obtaining a third point cloud by associating features of the second point cloud with context information;
A step of predicting the occupancy state of at least one voxel of the third point cloud;
The step of predicting the occupancy state of at least one voxel comprises the step of aggregating at least one feature of the third point cloud,
The step of aggregating at least one feature of the third point cloud comprises a step of using a first neural network,
Aggregating at least one feature of the third point cloud using the first neural network comprises using a first neural network parameter set together with the first neural network;
A step of generating a pruned point cloud by removing voxels of the third point cloud classified as empty according to the predicted occupancy state; and
A step of generating aggregated features by performing feature aggregation on the above pruned point cloud -
Performing feature aggregation on the above pruned point cloud involves using a second neural network,
Generating the aggregated features using the second neural network comprises using a second neural network parameter set together with the second neural network,
The above first neural network parameter set is identical to the above second neural network parameter set -
A method comprising: A method, further comprising the step of aggregating at least one feature of the second point cloud, in claim 42. In Article 43,
The step of aggregating at least one feature of the second point cloud comprises a step of using a third neural network,
Aggregating at least one feature of the second point cloud using the third neural network comprises using a third neural network parameter set together with the third neural network,
The above third neural network parameter set is identical to the above first neural network parameter set, method. A method according to any one of claims 42 to 44, wherein the initial upsampling comprises nearest neighbor upsampling. A method according to any one of claims 42 to 45, wherein associating features comprises obtaining the third point cloud by connecting features of the second point cloud with the context information. A method according to any one of claims 42 to 46, wherein the step of associating features comprises the step of obtaining the third point cloud by associating features of the second point cloud with the context information. A method according to any one of claims 42 to 47, wherein the context information is voxel-unit context information. A method according to any one of claims 42 to 48, further comprising the step of performing feature decoding on an input point cloud and a first bitstream to generate the first point cloud. A method, further comprising the step of performing a context-aware upsampling process on the aggregated features to generate a decoded point cloud, in claim 49. In Article 49,
A step of performing feature-residual transformation on the above pruned point cloud to generate a residual output; and
A method further comprising the step of adding the pruned point cloud to the residual output to generate a decoded point cloud. A method in claim 51, wherein the feature-residual transformation is performed on the aggregated features. A method according to any one of claims 42 to 52, wherein the step of predicting the occupancy state predicts the actual occupancy state of at least one voxel. A method according to any one of claims 42 to 53, wherein the step of predicting the occupancy state predicts a probability that the at least one voxel is occupied. A method according to any one of claims 42 to 54, wherein removing voxels of the third point cloud removes voxels using a voxel pruning process. In any one of claims 42 to 55, the step of predicting the occupancy state of at least one voxel comprises:
A step of processing the above-mentioned aggregated features with multilayer perceptron (MLP) layers to generate an MLP layer output;
A step of performing a softmax process on the above MLP layer output to generate softmax output values; and
A method comprising the step of performing threshold processing of the softmax output values to generate a predicted occupancy state of at least one voxel of the third point cloud. In claim 56, the threshold processing of the softmax output values is a method in which softmax output values exceeding 0.5 are converted into an output value of 1, and softmax output values less than or equal to 0.5 are converted into an output value of 0. In any one of claims 42 to 55, the step of predicting the occupancy state of at least one voxel comprises:
a step of aggregating at least one feature of the third point cloud; and
A method comprising the step of generating a predicted occupancy state of at least one voxel of the third point cloud based on the aggregated features. In any one of claims 56 to 58, the step of aggregating at least one feature of the third point cloud comprises:
A method comprising: repeating a cascading process one or more times;
A step of performing sparse 3D convolution of an input point cloud to generate a convolution output point cloud;
A step of generating a nonlinear output point cloud by performing a nonlinear activation process on the above convolution output point cloud; and
Including a step of preparing the nonlinear output point cloud as an input point cloud when there is a next cycle of the above cascading process,
The above third point cloud is the input point cloud for the first cycle of the above cascading process,
The final cycle of the above cascading process is a method for generating the aggregated features. A method in claim 59, further comprising the step of adding the third point cloud to the ReLU output point cloud of the last cycle of the cascading process. In any one of claims 56 to 58, the step of aggregating at least one feature comprises:
A step of performing sparse 3D convolution of an input point cloud to generate a convolution output point cloud; and
A method comprising the step of performing a nonlinear activation process on the convolution output point cloud to generate the aggregated features. A method according to any one of claims 59 to 61, wherein the nonlinear activation process comprises a rectifier linear unit (ReLU) activation process, and the nonlinear output point cloud comprises a ReLU output point cloud. In any one of claims 56 to 58, the step of aggregating at least one feature of the third point cloud comprises:
A step of repeating the first cascading process one or more times, wherein the first cascading process comprises:
A step of performing a first sparse 3D convolution of a first input point cloud to generate a first convolution output point cloud;
A step of generating a first nonlinear output point cloud by performing a first nonlinear activation process on the first convolution output point cloud; and
Including the step of preparing the first nonlinear output point cloud as the first input point cloud when there is a next cycle of the first cascading process,
The above third point cloud is the first input point cloud for the first cycle of the above first cascading process,
The last cycle of the first cascading process produces the first cascading process output;
A step of repeating a second cascading process one or more times, wherein said second cascading process comprises:
A step of performing a second sparse 3D convolution of the second input point cloud to generate a second convolution output point cloud;
A step of generating a second nonlinear output point cloud by performing a second nonlinear activation process on the second convolution output point cloud; and
Including the step of preparing the second nonlinear output point cloud as a second input point cloud when there is a next cycle of the second cascading process,
The above third point cloud is the second input point cloud for the first cycle of the second cascading process,
The last cycle of the second cascading process produces the second cascading process output;
A step of generating a connected output by connecting the output of the first cascading process and the output of the second cascading process; and
A method comprising the step of adding the third point cloud to the connected output to generate the aggregated features. In any one of claims 56 to 58, the step of aggregating at least one feature comprises:
A step of repeating the first cascading process one or more times, wherein the first cascading process comprises:
A step of performing a first sparse 3D convolution of a first input point cloud to generate a first convolution output point cloud;
A step of generating a first ReLU output point cloud by performing a first ReLU (rectifier linear unit) activation process on the first convolution output point cloud; and
Including a step of preparing the first ReLU output point cloud as the first input point cloud when there is a next cycle of the first cascading process,
The above third point cloud is the first input point cloud for the first cycle of the above first cascading process,
The last cycle of the first cascading process produces the first cascading process output;
A step of repeating a second cascading process one or more times, wherein said second cascading process comprises:
A step of performing a second sparse 3D convolution of the second input point cloud to generate a second convolution output point cloud;
A step of generating a second ReLU output point cloud by performing a second ReLU (rectifier linear unit) activation process on the second convolution output point cloud; and
Including a step of preparing the second ReLU output point cloud as a second input point cloud when there is a next cycle of the second cascading process,
The above third point cloud is the second input point cloud for the first cycle of the second cascading process,
The last cycle of the second cascading process produces the second cascading process output;
A step of generating a connected output by connecting the output of the first cascading process and the output of the second cascading process; and
A method comprising the step of adding the third point cloud to the connected output to generate the aggregated features. In any one of claims 56 to 58, the step of aggregating at least one feature comprises:
A step of performing a self-attention process on the above third point cloud;
A step of adding the third point cloud to the self-attention process output to generate an MLP process input;
A step of performing an MLP process on the above MLP process input; and
A method comprising the step of adding said MLP process input to said MLP process output to generate said aggregated features. In claim 65, the self-attention process generates output features based on k nearest neighbors of voxels of the third point cloud. A method according to any one of claims 56 to 58, wherein the step of aggregating at least one feature of the third point cloud comprises performing the feature aggregation process two or more times. A method according to any one of claims 42 to 67, wherein the first neural network parameter set and the second neural network parameter set are the same neural network parameter set, and the same neural network parameter set is used by at least the first neural network and the second neural network. A method according to any one of claims 42 to 68, wherein the first neural network parameter set and the second neural network parameter set are distinct but identical neural network parameter sets. As a method,
Step of obtaining the first point cloud;
A step of determining the occupancy state of at least one voxel of the first point cloud;
A step of generating a second point cloud by removing voxels of the first point cloud classified as empty based on the determined occupancy status;
A step of obtaining a third point cloud by associating features of the second point cloud with context information;
A step of obtaining a fourth point cloud by downsampling the third point cloud using initial downsampling; and
A step of outputting the above fourth point cloud as an encoded point cloud.
A method comprising: As a device,
processor; and
When executed by said processor, said device causes
Acquire the first point cloud;
Determine the occupancy state of at least one voxel of the first point cloud;
Based on the determined occupancy status, voxels of the first point cloud classified as empty are removed to generate a second point cloud;
The features of the second point cloud are associated with context information to obtain a third point cloud;
Using initial downsampling, the third point cloud is downsampled to obtain a fourth point cloud;
To output the above 4th point cloud as an encoded point cloud.
A non-transitory computer-readable medium storing instructions for operation.
A device comprising: As a method,
A step of accessing data including a first point cloud; and
A step of transmitting the data including the first point cloud
A method comprising: As a device,
An access unit configured to access data including a first point cloud; and
A transmitter configured to transmit said data including said first point cloud
A device comprising: As a device,
processor; and
A non-transitory computer-readable medium storing instructions that, when executed by said processor, cause said device to perform any one of the methods of claims 1 to 33 and claims 41 to 70.
A device comprising: As a device, at least one processor configured to perform the method of any one of claims 1 to 33 and claims 41 to 70.
A device comprising: A computer-readable medium storing instructions for causing one or more processors to perform the method of any one of claims 1 to 33 and claims 41 to 70, as a device.
A device comprising: As a device, at least one processor and at least one non-transitory computer-readable medium storing instructions for causing the at least one processor to perform a method according to any one of claims 1 to 33 and claims 41 to 70.
A device comprising: As a signal, a bitstream generated according to any one of claims 1 to 33 and claims 41 to 70
A signal containing .

Images