RU2847190C2 - Methods, devices and systems for encoding and decoding directional sound sources - Google Patents

Abstract

FIELD: physics.

SUBSTANCE: invention relates to computer engineering for processing audio data. Technical result is achieved by receiving an encoded basic audio signal, metadata of an encoded radiation pattern and metadata of an encoded audio object, wherein the audio object metadata includes 6DoF source orientation information; decoding the encoded base audio signal to determine the base audio signal; decoding the metadata of the encoded radiation pattern to determine a decoded radiation pattern; decoding audio object metadata; and rendering the core audio signal based on the metadata of the audio object and the decoded radiation pattern.

EFFECT: rendering a core audio signal based on metadata of an audio object and a decoded radiation pattern.

11 cl, 13 dwg

Description

[0001] The present invention claims priority to U.S. Patent Application No. 62/658,067, filed April 16, 2018; U.S. Patent Application No. 62/681,429, filed June 6, 2018; and U.S. Patent Application No. 62/741,419, filed October 4, 2018, which are incorporated herein by reference in their entireties.

AREA OF TECHNOLOGY

[0002] The present invention relates to encoding and decoding directional sound sources and auditory scenes based on a plurality of dynamic and/or moving directional sources.

BACKGROUND OF THE INVENTION

[0003] Real-world sound sources, whether natural or man-made (loudspeakers, musical instruments, voices, mechanical devices), emit sound anisotropically. Characterizing the radiation patterns (or "directivity") of sound sources can be important for proper rendering, particularly in the context of interactive environments such as video games and virtual/augmented reality (VR/AR) applications. In these environments, users typically interact with directional sound objects by walking over them, thereby changing their acoustic perspective relative to the generated sound (also known as 6 degrees of freedom (DoF) rendering). The user can also grab and dynamically rotate virtual objects, which again requires rendering different directions in the radiation pattern of the corresponding sound source(s). In addition to more realistically rendering the direct propagation effects from the source to the listener, radiation characteristics will also play a significant role in the higher-order acoustic coupling between the source and its environment (e.g., the virtual environment in a game), thereby affecting the reverberant sound (i.e., sound waves traveling back and forth, like an echo). As a result, such reverberation can influence other spatial cues, such as perceived distance.

[0004] Most audio game engines provide some way to represent and render directional sound sources, but are typically limited to a simple directivity index, relying on the definition of simple first-order cosine functions or "sound cones" (e.g., cosine functions raised to a power) and simple high-pass filters with a smooth rolloff. These representations are insufficient for representing real-world radiation patterns, and they are also not very suitable for a simplified/combined representation of multiple directional sound sources.

BRIEF DESCRIPTION OF THE INVENTION

[0005] Various methods for processing audio are disclosed herein. Some such methods may include encoding directional audio data. For example, some methods may include receiving a monophonic audio signal corresponding to an audio object and representing a radiation pattern corresponding to the audio object. The radiation pattern may, for example, contain audio levels corresponding to multiple sampling periods, multiple frequency bands, and multiple directions. Some methods may include encoding the monophonic audio signal and encoding the radiation pattern of a source to determine metadata of the radiation pattern. Encoding the radiation pattern may include determining a spherical harmonic transform of the radiation pattern representation and compressing the spherical harmonic transform to obtain metadata of the encoded radiation pattern.

[0006] Some such methods may include encoding multiple directional sound objects based on a cluster of sound objects. The radiation pattern may represent a centroid that reflects the average sound level for each frequency band. In some such implementations, multiple directional sound objects are encoded as a single directional sound object, the directivity of which corresponds to a time-varying energy-weighted average of the spherical harmonic coefficients of each sound object. Metadata of the encoded radiation pattern may indicate the position of the cluster of sound objects, i.e., the average position of each sound object.

[0007] Some methods may include encoding group metadata related to the radiation pattern of a group of directional sound objects. In some examples, the source radiation pattern may be scaled to the amplitude of the input radiation pattern in the frequency direction to determine the normalized radiation pattern. According to some implementations, spherical harmonic transform compression may include a singular value decomposition method, principal component analysis, discrete cosine transforms, data-independent bases, and/or removal of spherical harmonic transform coefficients that are above a threshold order of spherical harmonic coefficients.

[0008] Some alternative methods may include decoding audio data. For example, some such methods may include receiving an encoded base audio signal, encoded emission scheme metadata, and encoded audio object metadata, and decoding the encoded base audio signal to determine the base audio signal. Some such methods may include decoding the encoded emission scheme metadata to determine the decoded emission scheme, decoding the emission scheme metadata, and rendering the base audio signal based on the audio object metadata and the decoded emission scheme.

[0009] In some cases, the audio object metadata may comprise at least one of time-varying 3-degree-of-freedom (3DoF) or 6-degree-of-freedom (6DoF) source orientation information. The base audio signal may comprise multiple directional objects based on a cluster of objects. The decoded radiation pattern may represent a centroid that reflects an average value for each frequency band. In some examples, rendering may be based on applying subband gains based at least in part on the decoded radiation data to the decoded base audio signal. The encoded radiation pattern metadata may correspond to a time- and frequency-varying set of spherical harmonic coefficients.

[0010] According to some implementations, the encoded radiation pattern metadata may include audio object type metadata. The audio object type metadata may, for example, point to parametric data of a radiation pattern. The parametric data of a radiation pattern may comprise a cosine function, a sine function, and/or a cardioid function. In some examples, the audio object type metadata may point to data of a radiation pattern of a database. Decoding the encoded radiation pattern metadata to determine a decoded radiation pattern may include querying a directionality data structure that contains audio object types and corresponding data of a radiation pattern. In some examples, the audio object type metadata may point to dynamic data of a radiation pattern. The dynamic data of a radiation pattern may correspond to a time- and frequency-varying set of spherical harmonic coefficients. Some methods may include receiving the dynamic data of a radiation pattern before receiving the encoded base audio signal.

[0011] Some or all of the methods described herein may be performed by one or more devices in accordance with instructions (e.g., software) stored in one or more persistent storage media. Such persistent storage media may comprise storage devices such as those described herein, including, but not limited to, random access memory (RAM), read-only memory (ROM), etc. Accordingly, various inventive features of the subject matter described in the present invention may be implemented in one or more persistent storage media having software stored therein. The software may, for example, comprise instructions for controlling at least one device for processing audio data. The software may, for example, be configured to be executed by one or more components of a control system such as those described herein. The software may, for example, comprise instructions for performing one or more of the methods disclosed herein.

[0012] At least some aspects of the present invention may be implemented by means of an apparatus. For example, one or more devices may be adapted to perform, at least in part, the methods disclosed herein. In some implementations, the apparatus may comprise an interface system and a control system. The interface system may comprise one or more network interfaces, one or more interfaces between the control system and a memory system, one or more interfaces between the control system and another device, and/or one or more interfaces for external devices. The control system may comprise at least one of a general-purpose single- or multi-chip processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic device, a circuit on discrete components or a transistor logic circuit, or discrete hardware components. Accordingly, in some implementations, the control system may comprise one or more processors and one or more non-transitory storage media operatively connected to the one or more processors.

[0013] According to some such examples, the control system may be configured to receive, via an interface system, audio data corresponding to at least one audio object. In some examples, the audio data may comprise a monophonic audio signal, audio object position metadata, audio object size metadata, and a rendering parameter. Some such methods may include determining whether the rendering parameter indicates a position mode or a directional mode, and upon determining that the rendering parameter indicates a directional mode, rendering the audio data for playback through at least one loudspeaker in accordance with the directional pattern indicated by the position metadata and/or the size metadata.

[0014] In some examples, rendering audio data may include interpreting audio object position metadata as audio object orientation metadata. Audio object position metadata may, for example, include x,y,z coordinate data, spherical coordinate data, and/or cylindrical coordinate data. In some cases, audio object orientation metadata may include yaw, pitch, and roll data.

[0015] According to some examples, rendering audio data may include interpreting audio object size metadata as directionality metadata that corresponds to a directionality pattern. In some implementations, rendering audio data may include requesting a data structure that contains multiple directionality patterns and correlating position metadata and/or size metadata with one or more directionality patterns. In some cases, the control system may be configured to receive the data structure via an interface system. In some examples, the data structure may be received before the audio data. In some implementations, the audio data may be received in Dolby Atmos format. The audio object position metadata may, for example, correspond to world coordinates or model coordinates.

[0016] The details of one or more embodiments of the subject matter described in this specification are set forth in the accompanying drawings and the description below. Other features, aspects, and advantages will be apparent from the description, drawings, and claims. It should be noted that relative dimensions in the following figures may not be to scale. Like reference numerals and designations in different drawings generally indicate like elements.

BRIEF DESCRIPTION OF GRAPHIC MATERIALS

[0017] Fig. 1A is a block diagram illustrating blocks of a method for encoding audio according to one example.

[0018] Fig. 1B shows blocks of a process that may be implemented by a coding system for dynamically coding frame-by-frame directionality information for a directional audio object according to one example.

[0019] Fig. 1C shows blocks of a process that may be implemented by a decoding system according to one example.

[0020] Fig. 2A and 2B show the radiation patterns of a sound object in two different frequency bands.

[0021] Fig. 2C is a graph showing examples of normalized and non-normalized radiation patterns according to one example.

[0022] Fig. 3 shows an example of a hierarchy that includes audio data and various types of metadata.

[0023] Fig. 4 is a block diagram showing blocks of a method for decoding audio according to one example.

[0024] Fig. 5A shows a drum cymbal.

[0025] Fig. 5B shows an example of a speaker system.

[0026] Fig. 6 is a block diagram showing blocks of a method for decoding audio according to one example.

[0027] Fig. 7 shows one example of encoding a plurality of audio objects.

[0028] Fig. 8 is a block diagram showing examples of components of an apparatus that may be adapted to perform at least some of the methods described herein.

[0029] Similar reference numerals and designations in different drawings indicate similar elements.

DETAILED DESCRIPTION

[0030] An aspect of the present invention relates to the representation of complex radiation patterns and their efficient coding. Some such implementations may include one or more of the following:

1. Representation of general sound radiation patterns as time- and frequency-dependent N-th-order coefficients of a real-valued spherical harmonic (SPH) expansion (N>=1). This representation can also be extended to depend on the level of the sound reproduction signal. Unlike the case in which the directional source signal itself is represented as an HOA-like PCM representation, the monophonic object signal can be encoded separately from its directivity information, and it is represented as a set of time-dependent scalar SPH coefficients in subbands.

2. An efficient coding scheme to reduce the bit rate required to represent this information.

3. A solution that dynamically combines emitter patterns so that a scene created by multiple sound emitters can be represented by an equivalently reduced number of emitters, while maintaining perceptual quality during rendering.

[0031] An aspect of the present invention relates to the representation of common radiation patterns to supplement metadata for each monophonic audio object with a set of time/frequency dependent coefficients representing the directivity of the monophonic audio object projected onto a basis of N-th order (N>=1) spherical harmonics.

[0032] First-order radiation patterns can be represented by a set of 4 scalar gains for a given set of frequency bands (e.g., 1/3-octave). A set of frequency bands can also be referred to as an interval or subband. The intervals or subbands can be defined based on a windowed Fourier transform (STFT) or a perceptual filter bank for a single frame of data (e.g., 512 samples as in Dolby Atmos). The resulting pattern can be rendered by evaluating the spherical harmonic decomposition in the desired directions around the object.

[0033] In general, this radiation pattern represents a characteristic of the source and may remain constant over time. However, to represent a dynamic scene in which objects rotate or change, or to ensure that data can be accessed randomly, it may be advantageous to update this set of coefficients at regular intervals. In the context of dynamic auditory scenes with moving objects, the result of object rotation can be directly encoded in time-varying coefficients without the need for explicit separate encoding of the object's orientation.

[0034] Each type of sound source has a characteristic radiation/emission pattern, which typically differs in frequency band. For example, a violin may have a radiation pattern very different from that of a trumpet, drum, or bell. Moreover, a sound source, such as a musical instrument, may produce different emissions at performance levels, such as very quiet (pianissimo) and very loud (fortissimo) . As a result, the radiation pattern may also be a function not only of the direction around the sounding object, but also of the pressure level of the sound signal it emits, with the pressure level also being variable over time.

[0035] Accordingly, instead of simply representing the sound field at a point in space, some implementations employ encoding of audio data that correspond to the radiation patterns of audio objects so that they can be rendered from a different vantage point. In some cases, the radiation patterns may represent time- and frequency-varying radiation patterns. The audio input to the encoding process may, in some cases, include multiple channels (e.g., 4, 6, 8, 20, or more channels) of audio data from directional microphones. Each channel may correspond to data from a microphone at a specific location in space around the sound source, from which the radiation pattern can be derived. Assuming that the relative position from each microphone to the source is known, this can be achieved by numerically fitting a set of spherical harmonic coefficients such that the resulting spherical function best matches the detected energy levels in different subranges of each microphone input signal. For example, see the methods and systems described in connection with patent application No. PCT/US2017/053946, "Method, Systems, and Apparatus for Determining Audio Representations," by Nicolas Tsingos and Pradeep Kumar Govindaraju, which is incorporated herein by reference. In other examples, the radiation pattern of an audio object can be determined through numerical simulation.

[0036] Instead of simply encoding audio data from directional microphones at the sample level, some implementations include encoding monophonic signals of audio objects via corresponding radiation pattern metadata that represents radiation patterns for at least some of the encoded audio objects. In some implementations, the radiation pattern metadata may be represented as spherical harmonic data. Some such implementations may include a smoothing process and/or a data compression/reduction process.

[0037] Fig. 1A is a block diagram showing blocks of a method for encoding audio according to one example. Method 1 may, for example, be implemented by a control system (such as control system 815, which is described below with reference to Fig. 8) that includes one or more processors and one or more read-only memories. As with other described methods, not all blocks of method 1 are necessarily performed in the order shown in Fig. 1A. Furthermore, alternative methods may include more or fewer blocks.

[0038] In this example, block 5 includes receiving a monophonic audio signal corresponding to an audio object, and also receiving a representation of an emission pattern corresponding to the audio object. According to the present implementation, the emission pattern comprises audio levels corresponding to multiple sampling periods, multiple frequency bands, and multiple directions. According to the present example, block 10 includes encoding the monophonic audio signal.

[0039] In the example shown in Fig. 1A, block 15 includes encoding a source radiation pattern to determine radiation pattern metadata. According to the present implementation, encoding a representation of the radiation pattern includes determining a spherical harmonic transform of the representation of the radiation pattern and compressing the spherical harmonic transform to obtain encoded radiation pattern metadata. In some implementations, the representation of the radiation pattern may be scaled to the amplitude of the input radiation pattern in the frequency direction to determine a normalized radiation pattern.

[0040] In some cases, spherical harmonic transform compression may involve discarding some higher-order spherical harmonic coefficients. Some such examples may include discarding spherical harmonic transform coefficients that are above a threshold order of spherical harmonic coefficients, such as above the 3rd order, above the 4th order, above the 5th order, etc.

[0041] However, some implementations may include alternative and/or additional compression methods. According to some such implementations, spherical harmonic transform compression may include a singular value decomposition method, principal component analysis, discrete cosine transforms, data-independent bases, and/or other methods.

[0042] According to some examples, method 1 may also include encoding multiple directional sound objects as a group or "cluster" of sound objects. Some implementations may include encoding group metadata related to the radiation pattern of the group of directional sound objects. In some cases, multiple directional sound objects may be encoded as a single directional sound object, the directivity of which corresponds to a time-varying energy-weighted average of the spherical harmonic coefficients of each sound object. In some such examples, the encoded radiation pattern metadata may represent a centroid that corresponds to the average sound level for each frequency band. For example, the encoded radiation pattern metadata (or associated metadata) may indicate the position of a cluster of sound objects, that is, the average position of each directional sound object in the cluster.

[0043] Fig. 1B shows blocks of a process that may be implemented by a coding system 100 for dynamically coding frame-by-frame directionality information for a directional audio object according to one example. The process may, for example, be implemented by a control system, such as a control system 815, which is described below with reference to Fig. 8. The coding system 100 may receive a monophonic audio signal 101, which may correspond to a monophonic object signal, as described above. The monophonic audio signal 101 may be encoded in block 111 and provided to a serialization block 112.

[0044] Block 102 may process static or time-varying directional energy samples at different sound levels in a set of frequency bands relative to a reference coordinate system. The reference coordinate system may be defined in a specific spatial coordinate system, such as a model spatial coordinate system or a world spatial coordinate system.

[0045] In block 105, frequency-dependent rescaling of the time-varying directional energy samples from block 102 may be performed. In one example, frequency-dependent rescaling may be performed in accordance with the example illustrated in Figs. 2A-2C, as described below. The normalization may be based on rescaling of amplitude, such as the direction of a high frequency relative to a low frequency.

[0046] The frequency-dependent rescaling may be re-normalized based on the assumed direction of acquisition of the base signal. Such assumed direction of acquisition of the base signal may represent the listening direction relative to the sound source. For example, this listening direction may be referred to as the viewing direction, where the viewing direction may be in a specific direction relative to the coordinate system (e.g., the forward direction or the rearward direction).

[0047] In block 106, the rescaled directivity output from block 105 may be projected onto a spherical harmonic basis, resulting in spherical harmonic coefficients.

[0048] In block 108, the spherical coefficients from block 106 are processed based on the instantaneous value of the sound level 107 and/or information from the rotation block 109. The instantaneous value of the sound level 107 may be measured at a specific time in a specific direction. The information from the rotation block 109 may indicate an (optional) rotation of the time-varying orientation 103 of the source. For example, in block 109, the spherical coefficients may be adjusted to take into account a time-dependent modification of the orientation of the source relative to the originally recorded input data.

[0049] Block 108 may further perform a target level determination based on an alignment that is determined relative to the direction of the intended capture direction of the base audio signal. Block 108 may output a set of rotated spherical coefficients that are aligned based on the target level determination.

[0050] In block 110, encoding of the radiation scheme may be based on projection onto a smaller subspace of spherical coefficients relative to the radiation scheme of the source, which provides metadata of the encoded radiation scheme. As shown in Fig. 1A, in block 110, an SVD decomposition and compression algorithm may be performed on the output data of the spherical coefficients by block 108. In one example, the SVD decomposition and compression algorithm of block 110 may be performed in accordance with the principles described in connection with equations 11-13, which are described below.

[0051] Alternatively, block 110 may include the option of using other methods, such as principal component analysis (PCA) and/or data-independent bases, such as 2D discrete cosine transform (DCT), to project the spherical harmonic representation. into space, resulting in lossy compression. The output data from block 110 may be a matrix T that represents a projection of the data into a smaller subspace of the input data, i.e., an encoded emission scheme T. The encoded emission scheme T, the encoded monophonic base audio signal 111, and any other object metadata 104 (e.g., x, y, z, optional source orientation, etc.) may be serialized in serialization block 112 to output an encoded bitstream. In some examples, the emission scheme may be represented by the following bitstream syntax structure in each encoded audio frame:

Byte freqBandModePreset (e.g. broadband, octave, wideband, 1/3-octave, normal).

It determines the number N and the values of the average frequency of the sub-ranges)

Byte order (order N spherical harmonics)

Int * coefficients ((N+1) *(N+1) * K values)

[0052] Such syntax may encompass different sets of coefficients for different sound source pressure/intensity levels. Alternatively, if directivity information is available at different signal levels and if the source level can no longer be determined during playback, a single set of coefficients may be dynamically generated. For example, such coefficients may be generated by interpolating between low-level coefficients and high-level coefficients based on the time-varying sound signal level of the object during encoding.

[0053] The input radiation pattern relative to the monophonic signal of an audio object may also be "normalized" with respect to a given direction, such as the primary response axis (which may be the direction from which it is recorded or the average of several recordings) and an encoded directivity, and the final rendering may need to match this "normalization." In one example, this normalization may be specified as metadata. Typically, encoding the underlying audio signal is desirable, which conveys a good representation of the timbre of the object if no directivity information is used.

Directional coding

[0054] An aspect of the present invention relates to the implementation of efficient coding schemes for directionality information, since the number of coefficients grows quadratically with the decomposition order. Efficient coding schemes for directionality information can be implemented for the final delivery of an auditory scene emission, for example over a network with limited bandwidth, to a final rendering device.

[0055] Assuming 16 bits are used to represent each coefficient, representing 4th-order spherical harmonics in 1/3-octave bands would require 25*31 ~= 12 kbits per frame. Updating this information at 30 Hz would require an information rate of at least 400 kbits/s, more than current object-based audio codecs currently require to transmit both audio and object metadata. In one example, the emission scheme could be represented as follows:

equation no. (1)

[0056] In equation No. (1) represents a discrete meridional angle and azimuth angle relative to the sound source, represents the total number of discrete angles and represents the spectral frequency. Figs. 2A and 2B show the radiation patterns of a sound object in two different frequency bands. In Fig. 2A, one can see, for example, the radiation pattern of a sound object in the frequency band from 100 to 300 Hz, while in Fig. 2B, one can see, for example, the radiation pattern of the same sound object in the frequency band from 1 kHz to 2 kHz. Low frequencies, as a rule, are relatively more omnidirectional, so the radiation pattern shown in Fig. 2A is relatively more circular than the radiation pattern shown in Fig. 2B. In Fig. 2A represents the radiation pattern in the direction of the main axis 200 of the response, whereas represents a radiation pattern in an arbitrary direction 205.

[0057] In some examples, the radiation pattern may be captured and determined by a plurality of microphones physically located around a sound source corresponding to a sound object, while in other examples, the radiation pattern may be determined by numerical modeling. In the example with a plurality of microphones, the radiation pattern may be time-varying, reflecting, for example, a live recording. The radiation pattern may be captured at various frequencies, including low (e.g., <100 Hz), mid (100 Hz < and >1 kHz), and high (>10 kHz) frequencies. The radiation pattern may also be referred to as a spatial representation.

[0058] In another example, the radiation pattern may reflect a normalization based on a captured radiation pattern at a particular frequency in a particular direction , such as:

equation no. (2)

[0059] In equation No. (2) represents the radiation pattern in the direction of the main response axis. Referring again to Fig. 2B, in one example one can see the radiation pattern and a normalized radiation scheme Fig. 2C is a graph showing examples of normalized and unnormalized radiation patterns according to one example. In this example, the normalized radiation pattern in the direction of the main response axis, which is represented as in Fig. 2C, has essentially the same amplitude within the illustrated frequency bands. In this example, the normalized radiation pattern in the 205 direction (shown in Fig. 2A), which is represented as in Fig. 2C, has relatively higher amplitudes with higher frequencies than the unnormalized radiation pattern, which is represented as in Fig. 2C. For a given frequency band, it can be assumed that the radiation pattern may be constant for convenience of notation, but in practice it may change over time, for example, for different bowing techniques used on stringed instruments.

[0060] A radiation pattern, or a parametric representation thereof, may be transmitted. The radiation pattern may be pre-processed prior to transmission. In one example, the radiation pattern or parametric representation may be pre-processed using a computational algorithm, examples of which are shown in Fig. 1A. After pre-processing, the radiation pattern may be decomposed into an orthogonal spherical basis based on, for example, the following:

equation no. (3)

[0061] In equation No. (3) represents spatial representation and represents a spherical harmonic representation that has fewer elements than the spatial representation. The conversion between And can be based on the use of, for example, real fully normalized spherical harmonics:

equation no. (4)

[0062] In equation No. (4) represents the associated Legendre polynomials, order , degree And

equation no. (5)

[0063] Other spherical bases may also be used. Any approach to performing the spherical harmonic transform of discrete data may be used. In one example, the least squares method may be used by first defining the transform matrix :

equation no. (6)

thereby fulfilling the relationship between the representation of spherical harmonics and the spatial representation as

, equation No. (7)

[0064] In equation No. (7) Spherical harmonic representations and/or spatial representations can be stored for further processing.

[0065] Pseudo-inversion may be a weighted least squares solution method of the form:

equation no. (8)

[0066]Ordered solutions can also be used in cases where the distribution of spherical samples includes large amounts of missing data. Missing data may correspond to areas or directions for which no available directional samples are available (for example, due to uneven microphone coverage). In many cases, the distribution of spatial samples is fairly uniform, so the identity matrix of weighting coefficients gives acceptable results. It is also often assumed that, thus the representation of spherical harmonicscontains fewer elements than the spatial representation, which provides the first stage of lossy compression, within which the radiation patterns are smoothed.

[0067] Discrete frequency bands will be discussed next. Matrix can be arranged in a column, so that each frequency band is represented by a column of the matrix

. equation No. (9)

[0068] That is, a spatial representation can be defined based on intervals/bands/sets of frequencies. Therefore, the representation of spherical harmonics can be based on the following:

equation no. (10)

[0069] In equation No. (10) represents the radiation pattern for all discrete frequencies in the spherical harmonic region. It is expected that the adjacent columns are highly correlated, leading to redundancy in the representation. Some implementations include further decomposition by factorizing the matrix in the form

equation No. (11)

[0070]Some embodiments may include performing a singular value decomposition (SVD), where And represent the left and right singular matrices andrepresents a matrix of singular values decreasing along their diagonal. They can accept or store information about the matrix V. Alternatively, principal component analysis (PCA) and data-independent bases such as 2D DCT can be used for projectioninto space, resulting in lossy compression.

[0071] Let's say In some examples, to achieve compression, the encoder may discard components corresponding to smaller singular values by computing the result based on the following:

equation no. (12)

[0072] In equation No. (12) represents a truncated copy The matrix T can represent a projection of the data into a smaller subspace of the input data. T represents the data of the encoded transmission scheme, which is then transmitted for further processing. On the encoding and reception side, in some examples, the matrix T can also be adopted as a low-rank approximation for can be recreated on the basis of:

equation No. (13)

In equation No. (13) represents a truncated copy The V matrix can be stored on the decoder side or transmitted to it.

[0073] Three examples of transmitting truncated decomposition and truncated right singular vectors are given below:

1. The transmitter can transmit coded radiation and truncated right-hand singular vectors for each object independently.

2. Objects can be grouped, for example, by degree of similarity, and And can be calculated as representative bases for a set of objects. Coded radiation , therefore, can be passed for each object, and And can be transferred to a group of objects.

3. Left and right singular matrices And can be pre-computed using a large database of representative data (e.g. training data), and information related to , can be stored on the receiver side. In some such examples, only coded radiation can be transmitted to each object. The DCT is another example of a basis that can be stored at the receiver end.

Spatial coding of directional objects

[0074] When a complex auditory scene containing many objects is encoded and transmitted, spatial coding techniques can be applied where individual objects are replaced by a smaller number of representative clusters in a way that best preserves the auditory perception of the scene. In general, replacing a group of sound sources with a representative "centroid" requires computing an aggregate/average value for each metadata field. For example, the position of a cluster of sound sources may be the average position of each source. By representing the radiation pattern of each source using a spherical harmonic decomposition as described above (e.g., with reference to Equations #1-12), it is possible to linearly combine a set of coefficients in each subband for each source to construct an average radiation pattern for the cluster of sources. By computing the loudness or energy-weighted average of the spherical harmonic coefficients over time, it is possible to construct a perceptually optimized, time-varying representation that better preserves the original scene.

[0075] Fig. 1C shows blocks of a process that may be implemented by a decoding system according to one example. The blocks shown in Fig. 1C may, for example, be implemented by a control system of a decoding device (such as control system 815, which is described below with reference to Fig. 8), which includes one or more processors and one or more read-only memories. In block 150, metadata and an encoded monophonic base audio signal may be received and deserialized. The deserialized information may comprise object metadata 151, an encoded base audio signal, and encoded spherical coefficients. In block 152, the encoded base audio signal may be decoded. In block 153, the encoded spherical coefficients may be decoded. The information about the encoded radiation scheme may comprise an encoded radiation scheme T and/or a matrix V . The matrix V will depend on the method used for projection in space. If the SVD algorithm is used in block 110 in Fig. 1B, the matrix V may be received or stored by the decoding system.

[0076] The object metadata 151 may contain information about the relative direction from the source to the listener. In one example, the metadata 151 may contain information about the distance and direction of the listener and the distance and direction of one or more objects relative to the 6DoF space. For example, the metadata 151 may contain information related to the relative rotation, distance, and direction of the source in the 6DoF space. In an example with multiple objects in clusters, the metadata field may display information related to a representative "centroid" that reflects the aggregate/average value of the cluster of objects.

[0077] Rendering module 154 may then render the decoded base audio signal and the decoded spherical harmonic coefficients. In one example, rendering module 154 may render the decoded base audio signal and the decoded spherical harmonic coefficients based on object metadata 151. Rendering module 154 may determine subband gains for the spherical coefficients of the radiation pattern based on information from metadata 151, such as relative directions from the source to the listener. Rendering module 154 may then render the base signals of the audio object based on the determined subband gains of the corresponding decoded radiation pattern(s), information 155 about the position of the source and/or listener (e.g., x, y, z, yaw, pitch, roll). The information about the position of the listener may correspond to the user's position and viewing direction in 6DoF space. Listener position information can be received from a source located near the VR playback system, such as an optical tracking device. This information corresponds to the position of the sound object and its spatial orientation. It can also be derived from a local tracking system, for example, if the user's hands are tracked and the virtual sound object is interactively manipulated, or if a tracked physical property/intermediate object is used.

[0078] Fig. 3 shows an example of a hierarchy in which audio data and various types of metadata are included. As in other figures presented herein, the reference numbers and types of audio data and metadata shown in Fig. 3 are presented only as an example. Some encoders may provide the full set of audio data and metadata shown in Fig. 3 (data set 345), while other encoders may provide only the location of the metadata shown in Fig. 3, for example, only data set 315, only data set 325, or only data set 335.

[0079] In this example, the audio data comprises a monophonic audio signal 301. The monophonic audio signal 301 represents one example of what is sometimes referred to herein as a "base audio signal." However, in some examples, the base audio signal may comprise audio signals corresponding to multiple audio objects that are included in the cluster.

[0080] In this example, the audio object position metadata 305 is expressed as Cartesian coordinates. However, in some alternative examples, the audio object position metadata 305 may be expressed using coordinates of other types, such as spherical or polar coordinates. Accordingly, the audio object position metadata 305 may include three degrees of freedom (3 DoF) position information. According to this example, the audio object metadata comprises audio object size metadata 310. In alternative examples, the audio object metadata may comprise audio object metadata of one or more other types.

[0081] In this implementation, the data set 315 includes a monophonic audio signal 301, audio object position metadata 305, and audio object size metadata 310. The data set 315 may, for example, be provided in the Dolby Atmos™ audio data format.

[0082] In this example, the data set 315 also includes an optional rendering parameter R. According to some disclosed implementations, the optional rendering parameter R may indicate whether at least some metadata of the audio object from the data set 315 should be interpreted in its "normal" sense (e.g., position or size metadata) or as directionality metadata. In some disclosed implementations, the "normal" mode may be referred to herein as a "position mode," and the alternative mode may be referred to herein as a "directivity mode." Some examples are described below with reference to FIGS. 5A-6.

[0083] According to this example, orientation metadata 320 contains angular coordinate information for expressing the yaw, pitch, and roll of the audio object. In this example, orientation metadata 320 denotes yaw, pitch, and roll as φ, ϴ, and ψ. Data set 325 contains sufficient information for the orientation of the audio object for six degrees of freedom (6 DoF) applications.

[0084] In this example, the data set 335 contains audio object type metadata 330. In some implementations, the audio object type metadata 330 may be used to indicate corresponding emission pattern metadata. The encoded emission pattern metadata may be used (e.g., by a decoder or a device that receives audio data from the decoder) to determine the decoded emission pattern. In some examples, the audio object type metadata 330 may indicate, essentially, the following: "I am a trumpet,""I am a violin," and the like. In some examples, the decoding device may access a database of audio object types and corresponding beam patterns. According to some examples, the database may be provided together with the encoded audio data or before the audio data is transmitted. Such audio object type metadata 330 may be referred to herein as "database beam pattern data."

[0085] According to some examples, the audio object type metadata may indicate parametric data of a polar pattern. In some examples, the audio object type metadata 330 may indicate a polar pattern corresponding to a cosine function of a specified degree, may indicate a cardioid function, etc.

[0086] In some examples, the audio object type metadata 330 may indicate that the radiation pattern corresponds to a set of spherical harmonic coefficients. For example, the audio object type metadata 330 may indicate that the spherical harmonic coefficients 340 are provided in the data set 345. In some such examples, the spherical harmonic coefficients 340 may represent a time- and/or frequency-varying set of spherical harmonic coefficients, such as described above. Such information may require the largest amount of data compared to the rest of the metadata hierarchy shown in Fig. 3. Therefore, in some such examples, the spherical harmonic coefficients 340 may be provided separately from the monophonic audio signal 301 and the corresponding audio object metadata. For example, 340 spherical harmonic coefficients may be provided at the beginning of the audio data transfer before initiating real-time operations (e.g., real-time rendering operations for a game, movie, musical performance, etc.).

[0087] According to some implementations, a device on the decoder side, such as a device that provides audio to a playback system, may determine the capabilities of the playback system and provide directionality information in accordance with these capabilities. For example, even if the entire data set 345 is provided to the decoder, in some such implementations, only the usable portion of the directionality information may be provided to the playback system. In some examples, the decoding device may determine which type(s) of directionality information to use in accordance with the capabilities of the decoding device.

[0088] Fig. 4 is a block diagram showing blocks of a method for decoding audio according to one example. The method 400 may, for example, be implemented by a control system of a decoding device (such as the control system 815, which is described below with reference to Fig. 8), which includes one or more processors and one or more read-only memories. As with other described methods, not all blocks of the method 400 are necessarily performed in the order shown in Fig. 4. Moreover, alternative methods may include more or fewer blocks.

[0089] In this example, block 405 includes receiving an encoded base audio signal, encoded emission scheme metadata, and encoded audio object metadata. The encoded emission scheme metadata may include metadata of audio object types. The encoded base audio signal may, for example, include a monophonic audio signal. In some examples, the audio object metadata may include 3DoF position information, 6DoF position information and source orientation information, audio object size metadata, etc. The audio object metadata may, in some cases, be time-varying.

[0090] In this example, block 410 includes decoding the encoded base audio signal to determine the base audio signal. In this document, block 415 includes decoding the encoded emission scheme metadata to determine the decoded emission scheme. In this example, block 420 includes decoding at least some of the other metadata of the encoded audio object. In this document, block 430 includes rendering the base audio signal based on the audio object metadata (e.g., position, orientation, and/or size metadata of the audio object) and the decoded emission scheme.

[0091] Block 415 may include various types of operations depending on the particular implementation. In some cases, the audio object type metadata may point to the data of a database radiation pattern. Decoding the encoded radiation pattern metadata to determine the decoded radiation pattern may include querying a directionality data structure that contains audio object types and corresponding radiation pattern data. In some examples, the audio object type metadata may point to parametric radiation pattern data, such as radiation pattern data corresponding to a cosine function, a sine function, or a cardioid function.

[0092] According to some implementations, the audio object type metadata may indicate dynamic beamforming data, such as a time- and/or frequency-varying set of spherical harmonic coefficients. Some such implementations may include receiving the dynamic beamforming data prior to receiving the encoded base audio signal.

[0093] In some cases, the base audio signal received in block 405 may contain audio signals corresponding to multiple audio objects contained in a cluster. According to some such examples, the base audio signal may be based on a cluster of audio objects, which may contain multiple directional audio objects. The decoded radiation scheme determined in block 415 may correspond to the centroid of the cluster and may represent an average value for each frequency band of each of the multiple directional audio objects. The rendering process of block 430 may include applying subband gains, at least partially based on the decoded radiation data, to the decoded base audio signal. In some examples, after decoding and applying directionality processing to the base audio signal, the signal may be further virtualized to its intended position relative to the listener's position using audio object position metadata and known rendering processes such as binaural rendering via headphones, rendering using the playback environment's speakers, etc.

[0094] As discussed above with reference to Fig. 3, in some implementations, the audio data may be accompanied by a rendering parameter (shown as R in Fig. 3). The rendering parameter may indicate whether at least some audio object data, such as Dolby Atmos metadata, should be interpreted in the normal manner (e.g., as position or size metadata) or as directionality metadata. The normal mode may be referred to as a "position mode," and the alternative mode may be referred to herein as a "directivity mode." Accordingly, in some examples, the rendering parameter may indicate whether at least some audio object metadata is interpreted as directionality metadata relative to the speaker, or as a position relative to a room or other playback environment. Such implementations may be particularly useful for rendering directionality using intelligent speakers with multiple drivers, such as described below.

[0095] Fig. 5A shows a drum cymbal. In this example, the drum cymbal 505 is shown to emit sound with a directional pattern 510 that has a substantially vertical primary response axis 515. The directional pattern 510 itself is also predominantly vertical with some degree of spread relative to the primary response axis 515.

[0096] Fig. 5B shows an example of a speaker system. In this example, the speaker system 525 comprises multiple speakers/transducers configured to emit sound in various directions, including an upward direction. The speaker located at the very top may, for example, be used in a conventional Dolby Atmos manner ("position mode") for position rendering, such as to provide sound reflection from the ceiling to simulate overhead/ceiling speakers (z=1). In some such cases, the corresponding Dolby Atmos rendering may include additional virtualization processing that enhances the perception of a sound object having a certain position.

[0097] In other use cases, the same upward-firing speaker(s) may be operated in a "directional mode," such as to simulate the directional pattern of a drum, symbols, or other audio object having a directional pattern similar to the directional pattern 510 shown in Fig. 5A. Some speaker systems 525 may be capable of beamforming, which may assist in constructing the desired directional pattern. In some examples, virtualization processing is not performed to reduce the perception of an audio object having a particular position.

[0098] Fig. 6 is a block diagram illustrating blocks of a method for decoding audio according to one example. Method 600 may, for example, be implemented by a control system of a decoding device (such as control system 815, which is described below with reference to Fig. 8), which includes one or more processors and one or more read-only memories. As with other methods described, not all blocks of method 600 are necessarily performed in the order shown in Fig. 6. Furthermore, alternative methods may include more or fewer blocks.

[0099] In this example, block 605 includes receiving audio data corresponding to at least one audio object, wherein the audio data includes a monophonic audio signal, audio object position metadata, audio object size metadata, and a rendering parameter. In this implementation, block 605 includes receiving this data via an interface system of the decoding device (such as the interface system 810 of Fig. 8). In some cases, the audio data may be received in Dolby Atmos™ format. The audio object position metadata may correspond to world coordinates or model coordinates, depending on the specific implementation.

[00100] In this example, block 610 includes determining whether the rendering parameter indicates a position mode or a directional mode. In the example shown in Fig. 6, if it is determined that the rendering parameter indicates a directional mode, in block 615, the audio data is rendered for playback (e.g., via at least one loudspeaker, via headphones, etc.) according to the directional pattern indicated by at least one of the position metadata or the size metadata. For example, the directional pattern may be similar to the pattern shown in Fig. 5A.

[00101] In some examples, rendering audio data may include interpreting audio object position metadata as audio object orientation metadata. The audio object position metadata may include Cartesian coordinate/x,y,z coordinate data, spherical coordinate data, or cylindrical coordinate data. The audio object orientation metadata may be yaw, pitch, and roll metadata.

[00102] According to some implementations, rendering audio data may include interpreting size metadata of an audio object as directional metadata that corresponds to a directional pattern. In some such examples, rendering audio data may include requesting a data structure that contains multiple directional patterns and associating at least one of the position metadata or the size metadata with one or more directional patterns. Some such implementations may include receiving the data structure through an interface system. According to some such implementations, the data structure may be received before the audio data.

[00103] Fig. 7 shows one example of encoding a plurality of audio objects. In one example, information 701, 702, 703 about object 1-n, etc. may be encoded. In one example, a representative cluster for audio objects 701-703 may be determined in block 710. In one example, a group of audio sources may be collected and represented by a representative "centroid" that includes calculating an aggregate/average value for a metadata field. For example, the position of a cluster of audio sources may be the average position of each source. In block 720, an emission pattern of a representative cluster may be encoded. In some examples, the emission pattern for a cluster may be encoded in accordance with the principles described above with reference to Fig. 1A or Fig. 1B.

[00104] Fig. 8 is a block diagram showing examples of components of an apparatus that may be adapted to perform at least some of the methods described herein. For example, apparatus 805 may be adapted to perform one or more of the methods described above with reference to Figs. 1A-1C, Fig. 4, Fig. 6, and/or Fig. 7. In some examples, apparatus 805 may be, or may include, a personal computer, a desktop computer, or another local device configured to provide audio processing. In some examples, apparatus 805 may be, or may include, a server. According to some examples, apparatus 805 may be a client device that is configured to communicate with a server via a network interface. Components of apparatus 805 may be implemented using hardware, software stored on a persistent storage medium, firmware, and/or combinations thereof. The types and numbers of components shown in Fig. 8, as well as in other figures disclosed herein, are provided for exemplary purposes only. Alternative implementations may include more and/or fewer components and/or different components.

[00105] In this example, the apparatus 805 comprises an interface system 810 and a control system 815. The interface system 810 may comprise one or more network interfaces, one or more interfaces between the control system 815 and the memory system, and/or one or more interfaces for external devices (such as one or more universal serial bus (USB) interfaces). In some implementations, the interface system 810 may comprise a user interface system. The user interface system may be configured to receive input from a user. In some implementations, the user interface system may be configured to provide feedback to the user. For example, the user interface system may comprise one or more display devices with corresponding touch and/or gesture detection systems. In some examples, the user interface system may comprise one or more microphones and/or speakers. According to some examples, the user interface system may include a device for providing haptic feedback, such as a motor, a vibrator, etc. The control system 815 may, for example, include a general-purpose single- or multi-chip processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic device, a discrete component circuit or a transistor logic circuit, and/or discrete hardware components.

[00106] In some examples, the apparatus 805 may be implemented as a single device. However, in some implementations, the apparatus 805 may be implemented as more than one device. In some such implementations, the functionality of the control system 815 may be implemented in more than one device. In some examples, the apparatus 805 may be a component of another device.

[00107] Various exemplary embodiments of the present invention may be implemented by hardware or special-purpose circuits, software, a logic device, or any other combination thereof. Some aspects may be implemented by hardware, while other aspects may be implemented by firmware or software that may be executed by a controller, microprocessor, or other computing device. In general, it should be understood that the present invention also encompasses an apparatus suitable for performing the methods disclosed above, such as an apparatus (spatial rendering module) having a memory device and a processor connected to the memory device, wherein the processor is adapted to execute instructions and perform the methods according to embodiments of the present invention.

[00108] While various aspects of exemplary embodiments of the present invention are illustrated and described in block diagrams, flow charts, or using some other graphical representation, it should be understood that the blocks, apparatuses, systems, methods, and techniques described herein may be implemented, as non-limiting examples, by hardware, software, firmware, special-purpose circuitry or logic device, general-purpose hardware or controller, or other computing devices, or some combination thereof.

[00109] Additionally, the various blocks shown in the flow charts may be considered as method steps, and/or as operations that are results of the operation of computer program code, and/or as several related logic circuit elements designed to perform related function(s). For example, embodiments of the present invention include a computer program product comprising a computer program tangibly embodied on a computer-readable medium, in which the computer program comprising program codes is configured to perform the methods as described above.

[00110] In the context of the present invention, a computer-readable medium may be any tangible medium that can contain or store a program for use by or in connection with a system, apparatus, or device for executing instructions. A computer-readable medium may be a computer-readable signal medium or a computer-readable data medium. A computer-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the above. More specific examples of a computer-readable storage medium would include an electrical connection via one or more wires, a portable computer floppy disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc with write-once data (CD-ROM), an optical data storage device, a magnetic data storage device, or any suitable combination of the foregoing.

[00111] The computer program code for implementing the methods of the present invention may be written in any combination of one or more programming languages. These computer program codes may be provided to the processor of a general-purpose computer, a special-purpose computer, or another programmable data processing apparatus, such that the program codes, when executed by the processor of the computer or another programmable data processing apparatus, provide for the implementation of the functions/operations indicated in the block diagrams and/or structural diagrams. The program code may be executed entirely on the computer, partially on the computer as a stand-alone software package, partially on the computer and partially on a remote computer, or entirely on a remote computer or server.

[00112] Furthermore, although operations are depicted in a particular order, this should not be construed as a requirement that these operations be performed in the particular order shown, or in a sequential order, or that all of the illustrated operations be performed to achieve the desired results. In some circumstances, multitasking or parallel processing may be preferable. Similarly, while the above disclosure contains several specific implementation details, these should not be construed as limitations on the scope of any part of the present invention or any claim, but as descriptions of features that may be characteristic of particular embodiments of the present invention. Specific features described in this invention in the context of individual embodiments may also be implemented in combination in a single embodiment. Conversely, various features that are described in the context of a single embodiment may also be implemented in multiple embodiments individually or in any suitable subcombination.

[00113] It should be noted that the description and drawings illustrate only the principles of the proposed methods and apparatuses. Therefore, it should be taken into account that those skilled in the art will be able to design various circuits that, although not explicitly described or shown herein, implement the principles of the present invention and are included in its spirit and scope. In addition, all examples listed in this document are primarily expressly intended for pedagogical purposes only to assist the reader in understanding the principles of the proposed apparatuses and devices, as well as the concepts introduced by the inventors with the aim of developing this art, and they should be interpreted as taking place without being limited to the specifically listed examples and conditions. Moreover, all statements in this document listing the principles, aspects and embodiments of the present invention, as well as their specific examples, are intended to cover their equivalents.

Claims (20)

1. A method for decoding audio data, comprising: receiving an encoded base audio signal, metadata of an encoded emission scheme, and metadata of an encoded audio object, wherein the metadata of the audio object includes 6DoF source orientation information; decoding the encoded base audio signal to determine the base audio signal; decoding the metadata of the encoded emission scheme to determine the decoded emission scheme; decoding the audio object metadata; and Rendering the basic audio signal based on the audio object metadata and the decoded emission scheme. 2. The method according to claim 1, characterized in that the base audio signal contains several directional objects based on a cluster of objects, and wherein the decoded radiation pattern represents a centroid that reflects the average value for each frequency band. 3. The method according to paragraph 1, characterized in that the metadata of the coded radiation scheme correspond to a set of spherical harmonic coefficients that varies in time and frequency. 4. The method according to paragraph 1, characterized in that the metadata of the encoded emission scheme contains metadata of the type of sound object. 5. The method according to claim 4, characterized in that the metadata of the type of the sound object indicates parametric data of the directional pattern, and wherein the parametric data of the directional pattern comprises one or more functions selected from a list of functions consisting of a cosine function, a sine function, or a cardioid function. 6. The method according to claim 4, characterized in that the metadata of the type of the sound object indicate the dynamic data of the directivity pattern, and wherein the dynamic data of the directivity pattern correspond to a time- and frequency-varying set of spherical harmonic coefficients. 7. The method according to claim 6, characterized in that it further includes receiving dynamic data of the directional pattern before receiving the encoded base audio signal. 8. The method according to claim 1, characterized in that the rendering is based on applying sub-band gain factors, based at least in part on the decoded radiation scheme, to the decoded base audio signal. 9. The method of claim 4, wherein the metadata of the sound object type points to the data of the directionality pattern of the database, and wherein decoding the metadata of the encoded radiation pattern to determine the decoded radiation pattern includes requesting a directionality data structure that contains the types of the sound object and the corresponding data of the directionality pattern. 10. A non-volatile machine-readable medium with instructions stored thereon that, when executed by one or more processors, ensure that one or more processors execute the method according to paragraph 1. 11. An audio decoding apparatus comprising: interface system and a control system designed with the ability to: receiving, via an interface system, audio data corresponding to at least one audio object, wherein the audio data comprises a monophonic audio signal, audio object position metadata, audio object size metadata, and a rendering parameter, wherein the audio object position metadata comprises 6DoF source orientation information; determining whether the rendering parameter indicates a position mode or a directional mode, and upon determining that the rendering parameter indicates a directional mode, rendering the audio data for playback by at least one loudspeaker in accordance with a directional pattern indicated by at least one of the audio object position metadata or the audio object size metadata.