HK1233037B - Residual encoding in an object-based audio system - Google Patents

Description

Residual Coding in Object-Based Audio Systems

Related applications

This application claims priority to U.S. Provisional Patent Application No. 61/968,111, filed on March 20, 2014, and entitled “Residual Coding in Object-Based Audio Systems,” and U.S. Non-Provisional Patent Application No. 14/620,544, filed on February 12, 2015, and entitled “Residual Coding in Object-Based Audio Systems.”

Technical Field

The present invention generally relates to lossy, multi-channel audio compression and decompression, and more particularly to compressing and decompressing a downmixed multi-channel audio signal in a manner that facilitates upmixing a received decompressed multi-channel audio signal.

Background Art

Audio and audiovisual entertainment systems have evolved from humble beginnings, capable of reproducing monophonic audio through a single speaker. Modern surround sound systems are capable of recording, transmitting, and reproducing multiple channels through multiple speakers in a listener's environment (which can be a public theater or a more private "home theater"). Various surround sound speaker setups are available: these follow names such as "5.1 surround," "7.1 surround," and even 20.2 surround (where the number to the right of the decimal indicates the low-frequency effects channel). For each such configuration, various physical arrangements of speakers are possible; but generally, the best results will be achieved if the rendering geometry resembles the geometry assumed by the audio engineer who mixed and mastered the recorded channels.

Because a variety of rendering environments and geometries beyond those predicted by the mixing engineer are possible, and because the same content can be played back in a variety of listening configurations or environments, the diversity of surround sound configurations presents numerous challenges to engineers or artists who wish to present a faithful listening experience. "Channel-based" or (more recently) "object-based" approaches can be used to present a surround sound listening experience.

In a channel-based approach, each channel is recorded with the intention that it will be rendered during playback on the corresponding loudspeaker. During mixing, the physical setup of the intended loudspeakers is predetermined or at least approximately assumed. In contrast, in an object-based approach, multiple independent audio objects are recorded, stored, and transmitted separately, preserving their synchronization relationship but independent of any assumptions about the configuration or geometry of the intended playback loudspeakers or environment. Examples of audio objects would be individual instruments, ensemble parts such as a viola section perceived as a unified musical note, vocals, or sound effects. To preserve spatial relationships, the digital data representing the audio objects includes, for each object, certain data ("metadata") representing information associated with a particular sound source: for example, vector direction, approximation, loudness, motion, and range of the sound source can be symbolically encoded (preferably in a time-varying manner), and this information is transmitted or recorded along with the particular sound signal. The combination of the individual sound source waveforms and the associated metadata together comprises an audio object (stored as an audio object file). This approach has the advantage that it can be flexibly rendered in many different configurations; however, the burden is placed on the rendering processor ("engine") to calculate the appropriate mix based on the geometry and configuration of the playback speakers and environment.

In both channel-based and object-based approaches to audio, it is frequently desirable to send a downmix of the signal (A plus B) in such a way that the two independent channels (or objects, A and B) can be separated ("upmixed") during playback. One motivation for sending a downmix may be to maintain backward compatibility, so that the downmixed program can be played in mono, traditional two-channel stereo, or (more generally) on a system with fewer speakers than the number of channels or objects in the recorded program. In order to recover a higher diversity of channels or objects, an upmixing process is applied. For example, if one sends the sum C of signals A and B: (A+B), and if one also sends B, then the receiver can easily construct A: (A+B-B)=A. Alternatively, one can send the composite signals (A+B) and (A-B), and then recover A and B by taking a linear combination of the transmitted composite signals. Many existing systems use variations of this "matrix mixing" approach. These systems are quite successful in recovering discrete channels or objects. However, when a large number of channels, or in particular objects, are summed, it becomes difficult to adequately reproduce the individual discrete objects or channels without artifacts or unrealistically high bandwidth requirements. Because object-based audio often involves a very large number of independent audio objects, there are significant difficulties in efficiently upmixing to recover the discrete objects from the downmixed signal, especially where the data rate (or more generally, bandwidth) is constrained.

In most practical systems for the transmission or recording of digital audio, some method of data compression would be highly desirable. Data rates are always subject to some constraints, and it is always desirable to transmit audio more efficiently. This consideration becomes increasingly important when using a large number of channels - either as discrete channels or being upmixed. In this application, the term "compression" refers to a method of reducing the data requirements for transmitting or recording an audio signal, whether the result is a reduction in data rate or a reduction in file size. (This definition should not be confused with dynamic range compression, which is sometimes also called "compression" in other audio contexts not relevant here).

Existing methods for compressing downmix signals generally adopt one of the following two approaches: lossless coding or redundant description. Either of these two approaches can help with upmixing after decompression, but both have drawbacks.

Lossless and lossy encoding:

Assume that A, _B1 , _B2 , ..., _Bm are independent signals (objects) that are encoded in the bitstream and sent to the renderer. The distinguished object A will be called the base object, and B = _B1 , _B2 , ..., _Bm will be called the normal object. In object-based audio systems, we are interested in rendering objects simultaneously but independently, so that, for example, each object can be rendered at a different spatial location.

Backward compatibility is desirable: in other words, we need the coded stream to be interpretable by legacy systems that are neither object-based nor object-aware, or that can handle fewer channels. Such systems can only render composite objects or channels C = A + B ₁ + B ₂ + ... + B _m from the encoded (compressed) version E(C) of C. Therefore, we need the codestream to include E(C) being sent, followed by descriptions of individual objects, which are ignored by legacy systems. Thus, the codestream can include E(C) followed by descriptions of regular objects E(B ₁ ), E(B ₂ ), ..., E(B _m ). The base object A is then recovered by decoding these descriptions and setting A = CB ₁ - B ₂ - ... - B _m . However, it should be noted that most audio codecs used in practice are lossy, meaning that the decoded version Q(X) = D(E(X)) of an encoded object E(X) is only an approximation of X and is not necessarily identical to it. The accuracy of the approximation usually depends on the choice of codec and on the bandwidth (or storage space) available for the bitstream. Although lossless coding is possible, i.e., Q(X) = X, it usually requires much greater bandwidth or storage space than lossy coding. On the other hand, the latter can still provide a high-quality reproduction that is perceptually indistinguishable from the original object.

Redundant description:

An alternative approach is to include explicit encoding of some privileged object A in the codestream, which would then consist of E(C), E(A), E( _B1 ), E( _B2 ), ..., E( _Bm ). Assuming E is lossy, this approach may be more economical than using lossless encoding, but it is still not a very efficient use of bandwidth. The approach is redundant because E(C) is clearly related to the separately encoded objects E(A), E( _B1 ), E( _B2 ), ..., E( _Bm ).

Summary of the Invention

The lossy compression and transmission of a downmix composite signal (including the downmixed signal) having multiple tracks and objects is accomplished in a manner that reduces bit rate requirements while reducing upmix artifacts compared to redundant transmission or lossless compression. A compressed residual signal is generated and transmitted together with the compressed overall mix and at least one compressed audio object. In terms of reception and upmixing, the present invention decompresses the downmixed signal and other compressed objects, calculates an approximate upmixed signal, and corrects a specific base signal derived from the upmix by subtracting the decompressed residual signal. The present invention thus allows lossy compression to be combined with the downmixed audio signal for transmission over a communication channel (or for storage). Upon subsequent reception and upmixing, the additional base signal is recoverable in capable systems that provide multi-object performance (while older systems can easily decode the overall mix without upmixing). The method and apparatus of the present invention have two aspects: a) an audio compression and downmixing aspect, and b) an audio decompression/upmixing aspect, where compression should be understood to mean a method of bitrate reduction (or file size reduction), while downmixing means a reduction in channel or object count, and upmixing means an increase in channel count by restoring and separating previously downmixed channels or objects.

In terms of decompression and upmixing of the present invention, the present invention includes a method for decompressing and upmixing a compressed downmix composite audio signal. The method includes the following steps: receiving a compressed representation of a total mixed signal C, a set of compressed representations of a group of corresponding object signals {Bi} (the group having at least one member) and a compressed representation of a residual signal Δ; decompressing the compressed representation of the total mixed signal C, decompressing the compressed representation of the residual signal Δ and the group of object signals {Bi} to obtain a corresponding approximate total mixed signal C’, a set of approximate object signals {Bi’} and a reconstructed residual signal Δ’; subtractively mixing the approximate total mixed signal C’ and the entire group of approximate object signals {Bi’} to obtain an approximation R’ of the base signal R; and subtractively mixing the reconstructed residual signal Δ’ with an approximation R’ of the reference signal R to produce a corrected base signal A". In a preferred embodiment, at least one of the compressed representations of at least one Bi and the compressed representation of C is prepared by a lossy compression method.

In terms of compression and downmixing of the present invention, the present invention includes a method for compressing a composite audio signal, which composite audio signal includes a total mixed signal C, a group of at least one object signal {Bi} (the group has at least one member Bi) and a base signal A, wherein the total mixed signal C includes the base signal A mixed with the group of at least one object signal {Bi} according to the following steps: compressing the total mixed signal C and the group of at least one object signal {Bi} by a lossy compression method to respectively produce a compressed total mixed signal E(C) and a group of compressed object signals E({Bi}); decompressing the compressed total mixed signal E(C) and the group of compressed object signals E({Bi}) to obtain a reconstructed Q(C) and a group of reconstructed object signals Q({Bi}); subtractively mixing the reconstructed signal Q(C) and the entire group of object signals Q({Bi}) to produce an approximate base signal Q’(A); and subtracting a reference signal from the approximate base signal to produce a residual signal Δ, and then compressing the residual signal Δ to obtain a compressed residual signal Ec(Δ). The compressed total mixed signal E(C), the set of (at least one) compressed object signals E({Bi}) and the compressed residual signal Ec(Δ) are preferably transmitted (or equivalently, stored or recorded).

In one embodiment of the compression and downmixing aspects, the reference signal comprises a base mix signal A. In an alternative embodiment, the reference signal is an approximation of the base signal A obtained by compressing the base signal A using a lossy method to form a compressed signal E(A), and then decompressing the compressed signal E(A) to obtain the reference signal (which is an approximation of the base signal A).

This summary is provided to introduce a selection of concepts in a simplified form that are further described below in the detailed description. This summary is neither intended to identify key features or essential features of the claimed subject matter nor is it intended to be used to limit the scope of the claims. As used in this application, unless the context clearly requires otherwise, the term "group" is used to refer to a group having at least one member, but not necessarily having multiple members. This concept is commonly used in mathematical contexts and should not lead to ambiguity. These and other features and advantages of the present invention will be apparent to those skilled in the art from the following detailed description of the preferred embodiments taken in conjunction with the accompanying drawings, in which:

BRIEF DESCRIPTION OF THE DRAWINGS

FIG1 is a high-level block diagram depicting a generalized system known in the art for compressing and transmitting a composite signal including mixed audio signals in a backward-compatible manner;

2 is a flow chart showing the steps of a method for compressing a composite audio signal according to a first embodiment of the present invention;

3 is a flow chart illustrating the steps of a method for decompressing and upmixing an audio signal according to the decompression aspect of the present invention;

4 is a flow chart illustrating the steps of a method for compressing a composite audio signal according to an alternative embodiment of the present invention;

FIG5 is a schematic block diagram of an apparatus for compressing a composite audio signal in accordance with the method of FIG2 according to an alternative embodiment of the present invention; and

FIG. 6 is a schematic block diagram of an apparatus for compressing a composite audio signal in accordance with the method of FIG. 4 according to a first embodiment of the present invention.

DETAILED DESCRIPTION

The methods described herein relate to processing signals, and are particularly directed to processing audio signals that represent physical sounds. These signals can be represented by digital electronic signals. In this discussion, continuous mathematical formulas may be shown or discussed to illustrate the concepts; however, it should be understood that some embodiments operate in the context of a time series of digital bytes or words that form a discrete approximation to an analog signal or (ultimately) a physical sound. This discrete digital signal corresponds to a digital representation of a periodically sampled audio waveform. In an embodiment, a sampling rate of approximately 48,000 samples per second can be used. A higher sampling rate such as 96 kHz can be used alternatively. The quantization scheme and bit resolution can be selected to meet the needs of a specific application. The techniques and devices described herein can be applied interdependently in several channels. For example, they can be used in the context of a surround audio system with more than two channels.

As used herein, "digital audio signal" or "audio signal" does not describe a simple mathematical abstraction, but rather, in addition to its ordinary meaning, refers to information embodied in or carried by a non-transitory physical medium capable of being detected by a machine or device. This term includes recorded or transmitted signals and should be understood to include transmission using any form of encoding, including but not limited to pulse code modulation (PCM). The output or input may be encoded or compressed using any of a variety of known methods, including MPEG, ATRAC, AC3, or the proprietary method of DTS Corporation described in U.S. Patents 5,974,380, 5,978,762, and 6,487,535. Some modifications may be performed on the calculations to accommodate that particular compression or encoding method.

Overview

Figure 1 shows a general environment in which the present invention operates at a high level. As in the prior art, an encoder 110 receives a plurality of independent audio signals, arbitrarily referred to as A, B, downmixes the signals into a total mixed signal C (=A+B) using a mixer 120, compresses the downmixed signal using a compressor 130, and then transmits (or records) the downmixed signal in a manner that will allow a reasonable approximation of the signal to be reconstructed at a decoder 160. Although only signal B is shown in the figure (for simplicity), the present invention can be used for multiple independent signals or objects B1, B2, ..., Bm. Similarly, in the following description we refer to a group of objects B1, B2, ..., Bm; it should be understood that the group of objects includes at least one object, i.e. m>=1, and is not limited to a certain number of objects.

In addition to the encoder 110 and the decoder 160, FIG1 also shows a generalized transmission channel 150, which should be understood to include any equipment for transmitting or recording or storing a medium, in particular recording to a non-transitory machine-readable storage medium. In the context of the present invention, and more generally in communication theory, recording or storing combined with later playback can be considered a special case of information transmission or communication, with playback being understood to correspond to receiving and decoding the encoded information, typically at a later time, optionally in a different spatial location. Thus, the term "transmitting" can mean recording on a storage medium; "receiving" can mean reading from a storage medium; and "channel" can include information storage on a medium.

The signals are transmitted in a multiplexed format over the transmission channel, which is important for maintaining and preserving the synchronization relationship between the signals (A, B, C). The multiplexers and demultiplexers may include bit packing and data formatting methods known in the art. The transmission channel may also include other layers of information encoding or processing, such as error correction, parity checking, or other techniques suitable for the channel or physical layer described, for example, in the OSI layer model.

As shown, the decoder receives a compressed downmixed audio signal, demultiplexes the signal, decompresses the signal in an innovative way that allows acceptable reconstruction of the upmix to reproduce multiple independent signals (or audio objects), and then preferably upmixes the signal to restore the original signal (or as close as possible).

How it works:

Assume that A, _B1 , _B2 , ..., _Bm are independent signals (objects) that are encoded in the bitstream and sent to the renderer. The distinguished object A will be called the base object, and B = _B1 , _B2 , ..., _Bm will be called the regular object. We call a group of objects _B1 , _B2 , ..., _Bm ; however, it should be understood that the group contains at least one object (i.e., m>=1) and is not limited to a certain number of objects. In object-based audio systems, we are interested in rendering objects simultaneously but independently, so that, for example, each object can be rendered at a different spatial location.

For backward compatibility, we need the coded stream to be interpretable by legacy systems that are neither object-based nor object-aware. Such systems can only render the composite object C = A + B ₁ + B ₂ + ... + B _m from the encoded version E(C) of C. Therefore, we need to send a codestream consisting of E(C) followed by descriptions of the individual objects, which are ignored by legacy systems. In prior art approaches, the codestream would consist of E(C) followed by descriptions of regular objects E(B ₁ ), E(B ₂ ), ..., E(B _m ). The base object A is then recovered by decoding these descriptions and setting A = CB ₁ – B ₂ - ... - B _m . However, it should be noted that most audio codecs used in practice are lossy, meaning that the decoded version Q(X) = D(E(X)) of an encoded object E(X) is only an approximation of X, not necessarily identical to it. The accuracy of this approximation generally depends on the choice of codec {E, D} and on the bandwidth (or storage space) available for the codestream.

Therefore, it follows that when using a lossy encoder, the decoder will not have access to objects C, _B1 , _B2 , ..., _Bm , but will have access to approximate versions Q(C), Q( _B1 ), Q( _B2 ), ..., Q( _Bm ), and will only be able to estimate A as

Q'(A)=Q(C)-Q(B ₁ )-Q(B ₂ )-···-Q(B _m )

Such an approximation will suffer from the accumulation of errors in the individual lossy encodings. In practice, this often leads to unpleasant perceptible artifacts. In particular, Q'(A) may be a much worse approximation of A than Q(A), and its artifacts may be statistically correlated with other objects, while Q(A) is not. In practice, the residuals C–B1–B2, etc., will be audibly correlated with B1+B2+.. (for lossy compression). Our ears can pick up correlations that are difficult to detect algorithmically.

According to the present invention, some of the redundancies mentioned in connection with prior methods are avoided while still allowing acceptable reconstruction of A. Instead of including (the redundant signal) Q(A) in the codestream, we include the code E _c (Δ), where Δ is the residual signal:

Δ＝Q’(A)-A

and E _c is a lossy encoder for Δ (not necessarily the same as E). Let D _c be a decoder for E _c , and let

R(Δ)＝D _c (E _c (Δ))

On the decoder side, we get an approximation of A

Q _c (A) = Q' (A) - R (Δ)

The method of the first embodiment:

1. Encoder

The encoding method described mathematically above can be programmatically described as a sequence of actions, as shown in Figure 2. As previously described, at least one distinguished object A will be referred to as a base object, while _B1 , _B2 , ..., _Bm will be referred to as regular objects. For simplicity, we will refer to the regular objects collectively as B below, with the understanding that the set of all (at least one) regular objects _B1 , _B2 , ..., _Bm can be designated as {Bi}; in contrast, B = _B1 + _B2 + ... _Bm represents a mixture of regular objects _B1 , _B2 , ..., _Bm . The method begins with a mixed signal C = A + B. It should be clear that the mixing of A + B can be performed as a preliminary step, or the signals can be provided as pre-mixed. Signal A is also required; it can be received separately or reconstructed by subtracting B from C. The set of (at least one) regular objects {Bi} is also required and used by the encoder in the manner described below.

First, the encoder compresses (step 210) signals A, {Bi}, and C, respectively, using a lossy encoding method to obtain corresponding compressed signals represented by E(A), {E(Bi)}, and E(C), respectively. (The symbol {E(Bi)} indicates that each of the set of encoded objects corresponds to a corresponding original object belonging to the set of signals {Bi}, and each object signal is encoded separately by E). The encoder then decompresses (step 220) E(C) and {E(Bi)} using a method complementary to the method used to compress C and {Bi} to produce reconstructed signals Q(C) and {Q(Bi)}. These signals are approximate to the original C and {Bi} (different because they were compressed and then decompressed using a lossy compression/decompression method). Subsequently, {Q(Bi)} is subtracted from Q(C) using a subtractive mixing step 230 to produce a modified upmix signal Q'(A), which is an approximation of the original A, and Q'(A) is different from A due to errors introduced in the lossy encoding before mixing. Then, in a second mixing step 240, signal A (the reference signal) is subtracted from the modified upmix signal Q'(A) to obtain a residual signal Δ = Q'(A) - A (step 130). This residual signal Δ is then compressed (step 250) by a compression method, which we designate as _Ec , where _Ec is not necessarily the same compression method or device as E (used to compress signals A, {Bi}, or C in step 210). Preferably, to reduce bandwidth requirements, _Ec should be a lossy encoder for Δ that is selected to match the characteristics of Δ. However, in an alternative embodiment where bandwidth is less optimized, _Ec can be a lossless compression method.

Note that the method described above requires successive compression and decompression steps 210 and 220 (as applied to signals {Bi} and C). In these steps, as well as in the alternative methods described below, the computational complexity and time can be reduced in some examples by performing only the lossy portion of the compression (and decompression). For example, many lossy decompression methods for the DTS codec, such as that described in U.S. Patent 5,974,380, require the successive application of both the lossy steps (filtering into subbands, bit allocation, requantization in subbands) followed by the lossless steps (application of a codebook, entropy reduction). In such examples, it is sufficient to omit the lossless steps on both encoding and decoding and perform only the lossy steps. The reconstructed signal will still show all the effects of the lossy transmission, but many computational steps are saved.

The encoder then transmits (step 260) R = Ec(Δ), E(C), and {E(Bi)}. Preferably, the encoding method also includes the optional step of multiplexing or reformatting the three signals into a multiplexed package for transmission or recording. If some means is used to preserve or reconstruct the time synchronization of the three separate but related signals, then any of the known multiplexing methods can be used. It should be remembered that different quantization schemes can be used for all three signals and the bandwidth can be divided among the signals. Any of a number of known methods of lossy audio compression can be used for E, including MP3, AAC, WMA, or DTS (among others).

This approach offers at least the following advantages: First, the "error" signal Δ is expected to have less power and entropy than the original object. Due to its reduced power compared to A, the error signal Δ can be encoded with fewer bits than the object A, which aids reconstruction. Therefore, the proposed approach is expected to be more economical than the redundant description approach discussed above (in the Background section). Second, the encoder E can be any audio encoder (e.g., MP3, AAC, WMA, etc.), notably, the encoder can be, and in preferred embodiments is, a lossy encoder using psychoacoustic principles. (The corresponding decoder will, of course, also be a corresponding lossy decoder). Third, the encoder E _c need not be a standard audio encoder, but can be optimized for the signal Δ, which is not a standard audio signal. Indeed, perceptual considerations in the design and optimization of E _c will differ from those in the design of standard audio codecs. For example, perceptual audio codecs do not always seek to maximize the SNR across all parts of the signal; instead, they sometimes seek a more "constant" instantaneous SNR regime, where larger errors are tolerated when the signal is stronger. Indeed, this is the primary source of the artifacts found in Q'(A) caused by B _i . For E _c , we seek to eliminate as many of these artifacts as possible, so direct instantaneous SNR maximization seems more appropriate in this case.

The decoding method according to the present invention is illustrated in FIG3 . As a preliminary, optional step 300 , the decoder must receive and demultiplex the data streams to recover Ec(Δ), {E(Bi)}, and E(C). First, (step 310 ) the decoder receives the compressed data streams (or files) Ec(Δ), {E(Bi)}, and E(C). The decoder then decompresses each of the data streams (or files) Ec(Δ), {E(Bi)}, and E(C) (step 320 ) to obtain the reconstructed representations {Q(Bi)}, Q(C), and Rc(Δ) = Dc(Ec(Δ)), where Dc is the decompression method that is the inverse of the compression method Ec, and where the decompression method used for {E(Bi)} and E(C) is the complementary decompression method used for {Bi} and C. Signals Q(C) and {Q(Bi)} are subtractively mixed (step 330 ) to recover Q'(A) = Q(C) - ΣQ(Bi). This signal Q'(A) is an approximation of A and differs from the original A because it is reconstructed from a subtractive mix of Q(C) and {Q(Bi)}, both of which are transmitted using a lossy codec method. In the decoding and upmixing method of the present invention, the approximation signal Q'(A) is then improved by subtracting (step 340) the reconstructed residual R(Δ) to obtain Qc(A) = Q'(A) - R(Δ). The recovered replica signals Qc(A), Q(C), {Q(Bi)} can then be reproduced or output for reproduction as the upmix (A, {Bi}) (step 350). For systems with fewer channels, the downmix signal Q(C) is also available for output (or as a selection based on consumer control or preference).

It will be appreciated that the method of the present invention does require sending some redundant data. However, the file size (or bit rate requirement) of the method of the present invention is smaller than the file size (or bit rate requirement) required by either: a) using lossless encoding for all channels, or b) sending redundant descriptions of the lossy coded objects plus the lossy coded upmix. In one experiment, the method of the present invention was used to send the upmix A+B (for a single object B) along with the base channel A. The results are shown in Table 1. As can be seen, the redundant description (prior art) method would require 309KB to send the mix; in comparison, the method of the present invention would only require 251KB for the same information (plus some minimal overhead for multiplexing and header fields). This experiment does not represent a limit to the improvements that can be obtained by further optimizing the compression method.

As shown in FIG4 , in an alternative embodiment of the present method, the encoding method is different because the residual signal Δ is derived from the difference between Q′(A)=D(E(C))−ΣD(E(Bi)) and Q(A) (instead of A). This embodiment is particularly suitable in applications where the reconstruction of A is desired and expected to be approximately the same quality as the reconstructions of B and C (without striving for a higher fidelity reconstruction of A). This is often the case in audio entertainment systems.

Note that in an alternative embodiment, Q’(A) is the signal reproduced by taking the difference between a) an encoded and then decoded version of the C downmix, and b) the reconstructed basis object {Q(Bi)} reproduced by decoding the lossy coded base mix B.

Referring now to FIG. 4 , in an alternative method, an encoder compresses (step 410) signals A, {Bi}, and C, respectively, using a lossy encoding method to obtain three corresponding compressed signals, denoted by EA, {E(Bi)}, and E(C), respectively. The encoder then decompresses E(A) using a method complementary to the method used to compress A, producing Q(A), which is an approximation of A (different because it was compressed and then decompressed using a lossy compression/decompression method). The alternative method then decompresses both E(C) and {E(Bi)} using a corresponding method complementary to the method used to encode C and {Bi} (step 430). The resulting reconstructed signals Q(C) and {Q(Bi)} are approximations of the original {Bi} and C, differing due to imperfections introduced by the lossy encoding and decoding methods. The alternative method then subtracts ΣQ(Bi) from Q(C) in step 440 to obtain a difference signal Q’(A). Q'(A) is another approximation of A, which differs due to lossy compression being used for the transmitted downmix. The residual signal Δ is obtained by subtracting Q(A) from Q'(A) (step 450).

The residual signal Δ is then compressed (step 460) using an encoding method Ec (Ec may be different from E). As in the first embodiment described above, Ec is preferably a lossy codec suitable for the characteristics of the residual signal. The encoder then sends (step 470) R=Ec(Δ), E(C) and {E(Bi)} over a transmission channel, with the synchronization relationship preserved. Preferably, the encoding method also includes multiplexing or reformatting these three signals into a multiplexed package for transmission or recording. If some means is used to preserve or reconstruct the time synchronization of these three separate but related signals, then any of the known multiplexing methods can be used. It should be remembered that different quantization schemes can be used for all three signals and the bandwidth can be allocated between the signals. Any of the many known methods of audio compression can be used for E, including MP3, AAC, WMA or DTS (etc.).

A signal encoded by an alternative encoding method can be decoded using the same decoding method described above in conjunction with Figure 3. The decoder will subtract the reconstructed residual signal in order to improve the approximation of the upmixed signal, Q(A), thereby reducing the difference between the reconstructed replica signal Q(A) and the original signal A. The two embodiments of the present invention are united by the generality that they generate a residual or error signal Δ at the encoder, Δ representing the difference to be expected after decoding and upmixing the signal to extract the privileged object A. In both embodiments, the error signal Δ is compressed and transmitted (or equivalently, recorded and or stored). In both embodiments, the decoder decompresses the compressed error signal and subtracts it from the reconstructed upmixed signal, which approximates the privileged object A.

The methods of the alternative embodiments may have some perceived advantages in certain applications. In practice, which of the alternative embodiments is preferred may depend on the specific parameters of the system and the specific optimization goals.

In another aspect, the present invention includes an apparatus for compressing or encoding a mixed audio signal, as shown in FIG5 . In a first embodiment of the apparatus, signals C (=A+B object mixture) and B are provided at inputs 510 and 512 , respectively. Signal C is encoded by encoder 520 to produce encoded signal E(C); signal {Bi} is encoded by encoder 530 to produce a second encoded signal {E(Bi)}. E(C) and {E(Bi)} are then decoded by decoders 540 and 550 , respectively, to produce reconstructed signals Q(C) and {Q(Bi)}. The reconstructed signals Q(C) and {Q(Bi)} are subtractively mixed in mixer 560 to produce a difference signal Q'(A). This difference signal differs from the original signal A because it is obtained by mixing the reconstructed total mixture Q(C) and the reconstructed object {Q(Bi)}; artifacts or errors are introduced both because encoder 520 is a lossy encoder and because the signals are derived by subtraction (in mixer 560). The reconstructed signal Q’(A) is then subtracted from signal A (input to 570) and the difference Δ is compressed by a second encoder 580 to produce a compressed residual signal Ec(Δ), which in a preferred embodiment operates using a different method than the compressor 520.

As shown in Figure 6, in an alternative embodiment of an encoder device, signals C (=A+B object mixture) and B are provided at inputs 510 and 512, respectively. Signal C is encoded by encoder 520 to produce encoded signal E(C); signal {Bi} is encoded by encoder 530 to produce a second encoded signal {E(Bi)}. E(C) and {E(Bi)} are then decoded by decoders 540 and 550, respectively, to produce reconstructed signals Q(C) and {Q(Bi)}. The reconstructed signals Q(C) and {Q(Bi)} are subtractively mixed in mixer 560 to produce a difference signal Q'(A). This difference signal differs from the original signal A because it is obtained by mixing the reconstructed total mixture Q(C) and the reconstructed object {Q(Bi)}. Artifacts or errors are introduced because encoder 520 is a lossy encoder and because the signals are derived by subtraction (in mixer 560). Up to this point, the alternative embodiment is similar to the first embodiment.

In an alternative embodiment of the apparatus, the signal A received at input 570 is encoded by an encoder 572 (which may be the same encoder as the lossy encoders 520 and 530 or operate on the same principles), and the encoded output of 572 is then decoded again by a complementary decoder 574 to produce a reconstructed approximation Q(A), which differs from A due to the lossy nature of encoder 572. The reconstructed signal Q(A) is then subtracted from Q'(A) in a mixer 560, and the resulting residual signal is encoded by a second encoder 580 (using a method different from that used in lossy encoders 520 and 530). The outputs E(C), {E(Bi)}, and E(Δ) are then made available for transmission or recording, preferably in some multiplexed format or any other method that permits synchronization.

It will be clear that content encoded by the first or alternative method or encoding apparatus ( FIG. 6 ) can be decoded by the decoder of FIG. 3 . The decoder requires a compressed error signal, but need not be sensitive to the manner in which the error is calculated. This leaves room for future improvements in the codec without changing the decoder design.

The methods described herein can be implemented in consumer electronic devices, such as general-purpose computers, digital audio workstations, DVD or BD players, TV tuners, CD players, handheld players, Internet audio/video devices, game consoles, mobile phones, headphones, and the like. Consumer electronic devices may include a central processing unit (CPU), which may represent one or more types of processors, such as an IBM PowerPC, an Intel Pentium (x86) processor, and the like. Random access memory (RAM) temporarily stores the results of data processing operations performed by the CPU and is typically connected to the CPU via a dedicated memory channel. Consumer electronic devices may also include permanent storage devices, such as hard drives, which may also communicate with the CPU via an I/O bus. Other types of storage devices, such as tape drives or optical disk drives, may also be connected. A graphics card may also be connected to the CPU via a video bus and send signals representing display data to a display monitor. Peripheral data input devices, such as a keyboard or mouse, may be connected to the audio reproduction system via a USB port. A USB controller may convert data and instructions to and from the CPU for use by peripheral devices connected to the USB port. Additional devices, such as printers, microphones, speakers, headphones, and the like, may be connected to the consumer electronic device.

The consumer electronic device may utilize an operating system with a graphical user interface (GUI), such as WINDOWS from Microsoft Corporation of Redmond, Washington, MAC OS from Apple Inc. of Cupertino, CA, various versions of mobile GUI designed for mobile operating systems such as Android, and the like. The consumer electronic device may run one or more computer programs. Typically, the operating system and the computer program are tangibly embodied in a non-transitory computer-readable medium, such as one or more of a hard drive, fixed and/or removable data storage device. Both the operating system and the computer program may be loaded from the aforementioned data storage device into RAM for execution by the CPU. The computer program may include instructions that, when read and executed by the CPU, cause the CPU to perform the steps or features of the embodiments described herein.

The embodiments described herein can have many different configurations and architectures. Any such configuration or architecture can be easily substituted. Those skilled in the art will recognize that the above sequence is the most commonly used in computer-readable media, but there are other existing sequences that can be substituted.

The elements of an embodiment can be implemented by hardware, firmware, software or any combination thereof. When implemented as hardware, the embodiments described herein can be applied on an audio signal processor or distributed between various processing components. When implemented in software, the elements of the embodiment can include code segments that perform the necessary tasks. The software can include actual code that implements the operations described in an embodiment or code that simulates or emulates the operations. The program or code segment can be stored in a processor or machine accessible medium, or sent via a transmission medium by a computer data signal embodied in a carrier or a signal modulated by a carrier. Processor-readable or accessible medium or machine-readable or accessible medium can include any medium that can store, send or transmit information. In contrast, a computer-readable storage medium or non-transitory computer memory can include a physical computer machine storage device but does not include a signal.

Examples of processor-readable media include electronic circuits, semiconductor storage devices, read-only memories (ROMs), flash memories, erasable ROMs (EROMs), floppy disks, compact disk (CD) ROMs, optical disks, hard disks, fiber optic media, radio frequency (RF) links, and the like. Computer data signals may include any signal that can be transmitted via a transmission medium such as an electronic network channel, an optical fiber, air, electromagnetic waves, an RF link, and the like. Code segments may be downloaded via a computer network such as the Internet, an intranet, and the like. Machine-accessible media may be embodied in an article of manufacture. Machine-accessible media may include data that, when accessed by a machine, causes the machine to perform the operations described below. The term "data," in addition to its ordinary meaning, refers here to any type of information that is encoded for the purpose of being machine-readable. Thus, it may include programs, codes, files, and the like.

All or part of various embodiments can be implemented by software running in a machine, and this machine for example includes the hardware processor of digital logic circuit.Software can have several modules coupled to each other.The hardware processor can be a programmable digital microprocessor or a special programmable digital signal processor (DSP), a field programmable gate array, an ASIC or other digital processor.For example, in one embodiment, all of the steps of the method according to the present invention (or in terms of encoder or decoder) can be suitably implemented by one or more programmable digital computers that sequentially run all steps under software control.A software module can be coupled to another module to receive variables, parameters, independent variables (arguments), pointers etc. and/or to generate or transfer results, updated variables, pointers etc.A software module can also be a software driver or an interface that interacts with the operating system running on the platform.A software module can also include a hardware driver that is used to configure, set, initialize a hardware device, send data to the hardware device or receive data from the hardware device.

Various embodiments may be described as one or more processes, which may be depicted as flow charts, flow diagrams, structure diagrams, or block diagrams. Although a block diagram may depict operations as a sequential process, many operations may be performed in parallel or concurrently. Furthermore, the order of operations may be reset. A process terminates when its operations are completed. A process may correspond to a method, procedure, step, or the like.

Throughout this application, reference is frequently made to adding, subtracting, or "subtractively mixing" signals. It will be readily appreciated that signals can be mixed in a variety of ways with equivalent results. For example, to subtract an arbitrary signal F from G (G-F), one can use differential inputs to directly subtract, or equivalently flip one of the signals and then add (e.g., G+(-F)). Other equivalent operations are contemplated, some involving the introduction of phase offsets. Terms such as "subtracting" or "subtractively mixing" are intended to include such equivalent variations. Similarly, variations on the addition of signals are possible and contemplated as "mixing."

While several exemplary embodiments of the present invention have been shown and described, numerous modifications and alternative embodiments will occur to those skilled in the art. Such modifications and alternative embodiments may be conceived and made without departing from the spirit and scope of the present invention as defined in the appended claims.

Claims (27)

1. A method for decompressing and upmixing a compressed and downmixed composite audio signal, comprising the following steps: receiving a compressed representation of a total mixed signal C, a compressed representation of a residual signal Δ, and a set of compressed representations {Bi} of corresponding object signals, wherein the set of compressed representations of at least one object signal comprises at least one compressed representation of a corresponding object signal Bi, wherein Bi is one of the set of compressed representations {Bi} of the corresponding object signals, and the total mixed signal C is a mixture of the set of object signals {Bi} and a base signal A; Decompressing the compressed representation of the total mixed signal to obtain an approximate total mixed signal C'; decompressing the compressed representation of the residual signal Δ to obtain a reconstructed residual signal; decompressing the set of compressed representations of the object signals so as to obtain a complete set of approximated object signals {Bi'}, said set having as members one or more approximated object signals Bi'; subtractively mixing the approximated total mixed signal C' and the complete set of approximated object signals {Bi'} to obtain a first approximation A' of the basis signal; and The reconstructed residual signal is subtractively mixed with a first approximation of the base signal in order to obtain an improved approximation of the base signal. 2. The method of claim 1, wherein the set of compressed representations of object signals comprises one compressed representation of the corresponding object signal. 3. The method of claim 1, wherein at least one of the compressed representations is prepared by a lossy compression method. 4. The method of claim 3 , wherein the compressed representation of the residual signal Δ is prepared by: subtractively mixing a reference signal R with a reconstructed approximation A' of the base signal A to obtain a residual signal Δ representing the difference; and The residual signal Δ is compressed. The method of claim 4 , wherein the reference signal comprises a base signal A. The method of claim 4 , wherein the reference signal comprises an approximation of the base signal A. 7. The method of claim 1 , further comprising: At least one of the corrected base signal A", the reconstructed set of object signals Q({Bi}) and the approximate total mixed signal C' is reproduced as sound. 8. The method of claim 1, wherein The step of decompressing the set of compressed representations of the corresponding object signals comprises decompressing a plurality of compressed representations so as to obtain the complete set of approximated object signals {Bi'}; and The step of subtractively mixing the approximated total mixed signal C' and the complete set of object signals comprises subtracting the complete set of approximated object signals {Bi'} from C' to obtain a first approximation of the base signal. 9. The method of claim 8, wherein at least one of the compressed representations is prepared by a lossy compression method. 10. The method of claim 9, wherein the compressed representation of the residual signal Δ is prepared by: subtractively mixing the reference signal R with the reconstructed approximation A' of the base signal A to obtain a residual signal Δ representing the difference; and The residual signal Δ is compressed. The method of claim 10 , wherein the reference signal comprises a base signal A. The method of claim 10 , wherein the reference signal comprises an approximation of the base signal A. 13. The method of claim 8, further comprising: At least one of the corrected base signal A", the reconstructed set of object signals Q({Bi}) and the approximate total mixed signal C' is reproduced as sound. 14. A method for compressing a composite audio signal comprising a total mixed signal C, a set of compressed representations {Bi} of corresponding object signals, and a base signal A, wherein the total mixed signal C comprises the base signal A mixed with the set of compressed representations {Bi} of corresponding object signals, the set of compressed representations {Bi} of corresponding object signals having at least one member object signal Bi, the method comprising the following steps: compressing the total mixed signal C and the set of compressed representations {Bi} of the corresponding object signals using a lossy compression method to produce a compressed total mixed signal E(C) and a compressed set of object signals E({Bi}), respectively; Decompressing the compressed total mixed signal E(C) and the compressed set of object signals E({Bi}) to obtain a reconstructed signal Q(C) and a reconstructed set of object signals Q({Bi}); subtractively mixing the reconstructed signal Q(C) and a complete mixture of the set of reconstructed signals Q({Bi}) to produce an approximate basis signal Q'(A); subtracting a reference signal from the approximated base signal Q'(A) to generate a residual signal Δ; and The residual signal Δ is compressed to obtain a compressed residual signal Ec(Δ). 15. The method of claim 14, wherein the set of compressed representations {Bi} of respective object signals comprises only one object signal. 16. The method of claim 15, further comprising the steps of: A composite signal including the compressed total mixed signal E(C), the compressed set of object signals E({Bi}) and the compressed residual signal Ec(Δ) is transmitted. The method of claim 15 , wherein the reference signal comprises a base signal A. 18. The method of claim 15, wherein the reference signal comprises an approximation of a base signal A, wherein the approximation of the base signal A is obtained by compressing the base signal A using a lossy compression method and then decompressing it to obtain an approximation Q(A) of the base signal. 19. The method of claim 15, wherein the step of compressing the residual signal comprises compressing the residual signal using a method different from the method used to compress the total mixed signal C. 20. The method of claim 14, wherein the set of compressed representations {Bi} of corresponding object signals comprises a plurality of object signals. The method of claim 20 , wherein the reference signal comprises a base signal A. 22. The method of claim 20, wherein the reference signal comprises an approximation of a base signal A, wherein the approximation of the base signal A is obtained by compressing the base signal A using a lossy compression method and then decompressing it to obtain an approximation Q(A) of the base signal. 23. The method of claim 20, wherein the step of compressing the residual signal comprises compressing the residual signal using a method different from the method used to compress the total mixed signal C. 24. A method for improving digital audio reproduction by refining an approximation of an audio base signal A derived from an approximated total mixed signal C′ and a complete set of approximated object signals {Bi′} having at least one member signal Bi′, the method comprising the steps of: decompressing the compressed representation of the residual signal Δ to obtain a reconstructed residual signal Δ′; subtractively mixing the approximated total mixed signal C' and the complete set of approximated object signals {Bi'} to obtain a first approximation A' of the audio base signal A; and The first approximation A' of the audio base signal A is subtractively mixed with the reconstructed residual signal Δ' in order to obtain an improved approximation of the audio base signal A. 25. The method of claim 24, wherein the compressed representation of the residual signal Δ is prepared by: subtractively mixing the reconstructed approximation A' of the audio base signal A with the reference signal R to obtain a residual signal representing the difference; and The residual signal is compressed to obtain a compressed representation of the residual signal Δ. The method of claim 25 , wherein the reference signal comprises an audio base signal A. 27. The method of claim 25, wherein the reference signal comprises an approximation A' of the base signal, the approximation A' being prepared by compressing the audio base signal A using a lossy method and then decompressing it to obtain the reference signal R.