KR101144752B1 - video encoding/decoding method and apparatus thereof - Google Patents

Abstract

A video encoding / decoding method and apparatus thereof are disclosed. A method according to an embodiment encodes a plurality of view-by-view video images, applies multi-view video encoding to at least one of the plurality of view-by-view video images, and applies to at least one of the plurality of view-by-view video images. Generating scalable per-layer bitstreams by applying scalable video encoding; And generating the output bitstream by combining the generated per-layer layer bitstreams according to a predetermined order based on the view and the layer.

Description

Video encoding / decoding method and apparatus thereof

The disclosed technique relates to a video encoding / decoding technique, which is more particularly, but not exclusively, various kinds of network environments and terminals of various formats-various kinds of realistic displays ( For example, the terminal includes a terminal supporting stereoscopic display, a multi-view display, a view-selective display, and the like, and a terminal supporting various kinds of existing two-dimensional displays. The present invention relates to an encoding / decoding technique.

Recently, with the rapid development of video coding technology, users can enjoy high resolution and / or high quality video services in various application fields (eg, communication, broadcasting, storage media, etc.).

On the other hand, video coding techniques are mainly used to support two-dimensional flat panel display devices or to support a specific format, for example, a video format dedicated to a specific application / transmission environment / terminal, but in the future, multi-view video coding is used. Multi-view Video Coding (MVC) technology and Scalable Video Coding (SVC) technology are expected to spread rapidly.

The technical problem to be achieved by the disclosed technology is to provide a technology capable of efficiently processing image information in consideration of various types of network environments and various types of terminals.

In order to achieve the above technical problem, an aspect of the disclosed technology is to (a) encode a plurality of view-by-view video images, apply multi-view video encoding to at least one of the plurality of view-by-view video images, and Generating scalable per-layer bitstreams by applying scalable video encoding to at least one of per-view video images; And (b) combining the generated per-layer layer-specific bitstreams according to a predetermined order based on the view and the layer to generate an output bitstream.

In an embodiment, the step (a) may include applying multi-view video encoding and scalable video encoding to at least one of the plurality of view-by-view video images.

In one embodiment, applying the multi-view video coding in step (a) includes performing inter-view prediction based on video image information of another view.

In one embodiment, applying the scalable video coding in the step (a) is to encode the video image of the current layer based on the video image information obtained by the video encoding for the lower layer and the video image of the current layer. It includes.

In an embodiment, the step (a) may include setting a prediction structure for each layer and encoding scalable video according to the set layer-specific prediction structure.

In an embodiment, the step (a) may include determining whether a prediction structure of a current layer of the current view image is the same as a prediction structure of a lower layer of the current view image; And encoding the prediction structure of the current layer by determining whether to use the prediction structure of the lower layer according to the determination result.

In one embodiment, the scalable video encoding includes at least one of spatial scalable video encoding, temporal scalable video encoding, and quality scalable video encoding for the plurality of view-by-view images.

In one embodiment, the step (b) generates time-phased access units arranged by time-based bitstreams of time-phase hierarchies in the same time zone, and assigns the time-phased access units to the time sequence. Outputting the multi-view scalable video bitstreams listed accordingly.

Another aspect of the disclosed technology to achieve the above technical problem comprises the steps of: (a) receiving a bitstream; (b) dividing the received bitstream into layer-wise bitstreams for each view; And (c) performing multi-view scalable video decoding on the per-layer layered bitstreams using the inter-view information and the inter-layer information.

In an embodiment, the step (b) includes selecting a bitstream to be decoded according to a layer and a time point among the video bitstream.

In an embodiment, the step (c) may include: determining whether to perform inter-layer scalable video decoding according to whether an image of a lower layer or information on a lower layer of a current view image is present; Determining whether to perform multiview video decoding according to whether to perform view direction prediction coding; A multi-view scalable video decoding, a single-view scalable video decoding, a multi-view video decoding, and the like according to a result of determining whether the scalable video decoding is performed and whether or not the multi-view video decoding is performed. Performing one of the single-view video decoding.

In an embodiment, the step (c) includes decoding according to a prediction structure set for each layer.

In an embodiment, the step (c) may include performing multi-view video decoding on an image of the current view with reference to image information of a view other than the current view among the plurality of views. .

In one embodiment, the step (b) divides the received multi-view scalable video bitstream into time-phased access units listed in chronological order, and the time-phased access units in the same time zone in chronological order. And dividing the bitstream into layer-by-layer bitstreams.

Another aspect of the disclosed technology to achieve the above technical problem, the encoder for performing a multi-view scalable video encoding using the inter-view information and inter-layer information for a plurality of time-specific images; And a video output unit configured to output a multiview scalable video bitstream by combining the per-view layer bitstream generated by the scalable video encoding of the per-view images according to the order considering the view and the layer. Provide an encoding device.

Another aspect of the disclosed technology to achieve the above technical problem is a receiver for receiving a multiview scalable video bitstream; A splitter configured to divide the multiview scalable video bitstream into viewable layer-specific bitstreams based on a plurality of views and a plurality of layers of the multiview scalable video bitstream; And a decoder configured to perform multi-view scalable video decoding using the inter-view layer information and the inter-layer information on the per-layer hierarchical bitstreams.

Another aspect of the disclosed technology to achieve the above technical problem provides a computer-readable recording medium recording a program for implementing the disclosed video encoding method.

Another aspect of the disclosed technology to achieve the above technical problem provides a computer-readable recording medium recording a program for implementing the disclosed video decoding method.

One embodiment of the disclosed technique may have the effect of including the following advantages. However, not all the embodiments of the disclosed technology to include all, it should not be understood that the scope of the disclosed technology is limited by this.

Through the disclosed technology (e.g., multi-view scalable video encoding / decoding technology, hierarchical adaptive prediction technology, etc.), video information is integrated and processed and easily converted and transmitted to realistic terminals and existing 2D terminals in a ubiquitous environment. Can be.

1 illustrates an example of a prediction structure used in a multiview video coding technique.
FIG. 2 is a diagram for explaining a method of not using a scalable video coding technique and a method of using a scalable video coding technique.
3 illustrates a block diagram of a multiview scalable video encoding apparatus according to an embodiment.
4 is a block diagram of an apparatus for multiview scalable video decoding, according to an exemplary embodiment.
5 illustrates a schematic diagram of an immersive multiview scalable video service that can be implemented according to an embodiment.
6 illustrates a structure of an implemented encoder for multiview scalable video encoding according to an embodiment.
7 illustrates a detailed structure of an implemented encoder for multiview scalable video encoding according to an embodiment.
8 illustrates a structure for combining the output bitstreams according to an embodiment.
9 illustrates a configuration of a bitstream generated by multiview video encoding according to an embodiment.
10 illustrates a configuration of a bitstream generated by multiview scalable video encoding according to an embodiment.
11 is a diagram illustrating bitstreams according to layers according to an embodiment, according to an embodiment.
12 illustrates a detailed configuration of a multiview scalable video bitstream according to an embodiment.
13 illustrates a structure of an implemented decoder for multiview video decoding according to an embodiment.
14 illustrates a structure of an implemented decoder for multiview scalable video decoding according to an embodiment.
15 illustrates a structure of an implemented decoder for multiview scalable video decoding according to another embodiment.
16 is a flowchart of a method of performing decoding in units of pictures or slices in a sub decoder, according to an embodiment.
FIG. 17 illustrates information required for decoding a single image and scalable video between images according to an embodiment.
18 is a flowchart of a method of setting a prediction structure of a current layer, according to a layer-by-layer prediction structure, according to an embodiment.
19 is a flowchart of another method of setting a prediction structure of a current layer, according to a layer-by-layer prediction structure, according to an embodiment.
20 is a flowchart of a method of coding a prediction structure of a current layer by selectively using a prediction structure of a lower layer, according to a layer-by-layer prediction structure, according to another embodiment.
21 is a flowchart of another method of coding a prediction structure of a current layer by selectively using a prediction structure of a lower layer, according to another layer-specific prediction structure, according to another embodiment.
22 illustrates an example of a prediction structure of a spatial current layer and a lower layer in multiview scalable video coding having four views and two spatial layers, according to an embodiment.
23 illustrates another example of a prediction structure of a spatial current layer and a lower layer in multiview scalable video coding having four viewpoints and two spatial layers, according to an embodiment.
24 illustrates another embodiment of a prediction structure of a spatial current layer and a lower layer in multiview scalable video coding having four views and two spatial layers, according to an embodiment.
25 is a block diagram of a multiview scalable video encoder according to another embodiment.
26 is a block diagram of a multiview scalable video decoder according to another embodiment.
27 is a flowchart of a multiview scalable video encoding method according to an embodiment.
28 is a flowchart of a multiview scalable video decoding method, according to an embodiment.

Since descriptions of embodiments of the present invention are merely illustrated for structural to functional descriptions of the present invention, the scope of the present invention should not be construed as limited by the embodiments described in the present invention. That is, the embodiments of the present invention may be variously modified and may have various forms, and thus, it should be understood to include equivalents that may realize the technical idea of the present invention.

On the other hand, the meaning of the terms described in the present invention will be understood as follows.

Terms such as "first" and "second" are intended to distinguish one component from another component, and the scope of the present invention should not be limited by these terms. For example, the first component may be named a second component, and similarly, the second component may also be named a first component.

The term “and / or” should be understood to include all combinations that can be presented from one or more related items. For example, "first item, second item, and / or third item" means "at least one or more of the first item, second item, and third item", and means first, second, or third item. A combination of all items that can be presented from two or more of the first, second and third items as well as the third item.

When a component is referred to as being "connected" to another component, it should be understood that there may be other components in between, although it may be directly connected to the other component. On the other hand, when an element is referred to as being "directly connected" to another element, it should be understood that there are no other elements in between. On the other hand, other expressions describing the relationship between the components, such as "between" and "immediately between" or "neighboring to" and "directly neighboring to", should be interpreted as well.

Singular expressions described herein are to be understood to include plural expressions unless the context clearly indicates otherwise, and the terms "comprise" or "having" include elements, features, numbers, steps, operations, and elements described. It is to be understood that the present invention is intended to designate that there is a part or a combination thereof, and does not exclude in advance the possibility of the presence or addition of one or more other features or numbers, steps, actions, components, parts or combinations thereof. .

Each step described in the present invention may occur out of the stated order unless the context clearly dictates the specific order. That is, each step may occur in the same order as specified, may be performed substantially simultaneously, or may be performed in the reverse order.

Unless otherwise defined, all terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which the present invention belongs. Terms such as those defined in the commonly used dictionaries should be construed as having meanings consistent with the meanings in the context of the related art, and shall be interpreted as having ideal or overly formal meanings unless expressly defined in this application. Can't be.

First, the multi-view video coding technology is based on the existing video international standard MPEG-4 part 10 Advanced Video Coding (AVC; H.264) method, but is inputted from a plurality of cameras spaced at regular intervals in various forms. As a technology for efficiently encoding video images of a viewpoint, a realistic display device such as a 3D TV (3DTV) or a free viewpoint TV (FTV) is supported. In this multi-view video encoding technique, hierarchical B-pictures coding, which is a method performed to support temporal scalability in the Joint Scalable Video Coding (JSVC) scheme, is used in the time direction. In the view direction, inter-view prediction is used.

1 illustrates an example of a prediction structure used in a multiview video coding technique. In detail, eight viewpoint video images exist, and a size of a GOP (group of pictures) in the time direction is 8 (= N). The prediction structure of the case is shown.

In FIG. 1, S0, S0, S1, S2, S3, S4, S5, S6, S7 each represent one view, and T0, T1, T2, T3, ..., T16 represent the flow in time. Indicates. In the multi-view video coding technique, encoding that refers to video images of different views according to the arrow direction of FIG. 1, that is, prediction of the view direction (inter-view prediction) is performed.

Next, scalable video coding began standardization in March 2004 at the MPEG meeting of the International Organization for Standardization (ISO) / IEC (International Electrotechnical Commission) and decided to standardize on H.264 coding technology. After the decision, JVT (Joint Video Team) standardized it in January 2005. In July 2007, H.264 Amendment 3 was completed. The scalable video coding method is a technology for integrating video information in various types of terminals and various transmission environments and supports various spatial resolutions, various frame rates, and various image quality. It is a method of supporting data to be efficiently transmitted to various transmission environments and various terminals by generating one integrated data as much as possible.

FIG. 2 is a diagram for comparing and explaining a method of not using a scalable video coding technique (hereinafter, referred to as a first scheme) and a method of using a scalable video coding technique (hereinafter referred to as a second scheme). In FIG. 2, a video quality 200 having 4 CIF (4 × Common Intermediate Format, 704 × 576) resolution is converted into a CIF (352 × 288) class low-quality two-dimensional display device 210 or CIF class. It is assumed that the present invention is transmitted to the high quality 2D display device 220 and the 4CIF class high quality 2D display device 230, respectively.

According to the first scheme 240, the video encoders 242, 244, 246 perform respective encoding processes to conform to the format of the respective devices 210, 220, 230 so that each bitstream 252, 254 , 256). In addition, a complex process of transferring all the generated bitstreams 252, 254, and 256 is stored.

In contrast, according to the second scheme 260, the SVC encoder 265 performs an encoding process according to the scalable video coding scheme only once to generate the SVC bitstream 275, and then, respectively, in the SVC bitstream 275. The bitstream may be transmitted through an extraction process 285 to fit the devices 210, 220, and 230 of the apparatus.

Multi-view video coding standard, which is a method of efficiently coding multi-view video to support sensory display devices, and a scalable video coding standard that can efficiently and efficiently code and transmit various transmission environments and various types of terminals, In the ubiquitous environment, there is currently no method for integrating video information to realistic terminals and existing 2D terminals.

Although advances in video coding technology have made it possible for users to access high-definition and high-definition video information, they are only ways to support two-dimensional flat panel display devices, and they are realistic services that give users a three-dimensional effect or freely select a viewpoint. (Realistic Service) is not taken into account.

In the future, video information should be efficiently delivered in a ubiquitous environment in which various types of terminals are mixed in various transmission environments through processes such as convergence of broadcasting / communication and convergence of wired / wireless. However, since these video coding methods are designed to perform specific coding for a specific transmission environment and a specific terminal in each application, efficient processing is impossible.

If it is assumed that a single video information is serviced using a conventional video coding method, it can be expected to be a very difficult and complicated task because encoding must be repeatedly performed to support various transmission environments and terminals.

At present, interest and user desire for immersive content are very high, and various types of sensational display devices are expected to be commercialized. The user's interest in immersive content is rapidly increasing in the movie industry, and immersive display devices such as personal stereoscopic display devices and multi-view video display devices are being developed on various platforms.

Therefore, there is a need for a technology that processes sensory services and integrates video information that supports various transmission environments and various formats of various terminals so that realistic video contents can be efficiently delivered to various transmission environments and various terminals. Do.

3 to 29, a multi-view scalable video encoding / decoding technique according to each embodiment will be described in detail.

3 is a block diagram illustrating a configuration of a multiview scalable video encoding apparatus according to an embodiment.

In one embodiment, the multi-view scalable video encoding apparatus 300 includes an encoder 320 and an output unit 330. In another embodiment, as shown in FIG. The point scalable video encoding apparatus 300 may further include an acquirer 310.

The acquirer 310 acquires a plurality of video images having different viewpoints (hereinafter, referred to as view video images) and outputs view video images in a preset image processing unit. As an example of a method of acquiring a video image, the acquiring unit 310 includes a photographing module (eg, a multi-view camera or a plurality of cameras) to acquire viewpoint video images through photographing, and the acquiring unit 310 A method of receiving viewpoint video images from an external photographing module, and a method of reading viewpoint video images recorded on a storage medium (eg, hard disk, CD ROM, etc.), but are not limited thereto. Examples of the preset image processing unit may include a picture, a slice, a frame, a field, and the like, but are not limited thereto. In addition, in the present specification, for convenience, an image processing method according to the disclosed technology will be mainly described in picture units, but an image processing method using other image processing units (eg, a slice, etc.) is also within the scope of the present invention. Anyone in the field can fully understand.

The encoder 320 encodes a plurality of view video images provided from the acquirer 310, applies multi-view video coding to at least one of the plurality of view video images, and applies the plurality of view video images. For at least one of them, scalable video coding is applied to generate bitstreams per view layer. Here, the N view video images (first to Nth view video images) may have the same resolution or may not have the same resolution. In an embodiment, the encoder 320 may apply multi-view video coding and scalable video coding to at least one of the plurality of view-by-view video images.

As a method of applying multi-view video coding to a video image of a corresponding view, a method of encoding the video image of the corresponding view through inter-view prediction based on video image information of another view may be used. However, the present invention is not limited thereto. In the present specification, the video image information includes a reconstructed video image (for example, when scalable video coding is applied to a reconstructed video image of another view, a reconstructed video image of a lower layer, and a video image of another view, hierarchical reconstructed video image of another view. Etc.) and encoding information (for example, encoding information of another view, encoding information of a lower layer, and encoding information according to layers of another view point when scalable video coding is applied to a video image of another view). The encoding information is used as a term encompassing a motion vector, a block type used for encoding (eg, a block type used for motion estimation, a block type used for DCT), a transform coefficient, and the like.

In addition, as a method of applying scalable video coding to a video image of a corresponding view, the video of the current layer is based on the video image information obtained by encoding the lower layer of the corresponding view and the video image of the current layer of the corresponding view. A method of encoding an image may be included, but is not necessarily limited thereto. Examples of scalable video coding include spatial scalable video coding, temporal scalable video coding, and quality scalable video coding. According to an example of spatial scalable video coding, an image of a current layer and an image of a lower layer may be generated according to spatial resolution, and the image of the current layer may be predicted with reference to the image of the lower layer. In addition, according to an example of temporal scalable video coding, images of upper and lower layers may be generated according to frame rates, and upper and lower layers according to image quality may be distinguished according to an example of image quality scalable video coding.

According to an example of multi-view scalable video coding of the encoder 320, an image of a lower layer having a reduced resolution is generated by downsampling an input image at a predetermined time point, and temporally scalable video of the image of the lower layer is generated. Coding and quality scalable video coding may be performed. In this case, the encoder 320 according to an embodiment may perform spatial scalable video coding by referring to a result of temporally scalable video coding of an image of a lower layer, for scalable video coding of an image of a current layer of an input image. And temporally scalable video coding and quality scalable video coding may be selectively performed. In this case, the temporal scalable video coding of the lower layer and the spatial scalable video coding of the current layer refer to encoding information about images of viewpoints other than a predetermined viewpoint, thereby multi-view scale of the encoding performing unit 320. An example of flexible video coding may be performed.

In an embodiment, the encoding performing unit 320 may adaptively set a prediction structure for each layer and encode the prediction structure for each layer. For example, a temporal direction prediction structure or a view direction prediction structure may be adaptively determined for each layer according to the spatial layer or the image quality layer. Hereinafter, a prediction structure of an embodiment in which a prediction structure is individually set for each layer according to an embodiment is referred to as a 'layer prediction structure'.

According to an embodiment, the encoder 320 according to the prediction structure for each layer may set the prediction structure for each layer and then perform encoding according to the prediction structure for each layer for each layer. In another embodiment, the encoding performing unit 320 according to the prediction structure for each layer may repeatedly set the prediction structure individually for each layer and then perform encoding according to the set prediction structure.

In an embodiment, the encoder 320 according to the prediction structure for each layer determines whether the prediction structure of the current layer of the video image at the current view is the same as the prediction structure of the lower layer of the video image at the current view. According to the determination result, it may be determined whether to use the prediction structure of the lower layer to encode the prediction structure of the current layer. In this case, the referenced lower layer may be a lower layer below one layer of the current layer or a predetermined layer selected from a plurality of lower layers.

In one embodiment, the encoder 320 according to the prediction structure for each layer may set the view direction prediction structure of the spatial enhancement layer independently of the view direction prediction structure of the spatial base layer. For example, the encoder 320 according to the layer-by-layer prediction structure, the view direction prediction structure of the spatial base layer performs view direction prediction for all pictures, while for the spatial enhancement layer, non-anchor pictures or all the pictures. In this case, a prediction structure for not performing view direction prediction may be set.

In one embodiment, the encoder 320 is configured to predict the prediction structure of the current layer by using the encoding information of the lower layer according to the prediction structure for each layer, but the encoding information of the lower layer is determined by the prediction structure of the current layer. In the case of a structure that cannot be used for encoding (eg, when encoding information of a lower layer does not match the prediction structure of the current layer picture), encoding information of a lower layer may not be used.

In another embodiment, the encoder 320 may use only a portion that matches the prediction structure of the current layer. For example, if the prediction structure of the picture of the current layer allows only prediction in the time direction, and the encoding information of the lower layer is a result of performing both the prediction in the time direction and the view direction, the encoder 320 may encode the lower layer. The encoding according to the prediction structure of the current layer may be performed by referring to only prediction information in the temporal direction among the information. As another example, if the prediction structure of the current layer allows only prediction in the temporal direction and the encoding information of the lower layer is a result of performing the prediction in the view direction, the encoder 320 may refer to only the block type and the partition information of the lower layer. Can be.

The output unit 330 combines the per-layer layer-specific bitstreams generated by the encoder 320 according to a predetermined order based on the view and the layer, thereby generating an output bitstream (that is, a multi-view scalable video bitstream). Create According to an embodiment, the output unit 330 may generate access units according to time, which are arranged in bit streams according to time layers of the same time zone. An access unit is a kind of NAL unit. Timephased access units may be listed in chronological order to form a multiview scalable video bitstream. A method for constructing a multiview scalable video bitstream will be described in more detail with reference to FIGS. 9 to 13.

In one embodiment, the multi-view scalable video coding apparatus 300 may identify identification information of spatial scalability information and temporal scalability in order to identify a bitstream for each view layer of the access unit. ) May include a syntax including identification information of the information, identification information of the quality scalability information, and identification information of the bitstream for each layer per view. More details about this identification information will be described later.

4 is a block diagram illustrating a configuration of a multiview scalable video decoding apparatus according to an embodiment.

In one embodiment, the multi-view scalable video decoding apparatus 400 includes a divider 420 and a decoder 430. In another embodiment, as shown in FIG. The point scalable video decoding apparatus 300 may further include a receiver 410.

The receiver 410 receives a bitstream (that is, a multiview scalable video bitstream) encoded by a multiview scalable video coding scheme. Examples of the reception method may include a method of receiving a bitstream through wired / wireless communication or broadcasting, a method of reading a bitstream recorded in a storage medium, and the multiview scalable video decoding apparatus 400 is installed. It can be fully understood by those skilled in the art that various reception methods may exist depending on the environment.

In one embodiment, the multiview scalable video decoding apparatus 400 may select bitstreams of a portion to be decoded with respect to the received bitstream. For example, according to the decoder request or the reproduction capability of the display device, the bitstream to be decoded may be detected according to a specific format such as spatial resolution, temporal resolution, image quality, and the like. Since a multi-view scalable video bitstream according to an embodiment supports spatial scalability, temporal scalability, and quality scalability, the multi-view scalable video bitstream corresponds to a video suitable for a specific format such as spatial resolution, temporal resolution, and image quality. The bitstream may be selected from a multiview scalable video bitstream.

The divider 420 may convert the bitstream received by the bitstream receiver 410 into bitstreams per view layer, based on a plurality of views and a plurality of layers according to multiview scalable video coding. Divide.

The bitstream splitter 420 according to an embodiment divides the bitstream into timephased access units listed in a time sequence, and timephased access units of timeline hierarchical bitstreams of the same timezone in a time sequence. Can be divided into In addition, the bitstream splitter 420 according to an embodiment may divide the bitstream for each layer according to one or more layers from each time zone access unit.

The splitter 420 according to an embodiment may include: identification information of spatial scalability information, identification information of temporal scalability information, identification information of image quality scalability information, and a bitstream for each layer in view of a syntax for a bitstream According to the identification information of, the layer-specific bitstream for each view of the access unit may be identified and divided. The divider 420 selectively selects a desired bitstream fragment using identification information of spatial scalability information, identification information of temporal scalability information, identification information of image quality scalability information, and identification information of a bitstream for each layer in each view. Can be extracted with

The decoder 430 performs multi-view scalable video decoding using the inter-view layer information and the inter-layer information on the view-by-layer layer bitstreams divided by the bitstream divider 420.

The decoder 430 according to an embodiment may perform multi-view video decoding on a video image of a current view with reference to video image information of a view other than the current view among a plurality of views. In addition, the decoder 430 according to an embodiment may use the scalable video using at least one of a video image of a current layer, a video layer information of a lower layer, and inter-layer prediction information between the current layer and a lower layer. Decoding can be performed. The scalable video decoding according to an embodiment may include at least one of spatial scalable video decoding, temporal scalable video decoding, and quality scalable video decoding for a plurality of view-by-layer layered bitstreams.

The decoder 430 according to an embodiment determines whether to perform scalable video decoding according to whether the lower layer video image or the lower layer video image information is present with respect to the video image of the current view, and predicts the view direction. It may be determined whether to perform multiview video decoding according to whether to perform coding. According to the determination result, the decoding performing unit 430 according to an embodiment performs one of a multiview scalable video decoding, a single view scalable video decoding, a multiview video decoding, and a single view video decoding on the current view image. can do.

The decoder 430 according to an embodiment may perform decoding according to the prediction structure for each layer that individually sets the prediction structure for each layer. Therefore, the prediction structure of the current layer can be predicted by selectively referring to the prediction structure of the lower layer.

Accordingly, the decoder 430 according to the prediction structure for each layer may set the prediction structure for each layer and then decode according to the prediction structure for each layer. In addition, the decoding performing unit 430 according to the prediction structure for each layer may repeatedly set the prediction structure and decode each layer repeatedly.

In addition, the decoder 430 according to the prediction structure for each layer determines whether the prediction structure of the current layer of the current view image is the same as the prediction structure of the lower layer of the current view image, and predicts the lower layer according to the determination result. By determining whether to refer to the structure, the prediction structure of the current view image may be decoded. In this case, the referenced lower layer may be a lower layer below one layer of the current layer or a predetermined layer selected from a plurality of lower layers.

If the encoding information of the lower layer does not match the prediction structure of the current layer according to the prediction structure for each layer of the decoding execution unit 430, the decoding execution unit 430 does not use the encoding information of the lower layer or A prediction structure of the current layer may be predicted by referring to only a portion of the encoding information that matches the prediction structure of the current layer.

According to the layered prediction structure, the decoding performing unit 430 may determine the view direction prediction structure of the spatial enhancement layer independently of the view direction prediction structure of the spatial base layer.

The video decoded by the decoder 430 according to an embodiment may be reconstructed into multi-view videos of various formats having various resolutions, various viewpoints, various image quality, and various frame rates. Multi-view video of various formats of various viewpoints, various image quality, and various frame rates may be reproduced by supportable display devices.

By the multi-view scalable video encoding apparatus 300 according to an embodiment, various viewpoints including a conventional two-dimensional display, a stereoscopic display, a multi-view video display, a free view selectable display, QVGA, SD, HD, Full Content in a variety of formats, including various screen sizes including HD, various image quality including VCD, DVD, HDTV, and various temporal resolutions including 5Hz, 15Hz, 30Hz, 60Hz, etc., can be encoded and transmitted as bitstreams. have.

In addition, the multi-view scalable video decoding apparatus 400 according to an embodiment may extract a content by selecting a bitstream corresponding to the content having a desired format from the received bitstreams. Accordingly, content suitable for each environment may be provided to display devices capable of supporting various viewpoints, various screen sizes, various image quality, and various temporal resolutions.

Therefore, according to the multi-view scalable video encoding apparatus 300 and the multi-view scalable video decoding apparatus 400 according to an embodiment, various views, Content suitable for each format can be provided to display devices capable of supporting various screen sizes, various image quality, and various temporal resolutions, and image information of various formats is integrated and processed through one unit of bitstream for efficient transmission and Can be received.

5 illustrates a schematic diagram of an immersive multiview scalable video service that can be implemented according to an embodiment.

The multi-view scalable video encoding apparatus 300 according to an embodiment and the multi-view scalable video decoding apparatus 400 according to an embodiment may support various viewpoints, various resolutions, various image quality, and various frame rates. The immersive video content can be efficiently delivered to various transmission environments and various terminals.

By the realistic multiview scalable video service 500, video content of various formats using the multiview image content 510 of HD resolution may be provided. The encoder operation step 520 of the realistic multiview scalable video service 500 may be implemented by the multiview scalable video encoding apparatus 300 according to an embodiment, and the bitstream extraction step 540 may include It may be implemented by the multi-view scalable video decoding apparatus 400 according to the embodiment.

The encoder operation step 520 generates a bitstream 530 by performing a single encoding on the multi-view video content 510 of HD resolution. The bitstream is transmitted to the bitstream extraction step 540, and the bitstream is extracted to fit the environment to display devices supporting various resolutions, various image quality, various temporal resolutions, and various viewpoints.

Among the extracted video contents, the low quality and fixed view video content may be provided to the small 2D display device 550, and the high definition and fixed view video content may be provided to the small 2D display device 551. The video content may be provided to the SD class 2D display device 552 and the high quality and fixed view video content may be provided to the HD class 2D display device 553 for playback.

In addition, high-definition and view-selectable video content among the extracted video content may be provided to the HD-level two-dimensional display device 554, and the high-definition video content to the small stereo display device 555, the HD-level stereo display device ( 556 and the multi-view display device 558, the low quality video content may be provided to the multi-view display device 557 and played back. The display devices 554, 555, 556, 557, and 558 are realistic display devices, and may play multi-view video content.

Various kinds of applications may be easily supported according to the realistic multi-view scalable video service 500 according to an embodiment. Types of viewpoints include conventional two-dimensional displays, stereoscopic display multi-view video display devices, free viewpoint selectable displays, and the types of image quality include QVGA, SD, HD, and Full HD. There are VCD, DVD, HDTV, and the like, and frame rates include 5 Hz, 15 Hz, 30 Hz, and 60 Hz.

In the ubiquitous computing environment, the realistic multi-view scalable video service 500 according to an embodiment may support various viewpoints, various image quality, various resolutions, and various frame rates in an integrated manner, and efficiently each environment. Video content suitable for.

6 to 8, a method of encoding the multiview scalable video encoding apparatus 400 according to an embodiment will be described in detail.

6 illustrates a structure of an encoder implemented for multiview video encoding according to an embodiment.

6 illustrates a detailed configuration of an encoder according to an embodiment.

Referring to FIG. 6, the encoder 700 generates view points generated from first to Nth sub-encoders 710, 720, 730, and 740 encoding respective first to Nth view video images. And a multiplexer (MUX) 750 for multiplexing the bitstream for each layer to generate an output bitstream. The first to N th denier coders 710, 720, 730, and 740 of FIG. 6 correspond to the encoder 320 of FIG. 3, and the multiplexer 750 of FIG. 6 is connected to the output unit 330 of FIG. 3. Can correspond.

Each denier coder (710, 720, 730, 740) is capable of single-view video coding, spatial scalability to provide various screen resolutions, temporal scalability to provide various frame rates, and various image quality. Scalable video coding is available that supports a variety of scalability features, including quality scalability capabilities. In addition, at least one of these gyrocoders (second to Nth gypsy coders in FIG. 6) may include one or more other images encoded and reconstructed by another gyrocoder (the first gycoder in FIG. 6). Reference encoding is possible. Encoding of the combination of these various functions in various ways may be performed by the respective denier coders 710, 720, 730, and 740.

In an embodiment, as shown in FIG. 6, the denier coders 710, 720, 730, and 740 may exist physically or logically separately according to the number of input images. In this case, the denier coders 710, 720, 730, and 740 may perform each encoding process in parallel.

In another embodiment, the denier coders 710, 720, 730, and 740, as described later in FIG. 25, have only one physically or logically and are required in each image sequentially according to the coding order of each image. It is also possible to perform a form of encoding.

The output data after encoding is performed in each denier coder 710, 720, 730, and 740 (that is, the layer-by-view layer-specific bitstream) is combined through a multiplexer 750 into one integrated bitstream (ie, Multi-view scalable bitstream).

FIG. 7 is a diagram for describing a detailed structure of an encoder 700 for multiview scalable video coding of FIG. 6.

The encoder 800 receives a video image (first to third video image) according to three viewpoints, and as shown in FIG. 7, scalability to support two spatial scalable layers for each viewpoint. Multi-view scalable video coding that supports the function may be performed. In addition, the encoder 800 may support an image quality scalability function and a temporal scalability function for each spatial scalable layer, and perform view direction prediction coding with reference to different view image information.

Each of the first to third non-coding coders 810, 820, and 830 generates down-sampling images of a desired low resolution by performing down-sampling on the input image to support two spatial scalability functions. can do.

In the first denier coder 810, the downsampling unit 811 performs downsampling on the first view video image, and the VC unit 813 encodes the downsampled first view video image. In one embodiment, the VC unit 813 may perform temporal scalability video coding. In this case, the output of the VC unit 813 includes encoding information in which a temporal scalability function is supported for the spatial base layer (that is, the spatial lower layer). An example of a temporal scalability video coding method may be a method using a hierarchical B-pictures structure, such as H.264, but is not limited thereto.

The QS-VC unit 815 is based on at least one of the down-sampled first view video image and the encoding result data (that is, encoding information) of the VC unit 813, and the image quality for supporting the image quality scalability function. Perform scalable video coding. The output of the QS-VC unit 815 corresponds to encoding information in which the image quality scalability function is supported for the spatial base layer or the spatial base layer in which the temporal scalability function is supported.

The SS-VC unit 817 applies spatial scalable video coding to the first view video image. The SS-VC unit 817 performs spatial prediction by performing predictive coding by referring to information (ie, reconstructed video and video video information including encoded information) of the downsampled first view video video that has already been encoded. Encoding information about a layer (ie, a spatial upper layer) may be generated. In addition, the SS-VC unit 817 may selectively perform temporal scalability coding together with spatial scalable video coding. In this case, the output of the SS-VC unit 817 is encoding information in which a temporal scalability function is supported for the spatial enhancement layer.

The QS-VC unit 819 is based on the first view video image and the video image information (eg, reconstructed image and encoding information) provided from the SS-VC unit 817, and the image quality scaler for the first view video image. Perform capability video coding. The output of the QS-VC unit 819 corresponds to encoding information supported by the image quality scalability function for the spatial enhancement layer or encoding information supported by the image quality scalability function to the spatial enhancement layer supported by the temporal scalability function. do.

In the second denier coder 820, the downsampling unit 821 performs downsampling on the second viewpoint video image, and the MVC unit 823 is the first viewpoint video image provided from the first denier coder 810. The downsampled second view video image is encoded by referring to information (ie, video image information). MVC (Multi-view Video Coding, H.264 Amendment 4) method, in addition to H.264 (MPEG-4 Part 10 Advanced Video Coding (AVC)) method, another time point of the same time zone that is already encoded and reconstructed by another encoder Refers to the video information of the encoding method. When performing video coding according to the MVC scheme, it may or may not refer to image information of another viewpoint of the same time zone that is already encoded in each picture unit, picture type unit, or image sequence unit. In one embodiment, similar to the VC unit 813, the MVC unit 823 may perform temporal scalability video coding.

The M & SS-VC unit 827 provides information about the first viewpoint video image provided from the first denier coder 810 as well as the second viewpoint video image, that is, the video image information, and the MVC unit 823. The spatial scalable video coding and the multiview video coding are performed on the second view video image based on the video image information. In other words, unlike the encoding information of the first non-coder 810, the encoding information of the second non-coder 820 reflects the result data of the view direction prediction coding. Since the QS-VCs 825 and 829 are described on the same principle as the QS-VCs 815 and 819, the following description is omitted.

Since the third non-coder 830 is described in the same principle as the second non-coder 820, a description thereof will be omitted.

Data output from each sub-encoder 810, 820, 830 is output from the multiplexer MUX 850 as one integrated bitstream. That is, the multiplexer 850 of FIG. 7 may correspond to the multiplexer 750 of FIG. 6.

Therefore, the encoder 800 receives three viewpoint video images of different viewpoints input from different cameras, and encodes and generates the encoding to support the spatial scalability function, the temporal scalability function, and the image quality scalability function. The integrated bitstream may be provided as a bitstream corresponding to various formats according to multi-view, spatial resolution, frame rate, and image quality.

For example, first, an H.264 bitstream of the resolution of a downsampled image of a first view video image may be provided. In addition, an SVC-based bitstream capable of providing spatial, temporal, and image quality scalability of the first view video image may be provided. In addition, a bitstream of an MVC scheme from two viewpoints to three viewpoints of downsampled resolution may be provided. A bitstream that supports images from two viewpoints to three viewpoints of downsampled resolution and includes various scalability in each image may be provided. In addition, a bitstream including video data from two viewpoints to three viewpoints having the same resolution as each input view video image and having various scalability in each image may be provided.

Further extending from the embodiment of the encoder 800, the multi-view scalable video encoding apparatus 300 according to an embodiment may receive any number of viewpoint video images, and scale provided by each sub-encoder. Bitstreams having a flexible layer may generate a multiview scalable video bitstream composed of more various combinations.

The multi-view scalable video bitstream output in the embodiment according to the encoder 800 may have backward compatibility with various video coding methods. As an example, the output bitstream may be compatible with single view-point display terminals using H.264 (MPEG-4 Part 10 Advanced Video Coding (AVC)), which is a conventional video coding method. . As another example, the output bitstream may be compatible with single-view display terminals using SVC (Scalable Video Coding, H.264 Amendment 3) scheme. As another example, the output bitstream may be compatible with stereoscopic display terminals or multi-view display terminals using MVC (Multi-view Video Coding, H.264 Amendment 4) scheme.

Hereinafter, a structure of a bitstream generated by the multiview scalable video encoding apparatus 300 according to an embodiment will be described with reference to FIGS. 8 through 12.

8 illustrates a structure for combining the output bitstreams according to an embodiment.

The multiplexer 900 of the present embodiment may correspond to the multiplexer 750 of FIG. 6. In one embodiment, the multiplexer 900 receives N (N> 1) bitstreams generated from a plurality of denier coders, decomposes each bitstream into predetermined units, and combines the decomposed bitstreams. One multiview scalable video bitstream may be generated.

The bitstreams input to the multiplexer 900 are encoded in a form that can support various views and various layers (layers according to supported scalability functions). For example, the image may be a bitstream having one image quality at a single point of view, or may be a bitstream supporting various scalability such as spatial scalability, temporal scalability, and image quality scalability. It may also be a bitstream supporting multiple views.

9 illustrates a bitstream generated by multiview scalable video encoding.

The bitstream 1000 is a bitstream including N bitstreams input from the multiplexer 900 as one multiview scalable video bitstream. N bitstreams of bitstream 0, bitstream 1, bitstream 2, and bitstream N are each divided into bitstream fragments of a predetermined unit, and bitstream fragments representing screens of the same time zone are combined to form an entire bitstream. .

For example, bitstream 0, bitstream 1, bitstream 2 to bitstream N are divided into divided bitstream fragments, whereby bitstream fragments 1010, 1020, 1030 to 1040 are combined in sequence for the same time zone, and in chronological order. Accordingly, the bitstream fragments 1050, 1060, and 1070 to 1080 may be combined in the same time slot in order.

The divided bitstream fragment may be, for example, a bitstream corresponding to a picture, or may be a bitstream corresponding to a slice constituting one screen. Each bitstream fragment may be a per-layer layer-specific bitstream identified by one view and one layer.

10 illustrates a configuration of a bitstream generated by multiview scalable video encoding according to an embodiment.

The N bitstreams 1110, 1120, 1130, and 1140 are composed of X sub-access units (Sub-AUs) that are randomly accessible in time. The bitstream output unit 330 according to an embodiment may decompose the bitstreams 1110, 1120, 1130, and 1140 into sub-access units, and combine the substreams of the same time zone into one access unit 1152. , 1154, 1156, and sequentially combine the configured access units 1152, 1154, and 1156 in time to form a multiview scalable video bitstream 1150.

One example of a sub-access unit constituting each bitstream may consist of one picture or slice at a single point in time. In addition, another example of the sub-access unit may be composed of spatial or image quality scalability layers for one picture, and as another example, may be composed of a picture or slice of one of several views. .

11 is a diagram illustrating bitstreams according to layers according to an embodiment, according to an embodiment.

According to an embodiment, the encoder 320 codes three viewpoint video images having different viewpoints in the order of viewpoint 0, viewpoint 2, and viewpoint 1, and each viewpoint includes two spatial scalability layers DId 0. And DId 1), and each spatial scalability layer further includes three bitstreams composed of two image quality scalability layers (QId 0 and QId 1) at a time point 0 bitstream 1200 and a time point 2 bitstream ( 1260), and output in the order of the time point 1 bitstream 1230. A method of configuring a multiview scalable video bitstream when the output unit 330 receives the bitstreams 1200, 1260, and 1230 is described in more detail with reference to FIG. 12.

One block of FIG. 11 represents a NAL unit, and one raw unit is composed of a picture or a slice representing one screen, and a residual signal or spatial space for improving the quality of one intact screen or a lower layer. Residual signals may be included to increase the resolution. The bitstreams of the respective view video images are labeled VId 0, VId 1, and VId 2 in order to distinguish them by view, and the horizontal direction indicates a time sequence and is denoted by T0, T1, T2. The vertical direction represents a spatial and image quality layer of the same time, the spatial base layer is denoted as DId 0, the spatial enhancement layer is denoted as DId 1, the image quality base layer is denoted as QId 0, and the image quality enhancement layer is denoted as QId 1.

Each bitstream consists of sub-access units that can be accessed at any time, which is composed of raw units including spatial and image quality layers at the same time. For example, the day unit 1216 of DId 0 and QId 0 of time T0, the day unit 1211 of DId 0 and QId 1 of T0, the day unit 1206 of DId 1 and QId 0 of T0, and the DId 1 of T0 and Day unit 1201 of QId 1 becomes the sub-access unit for time T0, DId 0 and QId 0 1217 of time T1, DId 0 and QId 1 1212 of T1, DId 1 and QId 0 of T1. 1207, the day unit of DId 1 and QId 1 1202 of T1 becomes the sub-access unit for time T1.

12 illustrates a detailed configuration of a multiview scalable video bitstream according to an embodiment.

In one embodiment, the output unit 330 combines the sub-access units obtained in the bitstream at each time point with the sub-access units at different time points in the same time T, so that It consists of an access unit.

As an example, the access unit 1310 at time T0 may include sub-access units 1216, 1211, 1206, 1201 of T0 of VId0, sub-access units 1246, 1241, 1236, 1231 of T0 of VId1, VId. Two sub-access units 1276, 1271, 1266, 1261. An example of the order of combining the sub-access units into an access unit is sub-access units 1216, 1211, 1206, 1201 of T0 of VId 0, which is the view number order, sub-access units of T0 of VId 1. 1246, 1241, 1236, 1231, and sub-access units 1276, 1271, 1266, 1261 of T0 of VId 2. As another example, the sub-access units 1216, 1211, 1206, 1201 of T0 of VId 0, in which the bitstream is coded, and the sub-access units 1276, 1271, 1266, 1261 of T0 of VId 2. ) May be configured in the order of the sub-access units 1246, 1241, 1236, and 1231 of T0 of VId 1.

Access unit 1320 at time T1 is sub-access units 1217, 1212, 1207, 1202 of T1 of VId0, sub-access units 1247, 1242, 1237, 1232 of T1 of VId1, and T1 of VId2. Of sub-access units 1277, 1272, 1267, and 1262.

The access unit 1330 at time T2 is the sub-access units 1218, 1213, 1208, 1203 of T2 of VId0, the sub-access units 1248, 1243, 1238, 1233 of T2 of VId1, and T2 of VId2. Of sub-access units 1278, 1273, 1268, 1263.

Access unit 1340 at time T3 includes sub-access units 1219, 1214, 1209, 1204 at T3 of VId0, sub-access units 1249, 1244, 1239, 1234 at T3 of VId1, and T3 at VId2. Of sub-access units 1279, 1274, 1269, and 1264.

Access unit 1350 of time TX is sub-access units 1220, 1215, 1210, 1205 of TX of VId0, sub-access units 1250, 1245, 1240, 1235 of TX of VId1, TX of VId2. Of sub-access units 1280, 1275, 1270, 1265.

When each layer of time-specific bitstreams is composed of one or more kinds of layer-specific bitstreams, each time zone-specific access unit may include a layer-specific bitstream.

In one embodiment, the output unit 330 configures the multi-view scalable video bitstream 1330 by arranging the configured access units 1320, 1330, 1340, and 1350 in time order.

Among the multi-view scalable video bitstreams, information necessary for distinguishing sub-access units for each view video image and arranging the sub-access units in the access unit in order may be indicated in the bitstream. As an example of the implementation of the notation method, the syntax shown in Table 1 may be added to the NAL unit header.

nal_unit_header_smvc () { ..... dependency_id quality_id temporal_id Bitstream distinction_id ..... }

The multi-view scalable video encoding apparatus 300 according to an embodiment may include a syntax 'dependency_id' representing spatial scalability information in order to identify a sub-access unit in each bitstream in a raw unit header, and image quality scalability. A syntax 'quality_id' indicating information and a syntax 'temporal_id' indicating temporal scalability information may be added.

In addition, the multi-view scalable video encoding apparatus 300 according to an embodiment may include, in the raw unit header, each raw unit is a bitstream of a certain viewpoint video image in order to distinguish a combination order between viewpoint video images in the access unit. 'Bitstream discrimination_id' may be added to indicate whether the unit is a. An example of 'bitstream discrimination_id' may include 'view_id' indicating identification information of a current view among video images of various views.

Hereinafter, a decoding method of the multiview scalable video decoding apparatus 400 according to an embodiment will be described in more detail with reference to FIGS. 13 to 17.

13 illustrates a structure of an implemented decoder for multiview video decoding according to an embodiment.

The decoder 1400 receives a multiview scalable video bitstream generated by an encoder or a bitstream extracted for a predetermined function from the bitstream. The decoder 1400 extracts and decodes data of an image of a level (various viewpoints, various image quality, various spatial resolutions, and various temporal resolutions) to be provided in the bitstream, and decodes M (N ≥ M ≥ 1) viewpoints. The decoded video may be output. The decoder 1400 may implement the decoder 430 of the multiview video decoding apparatus 400 according to an embodiment.

14 illustrates a structure of an implemented decoder for multiview scalable video decoding according to an embodiment.

The decoder 1500 is an embodiment of implementing the decoder 430 of FIG. 4, wherein a multiview scalable video bitstream or a bitstream selected or extracted from the bitstream for various applications is input to the decoder 1500. do. The input bitstream is separated into data corresponding to M view images through a demultiplexer (DEMUX) 1510. Each separated data is input to one of the corresponding first to Mth decoders 1520, 1530, 1540, and 1550, respectively. The decoders 1520, 1530, 1540, and 1550 decode the respective data to output video images 1 to M decoded for each view.

The decoders 1520, 1530, 1540, and 1550 can decode a bitstream that supports a single viewpoint, and support various scalability functions such as spatial scalability, temporal scalability, and quality scalability. Can be decoded for the bitstream. Also, the decoders 1520, 1530, 1540, and 1550 may perform multi-view video decoding on the bitstream according to predictive decoding referring to image information of one or more different views decoded through other decoders. Can be. The decoders 1520, 1530, 1540, and 1550 are decoders capable of decoding a bitstream in which various functions described above are combined in various ways.

For example, the decoders 1520, 1530, 1540, and 1550 may perform decoding on a bitstream supporting a single view coded according to the H.264 scheme. In addition, the decoders 1520, 1530, 1540, and 1550 may decode a bitstream supporting a spatial scalability function, a temporal scalability function, and a quality scalability function according to the SVC scheme. The decoders 1520, 1530, 1540, and 1550 may perform decoding on the bitstream by referring to image information of another view decoded through another sub-decoder according to an MVC scheme. In addition, the decoders 1520, 1530, 1540, and 1550 may configure a decoder to decode a bitstream having a combination of H.264, SVC, and MVC functions in various ways. Can be.

The sub-coders 1520, 1530, 1540, and 1550 may physically or logically perform decoding according to the shape of each bitstream sequentially in the order in which the image for each view is decoded. Also, the decoders 1520, 1530, 1540, and 1550 may exist physically or logically separately according to the number of images for each view to be output corresponding to the input bitstream. In this case, the decoding operations of the decoders 1520, 1530, 1540, and 1550 may be performed in parallel.

15 illustrates a structure of an implemented decoder for multiview scalable video decoding according to another embodiment.

The decoder 1600 is another embodiment of implementing the decoding performing unit 430 and corresponds to a structure in which the bitstream selection module 1610 is added to the decoder 1500. The bitstream selection module 1610 may be required in the bitstream according to the decoding capability of the current terminal, for example, the screen resolution, the frame rate, the computing performance, the power consumption, the number of views, or the like in the input bitstream. Only parts can be selected.

For example, if the bitstream received from the broadcasting network is a multiview scalable video bitstream that supports spatial resolution of 1920 x 1080, but the current display terminal supports the resolution of 1280 x 720, the bitstream selection module 1610 ) May select only a bitstream corresponding to a resolution of 1280x720 from the received bitstream and perform the remaining decoding operations.

16 is a flowchart of a method of performing decoding in each picture or slice unit in each subdecoder according to an embodiment.

When the decoder 430 according to an embodiment includes a plurality of view-specific sub-decoders, each of the sub-decoders may be processed with respect to an image of the current view in processing units of each picture, slice, etc. according to the flowchart 1700. Decoding can be performed. For convenience of description, decoding is performed on an image of a current view in picture units, but an image processing unit of multi-view scalable video coding according to an embodiment is not limited to a picture, and includes a picture, a slice, a frame, It is possible to cover various processing units used in the technical fields such as fields and blocks.

In step 1710, it is determined whether a lower layer exists for a picture to be currently decoded. If there is a lower layer, the decoding operation proceeds to step 1720 and, if not present, to step 1750.

In operation 1720, it is determined whether a picture to be decoded currently performs view direction prediction between video images of different views decoded by another sub-decoder. If the viewpoint direction prediction is performed, the decoding operation proceeds to step 1730, and if not performed to step 1740.

In step 1730, single view and multiview scalable video decoding is performed on the current picture. That is, for the current picture, decoding is performed to refer to prediction of video image information of lower layers and / or video image information of another viewpoint obtained through decoding of another sub decoder, or is currently decoded by the current sub decoder. And decoding may be performed by referring to the video image information of the current view as the prediction data. For example, single-view and multi-view scalable video decoding may be implemented in a multi-view scalable video coding (MSVC) scheme in which the functions of the SVC scheme and the MVC scheme are combined.

In step 1740, single view scalable video decoding is performed on the current picture. That is, decoding is performed by referring to video image information of a lower layer as prediction data with respect to the current picture, or decoding is performed by referring to video image information of the current layer and the current view that have already been decoded by the current sub-coder as prediction data. Can be. For example, single view scalable video decoding may be implemented in an SVC manner.

In operation 1750, it is determined whether the current picture to be decoded performs view direction prediction with an image of another view decoded by another sub decoder. If the view direction prediction is to be performed, the decoding operation proceeds to step 1760, and if not performed to step 1770.

In step 1760, single view and multiview video decoding is performed on the current picture. That is, decoding is performed by referring to the information of another viewpoint image decoded by another sub-coder as prediction data, or decoded by referring to the information of the current viewpoint video image decoded by the current sub-coder as prediction data. This can be done. For example, single view and multiview video decoding may be implemented in an MVC manner.

In step 1770, single view video decoding is performed on the current picture. That is, decoding may be performed on the current picture by referring to the information of the current view image decoded by the current sub decoder as prediction data. For example, single view video decoding may be implemented in an H.264 manner.

17 illustrates information required for single view and inter-view scalable video decoding, according to an embodiment.

The decoder 1800 according to the single view and the inter-view scalable video decoding scheme according to an embodiment first receives a bitstream of a processing unit such as a picture and a slice. The decoder 1800 performs decoding by using information of a lower layer, image information of a different view, and image information of a current view as prediction data, in preparation for a current layer and a current view of a video image corresponding to a bitstream, Video data such as a decoded picture or slice may be output.

The lower layer information according to an embodiment may include encoding information about various lower layers such as texture information, a motion vector, a reference picture number, and residual data information.

The image information of another viewpoint according to an embodiment may be encoding information about a video image of another viewpoint decoded by a decoder of another viewpoint, and may be a decoded video image itself or a part of the other viewpoint video image. In addition, the image information of another viewpoint may be motion information such as a motion vector or a reference picture number of another viewpoint image, and may include encoding information about various other images.

The image information of the current view according to an embodiment may be encoding information about a video image of the current view decoded by the current decoder for the current view, and may be an image itself or a portion thereof that is already decoded by the current decoder. Also, the image information of the current view includes encoding information of various video views of the current view, such as motion vectors of the current view image, which is already decoded, motion information such as a reference picture number, and information of regions or pixels already decoded in the current image. can do.

Multi-view scalable video encoding according to an embodiment integrates encoding information of multi-view images and generates a bitstream supporting a spatial scalable layer, a temporal scalable layer, and an imageable scalable layer. By multi-view scalable video decoding according to an embodiment, the spatial scalable layer, the temporal scalable layer, and the quality scalable layer correspond to video images of a desired spatial layer, temporal layer, and quality layer from the bitstream. The bitstream can be selectively extracted and decoded.

Hereinafter, the prediction structure for each layer of the encoder 320 and the decoder 430 according to an embodiment will be described in detail with reference to FIGS. 18 to 24.

18 illustrates a flowchart 1900 of a method of setting a prediction structure of a current layer, according to a layer-by-layer prediction structure, according to an embodiment.

In an embodiment, the encoder 320 may set the prediction structure in the time direction and / or the view direction of the current layer for each layer, and then repeat the operation of coding the current layer according to the set prediction structure for each layer. Can be. The decoder 430 also has the same principle as the encoder 320 according to the above-described embodiment, and sets the prediction structure of the time direction and / or the view direction of the current layer for each layer, and then sets the current layer according to the prediction structure. The decoding operation may be performed.

In step 1910, the prediction structure of the current layer is set.

In step 1920, coding of the current layer is performed according to the prediction structure of the current layer set in step 1910.

Steps 1910 and 1920 are repeated by the number of layers. The hierarchy can be any kind of hierarchy or a specific kind of hierarchy.

19 illustrates a flowchart 2000 of another method of setting a prediction structure of a current layer, according to a layer-by-layer prediction structure, according to an embodiment.

In an embodiment, the encoder 320 sets a prediction structure in each time direction and / or a view direction for all layers before performing coding, and then sequentially predicts the structure previously set for each layer. Can be coded according to The decoder 430 also has the same principle as the encoder 320 according to the above-described embodiment, and sets a prediction structure in the time direction and / or the view direction for all layers, and then sequentially advances each layer. It can be decoded according to the set prediction structure.

In step 2010, each layer-specific prediction structure is set for all layers.

In step 2020, coding of the current layer is performed according to the prediction structure set in step 2010. The operations in which coding is performed according to the prediction structure set in step 2010 are sequentially repeated for each layer by the number of layers. The hierarchy can be any kind of hierarchy or a specific kind of hierarchy.

According to the prediction structure for each layer according to an embodiment, coding is performed by performing prediction coding according to a prediction structure of a temporal direction or a view direction that is adaptively set for each scalable layer of various layers such as a spatial layer, a temporal layer, and an image quality layer. Can improve random access performance.

Various schemes can be used to set the prediction structure of each layer. In order to set the prediction structure of each layer, the encoder 320 according to an embodiment may receive a user command and set a prediction structure of a time direction and / or a view direction of the current layer. In addition, the encoder 320 according to an embodiment may follow a predetermined prediction structure for each layer in order to set the prediction structure of each layer. For example, when spatial scalable video coding of two spatial layers is performed, a spatial base layer adopts both a temporal prediction structure and a view direction prediction structure, and a spatial enhancement layer. In the enhancement layer, only a prediction structure in the time direction may be adopted.

The decoder 430 according to an embodiment may set a prediction structure of the current layer based on the information transmitted from the encoder. In addition, the decoder 430 according to an embodiment may set the prediction structure for each layer to a prediction structure that is previously determined by the encoder and the decoder together.

20 is a flowchart 2100 of a method of coding a prediction structure of a current layer by selectively using a prediction structure of a lower layer, according to a layer-by-layer prediction structure, according to another embodiment.

According to the prediction structure for each layer according to another embodiment, the encoder 320 and the decoder 430 according to an embodiment predict the lower layer to code the prediction structure of the current layer. The structure can optionally be used.

In step 2110, it is determined whether to use the prediction structure of the lower layer below one layer for coding the prediction structure of the current layer. If the prediction structure of the lower layer is used, the process proceeds to step 2120, and if not, proceeds to step 2130.

In operation 2120, the prediction structure of the current layer is set to be the same as that of the lower layer, and in operation 2130, the prediction structure of the current layer is separately set.

According to the method of setting the prediction structure of the current layer selectively referring to the prediction structure of the lower layer described above with reference to FIG. 20, the information on whether to use the prediction structure of the lower layer directly in the prediction structure of the current layer may be used. It may be determined whether to refer to the prediction structure. The multi-view scalable video encoding apparatus 300 according to an embodiment may code information on whether to use a prediction structure of a lower layer for coding a prediction structure of a current layer as a flag.

The multi-view scalable video decoding apparatus 400 according to an embodiment extracts flag information from the received bitstream, and when the flag information is 1, the prediction structure of the current layer of the corresponding bitstream may be compared with the prediction structure of the lower layer. In the case of 0, the prediction structure of the current layer may be separately set.

Assuming that the information of the prediction structure is coded for each layer, the multiview scalable video encoding apparatus 300 and the multiview scalable video decoding apparatus 400 according to an embodiment are shown in Table 2 below. You can use syntax.

layer_prediction_structure () { use_lower_layer_pred_struct_flag if (use_lower_layer_pred_struct_flag = = 0) { (Set forecast information for the current layer) ... } }

The syntax 'layer_prediction_structure' for coding the information of the layer-by-layer prediction structure may include 'use_lower_layer_pred_struct_flag' which is flag information on whether to use the prediction structure of the lower layer for coding the prediction structure of the current layer.

If the prediction structure of the current layer directly uses the prediction structure of the lower layer, 'use_lower_layer_pred_struct_flag' is set to 1 so that the prediction structure of the current layer may be set to be the same as the information of the lower layer.

If the prediction structure of the lower layer is not referred to to predict the prediction structure of the current layer, 'use_lower_layer_pred_struct_flag' is set to 0 to separately set prediction information of the current layer.

For example, the prediction structure of the current layer may be both a prediction structure in the view direction and a prediction structure in the time direction. As another example, the prediction structure in the view direction may be a prediction structure only in the view direction, and in another example, the prediction structure in the time direction only. .

FIG. 21 illustrates a flowchart 2200 of another method of coding a prediction structure of a current layer by selectively using a prediction structure of a lower layer, according to the layer-by-layer prediction structure, according to another embodiment. For convenience, the operation of the encoder 320 will be mainly described, and the operation of the decoder 430 corresponding thereto will be described on the same principle.

The encoder 320 according to an embodiment may be another example of a method of selectively using a prediction structure of a lower layer to set a prediction structure of a current layer. The prediction of the Nth lower layer based on the current layer is performed. If the structure is the same prediction structure as the prediction structure of the current layer, the prediction structure of the current layer may be coded using the prediction structure of the Nth lower layer.

In step 2210, it is determined whether to use the prediction structure of a specific layer among the lower layers in the coding of the prediction structure of the current layer. If a prediction layer of a specific layer is used, the process proceeds to step 2220, and if not, proceeds to step 2230.

In step 2220, the prediction structure of the current layer is set equal to the prediction structure of a specific Nth lower layer. In step 2230, the prediction structure of the current layer is set separately.

According to the method of the flowchart 2200, the multi-view scalable video decoding apparatus 400 according to an embodiment decodes information on whether to use a prediction structure of a lower layer in coding of a prediction structure of a current layer, thereby decoding the N th. It may be determined whether to use a prediction structure of a lower layer. The multi-view scalable video encoding apparatus 300 according to an embodiment may code information on whether to use the prediction structure of the Nth lower layer as a prediction structure of the current layer with a flag. If the flag information is 1, the prediction structure of the current layer is coded in the same manner as the prediction structure of the Nth lower layer. If the flag information is 0, the prediction structure of the current layer is not coded in the same manner as the prediction structure of the lower layer. The prediction structure of is set separately.

When the prediction structure of the current layer is coded in the same manner as the prediction structure of the lower layer, information indicating the lower layer to be predicted may be coded. For example, as information indicating a lower layer to be predicted, a value obtained by coding a difference value between a current layer and a lower layer may be coded.

Assuming that the information of the prediction structure is coded for each layer, the multiview scalable video encoding apparatus 300 and the multiview scalable video decoding apparatus 400 according to an embodiment are shown in Table 3 below. You can use syntax.

layer_prediction_structure () { use_lower_layer_pred_struct_flag if (use_lower_layer_pred_struct_flag = = 0) { (Set forecast information for the current layer) ... } else { delta_lower_layer_minus1 } }

The syntax 'layer_prediction_structure' for coding the information of the layer-specific prediction structure may include 'use_lower_layer_pred_struct_flag', which is flag information on whether to use the prediction structure of the lower layer in the coding of the prediction structure of the current layer. If the coding of the prediction structure of the current layer uses the prediction structure of the lower layer, 'use_lower_layer_pred_struct_flag' may be set to 1 and the prediction structure of the current layer may be set equal to the information of the lower layer. If the prediction structure of the current layer does not refer to the prediction structure of the lower layer, 'use_lower_layer_pred_struct_flag' is set to 0 and prediction information of the current layer is set separately.

In addition, when 'use_lower_layer_pred_struct_flag' is 1, 'delta_lower_layer_mius1' indicating a value obtained by subtracting -1 from a difference value between a lower layer to be predicted and the current layer may be coded. For example, when coding the prediction structure of the current layer using the prediction structure of the lower layer, the value of 'delta_lower_layer_minus1' becomes zero.

According to the prediction structure for each layer according to an embodiment, when the information of the prediction structure of the current layer is coded using the prediction structure of the lower layer, a case where the prediction structure of the current layer does not match the prediction structure of the lower layer may occur. Can be.

For example, when performing predictive coding of a block of each current layer in the current picture, if the predictive coding result of the block of the lower layer does not match the prediction structure of the picture of the current layer, the lower layer is used for predictive coding of the block of the current layer. The information of the layer may not be used. For example, if the prediction structure of the picture of the current layer allows only prediction in the temporal direction and the prediction coding result of the block in the lower layer performs prediction in the view direction, the information of the lower layer may not be used in the upper layer block. have.

As another example, when performing prediction coding of blocks of each current layer in the current picture, if the prediction coding result of the blocks of the lower layer does not match the prediction structure of the picture of the current layer, the information of the picture of the current layer may be The current layer may be predicted using only a portion that fits the prediction structure. For example, if the prediction structure of the picture of the upper layer allows only prediction in the time direction and the prediction coding result of the block in the lower layer performs both the prediction in the time direction and the view direction, the information in the lower layer is included in the block of the current layer. Only prediction information in the time direction may be used.

As another example, when performing prediction coding of a block of each current layer in the current picture, if the prediction coding result of the block of the lower layer does not match the prediction structure of the picture of the current layer, the type of the block or Only the partition information of the block is referenced and can be used for predictive coding of the current layer. For example, if the prediction structure of the picture of the upper layer allows only prediction in the temporal direction and the prediction coding result of the block in the lower layer performs prediction in the view direction, the type or block of the block of the block in the lower layer in the current layer. Only the partition information of can be referred to and used.

22 illustrates an example of a prediction structure of a spatial current layer and a lower layer in multiview scalable video coding having four views and two spatial layers, according to an embodiment.

As another example of the prediction structure for each layer, in multi-view scalable video coding supporting two spatial layers, the view direction prediction structure may be adaptively implemented for each layer. The base layer and the enhancement layer correspond to an upper layer and a lower layer, respectively, of two spatial layers. The size of the GOP in the temporal direction is 8, and pictures of views are encoded for four views S0, S1, S2, and S3.

According to multi-view scalable video coding that does not use the prediction structure for each layer, the prediction structures of the spatial base layer 2300 and the spatial enhancement layer 2350 are the same. The spatial enhancement layer does not decode all the lower layers to predict the texture information. Instead, the coding is performed by mainly predicting the motion information according to the prediction structure of the lower layer and the residual data information predicted using the motion information. In general, it is common to borrow the prediction structure of the spatial base layer as it is.

However, if all of the spatial layers 2300 and 2350 are coded according to the same prediction structure, random access performance due to view direction prediction coding, such as multiview video coding, is reduced.

23 illustrates another example of a prediction structure of a spatial current layer and a lower layer in multiview scalable video coding having four viewpoints and two spatial layers, according to an embodiment.

According to multi-view scalable video coding according to the prediction structure for each layer, the prediction structure of the spatial enhancement layer 2450 may be set independently of the prediction structure 2400 of the spatial base layer. The prediction structure 2400 of the spatial base layer is the same as the prediction structure 2300 of the spatial base layer described above, while the prediction structure of the spatial enhancement layer 2450 does not perform view direction prediction for non-anchor pictures. The prediction structure has been adopted. Since the view direction prediction is not performed in the non-anchor picture, an improvement may be expected in terms of random access performance of the spatial enhancement layer.

24 illustrates another embodiment of a prediction structure of a spatial current layer and a lower layer in multiview scalable video coding having four views and two spatial layers, according to an embodiment.

According to the prediction structure for each layer, the prediction structure of the spatial enhancement layer 2550 may also be set independently of the prediction structure 2500 of the spatial base layer. The prediction structure 2500 of the spatial base layer is the same as the prediction structure 2300 of the spatial base layer described above, while the prediction structure of the spatial enhancement layer 2550 is in all view directions regardless of anchor pictures and non-anchor pictures. Prediction structures that do not make predictions are adopted. Since the view direction prediction is not performed in the spatial enhancement layer, the best performance can be said in terms of random access performance.

Since the user does not always need the image of every view, but needs an image of any desired view or views, the random access function is useful in the actual video transmission environment. For example, a user of a terminal supporting one viewpoint needs to select an arbitrary desired viewpoint, and a user of a terminal supporting stereo needs to select two arbitrary viewpoints.

Equation 1 shows the complexity of the viewpoint random access.

C is the number of pictures at the moment you want to access, M is the number of pictures that need to be decoded, including the spatial base layer, and h _base and w _base are the height and width of each picture in the spatial base layer, respectively. H _ehn and w _enh represent the height and width of each picture of the spatial enhancement layer, respectively. r _base represents the frame rate of the spatial base layer, and r _enh represents the frame rate of the spatial enhancement layer. GOP_size represents the size of the time-specific temporal GOP, α is the number of time of continuous reference to the anchor picture of the current point in time, β is the number of time of the non-anchor picture references in a row at the current time.

Applying Equation 1 to the hierarchical prediction structures 2350, 2450, and 2550 of the spatial enhancement layers of FIGS. 22, 23, and 24, all other variables are the same and only α and β have different values. Therefore, the smaller the number α of consecutively referenced viewpoints α of the anchor picture of the current view and the number β of consecutively referenced viewpoints of the non-anchor picture of the current viewpoint are smaller, the number of pictures that must be decoded including the spatial base layer. It can be seen that M is small and, accordingly, the complexity of the viewpoint random access is reduced.

For example, in the prediction structure 2350 of the spatial enhancement layer of FIG. 22, in order to access a picture at the S1 view of the spatial enhancement layer, decoding of the spatial enhancement layer except for motion compensation is performed on the picture at the same S1 view of the spatial enhancement layer. In the same spatial enhancement layer, since the S1 view refers to the S0 view and the S2 view, a complex coding process that requires decoding of all pictures of the S0, S1, and S2 views is required. Therefore, the number α of the continuous reference of the anchor picture of the current view with respect to the viewpoint S1 and the number β of the consecutive reference of the non-anchor picture of the current view are both two.

In contrast, in the prediction structure 2450 of the spatial enhancement layer of FIG. 23, the S1 viewpoint is the number α of consecutively referenced viewpoints of the anchor picture of the current viewpoint and the number β of the consecutive reference points of the non-anchor picture of the current viewpoint is 2 and 0, respectively, and in the prediction structure 2550 of the spatial enhancement layer of FIG. 24, the number α of consecutively referenced viewpoints of the anchor picture of the current viewpoint with respect to the viewpoint of S1 and the viewpoint of continuous reference of the non-anchor picture of the current viewpoint. The number β of all is zero.

Accordingly, the random access complexity is the largest in the prediction structure 2350 of the spatial enhancement layer of FIG. 22, and the random access complexity is the smallest in the prediction structure 2550 of the spatial enhancement layer of FIG. 24. That is, it can be confirmed that the prediction structure for each layer according to an embodiment has an effect in terms of improving random access performance.

25 is a block diagram of a multiview scalable video encoder according to another embodiment.

The multi-view scalable video encoding apparatus 2600 according to another embodiment includes a downsampling unit 2610, an encoding unit 2620, a high quality scalable video coding unit 2630, and a bitstream combination unit 2640. . When the multiview scalable video encoding apparatus 2600 according to another embodiment and the multiview scalable video encoding apparatus 300 according to an embodiment are compared, the downsampling unit 2610, the encoding unit 2620, and the image quality may be compared. The scalable video coding unit 2630 may correspond to the encoding performing unit 320, and the bitstream combining unit 2640 may correspond to the bitstream output unit 330.

The multi-view scalable video encoding apparatus 2600 may exist as one independent device and sequentially receive N images. In addition, a plurality of multiview scalable video encoding apparatuses 2600 may exist in parallel to receive N images.

The input view video image may be input to the downsampling unit 2610 to create a lower layer for supporting spatial scalability. The downsampling unit 2610 may be used to downsample an input image to generate an image of a spatial lower layer. If the spatial scalability is not supported, the input view video image may be input to the encoder 2620 without passing through the downsampling unit 2610. The encoder 2620 may receive input video image information, video information of a lower view, lower layer information, and interlayer layer information.

The other viewpoint image information may be a reconstructed image or coded information of another viewpoint image after encoding another viewpoint image. When different view image information is input, the encoder 2620 may perform multi-view video coding on an image of the current view using the different view image information. In addition, the lower layer information may be encoding information such as a reconstructed image of the lower layer or a motion vector or macroblock type of the lower layer. When lower layer information is input, the encoder 2620 may perform scalable video coding on an image of a current layer by using information of a lower layer.

When inter-layer prediction information is input, the encoder 2620 may perform encoding according to the “layer prediction structure”.

The information encoded by the encoder 2620 may be used to encode an image of another layer or another viewpoint. If the multiview scalable video encoding apparatus 2600 is configured as an independent encoding apparatus, encoding information may be input to the encoding unit 2620 again. If the multi-view scalable video encoding apparatus 2600 is one of a plurality of encoders, the multi-view scalable video encoding apparatus 2600 may be input to an encoder for an image of another layer or a different view.

After encoding by the encoder 2620, if the encoder intends to support image quality scalability, the encoder branches to the image quality scalable video coding unit 2630. The image quality scalable video coding unit 2630 performs image quality scalable video coding using information encoded by the encoding unit 2620 and an image input to the encoding apparatus. The information encoded by the encoder 2620 may include a reconstructed image, coded information, and the like. The information encoded by the image quality scalable video coding unit 2630 may output encoding information for another layer or another view image, similarly to the encoding unit 2620.

The resulting bitstream encoded by the encoder 2620 or the image quality scalable video coding unit 2630 is input to the bitstream combiner 2640. The bitstream combination unit 2640 may include a bitstream encoded by the encoding unit 2620, a bitstream encoded by the image quality scalability encoding unit 2630, a bitstream of an upper or lower layer, and bits of another view image. You can receive a stream.

The bitstream combiner 2640 configures the bitstream according to the bitstream structure described above with reference to FIGS. 10 to 13 by using the received bitstreams, and outputs one realistic multiview scalable video bitstream. Can be. In this case, the bitstream combiner 2640 may use one realistic multiview scalable video bitstream to generate 'bitstream summary information' necessary for selecting or extracting necessary information from the multiview scalable video bitstream in the decoding step. Can be combined in addition.

26 is a block diagram of a multiview scalable video decoder according to another embodiment.

The multi-view scalable video decoding apparatus 2700 according to another embodiment includes a bitstream selector 2710, a bitstream decomposer 2720, and a decoder 2730. When comparing the multiview scalable video decoding apparatus 2700 according to another embodiment with the multiview scalable video decoding apparatus 400 according to an embodiment, the bitstream decomposing unit 2720 corresponds to the splitter 420. In addition, the decoder 2730 may correspond to the decoder 430.

The bitstream input to the multiview scalable video decoding apparatus 2700 may be input to the bitstream selector 2710 or may be branched to the bitstream decomposer 2720. The input bitstream may be a multiview scalable video bitstream.

The bitstream selector 2710 may receive the bitstream selection information and the bitstream and select and output a part of the bitstream using the bitstream selection information. The bitstream selected by the bitstream selector 2710 or the bitstream input to the multiview scalable video decoding apparatus 2700 is input to the bitstream decomposer 2720 or branches to the decoder 2730.

The bitstream decomposition unit 2720 receives the bitstream and the bitstream summary information, decomposes the bitstream using the bitstream summary information, and selects or extracts necessary information from the realistic multiview scalable video bitstream, The extracted information is input to the decoding unit 2730.

The decoder 2730 may receive the undecomposed bitstream from the 'one independent decoder' and decode the bitstream. In addition, the decoder 2730 may be configured of 'multiple decoders', and the 'multiple decoders' may respectively receive a bitstream decomposed by the bitstream splitter 2720 to perform decoding. If composed of 'multiple decoders', each bitstream passed through the bitstream splitter 2720 is input to a corresponding decoder among 'multiple decoders'.

The decoder 2730 may receive, as input, viewpoint image information different from the bitstream, lower layer information, and inter-layer prediction information. Various decoding methods may be performed according to the bitstream input to the decoding unit 2730. If the H.264 method is performed, other view image information, lower layer information, and inter-layer prediction information other than the bitstream may not be used. In addition, when the SVC scheme is performed, decoding may be performed using lower layer information. When the MVC scheme is performed, decoding may be performed using image information of another viewpoint. When the MSVC scheme is performed, decoding may be performed using other viewpoint image information, lower layer information, and inter-layer prediction information.

The information decoded by the decoder 2730 may be output to be used by other decoders as current decoding information for use in decoding another layer or another view image. The current decoding information may include decoding information of the current scalable layer, a decoded image, and the like.

If the decoding unit 2730 is configured independently, the decoding information to be output may be input to the current decoding unit 2730 again, and in the case of a plurality of decoders, the decoding information may be input to other decoders.

27 is a flowchart of a multiview scalable video encoding method according to an embodiment.

In operation 2810, viewpoint-specific images of the plurality of viewpoints, that is, viewpoint video images, are input.

In operation 2820, multi-view scalable video coding using inter-view information and inter-layer information is performed on the plurality of views. Multiview video coding and scalable video coding may be performed on the image of the current view in combination. According to the multi-view scalable video coding, encoding may be performed on an image of the current view by using at least one of an image of the current layer of the current view, information of a lower layer, and inter-layer information between the current layer and the lower layer. have. Multi-view scalable video coding may be encoded according to a hierarchical prediction structure adaptively configured for each layer.

In operation 2830, the coding result bitstream is output by combining the bitstream for each view layer generated by scalable video coding for each of the view video images in the order considering the view and the layer. Time-phased access units that are arranged by the time-phase layered bitstreams of the same time zone may be generated according to the time sequence, and a bitstream in which time-domain access units are integrated into one unit listed according to the time sequence may be output. Time zone access units consist of hierarchical sub-access units.

28 is a flowchart of a multiview scalable video decoding method, according to an embodiment.

In step 2910, a bitstream encoded by a multiview scalable video coding scheme is received.

In operation 2920, the bitstream is divided into view-by-layer layer-based bitstreams based on the plurality of viewpoints and the plurality of layers according to the multiview scalable video coding. Among the received bitstreams, a bitstream requiring decoding may be selected according to a layer and a viewpoint. The received bitstream may be divided into time-phased access units listed in chronological order, and the time-phase access units may be divided into time-phase layered bitstreams of the same time zone in chronological order. From the time-phased access unit, hierarchical sub-access units of the corresponding time zone may be divided.

In operation 2930, multi-view scalable video decoding is performed on the inter-view layer-specific bitstreams using inter-view information and inter-layer information. In multi-view scalable video decoding, decoding is performed using at least one of an image of a current layer of a current view, information of a lower layer, and inter-layer information between the current layer and a lower layer to restore the current layer of the current view. have.

The multi-view scalable video decoding method determines whether multi-view video decoding is performed according to whether the inter-layer scalable video decoding is performed according to the existence of the lower layer and whether the view direction prediction coding is performed, and the current view according to the determination result. One of the multi-view scalable video decoding, the single-view scalable video decoding, the multi-view video decoding, and the single-view video decoding may be performed on the image. Multi-view scalable video decoding may perform prediction decoding according to a prediction structure set for each layer.

The multi-view scalable video encoding method and apparatus according to an embodiment of the present invention may generate a bitstream by performing one-time encoding on multi-view video content having HD resolution. Including various 2D displays, stereoscopic displays, multi-view video display devices, free viewpoint selectable displays, various screen sizes including QVGA, SD, HD, Full HD, VCD, DVD, HDTV, etc. The contents of various formats such as various image quality, various temporal resolution including 5 Hz, 15 Hz, 30 Hz, 60 Hz, etc. may be encoded and transmitted in the bitstream.

In addition, the multi-view scalable video decoding method and apparatus according to an embodiment of the present invention, by selecting a bitstream corresponding to the content of the desired format from the received bitstreams to extract the content to be transmitted to the display devices Can be. Accordingly, content suitable for each environment may be provided to display devices capable of supporting various viewpoints, various screen sizes, various image quality, and various temporal resolutions.

Therefore, by using multi-view scalable video coding according to an embodiment, immersive video content can be efficiently delivered to various transmission environments and various terminals.

The invention can also be embodied as computer readable code on a computer readable recording medium. The computer readable recording medium includes all kinds of recording devices in which packets which can be read by a computer system are stored. Examples of computer-readable recording media include ROM, RAM, CD-ROM, magnetic tape, floppy disks, optical data storage devices, and the like, which are also implemented in the form of carrier waves (for example, transmission over the Internet). Include. In addition, the computer-readable recording medium may be distributed over network-connected computer systems so that computer readable codes can be stored and executed in a distributed manner. In addition, functional programs, codes, and code segments for implementing the present invention can be easily inferred by programmers in the art to which the present invention belongs.

The inventors of the present invention have been described with reference to the embodiments shown in the drawings for clarity, but this is merely exemplary, and those skilled in the art may various modifications and other equivalent embodiments therefrom. Will understand. Therefore, the true technical protection scope of the present invention will be defined by the appended claims.

Claims (39)

(a) encoding a plurality of view-by-view video images, applying multi-view video encoding to at least one of the plurality of view-by-view video images, and scalable video to at least one of the plurality of view-by-view video images Generating encoding per view layer by bit by applying encoding; And
(b) combining the generated per-layer layer-specific bitstreams according to a predetermined order based on the view and the layer to generate an output bitstream;
In step (a),
And setting a prediction structure for each layer and scalable video encoding according to the set prediction structure for each layer, wherein the view direction prediction structure of the spatial enhancement layer is set independently of the view direction prediction structure of the spatial base layer. The video encoding method according to claim 1, wherein the view direction prediction is performed on all pictures of the spatial base layer, and the view direction prediction is not performed on non-anchor pictures or all pictures on the spatial enhancement layer. According to claim 1, wherein the step (a),
And applying multi-view video encoding and scalable video encoding to at least one of the plurality of view-by-view video images. The method of claim 1,
Applying the multiview video coding in step (a) comprises performing inter-view prediction based on video image information of another view. The method of claim 1,
In the step (a), applying the scalable video coding includes encoding a video image of a current layer based on video image information obtained by video encoding for a lower layer and a video image of a current layer. Video encoding method. delete According to claim 1, wherein the step (a),
A video encoding method comprising setting a layer-by-layer prediction structure for all layers and encoding the layer-by-layer according to the set layer-specific prediction structure. According to claim 1, wherein the step (a),
And encoding the layered prediction structure repeatedly for each layer. According to claim 1, wherein the step (a),
Determining whether a prediction structure of a current layer of a current view image is the same as a prediction structure of a lower layer of the current view image; And
And determining whether to use the prediction structure of the lower layer according to the determination result, and encoding the prediction structure of the current layer. The method of claim 8,
And the lower layer is a lower layer below one layer of the current layer or a predetermined layer selected from the plurality of lower layers. delete delete According to claim 1, wherein the step (a),
If the prediction structure of the lower layer does not match the prediction structure of the current layer, the current structure is not referred to by using the prediction structure of the lower layer or by referring to only the portion of the prediction structure of the lower layer that matches the prediction structure of the current layer. A video encoding method comprising predicting a prediction structure of a layer. The method of claim 1,
The scalable video encoding performs at least one of spatial scalable video encoding, temporal scalable video encoding, and quality scalable video encoding on the plurality of view-by-view images. The method of claim 13, wherein step (a) comprises:
Video according to the spatial scalable video encoding, wherein the image of the current layer and the image of the lower layer are generated according to the resolution, and the image of the current layer is predicted with reference to the image of the lower layer. Encoding Method. The method of claim 13,
And according to the temporal scalable video encoding, an image of a current layer and an image of a lower layer are generated according to a frame rate, and predicting the image of the current layer with reference to the image of the lower layer. Video encoding method. According to claim 1, wherein step (b),
Generating time-phase access units arranged in time-layer hierarchical bitstreams of the same time zone according to time order, and outputting a multi-view scalable video bitstream in which the time-phase access units are arranged in time order. Video encoding method comprising a. The method of claim 16, wherein step (b)
When each time-level layered bitstream is composed of a bitstream according to an image quality layer and a spatial layer, each time-phase access unit includes a bitstream for each layer according to an image quality layer and a spatial layer. Video encoding method. The method of claim 16,
As syntax information for the multiview scalable video bitstream, in order to identify a bitstream for each view layer of the access unit, identification information of spatial scalability information, identification information of temporal scalability information, and image quality scalability information And setting the identification information of the bitstream and the identification information of the bitstream per layer. (a) receiving a bitstream;
(b) dividing the received bitstream into layer-wise bitstreams for each view; And
(c) performing multi-view scalable video decoding on the per-layer layered bitstreams using the inter-view information and the inter-layer information,
In step (c),
And decoding according to a prediction structure set for each layer, wherein the view direction prediction structure of the spatial enhancement layer is set independently of the view direction prediction structure of the spatial base layer, and the view direction for all pictures for the spatial base layer. Performing prediction and not performing view direction prediction on non-anchor pictures or all pictures with respect to the spatial enhancement layer. 20. The method of claim 19, wherein step (b) comprises:
Selecting a bitstream to be decoded according to a layer and a viewpoint of the video bitstream. The method of claim 19, wherein step (c) is
Determining whether to perform inter-layer scalable video decoding according to whether an image of a lower layer or information of a lower layer with respect to a current view image is present;
Determining whether to perform multiview video decoding according to whether to perform view direction prediction coding; And
Multi-view scalable video decoding, single-view scalable video decoding, multi-view video decoding, and single for the current view image according to a result of determining whether the scalable video decoding is performed and whether to perform the multiview video decoding. And performing one of the viewpoint video decoding. delete The method of claim 19, wherein step (c) is
A video decoding method comprising setting a layer-by-layer prediction structure for all layers and decoding each layer according to the set layer-specific prediction structure. The method of claim 19, wherein step (c) is
Iteratively for each layer, video decoding method characterized in that for setting and decoding the prediction structure for each layer. The method of claim 19, wherein step (c) is
Determining whether a prediction structure of a current layer of a current view image is the same as a prediction structure of a lower layer of the current view image; And
And determining whether to use the prediction structure of the lower layer according to the determination result and decoding the prediction structure of the current layer. The method of claim 25,
And the lower layer is a lower layer below one layer of the current layer or a predetermined layer selected from the plurality of lower layers. The method of claim 25, wherein step (c) comprises:
If the prediction structure of the lower layer does not match the prediction structure of the current layer, the current structure is not referred to by using the prediction structure of the lower layer or by referring to only the portion of the prediction structure of the lower layer that matches the prediction structure of the current layer. Video decoding method for predicting the prediction structure of the layer. The method of claim 19, wherein step (c) is
Performing multi-view video decoding on an image of the current view with reference to image information of a view other than the current view among the plurality of views. The method of claim 28, wherein step (c) comprises:
Performing the scalable video decoding using at least one of an image of a current layer of the current view, information of a lower layer of the current view, and inter-layer prediction information between the current layer and the lower layer. Video decoding method. The method of claim 29,
The scalable video decoding performs at least one of spatial scalable video decoding, temporal scalable video decoding, and quality scalable video decoding on the plurality of view-by-layer hierarchical bitstreams. Way. delete delete 20. The method of claim 19, wherein step (b) comprises:
Dividing the received multi-view scalable video bitstream into time-phased access units listed in chronological order, and dividing the time-phase access units into time-layer hierarchical bitstreams of the same time zone in chronological order. Video decoding method comprising a. The method of claim 33, wherein step (b) comprises:
And dividing the bitstream for each layer according to the image quality layer and the spatial layer from each time zone access unit. The method of claim 33, wherein step (b) comprises:
Among the syntax information of the multi-view scalable video bitstream, identification information of spatial scalability information, identification information of temporal scalability information, identification information of image quality scalability information, and identification information of the bitstream for each layer of each view may be determined. And identifying and dividing the bitstream for each view layer of the access unit. An encoder for performing multi-view scalable video encoding using the inter-view information and the inter-layer information on the plurality of view-by-view images; And
An output unit for outputting a multi-view scalable video bitstream by combining the view-by-layer layer bitstream generated by the scalable video encoding of the view-by-view images according to the order considering the view and the layer,
The encoder,
A prediction structure is set for each layer and scalable video encoding is performed according to the set prediction structure for each layer, but the view direction prediction structure of the spatial enhancement layer is set independently of the view direction prediction structure of the spatial base layer. And a view direction prediction for all pictures for the layer and a view direction prediction for non-anchor pictures or all pictures for the spatial enhancement layer. A receiver configured to receive a multiview scalable video bitstream;
A splitter configured to divide the multiview scalable video bitstream into viewable layer-specific bitstreams based on a plurality of views and a plurality of layers of the multiview scalable video bitstream; And
A decoder configured to perform multi-view scalable video decoding using the inter-view information and the inter-layer information on the per-layer hierarchical bitstreams;
The decoding unit,
The decoding is performed according to the prediction structure set for each layer, but the view direction prediction structure of the spatial enhancement layer is set independently of the view direction prediction structure of the spatial base layer, and the view direction prediction is performed on all pictures for the spatial base layer. And do not perform view direction prediction on non-anchor pictures or all pictures with respect to the spatial enhancement layer. 19. A computer-readable recording medium having recorded thereon a program for implementing the method of any one of claims 1 to 4, 6 to 9, or 12 to 18. A computer-readable recording medium having recorded thereon a program for implementing the method according to any one of claims 19 to 21, 23 to 30, or 33 to 35.

Images