JP7587391B2 - Video encoding device and program - Google Patents

Description

The present invention relates to a video encoding device and program that encodes and transmits video with ultra-high definition, large screen, and wide viewing angle characteristics.

There is a known video-related content generating device that analyzes an input video signal, detects the position coordinates of video objects related to a video scene, and generates video-related content by associating it with character string data and audio data related to the content of the video that has been prepared in advance (see, for example, Patent Document 1). Therefore, before encoding the video signal, it is possible to obtain the video signal that serves as the original signal, video metadata indicating the position coordinates of video objects, and audio data related to the content of the video.

In addition, various techniques are known for detecting the position coordinates and shape coordinates of a video object from the video frames of an input video signal (see, for example, Patent Document 2).

In recent years, immersive media (highly realistic media) has been expected to be the video media of the future. In addition to immersive media, "Diverse Vision" is being considered as a service that allows anyone to watch and experience a variety of content using their preferred device regardless of time or place (see, for example, Non-Patent Document 1).

These future video systems will provide a highly realistic video experience, and the images used in these video systems will have characteristics such as ultra-high definition, large screens, and wide viewing angles. The amount of data for such images will be extremely large, and to realize services that code and transmit images with the characteristics of ultra-high definition, large screens, and wide viewing angles, an extremely efficient, high-performance video encoding device is required.

In addition, audio coexists in conjunction with video in all video content, not just immersive media. There are various methods for expressing audio data depending on the system, but for future advanced content, systems such as object-based audio systems are being considered in which speakers and sound sources are defined as objects, and various information associated with them, such as playback position information for audio objects, is added as metadata (see, for example, Non-Patent Document 2).

For example, a real-time transmission device for audio metadata adds audio metadata to an audio signal (see, for example, Non-Patent Document 3). The ITU-R specifies the Audio Definition Model (ADM) as an international standard for audio metadata, and the ADM in this audio metadata consists of a content section that describes the program contents, and a format section that describes speaker placement and object playback positions, etc. Metadata that changes over time, such as the playback position of an object, is called dynamic metadata, and can be handled separately from static metadata such as the content section, which does not change throughout the program.

JP 2004-15523 A JP 2001-307091 A

“Diverse Vision: Media Technology from 2030 to 2040”, [online], NHK Broadcasting Technology Research Laboratories, Open House 2019, held on May 30, 2019, [retrieved on September 18, 2020], Internet: https://www.nhk.or.jp/strl/open2019/tenji/e1.html “Object-based acoustics”, [online], NHK Science and Technology Research Laboratories, Giken Dayori, January 2019 issue, [Retrieved September 18, 2020], Internet: https://www.nhk.or.jp/strl/publica/giken_dayori/166/5.html Kubo and Oide, "Development of a real-time transmission device for audio metadata in object-based audio", 2019 Institute of Image Information and Television Engineers Winter Conference, 22A-3, presented on December 13, 2019

As mentioned above, in order to provide services that code and transmit ultra-high definition, large screen, and wide viewing angle video using transmission channels with limited transmission capacity, such as broadcasting and distribution, video coding that compresses data according to the transmission capacity is essential. Previous video coding technologies, such as 8K video coding, have either coded each frame uniformly regardless of the video content, or divided a frame into several regions and assigned the amount of code mainly based on the characteristics of the video signal components, such as the difficulty of coding each region.

Incidentally, it is expected that ultra-high definition, large screen, and wide viewing angle images exceeding 8K will be used in immersive media. When watching such ultra-high definition, large screen, and wide viewing angle images, the viewer does not watch the entire screen uniformly, but mainly focuses on a part of the image depending on the content of the image. For this reason, in order to effectively utilize limited transmission capacity, it is not always necessary to uniformly encode the entire screen, and in many cases, the quality of the image content can be maintained high by maintaining high quality only in the region of interest (ROI) of the image where the viewer is focusing. Therefore, when encoding a video signal, it is effective to control the image quality by increasing or decreasing the bit rate according to the region of interest, which is the area where the viewer is focusing, and to efficiently encode huge amounts of video data.

However, with conventional technology, it is necessary to determine the area that the viewer is gazing at (gaze area) and notify the encoding device of that information to control the video encoding device. Also, the gaze area often differs from viewer to viewer, and the video encoding device must be operated using different controls for each viewer's gaze area.

On the other hand, if the gaze area could be automatically guided according to the video, there would be no need to notify the video encoding device of information indicating the gaze area as additional information. Moreover, this information could be actively utilized as an element that effectively enhances the viewing experience of video and audio content.

For example, if a viewer is called out to or hears a loud sound from a certain direction while watching content, it is a natural reaction for the viewer to turn their gaze in the direction of the sound and move the area of attention. Using this, it is possible to link a video object in the input video with an audio object, define the area around the video object in the direction from which the sound from the audio object is produced as the area of attention, and control the encoding so that the encoding quality of the area of attention is kept higher than other areas, thereby improving the subjective quality.

In other words, there is a need for a video encoding device that can guide the area of interest in a video in accordance with video and audio, and encode the video while maintaining high quality of the video content.

However, while the object-based audio system discussed above is designed to add various pieces of information associated with audio objects, such as playback position information, as metadata, it is not yet possible to link these audio objects to video objects in two-dimensional video and use them in video encoding.

In view of the above problems, the object of the present invention is to provide a video encoding device and program that guides the gaze area of a video in accordance with video and audio, and encodes the video while maintaining high quality of the video content.

The video encoding device of the present invention is a video encoding device that encodes video so as to guide a focus area of the video in accordance with video and audio, and includes a video object position/shape extraction unit that inputs video before encoding processing and extracts position coordinates and shape coordinates of a video object in a video frame to be processed that constitutes the video from accompanying video metadata including information indicating position coordinates of the video object in the video frame or by video analysis, and generates position coordinates in the video frame to be processed for an audio object that is a sound source for each video frame to be processed based on audio metadata accompanying the video, and links the audio object to the nearest video object based on the position coordinates of the video object, and inputs the audio of the audio object in the video frame to be processed or references the video metadata to detect the loudness of the audio object, and extracts the position coordinates and shape coordinates of the audio object in the video frame to be processed. The system is characterized by comprising: an audio position generation unit that associates the position coordinates of a video object linked to an audio object in a video frame with information on the volume of the video object; an induced gaze area generation unit that determines a video object to be gaze-guided among the video objects linked as sound sources for each video frame to be processed based on the position coordinates of the video object linked to the audio object, the volume information of the sound, and the shape coordinates of the video object based on a predetermined criterion; an encoding control information generation unit that controls the amount of code when encoding the video frame to be processed based on the information on the induced gaze area so as to improve the encoded image quality of the induced gaze area more than other areas; and a video encoding unit that encodes the video based on the control of the amount of code and generates an encoded stream including encoding parameters indicating at least the control information on the amount of code.

The video encoding device of the present invention is also characterized in that the viewing environment of the receiver of the encoded stream is assumed and determined in advance, and the audio metadata includes predetermined position information of a sound source for each audio object in the video frame to be processed in a virtual space that is predetermined for the assumed viewing position of the receiver and the assumed position of the display device.

The video encoding device of the present invention is also characterized in that the video objects in the video metadata and the audio objects in the audio metadata are linked in advance before the encoding process.

In the video encoding device of the present invention, the audio metadata includes information indicating the loudness of the audio object.

The video encoding device of the present invention is further characterized in that it is connected online to a receiving system that receives the encoded stream in a bidirectional manner and further comprises a means for acquiring information indicating either or both of the actual viewing position and the viewing direction from the actual viewing position from the receiving system, and the induced gaze area generating unit further comprises a means for determining the induced gaze area by correcting either or both of the assumed viewing position and the direction from the assumed viewing position obtained from the audio metadata.

Furthermore, the program of the present invention is configured as a program for causing a computer to function as the video encoding device of the present invention.

The present invention not only makes it possible to compress the amount of encoded data without reducing subjective quality, but also makes it possible to produce content that more strongly reflects the director's intentions and leads to an effective viewing experience.

1 is a block diagram showing a schematic configuration of a video encoding device according to an embodiment of the present invention; 4 is a flowchart showing a video encoding process in a video encoding device according to an embodiment of the present invention. 3A and 3B are diagrams showing examples of information obtained from video metadata and audio metadata, respectively, in a video encoding device according to an embodiment of the present invention. FIG. 2 is a diagram for explaining the operation of a video encoding device according to an embodiment of the present invention.

(Device configuration)
Hereinafter, the configuration of a video encoding device 1 according to an embodiment of the present invention will be described with reference to the drawings. Fig. 1 is a block diagram showing a schematic configuration of a video encoding device 1 according to an embodiment of the present invention. The video encoding device 1 shown in Fig. 1 is a device that links a video object and an audio object in an input video, determines the periphery of the video object in the direction in which a sound is generated by the audio object as a gaze area, and controls encoding so as to maintain the encoding quality of the gaze area higher than other areas, and includes a video object position/shape extraction unit 2, an audio position generation unit 3, an induced gaze area generation unit 4, an encoding control information generation unit 5, and a video encoding unit 6.

The video object position/shape extraction unit 2 inputs pre-encoding video (a pre-encoding video file shown in units of one video frame or GOP (group of pictures)), extracts the position coordinates and shape coordinates of the video object in the video frame to be processed from the associated video metadata including information indicating the position coordinates of the video object in the video frame, or by video analysis, and outputs these to the audio position generation unit 3 and the induced gaze area generation unit 4, respectively. The position coordinates of the video object in the video frame to be processed can be any position that identifies a representative position of the video object, for example the center of gravity of the video object.

When the video object position/shape extraction unit 2 receives associated video metadata, it obtains the position coordinates of the video object in the video frame being processed from the video metadata, and extracts the shape (shape coordinates) of the video object located at the position coordinates of the video object. When the video object position/shape extraction unit 2 cannot obtain the position coordinates of the video object in the video frame being processed from the video metadata, it extracts both the position and shape of the video object from the video frame being processed. Any method can be used to extract the position and shape coordinates of the video object in the video frame being processed, and the techniques described in Patent Documents 1 and 2, for example, can be used.

The audio position generation unit 3 inputs audio metadata associated with the video, generates position coordinates in the video frame to be processed for the audio object that is considered to be a sound generator based on the audio metadata for each video frame to be processed, selects the video object closest to the position coordinates of the audio object by referring to the position coordinates of the video object obtained from the video object position/shape extraction unit 2, and links the audio object to the video object. The audio position generation unit 3 also inputs the audio of the audio object that is considered to be a sound generator in the video frame to be processed, detects the loudness of the audio object (e.g., average or maximum audio power), and outputs the position coordinates of the video object linked to the audio object in the video frame to be processed and information on the loudness of the sound to the induced gaze area generation unit 4 in association with each other. However, when the audio metadata includes information indicating the loudness of the audio object, audio input can be omitted.

In this embodiment, it is assumed that the video object in the video metadata and the audio object in the audio metadata are not linked at the production stage. The audio metadata includes position information of a sound source that is predetermined for each audio object in the video frame to be processed in a virtual space that is predetermined for the expected viewing position of the receiving side and the expected position of the display device. Therefore, the audio position generation unit 3 can generate position coordinates based on the expected viewing position in the video frame to be processed for the audio object that is a sound source based on the audio metadata, and can link it to the video object in the nearest position based on the position coordinates of the video object. On the other hand, as in the above-mentioned object-based audio system, when the video object in the video metadata and the audio object in the audio metadata are linked in advance at the production stage, such as video production by CG (Computer Graphics), the audio position generation unit 3 omits this linking process and outputs the position coordinates of the video object that is linked in advance to the audio object in the video frame to be processed and the information on the volume of the sound to the induced gaze area generation unit 4 in association with each other.

Based on the position coordinates and volume information of the video object linked to the audio object in the video frame to be processed obtained from the audio position generation unit 3, and the shape coordinates of the video object obtained from the video object position and shape extraction unit 2, the induced gaze area generation unit 4 determines the video object to be gaze-guided among the video objects linked as sound sources, based on predetermined criteria such as the content of the audio such as a "call" and the position and size of the video object, for each video frame to be processed, determines an induced gaze area to surround the shape, and outputs information on the induced gaze area to the coding control information generation unit 5. Here, "surrounding the shape of the video object" may be an area of the shape of the video object to be gaze-guided, an area tracing a few pixels around the shape of the video object, or a predetermined round or angular area including the entire shape of the video object.

The encoding control information generating unit 5 generates encoding control information based on the information on the induced gaze area obtained from the induced gaze area generating unit 4 so as to improve the encoding image quality of the induced gaze area more than other areas, and controls the amount of code when encoding the video frame to be processed in the video encoding unit 6.

The video encoding unit 6 encodes the input video in units of one video frame or one GOP based on the control of the code amount by the encoding control information generation unit 5, generates an encoded stream including encoding parameters (including encoding control information) that at least indicate the control of the code amount, and transmits it to a video decoding device (not shown).

(Device operation)
The video encoding process and operation in the video encoding device 1 of this embodiment will be described below based on Fig. 2 with reference to Figs. 3 and 4. Fig. 2 is a flowchart showing the video encoding process in the video encoding device 1 of one embodiment according to the present invention. Also, Figs. 3(a) and 3(b) are diagrams showing examples of information obtained from video metadata and audio metadata, respectively, in the video encoding device 1 of one embodiment according to the present invention. And Fig. 4 is a diagram explaining the operation in the video encoding device 1 of one embodiment according to the present invention.

Referring to FIG. 2, the video encoding device 1 of this embodiment first inputs pre-encoding video represented in units of one video frame or GOP (group of pictures) by the video object position/shape extraction unit 2 (step S1), and extracts the position coordinates and shape coordinates of the video object in the video frame to be processed from the associated video metadata including information indicating the position coordinates of the video object in the video frame, or by video analysis (step S2). The position coordinates of the video object in the video frame to be processed can be any position that identifies a representative position of the video object, for example the center of gravity of the video object.

Here, when the video object in the video metadata and the audio object in the audio metadata are linked in advance before the encoding process, there is no need to assume and determine the viewing environment on the receiving side in advance. However, in this embodiment, as shown in FIG. 4, the viewing environment on the receiving side of the encoded stream generated by the video encoding device 1 is assumed and determined in advance. For example, in world coordinates (X, Y, Z) with the assumed viewing position as the origin, the position of the display device (display) from the assumed viewing position and the size (H, V) of the video frame Fn corresponding to that display device (display) are defined. In other words, the position coordinates of the video frame Fn are also defined on the world coordinates (X, Y, Z). Note that the example shown in FIG. 4 is just one example, and the definitions of the assumed viewing position and world coordinates (X, Y, Z) can be determined arbitrarily.

For this reason, when the video object position/shape extraction unit 2 receives associated video metadata, it obtains the position coordinates of the video object in the video frame being processed from the video metadata, and extracts the shape (shape coordinates) of the video object located at the position coordinates of the video object. As shown in FIG. 3(a), the video metadata can be such that, for each predefined video object in a video frame Fn (n indicates the frame number), the position coordinates of each video object can be extracted within the size (H, V) of the video frame Fn, such that the center of gravity coordinates of video object number #1 are (h1, v1), the center of gravity coordinates of video object number #2 are (h2, v2), etc.

In addition, when the position coordinates of a video object in a video frame to be processed cannot be obtained from the video metadata, the video object position/shape extraction unit 2 extracts both the position and shape of the video object from the video frame to be processed. In this case, a video object number that identifies each video object in video frame Fn is provisionally set.

Next, the video encoding device 1 of this embodiment inputs audio metadata associated with the video by the audio position generation unit 3, generates position coordinates in the video frame to be processed for the audio object that is considered to be a sound generator based on the audio metadata for each video frame to be processed, and links it to the nearest video object based on the position coordinates of the video object obtained from the video object position/shape extraction unit 2, and inputs the audio of the audio object that is considered to be a sound generator in the video frame to be processed, detects the loudness of the audio object (e.g., average or maximum audio power), and associates the position coordinates of the video object linked to the audio object in the video frame to be processed with the information on the loudness of that sound (step S3). However, if the audio metadata includes information indicating the loudness of the audio object, audio input can be omitted.

In this embodiment, an example will be described assuming that the video object in the video metadata and the audio object in the audio metadata are not linked at the production stage. The audio metadata in this case includes position information of the sound source predefined for each audio object in the video frame to be processed in a virtual space predefined for the expected viewing position of the receiving side and the expected position of the display device. Therefore, as shown in FIG. 3B, the sound source direction can be specified based on the predefined expected viewing position for each audio object in the video frame to be processed. For example, the sound source direction based on the expected viewing position of audio object number #1 is (x1, y1, z1), and the sound source direction based on the expected viewing position of audio object number #2 is (x2, y2, z2) for each predefined audio object in a certain video frame Fn. Since the display position from the expected viewing position is defined in world coordinates (X, Y, Z), the position coordinates of each audio object in a certain video frame Fn can be extracted.

Then, the audio position generation unit 3 can select a video object having position coordinates that are closest to the position coordinates of each audio object in a certain video frame Fn, and link the audio object that is the sound generator to the video object. Here, as shown in FIG. 4, for convenience of explanation, video object #1 is linked to audio object #1, and video object #1 is linked to audio object #1. In other words, the audio position generation unit 3 selects a video object having position coordinates that are closest to the position coordinates of each audio object in a certain video frame Fn, whether it is a provisionally set video object number or a predefined video object number, and can link the audio object that is the sound generator to the video object. Note that the number of objects to be linked is not limited to one, and may be multiple.

The audio position generation unit 3 detects the loudness (e.g., average or maximum sound power) of the audio object from the input audio, and can associate the position coordinates of the video object linked to the audio object in the video frame to be processed with the loudness information. On the other hand, as in the above-mentioned object-based audio system, when the video object in the video metadata and the audio object in the audio metadata are linked in advance at the production stage, the audio position generation unit 3 can omit this linking process and associate the position coordinates of the video object linked in advance to the audio object in the video frame to be processed with the loudness information.

Next, in the video encoding device 1 of this embodiment, the induced gaze area generation unit 4 determines, based on the position coordinates and sound volume information of the video object linked to the audio object in the video frame to be processed obtained from the audio position generation unit 3, and the shape coordinates of the video object obtained from the video object position and shape extraction unit 2, a video object to be gaze-induced from among the video objects linked as sound sources, based on a predetermined criterion, for each video frame to be processed, and determines an induced gaze area to surround the shape of that video object (step S4).

For example, the induced gaze area generating unit 4 ranks which video objects are likely to be gazed at based on predetermined criteria such as the content of the voice such as a "call" and the position and size of the video object, and determines the video objects within a predetermined rank (for example, within two as shown in FIG. 4) from the highest ranked ones determined to be likely to be gazed at as the induced gaze area for guiding the gaze (determined to surround the shape of the video object). Note that in FIG. 4, the induced gaze area is a predetermined angular area that includes the entire shape of the video object, but it may be an area of the shape of the video object itself that is to be gazed at, an area that traces several pixels around the shape of the video object, or a predetermined circular area that includes the entire shape of the video object. In this way, the induced gaze area generating unit 4 can determine which video object to guide the gaze to. In addition, it is preferable to limit the number of induced attention areas to a predetermined number, such as one or more, in order to reduce the amount of code, and when multiple induced attention areas overlap each other, it is preferable to redefine them as a single induced attention area in terms of processing load and quality.

Next, in the video encoding device 1 of this embodiment, the encoding control information generation unit 5 controls the amount of code when encoding the video frame to be processed in the video encoding unit 6 based on the information on the induced attention area obtained from the induced attention area generation unit 4 so as to improve the encoded image quality of the induced attention area more than other areas. Then, based on the control of the amount of code by the encoding control information generation unit 5, the video encoding unit 6 encodes the input video in units of one video frame or one GOP, generates an encoded stream including encoding parameters (including encoding control information) indicating at least the control information of the amount of code, and transmits it to the video decoding device (not shown) (step S5).

In this way, the coding control information generator 5 controls the amount of code so as to improve the coding quality of the induced attention area more than other areas. Specifically, one method is to allocate a larger amount of code to the attention area.

Then, the video encoding unit 6 encodes the input video by controlling the code amount by the encoding control information generation unit 5, and outputs the encoded stream together with the encoding parameters to the outside. Any encoding method may be used, but it is assumed that the code amount (value of the quantization parameter) can be changed depending on the position on the screen. The value of the quantization parameter can be transmitted as an encoding parameter. The area to be changed is assumed to be in units of blocks, etc., according to the encoding process.

Although not shown in the figure, the video decoding device that receives and decodes the encoded stream transmitted from the video encoding device 1 of this embodiment may be configured to decode the encoded stream based on the encoding parameters obtained from the video encoding device 1, and may use a device similar to that used for existing video decoding processes.

In this way, the video encoding device 1 of this embodiment is configured to determine and encode an induced gaze area for video encoding based on audio information (audio metadata and audio input), thereby reducing the amount of encoded data while maintaining subjective quality, and providing an effective viewing experience that more strongly reflects the director's intentions on the receiving side.

Therefore, the video encoding device 1 according to the present invention not only makes it possible to compress the amount of encoded data without reducing subjective quality, but also makes it possible to produce content that more strongly reflects the director's intentions and leads to an effective viewing experience.

The video encoding device 1 according to the present invention can be configured by a computer, and a program for making each processing unit of the video encoding device 1 function can be suitably used. Specifically, a control unit for controlling each processing unit of the video encoding device 1 can be configured by a central processing unit (CPU) in a computer, and a storage unit for appropriately storing a program required for operating each processing unit can be configured by at least one memory. That is, by having such a computer execute the program by the CPU, the function of each processing unit of the video encoding device 1 can be realized. Furthermore, a program for realizing the function of each processing unit of the video encoding device 1 can be stored in a predetermined area of the above-mentioned storage unit (memory). Such a storage unit can be configured by a RAM or ROM inside the device, or can be configured by an external storage device (for example, a hard disk). In addition, such a program can be configured as part of the software on the OS used by the computer (stored in a ROM or an external storage device). Furthermore, a program for making such a computer function as each processing unit of the video encoding device 1 can be recorded on a computer-readable recording medium. Additionally, each processing unit of the video encoding device 1 can be configured as part of hardware or software, and can be realized by combining each part.

Although the present invention has been described above by giving examples of specific embodiments, the present invention is not limited to the above-mentioned embodiments, and various modifications are possible without departing from the technical concept thereof. For example, in the above-mentioned embodiment, an example was mainly described in which the acoustic level of each audio object is detected based on audio input, but by making the audio metadata describe the acoustic level of each audio object, it is possible to omit the audio input and detect the video object that is the sound source.

The video encoding device 1 according to the present invention can also be configured to be connected online to enable two-way communication with a receiving system (including a video decoding device, a display device, and peripheral devices) that receives the encoded stream generated by the video encoding device 1. In this case, the video encoding device 1 further includes a means for acquiring information indicating either or both of the "actual viewing position" and the "viewing direction from the actual viewing position" by using an upstream line from the receiving system to the video encoding device 1. The induced gaze area generating unit 4 in the video encoding device 1 can also determine the induced gaze area by correcting either or both of the "assumed viewing position" and the "direction from the assumed viewing position" obtained from the audio metadata.

For example, in a configuration in which the video encoding device 1 according to the present invention and a receiving system (including a video decoding device, a display device, and peripheral devices) are connected online to enable two-way communication, a configuration can be adopted in which VR (Virtual Reality) goggles are used as peripheral devices in the receiving system. In this case, when the gaze direction is obtained from an inertial measurement unit (IMU) such as an acceleration sensor and a gyroscope provided in the VR goggles, the induced gaze area generation unit 4 in the video encoding device 1 can determine the induced gaze area by correcting the "direction from the expected viewing position" obtained from the audio metadata.

In addition, in a configuration in which the video encoding device 1 according to the present invention and a receiving system (including a video decoding device, a display device, and peripheral devices) are connected online to enable two-way communication, a receiving environment detection sensor that measures the relative positional relationship between the display device and the actual viewing position can be provided as a peripheral device in the receiving system. In this case, the induced gaze area generation unit 4 in the video encoding device 1 can be configured to measure the actual viewing position using this receiving environment detection sensor and transmit information indicating the actual viewing position to the video encoding device 1 side using an upstream line, thereby correcting the "assumed viewing position" obtained from the audio metadata to determine the induced gaze area.

The present invention not only makes it possible to compress the amount of encoded data without reducing subjective quality, but also enables the production of content that more strongly reflects the director's intentions and leads to an effective viewing experience, making it useful for the encoding and transmission of video with ultra-high definition, large screen, and wide viewing angle characteristics.

Reference Signs List 1 Video encoding device 2 Video object position/shape extraction unit 3 Audio position generation unit 4 Guidance gaze area generation unit 5 Encoding control information generation unit 6 Video encoding unit

Claims (6)

A video encoding device that encodes a video so as to guide a gaze area of the video according to the video and audio,
a video object position/shape extraction unit that receives a video before encoding processing and extracts position coordinates and shape coordinates of a video object in a video frame to be processed that constitutes the video from associated video metadata including information indicating position coordinates of the video object in the video frame or by video analysis;
an audio position generation unit which generates position coordinates in the video frame to be processed for an audio object that is a sound source for each video frame to be processed based on audio metadata accompanying the video, links the audio object to a nearest video object based on the position coordinates of the video object, inputs the audio of the audio object in the video frame to be processed or detects the loudness of the audio object by referring to the video metadata, and associates the position coordinates of the video object linked to the audio object in the video frame to be processed with information on the loudness of the audio object;
an induced gaze area generating unit which, based on the position coordinates of the video object associated with the audio object, information on the volume of the sound, and the shape coordinates of the video object, determines a video object to be gaze-induced from among the video objects associated as sound sources for each video frame to be processed, based on a predetermined criterion, and determines an induced gaze area to surround the shape of the video object;
a coding control information generating unit that controls a code amount when coding a video frame to be processed based on information of the induced attention area so as to improve the coding image quality of the induced attention area more than other areas;
a video encoding unit that encodes the video based on the control of the code amount and generates an encoded stream including encoding parameters that indicate at least control information of the code amount;
A video encoding device comprising: A viewing environment on a receiving side of the encoded stream is assumed and determined in advance,
The video encoding device described in claim 1, characterized in that the audio metadata includes position information of a sound source predetermined for each audio object in the video frame to be processed in a virtual space predetermined for the expected viewing position of the receiving side and the expected position of the display device. The video encoding device according to claim 1, characterized in that the video objects in the video metadata and the audio objects in the audio metadata are linked in advance before the encoding process. The video encoding device according to any one of claims 1 to 3, characterized in that the audio metadata includes information indicating the loudness of an audio object. The apparatus further includes a means for acquiring information indicating an actual viewing position and/or a viewing direction from the actual viewing position, the information being connected online to a receiving system that receives the encoded stream and capable of bidirectional communication, from the receiving system;
The video encoding device according to claim 1 , wherein the induced gaze area generation unit further comprises means for determining the induced gaze area by correcting either or both of the assumed viewing position and the direction from the assumed viewing position obtained from the audio metadata. A program for causing a computer to function as a video encoding device according to any one of claims 1 to 5.