WO2016167160A1

WO2016167160A1 - Data generation device and reproduction device

Info

Publication number: WO2016167160A1
Application number: PCT/JP2016/061159
Authority: WO
Inventors: 北浦　竜二
Original assignee: シャープ株式会社
Priority date: 2015-04-17
Filing date: 2016-04-05
Publication date: 2016-10-20

Abstract

The present invention makes it possible for the viewer of a dynamic image for display that is cut out from an omnidirectional dynamic image generated by a moving shot, to confirm the state of each visual field around the same time when a plurality of different directions are seen from the same shooting spot. A data generation device (100) is provided with a basic-bearing data generation unit (111) for generating, as to each frame of an omnidirectional dynamic image, basic-bearing data to be referenced by a reproduction device for the purpose of cutting out an image to be displayed from the frame, and an extended-bearing data generation unit (112) for generating extended-bearing data to be referenced by the reproduction device for the said purpose.

Description

Data generating apparatus and reproducing apparatus

The present invention mainly relates to a data generation device that generates data relating to an omnidirectional image and a reproduction device that reproduces an image cut out from the omnidirectional image.

In recent years, a method using a camera equipped with a wide-angle fisheye lens or an omnidirectional mirror, a method of photographing the same scene from a plurality of cameras with different viewpoint positions, and connecting the captured images of each camera, etc. An omnidirectional image with a field of view can be taken. The image provider uses a large number of omnidirectional images obtained by shooting at various locations, so that the viewer can view the field of view according to an arbitrary viewpoint position and an arbitrary line-of-sight direction specified by the viewer. Can be browsed.

For example, in Patent Document 1 below, image data indicating the state of the field of view is extracted from omnidirectional images at various viewpoint positions held by the supply center, and the extracted image data is transmitted to the terminal device of the viewer. Techniques to do this are disclosed. Google Inc. (Headquarters: Google Inc., California, USA) is a "Street View (registered trademark)" still image that shows the state of view according to any viewpoint position and any line-of-sight direction on the road. A service that displays is displayed.

Japanese Patent Publication “Japanese Laid-Open Patent Publication No. 2001-008232 (published on January 12, 2001)” Japanese Patent Publication “JP 2013-090257 A (published May 13, 2013)”

The conventional image display technique using an omnidirectional image in Patent Document 1 and “Street View” is a technique for presenting a still image. That is, in Patent Document 1 and “Street View”, when a viewer designates an arbitrary line-of-sight direction from an arbitrary viewpoint position, a still image corresponding to the designation is presented.

Due to the fact that the image viewed by the viewer is not a moving image taken while moving, the technique of Patent Document 1 and the technique of “street view” have the following problems.

In other words, even if a viewer browses a plurality of still images obtained by “photographing at various points in an area where the scenery can be seen”. There is a problem that it is difficult to get a sense of realism as if you are in a visible area. In addition, there is a problem that it is difficult to feel the fun of the content from the image viewed by the viewer.

By the way, the invention of Patent Document 2 is based on an operation of moving the current location on a map along a virtual route, and a plurality of images obtained by photographing the surroundings while moving a real-world route corresponding to the virtual route. The following processing is performed for omnidirectional still images.

That is, in the invention of Patent Document 2, “a visual field image showing a visual field in the sight line direction designated by the viewer is cut out from the omnidirectional still image for the plurality of omnidirectional still images, and the cut out visual field image is displayed. By performing the “processing”, a moving image showing the surroundings viewed from the shooting point at each shooting time is displayed.

This technique of Patent Document 2 solves the problems of the technique of Patent Document 1 and the “street view” technique described above to some extent.

Moreover, in the invention of Patent Document 2, when a viewer who is browsing the moving image alone desires to see the state of a certain gaze direction, an operation to change the gaze direction is performed as necessary. Thus, it is possible to see the state of the certain gaze direction.

However, in the invention of Patent Document 2, a viewer who wants to confirm the state of view when viewing a certain direction from a certain point and the state of view when viewing another direction from the same point at the same time. There is a problem that the request cannot be satisfied. This is because the invention disclosed in Patent Document 2 cannot display the visual field image showing the former visual field and the visual field image showing the latter visual field at the same time.

The present invention has been made in view of the above problems, and its main purpose is to allow viewers of a moving image for display cut out from an omnidirectional moving image generated by moving shooting to have a plurality of different images from the same shooting point. An object of the present invention is to realize a data generation device that generates data that enables the state of each field of view when viewing a direction to be confirmed at the same time. It is also included in the object of the present invention to realize a playback device that allows the viewer to check the state of each field of view when viewing a plurality of different directions from the same shooting point.

A data generation device according to an aspect of the present invention provides a plurality of images that a playback device should refer to in order to extract a plurality of images to be displayed from a target frame for all or a part of frames of an omnidirectional video generated by moving shooting. A data generation unit for generating the data, wherein the data generation unit generates azimuth data indicating the azimuth for each of a plurality of different azimuths, thereby generating a plurality of azimuth data as the plurality of data, Each of the plurality of azimuth data is referred to by the playback device to cut out an image showing a field of view when viewing the azimuth indicated by the azimuth data from the shooting point of the target frame as the display target image. Data.

A playback device according to an aspect of the present invention includes a plurality of azimuth data to be referred to in order to extract a plurality of images to be displayed from a target frame for all or some frames of an omnidirectional video generated by moving shooting. A data reference processing unit to be referenced, and each of the plurality of azimuth data includes a field of view when the data reference processing unit views the azimuth indicated by the azimuth data from the shooting point of the target frame as the display target image. This is data that should be referred to in order to cut out an image showing the above situation, and includes a reproduction processing unit that reproduces a plurality of images cut out from the target frame for all or some of the frames.

The data generation device according to an aspect of the present invention is a view of each field of view when viewing a plurality of different directions from the same shooting point to a viewer of a display moving image cut out from an omnidirectional moving image generated by moving shooting. It is to realize a data generation apparatus that generates data that enables confirmation at the same time.

Also, the playback device according to an aspect of the present invention has an effect that the viewer can check the state of each field of view when viewing a plurality of different directions from the same shooting point at the same time.

It is a block diagram which shows the structure of the free viewpoint moving image data generator which concerns on Embodiment 1 of this invention. It is explanatory drawing for demonstrating the map data which the said free viewpoint moving image data production | generation apparatus uses. It is a flowchart which shows an example of operation | movement of the said free viewpoint moving image data generation apparatus. It is a figure which consists of two flowcharts which showed two specific processes in the flowchart of FIG. 3 in detail. An example of a visual field image showing a state of view when a certain direction is seen from a certain photographing point, which is cut out from an omnidirectional image photographed at a certain photographing point by the free viewpoint moving image reproducing device according to the second embodiment of the present invention. FIG. The content creator who is the user of the free viewpoint video data generation apparatus causes the free viewpoint video data generation apparatus to generate metadata for causing the free viewpoint video playback apparatus to reproduce a recommended scene by any operation. It is a figure for demonstrating. It is a figure which shows that the subject which a content producer recommends browsing exists in the circumference | surroundings of a certain area in an imaging | photography route. It is a figure which shows an example of the visual field image which shows the mode of a visual field when it sees another direction from this certain imaging | photography point cut out from the omnidirectional image image | photographed at the said certain imaging | photography point with the said free viewpoint moving image reproduction apparatus. . It is the figure which showed typically an example of the data structure of free viewpoint moving image data. It is the figure which showed typically an example of the data structure of the metadata contained in free viewpoint video data. It is the figure which showed typically another example of the data structure of the metadata contained in free viewpoint moving image data. It is a block diagram which shows the structure of the said free viewpoint moving image reproduction apparatus. It is a flowchart which shows an example of operation | movement of the said free viewpoint moving image reproduction apparatus. It is the flowchart which showed the specific process in the flowchart of FIG. 13 in detail. It is the figure which illustrated the display mode of the visual field image by the said free viewpoint moving image reproducing device set to the 1st display mode. It is a flowchart which shows another example of operation | movement of the said free viewpoint moving image reproduction apparatus. It is the flowchart which showed the specific process in the flowchart of FIG. 16 in detail. It is the figure which illustrated the display mode of the visual field image which shows the mode of the field of view at the time of seeing the above-mentioned certain direction from the above-mentioned certain photography point by the above-mentioned free viewpoint animation reproducing device set as the 2nd display mode. FIG. 17 is a flowchart showing another specific step in the flowchart of FIG. 16 in detail. It is the figure which illustrated the display mode of the visual field image which shows the mode of a field of view at the time of seeing the above-mentioned other direction from the above-mentioned certain photography point by the above-mentioned free viewpoint animation reproducing device set as the 2nd display mode. It is the schematic of the system which concerns on Embodiment 3 containing the free viewpoint moving image data production | generation apparatus which concerns on Embodiment 1, and the free viewpoint moving image reproducing device which concerns on Embodiment 2. FIG.

<Embodiment 1>
Embodiment 1 which is a preferred embodiment of a data generation apparatus according to the present invention will be described below in detail with reference to the accompanying drawings. In addition, the structure which attached | subjected the same code | symbol also in different drawing shall be abbreviate | omitted, and the description is abbreviate | omitted.

(Outline of free viewpoint video data generator)
The free viewpoint moving image data generation device according to the present embodiment generates free viewpoint moving image data that is referenced by a free viewpoint moving image playback device according to a second embodiment to be described later for playing back a free viewpoint moving image.

The free viewpoint moving image data generated by the free viewpoint moving image data generating device includes omnidirectional moving image data input to the free viewpoint moving image data generating device. This omnidirectional video data is generated by shooting (moving shooting) while moving a shooting route including a plurality of shooting points.

Noteworthy in the present embodiment is that the following metadata is included in the free viewpoint video data generated by the free viewpoint video data generation device.

That is, the metadata that allows the user of the free viewpoint video playback device according to Embodiment 2 to confirm the state of each field of view when viewing a plurality of different directions from the same shooting point at the same time is free viewpoint video. It is included in the data. Hereinafter, this metadata is also referred to as reproduction control data.

Note that specific examples of the free-viewpoint video data generating device according to the present embodiment include a broadcasting device that generates the free-viewpoint video data, a server on the cloud, and software that generates the free-viewpoint video data. An installed PC (Personal Computer) or the like can be mentioned.

(Configuration of free viewpoint video data generation device)
FIG. 1 is a block diagram showing the configuration of a free-viewpoint video data generation device 100 (hereinafter abbreviated as “data generation device 100”) according to an embodiment of the present invention. First, an outline of the configuration of the data generation device 100 will be described with reference to this figure.

The data generation device 100 includes a control unit 110 and an operation reception unit 120.

The control unit 110 is a CPU and controls the entire data generation apparatus 100 in an integrated manner.

The operation reception unit 120 is an operation device that receives an operation by a content creator (a user of the data generation device 100).

The control unit 110 executes a specific program, thereby performing a basic azimuth data generation unit 111 (data generation unit), an extended azimuth data generation unit (data generation unit) 112, a shooting point data conversion processing unit 113, and an encoding process. Unit 114, reproduction control data generation unit 115, and multiplexing processing unit 116.

Note that the encoding processing unit 114 and the multiplexing processing unit 116 may be realized by hardware (LSI) instead of software.

For each image frame (each omnidirectional image) of omnidirectional video data input from the outside, the basic azimuth data generation unit 111 cuts out the visual field image to be reproduced from the target frame by the free viewpoint video reproduction device according to the second embodiment. Data to be referred to is generated.

This field-of-view image is an image showing the state of the field of view when viewing a certain direction (the direction specified by the content creator or the default direction) from the shooting point of the target frame.

Specifically, the basic azimuth data generation unit 111 generates azimuth data indicating the certain azimuth (hereinafter referred to as basic azimuth data) as the data. Specific contents of the basic azimuth data will be described later.

When the content creator specifies the certain direction, the subject that the user wants to show most to the user of the free-viewpoint video playback device according to the second embodiment is selected from the various subjects included in the target frame. The certain direction is designated so as to be included in the field-of-view image.

The extended azimuth data generation unit 112 generates, for each image frame of the omnidirectional video data, data to be referred to so that the free viewpoint video playback device according to the second embodiment cuts out the playback target view image from the target frame.

This field-of-view image is an image showing the state of the field of view when another azimuth (the azimuth designated by the content creator or the default azimuth) is viewed from the shooting point of the target frame.

Specifically, the extended azimuth data generation unit 112 generates azimuth data indicating another azimuth (extended azimuth data indicating an azimuth different from the azimuth indicated by the basic azimuth data) as the data. Specific contents of the extended azimuth data will be described later.

The extended azimuth data generation unit 112 may generate only one extended azimuth data as the extended azimuth data related to the target frame, or may generate a plurality of extended azimuth data indicating different azimuths.

The shooting point data conversion processing unit 113 outputs map data input from the outside to the multiplexing processing unit 116.

Further, for each frame of the omnidirectional video data, the shooting point data conversion processing unit 113 corresponds to the shooting point in the map indicated by the map data input from the outside with the GPS coordinate data indicating the shooting point of the frame. It is converted into coordinate data (map coordinate data) indicating the position.

The GPS coordinate data may be included in the omnidirectional video data as additional information of the omnidirectional video data. That is, the GPS coordinate data may be generated by this camera when shooting an omnidirectional video by a camera incorporating a GPS module.

In addition, as additional information of the omnidirectional video, a type of “coordinate data indicating a shooting point” different from the GPS coordinate data may be included. For example, the coordinate data may be generated by using a GPS module and a gyro sensor and / or a vehicle speed sensor in combination, or may be manually input as additional information of an omnidirectional video.

Here, the map data will be described with reference to FIG. FIG. 2 is an explanatory diagram for explaining the map data.

The map data representing the map 10 in FIG. 2 is two-dimensional map image data indicating a map of the area where the above-described shooting route is located. As map data, map data, such as a sightseeing map and a road map, is mentioned, for example.

In the present embodiment, the map coordinate data (shooting point data) corresponding to the shooting point of the first frame of the omnidirectional video data is a coordinate indicating the point 11 on the map indicated by the map data in FIG. And Also, map coordinate data (shooting point data) corresponding to the shooting point of the last frame of the omnidirectional video data is assumed to be a coordinate indicating the point 13 on the map indicated by the map data in FIG.

Furthermore, each map coordinate data (photographing point data) corresponding to the photographing points of the remaining frames of the omnidirectional video data is coordinates on a line 12 connecting the

points

11 and 13 in FIG. .

In this embodiment, the shooting point data conversion processing unit 113 is configured to perform the process of converting the GPS coordinate data indicating the shooting point of the target frame into map coordinate data for all frames. You may perform only about some frames (key frame).

In this case, the shooting point data conversion processing unit 113, for each frame that is not a key frame, map coordinate data indicating the shooting point of the target frame, map coordinate data indicating a shooting point of the key frame immediately before the target frame, and the target You may produce | generate by the interpolation process using the map coordinate data which shows the imaging | photography point of the key frame immediately after a flame | frame.

For example, the shooting point data conversion processing unit 113 may indicate that “the camera that shoots an omnidirectional video has the immediately preceding key frame over a period from the shooting time of the immediately preceding key frame to the shooting time of the immediately following key frame. Map coordinate data indicating a shooting point of a frame that is not a key frame may be generated on the assumption that it has moved at a constant speed on a line segment connecting the shooting point of the key frame and the shooting point of the immediately following key frame.

Alternatively, the shooting point data conversion processing unit 113 generates map coordinate data indicating the shooting point of the target frame based on speed information (speed vector) input by the content creator for the target frame for each frame that is not a key frame. May be.

That is, the shooting point data conversion processing unit 113 indicates that “a camera that shoots an omnidirectional video has moved from the shooting point of the frame immediately before the target frame at a speed indicated by the speed information for a certain period of time. On the assumption that the shooting point has been reached, map coordinate data indicating the shooting point of the target frame may be generated. The fixed time may be a time corresponding to a quotient obtained by dividing the total shooting time of the omnidirectional video by the total number of frames n of the omnidirectional video.

The map coordinate data regarding the frame is composed of the frame number of the frame and the map coordinate value indicating the shooting point of the frame, but using the shooting time of the frame indicated by the frame number instead of the frame number. Also good.

The encoding processing unit 114 encodes omnidirectional moving image data input from the outside, and outputs the encoded omnidirectional moving image data to the multiplexing processing unit 116.

Here, as the encoding method used by the encoding processing unit 114, for example, the MPEG-2 method, which is a method standardized by MPEG (Motion Picture Experts Group), H.264, and H.264. Any method can be applied as long as it is a method for encoding a moving image, such as the H.264 method or the HEVC (High-efficiency-Video Coding) method. Since these encoding methods do not directly characterize the present invention and are known techniques, detailed description thereof will be omitted.

Note that the encoding processing unit 114 is not essential in the data generation apparatus according to the present invention. That is, the data generation apparatus according to another embodiment different from the first embodiment may output the input omnidirectional moving image data without compression without encoding.

When the reproduction control data generation unit 115 receives basic azimuth data, one or more extended azimuth data, and shooting point data (map coordinate data), the reproduction control data generation unit 115 generates reproduction control data including these data, and the generated reproduction control data. Is output to the multiplexing processing unit 116. The specific contents of the reproduction control data will be described later.

The multiplexing processing unit 116 generates free viewpoint moving image data by multiplexing map data, reproduction control data for each frame, and encoded omnidirectional moving image data that are input from the outside, The generated free viewpoint video data is output to the outside.

The free-viewpoint video data (distribution data) generated by the multiplexing processing unit 116 is automatically transmitted to the free-viewpoint video playback apparatus according to the second embodiment or manually using a removable recording medium. Distributed.

(Operation of Data Generation Device 100)
Next, the operation of the data generation device 100 will be described with reference to FIG. FIG. 3 is a flowchart illustrating an example of the operation of the data generation device 100.

The data generation device 100 starts the operation according to the flowchart of FIG. 3 at the timing when the omnidirectional video data and the map data are input from the outside.

As shown in FIG. 3, the data generation device 100 performs the processing from step S1 to step S6 for each frame of the omnidirectional video data.

That is, in step S1, the shooting point data conversion processing unit 113 converts the GPS coordinate data indicating the shooting point of the target frame into map coordinate data using the map data input from the outside, and the reproduction control data generation unit 115. Output to.

After step S1, the basic azimuth data generation unit 111 generates basic azimuth data (step S2).

Step S2 will be specifically described with reference to FIG.

FIG. 4 (a) shows a flowchart showing step S2 in detail.

4A, in step S21, the basic orientation data generation unit 111 determines whether the operation reception unit 120 has received an operation for generating basic orientation data.

When it is determined that the operation receiving unit 120 has received an operation for generating basic azimuth data, the basic azimuth data generating unit 111 proceeds to step S22, and the operation receiving unit 120 performs an operation for generating basic azimuth data. If it is determined that it has not been received, the process proceeds to step S23.

In step S22, the basic azimuth data generation unit 111 generates basic azimuth data corresponding to the contents of the above-described operation (specifically, an operation for inputting the cut-out center coordinates described below). In step S23, the basic azimuth data generation unit 111 automatically generates basic azimuth data.

Here, the specific contents of the orientation data (basic orientation data and extended orientation data) will be described with reference to FIG. FIG. 5 is a view image showing a state of view when a certain direction is viewed from a certain photographing point, which is cut out from an omnidirectional image photographed at a certain photographing point by the free viewpoint moving image reproducing device according to the second embodiment. It is a figure which shows an example.

The omnidirectional video data according to the present embodiment is configured to include all 360-degree videos, but the free-viewpoint video playback device according to the second embodiment (for example, a large display such as a TV or a projector, a tablet, a smartphone, or the like) The small display or the head mounted display) cannot display the entire image. That is, the free viewpoint moving image playback apparatus according to the second embodiment needs to cut out a part (view image) from the target frame and display the view image for each frame of the omnidirectional moving image.

In order to enable this, the basic azimuth data generation unit 111 performs an operation on the target frame (omnidirectional image) corresponding to the center position of the visual field image that the content creator wants the free viewpoint video playback apparatus according to the second embodiment to cut out. Direction data including the coordinates (cutout center coordinates) is generated.

Thereby, the free viewpoint video playback device according to the second embodiment acquires orientation data including the cut-out center coordinates corresponding to the point 15 at the center position of the image 14 in FIG. 5 as the orientation data related to the target frame. The image 14 can be cut out from the target frame as a visual field image.

So far, the specific contents of the bearing data have been described.

Note that the basic azimuth data generation unit 111 may proceed to step S22 when an operation of inputting a cut-out center coordinate is accepted, or when another type of operation is accepted. The other type of operation will be described below with reference to FIG. FIG. 6 is an explanatory diagram which is referred to for explaining the different operation.

The basic azimuth data generation unit 111 may proceed to step S22 when an operation for designating an arbitrary point on the map 10 indicated by the map data (for example, the point 16 in FIG. 6) is received.

In this case, in step S22, the basic azimuth data generation unit 111 may specify the azimuth in which the point designated by the above operation is located as seen from the point on the map 10 corresponding to the shooting point of the target frame. Then, the basic azimuth data generation unit 111 may generate basic azimuth data including a value indicating the azimuth instead of the cut-out center coordinates.

Alternatively, in step S22, the basic azimuth data generation unit 111 includes a coordinate value indicating a point on the map corresponding to the shooting point of the target frame and a coordinate value indicating a point designated by the operation on the map. Such basic azimuth data may be generated.

Referring back to the flowchart of FIG. 3, the description of the operation of the data generation device 100 is continued.

After step S2, the extended azimuth data generation unit 112 generates extended azimuth data (step S3), and the process proceeds to step S4.

In step S4, the data generating apparatus 100 returns to step S3 when an operation for further generating extended azimuth data is received, and returns to step S5 when not receiving an operation for further generating extended azimuth data. move on.

Step S3 will be specifically described with reference to FIG.

FIG. 4 (b) shows a flowchart showing step S3 in detail.

As can be seen from FIG. 4B, in step S31, the extended orientation data generation unit 112 determines whether or not the operation receiving unit 120 has received an operation for generating extended orientation data.

When it is determined that the operation receiving unit 120 has received an operation for generating extended azimuth data, the extended azimuth data generating unit 112 proceeds to step S32, and the operation receiving unit 120 performs an operation for generating extended azimuth data. If it is determined that it has not been received, the process proceeds to step S33.

In step S32, the extended azimuth data generation unit 112 expands the azimuth data according to the content of the above-described operation (specifically, an operation for inputting the cut-out center coordinates different from the cut-out center coordinates included in the basic direction data). Is generated.

In step S33, the extended azimuth data generation unit 112 automatically generates extended azimuth data including cutout center coordinates different from the cutout center coordinates included in the basic azimuth data.

For example, in step S33, the extended azimuth data generation unit 112 may automatically calculate four extended azimuth data each indicating the east, west, south, and north directions, or automatically generate only one extended azimuth data indicating a predetermined direction. May be calculated automatically. Alternatively, in step S33, the extended azimuth data generation unit 112 may automatically generate extended azimuth data including cut-out center coordinates corresponding to a combination of a predetermined azimuth angle and a predetermined elevation angle.

Alternatively, the extended azimuth data generation unit 112 may automatically generate extended azimuth data such that the coordinates in the moving area in the omnidirectional video become the cut-out center coordinates.

For example, the extended azimuth data generation unit 112 may calculate a motion vector in pixel units or block units with reference to the forward or backward frame of the omnidirectional video. Then, in any of the following cases, the extended azimuth data generation unit 112 generates extended azimuth data such that the coordinates of the center or the center of gravity of each of the following one or more areas become the cut center coordinates. It may be automatically generated.
When one or more of a plurality of predetermined regions (blocks) are regions in which the value indicating the magnitude of the motion vector is a value greater than or equal to a predetermined size When the region is a region where there are a predetermined number or more of pixels whose value indicating the magnitude of the motion vector is a value equal to or greater than the predetermined size,
Further, for example, when there are one or more areas having a predetermined size or more composed of the following pixels or blocks, the extended orientation data generation unit 112 has the center or the center of gravity of each of the one or more areas described above. Extended azimuth data may be automatically generated so that these coordinates become the cut-out center coordinates.
Pixel: The direction of the calculated motion vector is different from the direction of the motion vector of each pixel around the area. Pixel block: The direction of the calculated motion vector is different from the direction of the motion vector of each block around the area. If the image data obtained by photographing the subject included in the omnidirectional video is input by the user before the start of the operation according to the flowchart of FIG. You may perform the process which compares image data and an omnidirectional moving image.

Specifically, the extended azimuth data generation unit 112 may compare the image data with a target frame in the omnidirectional video to determine whether or not the subject image is included in the target frame. . Then, when the extended orientation data generation unit 112 determines that the subject image is included in the target frame, the extended orientation data generation unit 112 determines the region including the subject image among the plurality of predetermined regions. Extended azimuth data may be automatically generated such that the coordinates of the center or the center of gravity become the cut-out center coordinates.

It should be noted that the image data input by the user may be image data taken at an opportunity different from the shooting of the omnidirectional video. Alternatively, the image data may be data of an image captured by a user when a desired field-of-view image of a desired frame (omnidirectional still image) in the omnidirectional video is displayed by a playback device.

The extended azimuth data generation unit 112 may generate extended azimuth data in n steps S33, or generate extended azimuth data only in some steps S33 of the n steps S33. May be.

By the way, as can be seen from the description of step S3 and step S4, in the present embodiment, the operation receiving unit 120 can receive input of a plurality of different cut-out center coordinates. The extended azimuth data generation unit 112 generates a plurality of extended azimuth data by generating extended azimuth data that includes the cut out center coordinates for each of the plurality of input cut out center coordinates.

Here, a case where a content creator inputs a plurality of different cut-out center coordinates will be described below with reference to FIGS.

FIG. 7 is a diagram showing that there are a plurality of subjects that the content creator recommends browsing around a certain section in the shooting route. FIG. 8 shows a visual field image showing the state of the field of view when a different direction (the direction of the subject 18) is seen from a certain photographing point, which is cut out from a frame photographed at the certain photographing point by the free viewpoint moving image reproducing device. It is a figure which shows an example.

7, it is assumed that there are two

subjects

18 and 19 around the photographing path represented by the line 12, and the two

subjects

18 and 19 are shown in a certain frame (omnidirectional image). The content creator wants to show both subjects to the viewer.

In this case, if the content creator inputs the two cut-out center coordinates in order to cause the free-viewpoint video playback apparatus according to the second embodiment to cut out the field-of-view image of the subject 18 and the field-of-view image of the subject 19. Good.

For example, it is assumed that the point on the map 10 represented by the map data corresponding to the point where the subject 18 exists is the point 20, and the point on the map 10 corresponding to the point where the subject 19 exists is the point 21. The subject 18 is shown in P frames taken at P shooting points corresponding to the part included in the region 22 in the line 12, and the part included in the region 23 in the line 12. It is assumed that the subject 19 is shown in Q frames taken at Q shooting points corresponding to.

In this case, the content creator, for each of the P frames, displays a field image (for example, field image 24 in FIG. 18) in which the subject 18 is captured from the target frame (omnidirectional image) on the free viewpoint video reproduction device. What is necessary is just to input the cutting center coordinate (for example, the coordinate in this omnidirectional image corresponding to the point 25 of FIG. 18) which makes it cut out. Similarly, the content creator may input, for each of the Q frames, a cut-out center coordinate that causes the free-viewpoint video playback device to cut out a field-of-view image showing the subject 19 from the frame.

As a result, the extended azimuth data generation unit 112 generates extended azimuth data including a certain cutout center coordinate and extended azimuth data including another cutout center coordinate for a frame in which the two

subjects

18 and 19 are captured. become.

As described above, the case where the content creator inputs a plurality of different cut-out center coordinates has been described.

Note that the expanded orientation data may include identification information corresponding to the orientation represented by the X component of the cut-out center coordinates.

As described above, since the data generation apparatus 100 is configured to include a plurality of extended azimuth data for each frame, the content creator can highlight a part of the frames in the omnidirectional video. When a certain scene or a plurality of subjects that the viewer is interested in or wants to show to the viewer are included, it is possible to prevent the viewer from overlooking them.

In addition, the viewer can view a visual field image of a desired orientation among a plurality of orientations corresponding to a plurality of extended orientation data. For example, it is possible to browse recommended sightseeing spots, famous landmarks on the road map, facial expressions with highlights of one's acquaintances and family, and the like.

In step S5, the reproduction control data generation unit 115 generates reproduction control data including the input shooting point data, basic azimuth data, and extended azimuth data.

Here, the reproduction control data generated by the reproduction control data generation unit 115 will be described in detail with reference to FIGS.

FIG. 9 is a diagram schematically showing an example of the data structure of the free-viewpoint video data generated by the data generation device 100. FIG. 10 is a diagram schematically illustrating an example of the data structure of the reproduction control data included in the free viewpoint moving image data. FIG. 11 is a diagram schematically illustrating another example of the data structure of the reproduction control data included in the free viewpoint moving image data.

As shown in FIG. 9, the free viewpoint moving image data 26 includes omnidirectional moving image data and map data, and for each i from 1 to n, a frame whose frame number is i (hereinafter also referred to as a frame i). Eye-gaze direction control data Vi.

For example, the reproduction control data V1 includes shooting point data P1 indicating the shooting point of the frame 1 and basic azimuth data O1 related to the frame 1. For each j from 1 to m, the extended azimuth data E1j related to the frame 1 is included. (M, n, i, j are positive integers).

It should be noted that one or more extended azimuth data may be included in each reproduction control data of the free viewpoint moving image data generated by the data generation device 100 of the present embodiment. Alternatively, as in the free viewpoint moving image data of FIG. 9, the extended azimuth data may not be included in a part of the reproduction control data (reproduction control data Vn).

The azimuth data included in the reproduction control data will be specifically described again with reference to FIGS.

As shown in FIG. 10, the basic azimuth data includes cut-out center coordinates and identification information for identifying the basic azimuth data from the extended azimuth data. Each extended azimuth data includes cut-out center coordinates and identification information for identifying the extended azimuth data from basic azimuth data and other extended azimuth data.

Note that the data structure of the basic azimuth data and the extended azimuth data is not limited to the data structure shown in FIG.

That is, the basic azimuth data and the extended azimuth data may include priority display order data as shown in FIG. The priority display order data indicates the order in which the field of view image cut out by the orientation data including the priority display order data is to be displayed among the plurality of field of view images that can be extracted from the frame i. It is the data shown.

For example, if the value of the priority display order data included in certain azimuth data of the reproduction control data Vi is 1, this means that the azimuth video reproduction device can select the azimuth among a plurality of field images that can be extracted from the frame i. It shows that the visual field image cut out by the data should be displayed first. Similarly, the value of the priority display order data included in certain azimuth data of the reproduction control data Vi is k. The free viewpoint moving image reproduction apparatus can select the view field image that can be cut out from the frame i. This indicates that the field-of-view image cut out by the azimuth data is to be displayed k-th (k is an integer of 2 or more).

When the content creator wants the free-viewpoint video playback device to display a plurality of field-of-view images cut out from the frame i so that they can be simultaneously viewed, the same value is used for the plurality of orientation data for cutting out the plurality of image data. May be included.

Also, as shown in FIG. 11, viewpoint data may be included in each azimuth data.

The viewpoint restriction data includes information indicating whether or not there is a viewpoint restriction (a restriction for allowing the free-viewpoint video playback device to cut out the visual field image only from a part of the entire area of the omnidirectional image). Also good. The viewpoint restriction data including information indicating that there is a viewpoint restriction may include information indicating the partial area.

Then, the data generation device 100 allows the content creator to determine whether or not the viewpoint is restricted, and the above-mentioned partial area (for example, an area that the viewer in the omnidirectional image is particularly interested in, or a viewer in the omnidirectional image). The user may be configured to perform an operation of designating a region that may be browsed.

When the data generation device 100 is configured in this way, the content creator can allow the viewer who is the user of the free viewpoint video playback device to browse only the visual field image in the region that is particularly desired to be noticed. Alternatively, the content creator can restrict browsing of the field-of-view images in other areas that the viewer does not want to browse. An area that the content creator does not want the viewer to browse includes an area that includes a scene that the viewer does not want to browse from the viewpoint of confidentiality, portrait rights, customs, or the like.

In addition, it is desirable that the data generation apparatus 100 that performs streaming distribution of omnidirectional video in real time and the data generation apparatus 100 that distributes omnidirectional video by broadcast have the above-described configuration.

In step S6, the encoding processing unit 114 encodes the input frame (omnidirectional image) using a preset encoding method (for example, HEVC), and outputs encoded data of the omnidirectional image.

The data generation device 100 proceeds to step S7 after performing the above steps S1 to S6 for each image frame.

In step S7, the multiplexing processing unit 7 multiplexes the encoded data of each frame, the map data, and the reproduction control data of each frame to generate free viewpoint moving image data, and the data generating apparatus 100 ends the operation.

(Advantages of the data generation device 100)
As can be seen from the above description, the data generation device 100 generates free-viewpoint video data using omnidirectional video data, map data, and playback control data for each frame.

The reproduction control data for each frame includes shooting point data indicating a point on the map data corresponding to the shooting point of the target frame, basic orientation data for extracting a field-of-view image from the target frame, and extended gaze direction data. Yes.

As will be described in the second embodiment, the free-viewpoint video playback device, with respect to each frame of the omnidirectional video data included in the free-viewpoint video data created in this way, partly from the target frame (a subject, scenery, scene with a highlight) The reproduction control data relating to the target frame is used in order to cut out and display the field-of-view image in which the image is captured.

Thereby, the data generation device 100 can reduce (or eliminate) the time and effort required for the viewer to search for a subject, scenery, or scene with a highlight. In addition, the data generation apparatus 100 can solve the problem that the time when the scene can be reproduced has passed while searching for a scene with highlight, and the viewer misses the scene.

In addition, the content creator uses the data generation device 100 to create azimuth data for displaying a field-of-view image including the subject on the free-viewpoint video playback device for a plurality of subjects desired to be shown to the viewer. Can do. The content creator can also control the display priority order of a plurality of field-of-view images in each frame by using the data generation device 100, so that content satisfying various viewers with different preferences. Can be produced and distributed.

(Modification of Embodiment 1)
The data generation apparatus may be configured to receive omnidirectional image data (still image data) and output free viewpoint still image data.

In this case, the content creator can use this data generation device for the purpose of displaying advertisements using signage or the like. That is, the content creator can sell the right to determine the display priority of the visual field image for each of the multiple visual field images that signage cuts out from the omnidirectional image data to the advertiser or the like.

In addition, when the signage is configured so that an advertisement video or message can be superimposed on each visual field image, the content creator signs the advertising video or message into the target visual field image for each line-of-sight image. The right to superimpose display may be sold.

(Appendix 1 of Embodiment 1)
The free viewpoint moving image data generating apparatus according to the present embodiment generates free viewpoint moving image data that is referenced by a free viewpoint moving image reproduction system according to an embodiment different from the second embodiment to be described later to reproduce a free viewpoint moving image. You may comprise.

Such a free-viewpoint video playback system has, for example, a plurality of display devices (display, projector, etc.) surrounding the viewer and a structure that allows the viewer to face a predetermined direction when the viewer is sitting. And a seating system.

In this case, the basic azimuth data generated by the free viewpoint moving image data generating device is displayed on a display device (display device positioned in the predetermined direction as viewed from the seat) located in front of the viewer of the free viewpoint moving image playback system. Cutout center coordinates that are coordinates on the image frame (omnidirectional image) corresponding to the center position of the displayed visual field image may be included.

The extended azimuth data includes coordinates (cutout center coordinates) on the image frame (omnidirectional image) corresponding to the center position of the field-of-view image displayed on the display device in front of the viewer of the free viewpoint video playback device. May be included.

(Appendix 2 of Embodiment 1)
Regarding step S23, the basic azimuth data generation unit 111 may calculate the cut-out center coordinates included in the basic azimuth data by performing image processing on the omnidirectional moving image data.

For example, the basic azimuth data generation unit 111 may divide two adjacent image frames (omnidirectional images) into a plurality of regions with the same division pattern, and acquire a motion vector of each region by lock matching or the like.

Then, the basic azimuth data generation unit 111 determines which area is the area in the moving direction of the camera from the direction and magnitude of the motion vector of each area, and the center coordinates of the area determined to be the moving direction May be the cut-out center coordinates included in the basic azimuth data.

Further, with regard to step S22 and step S23, the basic orientation data generation unit 111 may divide the image frame (omnidirectional image) into a plurality of regions. When the photographer appears in a certain area in the omnidirectional image, the basic azimuth data generation unit 111 recognizes that the area is the direction opposite to the moving direction of the camera. It may be determined whether the region is in the traveling direction.

The basic orientation data generation unit 111 may use the center coordinates of the area determined to be the traveling direction as the cut-out center coordinates included in the basic orientation data.

It should be noted that the area where the photographer moves in the omnidirectional image may be designated by the operation of the content creator, and the basic azimuth data generation unit 111 selects the subject shown throughout the photographic period. , It may be recognized as a photographer.

(Appendix 3 of Embodiment 1)
Step S33 is not essential in the data generation device according to the present invention. That is, the extended azimuth data generation unit 112 may not generate the extended azimuth data when it is determined that the operation reception unit 120 has not received an operation for generating the extended azimuth data.

Further, step S4 is not essential in the data generation apparatus according to the present invention. That is, the extended azimuth data generation unit 112 may generate only one extended azimuth data for each frame.

(Appendix 4 of Embodiment 1)
Although the priority display order data may be input from the outside by the content creator, the present invention is not limited to such a configuration.

That is, the data generation device 100 may determine the priority display order data based on the preference information of a specific viewer that is input in advance, or a number of viewers who have already viewed the target omnidirectional video. It may be generated with reference to browsing information (big data indicating which part of the omnidirectional image is viewed by many viewers at each reproduction time of the omnidirectional video).

Alternatively, when the time when the omnidirectional video is reproduced is determined (for example, when free viewpoint video data is distributed by broadcasting), the data generation device 100 determines the priority display order based on the season and time zone at the time of reproduction. Data may be generated. By doing in this way, the data generation device 100 preferentially displays, for example, a visual field image including a subject or a landscape that is difficult to see unless it is in the season or time zone during playback on the free viewpoint video playback device. Can do. In particular, it is possible to provide an optimal display for tourist attraction and navigation.

(Appendix 5 of Embodiment 1)
The viewpoint restriction data may include level information related to viewpoint restriction instead of information indicating presence / absence of viewpoint restriction.

The level represented by the level information may be any of the following three levels, for example.
・ A level that allows viewers to display any field-of-view image in an omnidirectional image unconditionally on a free-viewpoint video playback device. ・ Browsing that meets a predetermined condition (for example, paid for the content creator) Only for the user, a level at which the arbitrary viewpoint image can be displayed on the free-viewpoint video playback apparatus. The viewer can display only the viewfield image in a part of the omnidirectional image on the free-viewpoint video playback apparatus. In this way, the content creator can allow the viewer who does not satisfy the predetermined condition to browse only a part of the visual field image.

Also, the content creator can loosen the viewpoint restriction according to the amount of money paid by the viewer to the content creator.

(Appendix 6 of Embodiment 1)
The reproduction control data may not be included in the free viewpoint moving image data.

For example, when the data generation apparatus 100 is implemented in the form of a PC having a function of distributing free viewpoint video data, the data generation apparatus 100 includes free viewpoint video data including map data and omnidirectional video data, omnidirectional The reproduction control data (reproduction control data set) of each frame of the moving image data may be distributed individually.

In this case, the content creator who is the user of the data generation device 100 can send two different reproduction control data sets created according to his / her preference to the partner to which the free viewpoint video data is distributed. Then, the content creator causes the other PC to play the omnidirectional video based on the first playback control data set and to play the omnidirectional video based on the second playback control data set. Can do.

For example, a content creator causes a partner to see a certain landscape (scenery that appears in the field-of-view image displayed by the former playback) when shooting an omnidirectional video, and is watching it when shooting a omnidirectional video. It is possible to make the other party see another landscape (the landscape shown in the visual field image displayed by the latter reproduction).

Note that the playback control data may include information related to the content creator (information related to the copyright such as the name of the content creator). This mechanism is useful when the content creator desires to handle the reproduction control data itself as a secondary work.

(Additional Item 7 of Embodiment 1)
In the first embodiment, the data generation apparatus 100 that handles omnidirectional video data has been described. However, the data generation apparatus according to the present invention is not limited to such a configuration. That is, the range of the data generation device according to the present invention is created by photographing a three-dimensional three-dimensional model (three-dimensional three-dimensional model using CG or a plurality of directions) instead of omnidirectional video data. A data generation apparatus that handles a three-dimensional three-dimensional model) is also included.

<Embodiment 2>
Embodiment 2 which is a preferred embodiment of a playback apparatus according to the present invention will be described below in detail with reference to the accompanying drawings. In addition, the structure which attached | subjected the same code | symbol also in different drawing shall be abbreviate | omitted, and the description is abbreviate | omitted.

(Outline of free viewpoint video playback device)
The free viewpoint video playback apparatus according to the present embodiment is an apparatus that plays back an omnidirectional video by referring to the free viewpoint video data generated by the data generation apparatus 100 according to the first embodiment. It is not structured to display the entire image frame.

The free-viewpoint video playback apparatus according to this embodiment is configured to display a field-of-view image by cutting out the field-of-view image from the target frame for each frame of the omnidirectional video.

Note that the free viewpoint video playback device may be a device having a touch panel function such as a smartphone or a tablet.

Alternatively, the free-viewpoint video playback device includes a magneto-optical disk typified by DVD, Blu-ray (registered trademark) Disc, and / or a semiconductor memory typified by USB memory, SD (registered trademark) card, etc. An apparatus that reads and reproduces content data from an electronic medium may be used.

Alternatively, the free-viewpoint video playback device may be a television receiver that receives broadcast waves of TV broadcasts, or a device that receives content data distributed from the Internet or other communication lines.

Alternatively, the free-viewpoint video playback device is configured to include an HDMI (registered trademark) Multi-Media Interface (HDMI) (registered trademark) receiver that receives an image signal from an external device such as a Blu-ray (registered trademark) disc player. It may be.

That is, the free viewpoint video playback device may be any device as long as it has a function of receiving content data from outside and playing back the input content data.

(Configuration of free viewpoint video playback device)
FIG. 12 is a block diagram showing a configuration of a free-viewpoint video playback apparatus 200 (hereinafter abbreviated as “playback apparatus 200”) according to the present embodiment. First, the outline of the configuration of the playback apparatus 200 will be described with reference to FIG.

The playback device 200 includes a control unit 210, a display unit 220, and an operation reception unit 230.

The control unit 210 is a CPU and controls the entire playback apparatus 200 in an integrated manner.

The display unit 220 is a display on which a visual field image is displayed.

The operation accepting unit 230 is an operation device that accepts an operation by a viewer of the omnidirectional video (a user of the playback device 200).

The control unit 210 functions as a demultiplexing processing unit 211, a map display processing unit 212, a decoding processing unit 213, an orientation data analysis unit 214, and an image cutout unit 215 by executing a specific program.

Note that the decoding processing unit 213 and the demultiplexing processing unit 211 may be realized by hardware (LSI) instead of software.

When the demultiplexing processing unit 211 receives an input of free viewpoint moving image data from the outside, the demultiplexing processing unit 211 performs demultiplexing processing on the free viewpoint moving image data, thereby reproducing map data and each frame from the free viewpoint moving image data. Control data and encoded omnidirectional video data are extracted.

The demultiplexing processing unit 211 outputs the map data and the shooting point data of each frame to the map display processing unit 212, outputs each encoded frame to the decoding processing unit 213, and sets the azimuth data group of each frame as a direction. The data is output to the data analysis unit 214.

The map display processing unit 212 displays a map represented by the map data on the display unit 220, and displays a line indicating the shooting route on the map using the shooting point data of each frame.

The decoding processing unit 213 decodes each frame input from the outside and outputs each decoded frame to the image cutout unit 215.

The azimuth data analysis unit 214 selects, for each frame, all or part of one or more azimuth data related to the frame. The azimuth data analysis unit 214 performs this selection process based on a browser operation or automatically.

The azimuth data analysis unit 214 outputs the cutout coordinates included in the azimuth data to the image cutout unit 215 for each selected azimuth data.

The image cutout unit 215 cuts out one or more visual field images from each frame with reference to one or more cutout coordinates regarding the frame. In other words, the image cutout unit 215, for each of the one or more cutout center coordinates referred to, displays a visual field image in a region having a predetermined length and width in the region centered on the cutout center coordinate. , Cut out from the frame.

The image cutout unit 215 displays, for each frame, one or more visual field images cut out from the frame on the display unit 220 within the period of the frame.

(Operation of the playback device 200)
Next, the operation of the playback apparatus 200 will be described.

The playback apparatus 200 has the following first mode and second mode, and the operation in the first mode is different from the operation in the second mode.
First mode: The field-of-view image cut out with reference to the cut-out center coordinates of the basic azimuth data is displayed in full screen, and the field-of-view image cut out with reference to the cut-out center coordinates of the extended azimuth data is displayed in a small screen (wipe display). Mode second mode: The field-of-view image cut out with reference to the cut-out center coordinates of the basic azimuth data is displayed by default, and the cut-out center coordinates of the extended azimuth data are referenced based on the operation of switching the view-field image displayed by the viewer. First, the operation of the playback device set to the first mode will be described with reference to FIGS. 13 to 15. FIG.

FIG. 13 is a flowchart showing the operation, and FIG. 14 is a flowchart showing in detail one step of the flowchart of FIG. FIG. 15 is a diagram illustrating a display mode (PinP display) of the visual field image by the playback device 200 set in the first display mode.

The playback device 200 starts the operation according to the flowchart of FIG. 13 at the timing when the free viewpoint video data is input from the outside.

In step S41, the multiplexing processing unit 7 performs a demultiplexing process on the free viewpoint moving image data, so that the map data, the reproduction control data of each frame, and the encoded omnidirectional moving image are converted from the free viewpoint moving image data. Extract the data.

As shown in FIG. 13, the playback device 200 performs the processing from step S43 to step S46 for each frame of the omnidirectional video data within the period of the frame.

In step S43, the decoding processing unit 213 decodes the frame i (omnidirectional image) using a preset decoding method (for example, HEVC), and outputs the decoded frame i to the image clipping unit 215.

In step S44, the orientation data analysis unit 214 selects one or more orientation data included in the reproduction control data Vi. The azimuth data analysis unit 214 extracts the cutout center coordinates from the azimuth data for each of the selected one or more azimuth data, and outputs the extracted cutout center coordinates to the image cutout unit 215.

Step S44 will be specifically described with reference to FIG.

First, in step S441, the azimuth data analyzing unit 214 determines whether or not priority display order data is included in each of the one or more azimuth data of the reproduction control data Vi.

If the direction data analysis unit 214 determines that the priority display order data is included in each direction data of the reproduction control data Vi, the process proceeds to step S442, and the priority display order data is included in each direction data of the reproduction control data Vi. If it is determined that it is not included, the process proceeds to step S443.

In step S442, the azimuth data analysis unit 214 selects azimuth data including priority display order data indicating the highest priority display order among all the azimuth data included in the reproduction control data Vi, and step S444. Proceed to

In step S443, the azimuth data analysis unit 214 selects basic azimuth data by referring to the identification information of each azimuth data included in the reproduction control data Vi.

In step S444, the azimuth data analyzing unit 214 selects the azimuth data selected in step S442 or step S443 (specific azimuth data which is the azimuth data that the playback device 200 should use most preferentially to cut out a plurality of field images). ) To extract the cut-out center coordinates.

The orientation data analysis unit 214 outputs the extracted cut-out center coordinates to the image cut-out unit 215. The orientation data analysis unit 214 proceeds to step S445 after step S444.

In step S445, the azimuth data analysis unit 214 determines whether or not the extended azimuth data is included in the reproduction control data Vi.

The reproducing device 200 proceeds to step S446 when it is determined that the extended control data Vi is included in the playback control data Vi, and proceeds to step S45 when it is determined that the extended control data Vi is not included in the playback control data Vi. Proceed to

In step S446, the azimuth data analyzing unit 214 determines whether or not the operation receiving unit 230 has received an operation for causing the playback device 200 to select the extended azimuth data.

When the operation accepting unit 230 determines that the operation has been accepted, the reproducing device 200 proceeds to step S447, and when the operation accepting unit 230 determines that the operation has not been accepted, the reproducing device 200 proceeds to step S45.

In step S447, the azimuth data analysis unit 214 selects extended azimuth data according to the operation. Further, in step S448, the azimuth data analysis unit 214 extracts cutout center coordinates from the extended azimuth data selected in step S447, and outputs the extracted cutout center coordinates to the image cutout unit 215.

The reproducing device 200 proceeds to step S45 after step S448.

Returning to the flowchart of FIG. 13, the description of the operation of the playback device 200 will be continued.

In step S45, the image cutout unit 215 refers to the cutout center coordinates output in step S444 and cuts out the view image for full screen display from the frame i.

When the step immediately preceding step S45 is step S448, in step S45, the image cutout unit 215 refers to the cutout center coordinates output in step S448, and displays the field-of-view image for sub-screen display from the frame i. cut.

The image cutout unit 215 displays the full-screen view field image in full screen after step S45. In addition, when the most recent step of step S45 is step S448, the image cutout unit 215 superimposes and displays a visual field image for sub-screen display on the visual field image.

As a result, for example, as shown in FIG. 15, the display unit 220 displays a full-screen display field-of-view image (full-screen image) 27 and a sub-screen display field-of-view image (wipe image) 28. Become.

(Appendix 1 regarding the first mode)
In step S44, the playback apparatus 200 may be configured to accept an operation for displaying an image of an arbitrary desired orientation included in the frame i (omnidirectional image) as a visual field image.

In this case, the playback device 200 may perform the following processing within the period of the frame i. That is, the playback device 200 may display the visual field image over a predetermined valid period, and perform the process of step S45 after the expiration date.

The above operation may be an operation for inputting the value of the cutout center coordinate in the omnidirectional image, or an operation for specifying the cutout center coordinate (mouse click or touch operation). In addition, when the playback apparatus 200 receives a mouse operation, a flick operation, or an operation of pressing a controller button during the predetermined valid period, the playback device 200 changes the displayed field image of one direction to a field image of another direction. It may be changed.

That is, the playback device 200 converts a field image of a certain direction being displayed into a field image of another direction according to the amount and direction of movement of the mouse, a field image of another direction according to the amount and direction of flick, or The viewing image may be changed to a different azimuth image according to the type of button to be pressed.

(Appendix 2 regarding the first mode)
The image cutout unit 215 may perform a process of correcting the cutout center coordinates as necessary for each frame except the first frame.

For example, when the distance between the cutout center coordinate C1 related to the frame immediately before the target frame and the cutout center coordinate C2 related to the target frame exceeds a predetermined value, it is desirable to correct the cutout central coordinate related to the target frame.

That is, it is desirable to correct the cut-out center coordinates for the frame immediately before the target frame from the cut-out center coordinates C1 to the following cut-out center coordinates C3.
Cut-out center coordinates C3: coordinates on a line segment connecting the coordinates C1 and C2, and the distance between the coordinates C1 is the predetermined value. In addition, this is because the viewer can always grasp which field of view image is displayed, and also to prevent the viewer from getting sick.

(Appendix 3 regarding the first mode)
The size of the visual field image cut out from the omnidirectional image by the image cutout unit 215 may be a preset size corresponding to the size of the display area of the display unit 220.

Alternatively, the reproduction control data Vi may include size information indicating the size of the visual field image cut out from the omnidirectional image by the image cutout unit 215. In this case, the image cutout unit 215 cuts out the visual field image with reference to the size information and the cutout center coordinates.

(Appendix 4 regarding the first mode)
The image cutout unit 215 may perform a process of correcting distortion (distortion due to a lens or a mirror) with respect to the omnidirectional image, and may cut out the visual field image from the corrected omnidirectional image. Note that the distortion correction method does not directly characterize the present invention, and a known method can be applied, and thus detailed description thereof is omitted.

(Appendix 5 regarding the first mode)
In S444, the azimuth data analysis unit 214 may output control information indicating that the field-of-view image to be cut out with reference to the cut-out center coordinates together with the cut-out center coordinates should be displayed in full screen. .

In S <b> 448, the orientation data analysis unit 214 may output control information indicating that the visual field image to be cut out with reference to the cut-out center coordinate should be wipe displayed together with the cut-out center coordinate.

In this case, the image cutout unit 215 may determine whether to view the visual field image to be cut out by referring to the control information with reference to the cutout center coordinates acquired together with the control information to be displayed in full screen or wiped.

In S444 and S448, the azimuth data analysis unit 214 may output the priority display order data included in the azimuth data together with the cutout center coordinates to the image cutout unit 215 together with the cutout center coordinates.

In this case, the image cutout unit 215 may specify priority display order data indicating the highest priority display order from the plurality of acquired priority display order data.

Then, the image cutout unit 215 may determine to display the visual field image to be cut out with reference to the cutout center coordinates acquired together with the priority display order data. In addition, the image cutout unit 215 may determine to wipe-display a field-of-view image to be cut out with reference to the cut-out center coordinates acquired together with other priority display order data.

(Appendix 6 regarding the first mode)
The image cutout unit 215 may compare the value of the X component of the cutout center coordinate of the basic azimuth data with the value of the X component of the cutout center coordinate of the extended gaze direction data. When the value of the former X component is smaller than the value of the latter X component, the image cutout unit 215 displays the wipe image 28 on the right end of the display, and the value of the former X component is greater than the value of the latter X component. May be displayed on the left end of the display.

By doing so, the viewer can see from the shooting point of the omnidirectional image whether the subject in the wipe image 28 is on the right side of the subject in the full screen image 27 or on the left side. I was able to understand it intuitively.

In addition, a field image in a certain direction in which a certain subject (a wonderful scenery) is displayed is displayed as a full screen image 27, and another subject related to the subject (the face of a person who is moved by seeing the scenery) is reflected. When a field image of a different orientation is displayed as the wipe image 28, the viewer can feel as if he / she is simultaneously viewing the wonderful scenery and the face of the person who is impressed by seeing it. .

Note that the display position of the wipe screen may be the upper end and the lower end instead of the left end and the right end.

In this case, the image cutout unit 215 may compare the value of the Y component of the cutout center coordinate of the basic orientation data with the value of the Y component of the cutout center coordinate of the extended gaze direction data. When the value of the former Y component is smaller than the value of the latter Y component, the image cutout unit 215 displays the wipe image 28 at the upper end of the display, and the value of the former Y component is greater than the value of the latter Y component. May be larger, the wipe image 28 may be displayed at the lower end of the display.

(Appendix 7 regarding the first mode)
The image cutout unit 215 may create an image button using the field-of-view image 28 and display the created image button instead of wiping the field-of-view image 28.

When this image button is pressed by the user, the image cutout unit 215 may perform the following processing after deleting the image button and the field-of-view image 27 displayed on the full screen. That is, the image cutout unit 215 may display the field image 28 in full screen, create an image button using the field image 27, and display the created image button.

Thereby, the user can quickly display the field-of-view image cut out with reference to the cut-out center coordinates of the extended orientation data on the playback device 200 in full screen.

(Appendix 8 regarding the first mode)
The playback device 200 may play back the omnidirectional video twice.

That is, the playback apparatus 200 displays the field-of-view image of each frame using only the cut-out center coordinates of the basic orientation data during the first playback, and uses only the cut-out center coordinates of the extended orientation data during the second and subsequent playbacks. The field-of-view image of each frame may be displayed. Further, the playback apparatus 200 may be a digital signage configured as described above.

In this case, the playback device 200 as digital signage can allow the viewer to browse various advertisements shown in the omnidirectional video without getting the viewer bored.

(Appendix 9 regarding the first mode)
In addition, when the reproduction control data Vi includes a plurality of direction data, the reproducing device 200 displays a button corresponding to the identification information included in the direction data for each of the plurality of direction data in a full screen display. It may be displayed on the field image.

The playback apparatus 200 may refer to the cut-out center coordinates of the orientation data identified by the identification information corresponding to the pressed button when any of the displayed buttons is pressed. Then, the playback apparatus 200 may switch the image to be displayed on the full screen to a reference image cut out using the cut-out center coordinates.

This makes it possible for the viewer to display a field-of-view image including his / her favorite scene in full screen as he wishes.

(Appendix 10 regarding the first mode)
The playback device 200 may display a button for setting whether or not to display a field-of-view image of the orientation indicated by the target extended orientation data for each extended orientation data.

In this case, the playback device 200 may wipe-display only the visual field image to be displayed based on the setting by the button.

Further, when there are a plurality of field images to be displayed in a certain frame based on the setting by the button, a plurality of field images are displayed according to the priority display order indicated by the priority display order data during the period of the frame. May be wiped in order, or a plurality of wipe screens may be prepared to wipe display all the field-of-view images.

(Advantages of the playback device 200 set to the first mode)
As can be seen from the above description regarding the first mode, the playback device 200 separates the omnidirectional video data, map data, and playback control data for each frame from the free-viewpoint video data generated by the data generation device 100. .

The playback apparatus 200 selects all or a part of the azimuth data included in the line-of-sight control data, and cuts out and displays a part of the omni-directional image (field-of-view image) using the cut-out center coordinates included in the line-of-sight direction data. To do.

As a result, the playback device 200 (a device having a display screen with a limited resolution and size, such as a tablet, a TV, a PC, a smartphone, or an HMD) can play back each frame (omnidirectional) For the image), a field image in a specific direction among the target omnidirectional images (field image that the content creator wants the viewer to browse) is displayed.

This allows the viewer to browse the visual field image that the content creator wants the viewer to browse if no special operation is performed.

In addition, even when the viewer performs an operation to display an image of a desired orientation as a visual field image, the visual field image that the content creator wants the viewer to browse after a predetermined period has elapsed Can do.

Next, the operation of the playback apparatus set to the second mode will be described with reference to FIGS.

FIG. 16 is a flowchart showing the operation, and FIG. 17 is a flowchart showing in detail one step (step S44) of the flowchart of FIG. FIG. 18 is a diagram illustrating a state in which the playback device 200 set in the second display mode displays a default visual field image on a map.

FIG. 19 is a flowchart showing in detail another process (step S48) of the flowchart of FIG. 16, and FIG. 20 shows the case where the playback device 200 set in the second display mode is designated by the user. It is the figure which illustrated a mode that the performed visual field image is displayed on a map.

The playback device 200 starts the operation according to the flowchart of FIG. 16 at the timing when the free viewpoint video data is input from the outside.

The playback device 200 proceeds to step S42 after performing step S41 already described.

In step S42, the map display processing unit 212 displays the map represented by the map data on the display unit 220, and displays a line indicating the shooting route on the map 29 using the shooting point data of each frame.

Thereafter, as shown in FIG. 16, the playback apparatus 200 performs the processing from step S43 to step S48 for each frame of the omnidirectional video data during the frame period.

That is, the reproducing device 200 proceeds to step S444A after performing step S43 already described.

In step S44A, the azimuth data analysis unit 214 automatically selects one azimuth data included in the reproduction control data Vi. The azimuth data analysis unit 214 extracts cutout center coordinates from the selected azimuth data, and outputs the extracted cutout center coordinates to the image cutout unit 215.

The specific processing in step S44A is as shown in FIG. 17, but the processing in steps S441 to S444 in FIG. 17 is the same as the processing in steps S441 to S444 in FIG. 13 already described. The description of the specific processing in step S44A is omitted.

After step S44A, in step S45, the image cutout unit 215 cuts out the default display visual field image 31 from the frame i with reference to the cutout center coordinates output in step S444.

The image cutout unit 215 displays the visual field image 31 for default display on the map 29 after step S45 (step S46).

The reproducing device 200 proceeds to step S47 after step S46.

In step S47, the map display processing unit 212 displays the symbol 30 indicating the shooting point of the frame i at the position on the map indicated by the shooting point data of the frame i.

In step S47, the map display processing unit 212 obtains each azimuth data not selected by the azimuth data analysis unit 214 in step S44A from the azimuth data analysis unit 214, and performs the following processing on each azimuth data regarding the frame i. I do.

That is, the map display processing unit 212 has a field of view when the azimuth indicated by the target azimuth data is viewed from the shooting point of the frame i from the cut-out center coordinates included in the target azimuth data and the shooting point data of the frame i. The position on the map of the subject to be entered is estimated, and the symbol 32 is displayed at the estimated position.

For example, the map display processing unit 212 refers to the cut-out center coordinates included in the target orientation data, extracts the field image from the frame i, and applies a known distance estimation technique to the field image. The distance between the shooting point of the frame i and the point where the subject in the field-of-view image exists may be estimated. Then, the map display processing unit 212 may estimate the position of the subject on the map from the shooting point data of the frame i, the cut-out center coordinates indicating the orientation, and the estimated distance.

The reproducing device 200 proceeds to step S48 after step S47.

In step S48, the playback device 200 performs a visual field image switching process. The visual field image switching process in step S48 will be specifically described with reference to FIG.

First, the direction data analysis unit 214 determines whether or not the extended control data is included in the reproduction control data Vi (step S481).

If the playback device 200 determines that the extended control data Vi does not include the extended orientation data, the playback device 200 ends step S48. If the playback control data Vi determines that the extended control data Vi includes the extended orientation data, the playback device 200 ends the process. The process proceeds to S482.

In step S482, the image cutout unit 215 performs the following processing on each extended orientation data included in the reproduction control data Vi.

That is, the image cutout unit 215 obtains the cutout center coordinates included in the target extended azimuth data from the azimuth data analysis unit 214, extracts the field image from the frame i with reference to the obtained cutout center coordinates, and extracts The displayed field-of-view image is displayed as a thumbnail.

In the present embodiment, in step S482, the image cutout unit 215 displays a broken line connecting the thumbnail 33 of the visual field image and the symbol 32 indicating the position of the subject in the visual field image on the map. indicate.

As a result, an image as shown in FIG. 18 is displayed on the display unit 220.

After step S482, the orientation data analysis unit 214 determines whether the operation reception unit 230 has received an operation for selecting any thumbnail.

When it is determined that the operation receiving unit 230 has not received an operation for selecting any thumbnail, the playback device 200 ends step S48, and the operation receiving unit 230 has received an operation for selecting any thumbnail. If it is determined, the process proceeds to step S484.

In step S484, the azimuth data analyzing unit 214 selects the extended azimuth data corresponding to the selected thumbnail 33 from one or more extended azimuth data related to the frame i, and the process proceeds to step S485.

In step 485, the azimuth data analysis unit 214 extracts the cutout center coordinates from the selected extended azimuth data, and outputs the extracted cutout center coordinates to the image cutout unit 215.

After step S485, the image cutout unit 215 refers to the cutout center coordinates output in step S485, cuts out the visual field image from the frame i (step S486), and cuts the displayed visual field image in step S486. Switch to view image (step S487)
In this embodiment, in step S487, the image cutout unit 215 displays a thick frame surrounding the selected thumbnail 33 (the field image corresponding to the thumbnail 33 is displayed as a result of the user selecting the thumbnail 33). A thick frame).

As a result, an image as shown in FIG. 20 is displayed on the display unit 220.

(Appendix 1 regarding the second mode)
As can be seen from FIGS. 18 and 19, the map display processing unit 212 displays a large symbol 30 indicating the shooting point of the frame i while the default visual field image is displayed, and the visual field image designated by the user. While the symbol is displayed, the symbol 30 indicating the shooting point of the frame i may be displayed small.

(Appendix 2 regarding the second mode)
In step S482, the image cutout unit 215 may perform the following process instead of performing the process of displaying the thumbnail.

That is, the image cutout unit 215 may display a button corresponding to the orientation indicated by the extended orientation data for each extended orientation data included in the reproduction control data Vi.

Specifically, the image cutout unit 215 may display a plurality of the above buttons side by side on the edge of the display screen.

Then, when any button is pressed, the image cutout unit 215 may display a visual field image having an orientation corresponding to the pressed button.

Alternatively, the image cutout unit 215 displays a pull-down menu that can display a field-of-view image indicated by any extended azimuth data among one or more extended azimuth data included in the reproduction control data Vi. Also good.

(Advantages of the playback device 200 set in the second mode)
According to the configuration of the playback device 200 set to the second mode, the viewer can display the shooting position of the current frame and the playback time of the current frame by viewing the map screen. The desired field-of-view image can be displayed while confirming the thumbnail of the field-of-view image, the location of the object in the field-of-view image, and the orientation in which the object is viewed from the shooting point. it can.

(Appendix 1 of Embodiment 2)
The playback device 200 is not only a playback device such as a television or a digital video recorder described above, but also a digital camera, a digital movie, a portable movie player, a mobile phone, a car navigation system, a portable DVD player, a PC, etc. Is widely applicable to devices that handle

In addition, the playback device according to the present invention is not limited to a playback device including a display, and a playback device that does not include a display and displays a moving image on an external display is also included in the scope of the playback device according to the present invention. .

(Appendix 2 of Embodiment 2)
Although the playback device 200 includes the first mode and the second mode, the playback device according to the present invention is not limited to such a configuration.

That is, a playback apparatus that performs only one of the first mode operation and the second mode operation is also included in the scope of the playback apparatus according to the present invention.

(Appendix 3 of Embodiment 2)
For example, the playback device 200 may play back an omnidirectional video as follows.

That is, the playback device 200 includes a virtual dome-shaped screen on which the movie texture of the omnidirectional video is pasted, and a virtual camera that is arranged at the center of the screen and that can change the orientation and position. It may be displayed in a certain area in the display screen, and a portion reflected in the virtual camera in the omnidirectional video may be displayed in another area in the display screen.

In this case, the playback apparatus 200 identifies the direction in which the virtual camera should face from the cut-out center coordinates included in the basic orientation data, and sets the orientation of the virtual camera so that it faces the same direction as the identified direction. I do not care.

Alternatively, the basic orientation data may include data indicating the direction in which the virtual camera should face instead of the cut-out center coordinates, and the playback device 200 sets the orientation of the virtual camera with reference to the data. May be.

In this case, in step S2, the data generation device 100 accepts an operation for designating the orientation of the virtual camera instead of the operation for inputting the cut-out center coordinates, and the data generated by the above operation is used instead of the cut-out center coordinates. Basic azimuth data as it is included is generated.

<Embodiment 3>
A free viewpoint moving image processing system according to still another embodiment of the present invention will be described with reference to FIG.

FIG. 21 is a schematic diagram of the free viewpoint moving image processing system according to the present embodiment.

As shown in FIG. 21, a free viewpoint video processing system 1 (hereinafter referred to as “system 1”) according to the present embodiment includes a data generation device 100 according to the first embodiment and a playback device 200 according to the second embodiment. Contains.

In the system 1, the data generation device 100 generates free viewpoint video data using the method described in the first embodiment, and the playback device 200 reads the free viewpoint video data and uses the method described in the second embodiment. Play an omnidirectional video.

The method of passing free viewpoint moving image data from the data generation device 100 to the playback device 200 may be a method using broadcasting or communication, or a method using a removable recording medium as a medium.

The system 1 may be owned by one user, or shared by a first user who owns the data generation device 100 and a second user who owns the playback device 200. There may be.

Then, the first user creates free viewpoint video data using the data generation device 100 of the system 1, and the second user browses the omnidirectional video using the playback device 200 of the system 1. May be.

The embodiment of the present invention has been described in detail with reference to the drawings. However, the specific configuration is not limited to this embodiment, and can be changed as appropriate within the scope of the effects of the present invention. It is. In addition, various modifications can be made without departing from the scope of the object of the present invention.

<Embodiment 4>
The control blocks of the data generation device 100 and the playback device 200 may be realized by a logic circuit (hardware) formed in an integrated circuit (IC chip) or the like, or realized by software using a CPU (Central Processing Unit). May be.

In the latter case, the data generation device 100 and the playback device 200 include a CPU that executes instructions of a program that is software that implements each function, and a ROM (in which the program and various data are recorded so as to be readable by a computer (or CPU)). A Read Only Memory) or a storage device (these are referred to as “recording media”), a RAM (Random Access Memory) for expanding the program, and the like. And the objective of this invention is achieved when a computer (or CPU) reads the said program from the said recording medium and runs it. As the recording medium, a “non-temporary tangible medium” such as a tape, a disk, a card, a semiconductor memory, a programmable logic circuit, or the like can be used. The program may be supplied to the computer via an arbitrary transmission medium (such as a communication network or a broadcast wave) that can transmit the program. The present invention can also be realized in the form of a data signal embedded in a carrier wave in which the program is embodied by electronic transmission.

The present invention is not limited to the above-described embodiments, and various modifications are possible within the scope shown in the claims, and embodiments obtained by appropriately combining technical means disclosed in different embodiments. Is also included in the technical scope of the present invention.

[Summary]
In the data generation device (for example, the data generation device 100) according to aspect 1 of the present invention, the playback device (for example, the playback device 200) uses the target frame for all or part of the frames of the omnidirectional video generated by moving shooting. A data generation unit (for example, basic azimuth data generation unit 111 and extended azimuth data generation unit 112) that generates a plurality of data (for example, basic azimuth data and extended azimuth data) to be referred to in order to extract a plurality of images to be displayed from The data generation unit generates a plurality of azimuth data as the plurality of data by generating azimuth data indicating the azimuth for each of a plurality of different azimuths, and the plurality of azimuths Each piece of data is indicated by the playback device as the display target image indicated by the direction data from the shooting point of the target frame. The data to be referred to excise the image showing the state of the field of view when viewed.

Here, the omnidirectional video refers to a video in which all or almost all directions from each shooting point on the moving shooting shooting route are shown. Further, it is assumed that the target frame is omnidirectional still image data shot at a certain shooting point.

According to the above configuration, the data generation device includes an image showing a state of view when viewing a certain direction from the shooting point, and a view showing a state of view when viewing another direction from the shooting point. A plurality of azimuth data for cutting out the image from the target frame is generated.

Further, the playback device cuts out these images from the target frame with reference to the plurality of azimuth data, and plays back these images in a period in which these images should be played back.

Therefore, the data generation device is configured such that each viewer when viewing a plurality of different directions from the same shooting point is viewed by a viewer (a user of the playback device) of a playback video clipped from the omnidirectional video data generated by moving shooting. It can be said that there is an effect of generating data that makes it possible to confirm the state of view at the same time.

In the data generation device according to aspect 2 of the present invention, in the above aspect 1, the data generation unit (for example, a portion including the basic azimuth data generation unit 111, the extended azimuth data generation unit 112, and the reproduction control data generation unit 115) The control data including the plurality of azimuth data (for example, reproduction control data) is generated, and the data generation unit adds the azimuth data to each of the plurality of azimuth data as the control data. Control data that includes identification information for identification from the orientation data may be generated.

According to the above-described configuration, the data generation device determines that the playback device specifies which azimuth data of the plurality of azimuth data is simply by referring to the azimuth data. It has the further effect of making it possible.

In the data generation device according to aspect 3 of the present invention, in the aspect 1 or 2, the data generation unit generates control data including the plurality of azimuth data, and the data generation unit As the control data, specific azimuth data (for example, basic azimuth data), which is azimuth data that the playback device should use most preferentially in order to cut out the image, is included in the plurality of azimuth data. Control data may be generated.

Here, the specific image among the plurality of images that can be cut out from the target frame using the plurality of azimuth data is the image that the user of the reproduction device most wants to view for the user of the data generation device. It shall be.

According to the above configuration, the data generation device generates the specific image as the specific orientation data by generating the orientation data for the playback device to cut out the specific image from the target frame. It is possible to make it easier for the user to touch the eyes.

In the data generation device according to aspect 4 of the present invention, in the above aspect 1 or 2, the display order of the image that the data generation unit extracts for each of the plurality of azimuth data with reference to the azimuth data. Display order data (e.g., priority display order data) is generated, and the playback device displays a plurality of images so that images with a relatively high display order are displayed relatively quickly during the period of the target frame. An image may be displayed.

According to the above configuration, the data generation device can cause the user of the reproduction device to view the plurality of images in the order that the user of the data generation device wants the user of the reproduction device to view the image. , There is an additional effect.

The data generation device according to aspect 5 of the present invention is the data generation apparatus according to any one of the aspects 1 to 4, wherein the omnidirectional video is a video generated by shooting while moving along a predetermined route. Distribution data generation processing for generating distribution data including omnidirectional video data and map data indicating a map (for example, map 10) of an area where the predetermined route (for example, a shooting route represented by line 12) is located. May be provided.

According to the above configuration, the data generation device has an additional effect that the user of the playback device can check the map of the area where the omnidirectional video is captured.

A playback apparatus (for example, playback apparatus 200) according to aspect 6 of the present invention refers to cutting out a plurality of display target images from the target frame for all or some of the omnidirectional video generated by moving shooting. A data reference processing unit (for example, an azimuth data analyzing unit 214) that refers to a plurality of data to be processed, and the plurality of data are a plurality of azimuth data indicating different azimuths, Is the data to be referred to in order for the data reference processing unit to cut out an image showing the state of the field of view when the azimuth indicated by the azimuth data is viewed from the shooting point of the target frame as the display target image. A reproduction processing unit (for example, the image clipping unit 215) that reproduces a plurality of images clipped from the target frame for all or some of the frames. Eteiru.

Therefore, the playback device has different views when viewing a plurality of different directions from the same shooting point to a viewer (a user of the playback device) of the playback video clipped from the omnidirectional video data generated by moving shooting. It can be said that there is an effect that the state of can be confirmed at the same time.

In the playback device according to aspect 7 of the present invention, in the aspect 6, the data reference processing unit refers to control data including the plurality of azimuth data, and the control data referred to by the data reference processing unit includes Specific orientation data, which is orientation data to be referred to most preferentially in order for the device to cut out the image, may be included in the plurality of orientation data.

Here, the specific image of the plurality of images that can be cut out from the target frame using the plurality of azimuth data is the image that the user of the playback device most wants to view for the producer of the omnidirectional video. Suppose that The specific image is an image cut out using the specific azimuth data.

According to the above configuration, the reproducing apparatus has an additional effect that the omnidirectional video can be reproduced in such a manner that the specific image is easily touched by the user's eyes.

In the playback device according to aspect 8 of the present invention, in the aspect 6 or 7, the data reference processing unit refers to the plurality of azimuth data, and the own device uses the azimuth data for each of the plurality of azimuth data. The reproduction processing unit refers to display order data indicating the display order of images to be cut out by reference, and the reproduction processing unit is configured to display the plurality of images having a relatively high display order relatively quickly in a target frame period. One image may be reproduced.

According to the above configuration, the reproducing apparatus has an additional effect that the user can browse the plurality of images in the order in which the creator of the omnidirectional video wants the user to browse.

In the playback device according to aspect 9 of the present invention, in any of the aspects 6 to 8, the omnidirectional video is a video generated by shooting while moving along a predetermined route, and the predetermined video A map display processing unit (for example, a map display processing unit 212) that displays a map of the region where the route of the map is located (for example, the map 29), and the map display processing unit applies a target for each frame of the omnidirectional video. During the frame period, information (for example, the symbol 30) indicating the shooting point of the target frame may be displayed on the map.

According to said structure, the said reproducing | regenerating apparatus can make a user grasp | ascertain the imaging | photography point which is a position which can see the to-be-photographed object (non-animals, such as a building) reflected in this visual field image at the time of the said visual field image display. , There is an additional effect.

In the playback device according to aspect 10 of the present invention, in the aspect 9 described above, the map display processing unit includes, for each frame of the omnidirectional video, information indicating the shooting point of the target frame and the playback process in the period of the target frame. Information (for example, symbol 30) indicating the position of the subject in the image to be cut out from the target frame by the unit may be displayed on the map.

According to the above configuration, the playback device can cause the user to grasp the approximate position of the subject (non-animal such as a building) shown in the visual field image when the visual field image is displayed. The effect which becomes.

The present invention can also be configured as follows.

(First configuration)
The omnidirectional image data having a 360 ° field of view, which is taken using an omnidirectional camera while moving along a predetermined route, and map data that is a map of the route are input, and the input omnidirectional data is used. An image data generation device that generates free viewpoint image data that enables reproduction of an image in an arbitrary line-of-sight direction from a shooting position corresponding to coordinates on map data,
A basic line-of-sight direction data generation unit that generates basic line-of-sight direction data, which is initial line-of-sight direction data, for each frame of omnidirectional image data;
An extended gaze direction data generating unit that generates at least one extended gaze direction data that is gaze direction data different from the basic gaze direction data for each frame of omnidirectional image data;
A map position data generating unit that generates map position data obtained by converting the shooting position of each frame of the omnidirectional image data into coordinates on the map data;
A line-of-sight direction control data generating unit that generates line-of-sight direction control data from the basic line-of-sight direction data, the extended line-of-sight direction data, and the map position data;
An encoding unit for encoding the omnidirectional image data;
A multiplexing unit that multiplexes the omnidirectional image data, the map data, and the line-of-sight direction control data, and generates the free viewpoint image data;
An image data generation apparatus comprising:

(Second configuration)
The basic line-of-sight direction data and the plurality of extended line-of-sight direction data include information for identifying each of the basic line-of-sight direction data and information about the line-of-sight direction when displaying the omnidirectional image. An image data generation device according to the configuration.

(Third configuration)
In the gaze direction control data, when the basic gaze direction data and at least one or more extended gaze direction data are included for each frame of omnidirectional image data, which gaze direction data is preferentially displayed. The image data generation device according to the first configuration, characterized in that the image data generation device includes information related to a ranking indicating the above.

(Fourth configuration)
In an image data reproduction apparatus for reproducing free viewpoint image data that enables reproduction of an image of a field of view in an arbitrary direction viewed from a viewpoint moving along a predetermined path,
An initial value for each frame of 360 ° field of view taken with an omnidirectional camera while moving along the route, map data that is a map of the route, and each frame of the omnidirectional image data. Map data indicating basic gaze direction data that is gaze direction data, at least one or more extended gaze direction data that is gaze direction data different from the basic gaze direction data, and the shooting position of each frame of the omnidirectional image data And obtaining free viewpoint image data in which line-of-sight direction control data including the map position data converted into the upper coordinates is multiplexed,
A separation unit that separates omnidirectional image data, the map data, and the line-of-sight direction control data from free viewpoint image data;
A gaze direction control unit that selects at least one or more basic gaze direction data or extended gaze direction data from the gaze direction control data, and outputs cut-out center coordinates of the selected extended gaze direction data;
An image cutout unit that obtains cutout center coordinates, cuts out part of the previous omnidirectional image data around the cutout center coordinates and generates a display image, and a display unit that obtains and displays the display image is provided. An image data reproducing apparatus characterized by the above.

(Fifth configuration)
The line-of-sight direction control unit outputs cut-out center coordinates of a plurality of line-of-sight direction data for the same frame of the omnidirectional image data, and simultaneously uses the cut-out center coordinates of the plurality of line-of-sight direction data from the image cut-out unit. The image data reproducing device according to the fourth configuration, wherein the display unit is notified of which of the display images generated is the main screen.

The present invention can be suitably applied to a device that distributes an omnidirectional video and a device that plays back an omnidirectional video.

100 Free-viewpoint video data generator (data generator)
110 Control Unit 111 Basic Direction Data Generation Unit (Data Generation Unit)
112 Extended orientation data generator (data generator)
115 Reproduction control data generation unit (data generation unit)
116 Multiplexing processing unit (distribution data generation processing unit)
200 Free viewpoint video playback device (playback device)
210 Control unit 212 Map display processing unit (map display processing unit)
214 Direction Data Analysis Unit (Data Reference Processing Unit)
215 Image cutout unit (reproduction processing unit)

Claims

A data generation unit that generates a plurality of data to be referred to in order for the playback device to extract a plurality of images to be displayed from the target frame for all or a part of the frames of the omnidirectional video generated by the moving shooting,
The data generation unit generates a plurality of azimuth data as the plurality of data by generating azimuth data indicating the azimuth for each of a plurality of different azimuths,
Each of the plurality of azimuth data is referred to by the playback device to cut out an image showing a field of view when viewing the azimuth indicated by the azimuth data from the shooting point of the target frame as the display target image. A data generation device characterized in that the data is power data.
The data generation unit generates control data including the plurality of azimuth data,
The data generation unit generates control data such that each of the plurality of azimuth data includes identification information for identifying the azimuth data from other azimuth data as the control data. The data generation device according to claim 1.
The data generation unit generates control data including the plurality of azimuth data,
The data generation unit may include, as the control data, specific azimuth data, which is azimuth data to be used most preferentially for the playback device to cut out the image, is included in the plurality of azimuth data. The data generation apparatus according to claim 1, wherein the control data is generated.
The data generation unit generates, for each of the plurality of azimuth data, display rank data indicating a display rank of an image to be cut out by referring to the azimuth data.
The playback apparatus is configured to display a plurality of images so that an image with a relatively high display order is displayed relatively quickly during a target frame period. Item 3. The data generation device according to Item 1 or 2.
The omnidirectional video is a video generated by shooting while moving along a predetermined route,
5. A distribution data generation processing unit that generates distribution data including the omnidirectional video data and map data indicating a map of an area where the predetermined route is located. The data generation device according to any one of the above.
A data reference processing unit that refers to a plurality of data to be referred to in order to cut out a plurality of images to be displayed from the target frame for all or some of the frames of the omnidirectional video generated by the moving shooting,
The plurality of data is a plurality of azimuth data indicating different azimuths,
Each of the plurality of azimuth data is for the data reference processing unit to cut out an image showing a field of view when viewing the azimuth indicated by the azimuth data from the shooting point of the target frame as the display target image. Data to refer to
A playback apparatus comprising: a playback processing unit that plays back a plurality of images cut out from a target frame for all or some of the frames.
The data reference processing unit refers to control data including the plurality of azimuth data,
The control data referred to by the data reference processing unit includes specific orientation data, which is orientation data that should be most preferentially used for the device to cut out the image, in the plurality of orientation data. The playback apparatus according to claim 6.
The data reference processing unit refers to the plurality of azimuth data, and refers to display order data indicating the display order of images that the own apparatus refers to the azimuth data for each of the plurality of azimuth data,
8. The reproduction processing unit reproduces the plurality of images so that an image having a relatively high display order is displayed relatively quickly during a target frame period. The playback device described in 1.
The omnidirectional video is a video generated by shooting while moving along a predetermined route,
A map display processing unit for displaying a map of an area where the predetermined route is located;
9. The map display processing unit according to any one of claims 6 to 8, wherein the map display processing unit displays, on the map, information indicating a shooting point of the target frame for each frame of the omnidirectional video during the target frame period. The playback apparatus according to claim 1.
The map display processing unit, for each frame of the omnidirectional video, during the period of the target frame, information indicating the shooting point of the target frame and the subject that is captured in the image to be cut out from the target frame by the reproduction processing unit The reproduction apparatus according to claim 9, wherein information indicating a position is displayed on the map.