JP6921703B2 - Encoding device, its control method, and control program, and imaging device - Google Patents

Description

The present invention relates to a coding device, a control method thereof, a control program, and an imaging device, and more particularly to an imaging device capable of encoding a video signal.

With the widespread use of digital cameras or digital camcorders, which are one of the imaging devices, still images or moving images are taken by a digital method. The image obtained by the digital method can be easily subjected to image processing by arithmetic processing using software, firmware, or the like. As one of the image processing, for example, aberration correction of the lens or adjustment of the dynamic range can be mentioned. Such an image processing technique is called computational photography.

For example, an imaging device that records depth information (called a depth map) generated by a depth estimation method represented by a DFD (Dept from Defocus) method or a stereo method, and performs image processing based on the depth map. Yes (see Patent Document 1).

Japanese Unexamined Patent Publication No. 2014-168227

By the way, since a huge amount of calculation is required when generating a depth map, the timing of generating the depth map is discrete with respect to the captured image in the case of continuous shooting of still images or recording of moving images.

The depth map is two-dimensional data having depth information corresponding to each point in the captured image, and the amount of the information increases in proportion to the number of pixels, so it is desirable to compress and record the depth map. Therefore, if the depth map is treated as a monochrome image, the compression technology used for image data can be applied, and by applying the same compression method as the captured image, compression recording is performed without adding a new encoder. can do.

However, when the captured image and the depth map are coded in a time-division manner by one encoder, so-called processing latency occurs with the coding process of the depth map. As a result, the coding process for the captured image is locally reduced, and it becomes difficult to keep the shooting interval constant during continuous shooting of still images, for example. In addition, the frame rate may decrease when shooting a moving image.

Therefore, an object of the present invention is to provide an image pickup apparatus capable of suppressing continuous shooting of still images and a decrease in frame rate of moving images when encoding a captured image and a depth map, a control method thereof, and a control program. To do.

In order to achieve the above object, the coding apparatus according to the present invention encodes the RAW data obtained by imaging and the depth map indicating the depth of each pixel of the RAW data to obtain the coded data. The apparatus includes a dividing means for dividing the depth map into a plurality of divided areas based on the ratio of the update interval of the RAW data and the update interval of the depth map, and the RAW data and the plurality of divided areas. It is characterized by having a coding means for encoding a depth map by time division.

According to the present invention, it is possible to suppress continuous shooting of still images and a decrease in the frame rate of moving images.

It is a block diagram for demonstrating the structure of an example of the image pickup apparatus according to 1st Embodiment of this invention. It is a figure which shows the structure of the RAW data which is the output of the image pickup part shown in FIG. It is a figure for demonstrating an example of the structure of the image pickup element provided in the image pickup part shown in FIG. It is a flowchart for demonstrating the coding processing of the RAW data and the depth map performed by the RAW data coding / decoding processing unit shown in FIG. It is a figure which shows an example of the RAW data and the depth map which are encoded by the RAW data coding / decoding processing part shown in FIG. It is a figure for demonstrating an example of the area division of the Debs map performed by the RAW data coding / decoding processing unit shown in FIG. It is a figure which shows an example about the result of the coding processing performed by the RAW data coding / decoding processing unit 103 shown in FIG. It is a block diagram which shows an example about the structure of the RAW data coding / decoding processing part used in the camera by 2nd Embodiment of this invention. It is a flowchart for demonstrating the RAW data and depth map coding processing performed by the RAW data coding / decoding processing unit provided in the camera by the 2nd Embodiment of this invention. It is a figure which shows an example of the re-region division of the Debs map performed by the RAW data coding / decoding processing unit provided in the camera by the 2nd Embodiment of this invention. It is a figure which shows an example about the result of the coding processing performed by the RAW data coding / decoding processing unit shown in FIG. It is a figure which shows an example in which the partial region shown in FIG. 10 is rearranged.

Hereinafter, an example of the image pickup apparatus according to the embodiment of the present invention will be described with reference to the drawings.

[First Embodiment]
FIG. 1 is a block diagram for explaining the configuration of an example of the image pickup apparatus according to the first embodiment of the present invention. FIG. 1A is a block diagram showing the configuration of the image pickup apparatus, and FIG. 1B is a block diagram showing the configuration of the RAW data coding / decoding processing unit shown in FIG. 1A.

The illustrated imaging device is, for example, a digital camera (hereinafter, simply referred to as a camera) 100 capable of capturing still images and moving images. The camera 100 includes a control unit 101, and the control unit 101 controls the entire camera 100.

The image pickup unit 102 includes a lens optical system and an image pickup element, and the lens optical system includes an optical lens and an aperture, as well as a focus control and a lens drive unit, and is capable of optical zooming. The image pickup device is, for example, a CCD image sensor or a CMOS sensor, and generates an electric signal according to an optical image (subject image) formed through the lens optical system. Then, the image sensor converts the electric signal into a digital signal and sends it as RAW data to the RAW data coding / decoding processing unit 103 and the depth map generation unit 104.

FIG. 2 is a diagram showing a configuration of RAW data which is an output of the imaging unit shown in FIG.

As shown, the RAW data is composed of four color elements, R (red), G1 (green), G2 (green), and B (blue) of the Bayer array.

FIG. 3 is a diagram for explaining an example of the configuration of an image pickup device provided in the image pickup unit shown in FIG. FIG. 3A is a plan view, and FIG. 3B is a side view.

In the image sensor, each of the pixels includes one microlens (ML) 200 and two photoelectric conversion units 201 and 202 associated with the ML 200. Then, each pixel of the image sensor receives light flux emitted from different regions of the exit pupil of the lens optical system. Here, RAW data (referred to as an A image) is output according to the output of the photoelectric conversion unit 20, and RAW data (referred to as a B image) is output according to the output of the electrical conversion unit 202.

With reference to FIG. 1 again, as shown in FIG. 1 (b), the RAW data coding / decoding processing unit 103 includes a RAW data coding processing unit 103a and a RAW data decoding processing unit 103b. Independently performs coding processing and decoding processing.

The RAW data coding processing unit 103a mainly outputs the RAW data input from the imaging unit 102 and the coded data obtained by encoding the depth map generated by the depth map generation unit 104 described later to the memory 105. .. The RAW data decoding processing unit 103b sends the RAW data, which is the decoding data obtained by decoding the encoded data stored in the memory 105, to the image processing unit 107, and outputs the decoded depth map to the memory 105.

The RAW data and the depth map are time-division-encoded / decoded as described later.

The depth map generation unit 104 generates a depth map by calculating the depth distance of each point (for each pixel) in the image using the A image and the B image by using a known DFD method, stereo method, or the like. Then, the depth map generation unit 104 outputs the depth map to the memory 105.

The memory 105 is used to store various data output from each unit constituting the camera 100. The memory 105 is, for example, a storage area composed of a volatile memory. The memory I / F unit 106 mediates memory access requests from each unit of the camera 100, and performs read / write control for the memory 105.

The image processing unit 107 performs image processing such as demosaic processing, noise removal processing, optical distortion correction processing, and color correction processing on the RAW data input from the RAW data coding / decoding processing unit 103 to perform image data. To generate. Then, the image processing unit 107 outputs the image data to the display unit 108 and the image data coding unit 109.

The image processing unit 107 performs image processing at a timing independent of the generation of the coded data by the RAW data coding / decoding processing unit 103. That is, the image processing unit 107 performs image processing on the RAW data obtained by decoding the coded data stored in the memory 105 by the RAW data coding / decoding processing unit 103 at independent timings. Further, the image processing unit 107 performs image alignment and image composition processing on the A image and the B image using the decoded depth map, and executes the refocus processing for the virtual focus distance.

The display unit 108 is, for example, a liquid crystal display or an organic EL display, and displays an image corresponding to the image data output from the image processing unit 107.

The image data coding unit 109a encodes the image data output from the image processing unit 107 according to the coding method of, for example, JPEG in the case of a still image and MPEG, for example, in the case of a moving image. The coded data obtained by the conversion process is output to the memory 105.

The recording processing unit 110 records various data such as coded data stored in the memory 105 on the recording medium 111. The recording medium 111 is, for example, a recording medium composed of a non-volatile memory.

FIG. 4 is a flowchart for explaining the RAW data and depth map coding processing performed by the RAW data coding / decoding processing unit shown in FIG. The processing related to the illustrated flowchart is performed under the control of the control unit 101.

Further, FIG. 5 is a diagram showing an example of RAW data and depth map encoded by the RAW data coding / decoding processing unit shown in FIG.

Here, RAW data of 4096 × 2160 pixels is recorded as a moving image of 60 frames / second. Further, it is assumed that the depth map is generated at an update interval of 15 frames / second, which is a quarter of the frame rate of RAW data. The RAW data can be recorded by encoding both the A image and the B image, but here, one RAW data is encoded for simplification of the description.

When the coding process is started, the RAW data coding / decoding processing unit 103 acquires the depth map update interval (frame rate) with respect to the frame rate of the RAW data from the depth map generation unit 104 (step S401). Although the update interval is acquired from the depth map generation unit 104 here, the update interval may be obtained by measuring the time when the first depth map is generated from the start of recording the RAW data. ..

Subsequently, the RAW data coding / decoding processing unit 103 divides the depth map area based on the ratio between the frame rate of the RAW data and the update interval of the depth map (step S402).

FIG. 6 is a diagram for explaining an example of region division of the Debs map performed by the RAW data coding / decoding processing unit shown in FIG. FIG. 6A is a diagram showing an example of division in the horizontal direction and the vertical direction, and FIG. 6B is a diagram showing an example of division in the horizontal direction. Further, FIG. 6C is a diagram showing an example of division in the vertical direction.

Here, since one depth map is generated for the RAW data of four frames, the depth map is divided into four division areas d [0] to d [3] as shown in FIG. When dividing, as shown in FIG. 6 (a), it is divided in the horizontal / vertical direction, or as shown in FIG. 6 (b), it is divided only in the horizontal direction. Further, as shown in FIG. 6C, it may be divided only in the vertical direction. If the area of each divided region is uniform, it may be divided into any of FIGS. 6 (a) to 6 (c).

Subsequently, the control unit 101 determines whether or not the recording of the coded data by the RAW data coding / decoding processing unit 103 has stopped (step S403). When the recording of the coded data is stopped (YES in step S403), the control unit 101 ends the coding process by the RAW data coding / decoding processing unit 103.

On the other hand, if the recording of the coded data is not stopped (NO in step S403), the control unit 101 controls the RAW data coding / decoding processing unit 103 to perform the RAW data coding process (step S404). Then, when the RAW data coding process is completed, the RAW data coding / decoding processing unit 103 determines the division area of the depth map to be encoded (step S405).

As shown in FIG. 6, an index number is added to each of the divided regions, and the RAW data coding / decoding processing unit 103 has an index counter that increments the index number. The RAW data coding / decoding processing unit 103 increments the index counter for each frame of the RAW data. Further, the RAW data coding / decoding processing unit 103 resets the index counter when the index number reaches the number of divisions. Then, the RAW data coding / decoding processing unit 103 determines the division area based on the count value of the index counter.

The depth map is stored in the memory 105 as described above. Therefore, the RAW data coding / decoding processing unit 103 sets information such as the dismap start address, starting point coordinates, and the number of horizontal / vertical pixels, and performs depth map coding processing of the divided region determined according to the information. (Step S406). Then, the process returns to step S403.

FIG. 7 is a diagram showing an example of the result of the coding process performed by the RAW data coding / decoding processing unit shown in FIG.

As described above, the RAW data coding / decoding processing unit 103 divides the depth map into regions based on the ratio of the frame rate of the RAW data and the update interval of the depth map. As a result, control is performed so that one depth map is distributed and encoded in the time axis direction. Therefore, as shown in FIG. 7, it is possible to level the coding process of the depth map and record the RAW data to the maximum extent.

Further, here, an example in which the frame rate of RAW data and the update interval of the depth map are fixed has been described. The update interval of the depth map may fluctuate with respect to the frame of the RAW data due to factors such as the degree of congestion of the access of the memory 105. In this case, if a buffer area for temporarily storing a plurality of depth maps is provided in the memory 105, the leveling of the coding timing of the depth maps can be easily adjusted.

However, since the update interval of the depth map fluctuates and the buffer area may overflow, it is desirable to control the coding process of the depth map by measuring the accumulated amount in the buffer area. For example, in the case of continuous shooting of still images, priority is given to recording the depth map, and when the accumulated amount of the buffer area exceeds a predetermined threshold value, the update interval of RAW data is changed. This prevents the buffer area for the depth map from overflowing.

Also, in the case of moving images, it is not desirable to change the frame rate of RAW data, so if the accumulated amount of the buffer area exceeds a predetermined threshold value, the newly input depth map should be discarded. You may control it.

As described above, in the first embodiment of the present invention, the depth map is divided into regions based on the ratio of the frame rate of the RAW data and the update interval of the depth map, and the depth map is encoded. As a result, the coding load related to the depth map can be leveled, and continuous shooting of still images and a decrease in the frame rate of moving images can be suppressed.

[Second Embodiment]
Subsequently, the camera according to the second embodiment of the present invention will be described. The configuration of the camera according to the second embodiment is the same as that of the camera shown in FIG. 1, but the configuration of the RAW data coding / decoding processing unit 103 is different.

FIG. 8 is a block diagram showing an example of the configuration of the RAW data coding / decoding processing unit used in the camera according to the second embodiment of the present invention.

The illustrated RAW data coding / decoding processing unit 800 includes a RAW data coding processing unit 800a and a RAW data decoding processing unit 800b. Each of the RAW data coding processing unit 800a and the RAW data decoding processing unit 800b has a 4-core configuration capable of parallel processing the color elements (R, G1, G2, and B) in the RAW data. As a result, when processing the depth map, the divided area of the depth map is processed in parallel.

As shown, the RAW data coding processing unit 800a has a plane separation unit 801 and first to fourth coding processing cores 802 to 805. Further, the RAW data decoding processing unit 800b has a plane synthesis unit 810 and first to fourth decoding processing cores 812 to 815.

The plane separation unit 801 separates the RAW data into color elements R, G1, G2, and B, and sends them to the first to fourth coding processing cores 802 to 805, respectively. The first, second, third, and fourth coding processing cores 802, 803, 804, and 805 perform coding processing on the color elements R, G1, G2, and B, respectively.

The first, second, third, and fourth decoding processing cores 812, 813, 814, and 815 receive the encoded data and perform decoding processing on the color elements R, G1, G2, and B, respectively. Then, the plane synthesizing unit 811 performs synthesizing processing on the decoded color elements R, G1, G2, and B.

FIG. 9 is a flowchart for explaining the RAW data and depth map coding processing performed by the RAW data coding / decoding processing unit provided in the camera according to the second embodiment of the present invention.

The processing related to the illustrated flowchart is performed under the control of the control unit 101. In FIG. 9, the same steps as those in the flowchart shown in FIG. 4 are designated by the same reference numerals, and the description thereof will be omitted. Further, here, as in the first embodiment, RAW data of 4096 × 2160 pixels is recorded as a moving image of 60 frames / second. Further, it is assumed that the depth map is generated at an update interval of 15 frames / second, which is a quarter of the frame rate of RAW data.

In step S402, the RAW data coding / decoding processing unit 800 divides the depth map into four regions, for example, as shown in FIG. 6A.

After the process of step S405, the RAW data coding / decoding processing unit 800 further subdivides the area-divided dismap in step S402 in order to perform the coding process in four parallels (step S906).

FIG. 10 is a diagram showing an example of redomain division of the Debs map performed by the RAW data coding / decoding processing unit provided in the camera according to the second embodiment of the present invention.

In the process of step S906, each of the divided regions d [0] to d [3] is further divided into four subregions. In the illustrated example, the divided area d [0] is divided into partial areas d [00] to d [03], and the divided area d [1] is divided into partial areas d [10] to d [13]. Further, the divided area d [2] is divided into partial areas d [20] to d [23], and the divided area d [3] is divided into partial areas d [30] to d [33].

As described above, the depth map is stored in the memory 105. Therefore, the RAW data coding / decoding processing unit 800 collectively sets information such as the start address of the dismap, the starting point coordinates, and the number of horizontal / vertical pixels for the four division areas, and determines the division according to the information. The area depth map coding process is performed (step S907). Then, the process returns to step S403.

FIG. 11 is a diagram showing an example of the result of the coding process performed by the RAW data coding / decoding processing unit shown in FIG.

As described above, the RAW data coding / decoding processing unit 800 divides the depth map into regions based on the ratio of the frame rate of the RAW data and the update interval of the depth map. Further, the RAW data coding / decoding processing unit 800 subdivides the division area based on the number of parallel processes of the coding processing. As a result, one depth map is distributed in the time axis direction, and control is performed to perform parallel coding processing. Therefore, as shown in FIG. 11, the RAW data can be recorded to the maximum by leveling the depth map coding process as compared with the first embodiment.

Here, the division area was subdivided as shown in FIG. 10 according to the number of parallel processes of the RAW data coding / decoding processing unit 800. On the other hand, the depth map having the four divided regions d [0] to d [3] shown in FIG. 10 may be rearranged.

FIG. 12 is a diagram showing an example in which the partial region shown in FIG. 10 is rearranged.

As shown in FIG. 12, a depth map is generated in which partial regions are interleaved and rearranged in pixel units at positions R, G1, G2, and B in the Bayer array. As a result, the parallel coding process can be performed by pretending to be RAW data without being aware of the four subregions.

As described above, in the second embodiment of the present invention, the depth map is divided into regions according to the ratio of the frame rate of the RAW data and the update interval of the depth map, and further divided based on the number of parallel processes of the coding process. The region is subdivided and the depth map is encoded. As a result, the coding load related to the depth map can be leveled, and continuous shooting of still images and a decrease in the frame rate of moving images can be suppressed.

Although the present invention has been described above based on the embodiments, the present invention is not limited to these embodiments, and various embodiments within the scope of the gist of the present invention are also included in the present invention. ..

For example, the function of the above-described embodiment may be used as a control method, and the encoding device may be made to execute this control method. Further, a program having the functions of the above-described embodiment may be used as a control program, and the control program may be executed by a computer provided in the encoding device. The control program is recorded on, for example, a computer-readable recording medium.

[Other Embodiments]
The present invention supplies a program that realizes one or more functions of the above-described embodiment to a system or device via a network or storage medium, and one or more processors in the computer of the system or device reads and executes the program. It can also be realized by the processing to be performed. It can also be realized by a circuit (for example, ASIC) that realizes one or more functions.

101 Control unit 102 Imaging unit 103,800 RAW data coding / decoding processing unit 104 Depth map generation unit 105 Memory 106 Memory I / F unit 107 Image processing unit 108 Display unit 110 Recording processing unit 111 Recording medium

Claims (11)

A coding device that obtains coded data by coding the RAW data obtained by imaging and the depth map showing the depth of each pixel of the RAW data.
A partitioning means for dividing the depth map into a plurality of partition areas based on the ratio of the update interval of the RAW data to the update interval of the depth map.
A coding means for coding the RAW data and the depth map having the plurality of divided regions by time division, and
A coding device characterized by having. The coding apparatus according to claim 1, wherein the depth map is temporarily stored in a memory. The dividing means further divides the divided area into a plurality of subregions.
The coding apparatus according to claim 1 or 2, wherein the coding means performs coding processing in parallel for each of the color elements of a fat map having the plurality of partial regions. The coding apparatus according to claim 3, wherein the dividing means divides a dividing region in the depth map into a plurality of partial regions based on the number of parallel processes in the coding means. The coding apparatus according to claim 4, wherein the color elements are arranged in a Bayer array, and the dividing means rearranges a partial region in the depth map so as to correspond to the Bayer array. When the RAW data is encoded at a predetermined update interval, if the amount of depth map stored in the memory exceeds a predetermined threshold value, the RAW data is characterized by having a control means for changing the update interval. The coding apparatus according to claim 2. It has a control means for discarding a depth map newly stored in the memory when the amount of the depth map stored in the memory exceeds a predetermined threshold value when the RAW data is encoded at a predetermined update interval. The coding apparatus according to claim 2. An imaging means that captures a subject and obtains RAW data,
A generation means for generating a depth map based on the RAW data, and
The coding apparatus according to any one of claims 1 to 7.
A recording means for recording the coded data obtained by the coding apparatus on a recording medium, and
An imaging device characterized by having. The imaging apparatus according to claim 8, further comprising a decoding means for decoding the encoded data recorded on the recording medium to obtain the decoded data. It is a control method of a coding apparatus that obtains coded data by coding a RAW data obtained by imaging and a depth map showing the depth of each pixel of the RAW data.
A division step of dividing the depth map into a plurality of division areas based on the ratio of the update interval of the RAW data and the update interval of the depth map, and
A coding step in which the RAW data and the depth map having the plurality of divided regions are coded by time division, and
A control method characterized by having. A control program used in a coding device that obtains coded data by coding a RAW data obtained by imaging and a depth map indicating the depth of each pixel of the RAW data.
In the computer provided in the encoding device,
A division step of dividing the depth map into a plurality of division areas based on the ratio of the update interval of the RAW data and the update interval of the depth map, and
A coding step in which the RAW data and the depth map having the plurality of divided regions are coded by time division, and
A control program characterized by executing.

Images