WO2013069445A1 - Three-dimensional imaging device and image processing method - Google Patents

Abstract

An image processing method related to an embodiment of the present invention is used for a three-dimensional imaging device that includes a single imaging optical system and a single imaging element, wherein the imaging element treats adjoining multiple pixels as one block, has a color filter for any one color per each block among color filters for three colors, and is capable of simultaneously obtaining multiple parallax images upon receiving rays of light which passed different regions of the imaging device. In this method, multiple parallax images are obtained from the imaging element, and when computing pixel values for defective pixels inside the multiple parallax images by means of interpolation, neighboring pixels which have the same parallax information and color information as the defective pixels are used as neighboring pixels to be used for interpolating the defective pixels.

Description

Stereo imaging device and image processing method

The present invention relates to a stereoscopic imaging apparatus and an image processing method, and more particularly to a technique for correcting defective pixels of a plurality of parallax images (3D images) photographed by a single photographing optical system.

2. Description of the Related Art Conventionally, there has been proposed an image processing device that can use an imaging device in which one microlens is assigned to a plurality of pixels and insert an arbitrary two-dimensional image in an arbitrary depth direction in a stereoscopic image. (Patent Document 1). This Patent Document 1 describes that a plurality of parallax images having different parallaxes are generated from a plurality of pixels to which one microlens is assigned.

Patent Document 2 discloses normal defect information that is information regarding pixel defects when the imaging device is set to the first light receiving sensitivity, and a second light receiving sensitivity that is higher than the first light receiving sensitivity. High-sensitivity defect information, which is information relating to pixel defects at the time of setting, is stored in a defect memory, and normal defect information or high-sensitivity defect information is used according to the setting of light receiving sensitivity for the image sensor at the time of imaging. An image correction apparatus that corrects pixel defects is described.

JP 2010-0608018 A JP 2005-311732 A

The imaging apparatus described in Patent Literature 1 can acquire a plurality of parallax images having different parallax from a plurality of pixels to which one microlens is assigned. However, the imaging device described in Patent Document 1 has pixels having different phase differences when correction is performed using peripheral pixels of the same color when correcting defective pixels (scratch pixels) in the plurality of parallax images. May be lost, and the stereoscopic effect may be lost.

The invention described in Patent Document 2 corrects white spot noise due to a long exposure time in a microscope camera or the like, and does not correct defective pixels in a plurality of parallax images.

The present invention has been made in view of such circumstances. When correcting defective pixels in a plurality of parallax images, a stereoscopic imaging apparatus and an image processing method capable of correcting defective pixels that do not impair the stereoscopic effect. The purpose is to provide.

In order to achieve the above object, a stereoscopic imaging apparatus according to an aspect of the present invention includes a single imaging optical system and a plurality of pixels adjacent in the horizontal and vertical directions as one block, and each block includes three primary colors. An image sensor with a color filter of any one of the color filters, which receives light that has passed through different areas of the photographic optical system and can simultaneously acquire multiple parallax images in the horizontal and vertical directions A defective pixel correction unit that calculates a pixel value of a defective pixel in an image acquired from the imaging device by interpolating a pixel value of a peripheral pixel of the defective pixel. The correction means uses peripheral pixels having the same parallax information and the same color information as the defective pixels of the plurality of parallax images as the peripheral pixels used for interpolation of the defective pixels to be corrected.

In the case of an image acquired from the image sensor according to this aspect, the peripheral pixel closest to the defective pixel in the image is not a pixel having the same parallax information and the same color information as the defective pixel. However, according to one aspect of the present invention, peripheral pixels having the same parallax information and the same color information as the defective pixels of the plurality of parallax images are used as the peripheral pixels used for interpolation of the defective pixel to be corrected. Thus, the stereoscopic effect can be prevented from being impaired by the defective pixel after correction. Moreover, according to one aspect of the present invention, each of the plurality of parallax images can be a color image.

In the stereoscopic imaging device according to another aspect of the present invention, the plurality of pixels in one block of the imaging element are four pixels arranged in a square shape adjacent to each other in the vertical and horizontal directions, and the imaging element is four pixels in one block. 1 has a microlens that divides the light incident on the microlens into pupils and guides them to the light receiving surfaces of four pixels. This microlens functions as pupil dividing means that divides a light beam that has passed through different areas of the photographing optical system and makes it incident on four pixels.

In the stereoscopic imaging apparatus according to still another aspect of the present invention, the defective pixel correction means includes a distance between the center position of the defective pixel to be corrected and the center positions of the four pixels in the vertical and horizontal directions around the defective pixel, and the defect It is determined whether or not the defective pixel is included in the shorter four pixels of the interval between the center position of the pixel and the center position of the four pixels in the diagonally upper right and diagonally lower right directions around the defective pixel. If the determination unit determines that the four pixels having the shorter interval do not include the defective pixel, the pixel value of the four pixels having the shorter interval is interpolated to obtain the pixel value of the defective pixel to be corrected. When the calculation means determines that the defective pixel is included in the four pixels with the shorter interval, the defective pixel of the shorter four pixels and the pixel facing the defective pixel are removed, and the defect to be corrected It is preferable to calculate the pixel value of the pixel. As a result, the position (center of gravity) of the pixel calculated by interpolation can be matched with the position of the defective pixel to be corrected.

In the stereoscopic imaging apparatus according to still another aspect of the present invention, the defective pixel correction means includes a distance between the center position of the defective pixel to be corrected and the center positions of the four pixels in the vertical and horizontal directions around the defective pixel, and the defect It is determined whether or not the defective pixel is included in the shorter four pixels of the interval between the center position of the pixel and the center position of the four pixels in the diagonally upper right and diagonally lower right directions around the defective pixel. When it is determined that the defective pixel is not included, including the determining means, the pixel value of the defective pixel to be corrected is calculated by interpolating the pixel values of the four pixels having the shorter interval, and the defective pixel is included. If it is determined, it is preferable to interpolate the pixel values of the longer four pixels to calculate the pixel value of the defective pixel to be corrected. For example, when a G pixel in a Bayer array is a defective pixel as a color filter array, four pixels in the diagonally upper right and diagonally lower right directions are the same color pixels closest to the defective pixel, but these pixels include the defective pixel. In this case, four pixels in the vertical and horizontal directions are used as correction pixels. In addition, when the R pixel or the B pixel in the Bayer array is a defective pixel, the four pixels in the vertical and horizontal directions are the same color pixels closest to the defective pixel, but when these pixels include the defective pixel. The four pixels in the diagonal upper right and diagonal lower right directions are used as correction pixels.

A stereoscopic imaging apparatus according to still another aspect of the present invention includes a detection unit that detects whether horizontal shooting or vertical shooting, and, when horizontal detection is detected by the detection unit, performs vertical pixel mixing among a plurality of parallax images. When the vertical shooting is detected, the image processing apparatus further includes a pixel mixing unit that mixes the pixels in the horizontal direction among the plurality of parallax images, and the defective pixel correction unit detects a defect when the horizontal shooting is detected by the detection unit. The pixel value of the defective pixel to be corrected is calculated by interpolating the pixel value of the pixel in the vertical direction around the pixel, and when vertical shooting is detected, the pixel value of the pixel in the horizontal direction around the defective pixel is interpolated. It is preferable to calculate the pixel value of the defective pixel to be corrected.

The plurality of viewpoint images include a parallax image in the horizontal direction and a parallax image in the vertical direction.In this aspect, depending on whether horizontal shooting or vertical shooting is performed, vertical pixel mixing among the plurality of parallax images, Alternatively, pixel mixing in the horizontal direction is performed. For this reason, the pixel value of the defective pixel to be corrected can be calculated using only the pixels in the vertical direction around the defective pixel or only the pixels in the horizontal direction around the defective pixel. Thereby, the interpolation calculation can be simplified while maintaining the parallax information required for the defective pixel.

A stereoscopic imaging apparatus according to still another aspect of the present invention includes a detection unit that detects whether horizontal shooting or vertical shooting, and, when horizontal detection is detected by the detection unit, performs vertical pixel mixing among a plurality of parallax images. When the vertical shooting is detected, the image processing apparatus further includes a pixel mixing unit that mixes the pixels in the horizontal direction among the plurality of parallax images, and the defective pixel correction unit detects a defect when the horizontal shooting is detected by the detection unit. The pixel value of the adjacent pixel in the vertical direction of the pixel is set as the pixel value of the defective pixel to be corrected, and when vertical shooting is detected, the pixel value of the adjacent pixel in the horizontal direction of the defective pixel is set as the pixel value of the defective pixel to be corrected. It is preferable to do. Thereby, it is possible to prevent false resolution by applying the pixel value of the adjacent pixel (closest pixel) of the defective pixel while maintaining the parallax information required for the defective pixel.

A stereoscopic imaging apparatus according to still another aspect of the present invention includes a detection unit that detects whether horizontal shooting or vertical shooting, and when horizontal detection is detected by the detection unit, pixel addition in a vertical direction among a plurality of parallax images is performed. And a pixel addition unit that performs horizontal pixel addition among a plurality of parallax images when the vertical shooting is detected, and the defective pixel correction unit detects a defect when the horizontal shooting is detected by the detection unit. The pixel value of the defective pixel to be corrected is calculated by interpolating the pixel value of the pixel in the vertical direction including the pixel adjacent to the pixel in the vertical direction. When vertical shooting is detected, the pixel in the horizontal direction of the defective pixel is included. It is preferable to interpolate the pixel values of the pixels in the left-right direction to calculate the pixel value of the defective pixel to be corrected. Since the pixel value of the defective pixel is calculated by interpolating the pixel value of the pixel including the adjacent pixel of the defective pixel, false resolution is unlikely to occur, and the center of gravity of the pixel calculated by interpolation is determined as the defect to be corrected. It can be made to coincide with the position of the pixel position.

In the stereoscopic imaging apparatus according to still another aspect of the present invention, the defective pixel correction unit includes a determination unit that determines whether adjacent pixels in the vertical and horizontal directions of the defective pixel to be corrected are defective pixels. When horizontal shooting is detected and an adjacent pixel in the vertical direction is determined as a defective pixel, the pixel value of the defective pixel to be corrected is interpolated by interpolating the pixel values of the closest pixels in the upward and downward directions of the defective pixel to be corrected. When the pixel value is calculated and the vertical photographing is detected by the detection means and the adjacent pixel in the left-right direction is determined as the defective pixel, the pixel value of the pixel closest to the left and right of the defective pixel to be corrected is determined. It is preferable to calculate the pixel value of the defective pixel to be corrected by interpolation. That is, when the adjacent pixel closest to the defective pixel is a defective pixel (when defective pixels are continuous), in the case of horizontal shooting instead of the adjacent pixel, the upper and lower directions of the defective pixel to be corrected are the highest. The pixel value of the defective pixel is calculated by interpolating the pixel value of the near pixel, and in the case of vertical shooting, the defective pixel is interpolated by the pixel value of the pixel nearest to the left and right of the defective pixel to be corrected. The pixel value is calculated.

In the stereoscopic imaging device according to still another aspect of the present invention, the imaging element is a red (R), green (G), or blue (B) filter, and is adjacent to the R filter and the R filter. A first line in which one G filter is repeatedly arranged in the horizontal direction, and a second line in which a B filter and a second G filter adjacent to the G filter are repeatedly arranged in the horizontal direction. A color imaging device having a Bayer array color filter in which lines are alternately arranged in the vertical direction, and a first G pixel pixel provided with a first G filter around a defective pixel to be corrected A step detecting means for detecting a step between the value and the pixel value of the second G pixel provided with the second G filter, and the defective pixel correcting means has a predetermined step detected by the step detecting means. Having a discriminating means for discriminating whether or not the threshold value is below In the case where the defective pixel to be corrected is the first G pixel or the second G pixel, if the level difference is determined to be equal to or less than a predetermined threshold value, The pixel value of the defective pixel to be corrected is calculated by interpolating the pixel values of the second G pixel or the second G pixel. It is preferable to calculate the pixel value of the defective pixel to be corrected by interpolating the pixel values of the four first G pixels or the second G pixels in the direction.

In the case of the Bayer array, a first G pixel (hereinafter referred to as “ _Gr pixel”) provided with a first G filter adjacent in the left-right direction of the R pixel provided with the R filter, and B There is a second G pixel (hereinafter referred to as a “ _Gb pixel”) provided with a second G filter adjacent to the B pixel in which the filter is provided in the left-right direction. Thus, for example, the pixel value of G _r pixels by light leakage (color mixture) from R pixel becomes larger than the pixel value of G _b pixels adjacent, stepped between the G _r pixels and G _b pixel May occur. According to yet another aspect of the present invention, difference in level between the G _r pixels and G _b pixel, if less than a predetermined threshold value, the closest the same color to the defective pixel (G _r pixels or G _b pixels) upper right is a pixel, the four G _b pixels or G _r pixels in the oblique lower right direction using the interpolation operation, whereas, when exceeding a predetermined threshold value, pixels in the same positional relationship as the defective pixel (defective pixel if There is G _r pixels, G _r pixels around, if G _b pixels, to become near G _b pixels), four G _r pixels or G _b pixels in the vertical and horizontal directions of the defective pixel interpolation operation I am trying to use it.

A stereoscopic imaging apparatus according to still another aspect of the present invention includes a detection unit that detects whether horizontal shooting or vertical shooting, and, when horizontal detection is detected by the detection unit, performs vertical pixel mixing among a plurality of parallax images. When vertical shooting is detected, pixel mixing means for performing horizontal pixel mixing among a plurality of parallax images, and steps of output signals of a plurality of pixels in a block around a block to which a defective pixel to be corrected belongs A step detecting means for detecting the step, and the defective pixel correcting means has a determining means for determining whether or not the step detected by the step detecting means is equal to or less than a predetermined threshold value. When it is determined that the horizontal shooting is detected by the detection unit when the detection unit detects that the pixel is equal to or less than the threshold value, the pixel value of the adjacent pixel in the vertical direction of the defective pixel is set as the pixel value of the defective pixel to be corrected. Left and right adjacent to When the pixel value of the defective pixel to be corrected is set as the pixel value of the defective pixel to be corrected and the level difference exceeds the predetermined threshold by the determination unit, the peripheral pixels having the same positional relationship as the defective pixel to be corrected in the block It is preferable to calculate the pixel value of the defective pixel to be corrected by interpolating the pixel value.

The color filters of the same color are arranged for a plurality of pixels in one block, but the color combinations of the color filters of adjacent blocks differ depending on the position of the block. Therefore, the pixel values of a plurality of pixels in the block may be different due to light leakage (mixed color) from pixels of a specific color. Originally, the output of a plurality of pixels in a block differs depending on the parallax, but if there is a tendency in the level difference between the pixels in the surrounding blocks, it is considered that a step due to color mixing has occurred. Therefore, according to still another aspect of the present invention, when a step between a plurality of pixels in a peripheral block is determined to be equal to or less than a predetermined threshold, and in the case of horizontal shooting, adjacent pixels in the vertical direction of a defective pixel are detected. The pixel value is the pixel value of the defective pixel to be corrected, and in the case of vertical shooting, the pixel value of the adjacent pixel in the left-right direction of the defective pixel is the pixel value of the defective pixel to be corrected. On the other hand, if it is determined that the step exceeds a predetermined threshold value, the pixel value of the defective pixel to be corrected is calculated by interpolating the pixel values of the peripheral pixels having the same positional relationship as the defective pixel to be corrected in the block. Like to do.

The invention according to still another aspect of the present invention is any one of a single photographing optical system and a color filter including a plurality of pixels adjacent in the horizontal and vertical directions as one block and including three primary colors for each block. An image pickup device provided with a color filter of a color, which receives light that has passed through different regions of the photographing optical system and can obtain a plurality of horizontal and vertical parallax images simultaneously. In the image processing method of a stereoscopic imaging apparatus, a process of acquiring a plurality of parallax images output from an imaging device, and interpolating pixel values of defective pixels in the plurality of parallax images with pixel values of peripheral pixels of the defective pixels Using the peripheral pixels having the same parallax information and the same color information as the defective pixels of the plurality of parallax images as the peripheral pixels used for interpolation of the defective pixels to be corrected, and Including There.

An image processing method according to still another aspect of the present invention includes a step of detecting whether the photographing by the stereoscopic imaging device is horizontal photographing or vertical photographing, and when horizontal photographing is detected, pixels in a vertical direction among a plurality of parallax images. And a step of mixing pixels in a horizontal direction among a plurality of parallax images when vertical shooting is detected, and the step of correcting a defective pixel includes detecting defective pixels when horizontal shooting is detected. The pixel value of the defective pixel to be corrected is calculated by interpolating the pixel value of the vertical pixel around the pixel, and when vertical shooting is detected, the pixel value of the pixel in the horizontal direction around the defective pixel is interpolated. It is preferable to calculate the pixel value of the defective pixel to be corrected.

An image processing method according to still another aspect of the present invention includes a step of detecting whether the photographing by the stereoscopic imaging device is horizontal photographing or vertical photographing, and when horizontal photographing is detected, pixels in a vertical direction among a plurality of parallax images. And a step of mixing pixels in a horizontal direction among a plurality of parallax images when vertical shooting is detected, and the step of correcting a defective pixel includes detecting defective pixels when horizontal shooting is detected. The pixel value of the adjacent pixel in the vertical direction is set as the pixel value of the defective pixel to be corrected, and when vertical shooting is detected, the pixel value of the adjacent pixel in the horizontal direction of the defective pixel is set as the pixel value of the defective pixel to be corrected. It is preferable.

In the image processing method according to still another aspect of the present invention, the step of correcting the defective pixel includes a step of determining whether adjacent pixels in the vertical or horizontal direction of the defective pixel to be corrected are defective pixels, Is detected, and the adjacent pixel in the vertical direction is determined as a defective pixel, the pixel value of the defective pixel to be corrected is interpolated by interpolating the pixel values of the pixels closest to each other in the upward and downward directions of the defective pixel to be corrected. When vertical shooting is detected and the adjacent pixel in the left-right direction is determined to be a defective pixel, the pixel value of the pixel closest to the correction target defective pixel in the left direction and the right direction is interpolated. It is preferable to calculate the pixel value of the defective pixel.

According to the present invention, when a plurality of horizontal and vertical parallax images are simultaneously photographed through a single photographing optical system and a single image sensor, they are used for correcting defective pixels in these images. Since the peripheral pixels having the same parallax information as the defective pixels in the plurality of parallax images are used as the peripheral pixels, the stereoscopic effect can be prevented from being impaired by the corrected defective pixels.

The top view which shows embodiment of the image pick-up element based on this invention 1 is an enlarged view of a main part showing an embodiment of an image sensor according to the present invention. The principal part side view which shows embodiment of the image pick-up element based on this invention 1 is a block diagram showing an embodiment of a stereoscopic imaging apparatus according to the present invention. The top view used in order to explain the pixel of 4 pixel 1 micro lens and the addition method of the pixel The figure used in order to explain the pixel of 4 pixel 1 micro lens and the addition method of the pixel The figure used in order to demonstrate 1st Embodiment of the image processing method with respect to a defective pixel The figure used in order to demonstrate 1st Embodiment of the image processing method with respect to a defective pixel The figure used in order to demonstrate 2nd Embodiment of the image processing method with respect to a defective pixel The figure used in order to demonstrate 2nd Embodiment of the image processing method with respect to a defective pixel The figure used in order to demonstrate 3rd Embodiment of the image processing method with respect to a defective pixel The figure used to explain the fourth embodiment of the image processing method for defective pixels (horizontal photography) The figure used to explain the fourth embodiment of the image processing method for defective pixels (vertical shooting) The figure used to explain the fifth embodiment of the image processing method for defective pixels (horizontal photography) The figure used to explain the fifth embodiment of the image processing method for defective pixels (vertical shooting) The figure used to explain the sixth embodiment of the image processing method for defective pixels (horizontal photography) The figure used to explain the sixth embodiment of the image processing method for defective pixels (vertical shooting) The figure used for describing 7th Embodiment of the image processing method with respect to a defective pixel The figure used for describing 7th Embodiment of the image processing method with respect to a defective pixel Diagram used to explain a step between the G _r pixels and G _b pixels in bayer array The flowchart which shows 8th Embodiment of the image processing method with respect to a defective pixel. The figure used for describing 8th Embodiment of the image processing method with respect to a defective pixel The figure used in order to explain the level difference of four pixels in a four-pixel one microlens The flowchart which shows 9th Embodiment of the image processing method with respect to a defective pixel. The figure used to explain the ninth embodiment of the image processing method for defective pixels The figure used to explain the ninth embodiment of the image processing method for defective pixels

Hereinafter, embodiments of a stereoscopic imaging device and an image processing method according to the present invention will be described with reference to the accompanying drawings.

[Image sensor]
FIG. 1A is a plan view showing an embodiment of an image sensor according to the present invention. 1B is an enlarged view of the main part of FIG. 1A, and FIG. 1C is a side view of the main part of FIG. 1B.

As shown in FIGS. 1A to 1C, the image pickup device 1 is a color image sensor of CCD (Charge CoupledupDevice) or CMOS (Complementary Metal Oxide Semiconductor), and is arranged in a row direction and a column direction on a semiconductor substrate. A large number of photoelectric conversion elements (photodiodes) PD (FIG. 1C), a microlens L, and color filters of a plurality of colors (three primary colors) of red (R), green (G), and blue (B) are provided. .

In FIG. 1A, one microlens L is arranged above four photodiodes PD (four pixel ABCD) in the upper left, lower left, upper right, and lower right, and has a configuration of a four pixel one microlens. In this embodiment, if the stereoscopic image pickup apparatus is set up so as to perform normal horizontal shooting and the plan view of the image pickup element 1 when viewed from the subject side is FIG. The upper left and lower left pixels and the upper right and lower right pixels in the lens are called X-direction (left and right direction) pixels, and the upper left and upper right pixels and the lower left and lower right pixels are Y direction (vertical direction) pixels. That's it. Further, “upward” of the four-pixel ABCD means a direction in which light is incident on each pixel (imaging device 1).

4 pixels 1 micro lens is made into 1 block, and the color filter of any one color of R, G, and B is arrange | positioned for every block. That is, color filters of the same color are disposed on the four photodiodes PD of the four-pixel one-microlens.

As shown in FIG. 1A, color filters are arranged in order of RG _r RG _r on the four pixel 1 microlenses on the odd lines l1, l3, l5,..., And even lines l2, l4, l6. The upper four-pixel 1 microlens is provided with color filters in the order of G _b BG _b B.

That is, when a 4-pixel 1-microlens (1 block) is regarded as 1 pixel, the color filter array is a Bayer array, and an image signal having different color information for each block is obtained.

As shown in FIG. 1B, the microlens L of the 4-pixel 1-microlens condenses the light flux on the light receiving surfaces of the four corners (four) of photodiodes PD. The upper left, lower left, upper right, lower right In each of the four directions, light with limited light flux (light that has been divided into pupils) is incident on four photodiodes PD.

That is, in FIG. 1C, the microlens L causes a light beam incident on the left side of the optical axis of the microlens L to enter the right photodiode PD, and causes a light beam incident on the right side to enter the left photodiode PD. .

According to this imaging device 1, four parallax images (3D images) having parallax in the horizontal and vertical directions (X direction and Y direction) can be generated based on the output signal output from the four-pixel one-microlens. . In addition, when four parallax images are added, a plane image (2D image) can also be generated.

[Stereoscopic imaging device]
FIG. 2 is a block diagram showing an embodiment of a stereoscopic imaging apparatus according to the present invention.

The stereoscopic imaging apparatus 10 is provided with the imaging element 1 shown in FIGS. 1A to 1C and can capture 2D images and 3D images. The operation of the entire apparatus is performed by a central processing unit (CPU) 40. Overall control.

The stereoscopic imaging device 10 is provided with operation units 38 such as a shutter button, a mode dial, a playback button, a MENU / OK key, a cross key, and a BACK key. A signal from the operation unit 38 is input to the CPU 40, and the CPU 40 controls each circuit of the stereoscopic imaging device 10 based on the input signal. For example, lens driving control, aperture driving control, photographing operation control, image processing control, image processing Data recording / reproduction control, display control of the liquid crystal monitor 30 for stereoscopic display, and the like are performed.

The shutter button is an operation button for inputting an instruction to start shooting, and is configured by a two-stroke switch having an S1 switch that is turned on when half-pressed and an S2 switch that is turned on when fully pressed. The mode dial is selection means for selecting a 2D shooting mode, a 3D shooting mode, a scene position such as a person, a landscape, a night view, a moving image mode, and the like.

The playback button is a button for switching to a playback mode in which a still image or a moving image of a plurality of parallax images (3D images) and planar images (2D images) that have been photographed and recorded are displayed on the liquid crystal monitor 30. The MENU / OK key is an operation key having both a function as a menu button for instructing to display a menu on the screen of the liquid crystal monitor 30 and a function as an OK button for instructing confirmation and execution of the selection contents. It is. The cross key is an operation unit for inputting instructions in four directions, up, down, left, and right, and functions as a button (cursor moving operation means) for selecting an item from the menu screen or instructing selection of various setting items from each menu. To do. The up / down key of the cross key functions as a zoom switch for shooting or a playback zoom switch in playback mode, and the left / right key functions as a frame advance (forward / reverse feed) button in playback mode. . The BACK key is used to delete a desired object such as a selection item, cancel an instruction content, or return to the previous operation state.

In the photographing mode, the image light indicating the subject is imaged on the light receiving surface of the image sensor 1 through the single photographing optical system (zoom lens) 12 and the diaphragm 14. The photographing optical system 12 is driven by a lens driving unit 36 controlled by the CPU 40, and performs focus control, zoom control, and the like. The diaphragm 14 is composed of, for example, five diaphragm blades, and is driven by a diaphragm drive unit 34 controlled by the CPU 40. For example, the diaphragm value is controlled in seven steps from 1 to an aperture value from a diaphragm value F1.4 to F11.

The CPU 40 controls the aperture 14 via the aperture drive unit 34, and controls the charge accumulation time (shutter speed) in the image sensor 1 and image signal readout from the image sensor 1 via the device controller 32. Etc.

The signal charge accumulated in the image sensor 1 is read out as a voltage signal corresponding to the signal charge based on a read signal applied from the device control unit 32. The voltage signal read from the image sensor 1 is applied to the analog signal processing unit 18 where the R, G, B signals for each pixel are sampled and held, and the gain designated by the CPU 40 (corresponding to ISO sensitivity). And then added to the A / D converter 20. The A / D converter 20 converts R, G, and B signals that are sequentially input into digital R, G, and B signals and outputs them to the image input controller 22.

The digital signal processing unit 24 performs gain control processing including gamma correction processing, offset processing, white balance correction, sensitivity correction, gamma correction processing, and synchronization processing (primary color filter) on a digital image signal input via the image input controller 22. Predetermined signal processing such as interpolating the spatial deviation of the color signals due to the arrangement of color signals and converting the color signals into simultaneous equations), YC processing (processing for generating luminance data and color difference data of image data), sharpness correction, etc. I do.

In FIG. 2, 46 is a ROM (EEPROM (Electrically Erasable Programmable Read-Only Memory)) in which various parameters and tables used for camera control programs, defect information of the image sensor 1, image processing, and the like are stored. .

The defect information is a correction indicating the positional relationship of one or a plurality of peripheral pixels of the same color used for correcting the defective pixel based on the coordinate data (X coordinate data, Y coordinate data) of the defective pixel and the color of the defective pixel. Pattern. The coordinate data of the defective pixel is created for each defective image determined to be a defective pixel by evaluating the output data of the imaging device 1 on a pixel-by-pixel basis before shipping the image sensor 1 from the factory, Recorded in the ROM 46. Note that a defective pixel that occurs later may be detected, and coordinate data of the defective pixel may be registered.

In addition, the digital signal processing unit 24 includes a defective pixel correction unit, and corrects defective pixels based on defect information stored in the ROM 46. The details of the defective pixel correction in the digital signal processing unit 24 will be described later.

The image data processed by the digital signal processing unit 24 is output to a VRAM (Video Random Access Memory) 50. The VRAM 50 includes an A area and a B area each storing image data representing an image for one frame. In the VRAM 50, image data representing an image for one frame is rewritten alternately in the A area and the B area. Of the A area and B area of the VRAM 50, the written image data is read from an area other than the area where the image data is rewritten. The image data read from the VRAM 50 is encoded by the video encoder 28 and is output to a stereoscopic display liquid crystal monitor 30 provided on the back of the camera, whereby a 2D / 3D subject image (live view image) is liquid crystal. It is displayed on the display screen of the monitor (LCD) 30.

The liquid crystal monitor 30 is a stereoscopic display unit that can display a stereoscopic image (left and right parallax images) as a directional image having a predetermined directivity by a parallax barrier, but is not limited thereto, and uses a lenticular lens. It is also possible that the left viewpoint image and the right viewpoint image can be individually viewed by wearing dedicated glasses such as polarized glasses or liquid crystal shutter glasses.

Further, when the shutter button of the operation unit 38 is first pressed (half-pressed), the CPU 40 starts an AF (Automatic Focus) operation and an AE (Automatic Exposure adjustment) operation, Control is performed so that the focus lens in the photographing optical system 12 comes to the in-focus position. The image data output from the A / D converter 20 when the shutter button is half-pressed is taken into the AE detection unit 44.

The AE detection unit 44 integrates the G signals of the entire screen or integrates the G signals that are weighted differently in the central portion and the peripheral portion of the screen, and outputs the integrated value to the CPU 40. The CPU 40 calculates the brightness of the subject (shooting EV value) from the integrated value input from the AE detection unit 44, and sets the aperture value of the diaphragm 14 and the electronic shutter (shutter speed) of the image sensor 1 based on the shooting EV value. It is determined according to a predetermined program diagram.

Here, in the program diagram, shooting (exposure) conditions that are a combination of aperture value and shutter speed, or a combination of these and shooting sensitivity (ISO sensitivity) are designed according to the brightness of the subject. Is. By shooting under the shooting conditions determined according to the program diagram, an image with appropriate brightness can be taken regardless of the brightness of the subject.

The CPU 40 controls the aperture 14 via the aperture drive unit 34 based on the aperture value determined according to the program diagram, and accumulates charges in the image sensor 1 via the device control unit 32 based on the determined shutter speed. Control the time.

The AF processing unit 42 is a part that performs AF processing (for example, contrast AF processing, phase AF processing). When contrast AF processing is performed, for example, high-frequency components of image data in a predetermined focus area of image data are extracted, and an AF evaluation value indicating an in-focus state is calculated by integrating the high-frequency components. . AF control is performed by controlling the focus lens in the photographic optical system 12 so that the AF evaluation value is maximized. Further, when performing phase difference AF processing, a phase difference of image data in a predetermined focus area among a plurality of parallax image data corresponding to a 4-pixel 1 microlens is detected, and information indicating this phase difference is obtained. The defocus amount is obtained based on this. AF control is performed by controlling the focus lens in the photographing optical system 12 so that the defocus amount becomes zero.

When the AE operation and the AF operation are completed and the shutter button is pressed in the second stage (full press), the image data output from the A / D converter 20 in response to the press is stored in the memory from the image input controller 22. (SDRAM (Synchronous Dynamic Random Access Memory)) 48 is input and temporarily stored.

The image data temporarily stored in the memory 48 is appropriately read out by the digital signal processing unit 24. The digital signal processing unit 24 performs predetermined signal processing such as defective pixel correction, white balance correction, gamma correction processing, synchronization processing, YC processing, and sharpness correction on the read digital image signal, and performs YC processing. The YC data is stored in the memory 48 again.

As shown in FIG. 3A, assuming that each pixel of the 4-pixel 1-microlens is A, B, C, and D, the digital signal processing unit 24 obtains four sheets of image data for each of A, B, C, and D. Generate. Next, when the stereoscopic imaging device 10 is taken horizontally (when shooting horizontally), the image data of A and C are added to generate a left eye display image (left parallax image), and B and The image data for D is added to generate a right-eye display image (right parallax image).

On the other hand, when the stereoscopic imaging device 10 is taken vertically (when shooting vertically), the image data for A and B are added to generate a left eye display image (left parallax image), and C and D The image data is added to generate a right eye display image (right parallax image).

The stereoscopic imaging device 10 is provided with a vertical / horizontal detection unit 49 that detects the orientation (vertical / horizontal) of the stereoscopic imaging device 10 and selectively adds the pixels based on the orientation of the stereoscopic imaging device 10 during 3D shooting. To do. A 2D image can also be generated by adding image data of A, B, C, and D.

The vertical / horizontal detection unit 49 can be constituted by a gravity sensor or a gyro sensor. In addition, there is a recording mode in which image data for four sheets A, B, C, and D is recorded as it is without performing pixel addition regardless of whether the image is taken horizontally or vertically.

The YC data of the plurality of parallax images stored in the memory 48 is output to the compression / decompression processing unit 26, where predetermined compression processing such as JPEG (joint photographic experts group) is executed, and further, a multi-picture file (MP File: a file in a format in which a plurality of images are connected) is generated, and the MP file is recorded on the memory card 54 via the media controller 52. Note that one piece of YC data generated in the 2D shooting mode and stored in the memory 48 is output to the compression / decompression processing unit 26, where a predetermined compression process such as JPEG is executed, and then Exif (Exchangeable image file format) file is generated and recorded on the memory card 54 via the media controller 52.

[Correction of defective pixels]
Next, the defective pixel correction processing in the defective pixel correction unit of the digital signal processing unit 24 will be described.

Image data (mosaic image data) of RG _r G _b B pixels corresponding to the pixels A, B, C, D of the four-pixel 1 microlens of the image sensor 1 shown in FIG. 1A and the color filter array (Bayer array) ) Is temporarily stored in the memory 48 via the image input controller 22.

The defective pixel correction unit identifies the defective pixel based on the coordinate data of the defective pixel in the defect information stored in the ROM 46. Further, the defective pixel correcting unit determines whether the color of the specified defective pixel is RG _r G _b B, or the specified defective pixel is each pixel A, B, C, Which pixel of D is identified (that is, which of the four parallax images belongs to).

Also, the defect information includes the defect pixel color type (R (B), G _r, G _b ) and the position (pixel A, B, C, D) within the 4-pixel 1 microlens. A correction pattern indicating the positional relationship of one or a plurality of peripheral pixels used for correcting a defective pixel with respect to the defective pixel is included. The defective pixel correction unit reads a correction pattern corresponding to the color of the defective pixel to be corrected and the types of the pixels A, B, C, and D.

The defective pixel correction unit identifies peripheral pixels used for correcting the defective pixel based on the coordinate data of the defective pixel to be corrected and the read correction pattern. The defective pixel correction unit calculates a pixel value of the defective pixel by interpolating an image value that is an image signal of the specified peripheral pixel.

<First Embodiment>
4A and 4B are diagrams used to explain the first embodiment of the image processing method for a defective pixel, and more particularly, a diagram showing a relationship between the defective pixel and peripheral pixels used for correcting the defective pixel. is there.

Now, as shown in FIG. 4A, when a pixel indicated by diagonal lines (R pixel A) is a defective pixel, four peripheral pixels (white background) of the same color closest to the defective pixel and the pixel A in the 4-pixel 1 microlens. The pixel A) indicated by is used as a pixel for interpolation.

In the case of the Bayer array, the R pixel closest to the R defective pixel exists at an equidistant position in the vertical and horizontal directions of the defective pixel. Therefore, the pixel value at the position of the defective pixel can be calculated by arithmetically averaging the pixel values of the four peripheral pixels.

When the defective pixel is the pixel A as described above, the pixel A of the same color as the peripheral pixels (the pixel A having the same phase among the pixels A, B, C, and D having different phases in the 4-pixel 1 microlens) is selected. By using the correction, the stereoscopic effect is prevented from being impaired by the corrected defective pixel.

Similarly, when the defective pixel is the pixel B, C, or D, the peripheral pixel used for correction also uses the pixel B, C, or D. In the Bayer array, since the array of R pixels and the array of B pixels have the same array, even when the B pixel is a defective pixel, correction can be performed in the same manner as described above.

Also, when a pixel indicated by hatching (pixel A G _b) is a defective pixel as shown in FIG. 4B, the nearest same color to the defective pixel, and four peripheral pixels of the same phase (pixel A shown in white) Use pixels for interpolation.

If the Bayer array, the closest G pixel to the defective pixel of the G _b is upper right of the defective pixel is present in the same distance in the lower right diagonal direction. Therefore, the pixel value at the position of the defective pixel can be calculated by arithmetically averaging the pixel values of the four peripheral pixels.

Note that the G _b pixels and G _r pixels, a pixel adjacent to the left-right direction, but there is a difference in the B pixel or an R pixel, the same G pixel, when the Bayer array, closest to G _b pixel Since the G pixel is G _r and the G pixel closest to the G _r pixel is the G _b pixel, these diagonal G pixels are used. Further, by using the G pixel having the same phase for correcting the defective G pixel, the stereoscopic effect is not impaired by the corrected defective G pixel.

<Second Embodiment>
FIG. 5A and FIG. 5B are diagrams used to explain the second embodiment of the image processing method for a defective pixel, and more particularly, a diagram showing the relationship between the defective pixel and peripheral pixels used for correcting the defective pixel. is there.

The second embodiment shown in FIGS. 5A and 5B is different from the first embodiment shown in FIGS. 4A and 4B in that defective pixels are included in peripheral pixels used for correcting defective pixels. Is different.

Whether or not the peripheral pixel used for correcting the defective pixel includes a defective pixel is determined by specifying the peripheral pixel used for correcting the defective pixel and then registering the coordinate data of the specified peripheral pixel in the ROM 46 This can be done depending on whether or not the coordinate data of the defective pixel is matched.

Of the four peripheral pixels above, below, left, and right of the defective pixel indicated by the diagonal lines in FIG. 5A, if the upward pixel (the pixel indicated by “x”) in FIG. 5A is a defective pixel, two peripheral pixels in the horizontal direction Are used for interpolation. In this case, the pixel in the lower direction that is symmetrical to the pixel in the upper direction across the defective pixel to be corrected is not a defective pixel, but is not used for interpolation.

Thereby, when the pixel value of the defective pixel is calculated by the interpolation calculation, the position (center of gravity) of the pixel calculated by interpolation can be matched with the position of the defective pixel to be corrected.

Further, upper right pixel (pixel A G _b) indicated by oblique lines in FIG. 5B, among the four pixels in the lower right diagonal direction (pixels indicated by × marks) the upper right direction of the pixel on Figure 5B is a defective pixel In this case, two peripheral pixels in the diagonally lower right direction are used as pixels for interpolation. In this case, the pixel in the lower left direction that is symmetrical to the pixel in the upper right direction across the defective pixel to be corrected is not a defective pixel, but is not used for interpolation.

Thereby, when the pixel value of the defective pixel is calculated by the interpolation calculation, the position (center of gravity) of the pixel calculated by interpolation can be matched with the position of the defective pixel to be corrected.

<Third Embodiment>
FIG. 6 is a diagram used for explaining the third embodiment of the image processing method for a defective pixel, and particularly shows a relationship between the defective pixel and peripheral pixels used for correcting the defective pixel.

The third embodiment shown in FIG. 6 is different from the second embodiment shown in FIG. 5B in that peripheral pixels used for correcting defective pixels are different.

That is, when the pixel indicated by hatching in FIG. 6 (pixel A of G _b) is a defective pixel, four peripheral pixels of the closest same color and the same phase to the defective pixel is present upper right, obliquely lower right direction . Among these four peripheral pixels, for example, when the pixel in the upper right direction in FIG. 6 (the pixel indicated by a cross) is a defective pixel, these four peripheral pixels in the oblique direction are used for correcting the defective pixel. Instead, the four pixels having the same color and the same phase in the vertical and horizontal directions of the defective pixel are set as peripheral pixels used for correcting the defective pixel.

These four peripheral pixels are peripheral pixels that are not closest to the G defective pixel in the same color and phase, but are present in the same color and the same phase at the same distance.

Therefore, the pixel value at the position of the defective pixel can be calculated by arithmetically averaging the pixel values of the four surrounding pixels. Note that since the number of pixels used for interpolation is larger than in the embodiment shown in FIG. 5B, the influence of false resolution can be reduced.

<Fourth Embodiment>
FIG. 7A and FIG. 7B are diagrams used to explain the fourth embodiment of the image processing method for a defective pixel, and in particular, are diagrams showing the relationship between the defective pixel and the peripheral pixels used for correcting the defective pixel. is there.

As shown in FIG. 3B, during horizontal shooting, the image data of the pixels A and C of the 4-pixel 1-microlens are added to generate a left parallax image, and the image data of the pixels B and D are added to generate a right parallax image. In the case of generating the left parallax image and the right parallax image, the parallax information in the vertical direction is canceled and the parallax image is only in the horizontal direction.

Therefore, when the pixels A and C and the pixels B and D in the vertical direction within the 4-pixel 1 microlens are added (pixel mixture) during horizontal shooting, for example, when the pixel indicated by the oblique lines in FIG. 7A is a defective pixel. Are pixels having the same color and the same phase closest to the defective pixel, and two peripheral pixels in the vertical direction are used as pixels for interpolation.

The parallax image to which pixels are added during horizontal shooting tends to have parallax in the horizontal direction and no parallax in the vertical direction. For this reason, it is possible to prevent deterioration of the parallax information of the corrected pixel by performing correction using the pixels in the vertical direction.

In addition, when the pixels A and B and the pixels C and D in the left and right direction in the 4-pixel 1 microlens are added (pixel mixture) during vertical shooting, for example, when the pixel indicated by the oblique line in FIG. 7B is a defective pixel Are pixels having the same color and the same phase closest to the defective pixel, and two peripheral pixels in the left-right direction are used for interpolation.

The parallax image to which pixels are added during vertical shooting tends to have parallax in the vertical direction and no parallax in the horizontal direction. For this reason, it is possible to prevent deterioration of the parallax information of the corrected pixel by performing correction using the left and right pixels.

The “lateral” parallax refers to the parallax in the left-right direction (pixel AB or pixel CD direction) on FIG. 1A, and the “vertical” parallax refers to the vertical direction (pixel) on FIG. 1A. The parallax in the direction of AC or pixel BD).

<Fifth Embodiment>
FIGS. 8A and 8B are diagrams used to explain the fifth embodiment of the image processing method for defective pixels, and particularly show the relationship between the defective pixels and the peripheral pixels used to correct the defective pixels. is there.

The fifth embodiment shown in FIGS. 8A and 8B relates to a defective pixel correction method in the case where pixels are added according to horizontal shooting or vertical shooting as in the fourth embodiment shown in FIGS. 7A and 7B. .

At the time of horizontal photographing, when the pixel indicated by the oblique line in FIG. 8A is a defective pixel, the pixel adjacent to the vertical direction in the four-pixel 1 microlens including the defective pixel is set as the pixel value of the defective pixel to be corrected. To do.

The four pixels in the four-pixel 1 microlens are pixels of the same color, although the phases (parallax information) are different, and it is only necessary to obtain a parallax image in the horizontal direction during horizontal shooting. For this reason, as the parallax information in the horizontal direction, the pixel value of the adjacent pixel in the same vertical direction is directly used as the pixel value of the defective pixel. Further, since the adjacent pixel is closest to the defective pixel, false resolution due to the corrected pixel is less likely to occur.

Similarly, when the pixel indicated by the oblique line in FIG. 8B is a defective pixel at the time of vertical photographing, a pixel adjacent to the vertical direction among the four-pixel 1 microlens including the defective pixel is determined as a defective pixel to be corrected. Value.

The four pixels in the four-pixel 1 microlens are pixels of the same color, although the phases (parallax information) are different, and it is only necessary to obtain a parallax image in the horizontal direction during horizontal shooting. For this reason, as the parallax information in the horizontal direction, the pixel value of the adjacent pixel in the same vertical direction is directly used as the pixel value of the defective pixel. Further, since the adjacent pixel is closest to the defective pixel, false resolution due to the corrected pixel is less likely to occur.

<Sixth Embodiment>
9A and 9B are diagrams illustrating a sixth embodiment of an image processing method for a defective pixel, and particularly a diagram illustrating a relationship between a defective pixel and peripheral pixels used for correcting the defective pixel.

The sixth embodiment shown in FIGS. 9A and 9B is a modification of the fifth embodiment shown in FIGS. 8A and 8B. That is, as shown in FIG. 9A, during horizontal shooting, the pixels adjacent to each other in the vertical direction among the four-pixel 1 microlens, the adjacent pixels, and the closest pixel of the same color on the opposite side of the adjacent pixel across the defective pixel. The pixels used for correcting the defective pixels are calculated, and the pixel values of these pixels are calculated by interpolating the pixel values of these pixels.

At the time of interpolation calculation, by taking a weighted average using a weighting coefficient corresponding to the distance from the defective pixel, the center of gravity of the pixel calculated by interpolation can be made to coincide with the position of the defective pixel to be corrected. Further, since adjacent pixels are used for interpolation, it is possible to make it difficult for false resolution to occur.

Similarly, as shown in FIG. 9B, during vertical shooting, adjacent pixels adjacent in the horizontal direction among the four-pixel 1 microlens and the closest pixel of the same color on the opposite side of the adjacent pixel across the defective pixel are defective. The pixels used for correcting the pixels are calculated, and the pixel values of these pixels are calculated by interpolating the pixel values of these pixels.

<Seventh Embodiment>
10A and 10B are diagrams illustrating a seventh embodiment of an image processing method for a defective pixel, and particularly a diagram illustrating a relationship between the defective pixel and a peripheral pixel used for correcting the defective pixel.

The seventh embodiment shown in FIGS. 10A and 10B is a modification of the sixth embodiment shown in FIGS. 9A and 9B. In the case where the adjacent pixels adjacent to the defective pixel to be corrected are also defective pixels. It shows the processing method.

In this case, as shown in FIG. 10A, in the horizontal shooting, the opposite side of the defective pixel to be corrected is sandwiched in place of the defective pixel adjacent to the vertical direction in the four-pixel 1 microlens (the pixel indicated by x). The closest pixel of the same color is used as a pixel used for correcting a defective pixel.

Similarly, as shown in FIG. 10B, during vertical shooting, the opposite side of the defective pixel to be corrected is sandwiched in place of the defective pixel adjacent to the horizontal direction in the four-pixel 1 microlens (the pixel indicated by a cross). The closest pixel of the same color is used as a pixel used for correcting a defective pixel.

<Eighth Embodiment>
For color filter array is a Bayer array as described above, the G _r pixels and G _b pixel, is a pixel of the same color (G), a pixel adjacent to the left-right direction, a difference of B pixel and R pixel is there. As a result, greater than the pixel value of G _b pixel a pixel value of G _r pixels by leakage of light from the R pixel (color mixing) are adjacent the shooting, and G _r pixels as shown in FIG. 11 G There may be a step between the _b pixel.

FIG. 12 is a flowchart showing an eighth embodiment of an image processing method for defective pixels.

12, the defective pixel to be corrected (flaw pixels), in the case of G _r pixels or G _b pixels, calculates a difference (level difference) of pixel values of the G _r pixels and G _b pixels around the defective pixels ( Step S10). Note that it is preferable to detect a tendency of the step by calculating a plurality of steps around the defect pixel to be corrected and calculating an average value thereof.

Subsequently, it is determined whether or not the calculated level difference is equal to or less than a predetermined threshold (step S12). If the step is equal to or less than a predetermined threshold (in the case of “Yes”), as shown by the solid line arrow in FIG. 13, the periphery of the defective pixel that is the same color and the same phase pixel as the defective pixel upper right, and the pixels to be used for correcting the (G _b is G _r in the case of defective pixel G _b, defective pixel in the case of G _r) 4 single pixel in the lower right diagonal direction, the pixels of the pixel of The pixel value of the defective pixel is calculated by interpolating the value (step S14).

On the other hand, when the step exceeds a predetermined threshold value (in the case of “No”), as shown by the dotted arrow in FIG. 13, the pixel is the second closest to the defective pixel and has the same phase. and pixels to be used for correcting the (G _r is the case when the defective pixel is G _b of G _b, defective pixel G _r) 4 single pixel in the vertical and horizontal directions around the defective pixel, the pixels of the pixel The pixel value of the defective pixel is calculated by interpolating the value (step S16).

If the step of the pixel values of the G _r pixels and G _b pixel is small, but may use any of the pixels, to reduce the spurious resolution by using a diagonal direction of the pixel closer to the defective pixel it can, on the other hand, if the step of the pixel values of the G _r pixels and G _b pixel is large, which enables the calculation of receiving no pixel values the influence of color mixture by using a pixel in the vertical and horizontal directions .

<Ninth Embodiment>
The four pixels A, B, C, and D in the four-pixel 1 microlens are different in adjacent color (A is different in the upper and left colors, B is different in the upper and right colors, etc.), and the color mixture is large Is affected by adjacent pixels, and steps are generated in the pixel values of the pixels A, B, C, and D as shown in FIG.

The four pixels within the four-pixel 1 microlens have different phases (parallax information) and therefore output levels are different (in the case of a focused pixel, there is no phase shift, so the output levels are equal. However, if the surrounding pixels also tend to have a level difference, it is considered that a step due to color mixing has occurred.

FIG. 15 is a flowchart showing a ninth embodiment of the image processing method for defective pixels.

In FIG. 15, the level difference of the pixel values of the pixels A, B, C, and D in the same color 4 pixel 1 microlens around the defective pixel (scratch pixel) to be corrected is calculated (step S20). In addition, it is preferable to detect the tendency of the level difference by calculating the level difference between the pixels A, B, C, and D in the plurality of 4-pixel 1 microlenses around the defect pixel to be corrected and calculating the average value thereof.

Subsequently, it is determined whether or not the calculated steps of the pixels A, B, C, and D within the 4-pixel 1 microlens are equal to or less than a predetermined threshold value (step S22). When each step is equal to or smaller than a predetermined threshold (when “Yes”), as shown in FIG. 16A, the pixel value of the adjacent pixel in the vertical direction of the defective pixel is used as the pixel value of the defective pixel to be corrected. At the time of vertical shooting, the pixel value of the adjacent pixel in the left-right direction of the defective pixel is set as the pixel value of the defective pixel to be corrected (step S24). For example, when the defective pixel is the pixel A at the time of horizontal shooting, the pixel value of the pixel C in the same 4-pixel 1 microlens is used, and when the defective pixel is the pixel A at the time of vertical shooting, The pixel value of the pixel B of

On the other hand, when each step exceeds a predetermined threshold value (in the case of “No”), during horizontal shooting, pixel values of pixels having the same color and the same phase in the vertical direction of the defective pixel are interpolated as shown in FIG. 16B. The pixel value of the defective pixel to be corrected is calculated, and during vertical shooting, the pixel value of the defective pixel to be corrected is calculated by interpolating the pixel values of the same color and phase of the defective pixel in the left-right direction (step S26). For example, when the defective pixel is an R pixel A during horizontal shooting, the pixel A of the vertical R pixel is used as a correction pixel, and when the defective pixel is an R pixel A during vertical shooting, the horizontal direction The pixel A of the R pixel is a pixel used for correction.

When the level difference between the pixel values of the pixels A, B, C, and D in the 4-pixel 1 microlens is small, it is possible to reduce false resolution by using adjacent pixels in the same 4-pixel 1 microlens. On the other hand, when the level difference is large, pixel values that are not affected by the level difference can be calculated by using pixels having the same color and the same phase in the vertical and horizontal directions.

[Others]
In this embodiment, the case where the color filter array is a Bayer array has been described. However, the present invention is not limited to this, and the present invention can also be applied to correction of defective pixels of image pickup devices of other color filter arrays.

In addition, the imaging device of this embodiment can simultaneously acquire a plurality of (four) parallax images in the horizontal and vertical directions by a four-pixel one microlens. For example, four or more such as nine-pixel one microlens The present invention can also be applied to an image sensor that can simultaneously acquire parallax images.

Furthermore, the pupil dividing means may be a combination of 1 pixel 1 microlens and a light blocking member for limiting the light flux.

Further, it goes without saying that the present invention is not limited to the embodiment described above, and various modifications can be made without departing from the spirit of the present invention.

DESCRIPTION OF SYMBOLS 1 ... Imaging device, 10 ... Stereo imaging device, 12 ... Imaging optical system, 14 ... Diaphragm, 24 ... Digital signal processing part, 30 ... Liquid crystal monitor, 38 ... Operation part, 40 ... Central processing unit (CPU), 46 ... ROM 48 ... Memory 49 ... Vertical / horizontal detection unit 54 ... Memory card L ... Microlens PD PD Photodiode

Claims (14)

A single photographic optical system,
An imaging device in which a plurality of pixels adjacent in the horizontal and vertical directions are made into one block, and a color filter of any one of the color filters including three primary colors is arranged for each block, and the imaging optical system includes: A single image sensor that receives light that has passed through different areas and can simultaneously acquire a plurality of horizontal and vertical parallax images;
A defective pixel correction unit that calculates a pixel value of a defective pixel in an image acquired from the image sensor by interpolating a pixel value of a peripheral pixel of the defective pixel; and
The defective pixel correction unit uses a peripheral pixel having the same parallax information and the same color information as the defective pixel of the plurality of parallax images as a peripheral pixel used for interpolation of a defective pixel to be corrected. . The plurality of pixels in one block of the image sensor are four pixels arranged squarely adjacent vertically and horizontally,
The imaging device is a microlens disposed above the four pixels of the one block, and the microscopic light that is incident on the microlens is divided into pupils and guided to the light receiving surfaces of the four pixels, respectively. The stereoscopic imaging device according to claim 1, further comprising a lens. The defective pixel correcting means includes a distance between the center position of the defective pixel to be corrected and the center position of four pixels in the vertical and horizontal directions around the defective pixel, and the center position of the defective pixel and the periphery of the defective pixel. And determining means for determining whether or not a defective pixel is included in the shorter four pixels of the interval from the center position of the four pixels in the diagonally upper right and diagonally lower right directions, If it is determined that no defective pixel is included in the four pixels, the pixel value of the four pixels having the shorter interval is interpolated to calculate the pixel value of the defective pixel to be corrected, and the four pixels having the shorter interval are calculated. The pixel value of the defective pixel to be corrected is calculated by removing the defective pixel of the shorter four pixels and the pixel facing the defective pixel when it is determined that the defective pixel is included in The stereoscopic imaging apparatus according to 2. The defective pixel correcting means includes a distance between the center position of the defective pixel to be corrected and the center position of four pixels in the vertical and horizontal directions around the defective pixel, and the center position of the defective pixel and the periphery of the defective pixel. It includes determination means for determining whether or not a defective pixel is included in the shorter four pixels of the interval between the center position of the four pixels in the diagonally upper right and diagonally lower right directions, and the defective pixel is included. If it is determined that there is no defective pixel, the pixel value of the four pixels with the shorter interval is interpolated to calculate the pixel value of the defective pixel to be corrected, and if it is determined that the defective pixel is included, The stereoscopic imaging device according to claim 1, wherein a pixel value of the defective pixel to be corrected is calculated by interpolating pixel values of four pixels. A detection means for detecting whether it is horizontal shooting or vertical shooting;
Pixels that perform vertical pixel addition of the plurality of parallax images when horizontal detection is detected by the detection means, and pixels that perform horizontal pixel addition of the plurality of parallax images when vertical shooting is detected. And adding means,
When the detection unit detects horizontal shooting, the defective pixel correction unit calculates a pixel value of the defective pixel to be corrected by interpolating pixel values of vertical pixels around the defective pixel. 3. The stereoscopic imaging device according to claim 1, wherein when photographing is detected, a pixel value of the defective pixel to be corrected is calculated by interpolating a pixel value of a pixel in a horizontal direction around the defective pixel. A detection means for detecting whether it is horizontal shooting or vertical shooting;
Pixels that perform vertical pixel addition of the plurality of parallax images when horizontal detection is detected by the detection means, and pixels that perform horizontal pixel addition of the plurality of parallax images when vertical shooting is detected. And adding means,
When the detection unit detects horizontal shooting, the defective pixel correction unit uses a pixel value of an adjacent pixel in the vertical direction of the defective pixel as a pixel value of the defective pixel to be corrected, and detects vertical shooting. The stereoscopic imaging device according to claim 1, wherein a pixel value of a pixel adjacent to the defective pixel in the left-right direction is set as a pixel value of the defective pixel to be corrected. A detection means for detecting whether it is horizontal shooting or vertical shooting;
Pixels that perform vertical pixel addition of the plurality of parallax images when horizontal detection is detected by the detection means, and pixels that perform horizontal pixel addition of the plurality of parallax images when vertical shooting is detected. And adding means,
When the detection unit detects horizontal photographing, the defective pixel correction unit interpolates the pixel value of the vertical pixel including the vertical pixel adjacent to the defective pixel, and the pixel value of the defective pixel to be corrected When the vertical shooting is detected, the pixel value of the defective pixel to be corrected is calculated by interpolating the pixel value of the pixel in the horizontal direction including the adjacent pixel in the horizontal direction of the defective pixel. The stereoscopic imaging apparatus according to 2. The defective pixel correcting unit includes a determining unit that determines whether a vertical or horizontal adjacent pixel of a defective pixel to be corrected is a defective pixel, and horizontal detection is detected by the detection unit, and a vertical adjacent pixel is detected. Is determined as a defective pixel, the pixel value of the defective pixel to be corrected is calculated by interpolating the pixel value of the pixel closest to the correction target defective pixel in the upward and downward directions, and the detection means When vertical shooting is detected and the adjacent pixel in the left-right direction is determined to be a defective pixel, the pixel value of the pixel closest to the correction target defective pixel in the left direction and the right direction is interpolated to detect the correction target defect. The stereoscopic imaging apparatus according to claim 7, wherein a pixel value of a pixel is calculated. The image sensor is a red (R), green (G), or blue (B) filter, and an R filter and a first G filter adjacent to the R filter are repeatedly arranged in the horizontal direction. Bayer array in which the first line and the second line in which the B filter and the second G filter adjacent to the G filter are repeatedly arranged in the horizontal direction are alternately arranged in the vertical direction A color imaging device having a color filter of
The pixel value of the first G pixel in which the first G filter around the defective pixel to be corrected is arranged, and the pixel value of the second G pixel in which the second G filter is arranged Further comprising a step detecting means for detecting the step of
The defective pixel correcting means has a determining means for determining whether or not the step detected by the step detecting means is equal to or less than a predetermined threshold, and the defective pixel to be corrected is the first G pixel or the second G pixel. In the case of a G pixel, if the step is determined to be equal to or less than a predetermined threshold value, the four pixels of the first G pixel or the second G pixel in the oblique upper right and oblique lower right directions around the defective pixel When the pixel value of the defective pixel to be corrected is calculated by interpolating the value and it is determined that the step exceeds a predetermined threshold, the four first, vertical, left, and right directions around the defective pixel The stereoscopic imaging device according to claim 1, wherein a pixel value of the defective pixel to be corrected is calculated by interpolating a pixel value of a G pixel or the second G pixel. A detection means for detecting whether it is horizontal shooting or vertical shooting;
Pixels that perform vertical pixel addition of the plurality of parallax images when horizontal detection is detected by the detection means, and pixels that perform horizontal pixel addition of the plurality of parallax images when vertical shooting is detected. Adding means;
Step detecting means for detecting steps of output signals of a plurality of pixels in a block around a block to which the defective pixel to be corrected belongs, and
The defective pixel correction means has a determination means for determining whether or not the level difference detected by the level difference detection means is less than or equal to a predetermined threshold, and the level difference is determined to be equal to or less than a predetermined threshold by the determination means, and When the horizontal photographing is detected by the detection means, the pixel value of the adjacent pixel in the vertical direction of the defective pixel is set as the pixel value of the defective pixel to be corrected, and when the vertical photographing is detected, the horizontal value of the defective pixel is detected. When the pixel value of the adjacent pixel is set as the pixel value of the defective pixel to be corrected and the determining unit determines that the step exceeds a predetermined threshold, the same position as the defective pixel to be corrected in the block The stereoscopic imaging apparatus according to claim 1, wherein pixel values of the defective pixels to be corrected are calculated by interpolating pixel values of neighboring pixels having a relationship. An imaging device in which a single photographing optical system and a plurality of pixels adjacent in the horizontal and vertical directions are made into one block, and any one of the color filters including three primary colors is arranged for each block. In the image processing method of a stereoscopic imaging device having a single imaging element that receives light that has passed through different regions of the imaging optical system and can simultaneously acquire a plurality of horizontal and vertical parallax images,
Obtaining a plurality of parallax images output from the imaging device;
Calculating a pixel value of a defective pixel in the plurality of parallax images by interpolating a pixel value of a peripheral pixel of the defective pixel, the peripheral pixels used for interpolation of the defective pixel to be corrected as the plurality of pixels Using peripheral pixels having the same parallax information and the same color information as the defective pixels in the parallax images of
An image processing method including: Detecting whether shooting by the stereoscopic imaging device is horizontal shooting or vertical shooting;
When the horizontal shooting is detected, vertical pixel addition is performed among the plurality of parallax images, and when vertical shooting is detected, horizontal pixel addition is performed among the plurality of parallax images;
Further including
In the step of correcting the defective pixel, when the horizontal shooting is detected, the pixel value of the pixel in the vertical direction around the defective pixel is interpolated to calculate the pixel value of the defective pixel to be corrected, and the vertical shooting is performed. 12. The image processing method according to claim 11, wherein the pixel value of the defective pixel to be corrected is calculated by interpolating pixel values of pixels in the left and right directions around the defective pixel. Detecting whether shooting by the stereoscopic imaging device is horizontal shooting or vertical shooting;
When the horizontal shooting is detected, vertical pixel addition is performed among the plurality of parallax images, and when vertical shooting is detected, horizontal pixel addition is performed among the plurality of parallax images;
Further including
In the step of correcting the defective pixel, when the horizontal photographing is detected, the pixel value of the adjacent pixel in the vertical direction of the defective pixel is set as the pixel value of the defective pixel to be corrected, and when the vertical photographing is detected, The image processing method according to claim 11, wherein a pixel value of an adjacent pixel in the left-right direction of the defective pixel is a pixel value of the defective pixel to be corrected. The step of correcting the defective pixel includes a step of determining whether vertical or horizontal adjacent pixels of the defective pixel to be corrected are defective pixels, the horizontal shooting is detected, and the vertical adjacent pixels are defective. When the pixel is determined as a pixel, the pixel value of the defective pixel to be corrected is calculated by interpolating the pixel value of the pixel closest to the correction target defective pixel in the upward and downward directions, and the vertical photographing is detected. When the adjacent pixel in the left-right direction is determined to be a defective pixel, the pixel value of the defective pixel to be corrected is interpolated by the pixel value of the pixel closest to the correction target defective pixel in the left direction and the right direction. The image processing method according to claim 13 to calculate.

Images