WO2003101119A1

WO2003101119A1 - Image processing method, image processing program, image processor

Info

Publication number: WO2003101119A1
Application number: PCT/JP2003/006388
Authority: WO
Inventors: Kenichi Ishiga
Original assignee: Nikon Corporation
Priority date: 2002-05-24
Filing date: 2003-05-22
Publication date: 2003-12-04

Abstract

An image processing method comprising the image acquiring process of acquiring a first image that is represented by a coloring system consisting of a plurality of color components and consists of a plurality of pixels, each one of which has color information on at least one color component and which are disposed in a delta lattice form, the color information generation process of generating, by using the acquired first image color information, at least one new piece of color information in the same delta-lattice-form pixel positions as with the first image, the pixel position conversion process of converting color information on a plurality of pixels, including the generated color information and being in delta-lattice-form pixel positions, into color information in respective inter-pixel positions by performing a one-dimensional displacement processing between pixels arranged in one direction, and the output process of outputting, by using pixel-positions-converted color information, a second image where a plurality of pixels are disposed in a square lattice form.

Description

Description Image processing method, image processing program, image processing apparatus The disclosure of the following priority application is incorporated herein by reference.

Japanese Patent Application No. 2000, No. 1, 507, 888 (filed on May 24, 2002) Japanese Patent Application No. 159, 228, 2008 (No. 20) (Filed on May 31, 2012) Japanese patent application No. 200, No. 1, 592, 209 (filed May 31, 2002) Japanese patent application, filed in Japanese No. 1 592 250 (filed on May 31, 2002)

The present invention relates to an image processing method, an image processing program, and an image processing apparatus for processing image data obtained by a color filter having a delta arrangement. Background art

An electronic camera captures an image of a subject using an image sensor such as a CCD. In this image sensor, a Bayer arrangement in which three color filters of RGB (red, green, blue) are arranged as shown in FIG. 24 (a) is known. Also, a Delaunay arrangement arranged as shown in Fig. 24 (b) is known. Further, honeycomb arrangements arranged as shown in FIG. 24 (c) are also known. Various image processing methods are available for the image data obtained by the Bayer array, for example, US Pat. No. 5,552,827, US Pat. No. 5,629,734, and JP-A-2001-245314. Proposed.

On the other hand, with respect to image data obtained in a Dell array, an image processing method is proposed in Japanese Patent Application Laid-Open No. 8-340455, US Pat. No. 5,805,217, and the like.

However, US Pat. No. 5,805,217 does not perform image processing in accordance with the characteristics unique to the delta array. Also, Japanese Patent Application Laid-Open No. 8-340455 proposes only a simple image restoration method. Also, the various image processing methods proposed for the Bayer array cannot necessarily be applied to the Delta array as they are. Disclosure of the invention

The present invention provides an image processing method, an image processing program, and an image processing device that output high-definition square grid array image data based on image data obtained by a triangular grid color filter or the like such as a delta array. I do.

Further, the present invention provides an image for outputting image data capable of deriving a spatial color resolution inherent in a delta array or the like based on image data obtained by a triangular lattice color filter or the like in a delta array or the like. Provided are a processing method, an image processing program, and an image processing device.

In addition, the present invention provides image data obtained by performing high-resolution interpolation processing based on image data obtained by a triangular lattice color filter or the like such as a Derby array, and an image that is finely converted to another color system. Provided are an image processing method, an image processing program, and an image processing device for outputting data.

Further, the present invention provides an image processing method and an image processing program for outputting, for example, high-definition image data of a different color system based on image data obtained by a triangular lattice color filter such as a delta arrangement. An image processing device is provided.

A first image processing method according to the present invention is represented by a color system composed of a plurality of color components, one pixel includes a plurality of pixels having color information of at least one color component, and the plurality of pixels are triangular. An image acquisition procedure for acquiring a first image arranged in a lattice, and using the acquired color information of the first image, at least one new color information is converted into the same triangular lattice pixels as the first image. A procedure for generating color information at a position, and performing a one-dimensional displacement process between the pixels arranged in one direction with the color information of a plurality of pixels at a pixel position of a triangular lattice including the generated color information. And a second image in which a plurality of pixels are arranged in a square grid using the color information obtained by converting the pixel position into color information of each pixel position. Output procedure to perform

In the first image processing method, the new color information is preferably color information of a color component missing in each pixel of the first image among color components of a color system of the first image.

Further, it is preferable that the new color information is color information of a color system different from the color system of the first image. Further, it is preferable that the one-dimensional displacement process is performed using a one-dimensional filter including positive and negative coefficient values.

Further, it is preferable that the one-dimensional displacement processing is performed on the first image every other line in units of a line.

In addition, it is preferable that a similarity determination procedure for determining the strength of similarity in at least three directions is further provided, and in the color information generation procedure, new color information is generated according to the determined similarity strength. In this case, in the similarity determination procedure, it is preferable to calculate the similarity in at least three directions and determine the strength of the similarity in each direction based on the reciprocal of the similarity.

Further, based on the color information of a plurality of pixels at the pixel positions of the triangular grid including the generated color information, a color difference generating procedure for generating color information of a color difference component at the pixel positions of the triangular grid, and a color difference generating procedure. Preferably, the method further comprises a correction procedure for correcting the color information of the generated color difference component.

Further, based on the color information of a plurality of pixels at the pixel positions of the triangular lattice including the generated color information, a luminance generation procedure for generating color information of the luminance component at the pixel positions of the triangular lattice, and a luminance generation procedure It is preferable that the method further includes a correction procedure for correcting the generated color information of the luminance component.

In the second image processing method of the present invention, a plurality of pixels represented by first to n-th color components (n≥2) and having color information of one color component in one pixel are arranged in a triangular lattice. An image acquisition procedure for acquiring the obtained first image, an interpolation procedure for interpolating the color information of the first color component to the pixel where the first color component is missing, using the color information of the acquired first image, An output step of outputting a second image based on the color information of the first image and the interpolated color information, wherein the interpolation step includes: 1) a first color component with respect to an interpolation target pixel of the first image. 2) The curvature information of at least one of the first to n-th color components is determined by a fixed calculation, and interpolation is performed based on the average information and the curvature information. In the second image processing method, the image processing method may further include a similarity determination procedure for determining the level of similarity in at least three directions, and the interpolation procedure may include a first procedure based on the similarity determined in the similarity determination procedure. It is preferable that the calculation of the average information of the color components is made variable. It is preferable that the curvature information is obtained by a second derivative calculation. When a first image represented by the first to third color components is input, it is preferable to perform interpolation based on the curvature information of all of the first to third color components.

A third image processing method according to the present invention records a first image represented by a plurality of color components, in which a plurality of pixels having color information of one color component in one pixel are arranged in a non-rectangular shape. A first direction group similarity calculating step of calculating a similarity for each of a plurality of first direction groups using the color information of the first image; and a color information of the first image. A second direction group similarity calculation for calculating a similarity for each of a second direction group composed of a plurality of directions orthogonal to at least one direction of the first direction group and different from the first direction group, using A similarity determination procedure for determining the strength of similarity between the first direction groups by using the similarity of the first direction group and the similarity of the second direction group together.

In the third image processing method, it is preferable that the method further includes a color information generating step of generating at least one new color information at a pixel position of the first image based on the determination result of the similarity determining step. In this case, when the first image is represented by the first to third color components, the color information generating procedure includes the steps of: adding a second color component and / or a third color component to pixels having the first color component; Preferably, the information is generated. Further, it is preferable that the color information generating procedure generates color information of a luminance component different from the color information of the first image. It is preferable that the color information generating step generates color information of a color difference component different from the color information of the first image. At this time, when the first image is represented by the first to third color components, the color information generation procedure includes (1) a color difference component between the first color component and the second color component, and (2) It is preferable to generate color information of three types of color difference components, that is, a color difference component between the second color component and the third color component and (3) a color difference component between the third color component and the first color component.

The first direction group similarity calculating procedure, D l, D 2, calculated ~ similarity C _{D 1,} CD ₂ of N direction indicated by DN (N≥ 2), ~, a C _DN, In the second direction group similarity calculation procedure, D 1,, D 2 ', ..., DN' (D i 'is a direction orthogonal to D i, i = 1, 2,, ..., N) (N≥2) similarity CD, CD ₂ ', ~,

'Is calculated, and similarity determination procedures, the intensity of the similarity between the first direction group _{_{(C D 1' C DN /}} C: (C D2 in _{'ZC D 2):: ~} (CDN' ZC DN) Preferably, the determination is made using a function based on the expressed ratio.

Further, the pixels of the first image are arranged in a triangular lattice, and the first direction group similarity calculation procedure is performed. It is preferable to set both N and 3 for the second direction group similarity calculation procedure.

A fourth image processing method according to the present invention records a first image represented by a plurality of color components and in which a plurality of pixels each having color information of one color component are arranged in a non-rectangular shape. A first method of calculating a similarity composed of color information at a first pixel interval for each of a first direction group including a plurality of directions using a recording procedure and a color information of a first image. Using the direction group similarity calculation procedure and the color information of the first image, the similarity composed of the color information at the second pixel interval is converted into a second direction composed of a plurality of directions different from the first direction group. For each of the groups, the similarity between the first direction group and the second direction group is used together with the procedure for calculating the second direction group similarity to be calculated, and the similarity between the first direction groups is determined. And a similarity determination procedure.

In the fourth image processing method, it is preferable that the method further includes a color information generating step of generating at least one new color information at a pixel position of the first image based on the determination result of the similarity determining step.

In both the first direction group similarity calculation procedure and the second direction group similarity calculation procedure, it is preferable to calculate the similarity between the same colors composed of the color information between the same color components as the similarity. In this case, the first direction group consists of directions in which color information of the same color component is arranged at a first pixel interval, and the second direction group consists of color information of the same color component arranged at a second pixel interval. It is preferred that they consist of the directions in which they are placed.

The first image is represented by first to third color components, and both the first direction group similarity calculation procedure and the second direction group similarity calculation procedure include: (1) color information of only the first color component (2) Similarity component composed of color information of only the second color component (3) At least two types of similarity components composed of color information of only the third color component It is preferable to calculate the similarity using the similarity component.

Further, the first pixel interval is preferably longer than the second pixel interval.

Preferably, the first pixel interval is about three pixel intervals, and the second pixel interval is about two pixel intervals.

In the third or fourth image processing method, both the first direction group similarity calculation procedure and the second direction group similarity calculation procedure include not only the similarity calculated for the processing target pixel to be subjected to image processing, but also A class calculated for pixels surrounding the pixel to be processed It is preferable to calculate the similarity including the similarity.

In the first image, a plurality of pixels are preferably arranged in a triangular lattice. In addition, the first image is represented by first to third color components, and the first to third color components are preferably distributed at a uniform pixel density.

In a fifth image processing method according to the present invention, a plurality of pixels represented by first to n-th color components (n≥2) and each pixel having color information of one color component are arranged in a triangular lattice shape An image acquisition procedure for acquiring the obtained first image, an interpolation procedure for interpolating the first color component to a pixel lacking the first color component using the acquired color information of the first image, An output step of outputting a second image based on the color information of the image and the interpolated color information, wherein the interpolation step is such that the first color component is 2 with respect to the interpolation target pixel of the first image. The average information of the first color component is obtained by using the color information of the area including the pixel closest to the second, and interpolation is performed. In the fifth image processing method, the image processing method further includes a similarity determination procedure for determining the strength of the similarity in at least three directions, and the interpolation procedure includes a first procedure based on the similarity determined in the similarity determination procedure. Preferably, the average information of the color components is obtained.

According to a sixth image processing method of the present invention, a first image is obtained in which a plurality of pixels represented by a plurality of color components and each pixel has color information of one color component arranged in a triangular lattice shape. An image acquisition procedure and color information for generating color information of a color component different from the color information of the first image by performing weighted addition of the acquired color information of the first image with a variable coefficient value of zero or more. A generation step, and an output step of outputting a second image using the generated color information. The color information generation step includes: for a pixel to be processed of the first image, a color component of the pixel; Weighted addition of the color information in the area including the pixel whose different color component is second closest.

The sixth image processing method may further include a similarity determination procedure for determining the strength of similarity in at least three directions, and the color information generation procedure may be performed according to the similarity strength determined in the similarity determination procedure. It is preferable to make the coefficient value of the weighted addition variable. Further, when the first image is represented by first to third color components, and the pixel having the first color component of the first image is the processing target pixel, the color information generating procedure includes the processing target pixel, It is preferable to perform weighted addition of color information in a region including a pixel whose two color components are second closest and a pixel whose third color component is second closest. Also, after the color information generation procedure and before the output procedure, color information of a color component different from the color information of the first image generated in the color information generation procedure is filtered by a filter process including a predetermined fixed filter coefficient. It is preferable to further include a correction procedure for performing correction. In this case, it is preferable that the filter coefficients include positive and negative values.

In a seventh image processing method according to the present invention, a plurality of pixels represented by first to n-th color components (n≥2) and having color information of one color component in one pixel are arranged in a triangular lattice. An image obtaining procedure for obtaining the obtained first image, a color difference generating procedure for generating color information of a color difference component between the first color component and the second color component using the color information of the first image, An output step of outputting a second image using the color information of the generated color difference component, wherein the color difference generation step includes: outputting at least a second color to a pixel having the first color component of the first image. The color information of the color difference component is generated using the color information of the pixel whose component is the second closest.

In the seventh image processing method, the color difference generating procedure includes: (1) color information of a first color component of the pixel, and (2) a color information of a first color component of the first image. It is preferable that the color information of the color difference component is generated based on the average information of the color information of the second color component in a region including the pixel whose second color component is second closest to the pixel. In the color difference generation procedure, it is preferable that color information of a color difference component is further generated based on curvature information of a second color component for the pixel to be processed.

In addition, it is preferable that a similarity determination procedure for determining the level of similarity in at least three directions is further provided, and the color difference generation procedure generates color information of a color difference component according to the similarity strength.

In the fifth to seventh image processing methods, it is preferable that, in the output procedure, the second image is output to the same pixel position as the first image.

An eighth image processing method according to the present invention obtains a first image represented by first to third color components and in which a plurality of pixels having color information of one color component are uniformly distributed to one pixel. Image information, and color information of a color component different from the color information of the first image by weighting and adding the acquired color information of the first image with a variable coefficient value of zero or more. A generation step, and an output step of outputting a second image using the generated color information.The color information generation step includes the steps of: first, second, and third color components for all pixels of the first image; The color information is always weighted and added at a uniform color component ratio. In the eighth image processing method, the image processing method further includes a similarity determination procedure for determining the strength of the similarity in a plurality of directions, and the color information generation procedure is based on the strength of the similarity determined in the similarity determination procedure. It is preferable to make the coefficient value of the weighted addition variable.

In the first image, a plurality of pixels are preferably arranged in a triangular lattice. Also, after the color information generation procedure and before the output procedure, color information of a color component different from the color information of the first image generated in the color information generation procedure is filtered by a filter process including a predetermined fixed filter coefficient. It is preferable to further include a correction procedure for performing correction. In this case, it is preferable that the filter coefficients include positive and negative values.

The ninth image processing method of the present invention is an image acquisition procedure for acquiring a first image composed of a plurality of pixels represented by three or more types of color components and having one pixel having color information of one color component. And a color information generating procedure for generating color information of a luminance component and color information of at least three types of color difference components using the acquired color information of the first image, and a color information generating procedure. An output procedure for outputting the second image using the color information of the luminance component and the color information of the color difference component can be obtained.

The ninth image processing method further includes a conversion procedure of converting the color information of the luminance component and the color information of at least three types of color difference components into color information of three types of color components. Preferably, the second image is output using the color information of the three types of color components converted by the conversion procedure.

In addition, it is preferable that the color information of the luminance component and the color information of the color difference component generated in the color information generation procedure are color information of components different from the three or more types of color components of the first image. Further, the first image is represented by first to third color components, a plurality of pixels are uniformly distributed, and the color information generation procedure is as follows: (1) The color component ratio of the first to third color components is Color information of a luminance component composed of 1: 1: 1, (2) color information of a color difference component between a first color component and a second color component, and (3) a second color component and a third color. It is preferable to generate color information of a color difference component between the components and (4) color information of a color difference component between the third color component and the first color component.

The image processing apparatus further includes a similarity determination procedure for determining the degree of similarity in a plurality of directions. The color information generation procedure includes a color component report of a luminance component according to the similarity determined in the similarity determination procedure. It is preferable to generate the color information of at least three types of color difference components. Further, it is preferable that the first image has a plurality of pixels arranged in a triangular lattice. According to a tenth image processing method of the present invention, there is provided an image acquisition procedure for acquiring a first image including a plurality of pixels represented by three or more types of color components and having one pixel having color information of one color component. A color difference generation procedure for generating at least three types of color difference component color information using the acquired color information of the first image, and a correction process for performing correction processing on the generated color difference component color information. And an output step of outputting a second image using the corrected color difference component color information.

In this image processing method, the first image is represented by first to third color components, and the color difference generation procedure includes: 1) color information of a color difference component between the first color component and the second color component; It is preferable to generate color information of a color difference component between the second color component and the third color component, and 3) color information of a color difference component between the third color component and the first color component.

Also, the first image is represented by first to third color components, and the color difference generation procedure uses the color information of the first image to generate color information of a luminance component different from the color information of the first image. And

1) color information of the color difference component between the first color component and the luminance component, 2) color information of the color difference component between the second color component and the luminance component, and 3) between the third color component and the luminance component. It is preferable to generate the color information of the color difference component. In this case, in the first image, the first to third color components are evenly distributed to a plurality of pixels, and the color difference generation procedure determines that the color component ratio of the first to third color components is 1 as a luminance component. It is preferable to generate color information of a luminance component composed of 1: 1.

In the eighth to tenth image processing methods, it is preferable that, in the output procedure, the second image is output at the same pixel position as the first image.

A computer-readable computer program product according to the present invention has an image processing program for causing a computer to execute the procedure of the image processing method described in any of the above.

This computer program product is preferably a recording medium on which an image processing program is recorded.

An image processing device according to the present invention includes a control device that executes a procedure of the image processing method according to any one of the above. BRIEF DESCRIPTION OF THE FIGURES FIG. 1 is a functional block diagram of the electronic camera according to the first embodiment.

FIG. 2 is a flowchart showing an outline of image processing performed by the image processing unit in the first embodiment.

FIG. 3 is a diagram showing a positional relationship between pixels obtained by an image sensor in a delta arrangement. FIG. 4 is a diagram showing coefficients used for peripheral addition.

FIG. 5 is a diagram showing coefficient values used when obtaining the curvature information dR.

FIG. 6 is a diagram showing an achromatic spatial frequency reproduction region in a delta arrangement.

FIG. 7 is a diagram showing coefficient values used for the one-dimensional displacement processing.

FIG. 8 is a diagram illustrating pixel positions used in the calculation according to the second embodiment.

FIG. 9 is a diagram showing coefficient values used when obtaining the curvature information dR, dG, and dB. FIG. 10 is a flowchart showing an outline of image processing performed by the image processing unit in the third embodiment.

FIG. 11 is a diagram illustrating coefficient values of a low-pass filter.

FIG. 12 is a diagram showing coefficient values of another low-pass filter.

FIG. 13 is a diagram illustrating coefficient values of another low-pass filter.

FIG. 14 is a diagram showing Laplacian coefficient values.

FIG. 15 is a diagram showing coefficient values of other Laplacians.

FIG. 16 is a diagram showing coefficient values of other Laplacians.

Fig. 17 shows the concept of generating a luminance plane (Y) and three color difference planes (Cgb, Cbr, Crg) directly from the delta plane of the delta array, and then converting it to the original RGB color system. FIG. FIG. 18 is a flowchart showing an outline of the image processing performed by the image processing unit in the fourth embodiment.

FIG. 19 is a diagram showing a state where the Crg and Cbr components are obtained at the R position and the Cgb component is obtained at the nearest neighbor pixel.

FIG. 20 is a diagram illustrating a spatial frequency reproduction region of each of the RGB components of the delta array. FIG. 21 is a flowchart illustrating an outline of image processing performed by the image processing unit in the sixth embodiment.

FIG. 22 is a diagram defining adjacent pixels.

Figure 23 shows a program stored on a storage medium such as a CD-ROM or on the Internet. FIG. 3 is a diagram showing a state of being provided through a data signal.

FIG. 24 is a diagram showing a Bayer array, a delta array, and a honeycomb array of RGB color filters.

FIG. 25 is a diagram showing the concept of a process of interpolating image data obtained in a delta array on a triangular lattice and restoring the image data to a square lattice data.

FIG. 26 is a diagram illustrating the azimuth relationship of the similarity in the fifth embodiment. FIG. 27 is a flowchart showing the image restoration processing and the gradation processing. BEST MODE FOR CARRYING OUT THE INVENTION

First Embodiment 1

(Configuration of electronic camera)

FIG. 1 is a functional block diagram of the electronic camera according to the first embodiment. The electronic camera 1 includes an A / D conversion unit 10, an image processing unit 11, a control unit 12, a memory 13, a compression / decompression unit 14, and a display image generation unit 15. Also, an external unit such as a personal computer (PC) 18 via a memory card interface unit 17 for interfacing with a memory card (card-shaped removable memory) 16 and a predetermined cable or wireless transmission path. And an external interface unit 19 for interfacing with the external interface. These blocks are interconnected via a bus 29. The image processing unit 11 is composed of, for example, a one-chip microprocessor dedicated to image processing.

The electronic camera 1 further includes a photographing optical system 20, an image sensor 21, an analog signal processor 22, and a timing controller 23. An optical image of the subject obtained by the imaging optical system 20 is formed on the image sensor 21, and the output of the image sensor 21 is connected to the analog signal processor 22. The output of the analog signal processor 22 is connected to the A / D converter 10. The output of the control unit 12 is connected to the timing control unit 23. The output of the timing control unit 23 is an image sensor 21, an analog signal processing unit 22, an A / D conversion unit 10 and an image processing unit 1. Connected to 1. The image sensor 21 is composed of, for example, a CCD or the like. The electronic camera 1 further includes an operation unit 24 and a monitor 25 corresponding to a release button, a selection button for mode switching, and the like. The output of the operation unit 2 4 is the control unit 1 2 and the output of the display image generation unit 15 is connected to the monitor 25.

The monitor 18 and the printer 27 are connected to the PC 18, and the application program recorded on the CD-ROM 28 is installed in advance. The PC 18 is connected to a memory card interface unit (not shown) for interfacing with the memory card 16 in addition to a CPU, a memory, and a hard disk (not shown). It has an external interface (not shown) that interfaces with external devices such as the camera 1.

In the electronic camera 1 configured as shown in FIG. 1, when the operator selects a shooting mode via the operation unit 24 and presses the release button, the control unit 12 controls the evening imaging control unit 2. 3, the timing of the image sensor 21, the analog signal processor 22, and the AZD converter 10 is controlled. The image sensor 21 generates an image signal corresponding to the optical image. The image signal is subjected to predetermined signal processing in an analog signal processing unit 22, digitized in an AZD conversion unit 10, and supplied as image data to the image processing unit 11.

In the electronic camera 1 of the present embodiment, since the color filters of R (red), G (green), and B (blue) are arranged in a delta array (described later) in the image sensor 21, the image processing unit 1 The image data supplied to 1 is represented by the RGB color system. Each pixel constituting the image data has color information of any one of RGB components. The image processing unit 11 performs image processing such as gradation conversion and contour emphasis on such image data in addition to performing image data conversion processing described later. The image data on which such image processing has been completed is subjected to predetermined compression processing by the compression / decompression section 14 as necessary, and is recorded on the memory card 16 via the memory card interface section 17.

Image data that has undergone image processing is recorded on a memory card 16 without compression, or converted into the color system used by the monitor 18 and the printer 27 on the PC 18 side. Alternatively, the data may be supplied to the PC 18 via the external interface 19. When the playback mode is selected by the operator via the operation unit 24, the image data recorded in the memory card 16 is transferred to the memory card interface 17 via the memory card interface unit 17. It is read out and subjected to decompression processing by the compression / decompression unit 1 2, and the display image generation unit 1 Displayed on monitor 2 through 5.

The decompressed image data is not displayed on the monitor 25, but is converted to the color system used by the monitor 26 and the printer 27 on the PC 18 side, and the external interface is used. The data may be supplied to the PC 18 via the unit 19.

(Image processing)

Next, the process of performing interpolation processing on the image data obtained in the Dell array on a triangular lattice and restoring it to square lattice data that is easy to handle on a computer will be described. FIG. 25 is a diagram showing the concept of these processes. The triangular lattice refers to an arrangement in which pixels of the image sensor are arranged with a shift of 1/2 pixel for each row. It is a line that forms a triangle when the centers of adjacent pixels are connected. The center point of a pixel may be called a grid point. The delta arrangement in Fig. 24 (b) is arranged in a triangular lattice. An image obtained with the arrangement shown in Fig. 24 (b) may be called an image in which pixels are arranged in a triangular arrangement. The square lattice refers to an arrangement in which pixels of the image sensor are arranged without being shifted for each row. The arrangement forms a square when the centers of adjacent pixels are connected. The bay array in Fig. 24 (a) is arranged in a square lattice. An image obtained in the arrangement shown in Fig. 24 (a) may be called an image in which pixels are arranged in a rectangular (square) shape.

As will be described later, a process of shifting the delta array image data by 1/2 pixel every other line generates square grid data. Interpolating on a triangular lattice means performing interpolation on the state of image data obtained in a delta array. That is, it means to interpolate the color information of the color component missing at the pixel position of the delta array. FIG. 2 is a flowchart illustrating an outline of the image processing performed by the image processing unit 11. In step S1, an image obtained by the image sensor 21 in the delta arrangement is input. In step S2, the similarity is calculated. In step S3, similarity is determined based on the similarity obtained in step S2. In step S4, an interpolation value of a missing color component in each pixel is calculated based on the similarity determination result obtained in step S3. In step S5, the obtained RGB color image is output. The RGB color image output in step S5 is image data obtained on a triangular lattice. Next, in step S6, one-dimensional displacement processing is performed on the image data obtained on the triangular lattice. One-dimensional displacement processing is performed on every other row of image data, as described later. Do it. In step S7, square lattice image data is output by combining the image data subjected to the one-dimensional displacement processing and the image data not subjected to the one-dimensional displacement processing. Steps S1 to S5 are interpolation processing on a triangular lattice, and steps S6 and S7 are square processing. Hereinafter, details of these processes will be described.

Interpolation on square grid

In the following description, the interpolation process at the R pixel position will be described as a representative. FIG. 3 is a diagram showing a positional relationship between pixels obtained by the image sensor 21 in the delta arrangement. The pixels of the image pickup device 21 are arranged with a shift of 1/2 pixel for each row, and the color filters are arranged on each pixel at a ratio of RGB components of 1: 1: 1. That is, the colors are evenly distributed. For reference, the color filter in the bay array has RGB arranged in the ratio of 1: 2: 1 (Fig. 24 (a)).

A pixel having R component color information is called an R pixel, a pixel having B component color information is called a B pixel, and a pixel having G component color information is called a G pixel. The image data obtained by the image sensor 21 has only one color component for each pixel. The interpolation process is a process for calculating color information of other color components missing in each pixel by calculation. Hereinafter, a case where the color information of the G and B components is interpolated at the R pixel position will be described.

In FIG. 3, the pixel to be processed (pixel to be interpolated), which is the R pixel, is called Rctr. In addition, each pixel position existing around the pixel Rctr is expressed using an angle. For example, the B pixel existing in the 60-degree direction is expressed as B060, and the G pixel is expressed as G060. However, this angle is not exact but approximate. Also, the direction connecting 0 ° -180 ° is 0 ° direction, the direction connecting 120 ° -300 ° is 120 ° direction, the direction connecting 240 ° -60 ° is 240 ° direction, and the direction connecting 30 ° -210 ° is The direction connecting 30 degrees, the direction connecting 150-330 degrees is called the 150-degree direction, and the direction connecting 270-90 degrees is called the 270-degree direction.

1. Calculation of similarity

Calculate the similarities C000, C120, and C240 in the directions of 0, 120, and 240 degrees. In the first embodiment, as shown in Expressions (1), (2), and (3), the similarity between different colors composed of different color components is obtained.

C000 (I GOOO-Rct r I + 1 B 180-Rct r | + 1 (GOOO + B 180) / 2-Rct r I) / 3 ... (1)

Next, the accuracy of the similarity is increased by performing the peripheral addition of the similarity and considering the continuity with the peripheral pixels. Here, the above similarity obtained for each pixel is subjected to peripheral addition using the coefficients shown in FIG. 4 for the R position. The similarity with margin addition is written in small letters. [] Indicates a pixel position on the delta array viewed from the processing target pixel. Equation (4) shows the peripheral addition of the similarity in the 0-degree direction.

cOOO = (6 * C000 [Rctr]

+ C000 [R030] + C000 [R150] + CO0O [270]

+ C000 [R210] + C000 [300] + C000 [R090]) / 12 ... (4)

C120 in the 120-degree direction and c240 in the 240-degree direction are obtained in the same manner.

2. Similarity judgment

The smaller the value of the similarity is, the greater the similarity is. Therefore, the strength of the similarity in each direction is determined so as to continuously change at the reciprocal ratio of the similarity. That is, it is determined by (1 / cOOO): (1 / C120): (1 / C240). Specifically, the following weight coefficient is calculated.

When the similarity in each direction is expressed as a weighting coefficient w000, wl20, w240 normalized by 1, w000 = (cl20 * c240 + Th) / (cl20 * c240 + c240 * c000 + c000 * cl20 + 3 * Th) … (5) wl20 = (c240 * c000 + Th) / (cl20 * c240 + c240 * c000 + c000 * cl20 + 3 * Th) ... (6) w240 = (c000 * cl20 + Th) / (cl20 * c240 + c240 * c000 + c000 * cl20 + 3 * Th)... (7) However, the threshold Th is a term to prevent divergence and takes a positive value. Normally, it is sufficient to set Th = l, but it is better to increase this threshold for images with much noise such as high-sensitivity images. The weighting coefficients w000, wl20, and w240 are values according to the strength of similarity. In the Bayer arrangement, as shown in U.S. Pat.No. 5,552,827, U.S. Pat.No. 5,629,734, and JP-A-2001-245314, a continuous weighting coefficient There are two types of methods, a statistical determination method and a discrete determination method based on threshold value determination. In the Bayer array, the nearest neighbor G component exists densely in four directions with respect to the pixel to be interpolated, so it can be used with almost no problem using either the continuous judgment method or the discrete judgment method. However, in a delta array where the nearest G component exists only in three directions: 0, 120, and 240 degrees In this case, it is important to determine the direction continuously based on the weighting factor. The nearest neighbor G component is a pixel which has a G component at a side of the edge of the pixel to be interpolated. In FIG. 3, they are G0OO, G12O, and G240.

3. Interpolation value calculation

The interpolation values of the G component and the B component are calculated using the above weighting coefficients. The interpolated value consists of two items, average information and curvature information.

Gave = w00O * Gave000 + wl20 * Gavel2O½24O * Gave24O ... (8)

Bave = w000 * Bave000 + wl20 * Bavel20 + w240 * Bave240 ... (9)

here,

GaveOOO = (2 * G000 + G180) / 3 (10)

BaveOOO = (2 * B180 + B000) / 3 (11)

Gavel20 = (2 * G120 + G300) / 3 (12)

Bavel20 (2 * B300 + B120) / 3 (13)

Gave240 (2 * G240 + G060) / 3 (14)

Bave240 (2 * B060 + B240) / 3 (15)

It is.

dR = (6 * Rctr-R030-R090- 150- 210-R270-R330) / 12 ... (16)

Gctr = Gave + dR ... (17)

Bctr = Bave + dR ... (18)

Normally, in G interpolation of the Bay array, average information is obtained using the nearest neighbor G component. However, in the case of the delta arrangement, it has been experimentally found that the same problem occurs in which vertical lines and boundaries in the 30-degree and 150-degree directions become buzzy. For example, in the case of a vertical line boundary, it is assumed that the weighting factors are w000 = 0 and wl20 = w2400.5, and this corresponds to a portion where the averaging process is performed between the two directions. However, the average of the nearest neighbors alone will mean the two points on the left side at a point along the vertical line, and the average of the two points on the right side at the next adjacent diagonally upper right or lower right point. It is thought to be. Therefore, when averaging is performed according to the distance ratio from the interpolation target pixel while considering the second adjacent pixel, for example, in the case of a vertical line, the average is always taken from both the right and left sides. Direction, the bubbly of the 150-degree direction boundary improves dramatically. Therefore, as shown in Expressions (10) to (15), an averaging process including up to the second adjacent pixel is performed. This improves the spatial resolution in the directions of 30, 150, and 270 degrees. For example, in equation (10), G000 is the nearest neighbor pixel or the first neighbor pixel, and G180 is the second neighbor pixel. In equation (12), G120 is the nearest neighbor pixel or the first neighbor pixel, and G300 is the second neighbor pixel. The same applies to other expressions. The first adjacent pixel is a pixel separated by about 1 pixel pitch, and the second adjacent pixel is a pixel separated by about 2 pixel pitch. It can be said that Rc tr and G 120 are separated by about 1 pixel pitch, and that Rc tr and G300 are separated by about 2 pixel pitch. FIG. 22 is a diagram that defines adjacent pixels. “center” is a pixel to be processed, “nearest” is the nearest or nearest neighbor or the first neighboring pixel, and “2nd” is the second neighboring pixel.

Further, in the conventional technique of Bayesian interpolation, a term corresponding to the curvature information dR is generally calculated in consideration of the same directionality as the average information. However, in the delta array, the directionality of the average information (0, 120, and 240 degrees) and the direction of the curvature information that can be extracted (30, 150, and 270 degrees) do not match. In such a case, the curvature information in the 30-degree direction and the 150-degree direction are averaged to define the curvature information in the 0-degree direction, and the interpolation value may be calculated in consideration of the directionality of the curvature information as in the case of the Payer array. Although it is possible, it has been experimentally found that it is more effective to uniformly extract the curvature information without considering the directionality. Therefore, in the present embodiment, the average information considering the directionality is corrected using the curvature information having no directionality. As a result, it is possible to improve gradation clarity up to a high-frequency region in every direction.

The equation (16) for calculating the curvature information dR uses the coefficient values shown in FIG. Equation (16) obtains the difference between the interpolation target pixel Rctr and the peripheral pixel, the difference between the peripheral pixel on the opposite side and the interpolation target pixel Rctr, and further obtains the difference. Therefore, it is obtained by the second derivative operation.

The curvature information dR is information indicating the degree of change in the color information of a certain color component. When the value of the color information of a certain color component is plotted and represented by a curve, this is information indicating a change in the curve of the curve and the degree of the curve. That is, the concave of the change of the color information of the color component This is the amount reflecting the structural information on the convexity. In the present embodiment, the curvature information dR of the R pixel position is obtained using the Rctr of the pixel to be interpolated and the color information of the surrounding R pixels. The G component color information is used for the G pixel position, and the B component color information is used for the B pixel position.

As described above, the “interpolation processing of the G and B components at the R pixel position” was performed. `` Interpolation of B and R components at G pixel position '' is a symbol of `` interpolation of G and B components at R pixel position ''. R is G, G is B, and B is R. Exactly the same processing may be performed instead. Also, "interpolation processing of R and G components at B position" is a circular replacement of the symbol "interpolation processing of B and R components at G position" with G for B, B for R, and R for G. The same process may be performed. That is, the same arithmetic routine (subroutine) can be used in each interpolation process.

The image reconstructed as described above can bring out all the limit resolution performance of the delta array in the spatial direction. In other words, when viewed in the frequency space (k-space), as shown in Fig. 6, all hexagonal achromatic color reproduction regions of the delta array are resolved. Also, a clear image can be obtained in the gradation direction. This is an extremely effective method especially for images with many achromatic parts.

Square processing 1

Next, a description will be given of a process of restoring image data interpolated on a triangular lattice as described above into square lattice data which is easy to handle on a computer. As a conventional technique for restoring single-chip image sensor data shifted by 1/2 pixel on a row basis to a square grid data in which three colors are aligned, Japanese Patent Laid-Open Publication No. Are disclosed in JP-A-2001-103295 and JP-A-2000-194386. Japanese Patent Application Laid-Open No. 8-340455 discloses an example in which restored data is generated at a pixel position different from that of a triangular lattice, or restored data is generated on a virtual square lattice having a pixel density twice that of the triangular lattice. Also, an example is shown in which a half row of the triangular lattice is restored to the same pixel position, and the other half row is restored to a pixel position shifted by 1/2 pixel from the triangular lattice. However, this implements the interpolation processing directly at the square grid position, and applies a different interpolation processing when the distance between adjacent pixels of the triangular grid changes. On the other hand, in Japanese Patent Application Laid-Open No. 2001-103295, a square is formed by two-dimensional cubic interpolation, and in Japanese Patent Application Laid-Open No. 2000-194386. Generates virtual double-density square lattice data.

Although there are various methods as described above, in the first embodiment, a method for maximizing the performance of the delta array will be described.

In the first embodiment, as described above, first, the interpolation data is restored with the triangular lattice at the same pixel position as the delta arrangement. This makes it possible to bring out the spatial frequency limit resolution performance of the delta array. If the color difference correction processing and the edge enhancement processing are also performed in the triangular arrangement, the effect works well isotropically. Therefore, once the image is restored on the triangular grid, an RGB value is generated for each pixel.

Next, the image data generated on the triangular lattice is subjected to square conversion. It has been experimentally found that it is important to keep the original data as much as possible to maintain the resolution performance of the triangular lattice. Therefore, a displacement process that shifts half a line by half a pixel every other line is performed. The remaining half of the line is not processed, and the Nyquist resolution of the vertical line of the triangular arrangement is maintained. Experiments have shown that the displacement process can maintain the vertical line resolution of the triangular lattice with little problem if it is self-estimated by cue-pick interpolation within one dimension of the row to be processed, though there is some influence from the Nyquist frequency of the square lattice. Was.

That is, the processing of the following equation (19) is performed on the even-numbered rows, and then the substitution processing of the equation (20) is performed. Thus, the color information of the R component at the pixel position of [X, y] is obtained.

tmp_R [, y] = (― 1 * R [x- (3/2) pixel, y] + 5 * [x— (1/2) pixel, y]

+ 5 * [x + (1/2) pixel, y] -1 * [xl (3/2) pixel, y] 1/8 ... (19)

R [x, y] = tmp_R [x. Y]… (20)

The G and B components are obtained in the same manner. The processing of equation (19) is called cubic displacement processing. In addition, since the displacement processing is performed on the position in the one-dimensional direction, it is also called one-dimensional displacement processing. FIG. 7 is a diagram showing coefficient values used in equation (19). The one-dimensional displacement processing by the cubic described above can be said to be a processing of applying a one-dimensional filter consisting of positive and negative coefficient values.

The image restoration method as described above not only can maintain the limit resolution of the triangular arrangement to the maximum, but also the image restoration processing on the triangular grid can use the same processing routine for all pixels. Since only half of the rows need be subjected to simple one-dimensional processing, it is possible to achieve a simpler algorithm than the conventional technology without increasing the amount of data.

¾

* ¥ 4¾ ^ ^ ^ ^ ^

¾ Λ is 撇

,

1

D m

Post-processing is performed to remove false colors and return to the RGB color system. In addition, in order to compensate for the decrease in sharpness due to an optical low-pass filter, etc., edge enhancement processing is performed on the luminance component Y plane. In the case of a delta array, exactly the same post-processing may be applied.

However, in the third embodiment, a post-processing method suitable for the delta array will be described. The configuration of the electronic device 1 of the third embodiment is the same as that of FIG. 1 of the first embodiment, and the description thereof is omitted.

FIG. 10 is a flowchart illustrating an outline of image processing performed by the image processing unit 11 in the third embodiment. It is assumed that the interpolation processing is performed on a triangular lattice as in the first embodiment. Figure 10 starts when the RGB color image after interpolation processing is input. That is, steps S1 to S5 in FIG. 2 of the first embodiment are completed, and thereafter, the flowchart in FIG. 10 starts.

In step S11, the RGB color image data after the interpolation processing is input. In step S12, the RGB color system is converted to the YCrCgCb color system unique to the third embodiment. In step S13, a low-pass fill process is performed on the color difference plane (CrCgCb plane). In step S14, edge enhancement processing is performed on the luminance plane (Y plane). In step S15, when the false color on the color difference plane has been removed, conversion is performed to return the YCrCgCb color system to the original RGB color system. In step S16, the obtained RGB color image is output. The RGB color image data output in step S16 is image data obtained on a triangular lattice.

When performing the square processing on the image data obtained on the triangular lattice, the processing of steps S6 and S7 in FIG. 2 is performed as in the first embodiment. Hereinafter, the details of the processing in steps S12 to S15 will be described.

1. Color system conversion

Convert from RGB color system to YCrCgCb color system. However, the YCrCgCb color system is defined as represented by equations (28) to (31).

Y [i, j] = (R [i,] + G [i, j] + B [i, j]) / 3 ... (28)

Cr [i, j] = R [i,] -Y [i, j]-.. (29)

Cg [i, j] = G [i,] -Y [i, j] ... (30) Cb [i, j] = B [i, j] -Y [i, j] ... (31)

After taking an image of a circular zone plate and performing interpolation processing on a triangular grid, as shown in equation (28), performing conversion to treat RGB equally in the Y plane, the center of the hexagonal corner in Fig. 6 is obtained. The color moiré that occurs on the luminance plane Y completely disappears from the luminance plane Y. As a result, all false color elements can be included in the color difference components Cr, Cg, and Cb. In the third embodiment, three types of color difference components are prepared and handled instead of the two types of color difference components conventionally handled, so that all false color elements unique to the Dell array can be extracted. it can.

2. Color difference correction

Here, an example in which the mouth-to-mouth process is performed on the chrominance plane by the equation (32) as the chrominance correction processing will be described. When a color difference port one-pass filter is applied on a triangular lattice (triangular arrangement), the false color reduction effect can be obtained in all directions. Also, median filter processing on a triangular lattice may be used. Here, as shown in FIG. 22, [] represents a pixel position on the triangular lattice viewed from the pixel to be processed. FIG. 11 illustrates the coefficient values used in equation (32). The one-pass filter is not limited to the one shown here, and another filter may be used. Figures 12 and 13 show examples of other mouth-pass filters.

<Low-pass processing>

tmp—Cr [center] = l9 * Cr [center]

+ 2 * (Cr [nearestOOO] + Cr [nearest 120] + Cr [nearest240] + Cr [nearest 180] + Cr [nearest 300] + Cr [nearest060])

+ Cr [2nd000] ICr [2ndl20] + Cr [2nd240]

+ Cr [2ndl80] + Cr [2nd300] + Cr [2nd060]) / 27 ... (32)

The same applies to tmp_Cg and tnip-one Cb.

Cr [i, j] = tmp_Cr [i, j] ... (33)

Cg [i, j] = tmp_Cg [i, j] ... (34)

Cb [i, j] = tmp-Cb [i, j] ... (35)

3. Edge enhancement

Next, if edge enhancement processing is necessary for the luminance plane Y, the following processing is performed on the triangular lattice. Work.

Edges are extracted by Laplacian processing on a triangular lattice. FIG. 14 illustrates the coefficient values used for the Laplace Braun's processing of equation (36). It should be noted that the Laplacian is not limited to the one shown here, and another one may be used. Figures 15 and 16 show examples of other Labrians.

YH [center] = 16 * Y [center]

― (Y [nearest 000] + Y [nearest 120] + Y [nearest 240]

+ Y [nearest 180] + Y [nearest 300] + Y [nearest060])) / 12 ... (36) <Edge enhancement>

Y [center] = Y [center] + * YH [center] ... (37)

Here, K is a value greater than or equal to zero, and is a parameter for adjusting the level of edge enhancement. 4. Color system conversion

As described above, when the color difference correction processing is performed and the false color on the color difference surface is removed, the color system is returned to the original RGB color system.

R [i,〗] = Cr [i, j] + Y [i, j]… (38)

G [i, j] = Cg [i, j] + Y [i, j] ... (39)

B [i, j] = Cb [i, j] + Y [i, j] ... (40)

In this way, the luminance resolution is saved to the Y component generated at a ratio of R: G: B = 1: 1: 1, and three equally treated color difference components Cr, Cg, and Cb are introduced. This enables color difference correction with extremely high false color suppression capability. In addition, by performing the color difference correction processing and the luminance correction processing on the triangular lattice, it is possible to perform the correction processing suitable for the directionality of the Dell array.

—Fourth embodiment—

In the first embodiment, the process of interpolating the color information of the color component missing in each pixel on the triangular lattice has been described. In the fourth embodiment, an example of image restoration processing of a different system from the interpolation processing in the RGB plane of the first embodiment will be described. In this method, the luminance component and the color difference component are created directly from the Dell array without interpolating in the RGB plane. Ideally, the usefulness of separating one luminance plane and three chrominance planes described in the third embodiment is taken over, and the luminance plane maximizes the achromatic luminance resolution. The three color difference planes are responsible for maximizing the color resolution of the three primary colors.

The configuration of the electronic camera 1 according to the fourth embodiment is the same as that of FIG. 1 according to the first embodiment, and a description thereof will be omitted.

(Image processing)

Figure 17 shows the concept of generating the luminance plane (Y) and three color difference planes (Cgb, Cbr, Crg) directly from the delta plane in the Delhi array, and then converting to the original RGB color system FIG. FIG. 18 is a flowchart showing an outline of the image processing performed by the image processing unit 11 in the fourth embodiment.

In step S21, an image obtained by the delta-array image sensor 21 is input. In step S22, the similarity is calculated. In step S23, similarity is determined based on the similarity obtained in step S22. In step S24, a luminance plane (Y0 plane) is generated based on the similarity determination result obtained in step S23 and the delta array image data obtained in step S21. In step S25, a correction process is performed on the luminance plane (Y0 plane) obtained in step S24.

On the other hand, in step S26, color difference components Cgb, Cbr, and Crg are generated based on the similarity determination result obtained in step S23 and the image data of the Delaunay array obtained in step S21. I do. In step S26, all the color difference components Cgb, Cbr, and Crg have not yet been generated in all the pixels. In step S27, interpolation processing is performed on the color difference components that have not been generated based on the surrounding color difference components. As a result, the color difference plane of Cgb, Cbr, Crg is completed.

In step S28, the generated Y, Cgb, Cbr, and Crg color systems are converted to the RGB color system. In step S29, the converted RGB single image data is output. Steps S 21 to S 29 are all processes on a triangular lattice. Therefore, the RGB image data output in step S29 is triangular lattice image data. Hereinafter, details of these processes will be described. If the square processing is necessary, the same processing as steps S6 and S7 in FIG. 2 of the first embodiment is performed. 1. Calculation of similarity

First, the similarity is calculated. Here, the similarity obtained by an arbitrary method may be used. However, the most accurate one shall be used. For example, the similarity between different colors shown in the first embodiment, the similarity between same colors shown in the second embodiment, a combination thereof, or the similarity between different colors based on a color index or the like is used. Alternatively, the same color similarity may be switched and used.

2. Similarity judgment

The determination is made in the same manner as in the first embodiment.

3. Recovery value calculation

1) Generate luminance component

First, a luminance plane is generated by weighting and adding the delta plane so that R: G: B = 1: 1: 1: 1 always with a positive coefficient that changes according to the direction similarity. For the same reason as in the first embodiment, the range of the weighted addition takes the G component and the B component up to the second adjacent pixel. That is, when the equations (8) and (9) are rewritten using the definition equations of the equations (10) to (15) of the first embodiment, the equations (41) and (42) are obtained.

<G> _{2 ll} d = Gave = wOOO * GaveOOO + wl20 * Gavel20lw240 * Gave240 ... (41)

<B> 2 nd = Bave = w000 * Bave000 + wl20 * Bavel20 + w240 * Bave240 (42) Using these and the central R component, a luminance component Y0 is generated by equation (43).

Since the luminance component generated in this way is always generated using the positive direction weighting coefficient while including the center pixel at a constant color component ratio, the potential for gradation sharpness is extremely high, and the spatial resolution power is extremely high. A high image can be obtained, and an image can be obtained that is very smoothly connected to peripheral pixels without being affected by chromatic aberration. For example, when the similarity between different colors is adopted as the similarity for an achromatic color zone plate, the spatial resolution reaches the limit resolution of FIG. 6 as in the first embodiment.

2) Brightness component correction

Since the above-mentioned luminance plane is generated using only positive coefficients, a correction process using Laplacian is performed in order to extract the potential of gradation clarity contained therein. Since the luminance plane Y0 is designed to be connected to peripheral pixels extremely smoothly taking into account the directionality, it is not necessary to calculate a correction term that requires a new calculation in the correction process according to the directionality. Thus, a single process using a fixed bandpass filter may be used. triangle As shown in FIGS. 14 to 16 of the third embodiment, several methods are available for taking the Laplacian in the arrangement. However, if the degree of optimality is slightly increased, the luminance plane Y0 can only be generated by collecting the G and B components in the directions of 0, 120, and 240 degrees. Here, a case where the correction is performed using the Laplacian shown in FIG. 15 in the directions of 30 degrees, 150 degrees, and 270 degrees, which is independent of the above (Equation (44)). Let Y be the corrected brightness component.

Band pass processing>

YH [center] = I6 * Y0 [center]

― (Y0 [2nd030] + Y0 [2ndl50] + Y0 [2nd270]

+ Y0 [2nd210] + Y0 [2nd330] + Y0 [2nd090]) 1/12 ... (44)

Y [center] = Y [center] + k * YH [center] ... (45)

Here, k is a positive value and is usually set to 1. However, by setting the value to be larger than 1, the edge enhancement processing as shown in the third embodiment can be provided here.

3) Color difference component generation

Unlike the definition of the third embodiment, the three chrominance planes are generated directly from the delta plane independently of the luminance plane Y. Find three color difference components Cgb, Cbr, Crg. Here, Cgb = G-B, Cbr = B-R, and Crg = R-G are defined.

First, find Crg and Cbr at the R position. The difference between the R component of the central pixel and the G or B component of the peripheral pixel is calculated while taking into account the directionality.

Here, dG and dB are the same as those defined by the equations (24) and (25) in the second embodiment, and G> ₂ „ _d and B> ₂ „ _d are obtained by the equation (41) ( Same as 42). In the generation of the color difference components, as in the first embodiment, the average information including the pixels up to the second adjacent pixel is calculated, and the consistency with the luminance component is obtained. The resolution has been raised. Also, dG and dB are not necessarily required, but they are added because they have the effect of increasing color resolution and vividness. The color difference components Cgb and Crg at the G position and the color difference components Cbr and Cgb at the B position are obtained in the same manner. At this point, the Crg and Cbr components have been obtained at the R position, and the Cgb component has been obtained at the nearest pixel. FIG. 19 is a diagram showing this state.

Next, the Cgb component at the R position is obtained from the equation (48) using the Cgb component of the pixel around the R position (interpolation processing). At this time, the calculation is performed using the direction determination result obtained at the R position. Cgb [center] = w000 * (Cgb [nearestOOO] + Cgb [nearest 180]) / 1

+ wl20 * (Cgb [nearest 120] + Cgb [nearest300]) / 2

+ w240 * (Cgb [nearest240] + Cgb [nearest060]) / 2 ... (48) As described above, the four components of YCgbCbrCrg are obtained for all pixels. If necessary, the color difference planes Cgb, Cbr, and Crg may be subjected to correction processing such as a color difference port one-pass filter similar to that of the third embodiment to suppress false colors.

4) Color system conversion

Convert YCgbCbrCrg generated for each pixel to RGB color system. Y = (MG + B) / 3, Cgb = G−B, Cbr = B−R, Crg = R−G are not independent four equations, but satisfy the relationship of Cgb + Cbr + Crg = 0. In other words, the conversion method is not unique because it is a 4 to 3 conversion, but to suppress color moiré and maximize the brightness resolution and color resolution, all Y, Cgb, C br and Crg components are included, and the mutual moire canceling effect is used. In this way, all of the highest performances generated by the respective roles of Y, Cgb, Cbr, and Crg can be reflected in each of R, G, and B.

R [i, j] = (9 * Y [i, j] + Cgb [i, j] -2 * Cbr [i, j] + 4 * Crg [i, j]) / 9 ... (49)

G [i, j] = (9 * Y [i, j] + 4 * Cgb [i, j] + Cbr [i, j] -2 * Crg [i, j]) / 9 ... (50)

B [i, j] = (9 * Y [i, j] -2 * Cgb [i, j] + 4 * Cbr [i, j] + Crg [i, j]) / 9 ... (51)

As described above, the image restoration method according to the fourth embodiment has extremely high gradation clarity, and simultaneously achieves excellent luminance resolution performance and color resolution performance in the spatial direction, while reducing chromatic aberration. It has the effect of being strong against them. When the square processing is required, it can be performed in the same manner as in the first embodiment.

—Fifth Embodiment I

In the second embodiment, the similarity is determined by calculating the similarity between the same colors. At this time, only similarities in the directions of 0, 120 and 240 degrees were obtained. But the fifth implementation In the embodiment, similarities in the directions of 30, 150, and 270 degrees are also obtained. The configuration of the electronic camera 1 according to the fifth embodiment is the same as that of FIG. 1 of the first embodiment, and a description thereof will be omitted.

The description focuses on the case where G and B components are interpolated at the R pixel position. Also, refer to FIG. 8 of the second embodiment. At the R pixel position, there are three nearest neighbors of the G component at positions that point to 0, 120, and 240 degrees, and the second adjacent pixel points to 60, 180, and 300 degrees separated by two pixels. There are three in position. Similarly, for the B component, there are three nearest neighbors at 60, 180, and 300 degrees adjacent to each other, and the second adjacent pixel points to 0, 120, and 140 degrees separated by two pixels. There are three in position. In addition, the nearest neighbor pixel of the R component is 6 pixels at positions 30 degrees, 90 degrees, 150 degrees, 210 degrees, 270 degrees, and 330 degrees separated by 2 pixels, and the second adjacent pixel is separated by 3 pixels There are six positions at 0, 60, 120, 180, 240, and 300 degrees.

1. Calculation of similarity

1) Calculate the similarity between same colors at 3 pixel intervals

Calculate the similarities C000, C120, and C240 in the directions of 0, 120, and 240 degrees. The same-color similarity between the same color components in these directions cannot be defined shorter than three pixel intervals.

C000 = ((| R000-Rctr | + | 180- ctr |) / 2 + IG000-G1801 + | B000-B 18011/3 ... (52) C120 = i (| R120-Rctr | + | R300-Rctr |) / 2 + | G120-G300 | + | B120-B300 |) / 3 ... (53) C240 = ((| 240-Rctr | + | R060-Rctr |) / 2 + | G240-G060 | + | B240-B060 |) / 3 ... (54) The similarity between same colors defined in this way checks the directionality that matches the direction in which the G component and the B component are missing at the R pixel position. Therefore, it is considered that it has the ability to spatially resolve the horizontal image line in the 0-degree direction and the image structure in the 120- and 240-degree directions in the colored portion. However, it should be noted that, unlike the similarity between the same colors at a two-pixel interval in the Bayer array, the information is between pixels that are very far from each other at a three-pixel interval.

2) Calculate the similarity between two pixels at the same color.

Next, the similarities C030, C150, and C270 in the directions of 30 degrees, 150 degrees, and 270 degrees are calculated. The similarity between the same colors in these directions can be defined by a shorter two-pixel interval, unlike the 0, 120, and 240 degree directions.

C030 = ((| R030-Rctr | + | R210-Rctr |) / 2 + IG240-G000 | + | B060-B18011 / 3 ... (55) C150 = I (| R150-Rctr | + lR330-Rctr |) / 2 + | GOOO-G120 | + | B180-B300 |) / 3 ... (56) C270 = l (| R270-Rctr | + | R090 -Rctr |) / 2 + | G120-G240 | l | B300-B060 |) / 3 ... (57) However, the direction that does not match the direction in which the G component and the B component are missing at the R pixel position Since we are examining similarities, techniques are needed to utilize them effectively. 3) Similarity marginal addition

Further, by performing marginal addition on each of the above-described similarities and taking into account continuity with neighboring pixels, the precision of the similarity is increased. Here, peripheral addition is performed for the R pixel position for each of the above-described six similarities obtained for each pixel. The similarity after margin addition is written in lower case. Here, [] indicates the pixel position on the DELUXE array viewed from the processing target pixel. Equation (58) is the same as equation (4) in the first embodiment.

cOOO = (6 * C000 [Rctr]

+ C000 030] + C000 [R150] + C000 [R270]

+ C000 [R210] + C000 [300] + C000 [R090]) / 12 ... (58)

cl20, c240, c030, cl50, and c270 are obtained in the same manner.

FIG. 26 shows the azimuth relationship of the similarity described above.

2. Similarity judgment

The smaller the value of the similarity described above, the greater the similarity in that direction. However, the significance of judging similarity is the direction in which the G component and B component that do not exist in the processing target pixel exist, that is, the directions of 0, 120, and 240 degrees, and 30 and 150 degrees. It is not meaningful to determine the similarity in the 270 degrees direction. Therefore, it is conceivable to first determine the degree of similarity in the 0-, 120-, and 240-degree directions by using the similarity in the 0-, 120-, and 240-degree directions and continuously determine the reciprocal ratio. It is. That is, it is determined by (1 000): (1 / C120): (1 / C240).

For example, since the similarity cOOO has the ability to resolve chromatic horizontal lines, it is possible to extend the ky-axis direction of the frequency reproduction range of each of the RGB color components in the Derby arrangement of FIG. 20 to the limit resolution. That is, such a determination method can extend the color resolution of a chromatic image to the limit resolution of the vertices of the hexagon in FIG. However, since it is a similarity over a long distance of 3 pixel intervals, if the angle between them is an The directionality cannot be determined due to the influence of the wave number component, and especially in the direction of 30 degrees, 150 degrees, and 270 degrees, the most adverse effect occurs. Near the midpoint of each side of the hexagon in Figure 20 It can only exhibit color resolution that can be lost.

Therefore, in order to prevent such adverse effects of long-range correlation, the similarities C030, cl50, and C270 of short-range correlation are effectively used. However, simply taking the reciprocal can only determine the similarity in the directions of 30, 150, and 270 degrees, so in order to convert it to the similarity in the directions of 0, 120, and 240 degrees, the reciprocal Rather, it is interpreted that the value of the similarity itself expresses the similarity in the direction orthogonal to the 30 °, 150 °, and 270 ° directions, ie, the 120 °, 240 °, and 0 ° directions. Therefore, the similarity in the directions of 0, 120 and 240 degrees is determined by the following ratio.

(c270 / c000): (c030 / cl20): (cl50 / c240)

Expressing the similarity in the 0, 120, and 240-degree directions as weighting coefficients wOOO, vl20, and w240 standardized by 1,

w000 = (cl20 * c240 * c270 + Th)

/ (cl20 * c240 * c270 + c240 * cOOO * c030 + cOOO * cl20 * cl50 + 3 * Th) ... (59) wl20 = (c240 * c000 * c030 + Th)

/ (cl20 * c240 * c270 + c240 * cOOO * c030 + cOOO * cl20 * cl50 + 3 * Th) ... (60) w240 = (c000 * cl20 * cl50 + Th)

/ (cl20 * c240 * c270 + c240 * cOOO * c030 + cOOO * cl20 * cl50 + 3 * Th) ... (61) However, the constant Th takes a positive value in the term to prevent divergence. Normally, it is sufficient to set Th = l, but it is better to increase this threshold for images with much noise such as high-sensitivity images.

The 0 degree direction and the 270 degree direction, the 120 degree direction and the 30 degree direction, and the 240 degree direction and the 150 degree direction are orthogonal relations. This orthogonal relationship is expressed as a 0-degree direction 丄 270-degree direction, a 120-degree direction 方 30-degree direction, and a 240-degree direction 度 150-degree direction.

The similarity continuously determined using the similarities in the six directions has a spatial resolution that accurately reproduces all the hexagons in FIG. 20 with respect to the chromatic image. In addition, the spatial resolution based on the similarity between same colors can always be achieved without being affected by chromatic aberration included in the optical system because similarity between the same color components is observed.

3. Interpolation value calculation The interpolation value is obtained in the same manner as in the second embodiment.

As described above, in the fifth embodiment, it is possible to extract all the spatial color resolving power of each RGB single color originally included in the delta arrangement for any image. In addition, clear image restoration in the gradation direction is possible, and it shows strong performance even for systems containing chromatic aberration. The calculation of the similarity and the determination of the similarity in the fifth embodiment can be applied to the calculation of the similarity and the determination of the similarity in the fourth embodiment. Such an image restoration method has extremely high gradation clarity, achieves excellent luminance resolution performance and color resolution performance in the spatial direction at the same time, and exhibits an effect of being strong against chromatic aberration. In particular, it is possible to derive the highest color resolution performance of the delta arrangement while achieving clear gradation. In normal image processing, there is a trade-off in which edge enhancement processing is performed to increase gradation clarity, and the color resolution is reduced by that amount, resulting in fading or a loss of three-dimensional appearance. By introducing restoration values of the four components, these conflicting problems can be solved simultaneously.

—Sixth Embodiment—

In the Bayer array, the RGB signal is usually converted to YCbCr consisting of luminance and color difference to reduce false colors remaining in the image after the interpolation processing, and a color difference port-one-pass filter is applied to the Cb and Cr planes. Post-processing is performed to remove false colors by applying a color difference median filter or to return to the RGB color system. Even in the case of the delta arrangement, if the Nyquist frequency cannot be completely reduced in the optical low-pass filter, appropriate false color reduction processing is required to improve the appearance. In the sixth embodiment, a post-processing method that does not impair the color resolution performance, which is an excellent feature of the delta array, as much as possible will be described. The configuration of the electronic device 1 according to the sixth embodiment is the same as that shown in FIG. 1 of the first embodiment, and a description thereof will be omitted.

FIG. 21 is a flowchart illustrating an outline of image processing performed by the image processing unit 11 in the sixth embodiment. Interpolation processing is performed on a triangular lattice as in the first, second, fourth, and fifth embodiments. Figure 21 starts with the input of an RGB color image after interpolation. For example, steps S1 to S5 in FIG. 2 of the first embodiment are completed, and thereafter, the flowchart in FIG. 21 starts.

In step S31, the RGB color image data after the interpolation processing is input. Stay In step S32, the RGB color system is converted to the YCgbCbrCrg color system unique to the sixth embodiment. In step S33, a color determination image is generated. In step S34, a color index is calculated using the color determination image generated in step S33. In step S35, a color judgment of low saturation or high saturation is performed based on the color index of step S34.

In step S36, based on the color determination result in step S35, the mouth-pass filter to be used is switched to perform color difference correction. The color difference data to be corrected is generated in step S32. In step S37, conversion is performed to return the YCgbCbrCrg color system to the original RGB color system when the false colors on the color difference plane have been removed. In step S38, the obtained RGB color image data is output. The RGB color image data output in step S38 is image data obtained on a triangular lattice.

When performing the square processing on the image data obtained on the triangular lattice, the processing of steps S6 and S7 in FIG. 2 is performed as in the first embodiment. Hereinafter, the details of the processing in steps S32 to S37 will be described.

1. Color system conversion

Convert from RGB color system to YCgbCbrCrg color system. However, the YCgbCbrCrg color system is defined by the following equations (62) to (65).

Y [i, j] = (R [i, j] + G [i, j] + B [i, j]) / 3 ... (62)

Cgb [i, j] = G [i, j] -B [i, j] ... (63)

Cbr [i, j] = B [i, j] -R [i, j] ... (64)

Crg [i, j] = R [i, j] -G [i, j] ... (65)

Performing conversion that treats RGB equally in this way, for example, the color moiré that occurs at the center of the hexagonal corner in Fig. 6 in a square zone plate completely disappears from the luminance plane Y. All false color elements can be included in the color difference components Cgb, Cbr, and Crg.

2. Color evaluation

1) Generation of color judgment image

In order to distinguish false colors in achromatic areas from color areas as much as possible, To reduce the false color by applying a powerful color difference low-pass filter to the image. However, since this is a temporary color judgment image, it does not affect the actual image.

TCgb [center] = l9 * Cgb [center]

+ 2 * (Cgb [nearestOOO] + Cgb [nearestl20] + Cgb [nearest240]

+ Cg [nearestl80] + Cgb [nearest300] + Cgb [neares t060])

+ Cgb [2nd000] + Cgb [2ndl20] + Cgb [2nd240]

+ Cgb [2ndl80] + Cgb [2nd300] + Cgb [2nd060]] / 27 ... (66)

TCbr and TCrg are also calculated in the same way.

2) Calculation of color index

Next, the color index Cdiff is calculated using the image for color determination in which the false color is reduced, and the color evaluation is performed in pixel units.

Cdiff [i, j] = (| Cgb [i, j] l + ICbr [i, j] | + | Crg [i, j] |) / 3 ... (67)

3) Color judgment

The above continuous color index Cdiff is judged as a threshold value and converted to a discrete color index BW.

if Cdiff [i, j] ≤Th then BW [i, j] = 'a' (low saturation)

else then BW [i, j] = 'c' (high saturation part)

Here, the threshold Th is preferably set to about 30 for 256 gradations.

3. Adaptive color difference correction

Since the image could be divided into two regions by color judgment, false colors in the achromatic portion were strongly eliminated, while color resolution in the chromatic portion was subjected to a color difference correction process that preserved as much as possible. Here, the mouth-to-pass processing of Expressions (68) and (69) is performed, but the median fill processing may be used. [] Represents the pixel position on the triangular lattice viewed from the processing target pixel. The low-pass filter of Expression (68) uses the coefficient values shown in FIG. 11 of the third embodiment, and the single-pass filter of Expression (69) uses the coefficient values shown in FIG.

<Low-pass processing>

if BW [center] = 'a'

tmp_Cgb [center] = l9 * Cgb [center]

+ 2 * (Cgb [nearestOOO] ICgb [nearestl20] + Cgb [nearest240]

+ Cgb [nearestl80] + Cgb [nearest300] + Cgb [nearest060]) White

White

The relationship between the B method and the YCCC method as in the fourth embodiment) and the gradation processing are shown here. FIG. 27 is a flowchart showing the processing.

1) Linear gradation delta array data input (step S 4 1)

2) Gamma correction processing (Step S 42)

3) Image restoration processing (Step S43)

4) Inverse gamma correction processing (Step S44)

5) User gamma correction processing (step S45)

The user gamma correction process is a process of converting a linear gradation to an 8-bit gradation suitable for display output, that is, a process of compressing the dynamic range of an image to within the range of the output display device. Independently of this, once the gradation is converted to a certain gamma space and the image restoration processing corresponding to the first to sixth embodiments is performed, a better restoration result can be obtained. . There are the following methods for this gradation conversion.

Input signal x (0≤x≤xmax), Output signal y (0≤x≤ymax), Input image is delta surface y = ymaxv (x / xmax)

If the input signal is 12 bits, xmax = 4095, and the output signal should be, for example, 16 bits ymax = 65535.

Input signal y (0≤x≤ymax), Output signal x (0≤x≤xmax), Input image is RGB plane

x = xmax (y / y ax) ²

When the image restoration processing is performed in the square root type gamma space as described above, the following advantages can be simultaneously obtained from a viewpoint different from the technique of the image restoration processing.

1) Color boundary can be sharpened

(For example, suppression of color fringing at the red-white border, suppression of black border generation, etc.)

2) Suppress false colors around bright spots (extremely bright areas)

3) Improved direction determination accuracy

1) and 2) are the effects created by the involvement of the “interpolated value calculator” in the RGB method of image restoration processing and the “reconstructed value calculator” in the Y method. In addition, 3) is the effect produced by the involvement of the “similarity calculation unit” and “similarity determination unit” in the image restoration process. Toes In addition, the input signal x contains an error dx = kx (k is a constant determined by the ISO sensitivity) of the quantum fluctuation, and when converted to the square root gamma space, this error is reduced to the total gradation 0 by the error propagation law. Since the uniform error dy = constant can be handled over ≤y≤ymax, the direction determination accuracy is improved. This technique can be applied not only to the delta arrangement but also to the interpolation processing of the payer arrangement and other various file arrangements. In the above-described embodiment, since the Delaware array originally has higher color resolution of a single color than the Payer array, by inserting this gradation conversion processing before and after the image restoration processing, even better color clarity is produced. It becomes possible.

In the first embodiment, an example has been described in which image data interpolated on a triangular lattice is subjected to square processing. Also, in other embodiments, it has been shown that when the square processing is necessary, the processing is performed. However, the image data itself that has been subjected to the interpolation processing or the color system conversion on the triangular lattice may be used as it is.

In the third embodiment, an example is described in which correction processing is performed on RGB image data generated in the same manner as in the first embodiment, but the present invention is not necessarily limited to this. The same applies to RGB color image data generated in other embodiments. As described above, the first to sixth embodiments can be appropriately combined with each other. That is, in the first to sixth embodiments, similar direction determination processing, interpolation processing or direct generation processing of color difference planes, post-processing such as correction, square processing, and the like are described. An optimal image processing method and processing apparatus can be realized by appropriately combining the processes of the embodiments.

In the above-described embodiment, the description has been made on the assumption that the image sensor is of a single-plate type. The present invention can be applied to a two-chip image sensor. In the case of the two-plate system, for example, one color component is missing in each pixel, but the content of the above embodiment can be applied to the interpolation processing of this one missing color component. Conversion processing can be performed in the same manner. Further, the method of directly generating a luminance component and a color difference component from the delta array without through interpolation processing according to the fourth embodiment can be similarly applied to a two-chip image sensor.

In the above embodiment, various calculation formulas are shown for determining similarity. However, the present invention is not necessarily limited to the content shown in the embodiment. Similarity with other, appropriate formulas The determination may be made. In addition, various calculation formulas have been shown in the calculation of the luminance information, but it is not necessarily limited to the contents described in the embodiment. The luminance information may be generated by another appropriate calculation formula.

In the above embodiment, an example of a low-pass filter for color difference correction and the like and a band-pass filter for edge enhancement have been described. However, the present invention is not necessarily limited to the contents described in the embodiment. A low-pass filter / band-pass filter having another configuration may be used.

In the above embodiment, an example of an electronic camera has been described, but the present invention is not necessarily limited to this content. It may be a video camera for capturing moving images, a personal convenience store with an image sensor, a mobile phone, or the like. That is, the present invention can be applied to any device that generates a color image data by an image sensor.

When applied to a personal computer or the like, a program relating to the above-described processing can be provided through a recording medium such as a CD-ROM or a data signal through the Internet or the like. FIG. 23 is a diagram showing this state. The personal convenience store 100 will be provided with the program via CD-ROM 104. The personal computer 100 has a function of connecting to the communication line 101. The combination computer 102 is a server computer that provides the above program, and stores the program on a recording medium such as a hard disk 103. The communication line 101 is a communication line such as the Internet, personal computer communication, or a dedicated communication line. The computer 102 reads out a program using the hard disk 103, and reads the program. The program is transmitted to the personal computer 100 via the communication line 101 ₍ that is, the program is transmitted as a data signal on a carrier wave via the communication line 101. In this way, the program It can be supplied as a computer-readable computer program product in various forms, such as a carrier wave, etc. The main advantages of the above embodiment are summarized as follows.

After performing interpolation processing at the pixel positions of the triangular lattice of the acquired image, it is converted to pixel positions of the square lattice, so it is arranged in a square lattice while maintaining the maximum resolution of the triangle arrangement as much as possible. The output image data can be output.

とき When interpolating the color information of the missing color component of the first image arranged in a square grid. Since the curvature information of the color component is obtained by a fixed calculation and used, It is possible to improve the tone clarity up to the high frequency region. That is, it is possible to perform high-definition interpolation while maximizing the effectiveness of image data in which pixels are arranged in a triangular lattice.

A similarity is calculated for each of the first direction groups including a plurality of directions, and each of the second direction groups including a plurality of directions that are orthogonal to at least one direction of the first direction group and different from the first direction group. Are calculated and the similarity is determined. For example, since the similarity in three directions is continuously determined using the similarity in six directions in the delta array, all the hexagons in FIG. It has a reproducible spatial resolution. That is, it is possible to extract all the spatial color resolving power of each of the RGB single colors originally possessed by the Delhi array.

When the first color component is interpolated into the pixel where the first color component is missing, interpolation is performed using the color information of the area including the pixel whose first color component is second closest to the interpolation target pixel. Therefore, for example, the image of the 30-degree, 150-degree, and 270-degree (vertical line) boundary lines in Figure 3 is dramatically improved. That is, the spatial resolution in the directions of 30, 150, and 270 degrees is improved.

For all pixels of the first image represented by the first to third color components, the color information of the first to third color components is always weighted and added at a uniform (1: 1: 1) color component ratio. The color information of a color component different from the color information of the first image is generated. The color information of the color components generated in this way has an extremely high gradation clear potential, an image with high spatial resolution is obtained, and an image that is connected to peripheral pixels very smoothly without being affected by chromatic aberration is obtained. can get.

Claims

The scope of the claims

1. An image processing method,

A pixel is represented by a color system composed of a plurality of color components, and one pixel includes a plurality of pixels having color information of at least one color component, and the plurality of pixels represent a first image arranged in a triangular lattice. An image acquisition procedure to be acquired,

A color information generating step of generating at least one new color information at the same triangular lattice pixel position as the first image using the acquired color information of the first image; and Pixel position conversion for converting color information of a plurality of pixels at pixel positions in the triangular lattice including information into color information of a position between pixels by performing a one-dimensional displacement process between pixels arranged in one direction. Instructions,

An output step of outputting a second image in which a plurality of pixels are arranged in a square lattice using the color information whose pixel positions have been converted.

2. In the image processing method described in claim 1,

The new color information is color information of a color component missing in each pixel of the first image among color components of a color system of the first image.

3. In the image processing method described in claim 1,

The new color information is color information of a color system different from the color system of the first image.

4. The image processing method according to any one of claims 1 to 3,

The one-dimensional displacement processing is performed using a one-dimensional filter including positive and negative coefficient values.

5. In the image processing method according to any one of claims 1 to 4,

The one-dimensional displacement processing is performed on the first image every other line in line units.

6. The image processing method according to any one of claims 1 to 5, further comprising a similarity determination procedure for determining the degree of similarity in at least three directions,

In the color information generation procedure, the new color information is generated according to the determined similarity strength.

7. An image processing method according to claim 6,

In the similarity determination procedure, the similarity in at least three directions is calculated, and the strength of similarity in each direction is determined based on the reciprocal of the similarity.

8. The image processing method according to any one of claims 1 to 7,

A color difference generation step of generating color information of a color difference component at the triangular lattice pixel position based on the color information of a plurality of pixels at the triangular lattice pixel position including the generated color information;

A correction procedure for correcting the color information of the color difference components generated in the color difference generation procedure.

9. In the image processing method according to any one of claims 1 to 7,

A luminance generating step of generating color information of a luminance component at the triangular lattice pixel position based on the color information of a plurality of pixels at the triangular lattice pixel position including the generated color information;

And a correction procedure for correcting the color information of the luminance component generated in the luminance generation procedure.

1 0. An image processing method,

An image acquisition procedure for acquiring a first image in which a plurality of pixels represented by first to n-th color components (n≥2) and each pixel having color information of one color component are arranged in a triangular lattice shape When,

Using the acquired color information of the first image, the first color component is assigned to the pixel where the first color component is missing. An interpolation procedure for interpolating color information of color components;

An output step of outputting a second image based on the color information of the first image and the interpolated color information,

The interpolation procedure includes: regarding an interpolation target pixel of the first image,

1) The average information of the first color component is obtained by a variable operation,

2) The curvature information of at least one of the first to n-th color components is obtained by a fixed calculation,

The interpolation is performed based on the average information and the curvature information.

1 1. In the image processing method described in claim 10,

A similarity judgment procedure for judging the degree of similarity in at least three directions is further provided,

In the interpolation procedure, the calculation of the average information of the first color component is made variable according to the similarity strength determined in the similarity determination procedure.

1 2. In the image processing method according to claim 10 or 11,

The curvature information is obtained by a second derivative operation.

13. In the image processing method according to any one of claims 10 to 12, when the first image represented by the first to third color components is input, all of the first to third The interpolation is performed based on the curvature information of the color components.

1 4. An image processing method,

A recording procedure of recording a first image in which a plurality of pixels represented by a plurality of color components and each pixel having color information of one color component are arranged in a non-rectangular form;

A first direction group similarity calculation procedure for calculating a similarity for each of a plurality of first direction groups using the color information of the first image;

Using the color information of the first image, for each of the second direction groups that are orthogonal to at least one direction of the first direction group and that include a plurality of directions different from the first direction group. A second direction group similarity calculation procedure for calculating the similarity by using

A similarity determining step of determining the strength of similarity between the first direction groups by using the similarity of the first direction group and the similarity of the second direction group together.

15. In the image processing method described in claim 14,

The method further includes a color information generation step of generating at least one new color information at a pixel position of the first image based on a determination result of the similarity determination step.

16. In the image processing method described in claim 14 or 15,

The first direction group similarity calculating procedure, D l, D 2, ~, N direction (N≥ 2) similarity C _{D 1,} CD2 represented by DN, calculates-the C _DN,

In the second direction group similarity calculation procedure, D 1 ′, D 2 ′,..., DN ′ (D i ′ is a direction orthogonal to D i, i = 1, 2, 2, N) Similarity of direction (N≥2)

C D 1 ·, then D2 ', ~, then DN' "ST

The similarity determination step includes determining the strength of similarity between the first direction groups.

(CDI '/ CDI): ( _CD2 ' ZC CDN '/ CDN)

The determination is made using a function based on the ratio represented by

17. The image processing method according to any one of claims 14 to 16, wherein the pixels of the first image are arranged in a triangular lattice,

The first direction group similarity calculation procedure and the second direction group similarity calculation procedure are both set to 3.

1 8. An image processing method,

A first direction group similarity calculation for calculating a similarity composed of color information at a first pixel interval for each of the first direction groups composed of a plurality of directions using the color information of the first image; Instructions, Using the color information of the first image, a similarity composed of color information at a second pixel interval is calculated for each of a second direction group including a plurality of directions different from the first direction group. A second direction group similarity calculation procedure,

1 9. In the image processing method described in claim 18,

20. In the image processing method according to claim 18 or 19,

In both the first direction group similarity calculation procedure and the second direction group similarity calculation procedure, the same color similarity composed of color information between the same color components is calculated as the similarity.

21. The image processing method according to any one of claims 18 to 20, wherein the first image is represented by first to third color components,

Both the first direction group similarity calculation procedure and the second direction group similarity calculation procedure

1) Similarity component composed of color information of only the first color component

2) Similarity component composed of color information of only the second color component

3) Similarity component composed of color information of only the third color component

The similarity is calculated using at least two types of similarity components.

22. In the image processing method described in claim 20,

The first direction group includes directions in which color information of the same color component is arranged at the first pixel interval.

The second direction group includes directions in which color information of the same color component is arranged at the second pixel interval.

23. In the image processing method according to any one of claims 18 to 22, The first pixel interval is longer than the second pixel interval.

24. The image processing method according to any one of claims 18 to 22, wherein the first pixel interval is approximately 3 pixel intervals,

The second pixel interval is approximately two pixel intervals.

25. In the image processing method according to any one of claims 14 to 24, both the first direction group similarity calculation procedure and the second direction group similarity calculation procedure are subject to image processing. The similarity is calculated including not only the similarity calculated for the processing target pixel but also the similarity calculated for the peripheral pixels of the processing target pixel.

26. In the image processing method according to any one of claims 14 to 25, in the first image, a plurality of pixels are arranged in a triangular lattice.

27. The image processing method according to any one of claims 14 to 25, wherein the first image is represented by first to third color components,

The first to third color components are distributed at a uniform pixel density.

28. In the image processing method according to claim 15 or 19,

When the first image is represented by first to third color components, the color information generating step includes: providing the pixel having the first color component with the color information of the second color component and Z or the third color component. Generating,

29. In the image processing method according to claim 15 or 19,

The color information generating step generates color information of a luminance component different from the color information of the first image.

30. In the image processing method according to claim 15 or 19,

The color information generating step generates color information of a color difference component different from the color information of the first image.

31. In the image processing method according to claim 30,

When the first image is represented by first to third color components,

The color information generation procedure includes:

1) The color difference component between the first color component and the second color component

2) The color difference component between the second color component and the third color component

3) Color difference component between the third color component and the first color component

The color information of the three types of color difference components is generated.

3 2. An image processing method,

An interpolation procedure of interpolating the first color component to a pixel where the first color component is missing, using the acquired color information of the first image;

The interpolation step calculates average information of the first color component by using color information of an area including a pixel whose first color component is second closest to the interpolation target pixel of the first image, The interpolation is performed.

3 3. In the image processing method described in claim 32,

In the interpolation procedure, average information of the first color component is obtained according to the similarity strength determined in the similarity determination procedure.

3 4. An image processing method,

Multiple images represented by multiple color components and having color information of one color component per pixel An image acquisition procedure for acquiring a first image in which the elements are arranged in a rectangular grid,

A color information generating step of generating color information of a color component different from the color information of the first image by weight-adding the acquired color information of the first image with a variable coefficient value of zero or more;

An output step of outputting a second image using the generated color information, wherein the color information generation step includes: for a pixel to be processed of the first image, a color different from a color component of the pixel The color information in the area including the pixel whose component is the second closest is weighted and added.

35. In the image processing method described in claim 34,

In the color information generation procedure, the coefficient value of the weighted addition is made variable according to the similarity strength determined in the similarity determination procedure.

36. In the image processing method described in claim 34 or 35,

When the first image is represented by first to third color components, and a pixel having the first color component of the first image is a processing target pixel,

The color information generation procedure weights and adds color information in an area including the pixel to be processed, the pixel whose second color component is second closest, and the pixel whose third color component is second closest. .

37. The image processing method according to any one of claims 34 to 36, wherein after the color information generation step and before the output step, the first image generated in the color information generation step The image processing apparatus further includes a correction procedure for correcting color information of a color component different from the color information by a filter process including a predetermined fixed filter coefficient.

38. In the image processing method described in claim 37,

The filter coefficient includes positive and negative values.

3 9. An image processing method,

A color difference generation procedure for generating color information of a color difference component between a first color component and a second color component using the color information of the first image;

An output step of outputting a second image using the color information of the generated color difference component,

The color-difference generating step includes: using color information of a pixel having a second color component that is at least second closest to a pixel having a first color component of the first image; Generate

40. The method of claim 39, wherein:

The color difference generation step includes: for a processing target pixel having a first color component of the first image,

1) The color information of the first color component of the pixel and

2) The average information of the color information of the second color component in the region including the pixel whose second color component is second closest to the pixel

The color information of the color difference component is generated based on

41. In the image processing method according to claim 39 or 40,

The color difference generating step further includes generating color information of the color difference component based on curvature information of a second color component for the pixel to be processed.

42. The image processing method according to any one of claims 39 to 41, further comprising a similarity determination procedure for determining the strength of similarity in at least three directions,

The color difference generation step generates color information of the color difference component according to the similarity strength. I do.

43. In the image processing method according to any one of claims 32 to 42, in the output step, the second image is output at the same pixel position as the first image.

4 4. An image processing method,

An image acquisition procedure of acquiring a first image represented by first to third color components, in which a plurality of pixels having color information of one color component in one pixel are uniformly distributed,

An output step of outputting a second image using the generated color information, wherein the color information generation step includes, for all pixels of the first image, color information of first to third color components. Is always weighted and added at a uniform color component ratio.

45. In the image processing method of claim 44,

The method further includes a similarity determination step of determining the degree of similarity in a plurality of directions, wherein the color information generation step includes the step of: Make it variable.

46. In the image processing method according to claim 44 or 45,

In the first image, a plurality of pixels are arranged in a triangular lattice.

47. The image processing method according to any one of claims 44 to 46, wherein after the color information generation step and before the output step, the first image generated in the color information generation step The image processing apparatus further includes a correction procedure for correcting color information of a color component different from the color information by a filter process including a predetermined fixed filter coefficient.

48. In the image processing method of claim 47, The filter coefficients include positive and negative values.

4 9. An image processing method,

An image acquisition procedure for acquiring a first image composed of a plurality of pixels represented by three or more types of color components and having color information of one color component per pixel;

A color information generating step of generating color information of a luminance component and color information of at least three types of color difference components using the color information of the obtained first image;

An output step of outputting a second image using the color information of the luminance component and the color information of the color difference component generated in the color information generation procedure.

50. The method of claim 49, wherein:

A conversion step of using the color information of the luminance component and the color information of the at least three types of color difference components to convert to color information of three types of color components;

The output step outputs a second image using the color information of the three types of color components converted in the conversion step.

51. In the image processing method according to claim 49 or 50,

The color information of the luminance component and the color information of the color difference component generated in the color information generation procedure are color information of components different from the three or more types of color components of the first image.

52. The image processing method according to any one of claims 49 to 51, wherein the first image is represented by first to third color components, and a plurality of pixels are uniformly distributed in color. The color information generation procedure is as follows:

1) The color information of the luminance component in which the color component ratio of the first to third color components is 1: 1: 1,

2) color information of a color difference component between the first color component and the second color component;

3) color information of a color difference component between the second color component and the third color component;

4) Color information of the color difference component between the third color component and the first color component

Generate

53. The image processing method according to any one of claims 49 to 52, further comprising: a similarity determination step of determining a degree of similarity in a plurality of directions, wherein the color information generation step includes: Color information of the luminance component and color information of the at least three types of color difference components are generated according to the similarity strength determined in the sex determination procedure.

54. In the image processing method according to any one of claims 49 to 53, in the first image, a plurality of pixels are arranged in a triangular lattice.

5 5. An image processing method,

A color difference generation procedure for generating color information of at least three types of color difference components using the obtained color information of the first image;

A correction procedure of performing a correction process on the color information of each of the generated color difference components; and an output procedure of outputting a second image using the corrected color information of the color difference components.

56. In the image processing method according to claim 55,

The first image is represented by first to third color components,

The color difference generation procedure includes:

1) color information of a color difference component between the first color component and the second color component,

2) color information of a color difference component between the second color component and the third color component,

3) Color information of the color difference component between the third color component and the first color component

Generate

57. The method of claim 55, wherein:

The first image is represented by first to third color components, The color difference generation procedure includes:

Using the color information of the first image, generating color information of a luminance component different from the color information of the first image,

1) color information of a color difference component between the first color component and the luminance component,

2) color information of a color difference component between the second color component and the luminance component,

3) color information of a color difference component between the third color component and the luminance component;

Generate

58. The method of claim 57, wherein:

In the first image, the first to third color components are evenly distributed to a plurality of pixels, and the color difference generation step includes: as a luminance component, a color component ratio of the first to third color components. The color information of the luminance component composed of 1: 1: 1 is generated.

59. In the image processing method according to any one of claims 44 to 58, in the output step, the second image is output at the same pixel position as the first image.

60. A computer-readable computer program product has an image processing program for causing a computer to execute the procedure of the image processing method described in any one of claims 1 to 59.

61. A computer program product according to claim 60 is a recording medium on which the image processing program is recorded.

6 2. An image processing apparatus,

A control device for executing the procedure of the image processing method according to any one of claims 1 to 59 is provided.