JP6938282B2 - Image processing equipment, image processing methods and programs - Google Patents

Description

The present invention relates to an image processing device, an image processing method, and a program for reducing noise contained in image data.

In imaging devices such as digital still cameras and digital video cameras, dark current noise, thermal noise, shot noise, etc. are generated due to the characteristics of the imaging element and circuit, and random noise (random noise) is mixed in the image data obtained by imaging. do. In recent years, as the image sensor has become smaller and the number of pixels has increased, the pixel pitch has been minimized, so that noise is more noticeable. In particular, shot noise is remarkably generated in image data obtained by shooting with high sensitivity, which is a major factor in image quality deterioration. Therefore, various methods for achieving high image quality by reducing random noise have been proposed.

The most principle method of removing shot noise is a method of averaging a plurality of image data obtained by continuously shooting the same subject under the same shooting conditions. Based on this method, with respect to the reference points in the image data obtained by continuous shooting, a plurality of points included in the learning area of another image are referred to, and a plurality of points are referred to according to the degree of similarity with the window including the reference points. A method for removing noise by weighted averaging has been proposed (see Patent Document 1). In addition, a method of reducing shot noise by synthesizing a plurality of image data obtained by continuous shooting with similar pixels, and further performing noise reduction processing on the composite image data by utilizing the noise reduction amount. Has been proposed (see Patent Document 2).

Special Table 2007-536662 Japanese Unexamined Patent Publication No. 2014-236226

According to the method described in Patent Document 1, noise can be reduced while leaving the texture structure. However, since the edge rattling is also treated as one of the texture structures, the method described in Patent Document 1 may leave the edge rattling. Further, in the method described in Patent Document 2, although the noise reduction amount of the entire composite image data can be made uniform, the edge rattling is not considered, and as a result, the edge rattling is left. ..

Therefore, an object of the present invention is to suppress rattling of edges that occur in an output image in a noise reduction process performed using a plurality of images obtained by continuous shooting.

The image processing apparatus according to the present invention uses one image as a reference image, an image other than the one image as a reference image, and synthesizes a reference image with the reference image among a plurality of images obtained by photographing the same subject. The first noise reduction means for generating the composite image and executing the first noise reduction processing for reducing the noise contained in the reference image, and whether each pixel of the reference image belongs to the edge region or the non-edge region. An isotropic filter is applied to the area determination means for determining the image and the pixels in the composite image corresponding to the edge area of the reference image, and the pixels in the composite image corresponding to the non-edge area of the reference image are equal to It is characterized by comprising a second noise reduction means for performing a second noise reduction process for reducing noise included in the composite image by applying a square filter.

According to the present invention, it is possible to suppress the rattling of edges generated in the output image in the noise reduction processing performed by using a plurality of images obtained by continuous shooting.

It is a block diagram which shows the hardware structure of the image processing apparatus in 1st Embodiment. It is a block diagram which shows the logical structure of the image processing apparatus in 1st Embodiment. It is a flowchart which shows the flow of the image processing executed by the image processing apparatus which concerns on 1st Embodiment. It is a figure for demonstrating the color filter of a Bayer array. It is a flowchart which shows the synthesis process of step S302. It is a figure for demonstrating the synthesis process of step S302. It is a flowchart which shows the area determination process of step S303. It is a figure for demonstrating the local area which is the processing unit of area determination processing. It is a flowchart which shows the noise reduction process for each area of step S304. It is a figure for demonstrating the filter used for the edge direction determination. It is a figure for demonstrating the anisotropic filter used for the noise reduction processing by area. It is a figure for demonstrating the effect of 1st Embodiment.

Hereinafter, embodiments of the present invention will be described in detail. It should be noted that the components described in this embodiment show embodiments as examples of the present invention, and the scope of the present invention is not necessarily limited to them.

[Embodiment 1]
The configuration of the image processing apparatus according to the present embodiment will be described with reference to FIG. FIG. 1 is a block diagram showing a hardware configuration of the image processing apparatus according to the first embodiment. As shown in FIG. 1, the image processing apparatus 100 according to the present embodiment includes a CPU 101, a RAM 102, an HDD 103, a general-purpose interface (I / F) 104, a monitor 108, and a main bus 109. The general-purpose I / F 104 connects an image pickup device 105 such as a camera, an input device 106 such as a mouse and a keyboard, and an external memory 107 such as a memory card to the main bus 109.

The CPU 101 realizes various processes by expanding and executing various software (computer programs) stored in the HDD 103 in the RAM 102. In the present embodiment, as one of the computer programs, an image processing application (hereinafter, simply referred to as an image processing application) is deployed in the RAM 102. When the image processing application is started by the CPU 101, the user interface (UI) screen of the image processing application is displayed on the monitor 108.

Various data stored in the HDD 103 and the external memory 107, instructions from the input device 106, and the like are transferred to the RAM 102. Further, the image data captured by the image pickup apparatus 105 is transferred to the RAM 102. Here, it is assumed that the RAW image data is transferred as the image data. Further, the RAW image data stored in the HDD 103 or the external memory 107 may be transferred to the RAM 102. Further, the RAW image data received from the server device may be transferred to the RAM 102 via a network (not shown). In the following description, the image data may be simply referred to as an image. The image data and the like stored in the RAM 102 are subjected to various calculations based on commands from the CPU 101 according to the processing of the image processing application. The calculation result is displayed on the monitor 108 or stored in the HDD 103 or the external memory 107.

In the above configuration, in the present embodiment, a plurality of image data are input to the image processing application based on a command from the CPU 101, and image data with reduced noise is generated and output.

FIG. 2 is a diagram showing a logical configuration of the image processing device 100 according to the present embodiment. As shown in FIG. 2, the image processing device 100 includes an input unit 201, a storage unit 202, a processing unit 203, and an output unit 204. The processing unit 203 includes a synthesis processing unit 205, an area determination processing unit 206, a noise reduction processing unit 207, and a development processing unit 208.

The input unit 201 inputs data such as a plurality of image data, image processing parameters, and information indicating noise characteristics for each ISO sensitivity (hereinafter, referred to as noise characteristic information). Based on the instruction from the CPU 101, the image data processed by the processing unit 203, the image data stored in the storage unit 202, and the like are output to the monitor 108, the HDD 103, and the like. The plurality of image data includes reference image data that is a noise reduction target, and one or more reference image data in which the same subject as the reference image data is continuously photographed. The image processing parameters are parameters used in the composition processing, the area determination processing, and the noise reduction processing described later. The noise characteristic information is a noise dispersion value calculated in advance for each ISO sensitivity. The input unit 201 inputs these data from the image pickup apparatus 105, the HDD 103, the external memory 107, and the like based on the instruction from the CPU 101. Further, the image data captured by the image pickup device 105 may be directly input from the image pickup device 105 to the input unit 201. The image processing parameters and noise characteristic information may be set by the user. For example, a UI screen for setting such information may be displayed on the monitor 108, and the value set by the user via the UI screen may be stored in the HDD 103 or the external memory 107.

Based on the instruction from the CPU 101, the storage unit 202 stores a plurality of image data, image processing parameters, image data after image processing, and the like input by the input unit 201 in the RAM 102, HDD 103, external memory 107, and the like.

Based on the instruction from the CPU 101, the processing unit 203 acquires image data, image processing parameters, noise characteristic information, and the like from the input unit 201, and performs image processing on the image data. The processing unit 203 may read the data from the storage unit 202. The processing unit 203 outputs the image data after image processing to the storage unit 202 and the output unit 204. The details of the image processing in the processing unit 203 will be described later.

Based on the instruction from the CPU 101, the output unit 204 outputs the image data processed by the processing unit 203, the image data stored in the storage unit 202, and the like to the monitor 108, the HDD 103, and the like. The data output destination is not limited to these, and data may be output to, for example, an external memory 107 connected to the general-purpose I / F 104, or a server device or printer (not shown).

(Main processing flow)
Next, the details of each process of the image processing apparatus 100 will be described with reference to the flowchart of FIG. FIG. 3 is a flowchart showing the flow of image processing executed by the image processing apparatus 100 according to the first embodiment. The process shown in FIG. 3 is realized by the CPU 101 expanding the program (image processing application) stored in the HDD 103 into the RAM 102 and executing the program. Of course, a dedicated processing circuit or the like that plays a part or all of the components shown in FIG. 2 may be provided, and a part or all of the components shown in FIG. 2 may be realized by hardware.

In step S301, the input unit 201 inputs a plurality of image data, image processing parameters, noise characteristic information, and a filter used for edge direction determination (hereinafter, referred to as an edge direction determination filter) and stores them in the storage unit 202. .. In the present embodiment, the image data input by the input unit 201 is assumed to be RAW image data taken by an imaging device having a Bayer array color filter. As shown in FIG. 4, the color filter of the bayer array includes a row in which green (G) pixels and red (R) pixels are alternately repeated, and blue (B) pixels and green (G) pixels. This is a filter in which rows in which are repeated alternately are arranged alternately. However, the scope of application of this embodiment is not limited to image data taken by an image pickup apparatus having a Bayer array color filter. When generating color image data from RAW image data, any image data capable of noise reduction processing and pixel interpolation can be applied to image data captured by an imaging device having an arbitrary color filter. The image data to be input is the reference image data to be noise-reduced and one or more reference image data obtained by continuously shooting the same subject as the reference image data. In the following, among the image data obtained by continuous shooting, the first image will be referred to as reference image data, and the others will be referred to as reference image data. Further, the size of each of the plurality of RAW image data input by the input unit 201 is a width X pixel and a height Y pixel, and the input unit 201 is assumed to input N pieces of reference image data. Further, in each image data, the coordinates (pixel positions) of the upper left pixel are used as a reference (that is, (0,0)), and the coordinates of each pixel are used as (x, y). However, x and y are integers that satisfy 0 ≦ x <X and 0 ≦ y <Y.

In step S302, the synthesis processing unit 205 synthesizes the reference image data input by the input unit 201 in step S301 and all the reference image data to generate the composite image data (first noise reduction process). I do. The details of the synthesis process will be described later.

In step S303, the area determination processing unit 206 performs an area determination process on the reference image data using the reference image data input by the input unit 201 in step S301 and all the reference image data. The details of the area determination process will be described later.

In step S304, the noise reduction processing unit 207 performs region-specific noise reduction processing (second noise reduction processing) on the composite image data generated in step S302 based on the result of the region determination processing in step S303. .. The details of the noise reduction processing for each region will be described later.

In step S305, the development processing unit 208 performs development processing on the composite image data that has been noise-reduced in step S304. The developed image data is stored in the storage unit 202. Here, the development process includes a general correction process for correcting the output image so that it looks suitable. For example, demosaic processing for interpolating pixels, edge enhancement for increasing sharpness, gamma correction for correcting brightness, and color correction for increasing vividness are included. Since the details of the correction process are not the main focus of the present embodiment, the description thereof will be omitted. In step S306, the output unit 204 reads out the image data developed in step S305 from the storage unit 202 and outputs it.

Hereinafter, the synthesis process in step S302, the area determination process in step S303, and the area-specific noise reduction process in step S304 will be described.

(Composite processing)
FIG. 5 is a flowchart showing the synthesis process of step S302. Note that FIG. 5 shows a compositing process performed on one pixel of interest in the reference image data. Therefore, when performing the compositing process for all the pixels of the reference image data, the process shown in FIG. 5 (more specifically, the process of steps S502 to S514) may be repeatedly executed for all the pixels.

In step S501, the synthesis processing unit 205 initializes the synthesis candidate pixel list for recording the pixels that are candidates for synthesis, and sets the block size S1, the reference block size S2, the threshold value th1, the number of composite pixels Num, and the like. Enter the image processing parameters. In the present embodiment, the number N of a plurality of reference image data acquired by continuous shooting is set as the number of composite pixels Num.

In step S502, the synthesis processing unit 205 determines the block of interest centered on the pixel of interest. Here, the pixel of interest is R. FIG. 6A shows the reference image data 601. For example, when the size S1 = 5, a 5 × 5 rectangular area centered on the pixel of interest 602 is set as the block of interest 603.

In step S503, the synthesis processing unit 205 selects one reference image data to be processed from the plurality of reference image data.

In step S504, the synthesis processing unit 205 determines the search area from the reference image data of the processing target selected in step S503. The size of the search area is the same as the reference block size S2. Further, the reference block size S2 is the same as the block size S1 of interest. FIG. 6B shows reference image data 604. For example, when the size S2 = 5, a 5 × 5 rectangular area centered on the pixel 605 having the same coordinates as the pixel of interest 602 is the search area 606. Is set as.

In step S505, the synthesis processing unit 205 selects one pixel of the same color as the pixel of interest (pixel shown by the diagonal line in FIG. 6B) as a reference pixel from the search region determined in step S504.

In step S506, the synthesis processing unit 205 determines a reference block centered on the reference pixel selected in step S505. For example, when the size S2 = 5, and the pixel 607 is selected as the reference pixel, the reference block is a 5 × 5 rectangular area centered on the reference pixel 607 as shown in FIG. 6 (c). It is set as 608.

In step S507, the synthesis processing unit 205 calculates the sum (SAD) of the difference absolute values of the pixel values for each pixel for the block of interest determined in step S502 and the reference block determined in step S506. SAD is an abbreviation for Sum of Absorbed Difference.

In step S508, the synthesis processing unit 205 compares the SAD calculated in step S507 with the threshold value th1 input in step S501. When SAD <th1 (YES in step S508), the synthesis processing unit 205 adds the reference pixel selected in step S505 to the synthesis candidate pixel list (step S509). If SAD <th1 is not satisfied (NO in step S508), the process proceeds to step S510. In step S510, the synthesis processing unit 205 determines whether or not all the reference pixels in the reference block have been selected, that is, whether or not the processing has been completed for all the reference pixels in the reference block (step S510).

If the processing is not completed for all the reference pixels in the reference block (NO in step S510), the processing returns to step S505. On the other hand, when the processing is completed for all the reference pixels in the reference block (YES in step S510), the synthesis processing unit 205 has selected all the reference image data, that is, the processing is completed for all the reference image data. It is determined whether or not (step S511).

If the processing is not completed for all the reference image data (NO in step S511), the processing returns to step S503. On the other hand, if the processing is completed for all the reference image data (YES in step S511), the processing proceeds to step S512.

In step S512, the composition processing unit 205 sorts the composition candidate pixel list in ascending order of SAD. Then, in step S513, the synthesis processing unit 205 selects the pixels sorted in step S512 from the one with the smallest SAD to the Numth pixel. Further, in step S514, the synthesis processing unit 205 calculates the average value of the pixel of interest and the Num pixels selected in step S513 by the following equation (1).

Here, I ₀ is the pixel value of the pixel of interest, and I _i is the pixel value of the i-th pixel (i indicates the order selected in step S513). By performing the processing of steps S502 to S514 for all the pixels of the reference image data, the composite image data in which the pixel values derived by the above equation (1) are set for each pixel is generated.
Instead of the simple average as described above, for example, the reciprocal of the SAD of each selected pixel may be used as a weight, and the weighted average may be calculated by the equation (2) to calculate the average value.

Here, w _i is the weight of the _i -th pixel, and w i = 1 / SAD _i .

In the present embodiment, the difference absolute value sum (SAD) is used to determine the composite candidate pixel, but the method is not limited to this and any method can find a pixel having a high degree of similarity to the pixel of interest. Any method will do. For example, a case where the absolute difference value or the difference rate of the average value of the block of interest and the reference block is equal to or less than a predetermined threshold value may be determined as the synthesis candidate pixel. Alternatively, the absolute difference between the SAD and the average value may be used in combination.

Further, if the number of reference pixels added to the composition candidate pixel list in step S509 is less than Nu, the process of step S513 may be skipped. Then, in step S514, the equation (1) may be applied to all the reference pixels added to the composition candidate pixel list. Further, when the number of reference pixels added to the composition candidate pixel list is less than Nu, especially when the number of reference pixels is 0, the value of th1 is increased, and then the process of step S302 is performed again (the process of step S302 is performed again. (Synthesis processing) may be executed.

(Area judgment processing)
Next, the area determination process in step S303 will be described with reference to FIG. 7. FIG. 7 is a flowchart showing the area determination process in step S303.

In step S701, the area determination processing unit 206 initializes the area determination map that stores the result of the area determination, image processing parameters such as the size S3 of the local area used for the area determination, the threshold values th2 and th3, and noise characteristic information. Enter. The area determination map has the same size as the reference image data. ^{Further, here, the noise variance σ 2} according to the ISO sensitivity at the time of shooting is input as the noise characteristic information.

In step S702, the area determination processing unit 206 selects the pixel of interest to be processed from the reference image data. Then, in step S703, the area determination processing unit 206 derives the edge strength E of the pixel of interest selected in step S702. Hereinafter, a method for deriving the edge strength E will be described.

First, the area determination processing unit 206 determines a local area including a pixel of interest and a peripheral pixel. The size S3 of the local region is determined based on the parameters input in step S301. Here, S3 = 5, and the case where the local region is 5 × 5 pixels will be described as an example with reference to FIG. In FIG. 8A, the black pixel 801 represents the pixel of interest, the shaded pixel 802 represents the peripheral pixel, and the region 803 surrounded by the thick line represents the local region. The shape of the local region is not limited to the shape shown in FIG. 8A, and may be any shape, for example, the shape shown in FIG. 8B.

Next, the area determination processing unit 206 calculates the variance value V in the local area for each RGB using the equation (3).

Here, M indicates the number of pixels included in the local region, and r, g, and b indicate RGB values, respectively.

は局所領域に含まれる画素値の平均値を示している。ｉは局所領域における各画素を識別するためのインデックス番号であり、０≦ｉ≦Ｍ−１を満たす整数である。領域判定処理部２０６は、これらの分散値Ｖを、ステップＳ７０１で入力した撮影時のＩＳＯ感度に応じたノイズ分散値σ²で割ることで分散値の比率を算出し、これをエッジ強度Ｅとする。具体的には、式（４）によりエッジ強度Ｅが算出される。エッジ強度Ｅは値が大きいほど、鮮鋭度が高く、逆に小さいほど鮮鋭度が低いことを示す指標である。

Indicates the average value of the pixel values contained in the local area. i is an index number for identifying each pixel in the local region, and is an integer satisfying 0 ≦ i ≦ M-1. The area determination processing unit 206 calculates the ratio of the dispersion values by dividing ^{these dispersion values V by the noise dispersion value σ 2} according to the ISO sensitivity at the time of shooting input in step S701, and sets this as the edge strength E. do. Specifically, the edge strength E is calculated by the equation (4). The larger the value of the edge strength E, the higher the sharpness, and conversely, the smaller the value, the lower the sharpness.

Area determination unit 206, the calculated edge intensity E _R, E _G, the maximum value of E _B, obtained as edge intensities E.

In step S704, the area determination processing unit 206 compares the edge strength E derived in step S703 with the threshold value (first determination value) th2 input in step S701. If the edge strength E is smaller than the first determination value (NO in step S704), the process proceeds to step S711. When the edge strength E is equal to or higher than the first determination value (YES in step S704), the area determination processing unit 206 selects one reference image data to be processed from the plurality of reference image data. Then, the area determination processing unit 206 selects a pixel having the same coordinates as the pixel of interest selected in step S702 from the selected reference image data as a reference pixel. Then, the area determination processing unit 206 calculates SAD for both the local area centered on the pixel of interest in the reference image data and the local area centered on the reference pixel in the reference image data to be processed. .. It is assumed that the size of the local region at the time of calculating SAD is the same size as at the time of calculating the edge strength, that is, 5 × 5 pixels.

In step S707, the area determination processing unit 206 compares the SAD calculated for the two local areas in step S706 with the threshold value (second determination value) th3 input in step S701. If the SAD is equal to or greater than the second determination value (NO in step S707), the region determination processing unit 206 determines that the two local regions are not similar. Then, the process proceeds to step S710. If the SAD is smaller than the second determination value (YES in step S707), the region determination processing unit 206 determines that the two local regions are similar, and the processing is completed for the reference pixels of all the reference image data. Whether or not it is determined (step S708). If the processing is not completed for the reference pixels of all the reference image data (NO in step S708), the processing returns to step S705. If the processing is completed for the reference pixels of all the reference image data (YES in step S708), the processing proceeds to step S709.

In step S709, the area determination processing unit 206 determines that the pixel of interest selected in step S702 belongs to the edge region corresponding to the non-moving body edge (hereinafter, referred to as the non-moving body edge region), and determines that the pixel of interest belongs to the area determination map. "1" is output to the pixel having the same coordinates as the pixel of interest. In step S710, the area determination processing unit 206 determines that the pixel of interest selected in step S702 is a pixel belonging to the edge region corresponding to the moving object edge (hereinafter referred to as the moving object edge region), and determines that the pixel of interest is a pixel of interest in the area determination map. "2" is output to the pixel with the same coordinates as. In step S711, the area determination processing unit 206 determines that the pixel of interest selected in step S702 belongs to an area other than the edge area (referred to as a flat area or a non-edge area), and sets the pixel of interest in the area determination map. "0" is output to the pixels with the same coordinates.

In step S712, the area determination processing unit 206 determines whether or not the processing of steps S703 to S711 has been completed for all the pixels of the reference image data. If the processing is not completed for all the pixels of the reference image data (NO in step S712), the processing returns to step S702. When the processing is completed for all the pixels of the reference image data (YES in step S712), the area determination processing unit 206 ends the area determination processing.

In the present embodiment, the dispersion ratio is used for calculating the edge strength E, but the present invention is not limited to this, and a Sobel filter, a Laplacian filter, or the like may be used. That is, the calculation method of the edge strength E may be any calculation method as long as the value corresponding to the edge strength can be calculated.

(Noise reduction processing by area)
Next, with reference to FIG. 9, the area-specific noise reduction processing in step S304 will be described. FIG. 9 will be described with reference to a flowchart showing the noise reduction processing for each region in step S304.

In step S901, the noise reduction processing unit 207 inputs the area determination map generated in step S303, image processing parameters such as filter sizes S4 and S5 used for noise reduction processing, and the edge direction determination filter.

In step S902, the noise reduction processing unit 207 selects the pixel of interest to be processed from the reference image data input in step S301. In step S903, the noise reduction processing unit 207 determines whether the pixel of interest is a non-moving body edge (= “1”) based on the area determination map input in step S901, that is, the pixel of interest belongs to the non-moving body edge region. Is determined. If it is a non-moving body edge (YES in step S903), the process proceeds to step S905. If it is not a non-moving body edge (NO in step S903), the noise reduction processing unit 207 determines whether the pixel of interest is the moving body edge (= “2”) based on the area determination map input in step S901, that is, the pixel of interest. Determines if is a pixel belonging to the moving body edge region. If it is a moving body edge (YES in step S904), the process proceeds to step S906, and if it is not a moving body edge (NO in step S904), the process proceeds to step S907.

In step S905, the noise reduction processing unit 207 determines the direction of the edge with respect to the pixel of interest, and applies an anisotropic filter along the direction of the edge to the pixel of interest. Specifically, first, the noise reduction processing unit 207 applies a plurality of edge direction determination filters (for example, filters as shown in FIGS. 10A to 10D) prepared in advance to the pixels of interest one by one. Apply. In FIG. 10, the pixels painted in black indicate the pixels of interest, and the pixels in which the circles are drawn and the pixels in which the triangles are drawn indicate the pixels for determining the direction of the edge, respectively. Next, the noise reduction processing unit 207 calculates the sum of the pixel values of the pixels in the reference image data, which spatially corresponds to the pixels on which the circles of the edge direction determination filter are drawn. Similarly, the noise reduction processing unit 207 calculates the sum of the pixel values of the pixels in the reference image data, which spatially corresponds to the pixels on which the triangles of the edge direction determination filter are drawn. Then, the noise reduction processing unit 207 obtains the brightness difference on a straight line by calculating the difference between the sums of the calculated pixel values. Then, the noise reduction processing unit 207 sets the pixel direction in which the circle of the edge direction determination filter having the largest brightness difference is drawn (or the pixel direction in which the triangle is drawn) as the edge direction. Finally, the noise reduction processing unit 207 applies an anisotropic filter of size S3 extending in the direction of the edge to the pixel of interest. In FIG. 11A, when the direction indicated by the edge direction determination filter 1101 (the diagonally upward right direction indicated by the thick arrow in the figure) is determined as the direction of the edge in the pixel of interest 1102 (here, R). The state of is shown. FIG. 11B shows the anisotropic filter applied at this time. Here, the image data 1103 is the composite image data obtained by synthesizing the reference image data and all the reference image data in step S302. As shown in FIG. 11B, the focus pixel 1104 on the composite image data 1103 corresponding to the focus pixel 1102 is not extended in the same direction as the thick line arrow in FIG. 11 (a) (diagonally upward to the right). The isotropic filter 1105 is applied. Anisotropy filters are prepared for each of R, G, and B. The anisotropic filter 1105 shown in FIG. 11B is an anisotropic filter for R. By applying the anisotropic filter 1105 to the pixel of interest 1102, the pixel value of the R pixel in the broken line frame in the drawing is obtained. Is smoothed. Further, as the coefficient set in the anisotropic filter, any value may be set as long as it is a value that smoothes the pixel value in the edge direction.

In step S906, the noise reduction processing unit 207 determines the direction of the edge with respect to the pixel of interest in the same manner as in step S905, and applies an anisotropic filter along the direction of the edge. The anisotropic filter applied in step S906 (hereinafter referred to as a second anisotropic filter) has the same size as the anisotropic filter applied in step S905 (hereinafter referred to as a first anisotropic filter). Only different. More specifically, the second anisotropic filter differs from the first anisotropic filter only in the size in the edge direction. The size S5 of the second anisotropic filter> the size S4 of the first anisotropic filter. This is because it is difficult to find similar pixels in the image data of the moving body edge as compared with the non-moving body edge, so that the edge rattling tends to remain. Therefore, the size S5 of the anisotropic filter applied to the moving body edge region is set large. At that time, the size of the edge direction determination filter applied to the moving body edge region is also set large according to the size of the anisotropic filter applied to the moving body edge region after the size is increased. In FIG. 11C, the pixel value of the area determination map corresponding to the pixel of interest 1106 is “2 (moving object edge)”, and the edge direction of the pixel of interest is the horizontal direction (horizontal direction in the figure). An example of a second anisotropic filter in the case of determination is shown. Note that FIG. 11 (c) shows an anisotropic filter applied when the pixel of interest is R, as in FIG. 11 (b). As shown in FIG. 11 (c), the second anisotropic filter applied to the moving body edge region is more than the first anisotropic filter applied to the non-moving body edge region shown in FIG. 11 (b). The size in the edge direction is set large. A part of the second anisotropic filter may be used as the first anisotropic filter, and completely different filter coefficients are set for the second anisotropic filter and the first anisotropic filter. May be done.

In step S907, the noise reduction processing unit 207 applies an isotropic filter to the pixel of interest. This process is performed in order to suppress the SN step that may occur at the boundary between the edge region and the flat region by applying an anisotropic filter to the edge region. However, since the noise in the flat region has already been reduced in the synthesis process in step S302, the noise reduction effect of the isotropic filter is set to be weaker than that in the edge region. In this embodiment, a Gaussian filter is used as the isotropic filter.

In step S908, the noise reduction processing unit 207 determines whether or not the processing of steps S903 to S907 has been completed for all the pixels. When the processing is completed for all the pixels (YES in step S908), the noise reduction processing unit 207 ends the area-specific noise reduction processing. If the processing is not completed for all pixels (NO in step S908), the processing returns to step S902.

As described above, in the second noise reduction processing (region-specific noise reduction processing), in the present embodiment, when the composite image obtained by the first noise reduction processing (composite processing) is filtered. , Apply different filters to each of the edge and flat areas. In particular, in the present embodiment, the anisotropic filter is applied to the edge region in the composite image. Therefore, according to the present embodiment, it is possible to suppress the rattling of edges that occur in the output image in the noise reduction processing performed using a plurality of images. Further, in the present embodiment, for the edge region of the image of interest, the SAD with the region in the reference image corresponding to the edge region is derived, and the edge region is divided into a moving body edge region and a non-moving body edge region based on the derived SAD. It is further classified into. Then, anisotropic filters of different sizes are applied to the moving body edge region and the non-moving body edge region. As a result, it becomes possible to further suppress the rattling of the edge. FIG. 12 is a diagram for explaining the effect of the present embodiment. FIG. 12 shows an image (FIG. 12 (b)) in which a conventional noise reduction process that does not consider edge rattling is applied to an input image (FIG. 12 (a)), and noise reduction by the present embodiment. An image to which the processing is applied (FIG. 12 (c)) is shown. Focusing on the part surrounded by the broken line ellipse in the figure, it can be seen that the edge remains rattling in the image to which the conventional noise reduction processing is applied. On the other hand, in the image to which the noise reduction processing according to the present embodiment is applied, no rattling is observed at the edges, and it can be seen that the rattling of the edges is suppressed.

In the present embodiment, the case where the number of images obtained by continuous shooting and the number of composite pixels Nu are the same has been described, but an arbitrary number of composite pixels is set so as to satisfy the composite pixel number. If the compositing process is performed, the number of compositing pixels is not limited to those described above.

Further, in the present embodiment, among the images obtained by continuous shooting, the first obtained image is used as the reference image, but an image other than the first obtained image may be used as the reference image. For example, the image that most captures the main subject may be used as the reference image.

Further, in the present embodiment, among the pixels included in the composition candidate pixel list, Num pixels are synthesized in ascending order of SAD between the pixel of interest and the reference pixel, but the present invention is not limited to this. For example, Num pixels may be randomly selected from the composition candidate pixel list and combined.

Further, in the present embodiment, a pixel having the same coordinates as the pixel of interest is selected as the reference image from the reference image, but the present invention is not limited to this. For example, a large displacement may occur in an image obtained by handheld shooting. Therefore, when the input unit 201 inputs an image obtained by handheld shooting, the processes of steps S302 to S306 may be performed after each reference image is aligned with the reference image in step S301.

(Other embodiments)
The present invention supplies a program that realizes one or more functions of the above-described embodiment to a system or device via a network or storage medium, and one or more processors in the computer of the system or device reads and executes the program. It can also be realized by the processing to be performed. It can also be realized by a circuit (for example, ASIC) that realizes one or more functions.

Claims (9)

Of a plurality of images obtained by photographing the same subject, one image is used as a reference image, an image other than the one image is used as a reference image, and a composite image obtained by synthesizing the reference image with the reference image is generated. The first noise reduction means for executing the first noise reduction processing for reducing the noise included in the reference image, and the first noise reduction means.
A region determination means for determining whether each pixel of the reference image belongs to an edge region or a non-edge region, and
An isotropic filter is applied to the pixels in the composite image corresponding to the edge region of the reference image, and the isotropic filter is applied to the pixels in the composite image corresponding to the non-edge region of the reference image. An image processing apparatus comprising the second noise reducing means for performing a second noise reducing process for reducing the noise included in the composite image by applying The area determination means
For the reference image, the edge strength is derived for each local region composed of the pixel of interest and the peripheral pixels of the pixel of interest.
It is determined that the pixel of interest in the local region of the reference image whose derived edge strength is equal to or higher than the first determination value is a pixel belonging to the edge region, and the edge strength is based on the first determination value. The image processing apparatus according to claim 1, wherein a small pixel of interest in the local region of the reference image is determined to be a pixel belonging to the non-edge region. The area determination means
It is determined whether or not each of the local regions of the reference image having the edge intensity equal to or higher than the first determination value is similar to the region in the reference image corresponding to each of the local regions. ,
The pixel of interest in the local region determined to be similar is determined to be a pixel belonging to the non-moving body edge region, and the pixel of interest in the local region determined to be dissimilar belongs to the moving body edge region. The image processing apparatus according to claim 2, wherein the image processing apparatus is determined to be a pixel. The area determination means
The sum of the absolute values of the differences between the local region of the reference image and the region in the reference image corresponding to the local region, whose edge strength is equal to or higher than the first determination value, is derived.
When the derived absolute difference sum is smaller than the second determination value, it is determined that both regions are similar, and when the difference absolute sum is equal to or greater than the second determination value, both regions are determined to be similar. The image processing apparatus according to claim 3, wherein it is determined that they are not similar. The anisotropic filter applied to the pixels in the composite image corresponding to the moving body edge region of the reference image is applied to the pixels in the composite image corresponding to the non-moving body edge region of the reference image. The image processing apparatus according to claim 3 or 4, characterized in that the size in the edge direction is larger than that of the applied anisotropic filter. The area determination means
The edge direction is determined for each pixel belonging to the edge region of the reference image, and the edge direction is determined.
The image processing according to any one of claims 1 to 5, wherein an anisotropic filter in the edge direction determined for each pixel is applied to each pixel belonging to the edge region. Device. The first noise reducing means is
A synthesis candidate pixel to be applied to the one pixel is selected from a block composed of a pixel corresponding to one pixel of the reference image and peripheral pixels of the pixel in each reference image.
Using the composite candidate pixels selected for each pixel of the reference image, the pixel value of each pixel of the composite image is determined.
Claims 1 to 6, wherein the composite candidate pixels are a predetermined number of pixels selected from the blocks in each reference image in descending order of similarity to the one pixel. The image processing apparatus according to any one of the items. Of a plurality of images obtained by photographing the same subject, one image is used as a reference image, an image other than the one image is used as a reference image, and a composite image obtained by synthesizing the reference image with the reference image is generated. The first noise reduction step of executing the first noise reduction processing for reducing the noise included in the reference image, and
A region determination step for determining whether each pixel of the reference image belongs to an edge region or a non-edge region, and
An isotropic filter is applied to the pixels in the composite image corresponding to the edge region of the reference image, and the isotropic filter is applied to the pixels in the composite image corresponding to the non-edge region of the reference image. An image processing method comprising a second noise reduction step of performing a second noise reduction process for reducing the noise included in the composite image by applying the above. A program for causing a computer to function as the image processing device according to any one of claims 1 to 7.

Images