JP4875557B2 - Image processing apparatus, image processing method, and image processing program - Google Patents

Description

  The present invention relates to an image processing apparatus, an image processing method, and an image processing program that perform noise reduction processing on a moving image signal.

  A moving image signal is a signal composed of a plurality of frames (fields) captured at specified time intervals. Generally, each frame (field) has a correlation with a local space in the frame (in the field) and is adjacent to each other. A frame (field) can be assumed as a signal having a correlation with local time between frames (inter-field).

  Such a moving image signal can be obtained mainly by imaging an arbitrary subject with a video camera including an image sensor such as a CCD or a CMOS. Specifically, the subject image formed on the image sensor is output in a predetermined order as a signal photoelectrically converted in units of pixels, and the output analog image signal is amplified by a predetermined gain amount, and then a digital image is obtained by A / D conversion. In general, it is converted into a signal and subjected to predetermined image processing.

  In such a video camera, noise is superimposed on the captured image due to the characteristics of the image sensor. This noise is mainly shot noise due to the nature of photoelectric conversion. Here, the shot noise has an average amplitude proportional to the square root of the image signal value, and generally becomes random noise in the time direction and the spatial direction. This noise appears more conspicuously when the gain amount is increased when the amount of imaged light on the image sensor is insufficient.

  Conventionally, as noise reduction processing for a moving image signal on which random noise is superimposed, intra-frame (intra-field) noise reduction processing using spatial correlation and inter-frame (inter-field) noise reduction processing using temporal correlation Various proposals have been made.

  As the noise reduction processing using the time correlation described above, the cyclic noise reduction processing using the noise-reduced frame as a past frame is known as a method capable of obtaining a large noise reduction amount. However, since this method is premised on the correlation with the past frame, if the same processing is performed on a scene with a large scene change or movement with the past frame, the image of the previous frame will be displayed. There is a problem that afterimages are superimposed on the current frame. On the other hand, if control is performed to reduce the afterimage, there is a problem that a sufficient noise reduction effect cannot be expected.

  As a countermeasure, for example, Patent Document 1 discloses a noise reduction process that uses both intra-frame correlation and inter-frame correlation to improve noise reduction capability even for a scene with large motion.

  This noise reduction system is provided with an image memory for delaying one frame or one field, newly inputted center pixel data, pixel data in the vicinity of this center pixel data, and already recorded in the image memory. Noise-reduced pixel data is output by performing non-linear filter processing on pixel data in the vicinity of the central pixel data in the image data of one frame or one field before noise reduction. Here, the non-linear filter processing means that a large weighting factor is assigned to data of neighboring pixels having a high correlation with the central pixel data value, and conversely, a small weighting factor is assigned to a pixel having a low correlation to perform weighted averaging. Is.

According to the above method, it is possible to perform noise reduction using both intra-frame correlation and inter-frame correlation. In particular, when the captured image is in a stationary state, the number of pixels that contribute to averaging increases because the number of pixels having a large weighting coefficient increases in the previous frame and the previous field used for weighted averaging and the current field pixel. . As a result, it is possible to effectively reduce noise. Further, when there is a large movement in the captured image or when there is a scene change, a larger weighting factor is given to the current field pixel than to the pixel one frame before or one field before, so the weighting is performed mainly on the current field pixel. An average is made. As a result, the amount of noise reduction is smaller than that in a stationary state, but a large noise reduction effect is obtained as compared with cyclic noise reduction processing using pixels at the same position in the current frame and the past frame. It becomes possible.
JP-A-6-62283

  The noise reduction processing using time correlation as in the above prior art is substantially switched to noise reduction processing mainly using the field for scenes and scene changes with large movements. Therefore, the noise reduction amount decreases when the moving region suddenly changes from the still region or immediately after the scene change. Furthermore, a stable noise reduction amount cannot be obtained immediately after returning from the moving region to the stationary region, and a time delay of several frame periods is required before obtaining a stable noise reduction amount. That is, when trying to obtain a large noise reduction amount using a cyclic noise reduction process using time correlation, there arises a problem that the noise amount greatly fluctuates with time depending on the input moving image.

  The present invention has been made in view of the above problems, and is capable of effectively reducing noise and keeping resolution degradation to a minimum while maintaining a stable noise level, and image processing. It is an object to provide a method and an image processing program.

In order to solve the above problems, the present invention employs the following means.
An image processing apparatus according to the present invention is an image processing apparatus that performs noise reduction processing on a frame image or a field image input in time series. The processing target frame image or field image and the processing target frame image or field image Recording means for recording past and future frame images or field images, and first pixel extracting means for extracting a plurality of pixels in a predetermined area in the processing target frame image or field image recorded by the recording means; Second pixel extracting means for extracting a plurality of pixels in an area corresponding to the predetermined area in the past frame image or field image and the future frame image or field image recorded by the recording means; Extracted by the pixel extracting means of First distance calculating means for calculating a spatio-temporal distance between a target pixel in a fixed area, a plurality of pixels in the predetermined area, and a plurality of pixels in an area corresponding to the predetermined area extracted by the second pixel extracting means The pixel value of the pixel of interest in the predetermined area extracted by the first pixel extracting means, the pixel value of the plurality of pixels in the predetermined area extracted by the first pixel extracting means, and the second Second distance calculating means for calculating a distance between pixel values with pixel values of a plurality of pixels in an area corresponding to the predetermined area extracted by the pixel extracting means; and the time calculated by the first distance calculating means. Noise reduction means for reducing noise of the processing target frame image or field image based on a spatial distance and the inter-pixel value distance calculated by the second distance calculation means, That.

  According to the present invention, noise reduction processing of a moving image signal is performed using a spatial correlation within a frame and a temporal correlation between a past frame and a current frame and between a current frame and a future frame. Do. As a result, it is possible to suppress the noise reduction amount from reacting sensitively to fluctuations in time correlation, and to stably reduce noise while suppressing a reduction in resolution.

  An image processing program according to the present invention is an image processing program for causing a computer to perform noise reduction processing of a frame image or a field image input in time series, the processing target frame image or the field image and the processing target frame image. Alternatively, a recording process that records past and future frame images or field images with respect to a field image, and a first pixel that extracts a plurality of pixels in a predetermined region in the processing target frame image or field image recorded by the recording process. Pixel extraction processing; and second pixel extraction processing for extracting a plurality of pixels in a region corresponding to the predetermined region in the past frame image or field image and the future frame image or field image recorded by the recording processing. The above Spatio-temporal of a pixel of interest in the predetermined area extracted by one pixel extraction process, a plurality of pixels in the predetermined area, and a plurality of pixels in an area corresponding to the predetermined area extracted in the second pixel extraction process A first distance calculation process for calculating a distance; a pixel value of a pixel of interest in the predetermined area extracted by the first pixel extraction process; and a value of the predetermined area extracted by the first pixel extraction process. A second distance calculating process for calculating a distance between pixel values of a pixel value of a plurality of pixels and a pixel value of a plurality of pixels in an area corresponding to the predetermined area extracted in the second pixel extracting process; Noise that reduces noise in the processing target frame image or field image based on the spatiotemporal distance calculated by the distance calculation processing and the distance between the pixel values calculated by the second distance calculation processing. Characterized in that to execute a reduction process, to the computer.

  An image processing method according to the present invention is an image processing method for reducing noise of a frame image or a field image input in time series, and includes a processing target frame image or a field image and the processing target frame image or a field image. A recording step of recording past and future frame images or field images, and a first pixel extraction step of extracting a plurality of pixels in a predetermined region in the processing target frame image or field image recorded by the recording step; A second pixel extracting step of extracting a plurality of pixels in a region corresponding to the predetermined region in the past frame image or field image and the future frame image or field image recorded by the recording step; Extracted in the pixel extraction step of A first distance calculating step of calculating a spatiotemporal distance between a target pixel in a fixed region, a plurality of pixels in the predetermined region, and a plurality of pixels in a region corresponding to the predetermined region extracted in the second pixel extracting step; The pixel value of the pixel of interest in the predetermined region extracted in the first pixel extraction step, the pixel values of the plurality of pixels in the predetermined region extracted in the first pixel extraction step, and the second A second distance calculating step of calculating a distance between pixel values with pixel values of a plurality of pixels in a region corresponding to the predetermined region extracted in the pixel extracting step; and the time calculated by the first distance calculating step. A noise reduction step of reducing noise of the processing target frame image or field image based on the spatial distance and the distance between the pixel values calculated by the second distance calculation step. That.

  According to the present invention, it is possible to prevent the noise reduction amount from reacting sensitively to the fluctuation of the time correlation that occurs when reducing the noise of the moving image signal, and to suppress stable deterioration with little time fluctuation while suppressing a decrease in resolution. Reduction is possible.

[First Embodiment]
The image processing apparatus according to the first embodiment of the present invention will be described below with reference to the drawings.
FIG. 1 is a functional block diagram showing the basic configuration of the first embodiment of the present invention, and details will be described below.
The image processing apparatus according to this embodiment includes a switch 101, a memory group (recording unit) including a frame memory 102, a frame memory 103, a switch 104, a switch 105, a frame memory 109, a frame memory 110, and an N-line memory 106. , A noise reduction unit (noise reduction unit) 107, a switch 108, a switch 111, a control unit 112, a block extraction unit (first pixel extraction unit) 113, and a block extraction unit (second pixel extraction unit). 114.

A signal flow in the image processing apparatus having the above configuration will be described below.
A moving image signal captured and digitized by an image sensor of an imaging unit (not shown) is input to the switch 101. Here, the switch 101 is alternately connected to the frame memory 102 or the frame memory 103 every frame period according to a control signal of the control unit 112. Therefore, the moving image signal input to the switch 101 is stored in the frame memory 102 or the frame memory 103 for each frame period.

  The moving image signal may be a monochrome signal or a color signal. In the case of a color signal, even a synchronized signal composed of a plurality of colors per pixel (generally three colors) may be used with a single-plate image sensor. It may be an imaged signal consisting of one color per pixel before synchronization. In the following description, it is assumed that the moving image signal is a single color signal. However, in the case of a color signal, it is necessary to perform the processing described below for each color signal.

The frame memory 102 is connected to the switch 104 and the switch 105. The frame memory 103 is also connected to the switch 104 and the switch 105.
The switch 104 and the switch 105 are connected to the N-line memory 106 and the block extraction unit 114, respectively. Here, the control unit 112 controls the switch 104 and the switch 105 so that the output signals of the switch 104 and the switch 105 are switched every frame period. Specifically, when the frame memory 102 is connected to the N-line memory 106 via the switch 104, the frame memory 103 is connected to the block extraction unit 114 via the switch 105. Conversely, when the frame memory 102 is connected to the block extraction unit 114 via the switch 105, the frame memory 103 is connected to the N line memory 106 via the switch 104.

  The N line memory 106 temporarily stores a predetermined number of pixels above and below the noise reduction processing target pixel from the frame 102 or the frame memory 103. The N line memory 106 is connected to the input of the noise reduction unit 107 via the block extraction unit 113.

  A block extraction unit 113, a block extraction unit 114, and a control unit 112 are connected to the input of the noise reduction unit 107. The noise reduction unit 107 includes a predetermined region pixel of the processing target frame output from the block extraction unit 113 and a temporally future frame image stored in the frame memory 102 or the frame memory 103 output from the block extraction unit 114. Noise reduction processing for a pixel to be processed based on a predetermined area pixel of the image, a predetermined area pixel of a temporally past frame image stored in the frame memory 109 or the frame memory 110, and a control signal from the control unit 112 I do. The calculated noise reduction pixel is output as an output signal. Here, the predetermined areas output from the block extraction unit 113 and the block extraction unit 114 are spatially the same extraction position in each frame.

The output of the noise reduction unit 107 is connected to the switch 108 and the N line memory 106.
The output signal to the N line memory 106 is used to overwrite the processing target pixel value before noise reduction stored in the N frame memory 106 with the noise reduction pixel. This makes it possible to form a recursive filter that uses pixels after noise reduction for the current frame, and thus noise can be more effectively reduced.

  The connection of the switch 108 is switched to the frame memory 109 or the frame memory 110 for each frame period according to a control signal from the control unit 112. As a result, the noise reduction pixels calculated by the noise reduction unit 107 are recorded in the corresponding frame memory 109 or the frame memory 110. Here, the frame memory in which the noise reduction pixels are recorded is a frame memory different from the frame memory in which the past frame extracted by the block extraction unit 114 is stored.

  The switch 111 is switched and connected to the frame memory 109 and the frame memory 110 for each frame period by a control signal of the control unit 112. Here, the control unit 112 associates the switch 108 and the switch 111 and controls switching of each switch. Specifically, when the switch 108 is connected to the frame memory 109, the frame memory 110 is connected to the switch 111. On the other hand, when the switch 108 is connected to the frame memory 110, the frame memory 109 is connected to the switch 111.

  An output signal from the switch 111 is input to the block extraction unit 114. The block extraction unit 114 extracts a plurality of pixels in the peripheral area corresponding to the processing target pixel.

  The control unit 112 outputs a preset target noise reduction amount to the noise reduction unit 107 and also outputs a control signal for interlocking control of the switches 101, 104, 105, 108, and 111 as described above.

FIG. 3 is a schematic diagram illustrating a structure of a three-dimensional block (N _x × N _y × N _t pixels) including a processing target pixel processed by the noise reduction unit 107 and its peripheral pixels. Hereinafter, the definition of the three-dimensional block and the pixel P (r, t) in the area will be described with reference to FIG.

The three-dimensional block processed by the noise reduction unit 107 is N _x × N composed of the processing target pixel P (r ₀ , t ₀ ) and the surrounding area pixel P (r, t ₀ ) of the current frame (time t ₀ ). a current block 302 of _y pixels, the past consists frame (time _t 0 -1) pixels _{P (r 0, t 0 -1} ) and its surrounding area pixel _{P (r, t 0 -1)} N x × N y A future block of N _x × N _y pixels consisting of a past block 301 of pixels, a pixel P (r ₀ , t ₀ +1) of a future frame (time t ₀ +1), and its peripheral region pixel P (r, t ₀ +1) 303. Here, r means a position vector when the origin in each frame is r ₀ , r = (x, y), r ₀ = (x ₀ , y ₀ ), where N _t = 3 However, N _t may be an integer of 2 or more.

In view of the field interlace signal, the past block and the future block do not have pixels in the same spatial position as the current block. However, if an N _x × N _y pixel composed of the processing target pixel P (r ₀ , t ₀ ) and its peripheral region pixel P (r, t ₀ ) is defined as the current block, the past field (time t ₀ −1) A pixel in the future field (time t ₀ +1) is defined as a past block of N _x × N _y pixels composed of the pixel P (r ₀ ′, t ₀ −1) and its peripheral region pixel P (r ′, t ₀ −1). An N _x × N _y pixel composed of P (r ₀ ′, t ₀ +1) and its peripheral region pixel P (r ′, t ₀ +1) can be defined as a future block. In the case of the field, the current block, the past block, and the future block are strictly shifted by one line. However, in the following description, r = r ′, that is, the same position will be described.

FIG. 2 is a detailed block diagram of the noise reduction unit 107, and details thereof will be described based on the definition of the three-dimensional block.
The noise reduction unit 107 includes a past block memory 201, a future block memory 202, a current block memory 203, a distance calculation unit (first distance calculation unit and second distance calculation unit) 204, an inter-frame correlation calculation unit (inter-frame correlation). Amount calculation means) 205, correction coefficient calculation section (correction coefficient calculation means) 206, filter coefficient calculation section (weight coefficient calculation means) 207, filter processing section (weighted average value calculation means) 208, and noise reduction amount calculation section (noise) Reduction amount calculation means) 209.

A signal flow in the noise reduction unit 107 having the above configuration will be described below.
From the block extraction unit 113, the current block 302 including the processing target pixel P (r ₀ , t ₀ ) and its peripheral region N _x × N _y is output and stored in the current block memory 203. The block extraction unit 114 also includes a past consisting of P (r ₀ , t ₀ −1) that is spatially identical to the processing target pixel P (r ₀ , t ₀ ) and its peripheral region N _x × N _y. The block 301 is output and stored in the past block memory 201. Further, from the block extraction unit 114, a future block composed of P (r ₀ , t ₀ +1) and its peripheral region N _x × N _y that are spatially the same position as the processing target pixel P (r ₀ , t ₀ ). 303 is output and stored in the future block memory 202.

  The pixel data of the three-dimensional block stored in the three block memories 201, 202, and 203 is output to the distance calculation unit 204, the interframe correlation calculation unit 205, and the filter processing unit 208. Further, the pixel data in the current block memory 203 is further output to the correction coefficient calculation unit 206.

In the distance calculation unit 204, the pixel position (r, t) stored in the three input block memories 201, 202, and 203 and the position (r ₀ ) that is the processing target pixel recorded in the block memory 203. , T ₀ ), the following spatiotemporal distance Ds is calculated.
When t ≧ t ₀ Ds = α ₁ | t−t ₀ | + β | r−r ₀ |
When t <t ₀ Ds = α ₂ | t−t ₀ | + β | r−r ₀ |

Here, α ₁ , α _2, and β are predetermined coefficients of 0 or more, and || represents an absolute value. Further, | r−r ₀ | = √ {(x−x ₀ ) ² + (y−y ₀ ) ² }. The calculation of the spatial distance | r−r ₀ | may be performed by storing the calculation result in a ROM table (not shown) in advance when the block size (N _x × N _y ) is determined in advance. Is possible.

Α ₁ indicates a coefficient for converting the time of the future block into a distance, and α ₂ indicates a coefficient for converting the time of the past block into a distance. Here, the past block consists of pixels whose noise has already been reduced, and the future block consists of pixels whose noise has not been reduced yet. For this reason, it is possible to improve the noise reduction effect by setting the weighting factor of each pixel determined by the filter coefficient calculation unit 207 to be greater in weight with respect to the past block than the future block.

In the distance calculation unit 204, the pixel value P (r, t) at the position (r, t) stored in the three input block memories 201, 202, and 203 is recorded in the current block memory 203. The pixel value distance Dv is calculated based on the following calculation formula from the pixel value P (r ₀ , t ₀ ) at the target position (r ₀ , t ₀ ) of the noise reduction process.
Dv = | P (r, t) −P (r ₀ , t ₀ ) |

The distance calculation unit 204 multiplies the above Ds and Dv to calculate the distance D = Ds × Dv of the pixel P (r, t) in the three-dimensional block with respect to the processing target pixel P (r ₀ , t ₀ ). And output to the filter coefficient calculation unit 207.

The inter-frame correlation calculation unit 205 uses the pixel values P (r, t) of the pixels stored in the input three block memories 201, 202, and 203, to generate a frame between the current block 302 and the past block 301. and between the correlation value S _p, and the inter-frame correlation value S _f of the current block 302 and the future blocks 303 calculated as follows, and outputs the correction coefficient calculation section 206.
S _p = Σ _r | P (r, t ₀ −1) −P (r, t ₀ ) |
S _f = Σ _r | P (r, t ₀ +1) −P (r, t ₀ ) |
Here, sigma _r indicates the total sum in the block _{N x} × _{N y.}

A noise reduction amount calculation unit 209, an interframe correlation calculation unit 205, a control unit 112, a current block memory 203, and a filter coefficient calculation unit 207 are connected to the correction coefficient calculation unit 206. The correction coefficient calculation unit 206 includes a frame average noise reduction amount NR _ave calculated by the noise reduction amount calculation unit 209, a target noise reduction amount NR _target output from the control unit 112, and an inter-frame correlation calculation unit 205. The correlation value S _p between the current block 302 and the past block 301 that is output, the correlation value S _f between the current block 302 and the future block 303 that is output from the inter-frame correlation calculation unit 205, and the current block memory 203. Current pixel data is input. The correction coefficient calculation unit 206 calculates the correction coefficient T _i (where i = p, c, f) of the filter coefficient calculated by the filter coefficient calculation unit 207 using these input data.

The correction coefficient T _i is one coefficient T _p , T _c , and T _f for the current block 302, the past block 301, and the future block 303, respectively, and this correction coefficient T _i is sent to the filter coefficient calculation unit 207. Is output. Here, a three-dimensional block composed of three frames is taken as an example, but a three-dimensional block composed of arbitrary N frames may be used. In this case, N coefficients T _i are calculated and output to the filter coefficient calculation unit 207.

The filter coefficient calculation unit 207 receives the distance D output from the distance calculation unit 204 and the correction coefficients T _p , T _c , and T _f output from the correction coefficient calculation unit 206. The filter coefficient calculation unit 207 uses these data to filter coefficient C (r, r, t) corresponding to the pixel P (r, t) stored in the current block memory 203, the past block memory 201, and the future block memory 202. t) is calculated and output to the filter processing unit 208.

The filter processing unit 208 reads out the pixels P (r, t) stored in the three block memories 201, 202, and 203 in a predetermined order. Then, a product-sum operation is performed on the read pixel P (r, t) and the filter coefficient C (r, t) output from the filter coefficient calculation unit 207 to obtain a noise reduction pixel P _n (r ₀ , t ₀ ). Is calculated and output.

The noise reduction amount calculation unit 209 includes the noise reduction pixel P _n (r ₀ , t ₀ ) calculated and output by the filter processing unit 208 and the processing target pixel P (r ₀ , t ₀ ) from the current block memory 203. ) And. The noise reduction amount calculation unit 209 calculates a difference absolute value | P _n (r ₀ , t ₀ ) −P (r ₀ , t ₀ ) | between two pixel values, and integrates the difference absolute value for one frame. To do. Then, when the processing for one frame is completed, the frame average noise reduction amount NR _ave is calculated as follows and output to the correction coefficient calculation unit 206.
NR _ave = Σ _r0 | P _n (r ₀ , t ₀ ) −P (r ₀ , t ₀ ) | / total number of pixels in frame Here, NR _ave is an average value in the above example, but the total number of pixels in the frame In an apparatus in which the number does not change, the sum of absolute differences may be substituted.

Next, details of the correction coefficient calculation unit 206 will be described based on the functional blocks of FIG.
In the correction coefficient calculation unit 206, the subtractor 601 receives the frame average noise reduction amount NR _ave output from the noise reduction amount calculation unit 209 and the target noise reduction amount NR _target output from the control unit 112. The subtractor 601 subtracts these input values, and outputs the reduced error amount Er = NR _ave -NR _target , which is the subtracted value, to the reference correction coefficient determination unit 602.

The reference correction coefficient determination unit 602 determines a reference correction coefficient T _base based on the input reduction error amount Er. For example, when the threshold value is TH, the threshold value is changed in three steps by the following threshold determination.
In the case of Er> TH, the reference correction coefficient T _base = V ₁
When TH ≧ Er ≧ −TH Reference correction coefficient T _base = V ₂
In the case of Er <−TH, the reference correction coefficient T _base = V ₃
Here, V ₁ , V ₂ , and V ₃ are coefficients determined in advance such that V ₁ <V ₂ <V ₃ .

That is, the reference correction coefficient V ₂ standard when the absolute value of the threshold TH below Er, Er is a small reference correction coefficient V ₁ than standard if it becomes greater than the threshold TH, Er is the threshold value TH When the value is smaller than the negative value, the reference correction coefficient V ₃ larger than the standard is output to the multiplier 605.

  On the other hand, the current block 302 output from the current block memory 203 is input to the average value calculation unit 603. The average value calculation unit 603 calculates the average pixel value of the current block 302 and outputs it to the LUT_V 604. The LUT_V 604 converts the correction value Rv of the reference correction coefficient with respect to the average pixel value and outputs it to the multiplier 605.

An example of the relationship between the average pixel value and the reference correction coefficient correction value Rv is shown in FIG.
In the example of FIG. 4, the reference correction coefficient correction value Rv is larger than 1 when the pixel value is small, and the reference correction coefficient correction value Rv is smaller than 1 as the pixel value increases. That is, in the dark area, the setting of the noise correction amount is emphasized by increasing the reference correction coefficient, and in the bright area, the resolution is emphasized by decreasing the reference correction coefficient. Note that the reference correction coefficient may not depend on the pixel value by setting the correction value Rv = 1 regardless of the pixel value.

The multiplier 605 multiplies the reference correction coefficient T _base by the correction value Rv and outputs the result to the multiplier 607.

On the other hand, the inter-frame correlation value S _p, fixes and inter-frame correlation value S _f is coefficient ratio calculation of the future blocks 303 and current block 302 with the past block 301 and current block 302 which is output from the inter-frame correlation calculating unit 205 Input to the unit 606. For example, the correction coefficient ratio calculation unit 606 assigns the ratios R _p , R _c , and R _{f to} the past block 301, the current block 302, and the future block 303 as follows.
When S _p ≦ TH _p and S _f ≦ TH _f R _p = R _c = R _f
When S _p ≦ TH _p and S _f > TH _f R _p = R _c > R _f
When S _p > TH _p and S _f ≦ TH _f R _p <R _c = R _f
When S _p > TH _p and S _f > TH _f R _p = R _c = R _f
Here, R _p + R _c + R _f = 1.

The condition of S _p ≦ TH _p and S _f ≦ TH _f is a case where the temporal correlation is high for both the past block 301 and the future block 303 with respect to the current block 302, and the motion is in the block area that is temporally related. Corresponds to a small area. In this case, the correction coefficient is the same in both the time and space directions.

The condition of S _p ≦ TH _p and S _f > TH _f is a case where the past block 301 has a high time correlation and the future block 303 has a low time correlation with respect to the current block 302. It shows that the area is where sudden movements or scene changes occur in the time to the future. In this case, the correction coefficient of the past block 301 and the current block 302 is increased, and the correction coefficient of the future block is decreased.

The condition of S _p > TH _p and S _f ≦ TH _f is that the future block 303 is high in time correlation and the past block 301 is low in time correlation with respect to the current block 302. In this case, from the past It shows a region where a sudden movement or scene change has occurred in the time up to now. In this case, the correction coefficient of the future block 303 and the current block 302 is increased, and the correction coefficient of the past block 301 is decreased.

The conditions of S _p > TH _p and S _f > TH _f are when the time correlation is low for both the past block 301 and the future block 303 with respect to the current block 302. In this case, noise mixed in the image This indicates that there is too much or that the movement is large within the time from the past block 301 to the future block 303. In this case, the correction coefficient is the same in both the time and space directions.

Multiplier 607 multiplies output value T _base × Rv of multiplier 605 by output values R _p , R _c , and R _f from correction coefficient ratio calculation unit 606. Correction coefficient thus calculated _{T p = T base × Rv ×} R p, T c = T base × Rv × R c, and _{T f = T base × Rv ×} R f is output to the filter coefficient calculating section 207 Is done.
Here, in the above description, the example in which the reference correction coefficient, the correction value, and the block allocation ratio are all variably controlled has been shown. However, one or two of them can be variably controlled, and the others can be set to predetermined constants. .

Next, the filter coefficient calculation unit 207 will be described based on the graph of FIG. 5 and the functional block diagram of FIG.
The distance D output from the distance calculation unit 204 is output to the LUT_R ₁ 701, LUT_R ₂ 702,..., LUT_R _N 703 and the interpolation processing unit 705.
LUT_R ₁ 701, LUT_R ₂ 702,..., LUT_R _N 703 are tables corresponding to the graphs 501, 502, and 503 having different values of T in the rational function {T / (x + T)} shown in FIG. T is a value greater than zero.

Then, LUT_R ₁ 701 corresponding to the rational function of the graph 501 as T ₁ / (x + T ₁ ), LUT_R ₂ 702 corresponding to the rational function of the graph 502 as T ₂ / (x + T ₂ ), LUT_R ₂ 702 Let the rational function of the graph 503 be T _N / (x + T _N ), and the corresponding lookup table be LUT_R _N 703. Here _{T 1,} T 2, ..., corresponds to the correction _{factor T N} is outputted from the correction coefficient calculation unit _206, a relationship of _{T 1 <T 2 <... <} T N. That is, the above rational function is a function that decreases rapidly as the variable x increases when the correction coefficient is small, and a function that decreases gently as the variable x increases when the correction coefficient is large.

  Each look-up table is input as an address extracted by a predetermined number of upper bits with respect to the distance D output from the distance calculation unit 204. Each lookup table outputs the value of the lookup table stored in the address and the amount of inclination to the selection unit 704.

Here, in LUT_R ₁ 701, LUT_R ₂ 702,..., LUT_R _N 703, the rational function is sampled with a value R _j (a) sampled by a variable x = a and X = a + Δ separated by a predetermined interval Δ. It is a table in which the inclination α _j (a) when _j (a + Δ) is connected by a straight line and the starting point R _j (a) are recorded at the location of address a. Here, j = 1, 2,..., N.
That is, based on the distance D, addresses that refer to LUT_R ₁ 701, LUT_R ₂ 702,..., LUT_R _N 703 are determined, and the start point R _j (a) stored in the determined address a and the slope α _j (a ) Is output to the selection unit 704.

Selecting unit 704 selects a gradient with these N start _R j (a) and the slope α _j (a) and on the basis of the correction coefficient _{T i} 1 single start point R (a) α (a) , the interpolation processing unit Output to 705.
The interpolation processing unit 705 receives the start point R (a) and the inclination α (a) output from the selection unit 704 and the predetermined lower-order bits of the distance D, so that the weighting factor R ′ (D _, T _i ) are calculated and sequentially stored in the memory 706.

When the weight coefficients corresponding to the number of three-dimensional blocks N _x × N _y × N _t pixels are stored in the memory 706, these weight coefficients are input to the normalization coefficient calculation unit 707. The normalization coefficient calculation unit 707 calculates a normalization coefficient by integrating these weighting coefficients. The calculated normalization coefficient is converted into the reciprocal of the normalization coefficient by LUT_D708 without dividing the weighting coefficient stored in the memory 706 by the normalization coefficient. Here, LUT_D708 is a multiplication / division conversion table prepared in advance for normalization by multiplication.

  The reciprocal of the normalized coefficient thus converted is multiplied by the weight coefficient stored in the memory 706 by the multiplier 709 and output to the filter processing unit 208 as the final filter coefficient C (r, t). The

The filter coefficient C (r, t) corresponding to the pixel P (r, t) of the three-dimensional block N _x × N _y × N _t calculated in this way corresponds to the processing target pixel P (r ₀ , t ₀ ). The weight coefficient (filter coefficient) value is inversely proportional to the distance calculated from the distance between the pixel values and the spatiotemporal distance, and this weight coefficient is variably controlled in accordance with the correction coefficient, that is, the temporal correlation variation of the moving image signal. It has the characteristic that it can.

That is, when the pixel to be processed P (r ₀ , t ₀ ) is a pixel in the three-dimensional block N _x × N _y × N _t and a pixel having a close spatiotemporal distance (a pixel having high correlation), a large weight is given. Has a coefficient. On the other hand, a pixel having a large pixel value and spatiotemporal distance (a pixel having low correlation) has a small weighting factor. For this reason, the filter processing result in the filter processing unit 208 described below can obtain an effect equivalent to collecting only pixels having high correlation and taking an average. Therefore, even if a spatial edge and a temporal edge (when a scene change or a rapid movement occurs) are included in the block, it is possible to minimize the dullness of the edge due to averaging.

Further, since the weight coefficient allocation can be variably controlled by the correction coefficient, the noise reduction result of one frame before is reflected, and when the target noise reduction amount is not satisfied, the three-dimensional block N _{x is set} so that the specified noise reduction amount is obtained. It is possible to give a weighting coefficient that cancels the correlation amount between the pixel value and the distance in × N _y × N _t . In this case, it is equivalent to selecting more pixels in the three-dimensional block N _x × N _y × N _t and performing the averaging process, and the noise reduction effect can be increased. Conversely, if the noise reduction result of the previous frame exceeds the target noise reduction amount, the correlation value between the pixel value and the distance in the three-dimensional block N _x × N _y × N _t is set so as to be the target noise reduction amount. A stronger weighting factor can be given.

Even in a scene change in which the time correlation changes suddenly or in a scene in which a rapid movement occurs, the future and current block having the time correlation within the three-dimensional block N _x × N _y × N _t , or the past By controlling the correction coefficient so as to cancel the correlation amount between the pixel value and the spatio-temporal distance with respect to the current block, it is possible to prevent the noise reduction amount from rapidly deteriorating. Note that if the correction coefficient is used to control the correlation between the pixel value and the spatio-temporal distance, a side effect of reducing the resolution occurs. However, since the weight coefficient can be changed substantially continuously with respect to the correction coefficient as in the configuration of the present embodiment, the temporal and spatial resolution changes are controlled so that the user can visually It is possible to prevent the change from being recognized.

Next, details of the filter processing unit 208 will be described based on the functional block diagram of FIG.
The filter processing unit 208 includes a filter coefficient C (r, t) output from the filter coefficient calculation unit 207, pixel values P (r, t) stored in the block memories 201, 202, and 203, and a control. The noise model table output from the unit 112 is input.
The filter coefficient C (r, t) and the pixel value P (r, t) are processed by the product-sum operation unit (product-sum operation means) 801 as follows, and the pixel P (r ₀ , t ₀ ) is processed. The smoothed pixel value P ′ (r ₀ , t ₀ ) is output to the noise amount estimation unit (noise amount estimation unit) 802 and the coring processing unit 803.
_{P '(r 0, t 0} ) = Σ r Σ t C (r, t) P (r, t)

The noise amount estimation unit 802 stores the noise model table output from the control unit 112, and uses the smoothed pixel value P ′ (r ₀ , t ₀ ) output from the product-sum operation unit 801 as an address. The corresponding noise amount N _amp is output to the coring processing unit 803. Here, the noise model table is a table in which the amount of noise (an amount corresponding to the average amplitude of noise) is stored at the address position corresponding to the pixel value. The control unit 112 converts the predetermined noise model table into the pixel value. It is also possible to make a noise model table arbitrarily changed by multiplying with an arbitrary function as a variable.

The coring processing unit 803 includes a processing target pixel value P (r ₀ , t ₀ ), a pixel value P ′ (r ₀ , t ₀ ) after the smoothing process, and noise output from the noise amount estimation unit 802. A quantity is entered. The coring processing unit 803 performs the following coring determination, calculates a final noise reduction pixel value P _n (r ₀ , t ₀ ) for the pixel to be processed, and outputs it as an output signal.
When P (r ₀ , t ₀ ) −P ′ (r ₀ , t ₀ )> N _amp P _n (r ₀ , t ₀ ) = P (r ₀ , t ₀ ) −N _amp
When P (r ₀ , t ₀ ) −P ′ (r ₀ , t ₀ ) <N _amp P _n (r ₀ , t ₀ ) = P (r ₀ , t ₀ ) + N _amp
| P (r ₀ , t ₀ ) −P ′ (r ₀ , t ₀ ) | ≦ N _amp P _n (r ₀ , t ₀ ) = P ′ (r ₀ , t ₀ )

The filter processing unit 208 is configured to include the noise amount estimation unit 802. For example, the filter processing unit 208 includes only a simpler product-sum operation unit 801, and the final noise reduction pixel value P _n (r ₀ , T ₀ ) may be P ′ (r ₀ , t ₀ ).

  As described above, according to the image processing apparatus according to the present embodiment, the spatial correlation in the frame and the temporal correlation between the past frame and the current frame and between the current frame and the future frame are calculated. Utilizing this, noise reduction processing of the moving image signal is performed. Thereby, for example, even when a scene change occurs, it is possible to suppress a noise reduction amount from reacting sensitively to a change in time correlation, and it is possible to stably reduce noise while suppressing a reduction in resolution.

  Further, since the pixels after noise reduction are used for the pixels of the past frame and the current frame used for the cyclic noise reduction processing, the amount of noise reduction can be improved.

  In addition, the weighting factor is calculated based on the spatio-temporal distance and the distance between pixel values, and weighted average is performed by weighting the pixels that are highly correlated with the noise processing target pixel. It is possible to effectively reduce noise while minimizing dullness with respect to changes.

  In addition, the noise characteristic of the input signal is estimated by estimating the noise amount of the pixel of interest based on the product-sum operation value of the pixel values and weighting factors of a plurality of pixels in a predetermined area, and performing noise reduction processing based on the noise amount. In addition to effective noise reduction suitable for the image quality, it is possible to minimize resolution deterioration (dullness) at the edge portion.

  In addition, by correcting the weighting factor based on the noise reduction amount of the pixel of interest, it is possible to suppress the noise reduction amount from reacting sensitively to fluctuations in time correlation that occurs when reducing the noise of the moving image signal, Stable noise reduction with little time fluctuation can be achieved while suppressing a decrease in resolution.

  In addition, by correcting the weighting coefficient based on the pixel value of the target pixel, it is possible to control the noise reduction amount over a wide range in accordance with the pixel value in the current frame.

  In addition, by correcting the weighting coefficient based on the inter-frame correlation between the processing target frame image and the past and future frame images, the amount of noise reduction is reduced due to the temporal correlation variation that occurs when noise is reduced in the moving image signal. Responsive reaction is suppressed, and stable noise reduction with little time variation is possible.

  In the embodiment described above, processing by hardware is assumed, but it is not necessary to be limited to such a configuration. For example, a configuration in which processing is performed separately by software is also possible. In this case, the image processing apparatus includes a main storage device such as a CPU and a RAM, and a computer-readable recording medium on which a program for realizing all or part of the above processing is recorded. Then, the CPU reads out the program recorded in the storage medium and executes information processing / calculation processing, thereby realizing processing similar to that of the above-described image processing apparatus. Here, the computer-readable recording medium means a magnetic disk, a magneto-optical disk, a CD-ROM, a DVD-ROM, a semiconductor memory, or the like.

The flowchart shown in FIG. 11 shows the processing procedure of the first embodiment.
First, the input moving image signal is stored in the frame memory 102 or 103 as a future frame via the switch 101. (S1101)

  Next, it is determined whether or not the input moving image signal is the first frame (S1102). If it is the first frame, the process proceeds to S1113. If it is not the first frame, the three-dimensional block P (r, t) is extracted from the past, future, and current frames stored in the frame memory 102 or 103-109 or 110, and further stored in the N-line memory 106. The block 301, the future block 303, and the current block 302 are stored in the block memories 201, 202, and 203 (S1103).

Next, the three-dimensional block which is stored in the block memory 201, 202, 203, and calculates inter-frame correlation amounts _{S p} and _{S f} (S1104), the three-dimensional block noise processing target pixel P (r ₀ , t ₀ ) and its surrounding pixels are calculated as a spatiotemporal distance Ds and a distance Dv between pixel values, and the distance D is calculated by multiplying the two distances (S1105).

Then a 1-frame previous frame average amount of noise reduction _{NR ave} and target amount of noise reduction _{NR target,} pixel value P of the current block (r, _{t 0)} and three blocks from the inter-frame correlation amounts _{S p} and _{S f} Correction coefficients T _p , T _c and T _f are calculated (S1106), and a filter coefficient C (r, t) is calculated from the distance D and the correction coefficients T _p , T _c and T _f (S1107).

Next, a noise reduction pixel P _n (r ₀ , t ₀ ) is calculated based on the calculated filter coefficient C (r, t) and the three-dimensional block pixel P (r, t) (S1108), and the noise reduction pixel P _n (r ₀ , t ₀ ) is stored in an output buffer (not shown), the frame memory 109 or 110, and the N line memory 106 (S1109).

Next, a noise reduction amount is calculated from the sum of absolute differences between the noise reduction pixel P _n (r ₀ , t ₀ ) and the processing target pixel P (r ₀ , t ₀ ) (S1110), and the calculation of the noise reduction pixel is performed. It is determined whether or not the processing is completed for the number of pixels in the frame (S1111). As a result of the determination, if the processing for the number of frame pixels has not been completed yet, the processing returns to S1103, and the noise reduction processing for the next processing target pixel is repeated. When processing for the number of frame pixels is completed, an average value of the calculated noise reduction amounts is obtained, and a frame average noise reduction amount NR _ave is calculated (S1112). Then, the input / output of the switches 101, 104, 105, 108, and 111 for performing input / output control of the frame memory storing the current, past, and future frames is switched (S1113).

  Finally, it is determined whether or not the processing for the designated number of frames has been completed (S1114). If the processing has not been completed, the processing returns to S1101 to repeat the processing for the next frame, and it is determined that the processing for the designated number of frames has been completed. At that point, the noise reduction process ends.

In the above embodiment, the distance D is the product of the inter-pixel value distance Dv and the spatio-temporal distance Ds, and the weighting coefficient is calculated as a variable of one rational function. However, the inter-pixel value distance D _v and the spatio-temporal distance D _s are calculated. The weighting factor may be calculated by the product of the function values R (Dv) and R (Ds) after each is given to the function.

Furthermore, the function for converting the distance D into a weighting factor is not limited to a rational function, and for example, a Gaussian function Exp (−x ² / 2σ ² ) may be used. In this case, the same effect can be obtained by setting the distance D to x and the correction coefficient T to σ. Also, when σ is increased, the width of the Gaussian function becomes wider and acts to cancel the distance correlation. Conversely, when σ is reduced, the width of the Gaussian function is narrowed, and the distance correlation can be further emphasized.

[Second Embodiment]
Next, a second embodiment of the present invention will be described with reference to FIG.
The image processing apparatus according to the present embodiment is different from the first embodiment in that a motion compensation unit 901 is provided instead of the block extraction unit 114. Hereinafter, the image processing apparatus according to the present embodiment will not be described with respect to the points common to the first embodiment, and different points will be mainly described.

FIG. 9 is a functional block diagram showing the basic configuration of the second embodiment of the present invention.
A moving image signal captured by an image sensor of an imaging unit (not shown) and converted into digital data is input to the switch 101 and is alternately connected to the frame memory 102 or the frame memory 103 every frame period by a control signal of the control unit 112. Then, it is stored in the frame memory 102 or the frame memory 103 for each frame period.

  The outputs of the frame memory 102 and the frame memory 103 are connected to the inputs of the switch 104 and the switch 105, and are connected to the N line memory 106 and the motion compensation unit 901 by the control signal of the control unit 112. Here, when the frame memory 102 is connected to the N-line memory 106 via the switch 104, the frame memory 103 is connected to the motion compensation unit 901 via the switch 105. On the other hand, when the frame memory 102 is connected to the motion compensation unit 901 via the switch 105, the control unit 112 is controlled so that the frame memory 103 is connected to the N line memory 106 via the switch 104.

The N line memory 106 temporarily stores a predetermined number of pixels above and below the noise reduction processing target pixel from the frame 102 or the frame memory 103.
The output of the N line memory 106 is connected to the inputs of the noise reduction unit 107 and the motion compensation unit 901 via the block extraction unit 113.

  The noise reduction unit 107 is connected so that the output of the block extraction unit 113, the output from the motion compensation unit 901, and the control signal from the control unit 112 are input. The noise reduction unit 107 includes a predetermined region pixel of the processing target frame output from the block extraction unit 113, a predetermined region pixel of a frame image that is temporally compensated for by the motion compensation unit 901, and a control unit 112. The noise reduction processing is performed on the processing target pixel based on the control signal. Further, the noise reduction pixel calculated in this way is output as an output signal to a buffer memory of an image processing unit arranged in a subsequent stage (not shown).

  The output of the noise reduction unit 107 is connected to the switch 108 and the input of the N line memory 106. The output signal of the noise reduction unit 107 is used to overwrite the processing target pixel value before noise reduction stored in the N frame memory 106 with the noise reduction pixel. As a result, a recursive filter that uses pixels after noise reduction can be formed for the current frame, and noise can be reduced more effectively.

  The output of the switch 108 is switched to the frame memory 109 or the frame memory 110 for each frame period according to the control signal of the control unit 112, and the frame memory 109 or the frame memory 110 to which the noise reduction pixel of the noise reduction unit 107 corresponds. To be recorded.

  The frame memory 109 or the frame memory 110 is switched and connected to the motion compensation unit 901 via the switch 111 for each frame period according to a control signal from the control unit 112. Here, when the output of the switch 108 is connected to the frame memory 109, the frame memory 110 is connected to the switch 111. On the other hand, when the output of the switch 108 is connected to the frame memory 110, the control unit 112 controls so that the frame memory 109 is connected to the switch 111.

  The motion compensation unit 901 includes the current block 302 from the connected block extraction unit 113, the future that changes in time with respect to the processing target frame via the switch 104, the switch 105, and the switch 111, and the past A frame is input. The motion compensation unit 901 calculates a position where the correlation value is maximum between the current block 302 and the future and past frames. Then, the region having the highest correlation is extracted from the future and past frames, and is output to the noise reduction unit 107 as the future block 303 and the past block 301.

  The control unit 112 outputs a preset target noise reduction amount to the noise reduction unit 107, and also outputs a control signal for interlocking control of the switches 101, 104, 105, 108, and 111 as described above.

Details of the motion compensation unit 901 will be described below with reference to FIG.
An extracted image of a predetermined search range of the past frame input to the motion compensation unit 901 is stored in the search range storage memory 1001. In addition, an extracted image of a predetermined search range of the future frame is stored in the search range storage memory 1002.

On the other hand, the current block is input to a motion vector determination unit (motion vector determination unit) 1003 and a motion vector determination unit (motion vector determination unit) 1004. Also, an extracted image of the motion vector search range of the past frame (region larger than the current block size N _x × N _y ) is input from the search range storage memory 1001 to the motion vector determination unit 1003. In addition, an extracted image of the motion vector search range of the future frame (region larger than the current block size N _x × N _y ) is input from the search range storage memory 1002 to the motion vector determination unit 1004.

  The motion vector determination units 1003 and 1004 use the current block as a pattern matching reference image to perform pattern matching processing while moving at the pixel pitch within the search range extraction image, and move the position within the search range where the correlation is maximum. Determine as a vector. Here, as an example of the pattern matching process, a well-known method can be used in which the correlation is maximized at a position where the sum of absolute values of differences or the sum of squares is minimized.

The motion vectors determined by the motion vector determination units 1003 and 1004 are input to the block extraction units 1005 and 1006, respectively.
The block extraction unit 1005 receives the extracted image of the past frame stored in the search range storage memory 1001, extracts the past block based on the determined motion vector to the past frame, and outputs the extracted block to the noise reduction unit 107. .
The block extraction unit 1006 receives the extracted image of the future frame stored in the search range storage memory 1002, extracts the future block based on the determined motion vector to the future frame, and outputs it to the noise reduction unit 107. .

  As described above, according to the present embodiment, since the future block and the past block are selected by motion compensation for the current block, a three-dimensional block having a higher correlation than that of the first embodiment can be configured. As a result, the weighting coefficient for the future block and the past block calculated by the filter coefficient calculation unit 207 becomes larger, and the contribution to the weighted average increases. As a result, the amount of noise reduction is suppressed from reacting sensitively to fluctuations in time correlation that occur when moving image signals are reduced by cyclic noise reduction processing, and more stable noise reduction is performed. It becomes possible. Furthermore, it is possible to automatically adjust to the target noise reduction amount by adapting to the input moving image signal, and it is possible to obtain the effect that it can be instantly adapted even when the gain amount of the image sensor is changed. It becomes.

  In each of the above embodiments, the cyclic type (IIR type) in which all the pixels of the past block and some pixels of the current block are replaced with pixel values after noise reduction processing is described as an example. FIR type). In this case, the amount of noise reduction in the still region is lower than that in the cyclic type, but the convergence performance for suppressing the afterimage generated in the motion region is improved. Furthermore, although a progressive signal is assumed as an input signal in the first and second embodiments, a field interlace signal may be used. In this case, each of the above embodiments is configured by describing the frame image as a field image.

FIG. 3 is a functional block diagram illustrating the functions provided in the image processing apparatus according to the first embodiment of the present invention. It is a functional block diagram of the noise reduction part of the image processing apparatus shown in FIG. It is a figure which shows the three-dimensional block which consists of a present, past, and future block processed with a noise reduction part. It is a figure which shows an example of the relationship between a pixel value and a reference | standard correction coefficient correction value. It is a figure which shows the some rational function which changes with correction coefficients. FIG. 3 is a functional block diagram of a correction coefficient calculation unit shown in FIG. 2. FIG. 3 is a functional block diagram of a filter coefficient calculation unit shown in FIG. 2. FIG. 3 is a functional block diagram of a filter processing unit shown in FIG. 2. It is the functional block diagram which expanded and showed the function with which the image processing apparatus which concerns on the 2nd Embodiment of this invention is provided. FIG. 10 is a functional block diagram of a motion compensation unit shown in FIG. 9. It is a flowchart which shows the process sequence of 1st Embodiment.

Explanation of symbols

102 frame memory 103 frame memory 106 N line memory 107 noise reduction unit 109 frame memory 110 frame memory 113 block extraction unit 114 block extraction unit 204 distance calculation unit 205 inter-frame correlation calculation unit 206 correction coefficient calculation unit 207 filter coefficient calculation unit 208 filter Processing unit 209 Noise reduction amount calculation unit 705 Interpolation processing unit 707 Normalization coefficient calculation unit 802 Noise amount estimation unit 901 Motion compensation unit 1003 Motion vector determination unit 1004 Motion vector determination unit

Claims (11)

An image processing apparatus for reducing noise of a frame image or a field image input in time series,
A recording means for recording a processing target frame image or field image and past and future frame images or field images with respect to the processing target frame image or field image;
First pixel extraction means for extracting a plurality of pixels in a predetermined area in the processing target frame image or field image recorded by the recording means;
Second pixel extraction means for extracting a plurality of pixels in an area corresponding to the predetermined area in the past frame image or field image and the future frame image or field image recorded by the recording means;
A pixel of interest in the predetermined area extracted by the first pixel extracting means, a plurality of pixels in the predetermined area, and a plurality of pixels in an area corresponding to the predetermined area extracted by the second pixel extracting means. First distance calculating means for calculating a spatiotemporal distance;
The pixel value of the pixel of interest in the predetermined area extracted by the first pixel extracting means, the pixel values of the plurality of pixels of the predetermined area extracted by the first pixel extracting means, and the second pixel extraction Second distance calculating means for calculating a distance between pixel values with pixel values of a plurality of pixels in an area corresponding to the predetermined area extracted by the means;
Based on the spatio-temporal distance calculated by the first distance calculation means and the distance between the pixel values calculated by the second distance calculation means, noise of the processing target frame image or field image is reduced. Noise reduction means,
An image processing apparatus comprising: The recording means includes
Overwriting means for overwriting the recorded frame image or field image and the past frame image or field image with a noise-reduced image,
The first pixel extracting means includes
Extracting noise-reduced pixels for some of the pixels of the processing target frame image or field image,
The second pixel extracting means includes
The image processing apparatus according to claim 1, wherein pixels for which noise reduction processing has been performed on all pixels of the past frame image or field image are extracted. The noise reduction means includes
A weighting factor for calculating a weighting factor for a plurality of pixels in the predetermined area based on the spatiotemporal distance calculated by the first distance calculating unit and the distance between the pixel values calculated by the second distance calculating unit. A calculation means;
A weighted average value calculating means for calculating a weighted average value of the target pixel and a plurality of pixels in the predetermined region based on the weight coefficient calculated by the weight coefficient calculating means;
The image processing apparatus according to claim 1, further comprising: The noise reduction means includes
A noise amount estimating means for estimating a noise amount of the target pixel;
The image processing apparatus according to claim 1, wherein the noise of the target pixel is reduced based on the noise amount estimated by the noise amount estimation unit. The noise reduction means includes
Product-sum operation means for performing a product-sum operation on the pixel values of the plurality of pixels in the predetermined region and the weighting coefficient calculated by the weighting coefficient calculating means;
The noise amount estimating means includes
The image processing apparatus according to claim 4, wherein the noise amount is estimated based on a product-sum operation value calculated by the product-sum operation unit. The noise reduction means includes
Noise reduction amount calculation means for calculating a noise reduction amount based on pixel values before and after noise reduction processing of the target pixel;
Correction coefficient calculation means for calculating a correction coefficient based on the noise reduction amount calculated by the noise reduction amount calculation means;
Have
The image processing apparatus according to claim 3, wherein the weighting coefficient is corrected based on the correction coefficient calculated by the correction coefficient calculating unit. The noise reduction means includes
Correction coefficient calculation means for calculating a correction coefficient based on the pixel value of the target pixel extracted by the first pixel extraction means;
The image processing apparatus according to claim 3, wherein the weighting coefficient is corrected based on the correction coefficient calculated by the correction coefficient calculating unit. The noise reduction means includes
An inter-frame correlation amount calculating means for calculating an inter-frame correlation amount based on a plurality of pixels in a predetermined region of the processing target frame image or field image and a plurality of pixels in the predetermined region of the past and future frame images or field images;
Correction coefficient calculating means for calculating a correction coefficient based on the interframe correlation amount calculated by the interframe correlation amount calculating means;
Have
The image processing apparatus according to claim 3, wherein the weighting coefficient is corrected based on the correction coefficient calculated by the correction coefficient calculating unit. The second pixel extracting means includes
A motion vector detecting means for detecting a motion vector based on a correlation amount between the processing target frame image or field image and the past and future frame images or field images,
9. The method according to claim 1, wherein a plurality of pixels in a predetermined area are extracted from the past and future frame images or field images recorded in the recording unit based on the motion vector detected by the motion vector detecting unit. An image processing apparatus according to 1. An image processing program for causing a computer to perform noise reduction processing of a frame image or a field image input in time series,
A recording process for recording a processing target frame image or field image and past and future frame images or field images with respect to the processing target frame image or field image;
A first pixel extraction process for extracting a plurality of pixels in a predetermined area in the processing target frame image or field image recorded by the recording process;
A second pixel extraction process for extracting a plurality of pixels in an area corresponding to the predetermined area in the past frame image or field image and the future frame image or field image recorded by the recording process;
A pixel of interest in the predetermined area extracted in the first pixel extraction process, a plurality of pixels in the predetermined area, and a plurality of pixels in an area corresponding to the predetermined area extracted in the second pixel extraction process A first distance calculation process for calculating a spatiotemporal distance;
The pixel value of the target pixel in the predetermined area extracted by the first pixel extraction process, the pixel values of the plurality of pixels of the predetermined area extracted by the first pixel extraction process, and the second pixel extraction A second distance calculation process for calculating a distance between pixel values with pixel values of a plurality of pixels in an area corresponding to the predetermined area extracted by the process;
Processing for reducing noise in the processing target frame image or field image based on the spatiotemporal distance calculated by the first distance calculation processing and the distance between the pixel values calculated by the second distance calculation processing Noise reduction processing to
An image processing program for causing a computer to execute. An image processing method for reducing noise of a frame image or a field image input in time series,
A recording step of recording a processing target frame image or field image and past and future frame images or field images with respect to the processing target frame image or field image;
A first pixel extraction step of extracting a plurality of pixels in a predetermined region in the processing target frame image or field image recorded by the recording step;
A second pixel extraction step of extracting a plurality of pixels in a region corresponding to the predetermined region in the past frame image or field image and the future frame image or field image recorded by the recording step;
The pixel of interest in the predetermined region extracted in the first pixel extraction step, the plurality of pixels in the predetermined region, and the plurality of pixels in a region corresponding to the predetermined region extracted in the second pixel extraction step A first distance calculating step for calculating a spatiotemporal distance;
The pixel value of the target pixel in the predetermined region extracted in the first pixel extraction step, the pixel values of the plurality of pixels in the predetermined region extracted in the first pixel extraction step, and the second pixel extraction A second distance calculating step of calculating a distance between pixel values with pixel values of a plurality of pixels in a region corresponding to the predetermined region extracted in the step;
Based on the spatiotemporal distance calculated by the first distance calculating step and the inter-pixel value distance calculated by the second distance calculating step, noise in the processing target frame image or field image is reduced. Noise reduction process,
An image processing method.

Images