WO2008148924A1

WO2008148924A1 - Method for automatically improving images and sequences with spatially varying degradation

Info

Publication number: WO2008148924A1
Application number: PCT/ES2008/070106
Authority: WO
Inventors: Gabriel CRISTÓBAL PÉREZ; Salvador GABARDA TÉBAR
Original assignee: Consejo Superior De Investigaciones Científicas
Priority date: 2007-06-07
Filing date: 2008-05-29
Publication date: 2008-12-11
Also published as: ES2310136A1; ES2310136B1

Abstract

The invention relates to an automatic method for improving images, based on eliminating spatially varying degradation caused by adverse weather or environmental conditions.

Description

Method for the automatic improvement of images and sequences with spatially variant degradation.

SECTOR OF THE TECHNIQUE

There is a set of situations that in practice can hinder both the identification and recognition of objects in images, due to the presence of spatially varying degradations. Examples of such situations occur both in remote sensing and in video surveillance tasks, where the presence of fog, aerosols, cloud cover, etc. It can make the analysis of these images very difficult. The method proposed here allows to obtain a very good discrimination of the objects of interest, by eliminating in the images the types of degradations described above and which are difficult to treat by conventional methods of image processing. Most of these methods are based on the hypothesis that the present degradations are of the spatially invariant type, which in practice is usually not verified. The method proposed here is based on the extraction of a local measure of anisotropy in the images. Said measure facilitates the elimination of spatially degraded variations present in the images, such as fog, cloud cover, underwater navigation, etc. On the other hand, the method described here may have applications in other areas of interest, such as biomedical images, and more specifically the fundus scanning systems in the presence of cataracts in the lens, which makes it difficult to observe the retina in such cases. The proposed method would allow obtaining an improvement of the acquired images by reducing the effect produced by the turbidity of the lens. STATE OF THE TECHNIQUE

When image restoration tasks are carried out blindly, that is, without prior knowledge of the degradation process, these methods are called blind deconvolution methods [1]. In the event that the blur is not uniformly distributed over the image, said degradation affects the different regions of the same image differently. This scenario is called spatially variant blurring [2]. A particular case of spatially variant degradation occurs when multi-focal, multi-temporal or multi-sensory images are available, including the case of video sequences. In all these cases, various image fusion techniques are applicable [3-4]. In the area of elimination of degradations of the type of mist described above, various methods have been proposed, including adaptive methods and methods based on wavelets [5-8].

The invention procedure proposed here consists of a new fusion procedure between the different frames of the input sequence, which allows solving the problem of eliminating the degradations caused by the presence of mists, aerosols, etc. The advantages of the procedure proposed here are:

- Its low computational cost, compared to other methods that work with space-frequency distributions for the representation of the information contained in the images.

- Provides, implicitly, the definition of a quantitative measure of the improvement obtained.

On the other hand, in Annex I there are several invention patents that are related to this proposal.

References: [1] D. Kundur and D. Hatzinakos, "Blind Image Deconvolution." IEEE Signal Processing Magazine, 13 (3), 1996, pp. 43-64. [2] AN Rajagopalan and S. Chaudhuri, "Space-Variant Approaches to

Recovery of Depth from Defocused Images. "Computer Vision and Image

Understanding, VoI. 68, No. 3, 1997, pp. 309-329.

[3] D. Kundur, D. Hatzinakos, and H. Leung, "A Novel Approach to Multispectral Blind Image Fusion." In B. V. Dasarathy, editor, Sensor

Fusion: Architectures, Algorithms, and Applications, Proceedings of SPIE,

VoI 3067, 1997, pp. 83-93.

[4] Z. Zhang and R. S. Blum, "A categorization of multiscale-decomposition- based image fusion schemes with a performance study for a digital camera application", Proc. IEEE, 87, 1999, pp. 1315-1328

[5] Y. Du, B. Guindon and J. Cihlar, "Haze detection and removal in high resolution satellite images with wavelet analysis," IEEE Trans. Geosci

Remote Sensing, 40, 2002, pp. 210-217.

[6] Y. Zhang, B. Guindon and J. Cihlar "Development of a robust haze detection algorithm: assessment using temporally invariant targets", in

Proc. IGARSS Toronto, ON, Canada, June 24-28, 2002

[7] Y. Zhang and B. Guindon, "Quantitative assessment of a haze suppression methodology for satellite imagery: effect on land cover classification performance", IEEE Trans. On Geoscience and Remote Sensing, 41, 2003, pp. 1082-1089

[8] R. Richter, "Atmospheric correction of satellite data with haze removal including a haze / clear transition region", Comput. Geosci, 22, 1996, pp.

675-681

[9] E. Wigner, "On the Quantum Correction for Thermodynamic Equilibrium." Physical Review, VoI. 40, 749-759 (1932).

[10] T. A. C. M. Claasen and W. F. G. Mecklenbráuker, "The Wigner

Distribution - A Tool for Time-Frequency Analysis, "Parts l-lll. Philips J.

Research, VoI. 35, 217-250, 276-300, 372-389 (1980).

[11] K. H. Brenner, "A Discrete Version of the Wigner Distribution Function," Proc. EURASIP, Signal Processing II: Theories and

Applications, 307-309 (1983). [12] L. Galleani, L. Cohen, G. Cristóbal and B. Suter, "Generation and denoising of images with clouds," SPIE, 8-10 JuY 2002, Seattle, Washington, USA.

[13] C. E. Shannon and W. Weaver. The Mathematical Theory of Communication. The University of Illinois Press, Urbana, Chicago, London, 1949.

[14] Alfréd Rényi. "Some fundamental questions of information theory." In Pal Turan, editor, Selected Papers of Alfréd Rényi, volume 3, pp. 526-552. Akadémiai Kiadó, Budapest, 1976. (Originally: MTA III. Oszt. Kozl., 10, 1960, pp. 251-282).

[15] P. Flandrin, R. G. Baraniuk, O. Michel, "Time-frequency complexity and information", Proceedings of the ICASSP, vol. 3, 1994, pp. 329-332. [16] Y. Qu, Z. Pu, H. Zhao and Y. Zhao, "Comparison of different quality assessment functions in autoregulative illumination intensity algorithms", Optical Engineering, 45, 2006, pp. 117201

DESCRIPTION OF THE INVENTION Brief description of the invention.

The method proposed in this invention patent consists in obtaining an improvement in the quality of the images provided by a capture device, such as a conventional CCD camera. This procedure is characterized in its analysis phase by the use of a space-frequency distribution for the images. Subsequently, an unpublished image fusion algorithm is applied, based on a measure of anisotropy that allows obtaining an improved image. The usefulness of the proposed method extends to video surveillance, security, navigation, remote sensing and biomedicine applications. Detailed Description of the invention.

The method proposed in this invention patent consists of four stages: - Obtaining the sequence of the video images.

- Pre-processed of the sequence of the video images.

- Calculation of the anisotropy of the images.

- Fusion of images.

Obtaining the sequence of the video images.

Video sequences obtained and stored by usual procedures in digital imaging technology are considered. The sequences must be subdivided into frames for processing and analysis frame by frame.

Pre-processed of the sequence of video images. The frequency information of a signal can be obtained by associating a vector with a discrete position n of the signal with the discrete values provided by the Wigner pseudo-distribution (PWD). The discrete approximation of the Wigner distribution [9] used has been the one proposed in [10], similar to Brenner's expression [11]:

W [n, k] =

(1) where z ^* is the conjugate complex of the signal z, myk represent the discrete variables of time and frequency, respectively, and W [n, k] is a matrix where each row is a vector that represents the value of PWD in pixel n, for the different values of spatial frequency k. This expression can be interpreted as the discrete Fourier transform (DFT) of the product r [n, m] = z [n + m] z ^* [n - m], and is limited to the neighborhood interval [-Λ // 2, Λ // 2 - 1] of pixel n. The PWD presents coefficients with different magnitude for each position n, due to changes in the values of the signal according to the spatial variable.

In the present invention patent a method of image improvement invariant to the changes in brightness is described, based on the characterization of the content in frequencies of the mist by a PWD pattern, discernible from the knowledge only of the image itself. For this, the input sequence that presents the spatially variant degradation is processed pixel by pixel, resulting in an improved image in which the mist has been suppressed. In this way the pixels that show mist degradation are eliminated and replaced by others estimated from the input images. The proposed procedure is based on the use of a 1 D implementation of the Wigner distribution function indicated in Eq. (one ). This results in that for each pixel of the image a 1 D vector can be associated with the same number of components as the values that were used in its calculation. Said vector can be expressed through:

PWD [x, y] = w = [w_ _{N / 2} , ..., w _m _¡ \

(2)

For the calculation of this vector a one-dimensional data window is used on the image, to which the desired orientation can be given, this being the fundamental theoretical basis for measuring the anisotropy of the image that is analyzed, at the level of pixel

In practice, in the case of multi-temporal images and remote sensing, degradations are often found through cloud coverings or fog, which makes interpretation difficult. exist studies related to the statistics of clouds considered as noise [12].

In the proposed method, the calculation of the PWD described above for each pixel of the image is performed first. For this, the 1 D version of the PWD is applied in sliding windows of size Λ / = 8 pixels. In this way, the space-frequency information of the images is obtained simultaneously. In the event that the input sequence corresponds to a multi-temporal sequence, with the presence of mist, it is possible to use the PWD of homologous points, with different orientations of the windows on the image, to perform an anisotropy measurement, through from which it is possible to obtain a degradation free image. Basically, the pixels corresponding to the fog are replaced by fog-free pixels, from the use of the frames of the input sequence and after applying a fusion procedure between them.

Calculation of the anisotropy of the images.

As already indicated, the PWD presents coefficients with different magnitude for each position n, due to changes in the values of the signal according to its position. One way to quantify these differences between the PWD is by means of a measurement made in each position n, for which the Rényi entropy of said local PWD can be used.

The measure of entropy was initially proposed by Shannon [13] as a measure of the information content per symbol, from a stochastic information source. Subsequently, Rényi [14] extended this notion by introducing the concept of generalized entropy. The measure of Rényi's entropy applied to a space-frequency distribution (E-F) has the form:

(3) where n is the discrete spatial variable, k is the discrete frequency variable and a is a parameter whose recommended value must be equal to or greater than 2 [15]. In the Eq. (3) P represents the PWD defined in Eq. (one ). Although Rényi's measurements of the E-F distributions resemble the original entropy, they do not have the same properties that derive from the classical theory of information.

In order to adapt the values of a distribution such as PWD to the case of unit energy signals, it is necessary to perform some kind of normalization. The so-called quantum normalization is the one that has proven to be, in an experimental way, the most appropriate. The procedure consists in associating the PWD of a certain position n with a probability distribution function by means of the expression: P [n, k] = PWD [n, k] PWD ^* [n, k]

(4) together with a normalization step to achieve that the condition

∑∑P [n, k] = l is satisfied. n k

Substituting (4) in equation (3) for a = 3, it turns out

(5)

This measurement can be interpreted at the pixel level by means of:

(6)

The term P has to be normalized, at the pixel level, as follows:

Q [n, k] = PWD [n, k] PWD ^* [n, k]

(7)

(8) in order to verify the normalization condition,

(9) where M represents the number of samples to be processed and k represents

The frequency variable: -N 12 <k <N 12

Equation (6) provides an entropy value for each pixel of the image. Based on this, it is possible to define figures of merit for each pixel of the image given by R [n; θ _s ]. Here θ _s e [θ _{] r} θ ₂ , ..., θ _s ] has been introduced to represent S different orientations to measure the entropy, based on as many orientations of the PWD. In this way, the expected value of the entropy is obtained as the average of a certain number of analysis directions and as a consequence of the foregoing an anisotropy value. Specifically, the following orientations were chosen: (0, 30, 60, 90, 120 and 150 °). Finally, the variability (standard deviation) of the entropy values at the pixel level was chosen as a measure of the anisotropy of the images. So in the case that R [n; θ _s ] represent the entropy value for the pixel n of the image, measured according to the direction θ _s <≡ [θ _j , Q ₂ , ..., θ _s ], the standard deviation relative to the position n of the image , can be expressed by:

(10)

Where μ represents the mean of the R [n; ] _s ] defined by the expression,

(eleven )

The value σ [n] given by Eq. (10) is defined as an anisotropy index of the pixel n and measures the quality of the local information of that pixel. Averaged over the image allows defining a measure of the overall quality of the image considered. R values [n; θ _s ] of entropy in pixel n constitute a vector of dimension S, to which one more component element, R [n; θ _{s + 1} ] = λz [n], which provides information on the gray level z [n] of the pixel considered, weighted by a coefficient λ to be determined experimentally. This allows the algorithm to be more selective about the "color" of the noise. The value λ of is given by the following expression:

(12) Where A <≡ [0, 1] determines the importance of the mist, n is the size of the window used in the calculation of the entropy, B is the value of the level gray more characteristic of the fog and C is the maximum observed in the z image.

Fusion of the images In short, the proposed method is based on the fact that each pixel of the image has different anisotropy values, given by the expression (10). In this way, it is possible to define a fusion algorithm of a sequence of T images by means of the expression:

(13)

Here, DM represents a decision map that indicates the image t of the input sequence from which the value of the pixel n is taken to constitute the resulting image. This map allows an improved image to be constructed from the fusion of the input sequence by means of the selection of the highest quality pixels, according to a criterion of maximum anisotropy.

The present technique can be used in color images by simply applying the procedure described here to each of the color channels e.g. RGB (red, green and blue) or in any other color space, although the best results will be obtained in the luminance channel which is where the highest degree of discrimination can be obtained.

In the same way, the described technique can be applied to video sequences (that is, a succession of frames in time), applying it to a set of frames and moving them successively in time.

The technique described here can be applied in ophthalmology for the exploration of the fundus with the presence of cataracts, since these introduce a turbidity in the lens that can be modeled from a analogous to that of the cloud cover or fog in the case of images in remote perception. For this, it is necessary to have a set of frames and thus be able to apply the developed fusion technique.

Finally, it should be noted that in order to obtain the best results, it is preferable to process uncompressed images. In the event that these have been compressed with e.g. The method known under the name JPEG, the compression rate should be low and should not exceed 80: 100 so that the fusion process can provide satisfactory results.

EXAMPLES OF THE EMBODIMENT OF THE INVENTION

We present here, by way of example, three cases of practical use of the technique developed in the especially specific case of coastal video surveillance sequences and remote sensing. In fact, the first scenario consists of the improvement and suppression of noise in coastal video surveillance images (Figures 2-4). In the second scenario, the presence of the mist is even more severe than in the previous case (Fig. 5). However, as a result of applying the described method, it is possible to make a good discrimination of the characters of the registration of the vessel (Fig. 6). We must emphasize the improvement obtained in the quality of images compared to other existing methods that do not consider a modeling of the existing noise and that are based on the equalization of the histogram (Figures 2-3). The third scenario consists of a sequence of multi-temporal images represented in Figure 7. Figure 8 presents the result of the application of the method presented in which a very significant elimination of the cloud cover is observed. DESCRIPTION OF THE FIGURES.

Fig. 1. General scheme of the proposed method. After a previous registration stage, the anisotropy measurement of each of them is extracted, and then the image is improved after a fusion process.

Fig. 2. Example of a foggy frame taken from a video surveillance sequence, together with its gray level histogram Fig. 3. Result of performing a histogram equalization. Note that this operation does not produce a significant improvement, but instead a worsening of the contrast without a noise reduction Fig. 4 Result of applying the proposed method. Note how there is an improvement in contrast as well as a reduction in noise. The improvement of the image quality is reflected in the increase in the standard deviation of Table 1.

Fig. 5 Example of a frame with a more pronounced mist cover than in the previous example extracted from a video surveillance sequence, together with its gray level histogram Fig. 6. Result of applying the proposed method. Observe how both the improvement of the contrast and a reduction of the noise make it possible to discriminate the registration of the boat. The improvement of the image quality is reflected in the increase in the standard deviation of Table 1 [16]. Fig. 7. Sequence of the multi-temporal images corresponding to the area of the Mississippi Delta, from the MODIS satellite and provided by the Marine Remote Sensing of the University of South Florida. Fig. 8. Result of the elimination of the cloud cover.

Table 1

Claims

1. Automatic method of eliminating spatially variant degradations that appear in images or sequences of images due to adverse weather conditions such as fog, fog, aerial images with cloud cover, etc., characterized in that it consists of the following stages:

a) Obtaining the sequence of video images, b) Pre-processing of the sequence of video images, c) Calculation of the anisotropy of the images, d) Fusion of the images.

2. Automatic method described in claim 1, characterized in that in step b), claim 1, the Wigner pseudo-distribution is used, using the expression (1), 1 D implementation, of the Detailed Description in this Patent Report , discernible pattern from the knowledge of the image obtained in stage a).

3. Automatic method described in claim 1, also characterized in that in step b), claim 1, a window of unidimensional data of the image is used, in the calculation of each vector associated with Wigner's pseudo-distribution, according to The expression (2).

4. Automatic method described in claim 1, characterized in that in step c), claim 1, calculates the entropy of the images by the concept of generalized entropy of Rényi applied to a space-frequency distribution according to the expression

(3) of the Detailed Description of this Patent Mammal.

5. Automatic method described in claim 1, characterized in that in step d), claim 1, performs the fusion of the images of each image element (pixel) by constructing an improved image by means of the selection of the highest quality elements , according to a criterion of maximum entropy.

6. Automatic method described in claims 1, 2, 3, 4 and 5 characterized in that it allows its application to each of the frames of a video sequence.

7. Automatic method described in claims 1, 2, 3, 4, 5 and 6 characterized in that it allows the treatment of images of interest, such as for example the detection of the characters of a ship's license plate, in maritime video surveillance.

8. Automatic method described in claims 1, 2, 3, 4, 5, 6 and 7, characterized in that it is applicable to signals endowed with color.

9. Automatic method described in claims 1, 2, 3, 4, 5, 6, 7 and 8, characterized in that it is extensible to frames or video sequences that have been previously compressed, provided that the compression rate does not exceed 80: 100

10. Automatic method described in claims 1, 2, 3, 4, 5, 6, 7, 8 and 9, characterized by being extensible to the exploration, or examination, of fundus images when the existence of cataracts prevents Ia correct vision