METHOD FOR IN-SITU FOCUS-FUSION MULTI-LAYER SPECTRAL IMAGING
AND ANALYSIS OF PARTICULATE SAMPLES
This is a Continuation-in-Part of U.S. Patent Application No. 09/146,361, filed on
September 03, 1998.
FIELD AND BACKGROUND OF THE INVENTION
The present invention relates to methods of imaging and analysis of particles and, in particular, to a method for in-situ focus-fusion multi-layer spectral imaging and analysis of particulate samples.
When a sample featuring, for example, a particle, an aggregate of particles, or a dispersion of particles, has large layer or depth variations relative to changes in the distance from which it is viewed, an image of the sample exhibits a layer dependent or spatially varying degree of sharpness. This is referred to as a defocused image of the sample or scene, where some of the objects of the scene are in focus, while other objects of the scene are out of focus. Defocused images contain information potentially useful for scene analysis. The analysis of scenes from defocused images is of general interest in machine vision applications, for example, in active vision or robot vision where a camera actively explores a scene by continuously changing its position, i.e., field of view, relative to scene features. Additionally, scene analysis is of significant practical importance in chemical, pharmaceutical, biomedical, and biological imaging, and general microscopy image analysis, where layer or depth variations of imaged samples of chemicals, powders, frozen suspensions of powders, biological specimens, or other multi-layered particulate samples are typically large compared to imaging distances. Scene analysis is of particular applicability to depth
dependent particulate samples, where, tor instance, one or more layers ot microorganisms such as bacterial or fungal growth, exhibiting fluorescent emission properties in addition to the fluorescent emission properties of the particles themselves, is present on the particles, and there is a need for separation of imaging and analysis of the microorganisms from imaging and analysis of the host particles.
In conventional scene analysis using methods and systems for imaging particles, for example, for each scene, there is auto-focusing, where a best focal position is determined for use in analyzing or classifying particle properties. For some scenes, this is possible, and a focused image may be obtained in an automatic manner. Typically, an auto-focus module is coupled with a computer controlled mechanism that automatically changes the focal position, by moving along an axis parallel to the optical axis of the imaging or focusing sensor, thereby enabling identification of a good focal position. For other scenes, a good focal position is not guaranteed to exist and further image processing via focus-fusion is required. When a focused image of a spatially varying or depth dependent scene can not be generated by using such electro-mechanical microscopy means, whereby a single focal position can not be identified, a focused representation of the scene can be constructed by combining or fusing several defocused images of the same scene. This process is referred to as focus-fusion imaging, and the resulting image of such processing is referred to as a focus-fusion image. Defocused images, for example, those acquired during auto-focusing, are fused together such that each target in a given scene is in correct focus. Scene targets are detected by analyzing either the focused image, if it exists, or the focus-fusion image.
A current technique of imaging particles, for example, featuring chemical and/or biological species, is based on spectral imaging. In spectral imaging, a particulate sample is affected in a way, for example, excitation by incident ultraviolet light upon the sample, which causes the sample to emit light featuring an emission spectra. Emitted light is recorded by an instrument such as a scanning interferometer that generates a set of interferogram images, which in turn are used to produce a spectral image, or image cube, of the sample. A spectral image, or image cube, is a
three dimensional data set (a volume) of voxels in which two dimensions are the spatial dimensions of the sample and the third dimension is the wavelength of the imaged light, such that coordinates of a voxel in a spectral image or image cube may be represented as (x,y,λ). At any particular wavelength, a particulate sample is imaged in two dimensions, so that voxels corresponding to that wavelength constitute the pixels of a monochromatic image of the sample at that wavelength. The spectral image is analyzed to produce a two dimensional map of the chemical or biological composition, or of some other physicochemical property of the sample, for example, particle size distribution. An example of a method and system for real-time, on-line chemical analysis of particulate samples, for example, polycyclic aromatic hydrocarbon (PAH) particles in aerosols, in which the PAH sample is excited to emit light, for example fluorescence, is that of U.S. Patent No. 5,880,830, issued to Schechter, and manufactured by GreenNision Systems Ltd. of Tel Aviv, Israel, and is incorporated by reference for all purposes as if fully set forth herein. In the disclosed method, spectral imaging techniques are implemented to acquire an image and analyze the properties of fixed position PAH particles. As part of this method, PAH particles are first collected by drawing a large volume of air containing PAHs through a filter, followed by on-line scene analysis of the stationary particles. A method of calibration and real-time analysis of particles is described in U.S.
Patent Application No. 09/146,361 , filed September 03, 1998, and is incorporated by reference for all purposes as if fully set forth herein. The method described, is based on using essentially the same system of U.S. Pat. No. 5,880,830, for acquiring spectral images of static particles on a filter. Targets are identified in static particle images and are classified according to morphology type and spectrum type. Each target is assigned a value of an extensive property. A descriptor vector is formed, where each element of the descriptor vector is the sum of the extensive property values for one target class. The descriptor vector is transformed, for example, to a vector of mass concentrations of chemical species of interest, or of number concentrations of biological species of interest, using a relationship determined in a calibration
procedure. In the calibration procedure, spectral images of calibration samples of static particles having known composition are acquired, and empirical morphology types and spectrum types are inferred from the spectral images. Targets are identified in the calibration spectral images, classified according to morphology type and spectrum type, and assigned values of an extensive property. For each calibration sample, a calibration descriptor vector and a calibration concentration vector is formed. A collective relationship between the calibration descriptor vectors and the calibration concentration vectors is found using chemometric methods.
In the method of U.S. Patent Application No. 09/146,361 , standard spectra are determined empirically in the calibration procedure. In such analytical procedures, empirical calibration is quite important for leading to highly accurate results based on image analysis and classification, because spectra of adsorbed chemical species in general, and, of PAHs in particular, are known to be altered by the surfaces on which they are adsorbed, and by the presence of contaminants during sample preparation and image acquisition. Moreover, in the described method, the relationship between the descriptor vector and the concentration vector accounts explicitly and simultaneously for both morphologies and empirically determined spectra. This is particularly important in cases where fluorescence spectra of crystal particles are known to depend on crystal morphology, in general, and crystal size, in particular. Spectral imaging of spatially varying, depth dependent, or multi-layered samples of particles is not described in the above referenced methods and systems. Imaging and image analysis of a random single layer of a sample including particles are ordinarily straightforward. However, multi-layer imaging and image analysis of depth dependent particulate samples, for example, multi-layered dry particles, or particles in a frozen or immobilized suspension, are substantially more complex, for the reasons stated above. More often than not, images obtained of such particulate samples are defocused, and require special image processing techniques, such as focus-fusion, for obtaining useful information about the samples. Nevertheless, there are instances where it is necessary to obtain property and classification information of depth dependent particulate samples, in-situ, for example, as part of sampling an
industrial process. In principle, a sample of dispersed or multi-layered particles is amenable to three-dimensional imaging and scene analysis. In practice, however, for depth dependent samples of particles, spectral imaging as presently practiced would involve tedious methodologies and system manipulations, making acquisition of high resolution images impossible or at best impracticable.
Scene analysis via focus-fusion of defocused images, acquired by multi-layer spectral imaging of depth dependent particulate samples would be quite useful for detecting and classifying in-situ physicochemical information of the particles, such as particle size distribution, morphological features, including structure, form, and shape characteristics, and, chemical and biological composition, which ideally involve multi-layer three-dimensional image analysis. For fusing defocused images, current focus-fusion procedures and algorithms typically involve information and parameters relating only to the extent to which acquired images are either focused or defocused, without inclusion of additional information and parameters relating to particular properties or characteristics of the imaged object or sample. Characteristic sample physicochemical and spectral information and parameters can be quite relevant to imaging particulate samples, and ought to be included in a method of focus-fusion of acquired images of such samples. This is especially the case for images of chemical and/or biological particulate samples featuring layer dependent or spatially varying degree of sharpness.
There is thus a recognized need for, and it would be highly advantageous to have, a method for in-situ focus-fusion multi-layer spectral imaging and analysis of depth dependent particulate samples.
SUMMARY OF THE INVENTION
The present invention relates to a method for in-situ focus-fusion multi-layer spectral imaging and analysis of depth dependent particulate samples, where a given sample features chemical and/or biological species. A unique method of focus-fusion is applied to focused and defocused images acquired from multi-layer spectral imaging of a depth dependent particulate sample, in order to construct focused fused cube
spectral image representations of the imaged particles, thereby generating a focused image of essentially each particle in the sample. The method of the present invention features the use of a uniquely defined and calculated focus-fusion factor parameter which combines empirically determined particle morphological characteristics with empirically determined particle spectral characteristics, and is used in critical steps of image detection and image analysis classification. This uniquely determined parameter enables achievement of high levels of accuracy and precision in detection and classification of the sample, in general, and of the featured particles, in particular. The method includes collecting and analyzing physicochemical and multi-layer spectral data relating to the particles in the sample, including mapping of three-dimensional positions of particles, particle sizes, and characteristics of particle emission spectra. Scene information, in the form of spectral fingerprints, derived from analysis of focus-fusion of the multi-layer spectral images is further processed in order to generate usable in-situ physicochemical information of the particles, such as particle size distribution, moφhological features, including structure, form, and shape characteristics, and, chemical and biological composition. The focus-fusion multi-layer spectral image analysis includes a sophisticated classification procedure for extracting, on-line, useful information relating to particle properties and characteristics needed for generating a report applicable to monitoring or controlling an industrial process.
The method of the present invention enables multi-layer spectral imaging, multi-layer scene analysis, and multi-layer physicochemical characterization of particulate samples featuring depth dependency, which until now has not been described. The present invention is of significant practical importance in chemical, pharmaceutical, biomedical, and biological imaging, and general microscopy image analysis, where layer or depth variations of imaged samples of chemicals, powders, frozen suspensions of powders, biological specimens, or other multi-layered particulate samples are typically large compared to imaging distances.
According to the present invention, there is provided a method for multi-layer imaging and analyzing a sample featuring particles, imaged particles exhibiting a
spatially varying degree of shaφness, the method composing the steps of: (a) providing a spectroscopic imaging system, including a sample holder movable by a three dimensional translation stage; (b) defining imaging scenario parameters; (c) adjusting and setting the imaging system for imaging at a selected field of view, FON„ having central x, y coordinates; (d) focusing the imaging system by moving the translation stage an increment Δz, until receiving a shaφ gray level image of the sample at a selected focal distance Δz,; (e) at the selected FON, and at the selected Δz„ acquiring a cube image of the sample, the cube image featuring a plurality of pixels, each of the plurality of pixels having at least one common visual property, each of the plurality of pixels having a location in the cube image; (f) detecting a plurality of targets for each of the plurality of pixels, each of the plurality of targets created by a plurality of pixels having a pre-defined measured intensity above the imaged background intensity, each target defined as a Blob ; (g) calculating a set of empirically determined moφhological and biological parameters and a set of spectral parameters for each Blobk; (h) calculating a focus-fusion factor parameter, F„ from the set of moφhological and biological parameters and from the set of spectral parameters for each Blobk; (i) grouping all the Blobs into a single Blob neighborhood, according to a neighborhood distance parameter, ΔD; (j) calculating a set of inter-Blob distances, Δd.., for all the Blobs in the Blob neighborhood, each inter-Blob distance measured between each grouped Blobk and a neighboring grouped Blob) to the grouped Blob ; (k) selecting a set of high content Blobs from the Blob neighborhood, according to decisions made by using the focus-fusion parameter, the neighborhood distance parameter, and the set of the inter-Blob distances; (1) saving data of the cube image in a cube image database; (m) repeating step (d) through step (1) in same FON„ at another Δz, until a selected depth of the sample is incrementally imaged; (n) constructing a fused cube image from the cube image database and empirically determined background parameters, B,, and locate the position of each high content Blob with x, y coordinates of central gravity point; and (o) saving data of the fused cube image in a fused cube image database to be used for image analysis of the sample.
According to the present invention, there is provided a method for multi-layer imaging and analyzing a sample featuring particles, imaged particles exhibiting a spatially varying degree of shaφness, the method comprising the steps of: (a) at a selected field of view, FON„ and at a selected focal distance, Δz„ acquiring a cube image of the sample, the cube image featuring a plurality of pixels, each of the plurality of pixels having at least one common visual property, each of the plurality of pixels having a location in the cube image; (b) detecting a plurality of targets for each of the plurality of pixels, each of the plurality of targets created by a plurality of pixels having a pre-defined measured intensity above the imaged background intensity, each target defined as a Blobk; (c) calculating a set of empirically determined moφhological and biological parameters and a set of spectral parameters for each Blobk; (d) calculating a focus-fusion factor parameter, F,, from the set of moφhological and biological parameters and from the set of spectral parameters for each Blob; (e) grouping all the Blobs into a single Blob neighborhood, according to a neighborhood distance parameter, ΔD; (f) calculating a set of inter-Blob distances, Δd,., for all the Blobs in the Blob neighborhood, each inter-Blob distance measured between each grouped Blobk and a neighboring grouped Blob! to the grouped Blobk; (g) selecting a set of high content Blobs from the Blob neighborhood, according to decisions made by using the focus-fusion parameter, the neighborhood distance parameter, and the set of the inter-Blob distances; (h) saving data of the cube image in a cube image database; (i) repeating step (a) through step (h) in same FON„ at another Δz, until a selected depth of the sample is incrementally imaged; (j) constructing a fused cube image from the cube image database and empirically determined background parameters, B„ and locate the position of each high content Blob with x, y coordinates of central gravity point; and (k) saving data of the fused cube image in a fused cube image database to be used for image analysis of the sample.
BRIEF DESCRIPTION OF THE DRAWINGS
The invention is herein described, by way of example only, with reference to the accompanying drawings, wherein: FIG. 1 is a flow diagram of an exemplary preferred embodiment of the method for in-situ focus-fusion multi-layer spectral imaging and analysis of particulate samples, in accordance with the present invention; and
FIG. 2 is a schematic diagram illustrating implementation of selected steps of the preferred embodiment of the method for in-situ focus-fusion multi-layer spectral imaging and analysis of particulate samples, in accordance with the present invention.
DESCRIPTION OF THE PREFERRED EMBODIMENTS
The present invention is of a method for in-situ focus-fusion multi-layer spectral imaging and analysis of particulate samples. Steps and implementation of the method according to the present invention are better understood with reference to the drawings and the accompanying description. It is to be noted that illustrations of the present invention shown here are for illustrative puφoses only and are not meant to be limiting.
Referring now to the drawings, FIG. 1 is a flow diagram of an exemplary preferred embodiment of the method for in-situ focus-fusion multi-layer spectral imaging and analysis of particulate samples. In FIG. 1 , each generally applicable, principle step of the method of the present invention is numbered and enclosed inside a frame. A sub-step representing further of an indicated principle step of the method are indicated by a letter in parentheses. FIG. 2 is a schematic diagram illustrating implementation of selected steps of the preferred embodiment of the method for in-situ focus-fusion multi-layer spectral imaging and analysis of particulate samples. Referenced items shown in FIG. 2 relevant to understanding the method of FIG. 1 are referred to and described in the description of FIG. 1.
In Step (1), a sample 10 (FIG. 2) featuring particles is provided, and prepared for multi-layer spectral imaging and analysis. Sample 10, could be, for example, a
pure powder or a powder mixture, a frozen suspension of a powder, a biological specimen, or some other multi-layered particulate sample, and features a three dimensional topography using coordinate system 12 as a reference, whereby there are layer or depth variations along sample height 14 which are relatively large compared to imaging distances. Sample 10 is placed on a sample holder 16, where sample 10 and sample holder 16 are either exposed to ambient conditions, for example, a powdered sample resting on a glass slide without controlled environmental containment, or, are contained in a controlled environment, for example, a frozen suspension maintained at or below the freezing point temperature of such a frozen suspension. In Step (2), a spectroscopic imaging system 18, including a three dimensional translation stage 20 is provided. Examples of a spectroscopic imaging system 18, including peripheral apparatus, and control / data links, appropriate for implementation of the method of the present invention are fully described in U.S. Patent No. 5,880,830, issued to Schechter, and references cited therein, which are incoφorated by reference for all puφoses as if fully set forth herein. Spectroscopic imaging system 18 includes, among other components, an ultraviolet light illumination source, an optical system, a spectroscopic imaging device, a CCD camera having suitable sensitivity and dynamic range, a central control system, and control / data links. In spectroscopic imaging system 18, the light source illuminates particles of sample 10 homogeneously via the optical system, or by directly without inclusion of the optical system. The control system is based on a personal computer, and includes a frame grabber, for acquiring images from the CCD camera, as well as other hardware interface boards for controlling translation stage 20 and the other components of spectroscopic imaging system 18. The software of the control system includes a database of empirically determined moφhology types and spectrum types and codes for implementing the image processing and quantification algorithms described below.
Spectroscopic imaging system 18 includes a three dimensional translation stage 20 used for synchronized electronic three dimensional movement and positioning of sample holder 16, and therefore, sample 10. Translation stage 20 is in electronic communication with spectroscopic imaging system control devices via control / data
links 22. Usage of translation stage 20 enables spectroscopic imaging system 18 to automatically focus and image sample 10 in a plurality of pre-selected fields of view 24, FON„ and along a plurality of pre-selected focal planes or focal distances, Δz,, potentially spanning entire sample height 14. In sub-step (a) of Step (2), imaging scenario parameters to be used for image acquisition and analysis are defined. These include (i) sample physicochemical and biological parameters relating to particle chemical composition and biological composition, such as microorganism count per particle, and related chemistry, and particle moφhology relating to particle sizes and shapes, and, (ii) imaging system scanning parameters, including selected viewing or imaging range of the sample relating to fields of view 24, FON„ and depth dimension or focal distances, Δz)J5 pixel threshold intensity, Blob neighborhood distance, ΔD, and imaging time interval, Δt.
In sub-step (b) of Step (2), calibrations are performed on standard samples with known physicochemical, biological, and spectral imaging characteristics, according to methodology described in pending U.S. Patent Application No. 09/146,361, cited above, which is incoφorated by reference for all puφoses as if fully set forth herein.
Results of the calibrations are used as part of image analysis of test sample 10.
In Step (3), sample 10 is scanned by adjusting and setting spectroscopic imaging system 18 for spectral imaging at a selected field of view 24, FOV„ over sample 10, having central x, y coordinates, by moving translation stage 20 an increment of Δx and Δy.
In Step (4), imaging system 18 is focused by moving translation stage 20 an increment Δz, until receiving a shaφ gray level image of sample 10, at a selected focal distance Δz,r In Step (5), a plurality of cube (spectral) images, featuring pixels, of sample 10 and particles therein, are acquired. In each cube image, the pixels have at least one common visual property, and each pixel has a location. For acquisition of each cube image, in FON„ at Δzy, the following sub-steps are performed:
In sub-step (a), Blobs, if present in sample 10, are detected, where a Blobk 26 or 28 is defined as a detected target created by a plurality of pixels having a pre-defined
intensity above the imaged background or threshold intensity (defined in Step (2), sub-step (a)). Ordinarily, it is desired that detection of a Blobk 26 or 28 be indicative of either a focused or defocused image of a particle of sample 10. One of the primary tasks of the unique image acquisition procedure of the present invention is to distinguish meaningful or high content Blobs featuring useful information relating to particle characteristics from Blobs featuring non-particle information, such as sample or imaging system contamination.
In sub-step (b), there is calculating a set of moφhological and biological parameters, and a set of spectral parameters, for each Blobk 26 or 28, empirically determined from the Blob image data. Moφhological and biological parameters relate to, for example, the size, area, shape, microorganism count per particle, and x, y position coordinates of central gravity point of a given Blobk, which in turn, relate to particle characteristics in sample 10. Spectral parameters relate to emission characteristics of an imaged particle in sample 10. In sub-step (c), there is calculating a focus-fusion factor parameter, F„ from the set of moφhological and biological, and set of spectral parameters, for each Blobk 26 or 28. The focus-fusion factor parameter uniquely combines particle moφhological and biological information with spectral information, to be used in a decision step for discriminating Blobs from each other. This uniquely determined parameter enables achievement of high levels of accuracy and precision in detection and classification of the sample, in general, and of the featured particles, in particular.
In sub-step (d), all detected Blobs are grouped into a single Blob neighborhood, according to the neighborhood distance parameter, ΔD, defined in Step (2), sub-step (a). In sub-step (e), there is calculating a set of inter-Blob distances 30, Δdkb for all the detected Blobs in the Blob neighborhood, where Δdk] is measured between each grouped Blobk and its neighboring grouped Blobs, in the Blob neighborhood.
In sub-step (f), there is selecting high content Blobs 28 from all the Blobsk 26 and 28 in the Blob neighborhood, according to decisions made by using the focus-fusion factor parameter, F„ ΔD, and the set of Δdk]. These high content Blobs
are to be used in construction of a fused cube image of sample 10, ultimately, providing useful image content relating to particle characteristics.
In sub-step (g), cube image data is saved in a cube image database, for use in construction of a fused cube image of sample 10. In Step (6), Step (4) through Step (5) are repeated, in the same field of view,
FON,, for the selected range of focal distances, Δz,r, by movement of translation stage
20 in increments of Δz. This step enables the acquisition of multi-layer spectral image data of sample 10. For example, in FIG. 2, three cube images 30, 32, and 34, all in same FON, 24, are separately acquired using focal distances Δz,, 36, Δz,2 38, and Δz,3 40, respectively. This procedure illustrates the multi-layer spectral imaging of sample
10 featuring large layer or depth variations relative to changes in the imaging distance. In Step (7), a single fused cube image 42, is constructed, using the cube image database of Step (5), sub-step (g), and using empirically determined background parameters, B„ for selecting the background of fused cube image 42. In fused cube image 42, preferably, only high content Blobs 28 are retained and featured. In sub-step
(a), fused cube image data is saved in a fused cube image database, for use in image analysis algorithms (Step (9)).
In Step (8), there is acquiring and constructing additional fused cube (spectral) images of sample 10 in other fields of view, FONj, by repeating Step (3) through Step (7), until the selected sample viewing / imaging range is imaged. This is, in part, accomplished by programmed movement of translation stage 20 to other fields of view over sample 10, and in each field of view, incremental movement of translation stage
20 to different selected focal distances, Δzir
In Step (9), one or more image analysis algorithms are applied to the database of fused cube images. The plurality of fused cube images are analyzed for spectral fmgeφrints, whereby spectral data is related to applicable physicochemical and biological characteristics of sample 10.
In sub-step (a), detection, classification, and/or decision algorithms are used for image analysis of the fused cube image data. Examples of specific detection, classification, and/or decision algorithms suitable for image analysis in the method of
the present invention are fully described in U.S. Patent No. 5,880,830, issued to Schechter, and in pending U.S. Patent Application No. 09/146,361 , and references cited therein, which are incoφorated by reference for all puφoses as if fully set forth herein. Calibration data of standard samples with known physicochemical, biological, and spectral imaging characteristics are used as part of the image analysis. In the present invention, image analysis is based on uniquely combining physicochemical, for example, moφhological, and, chemical and biological composition data, with multi-layer spectral imaging data of sample 10 featuring particles. This unique combination enables achievement of high levels of accuracy and precision in detection and classification of the sample, in general, and of the featured particles, in particular. In sub-step (b), a statistical analysis report of the image analysis results is generated.
In Step (10), Step (3) through Step (9) are repeated for each pre-defined time interval, Δt. In sub-step (a), a report relating to time variation of the physicochemical, biological, and spectral imaging characteristics of sample 10 featuring particles is generated. This step further enables achievement of high levels of accuracy and precision in detection and classification of the sample.
While the invention has been described in conjunction with a specific embodiment thereof, it is evident that many alternatives, modifications and variations will be apparent to those skilled in the art. Accordingly, it is intended to embrace all such alternatives, modifications and variations that fall within the spirit and broad scope of the appended claims.