US20060023217A1

US20060023217A1 - Method and apparatus for producing a mosaic image

Info

Publication number: US20060023217A1
Application number: US11/216,031
Authority: US
Inventors: Arjun Bangalore; Jason Neiss
Original assignee: ChemImage Corp
Current assignee: ChemImage Corp
Priority date: 2004-05-28
Filing date: 2005-09-01
Publication date: 2006-02-02
Also published as: US20060013500A1; US7705981B2; US20090021730A1; US7479966B2

Abstract

The disclosure relates to method and apparatus for providing a mosaic image based on two or more sub-images. A method according to the disclosure includes irradiating a sample with light to thereby produce photons from the sample and forming two or more sub-images from the photons; the sub-images formed from photons of substantially the same wavelength. The plurality of sub-images of the sample can be combined in a mosaic-type formation to define an image of the sample. Similar images of the sample at different wavelengths can be combined to form a comprehensive mosaic image of the sample at all wavelengths.

Description

The instant application relates to Provisional Application No. 60/575,090 filed May 28, 2004 and application Ser. No. 10/812,233 filed Mar. 29, 2004. The specification of each application is incorporated herein in its entirety.

BACKGROUND

Spectroscopic imaging combines digital imaging and molecular spectroscopy techniques, which can include Raman scattering, fluorescence, photoluminescence, ultraviolet, visible and infrared absorption spectroscopies. When applied to the chemical analysis of materials, spectroscopic imaging is commonly referred to as chemical imaging. Instruments for performing spectroscopic (i.e. chemical) imaging typically comprise image gathering optics, focal plane array imaging detectors and imaging spectrometers.
In general, the sample size determines the choice of image gathering optic. For example, a microscope is typically employed for the analysis of sub micron to millimeter spatial dimension samples. For larger objects, in the range of millimeter to meter dimensions, macro lens optics are appropriate. For samples located within relatively inaccessible environments, flexible fiberscope or rigid borescopes can be employed. For very large scale objects, such as planetary objects, telescopes are appropriate image gathering optics.
For detection of images formed by the various optical systems, two-dimensional, imaging focal plane array (FPA) detectors are typically employed. The choice of FPA detector is governed by the spectroscopic technique employed to characterize the sample of interest. For example, silicon (Si) charge-coupled device (CCD) detectors or CMOS detectors are typically employed with visible, fluorescence and Raman spectroscopic imaging systems, while indium gallium arsenide (InGaAs) FPA detectors are typically employed with near-infrared spectroscopic imaging systems.
Spectroscopic imaging can be implemented by one of two methods. First, a point-source illumination can be provided on the sample to measure the spectra at each point of the illuminated area. Second, spectra can be collected over the an entire area simultaneously using an active optical imaging filter such as AOTF or LCTF. Here, the organic material in the optical filters are actively aligned by applied voltages to produce the desired bandpass and transmission function. Thus, spectral images of a sample are often require accumulation and combination of images of a sample at a number of wavelengths.
The ability to provide a spectral image of a sample is often limited by the field of view of the spectral imaging device. For microscopic imaging the field of view is deliberately made small in order to capture the details of the sample. Because the field of view is limited to a small region of the sample, several such images have to be combined in a mosaic to define the entire sample. Conventional technologies allow image stitching to obtain a panoramic view of individual sequential frames of a scenery in the X-direction. Conventional technologies are inoperable with n-dimensional images or high resolution chemical imaging formed from combination of images at different wavelengths.

SUMMARY OF THE DISCLOSURE

In one embodiment, the disclosure relates to a method for producing an image of an object from overlapping sub-images where each sub-image includes plural pixels each having an initial intensity value, comprising irradiating the object with light to thereby produce from the object scattered light for each of a plurality of wavelengths; producing a plurality of overlapping sub-images of the object for each of the plurality of wavelengths; and for a first and a second of said overlapping sub-images produced at one of said wavelengths: (i) determining an overlap region; (ii) determining a line of fusion within said overlap region; and (iii) stitching together said first and second sub-images to thereby produce a stitched image.
In another embodiment the disclosure relates to a method for producing a chemical image of an object from overlapping sub-images where each sub-image includes plural pixels each having an initial intensity value, the method comprising the steps of (a) irradiating the object with light to thereby produce from the object scattered light for each of a plurality of wavelengths; (b) producing a plurality of overlapping sub-images of the object for each of the plurality of wavelengths; and (c) for a first and a second of said overlapping sub-images produced at one of said wavelengths: (i) determining an overlap region; (ii) determining a line of fusion within said overlap region; and (iii) stitching together said first and second sub-images to thereby produce a stitched chemical image.
In still another embodiment, the disclosure relates to a method for producing a Raman image of an object from overlapping sub-images where each sub-image includes plural pixels each having an initial intensity value, the method comprising the steps of: (a) irradiating the object with light to thereby produce from the object Raman scattered light for each of a plurality of wavelengths; (b) producing a plurality of overlapping sub-images of the object for each of the plurality of wavelengths; and (c) for a first and a second of said overlapping sub-images produced at one of said wavelengths: (i) determining an overlap region; (ii) determining a line of fusion within said overlap region; (iii) stitching together said first and second sub-images to thereby produce a stitched Raman image by performing the steps of: (A) copying the intensity value of pixels located in said first sub-image between an edge opposite the line of fusion and the line of fusion; (B) copying the intensity value of pixels located in said second sub-image between the line of fusion and an edge opposite the line of fusion; and (C) creating a stitched chemical image by combining the copied intensity values from said first and second sub-images; (iv) determining if the line of fusion in the stitched Raman image meets a predetermined criteria for requiring correction; and (v) correcting the stitched Raman image if the predetermined criteria is met, by performing the steps of: (A) defining a window in the stitched Raman image which contains the line of fusion; (B) obtaining, for one of the pixels in said window, the intensity value of a corresponding pixel in each of the first and second sub-images; (C) determining a weighted sum intensity value for the corresponding pixels; and (D) replacing the intensity value of said one pixel in said window with the weighted sum intensity value.
Another embodiment the of the disclosure relates to a spectroscope for producing an image of an object from overlapping sub-images where each sub-image includes plural pixels each having an initial intensity value, the spectroscope comprising (a) a photon source for irradiating the object with light to thereby produce from the object scattered light for each of a plurality of wavelengths; (b) a photon detector for producing a plurality of overlapping sub-images of the object for each of the plurality of wavelengths; and (c) a processor programmed to perform a plurality of executable instructions, the instructions comprising: (i) determining an overlap region for a first and a second of said overlapping sub-images produced at one of said wavelengths; (ii) determining a line of fusion within said overlap region; and (iii) stitching together said first and second sub-images to thereby produce a stitched image.
A spectroscope according to one embodiment of the disclosure comprises (a) a photon source for irradiating the object with light to thereby produce from the object Raman scattered light for each of a plurality of wavelengths; (b) a photon detector for producing a plurality of overlapping sub-images of the object for each of the plurality of wavelengths; and (c) a processor programmed to perform a plurality of executable instructions, the instructions comprising: (i) determining an overlap region for a first and a second of said overlapping sub-images produced at one of said wavelengths; (ii) determining a line of fusion within said overlap region; (iii) stitching together said first and second sub-images to thereby produce a stitched Raman image by performing the steps of: (A) copying the intensity value of pixels located in said first sub-image between an edge opposite the line of fusion and the line of fusion; (B) copying the intensity value of pixels located in said second sub-image between the line of fusion and an edge opposite the line of fusion; and (C) creating a stitched chemical image by combining the copied intensity values from said first and second sub-images; (iv) determining if the line of fusion in the stitched Raman image meets a predetermined criteria for requiring correction; and (v) correcting the stitched Raman image if the predetermined criteria is met, by performing the steps of: (A) defining a window in the stitched Raman image which contains the line of fusion; (B) obtaining, for one of the pixels in said window, the intensity value of a corresponding pixel in each of the first and second sub-images; (C) determining a weighted sum intensity value for the corresponding pixels; and (D) replacing the intensity value of said one pixel in said window with the weighted sum intensity value.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a schematic illustration of a mosaic image of a sample;
FIG. 2 is a schematic representation of spectral image formation of a sample;
FIG. 3 shows a method for stitching two sub-images according to one embodiment of the disclosure; and
FIG. 4 schematically illustrates the process of forming a mosaic image of a sample from two sub-images according to one embodiment of the disclosure.

DETAILED DESCRIPTION

The embodiments disclosed herein relate to method and apparatus for producing a mosaic image from several frames or sub-images. The embodiments disclosed herein are suitable for use, among others, in forming a chemical image of a sample.
FIG. 1 is a schematic illustration of a mosaic image of a sample. In FIG. 1, the field of view captures sub-images 110, 120,130 and 140. A combination of sub-images 100, 120, 130 and 140 forms image 100. When applied to chemical imaging, each sub-image 110, 120, 130 and 140 can be collected at one wavelength; alternatively, they can be images at any one of the plurality of wavelengths at which sub-images can be collected. The sub-images may have about 5% spatial overlap. The overlap, if not removed, can cause distortion in the final mosaic image. When applied to chemical imaging methods such as Raman, fluorescence, etc., the overlap can cause serious misrepresentation or mischaracterization of the sample.
A first step in forming a mosaic image is to identify a region (or a window) that contains one or more common features in the sub-images. Next, the sub-images having the common feature are arranged next to each other as a montage or a mosaic (see FIG. 1.) In forming a montage the goal is forming a mosaic image with little or no spatial overlap among frames.
FIG. 2 is a schematic representation of spectral image formation of a sample. Referring to FIG. 2, each of image 1 and image 2 is shown to be a combination of sub-images taken at different wavelengths λ. By adjoining sub-images having the same wavelength, a mosaic image can be formed. By combining the mosaic images of the sample having different wavelengths, a complete image can be formed.
In method according to one embodiment, a spectral image of a sample is formed from overlapping sub-images, collected by irradiating the object with light of varying or constant wavelength. The photons reaching the sample are scattered by the sample, thereby forming scattered photons having different wavelengths than the incident photons. In one embodiment, the disclosure relates to forming a plurality of overlapping sub-images of the sample for each of the plurality of wavelengths. The image is compiled from several sub-images where each sub-image has an a similar wavelength. That is, a first and a second sub-image having the same wavelength are adjoined based on a common reference point to form an image. The process of adjoining the first and the second sub-images is interchangeably referred to as stitching.
Once formed, the image can be inspected for overlaps. Overlap estimation can be implemented by visual inspection or by conventionally-available software. Once the overlapping region is identified, a line of fusion (i.e., the stitch line) within the overlapping window can be identified. The line of fusion may include one or more feature common to both sub-images. To stitch the first and the second sub-images together according to one embodiment of the disclosure, a line of fusion is identified in the overlapping region and pixel intensity values to either side of the line of fusion are obtained to form an image.
FIG. 3 shows a method for stitching two sub-images according to one embodiment of the disclosure. In the exemplary embodiment of FIG. 3, where two sub-images are combined to form an image, the intensity value of each pixel located in the first sub-image between an edge opposite the line of fusion and the line of fusion is used. In one embodiment, the edge opposite the line of fusion for each sub-image is the non-overlapping edge. Specifically, sub images 310 and 320 are formed independently of each other. To combine sub-images, a common reference point can be identified in each sub-image and based thereon the sub-images can be adjoined to form a mosaic image. Next, overlapping region can be estimated visually or by software inspection. Line of fusion 330 can then be identified common to and between sub-images 310 and 320.
According to one embodiment of the disclosure, a mosaic image 350 is formed by copying pixels from each of the sub-images. To this end, pixels to the left of stitch line 330 along in sub-image 310 and pixels to the right of stitch line 330 in sub-image 320 are copied onto image 350 (FIG. 3B). For example, pixel 311 from sub-image 310 is copied to the left of stitch line 330 and pixel 321 from sub-image 320 is copied to the right of stitch line 330 of mosaic image 350 (FIG. 3B). Once the pixels are copied, a determination can be made as to whether additional correction is required. To this end mosaic image 350 or a portion thereof can be assessed against a predefined threshold to determine whether further correction is required.
If correction is deemed required, a window containing at least a portion of stitch line 330 can be defined. In FIG. 3B, window 340 containing stitch line 330 is shown. Within the window a first and a second exemplary pixels are selected such that each pixel falls on one side of stitch line 330. Referring to FIG. 3B, first exemplary pixel 313 and second exemplary pixel 323 are selected and the intensity value of each exemplary pixel is determined. Next, a weighted-sum intensity value is determined as a function of the intensity values of exemplary pixels 313 and 323 from the original sub-images 310 and 320. Finally, to correct the stitched image, the intensity value of each pixel in window 340 is replaced with the weighted-sum intensity value. While in the illustrative example of FIG. 3, the intensity of window 340 is identified and replaced with a weighted-sum intensity, the scope of the disclosure is not limited thereto.
The embodiments of the disclosure are particularly suitable for spectral imaging of a sample. As stated, obtaining a refined image of the entire sample may require a compilation of several sub-images. To produce a chemical image of an object from overlapping sub-images in accordance with one embodiment of the disclosure, the sample is illuminated by photons. The illuminating photons can have different wavelengths consistent with the intended form of spectroscopy. Once illuminated the sample scatters photons of different wavelength. Forming a spectroscopic image of the sample may require combining two or more sub-images having substantially identical wavelengths as a mosaic.
FIG. 4 schematically illustrates the process of forming a mosaic image of a sample from two sub-images according to one embodiment of the disclosure. In FIG. 4, sub-images 410 and 420 have been collected from a sample. Sub-images 410 and 420 can define Raman, Fluorescence, or other forms of chemical imaging. Moreover, the sub-images can define photons scattered from the sample at substantially the same wavelength. As can be seen from FIG. 4, sub-images 410 and 420 comprise an overlapping region 415. As before, the overlapping region can be identified through visual inspection or by software. Once an overlapping region in each of the sub-images has been identified, a line of fusion within each overlapping window 415 can be defined. An exemplary line of fusion 425 is shown in each of sub-images 410 and 420. By adjoining sub-images 410 and 420 along fusion line 415, mosaic image 430 can be formed. For reference, fusion line 415 as well as overlapping region 425 are also identified at image 430.
Once mosaic image 430 is formed, the image can be inspected for quality. The inspection can assess whether the line of fusion in the stitched image meets a predetermined criteria for requiring correction. If the overall intensity values at and on either side the line of fusion inside the window 415 is approximately similar, then the mosaic image would appear seamless. If not, there will be discontinuity of intensity values at the line of fusion which appear as a distinct line (also called a seam). Visual inspection or automated analysis of the mosaic image can reveal the presence or absence of seam. If the seam is present, then further correction is necessary for its removal.
Should the line of fusion in the stitched image fail to meet the predetermined criteria, the mosaic image can be corrected. According to an embodiment of the disclosure, the stitched image can be corrected by defining a window in the stitched image which contains the line of fusion. Referring to image 430, window 435 (shown in broken lines) surrounds line of fusion 425. Within window 435 a randomly-selected pixel can be selected and its corresponding pixels in each of first sub-image 410 and second sub-image 420 identified. A corresponding pixel is one that appeared in the original sub-image(s). Referring to the embodiment of FIG. 4, the intensity value associated with each of the corresponding pixels can be used to determine a weighted sum intensity value for the randomly-selected pixel within window 435. Once the weighted sum intensity value has been determined, the intensity value of the randomly-selected pixel can be replaced therewith in order to correct the stitched image 430.
According to another embodiment of the disclosure, the process of correcting the stitched image may include determining the weighted sum intensity value for all pixels within line of fusion 425, window 435 or overlapping area 415 and replacing each actual pixel intensity with the weighted sum intensity value determined based on the pixel's corresponding pixels as appearing in each original sub-image. Referring to FIG. 4, mosaic image 440 illustrates the corrected version of mosaic image 430. The mosaic image 440 provides an image of a sample at one wavelength. To obtain a comprehensive mosaic image of the sample, the process can be repeated for images having different wavelengths and the mosaic images can be combined to form a comprehensive mosaic image of the sample.
While the disclosure has been described in relation to exemplary embodiments and specific examples, it should be noted that the principles disclosed herein are not limited to the enumerated embodiments and include any variation, permutation and modification thereto.

Claims

1. A method for producing an image of an object from overlapping sub-images where each sub-image includes plural pixels each having an initial intensity value, comprising:

(a) irradiating the object with light to thereby produce from the object light for each of a plurality of wavelengths;

(b) producing a plurality of overlapping sub-images of the object for each of the plurality of wavelengths; and

(c) for a first and a second of said overlapping sub-images produced at one of said wavelengths:

(i) determining an overlap region;

(ii) determining a line of fusion within said overlap region; and

(iii) stitching together said first and second sub-images to thereby produce a stitched image.

2. The method of claim 1 further comprising the steps of:

(iv) determining if the line of fusion in the stitched image meets a predetermined criteria for requiring correction; and

(v) correcting the stitched image if the predetermined criteria is met.

3. The method of claim 2 wherein the step of correcting the stitched image comprises the steps of:

(A) defining a window in the stitched image which contains the line of fusion;

(B) obtaining, for one of the pixels in said window, the intensity value of a corresponding pixel in each of the first and second sub-images;

(C) determining a weighted sum intensity value for the corresponding pixels; and

(D) replacing the intensity value of said one pixel in said window with the weighted sum intensity value.

4. The method of claim 1 wherein the step of stitching together said first and second sub-images comprises steps of:

(A) copying the intensity value of pixels located in said first sub-image between an edge opposite the line of fusion and the line of fusion;

(B) copying the intensity value of pixels located in said second sub-image between the line of fusion and an edge opposite the line of fusion; and

(C) creating a stitched image by combing the copied intensity values from said first and second sub-images.

5. The method of claim 1 including the step of locating the object so that the image of the object will appear registered properly in the stitched image.

6. The method of claim 5 wherein the width of the stitched image is less than the combined width of the first and second sub-images.

7. A method for producing a chemical image of an object from overlapping sub-images where each sub-image includes plural pixels each having an initial intensity value, the method comprising the steps of:

(c) for a first and second of said overlapping sub-images produced at one of said wavelengths:

(i) determining an overlap region;

(ii) determining a line of fusion within said overlap region; and

(iii) stitching together said first and second sub-images to thereby produce a stitched chemical image.

8. The method of claim 7 further comprising the steps of:

(iv) determining if the line of fusion in the stitched chemical image meets a predetermined criteria for requiring correction; and

(v) correcting the stitched chemical image if the predetermined criteria is met.

9. The method of claim 8 wherein the step of correcting the stitched chemical image comprises the steps of:

(A) defining a window in the stitched chemical image which contains the line of fusion;

(C) determining a weighted sum intensity value for the corresponding pixels;

(D) replacing the intensity value of said one pixel in said window with the weighted sum intensity value; and

(E) replacing all the pixel intensity values of the said window with the weighted sum intensity values.

10. The method of claim 7 wherein the step of stitching together said first and second sub-images comprises the steps of:

(C) creating a stitched chemical image by combining the copied intensity values from said first and second sub-images.

11. The method of claim 7 including the step of locating the object so that the chemical image of the object will appear registered in the stitched chemical image.

12. The method of claim 11 wherein the width of the stitched chemical image is less than the combined width of the first and second sub-images.

13. The method of claim 7 wherein the chemical image is a Raman image.

14. A method for producing a Raman image of an object from overlapping sub-images where each sub-image includes plural pixels each having an initial intensity value, the method comprising the steps of:

(a) irradiating the object with light to thereby produce from the object Raman scattered light for each of a plurality of wavelengths;

(i) determining an overlap region;

(ii) determining a line of fusion within said overlap region;

(iii) stitching together said first and second sub-images to thereby produce a stitched Raman image by performing the steps of:

(C) creating a stitched chemical image by combining the copied intensity values from said first and second sub-images;

(iv) determining if the line of fusion in the stitched Raman image meets a predetermined criteria for requiring correction; and

(v) correcting the stitched Raman image if the predetermined criteria is met, by performing the steps of:

(A) defining a window in the stitched Raman image which contains the line of fusion;

15. The method of claim 14 including the step of locating the object so that the Raman image of the object will appear substantially in the center of the stitched Raman image.

16. The method of claim 15 wherein the width of the stitched Raman image is less than the combined width of the first and second sub-images.

17. The method of claim 15 further comprising the step of: (E) replacing all the pixel intensity value of the said window with the weighted sum intensity values.

18. An imaging spectroscope for producing an image of an object from overlapping sub-images where each sub-image includes plural pixels each having an initial intensity value, the spectroscope comprising:

(a) a photon source for irradiating the object with light to thereby produce from the object light for each of a plurality of wavelengths;

(b) a photon detector for producing a plurality of overlapping sub-images of the object for each of the plurality of wavelengths; and

(c) a processor programmed to perform a plurality of executable instructions, the instructions comprising:

(i) determining an overlap region for a first and a second of said overlapping sub-images produced at one of said wavelengths;

(ii) determining a line of fusion within said overlap region; and

19. The spectroscope of claim 18 wherein said processor is further programmed to execute the instructions of:

(v) correcting the stitched image if the predetermined criteria is met.

20. The imaging spectroscope of claim 19 wherein said processor is further programmed so that the instruction of correcting the stitched image comprises the instructions of:

(A) defining a window in the stitched image which contains the line of fusion;

(C) determining a weighted sum intensity value for the corresponding pixels;

(E) Replacing the pixel intensity value of the said window with the weighted sum intensity values.

21. The imaging spectroscope of claim 20 wherein the window is approximately 30 pixels wide.

22. The imaging spectroscope of claim 18 wherein said processor is further programmed so that the instruction of stitching together said first and second sub-images comprises the instructions of:

(C) creating a stitched image by combining the copied intensity values from said first and second sub-images.

23. The imaging spectroscope of claim 18 wherein said light produced from the object includes one or more of fluorescence light, reflected light, refracted light, transmitted light, and scattered light.

24. The imaging spectroscope of claim 18 wherein said image is a chemical image and said stitched image is a stitched chemical image.

25. The imaging spectroscope of claim 18 wherein said image is a Raman image and said stitched image is a stitched Raman image.

26. A imaging spectroscope for producing a Raman image of an object from overlapping sub-images where each sub-image includes plural pixels each having an initial intensity value, the method comprising the steps of:

(a) a photon source for irradiating the object with light to thereby produce from the object Raman scattered light for each of a plurality of wavelengths;

(ii) determining a line of fusion within said overlap region;

(v) correcting the stitched Raman image if the predetermined criteria is met, by performing steps of:

27. The method of claim 26 including the step of locating the object so that the Raman image of the object will appear substantially properly registered the stitched Raman image.

28. The method of claim 27 wherein the width of the stitched Raman image is less than the combined width of the first and second sub-images.

29. The method of claim 26 further comprising the step of: (E) replacing all the pixel intensity value of said window with the weighted sum intensity values.