WO2018154871A1

WO2018154871A1 - Observation device, observation system, and method for controlling observation device

Info

Publication number: WO2018154871A1
Application number: PCT/JP2017/040772
Authority: WO
Inventors: 宏樹網野; 聡司原; 健人原; 加藤　茂; 松音　英明; 児玉　裕; 野中　修
Original assignee: オリンパス株式会社
Priority date: 2017-02-21
Filing date: 2017-11-13
Publication date: 2018-08-30
Also published as: JP6330070B1; JP2018136178A

Abstract

Provided is an observation device (100), comprising: an imaging unit (151); a movement mechanism (160); and a stereoscopic information acquisition unit. The imaging unit (151) includes a first imaging optical system, which is a magnifying optical system and is a subject-side non-telecentric optical system in which an angle formed between one of principal rays on the subject side and the optical axis is 6° or more. The imaging unit (151) uses the first imaging optical system to image a sample (300) and acquire a first image. The movement mechanism (160) changes relative positions of the sample (300) and the imaging unit (151). The stereoscopic information acquisition unit acquires stereoscopic information on the sample (300) on the basis of a plurality of the first images acquired at different positions of the imaging unit (151) and information on respective imaging positions used to acquire the first images.

Description

Observation device, observation system, and control method for observation device

The present invention relates to an observation apparatus, an observation system, and an observation apparatus control method.

Generally, an apparatus is known in which a culture vessel is left in an incubator to obtain an image of cultured cells in the culture vessel. For example, Japanese Patent Application Laid-Open No. 2005-295818 discloses a technique related to a cell culture device that takes an image of the surface of a culture vessel while changing the relative position between the imaging device having a magnifying optical system and the culture vessel.

In addition, there is a demand for acquiring depth information relating to cultured cells. As a technique for acquiring the position of the subject of interest in the optical axis direction, for example, Japanese Unexamined Patent Application Publication No. 2014-238558 discloses the position of the subject of interest in the optical axis direction based on the parallax between images acquired by two cameras. A technique related to an imaging apparatus that acquires the image is disclosed.

Under such circumstances, an imaging device that images the surface of the culture vessel while changing the relative position between the imaging device and the culture vessel, and an imaging device that acquires the position in the optical axis direction of the subject of interest based on the parallax between the imaging And have different optical limitations.

An object of the present invention is to provide an observation apparatus, an observation system, and an observation apparatus control method capable of acquiring depth information of a subject of interest.

According to one aspect of the present invention, the observation apparatus is a first optical system that is an object-side non-telecentric optical system in which the angle formed by any one of the principal rays on the object side and the optical axis is 6 ° or more. An imaging optical system that captures a sample using the first imaging optical system to obtain a first image, a moving mechanism that changes a relative position between the sample and the imaging unit, Three-dimensional information for acquiring three-dimensional information about the sample based on a plurality of the first images acquired at different positions of the imaging unit and information on each imaging position at the time of acquisition of the first image. An acquisition unit.

According to an aspect of the present invention, the observation system includes the observation device and a controller that acquires a user operation result, outputs the operation result to the observation device, and acquires the observation result of the observation device.

According to one aspect of the present invention, the observation apparatus control method is an enlargement optical system and a subject-side non-telecentric optical system in which an angle formed between any principal ray on the subject side and the optical axis is 6 ° or more. Using the imaging unit including a certain first imaging optical system, imaging the sample by using the first imaging optical system to obtain the first image, and the relative relationship between the sample and the imaging unit Based on the change of position, the plurality of first images acquired at different positions of the imaging unit, and information on each imaging position at the time of acquisition of the first image, the sample And acquiring three-dimensional information.

According to the present invention, it is possible to provide an observation apparatus, an observation system, and an observation apparatus control method that can acquire depth information of a subject of interest.

FIG. 1 is a schematic diagram illustrating an example of an outline of the appearance of the observation system according to the first embodiment. FIG. 2 is a block diagram illustrating an outline of a configuration example of the observation system according to the first embodiment. FIG. 3 is a schematic diagram illustrating an example of a positional relationship between an imaging unit including the second imaging optical system according to the first embodiment and a sample. FIG. 4 is a schematic diagram illustrating another example of the positional relationship between the imaging unit including the second imaging optical system according to the first embodiment and the sample. FIG. 5 is a schematic diagram illustrating an example of a positional relationship between an imaging unit including the first imaging optical system according to the first embodiment and a sample. FIG. 6 is a schematic diagram illustrating another example of the positional relationship between the imaging unit including the first imaging optical system according to the first embodiment and the sample. FIG. 7 is a flowchart illustrating an example of the observation apparatus control process according to the first embodiment. FIG. 8 is a flowchart of an example of the count scan process according to the first embodiment. FIG. 9 is a schematic diagram illustrating an example of count scan processing information according to the first embodiment. FIG. 10 is a schematic diagram illustrating an example of a movement pattern of the image acquisition unit in the count scan processing according to the first embodiment. FIG. 11 is a flowchart illustrating an example of the 3D scan processing according to the first embodiment. FIG. 12 is a schematic diagram illustrating an example of 3D scan processing information according to the first embodiment. FIG. 13 is a schematic diagram illustrating an example of a movement pattern of the image acquisition unit in the 3D scan processing according to the first embodiment. FIG. 14 is a flowchart illustrating an example of a controller control process according to the first embodiment. FIG. 15 is a schematic diagram illustrating an example of a positional relationship between an imaging unit including the first imaging optical system according to the second embodiment and a sample. FIG. 16 is a schematic diagram illustrating an example of a side observation image according to the second embodiment. FIG. 17 is a flowchart illustrating an example of a side observation process according to the second embodiment.

[First Embodiment]
<Configuration of observation system>
(Outline of observation system)
A first embodiment of the present invention will be described with reference to the drawings. The observation system according to the present embodiment is a system for photographing a cell, a cell group, a tissue or the like in culture and recording the number, form, etc. of the cell or cell group. An example of an outline of the appearance of the observation system 1 is shown in FIG. 1 as a schematic diagram, and an outline of a configuration example of the observation system 1 is shown in FIG. 2 as a block diagram, and the configuration of the observation system 1 will be described with reference to these figures.

As shown in FIGS. 1 and 2, the observation system 1 includes an observation device 100 and a controller 200. As shown in FIG. 1, the observation apparatus 100 has a substantially flat plate shape. A sample 300 to be observed is arranged on the upper surface of the observation apparatus 100, and the observation apparatus 100 and the sample 300 are installed in, for example, an incubator.

For the following explanation, an X axis and a Y axis that are orthogonal to each other are defined in a plane parallel to a plane on which the sample 300 of the observation apparatus 100 is arranged, and a Z axis (observation axis) is orthogonal to the X axis and the Y axis. ) Is defined.

As shown in FIGS. 1 and 2, the observation apparatus 100 includes a housing 101, a transparent plate 102, an image acquisition unit 150, and a moving mechanism 160. A transparent plate 102 is disposed on the upper surface of the housing 101. The image acquisition unit 150 is provided inside the housing 101 and includes an imaging unit 151 and an illumination unit 155. As illustrated in FIG. 2, the imaging unit 151 includes an imaging optical system 152 and an imaging element 153. The imaging unit 151 generates image data based on an image (subject image) formed on the imaging surface of the imaging element 153 via the imaging optical system 152. The image acquisition unit 150 is moved by the moving mechanism 160 to change the relative position with the sample 300. The image acquisition unit 150 illuminates the sample 300 through the transparent plate 102 while being moved, and photographs the sample 300 to acquire an image of the sample 300. On the other hand, the controller 200 is installed outside the incubator, for example. The observation apparatus 100 and the controller 200 communicate with each other. The controller 200 controls the operation of the observation apparatus 100.

Note that the shooting position in the Z-axis direction may be changed by the moving mechanism 160, or may be changed by changing the in-focus position of the imaging optical system 152. The imaging optical system 152 is preferably a zoom optical system that can change the focal length.

Hereinafter, the case where the imaging optical system 152 is an expansion optical system and the optical axis of the imaging optical system 152 is parallel to the Z axis (observation axis) will be described as an example. The observation apparatus 100 according to the present embodiment performs a 3D scan process for acquiring depth information of the subject of interest, and a count scan process for acquiring the size, number, and the like of the subject of interest.

The imaging optical system 152 according to the present embodiment includes a first imaging optical system that is an optical system having non-telecentricity at least on the subject side. Hereinafter, an image captured using the first imaging optical system is referred to as a first image. For example, in the 3D scanning process, the observation apparatus 100 acquires the first image while moving the image acquisition unit 150 including the first imaging optical system. The observation apparatus 100 acquires, as depth information, the position in the optical axis direction (observation axis direction) of the first imaging optical system of the subject of interest included in the sample 300 based on parallax between imaging performed at different positions. To do. The depth information includes the position in the optical axis direction of the first imaging optical system related to the subject of interest, a three-dimensional model (3D model), and the like.

The imaging optical system 152 according to the present embodiment further includes a second imaging optical system that is an optical system having telecentricity at least on the subject side. Hereinafter, an image captured using the second imaging optical system is referred to as a second image. For example, in the count scan process, the observation apparatus 100 acquires the second image while moving the image acquisition unit 150 including the second imaging optical system. The observation apparatus 100 combines the second images captured at different positions, and acquires a wide range of high pixel images as if they were captured by capturing a wide range. In addition, the observation apparatus 100 acquires the size, number, and the like of the subject of interest based on the second image or the high pixel image.

(About imaging optics)
Here, with respect to imaging performed while the image acquisition unit 150 is moved, each of a case where the first imaging optical system is used and a case where the second imaging optical system is used will be described with reference to FIGS. 3 to 6. . The imaging optical system 152 according to the present embodiment includes a diaphragm 152a and a plurality of lenses including at least an objective lens 152b and an imaging lens 152c. The focal length on the image sensor 153 side of the imaging lens 152c is defined as a focal length Ft.

First, the case where the second imaging optical system is used as the imaging optical system 152 according to the present embodiment will be described. An example of the positional relationship between the imaging unit 151 including the second imaging optical system according to the present embodiment and the sample 300 is shown in FIG. 3 as a schematic diagram. The sample 300 includes a container 310 and a cell 324 that is a subject of interest. Here, a case where the second imaging optical system is a double-sided telecentric optical system will be described as an example. As shown in FIG. 3, the second imaging optical system according to the present embodiment matches the positions of the exit side focal point of the stop 152a and the objective lens 152b with the incident side focal point of the imaging lens 152c, so that the subject side And an optical system having telecentricity on both sides of the image side. In such a double-sided telecentric optical system, a principal ray that has entered parallel to the optical axis of the optical system (a ray passing through the center of the stop 152a) is emitted in parallel to the optical axis of the optical system.

For example, in the state shown in FIG. 3, the point P1 on the cell 324 is located on the optical axis of the second imaging optical system. Of the light rays emitted from the point P1, the light rays (chief rays) that pass through the second imaging optical system so as to pass through the center of the stop 152a and enter the imaging element 153 are principal rays of the second imaging optical system. This is a light ray R1 incident on the objective lens 152b in parallel to the optical axis. The light ray R1 enters the objective lens 152b in parallel with the optical axis of the second imaging optical system, passes through the center of the diaphragm 152a and the imaging lens 152c, and the second imaging optical on the image sensor 153. The light enters the point Q1 located on the optical axis of the system.

Further, from the state shown in FIG. 3, the moving mechanism 160 moves the optical axis of the second imaging optical system in the X direction by the first X movement amount ΔX1 while being maintained parallel to the Z axis (observation axis). FIG. 4 shows a schematic diagram of the state after being applied.

In the state shown in FIG. 4, among the light rays radiated from the point P1, the light rays (chief rays) that can enter the image sensor 153 through the second imaging optical system so as to pass through the center of the stop 152a are The light ray R1 ′ is incident on the objective lens 152b in parallel with the optical axis of the second imaging optical system. Of course, if the mechanism configuration can be tilted, it is possible to apply the tilting movement to the optical axis. Hereinafter, a point on the image sensor 153 on which the light ray R1 ′ is incident is referred to as Q1 ′. Here, when the point P1 is a corresponding point, the amount of change in the image position (second image movement amount) on the imaging surface of the imaging element 153 is the distance between the point Q1 and the point Q1 ′. Further, in the state shown in FIG. 4, a point on the optical axis of the second imaging optical system and whose position in the optical axis direction is equal to the point P1 is defined as a point P11.

Next, a case where the first imaging optical system is used as the imaging optical system 152 according to the present embodiment will be described. An example of the positional relationship between the imaging unit 151 including the first imaging optical system according to the present embodiment and the sample 300 is shown in FIG. 5 as a schematic diagram. Similar to the case shown in FIGS. 3 and 4, the sample 300 includes a container 310 and a cell 324 that is a subject of interest. Here, a case where the first imaging optical system is a subject-side non-telecentric optical system (image-side telecentric optical system) will be described as an example. As shown in FIG. 5, the first imaging optical system according to the present embodiment is configured to reduce the position of the objective lens 152b in the optical axis direction of the first imaging optical system in the double-sided telecentric optical system shown in FIG. This is an optical system in which the telecentricity is broken only on the subject side by moving to the 152a side. In such a subject-side non-telecentric optical system, a light beam (principal light beam) that enters the optical system and is deflected by the objective lens 152b and then passes through the center of the stop 152a is parallel to the optical axis of the optical system. It is injected. Since the exit side focal points of the aperture stop 152a and the objective lens 152b are different from each other, the principal ray (in FIG. 5) that passes through the object side non-telecentric optical system and enters through the portion other than the optical axis. The off-axis principal ray (shown by a broken line) has an optical axis and an inclination of the optical system between the subject of interest and the optical system.

For example, in the state shown in FIG. 5, the position of the point P1 on the cell 324 is on the optical axis of the first imaging optical system. Therefore, out of the light rays emitted from the point P1, the diaphragm 152a of the first imaging optical system. A ray (principal ray) that can enter the image sensor 153 through the center of the image is a ray R2 that passes through the optical axis of the first image pickup optical system and enters the point Q2 on the image sensor 153.

Further, from the state shown in FIG. 5, the moving mechanism 160 moves the first imaging optical system in the X direction by the second X movement amount ΔX2 while maintaining the optical axis of the first imaging optical system in parallel with the Z axis (observation axis). FIG. 6 shows a schematic diagram of the state after being applied.

In the state shown in FIG. 6, since the position of the point P1 is not on the optical axis of the first imaging optical system, the principal ray out of the rays emitted from the point P1 has an inclination of the angle φ with respect to the optical axis. It is a light ray R2 ′ that is incident on the objective lens 152b. Thereafter, the light ray R2 ′ is deflected by the objective lens 152b, passes through the center of the stop 152a, and enters the point Q2 ′ on the image sensor 153. Here, when the point P1 is a corresponding point, the amount of change in the image position (first image movement amount ΔX) on the imaging surface of the imaging element 153 is the distance between the point Q2 and the point Q2 ′. . In the state shown in FIG. 6, a point on the optical axis of the first imaging optical system and whose position in the optical axis direction is equal to the point P1 is defined as a point P12.

As described above with reference to FIGS. 3 and 4, when the second imaging optical system is used, each principal ray incident on the imaging element 153 is parallel to the optical axis of the second imaging optical system. 2 is incident on the imaging optical system 2. Therefore, when the relative position between the subject of interest and the optical axis position of the second imaging optical system changes, there is no change (parallax) in the direction of viewing the point P1 from the incident end of the second imaging optical system. When the relative position between the subject of interest and the optical axis position of the second imaging optical system changes, for example, the optical axis position (imaging position) of the second imaging optical system moves from position X1 to position X1 ′. It is. Further, the second image movement amount is equal to a distance obtained by multiplying the interval between the imaging positions at which the second image is acquired (first X movement amount ΔX1) by the magnification of the second imaging optical system. Even if the subject is uneven, the appearance on the imaging surface does not change. This can also be expressed as the second image movement amount not including the image movement amount caused by the parallax that occurs when the parallax exists.

As described above, in the imaging using the second imaging optical system, the pupil is infinite when viewed from the subject side, so that an image (second image) as if seen from a distance is obtained. For example, even when the relative position between the cell 324 and the objective lens 152b differs in the optical axis direction as in the case where the cell has irregularities, the second imaging optical system is parallel to the optical axis of the second imaging optical system. The optical path of the light incident on the system does not change. Therefore, when combining a plurality of second images acquired while changing the imaging position to synthesize a wide range of high-pixel images, it is easy to process the joints between the images. Needless to say, based on a wide range of images, it is easy to compare the count of the number of cells and the size of the cells present in the imaging region.

On the other hand, when the shape of the cell 324 or the cell group is uneven with respect to the optical axis direction of the imaging optical system 152, there is a demand for acquiring the depth information of the subject of interest. However, as described above, in the subject-side telecentric optical system such as the second imaging optical system, even if the relative position between the subject of interest and the optical system changes in the optical axis direction, the second image movement amount is Since the amount of image movement due to parallax is not included, the amount of change in the position of the subject of interest in the optical axis direction cannot be acquired. The observation apparatus 100 according to the present embodiment performs imaging using the first imaging optical system when it is desired to acquire depth information.

As described above with reference to FIG. 5 and FIG. 6, a range that is not located on the optical axis of the first imaging optical system among the principal rays incident on the imaging element 153 when the first imaging optical system is used. The principal ray emitted from the optical axis has an inclination with respect to the optical axis. Therefore, for example, when the relative position of the subject of interest and the optical axis position changes so that the optical axis position (imaging position) of the first imaging optical system moves from the position X2 to the position X2 ′, the first Change (parallax) in the direction of viewing the point P1 from the incident end of the optical system. Further, the first image movement amount ΔX is equal to a distance obtained by multiplying the interval between the imaging positions at which the first image is acquired (second X movement amount ΔX2) by the magnification of the first imaging optical system. However, when the subject has irregularities, the magnification on the imaging surface differs for a subject that is in a position before and after the focal plane on the subject side of the objective lens (a surface that includes P1 and is perpendicular to the optical axis). In the example of FIG. 6, the magnification on the imaging surface decreases as the distance from the objective lens 152b increases. That is, the ratio between the second X movement amount ΔX2 and the first image movement amount ΔX changes depending on the difference in the depth of the position of the point P1 to be set. This can also be expressed as the first image movement amount ΔX including an image movement amount caused by the parallax generated when the parallax exists.

As described above, in imaging using the first imaging optical system, due to parallax, for example, based on a plurality of first images including the same corresponding points and having different imaging positions on the same XY plane, for example. A first image movement amount ΔX including the image movement amount is obtained. Therefore, the observation apparatus 100 according to the present embodiment can acquire depth information about a subject of interest such as the cell 324 using the principle of triangulation based on the first image movement amount ΔX, for example. The depth information includes, for example, 3D information and 3D images including information such as thickness and unevenness. In the imaging using the first imaging optical system, such distance distribution and image information in each depth direction can be obtained, so that the object can be confirmed with various information. That is, the imaging unit 151 that acquires an image of the sample 300 using the observation optical system, the moving mechanism 160 that changes the relative position between the sample 300 and the imaging unit 151, and a plurality of images acquired at different positions of the imaging unit 151. And an observation apparatus provided with the three-dimensional information acquisition part which acquires this sample 300 as three-dimensional information based on the information regarding each imaging position at the time of acquisition of this image can be provided. The observation optical system only needs to have an imaging function, and may be applied such that enlargement is electronically enlarged.

(About obtaining depth information)
Here, acquisition of depth information based on the first image acquired in the state shown in FIG. 5 and the first image acquired in the state shown in FIG. 6 will be described. Note that the acquisition interval of these first images is sufficiently shorter than the temporal change of the phenomenon occurring inside the sample 300. For the sake of simplicity, a description will be given by taking as an example a case where the light beam on the incident side and the light exit side of the objective lens 152b are on the same line in the light beam R2 ′ passing through the first optical system shown in FIG. Accordingly, in the following description of the depth information acquisition, the triangle formed by the point S0, the point P1, and the point P12 indicating the center of the diaphragm 152a is the point K1 on the imaging lens 152c through which the point S0 and the light ray R2 ′ pass. And a right triangle composed of a point K0 indicating the center of the imaging lens 152c. Needless to say, even if the incident-side and exit-side rays of the objective lens 152b are not on the same line, the same processing as described below can be performed in consideration of the refraction angle in the objective lens 152b.

In the right triangle composed of the point S0, the point K1, and the point K0, the distance between the point K1 and the point K0 is equal to the first image movement amount ΔX. The first image movement amount ΔX is known because the corresponding point is detected and calculated as the movement amount of the corresponding point on the image plane in the image processing based on at least two first images. Further, the distance between the point S0 and the point K0 is the focal length Fo on the incident side of the imaging lens 152c and is known. On the other hand, in the right triangle composed of the point S0, the point P1, and the point P12, the distance between the point P1 and the point P12 is the distance between the optical axes (second X movement amount ΔX2) and is known.

Therefore, when the difference in position between the diaphragm 152a and the entrance pupil (image of the diaphragm 152a viewed from the subject side) can be ignored, the distance in the optical axis direction of the first imaging optical system between the point S0 and the point P1. Z is calculated as Z = Fo × ΔX2 / ΔX, and depth information of the cell 324 is acquired.

Note that the second imaging optical system shown in FIGS. 3 and 4 and the first imaging optical system shown in FIGS. 5 and 6 are shown as optical systems having telecentricity on the image side, respectively. However, it is not limited to this. As described above, the observation apparatus 100 according to the present embodiment uses the imaging optical system 152 as the first imaging optical system in order to switch whether an image acquired by imaging includes depth information of the cell 324 or not. Or a second imaging optical system. Therefore, even if the first imaging optical system or the second imaging optical system is an optical system having non-telecentricity on the image side, the same effect can be obtained.

In addition, since the first imaging optical system is an optical system having non-telecentricity on the subject side, for example, the first imaging optical system is moved in the optical axis direction of the first imaging optical system in order to focus. In this case, the corresponding point on the image sensor 153 moves. For example, the case where the first imaging optical system and the imaging element 153 are moved together in the direction away from the cell 324 (Z-direction) from the state shown in FIG. 6 will be described as an example. At this time, an angle formed by a straight line passing through the points P1 and S0 and a straight line passing through the points P12 and S0 is small. Therefore, the point K1 moves to the optical axis side, and the point Q2 ′ also moves to the optical axis side. In this way, even when the first imaging optical system is moved in the optical axis direction of the first imaging optical system for focusing or the like in the middle of repeating the imaging while being moved, the corresponding point is moved. Obviously, it is possible to acquire depth information.

For the sake of simplicity, the acquisition of depth information in the optical axis direction when the imaging optical system 152 is moved in the X direction and only focusing on a certain XY plane (the point where the point P1 is imaged). Although explained, it is not limited to this. It is obvious that the first image movement amount ΔX is acquired for a plurality of corresponding points included in the first image, and the depth information on the surface is acquired for the cell 324.

For the sake of simplicity, as shown in FIG. 5, a point P1 is positioned on the optical axis of the first imaging optical system as one of the first images used for obtaining depth information is captured. Although the case has been described as an example, the present invention is not limited to this. For example, in at least two first images used for acquisition of depth information, points that are not located on the optical axis at the time of each imaging may be used as corresponding points.

Generally, it is easy to acquire depth information for an arbitrary point on the subject of interest based on, for example, the result of autofocus. On the other hand, acquiring depth information for an arbitrary surface of a subject of interest based on the result of autofocusing depends on the desired resolution, but requires an autofocus operation for all points included in the surface. It is difficult. Under such circumstances, when the observation apparatus 100 according to the present embodiment acquires depth information using the first imaging optical system, the depth information includes depth information of the surface of the subject of interest, and the subject of interest It becomes easy to obtain a 3D model.

It should be noted that any principal ray on the subject side may be regarded as a subject-side telecentric optical system as long as the angle formed with the optical axis is 4 ° or less.

The subject-side non-telecentric optical system may be regarded as an optical system in which an angle formed by any principal ray on the subject side and the optical axis (angle φ in FIG. 6) is 6 ° or more. An image having a sufficient parallax between the plurality of first images can be obtained.

Furthermore, when taking a peripheral (side) image, it is preferable that the angle (angle φ in FIG. 6) formed by any principal ray on the subject side and the optical axis is 20 ° or more.

(About the sample)
A sample 300 that is a measurement target of the observation system 1 is, for example, as follows. The sample 300 includes, for example, a container 310, a culture medium 322, cells 324, and a reflection plate 360. A medium 322 is placed in the container 310, and cells 324 are cultured in the medium 322. The container 310 can be, for example, a petri dish, a culture flask, a multiwell plate, or the like. Thus, the container 310 is a culture container for culturing a biological sample, for example. The shape, size, etc. of the container 310 are not limited. The medium 322 may be a liquid medium or a solid medium. The measurement object is, for example, the cell 324, but this may be an adhesive cell or a floating cell. The cell 324 may be a spheroid or a tissue. Furthermore, the cell 324 may be derived from any organism, and may be a fungus or the like. Thus, the sample 300 includes a biological sample that is a living organism or a sample derived from a living organism. The reflection plate 360 is for illuminating the cells 324 by reflecting the illumination light incident on the sample 300 via the transparent plate 102, and is disposed on the upper surface of the container 310.

(About observation equipment)
The transparent plate 102 disposed on the upper surface of the casing 101 of the observation apparatus 100 is made of, for example, glass. The observation apparatus 100 is in a state in which the inside is sealed by a member including, for example, a housing 101 and a transparent plate 102. The sample 300 is placed on the transparent plate 102. FIG. 1 shows an example in which the entire upper surface of the housing 101 is formed of a transparent plate, but the observation apparatus 100 is provided with a transparent plate on a part of the upper surface of the housing 101. The other part of the upper surface may be configured to be opaque. In addition, transparency here shows that it is transparent with respect to the wavelength of illumination light.

The moving mechanism 160 includes a support portion 165, an X feed screw 161 for moving the support portion 165 in the X-axis direction, and an X actuator 162. The moving mechanism 160 further includes a Y feed screw 163 and a Y actuator 164 for moving the support portion 165 in the Y-axis direction. The moving mechanism 160 may include a Z feed screw and a Z actuator for moving the support portion 165 in the Z-axis direction. For the following explanation, the direction in which the support portion 165 moves away from the X actuator 162 is defined as the positive direction of the X direction (X + direction), and the direction of movement away from the Y actuator 164 is defined as the positive direction in the Y direction. (Y + direction), and the direction from the support 165 toward the sample 300 is the positive direction of the Z direction (Z + direction).

As shown in FIG. 1, the illumination unit 155 included in the image acquisition unit 150 is provided on the support unit 165 included in the moving mechanism 160. An imaging unit 151 is provided in the vicinity of the illumination unit 155. The illumination unit 155 includes an illumination optical system 156 and a light source 157. The illumination light emitted from the light source 157 is irradiated onto the sample 300 via the illumination optical system 156. The light source 157 includes, for example, an LED.

As shown in FIG. 2, the imaging unit 151 further includes a lens switching unit 154. For example, the lens switching unit 154 drives a lens included in the imaging optical system 152 in the optical axis direction so that the imaging optical system 152 becomes the first imaging optical system or the second imaging optical system. The lens switching unit 154 according to the present embodiment performs, for example, a count scan process that counts the number of cells 324, an imaging process that acquires a wide range of high pixel images by combining a plurality of images acquired at a plurality of positions, and the like. In this case, the imaging optical system 152 is used as the second imaging optical system. In addition, the lens switching unit 154 according to the present embodiment uses the imaging optical system 152 when, for example, 3D scan processing for acquiring depth information of the cell 324, imaging processing for acquiring a three-dimensional image of the cell 324, or the like is performed. 1 imaging optical system.

As described above, the observation apparatus 100 according to the present embodiment causes the lens switching unit 154 to switch the imaging optical system 152 according to the type of observation, and causes the moving mechanism 160 to change the position of the image acquisition unit 150 in the X direction and the Y direction. The sample 300 is repeatedly photographed while changing the optical axis of the imaging optical system 152 in the direction while being kept parallel to the observation axis, and a plurality of images are acquired.

The observation apparatus 100 further includes an observation side recording circuit 130. The observation-side recording circuit 130 records, for example, programs and various parameters used in each unit included in the observation apparatus 100 and data obtained by the observation apparatus 100. The observation-side recording circuit 130 temporarily records various data such as image data (pixel data), image data for recording, image data for display, and processing data during operation.

Furthermore, the observation-side recording circuit 130 records the focus position range in the optical axis direction of the imaging optical system 152 as, for example, a focus position range. For the focus position range, for example, a value corresponding to the size of the sample 300 or the like is set in advance, or is set by a user input.

The observation apparatus 100 further includes an image processing circuit 120. The image processing circuit 120 performs various image processing on the image data obtained by the imaging unit 151. Data after image processing by the image processing circuit 120 is recorded in, for example, the observation-side recording circuit 130 or transmitted to the controller 200. Further, the image processing circuit 120 may perform various analyzes based on the obtained image. For example, the image processing circuit 120 acquires depth information of a cell 324 or a cell group included in the sample 300 based on the obtained first image. For example, the image processing circuit 120 extracts an image of a cell 324 or a cell group included in the sample 300 based on the obtained second image, or calculates the number of cells or a cell group. The analysis result obtained in this way is also recorded in the observation-side recording circuit 130 or transmitted to the controller 200, for example.

In order to perform communication with such a controller 200, the observation apparatus 100 further includes an observation side communication apparatus 140. For this communication, wireless communication using, for example, Wi-Fi (registered trademark) or Bluetooth (registered trademark) is used. Further, the observation apparatus 100 and the controller 200 may be connected to each other by a wired communication to communicate with each other, or may be connected to an electrical communication line such as the Internet and communicate via an electrical communication line such as the Internet. It may be broken.

The observation apparatus 100 further includes an observation-side control circuit 110 and a clock unit 172.

The observation side control circuit 110 controls the operation of each unit included in the observation apparatus 100. In addition, the observation-side control circuit 110 acquires various information related to the operation of the observation apparatus 100, performs various determinations related to the operation of the observation apparatus 100, and notifies and alerts the user based on the determination result. Etc. As shown in FIG. 2, the observation-side control circuit 110 includes a position control unit 111, an imaging control unit 112, an illumination control unit 113, a communication control unit 114, a recording control unit 115, a measurement control unit 116, and a distance conversion unit 117. It has a function. The position control unit 111 controls the operation of the moving mechanism 160 and controls the position of the image acquisition unit 150. The imaging control unit 112 controls the operation of the imaging unit 151 included in the image acquisition unit 150 and causes the imaging unit 151 to acquire an image of the sample 300. The imaging control unit 112 includes a focus / exposure switching unit. The imaging control unit 112 performs focus adjustment by moving a focusing lens included in the imaging optical system 152 in the optical axis direction, for example. The focusing lens may be a lens having a variable focal length such as a liquid lens. A plurality of lenses with different focal points may be prepared for focusing. If the prepared lens is multi-lens, refocusing technology or the like can be used. The focus / exposure switching unit adjusts the exposure by controlling the operation of the diaphragm 152a, for example, and adjusts the zoom by controlling the operation of the lens in the optical axis direction. The illumination control unit 113 controls the operation of the illumination unit 155 included in the image acquisition unit 150. The communication control unit 114 manages communication with the controller 200 via the observation side communication device 140. The recording control unit 115 controls recording of data obtained by the observation apparatus 100. The measurement control unit 116 controls the entire measurement such as the timing and number of times of measurement. The distance conversion unit 117 acquires position information in the optical axis direction of the cell 324 that is the subject of interest, information on the unevenness of the cell 324, and the like as depth information based on the processing result of the image processing circuit 120, for example. The clock unit 172 generates time information and outputs it to the observation side control circuit 110. The time information is used for determination related to the operation of the observation apparatus 100 when recording acquired data, for example.

Note that the observation-side control circuit 110, the image processing circuit 120, the observation-side recording circuit 130, and the observation-side communication device 140 described above are arranged inside the casing 101 as a circuit group 104, for example, as shown in FIG. Is provided.

Moreover, the observation apparatus 100 according to the present embodiment includes functions as a three-dimensional information acquisition unit, a corresponding point acquisition unit, and a three-dimensional model generation unit. The three-dimensional information acquisition unit is used as the three-dimensional information about the sample 300 based on the plurality of first images acquired at the positions of the different imaging units 151 and the information related to the respective imaging positions at the time of acquisition of the first images. get. The three-dimensional information acquisition unit acquires information related to the unevenness of the sample 300, for example. The three-dimensional information acquisition unit acquires, for example, information related to each imaging position at the time of acquiring the first image from the distance conversion unit 117. The three-dimensional information includes depth information acquired by the distance conversion unit 117, for example. The corresponding point acquisition unit acquires the corresponding point based on the correlation between the plurality of first images acquired while the imaging unit 151 is moved by the moving mechanism 160, for example, and acquires the first image movement amount ΔX. . The three-dimensional model generation unit constructs a three-dimensional model of the subject of interest based on the depth information acquired by the distance conversion unit 117, for example. The functions as the three-dimensional information acquisition unit, the corresponding point acquisition unit, and the three-dimensional model generation unit can be realized by the observation-side control circuit 110 and / or the image processing circuit 120, for example.

As described above, by providing the image acquisition unit 150 that generates image data by imaging through the transparent plate 102 and the moving mechanism 160 that moves the image acquisition unit 150 in the housing 101, the reliability is high. The structure can be easily handled and cleaned, and can prevent contamination and the like.

(About the controller)
The controller 200 is, for example, a personal computer (PC), a tablet information terminal, or the like. FIG. 1 illustrates a tablet information terminal.

The controller 200 is provided with an input / output device 270 including a display device 272 such as a liquid crystal display and an input device 274 such as a touch panel. The input device 274 may include a switch, dial, keyboard, mouse, and the like in addition to the touch panel.

Further, the controller 200 is provided with a controller side communication device 240. The controller side communication device 240 is a device for communicating with the observation side communication device 140. The observation apparatus 100 and the controller 200 communicate with each other via the observation side communication apparatus 140 and the controller side communication apparatus 240.

The controller 200 includes a controller-side control circuit 210 and a controller-side recording circuit 230. The controller side control circuit 210 controls the operation of each part of the controller 200. The controller-side recording circuit 230 records, for example, programs used in the controller-side control circuit 210, various parameters, and data received from the observation apparatus 100.

The controller-side control circuit 210 has functions as a system control unit 211, a display control unit 212, a recording control unit 213, a communication control unit 214, and a network cooperation unit 215. The controller-side control circuit 210 may further have functions as a corresponding point acquisition unit and a three-dimensional model generation unit. The system control unit 211 performs various calculations related to control for measurement of the sample 300. The display control unit 212 controls the operation of the display device 272. The display control unit 212 causes the display device 272 to display necessary information and the like. The recording control unit 213 controls information recording in the controller-side recording circuit 230. The communication control unit 214 controls communication with the observation device 100 via the controller side communication device 240. The network cooperation unit 215 controls the cooperation between the observation system 1 and a network server or the like outside the observation system 1 such as a cloud provided on a telecommunication line such as the Internet. In the cooperation, the network cooperation unit 215 provides, for example, an observation result such as an image acquired by the observation device 100 or an observation result acquired by the controller 200 on the observation side communication device 140 or the controller side communication device 240 on the network. Sent to the server. For example, the network cooperation unit 215 causes the image processing circuit or the like included in the network server to perform processing such as cell count and depth information calculation based on the observation result, and acquires the result of the processing. Further, for example, the network cooperation unit 215 acquires information from an IoT device such as an incubator, an air conditioning facility, and a lighting facility connected to the Internet, and controls the IoT device.

The observation-side control circuit 110, the image processing circuit 120, and the controller-side control circuit 210 include an integrated circuit such as Central Processing Unit (CPU), Application Specific Integrated Circuit (ASIC), or Field Programmable Gate Array (FPGA). . The observation side control circuit 110, the image processing circuit 120, and the controller side control circuit 210 may each be configured by one integrated circuit or the like, or may be configured by combining a plurality of integrated circuits. Further, the observation side control circuit 110 and the image processing circuit 120 may be configured by one integrated circuit or the like. In addition, the position control unit 111, the imaging control unit 112, the illumination control unit 113, the communication control unit 114, the recording control unit 115, the measurement control unit 116, and the distance conversion unit 117 of the observation side control circuit 110 are each one integrated circuit. Etc., or a combination of a plurality of integrated circuits or the like. Further, two or more of the position control unit 111, the imaging control unit 112, the illumination control unit 113, the communication control unit 114, the recording control unit 115, the measurement control unit 116, and the distance conversion unit 117 are configured by one integrated circuit or the like. May be. Similarly, the system control unit 211, the display control unit 212, the recording control unit 213, the communication control unit 214, and the network link unit 215 of the controller-side control circuit 210 may each be configured with one integrated circuit or the like. These integrated circuits or the like may be combined. Further, two or more of the system control unit 211, the display control unit 212, the recording control unit 213, the communication control unit 214, and the network link unit 215 may be configured by one integrated circuit or the like. The operation of these integrated circuits is performed according to a program recorded in a recording area in the observation-side recording circuit 130 or the controller-side recording circuit 230 or the integrated circuit, for example.

Note that the observation-side recording circuit 130, the controller-side recording circuit 230, or each of the elements included in the observation-side recording circuit 130 is a non-volatile memory such as a flash memory, for example, but is not limited to Static Random Access Memory (SRAM) Such a volatile memory may be further included. In addition, the observation-side recording circuit 130 or each of the elements included therein and the controller-side recording circuit 230 or each of the elements included in the observation-side recording circuit 130 may be configured by one memory or the like, or a plurality of memories or the like may be combined. It may be configured. Further, a database or the like outside the observation system 1 may be used as a part of the memory.

<Operation of observation system>
An example of an observation apparatus control process performed by the observation apparatus 100 by communicating with the controller 200 is shown as a flowchart in FIG. 7, and the operation of the observation system 1 will be described with reference to this flowchart.

In step S101, the observation-side control circuit 110 stands by until a signal output from the controller 200 according to a user operation is received, for example.

In step S102, the observation-side control circuit 110 determines whether, for example, a power-on signal for turning on the power of the observation apparatus 100 or a power-off signal for turning off the power of the observation apparatus 100 is received from the controller 200. The process proceeds to step S103 if it is determined that the power ON / OFF signal has been received, and proceeds to step S104 if it is not determined that it has been received.

In step S103, when it is determined that the power-on signal is received in step S102, the observation-side control circuit 110 starts supplying power to each part of the observation apparatus 100, and receives the power-off signal in step S102. If determined, the supply of power to each unit of the observation apparatus 100 is terminated. Note that power is continuously supplied to the observation-side communication device 140 in any case in order to wait for communication. Thereafter, the process returns to step S101.

When observing the cell culture, etc., when the change over time of the observation object is gentle, observation such as imaging may be performed at time intervals as necessary. Therefore, such power control contributes to energy saving.

Note that the observation device 100 is a communication device with low power consumption such as Bluetooth Low Energy (BLE) for transmission / reception of control signals and the like, and high-speed communication such as Wi-Fi for transmission / reception of data such as observation results including images. And a device. In this case, when the power of the observation apparatus 100 is off, communication is waited by BLE or the like, and communication such as Wi-Fi is established with the controller 200 when the power is turned on at step S103. That's fine. Further, although it has been described that the power of the observation apparatus 100 is turned on or off based on the power ON / OFF signal output from the controller 200, the present invention is not limited to this. For example, during culturing of cells, observation such as imaging may be performed by turning on or off the observation apparatus 100 at a preset time interval such as every minute.

In step S104, the observation-side control circuit 110 determines whether or not control signals related to various settings are received from the controller 200, for example. The process proceeds to step S105 if it is determined that a control signal related to various settings has been received, and returns to step S101 if it is not determined that it has been received.

In step S105, the observation-side control circuit 110 sets each part of the observation device 100 according to the control signals related to various settings received by the observation-side communication device 140 in step S104. The information set here includes, for example, information related to an observation result such as an image acquired by the observation apparatus 100 or a transmission destination of the measurement result, imaging conditions, measurement conditions, and various parameters. Note that the transmission destination of the observation result or measurement result acquired by the observation apparatus 100 is, for example, the observation-side recording circuit 130 of the observation apparatus 100, the controller-side recording circuit 230 of the controller 200, a data server on the network, or the like. For example, if an observation result or a measurement result is transmitted to a cloud or the like constructed on the network in this way, not only information sharing between different users is facilitated, but also an analysis of an acquired image and an image outside the observation system 1 Processing can be performed.

In step S106, the observation-side control circuit 110 determines whether or not a control signal instructing execution of the count scan process is received from the controller 200, for example. The process proceeds to step S107 when it is determined that a control signal instructing execution of the count scan process has been received, and proceeds to step S108 when it is not determined that it has been received. In the count scan process, for example, a measurement start time or the like is determined in advance, and measurement may be started at the determined measurement start time.

In step S107, the observation-side control circuit 110 executes a count scan process and counts the number of cells 324. Details of the count scan process will be described later. Thereafter, the process proceeds to step S108.

In step S108, the observation-side control circuit 110 determines whether or not a control signal instructing execution of 3D scan processing is received from the controller 200, for example. The process proceeds to step S109 if it is determined that a control signal instructing execution of the count process has been received, and proceeds to step S110 if it is not determined that it has been received.

In step S109, the observation-side control circuit 110 executes a 3D scan process and acquires depth information such as three-dimensional information of the cell 324. Details of the 3D scanning process will be described later. Thereafter, the process proceeds to step S110.

In step S110, the observation-side control circuit 110 determines whether or not to end the processing related to observation or measurement based on, for example, a control signal output by the controller 200 in response to a user operation. The process proceeds to step S111 if it is determined to end, and returns to step S104 if it is determined not to end.

In step S111, the observation-side control circuit 110 determines whether or not a control signal that requests an observation result or a measurement result is received from the controller 200, for example. The observation result or the measurement result includes, for example, various data obtained by the observation apparatus 100 such as a measurement measurement value, an acquired image, a photographing position, and an analysis result. The shooting position includes an X coordinate, a Y coordinate, and a Z coordinate of the shooting position. The X coordinate and the Y coordinate are values used in the control of the moving mechanism 160 and can be acquired from the position control unit 111, for example. The Z coordinate is a value used for controlling the imaging optical system 152, and can be acquired from, for example, the imaging control unit 112, the distance conversion unit 117, and the like. The process proceeds to step S112 if it is determined that a control signal requesting an observation result or a measurement result has been received, and returns to step S101 if it is not determined that it has been received.

In step S112, the observation-side control circuit 110 transmits the results obtained by various observations and measurements such as an acquired image, the analysis results obtained by analyzing the results, and the like to the transmission destination set in step S105, for example. To do. Thereafter, the process returns to step S101.

Here, an example of the count scan process in step S107 of the observation apparatus control process is shown in FIG. 8 as a flowchart, and the operation of the observation system 1 during the count scan process will be described with reference to this.

In step S201, the observation-side control circuit 110 causes the lens switching unit 154 to make the imaging optical system 152 the second imaging optical system. Note that the second imaging optical system is an optical system having telecentricity on the subject side as described above. Thereafter, the process proceeds to step S202.

In step S202, the observation-side control circuit 110 executes pre-processing for starting a count scan based on, for example, the count-scan processing information recorded in the observation-side recording circuit 130. In the preprocessing, the observation side control circuit 110 moves the image acquisition unit 150 to the moving mechanism 160 and returns it to the XY start position of the count scan. In addition, the observation-side control circuit 110 controls the operations of the imaging optical system 152 and the imaging element 153 or the moving mechanism 160 so that the count scan can be started from the initial position in the Z direction. Thereafter, the observation side control circuit 110 starts a count scan.

Here, an example of count scan processing information according to the present embodiment is shown in FIG. 9, and information recorded as count scan processing information will be described with reference to this. The information is set in advance, for example, or set in step S105 of the observation apparatus control process.

As shown in FIG. 9, the count scan processing information includes information CSP relating to the count scan pattern, information CSJ relating to execution of the count scan processing, and information CSR obtained by the count scan processing. The information CSP related to the count scan pattern includes, for example, a count scan start condition CSP1, a start position CSP2, an end condition CSP3, a first X movement pitch CSP5, a first Y movement pitch CSP6, and movement in the X direction to movement in the Y direction. The first X → Y condition CSP10 which is a condition for switching to the X direction, and the first Y → X condition CSP11 which is a condition for switching from the movement in the Y direction to the movement in the X direction. Here, for example, the first X movement pitch CSP5 is a movement (imaging) interval in the X direction, and the first Y movement pitch CSP6 is a movement (imaging) interval in the Y direction. The image acquisition unit 150 according to the present embodiment captures images for each of these movement pitches and acquires a second image. The information CSJ related to the execution of the count scan process is, for example, when it is determined that there is an observation defect based on a first NG determination condition CSJ1, which is a determination condition for determining an observation defect, for example, the first NG determination condition CSJ1. A first retry determination condition CSJ2, which is a determination condition for determining whether or not to re-execute the count scan, is included. The information CSR obtained by the count scan process is recorded in association with each image (second image) acquired by using the second imaging optical system in the count scan process, for example. For example, the first result CSR1 includes the first frame CSR11, the first time CSR12 when the first frame CSR11 is acquired, the first AF information CSR13, and the first imaging condition CSR14. Note that the shooting conditions include exposure conditions such as shutter speed and aperture, and other shooting conditions. The photographing conditions here may be different for each photographing, may be different for each measurement, or may be common for all photographing. Further, the information may include information on a position where the second image is acquired, a result of counting the number of cells 324, and the like.

Here, an example of the movement pattern of the image acquisition unit 150 in the count scan processing according to the present embodiment is shown in FIG. 10 as a schematic diagram, and the movement of the image acquisition unit 150 in the count scan will be described with reference to this. Here, a case where the count scan process is executed while the image acquisition unit 150 is moved on the line CL1 shown in FIG. 10 will be described as an example.

As shown in FIG. 10, the observation-side control circuit 110 moves the image acquisition unit 150 to the start position CP1, and acquires the second image. The observation-side control circuit 110 moves the image acquisition unit 150 in the Y direction by the first Y movement pitch, and acquires the second image at the position after the movement. Thereafter, the observation-side control circuit 110 determines that the second image is in a state of being in a state corresponding to the first Y → X condition CSP11, for example, the image acquisition unit 150 exists at the position indicated by the point CP2. The acquisition and the movement of the image acquisition unit 150 are repeated. When it is determined that the state corresponds to the first Y → X condition CSP11, the observation side control circuit 110 switches the direction in which the image acquisition unit 150 is moved from the Y direction to the X direction. After the movement direction is switched to the Y direction, the observation-side control circuit 110 is in a state corresponding to the first X → Y condition CSP10, for example, the image acquisition unit 150 exists at the position indicated by the point CP3. Until it is determined, the process of moving the image acquisition unit 150 by the first X movement pitch and acquiring the second image is repeated. In this way, the observation-side control circuit 110 continues the count scan process until it is determined that the end condition CSP3 is satisfied, for example, when the image acquisition unit 150 reaches the point CP10.

Here, the operation of the observation system 1 during the count scan process will be described with reference again to FIG.

In step S203, the observation-side control circuit 110 determines the count scan state. In this determination, for example, when the image captured by the image processing circuit 120 is analyzed to detect an observation failure, or when the moving mechanism 160 detects an operation failure, it is determined that the count scan needs to be performed again. The As shown in FIG. 9, the determination condition is recorded in the observation-side recording circuit 130 as count scan processing information, for example. In addition, the specification which a user judges based on the live view (LV) display performed by the controller 200 by transmitting the image acquired at the time of count scanning to the controller 200 is also considered. The process proceeds to step S204 when it is determined that re-execution of the count scan is necessary, and proceeds to step S205 when it is not determined.

In step S204, the observation-side control circuit 110 alerts the user that an observation failure or the like has occurred in the count scan, or that the count scan needs to be re-executed, according to the determination result in step S203. A control signal is generated and transmitted to the controller 200. Thereafter, the process returns to step S202.

Note that the count scan processing performed after returning to step S202 may be performed again with the image acquisition unit 150 returned to the start position or performed again from the current position, for example, according to the determination result in step S203. To do.

In step S205, for example, the observation-side control circuit 110 determines whether or not the count scan in a predetermined entire region has ended based on the end condition CSP3 recorded in the observation-side recording circuit 130 as count scan processing information, for example. To do. The process proceeds to step S211 when it is determined that the entire region has been completed, and proceeds to step S206 when it is not determined.

In step S206, the observation-side control circuit 110 causes the imaging unit 151 to perform focus adjustment (Auto Focus: AF) on the cell 324 that is the subject of interest, and to perform an imaging operation to acquire a second image. Let Here, as described above with reference to FIGS. 3 and 4, since the second imaging optical system, which is an optical system having telecentricity, is used on the subject side in the count scan processing, the imaging optical system is used during AF. 152, the size and position of the cell 324 do not change even if the position in the optical axis direction of the image sensor 153 or the like changes. In addition, the observation-side control circuit 110 causes the observation-side communication device 140 to transmit the second image acquired as described above with reference to FIG. 9 to a preset transmission destination.

In step S207, the observation-side control circuit 110 causes the image processing circuit 120 to analyze the acquired second image, count the number of cells 324 or cell groups, and also display the cell count result as the observation-side recording circuit 130. Or to the controller 200. Thereafter, the process proceeds to step S208.

For example, when the controller 200 is the transmission destination of the second image and the controller 200 includes an image processing circuit, the cell count may be performed by the controller 200. For example, when a server on a network such as a cloud is a transmission destination of the second image and the cloud has a function corresponding to an image processing circuit, the cell count is performed outside the observation system 1. Also good. Further, the cell count may be performed based on a wide range of high pixel images synthesized based on the acquired second image after the count scan process of the entire region is completed.

In step S208, the observation-side control circuit 110 determines whether or not the current state satisfies the first X → Y condition CSP10 or the first Y → X condition CSP11 recorded as count scan processing information. judge. The process proceeds to step S209 when it is determined that the first X → Y condition CSP10 or the first Y → X condition CSP11 is satisfied, and the process proceeds to step S210 when it is not determined.

In step S209, the observation-side control circuit 110 switches the direction in which the image acquisition unit 150 is moved according to the determination result in step S208. Thereafter, the process proceeds to step S210.

In step S210, the observation-side control circuit 110 moves the image acquisition unit 150 by the first X movement pitch or the first Y movement pitch according to the movement direction at that time. Thereafter, the process returns to step S203.

In step S211, the observation-side control circuit 110 causes the observation-side communication device 140 to transmit an end signal to the controller 200 when it is determined in step S205 that the count scan has been completed in the entire area. In addition, the observation side control circuit 110 causes the image processing circuit 120 to synthesize a wide range of high pixel images based on the acquired second image. Thereafter, the process ends, and the process proceeds to step S108 of the observation apparatus control process.

Here, an example of the 3D scan process in step S109 of the observation apparatus control process is shown in FIG. 11 as a flowchart, and the operation of the observation system 1 during the 3D scan process will be described with reference to this flowchart.

In step S301, the observation-side control circuit 110 causes the lens switching unit 154 to make the imaging optical system 152 the first imaging optical system that is a non-telecentric optical system on the subject side. Thereafter, the process proceeds to step S302.

In step S302, the observation-side control circuit 110 executes pre-processing for starting 3D scanning based on, for example, 3D scan processing information recorded in the observation-side recording circuit 130. In the pre-processing, the observation-side control circuit 110 moves the image acquisition unit 150 to the moving mechanism 160 and returns it to the XY designated position for 3D scanning. In addition, the observation-side control circuit 110 controls the operations of the imaging optical system 152 and the imaging element 153, or the moving mechanism 160 so that 3D scanning can be started from a predetermined position in the Z direction. Thereafter, the observation-side control circuit 110 starts 3D scanning.

Here, an example of 3D scan processing information according to this embodiment is shown in FIG. 12, and information recorded as 3D scan processing information will be described with reference to this. The information is set in advance, for example, or set in step S105 of the observation apparatus control process. Note that, as described above, the 3D scan processing according to the present embodiment is executed using the first imaging optical system that is a subject-side non-telecentric optical system. In addition, the 3D scanning process according to the present embodiment is a process executed on an area (specific area) including a specific position designated by the user.

As shown in FIG. 12, the 3D scan processing information includes information TSP related to the 3D scan pattern, information TSJ related to execution of the 3D scan processing, and information TSR obtained by the 3D scan processing. The information TSP related to the 3D scan pattern is a designated position TSP1, a range setting TSP2, a second X movement pitch TSP5, a second Y movement pitch TSP6, and a second condition for switching from movement in the X direction to movement in the Y direction. X → Y condition TSP10, and a second Y → X condition TSP11 that is a condition for switching from movement in the Y direction to movement in the X direction. Here, for example, the second X movement pitch TSP5 is a movement (imaging) interval in the X direction, and the second Y movement pitch TSP6 is a movement (imaging) interval in the Y direction. For example, the observation-side control circuit 110 determines the specific area based on the designated position TSP1 and the range setting TSP2, and also determines the start position and end position of the 3D scan. The information TSJ related to the execution of the 3D scanning process is, for example, when it is determined that the observation is defective according to the second NG determination condition TSJ1, which is a determination condition for determining the observation defect, for example, the second NG determination condition TSJ1. A second retry determination condition TSJ2 that is a determination condition for determining whether or not to re-execute the 3D scan is included. The information TSR obtained by the 3D scanning process is recorded in association with each image (first image) acquired by using the first imaging optical system in the 3D scanning process. For example, the first result TSR1 includes the first frame TSR11, the first time TSR12 when the first frame TSR11 is acquired, the first depth information TSR13, and the first 3D shooting condition TSR14. . Note that the depth information may include the position of the cell 324 in the optical axis direction of the first imaging optical system, information on a 3D model that can be constructed based on the position information, and the like.

Here, an example of the movement pattern of the image acquisition unit 150 in the 3D scan processing according to the present embodiment is shown in FIG. 13 as a schematic diagram, and the movement of the image acquisition unit 150 in the 3D scan will be described with reference to this. Here, a case where the image acquisition unit 150 performs the 3D scan process while being moved on the line TL1 illustrated in FIG. 13 will be described as an example.

As shown in FIG. 13, the observation-side control circuit 110 moves the image acquisition unit 150 to the line TL1 in a region (specific region) that is generally centered on the designated position TSP1 indicated by the point TP0 according to the range setting TSP2. The first image is acquired at each position by picking up an image while moving it up. The movement of the image acquisition unit 150 during the 3D scan is substantially the same as the movement of the image acquisition unit 150 during the count scan described with reference to FIG. The image acquisition unit 150 starts moving from the start position TP1, and moves in the Y direction until the second Y → X condition TSP11 is satisfied at the point TP2. Thereafter, the robot moves in the X direction at the point TP3 until the second X → Y condition TSP10 is satisfied. In this way, the observation-side control circuit 110 continues the 3D scan process until the image acquisition unit 150 reaches the point TP10.

Here, the operation of the observation system 1 during the 3D scan process will be described with reference to FIG. 11 again.

In step S303, the observation-side control circuit 110 determines the 3D scan state in the same manner as in step S203 of the count scan process. The determination conditions used here are the second NG determination condition TSJ1 and the second retry determination condition TSJ2. The process proceeds to step S304 if it is determined that the 3D scan needs to be re-executed, and proceeds to step S305 if it is not determined.

In step S304, the observation-side control circuit 110 warns the user that an observation defect or the like has occurred in the 3D scan according to the determination result in step S303, and that it is necessary to re-execute the 3D scan. A control signal is generated and transmitted to the controller 200. Thereafter, the process returns to step S302.

In step S305, the observation-side control circuit 110 determines whether or not the 3D scan in the specific area has ended. The process proceeds to step S310 when it is determined that the 3D scan in the specific area has been completed, and proceeds to step S306 when it is not determined.

In step S306, the observation-side control circuit 110 causes the imaging unit 151 to perform an imaging operation to acquire a first image. Here, as described above with reference to FIGS. 5 and 6, in the 3D scanning process, the first imaging optical system which is an optical system having non-telecentricity on the subject side is used. In addition, the observation-side control circuit 110 causes the observation-side communication device 140 to transmit the acquired first image to a preset transmission destination.

In steps S307 to S309, the observation-side control circuit 110 performs the same processes as steps S208 to S210 in the count scan process. The observation-side control circuit 110 determines whether or not the second X → Y condition TSP10 or the second Y → X condition TSP11 is satisfied in step S307, and if it is determined that these conditions are satisfied. In step S308, the moving direction is switched. Thereafter, in step S309, the observation-side control circuit 110 moves the image acquisition unit 150 according to the movement direction and the value of the second X movement pitch TSP5 or the second Y movement pitch TSP6. Thereafter, the process returns to step S303.

In step S310, the observation-side control circuit 110, as described above with reference to FIGS. 5 and 6, based on the result of causing the image processing circuit 120 to perform image processing on the first image, the depth associated with the cell 324. Get information.

Note that the depth information includes information related to the unevenness of the cell 324 or the cell group in the optical axis direction of the first imaging optical system. Therefore, the depth information acquired here includes, for example, a 3D model indicating the three-dimensional shape of the cell 324 or the cell group generated by the three-dimensional model generation unit.

For example, when the controller 200 is the transmission destination of the first image and the controller 200 includes an image processing circuit, the depth information may be acquired by the controller 200. For example, when a server on a network such as a cloud is a transmission destination of the first image and the cloud has a function corresponding to an image processing circuit, the depth information is acquired outside the observation system 1. It may be broken.

In step S311, the observation-side control circuit 110 causes the observation-side communication device 140 to transmit an end signal to the controller 200. Thereafter, the process ends, and the process proceeds to step S110 of the observation apparatus control process.

An example of a controller control process performed by the controller 200 is shown in FIG. 14 as a flowchart, and the operation of the observation system 1 will be described with reference to this flowchart. The process illustrated in the flowchart of FIG. 14 starts, for example, in a state where the observation apparatus 100 is waiting for communication.

In step S401, the controller-side control circuit 210 generates display information for presenting various functions provided in the controller 200 to the user using text, icons, and the like, and causes the display device 272 to display the display information.

In step S402, the controller-side control circuit 210 determines whether the activation of the inspection application is instructed based on, for example, a control signal output from the input device 274 in accordance with a user operation result. Here, the inspection application is application software having a program for communicating with the observation apparatus 100 to control the observation apparatus 100. The controller control process proceeds to step S403 when it is determined that the activation of the inspection application is instructed, and returns to step S401 when it is not determined. The controller 200 is, for example, a tablet PC or a smartphone, and in this step, a telephone application or a mail application can be selected in addition to the inspection application. In the following description, only the case where the inspection application is selected will be described as an example.

In step S403, the controller side control circuit 210 accesses the designated camera. The designated camera is an imaging device to be controlled by the inspection application selected in step S402, for example. Hereinafter, in the present embodiment, the description will be continued assuming that the designated camera is the observation apparatus 100.

In step S <b> 404, the controller-side control circuit 210 performs an operation for turning on the observation apparatus 100 by the user or a power supply for the observation apparatus 100 based on, for example, a control signal output from the input apparatus 274 according to the operation result of the user. It is determined whether or not an operation to turn off (imaging ON / OFF operation) has been performed. The process proceeds to step S405 if it is determined that the imaging ON / OFF operation has been performed, and proceeds to step S406 if it is not determined.

In step S405, the controller-side control circuit 210 turns off the power ON signal for turning on the observation device 100 or the power to the observation device 100 based on the result of the user's imaging ON / OFF operation detected in step S404. The controller-side communication device 240 is caused to transmit a power OFF signal to the observation device 100. Thereafter, the process returns to step S403. Note that the process in this step corresponds to steps S102 to S103 in the observation apparatus control process.

In step S406, the controller-side control circuit 210, for example, based on a control signal output from the input device 274 in accordance with a user operation result, the transmission destination of an observation result or measurement result such as an image acquired by the observation device 100 by the user. It is determined whether or not various settings including information, shooting conditions, measurement conditions, and various parameters are performed. The process proceeds to step S407 if it is determined that various settings have been made, and proceeds to step S408 if it is not determined.

In step S407, the controller-side control circuit 210 causes the controller-side communication device 240 to transmit control signals related to various settings detected in step S406 to the observation device 100. Thereafter, the process proceeds to step S408. Note that the process in this step corresponds to steps S104 to S105 in the observation apparatus control process.

In step S408, the controller-side control circuit 210 determines whether or not the user has instructed execution of the count scan process based on, for example, a control signal output from the input device 274 in accordance with the result of the user operation. The process proceeds to step S409 when it is determined that execution of the count scan process is instructed, and proceeds to step S410 when it is not determined.

In step S409, the controller-side control circuit 210 causes the controller-side communication device 240 to transmit a control signal instructing execution of the count scan process to the observation device 100. Thereafter, the process proceeds to step S410. Note that the process in this step corresponds to steps S106 to S107 in the observation apparatus control process.

In step S410, the controller-side control circuit 210 determines whether or not the user has instructed the execution of the 3D scanning process based on, for example, a control signal output from the input device 274 in accordance with the result of the user operation. The controller-side control circuit 210 also determines whether or not the user has set the designated position TSP1, the range setting TSP2, and the like related to the specific area where the 3D scan is executed. The process proceeds to step S411 when it is determined that the execution of the 3D scan process is instructed or the setting related to the specific area is performed, and when it is not determined, the process proceeds to step S412.

In step S411, if the execution of the 3D scan process is instructed in step S410, the controller side control circuit 210 causes the controller side communication device 240 to transmit a control signal instructing the execution of the 3D scan process to the observation apparatus 100. . When it is determined in step S410 that the setting related to the specific area has been performed, the controller-side control circuit 210 causes the controller-side communication device 240 to transmit the control signal related to the setting to the observation device 100. Thereafter, the process proceeds to step S412. Note that the process of this step corresponds to steps S108 to S109 of the observation apparatus control process.

In step S412, the controller-side control circuit 210 determines whether to receive a measurement result or the like from the outside of the controller 200, for example, according to the result of the user operation. The process proceeds to step S413 when it is determined that the measurement result is received, and proceeds to step S414 when it is not determined.

In step S413, the controller-side control circuit 210 acquires the measurement results and the like acquired by the observation apparatus 100, and displays the measurement results and the like on the display device 272. Note that the measurement result or the like may be acquired from the observation apparatus 100, or may be acquired from the transmission destination of the measurement result of the observation apparatus 100 set in step S406. Thereafter, the process proceeds to step S414. Note that the process of this step corresponds to steps S111 to S112 of the observation apparatus control process.

In step S414, the controller-side control circuit 210 determines whether or not to end the inspection application, for example, according to the operation result of the user. If it is determined that the process is to end, the inspection application is ended and the process returns to step S401. If it is determined that the process is not ended, the process returns to step S403.

The lens switching unit 154 according to the present embodiment is driven in the optical axis direction of the imaging optical system 152 of the lens included in the imaging optical system 152, and the first imaging optical system and the second imaging optical system are driven. Although described as switching, it is not limited to this. For example, the observation apparatus 100 may include a first imaging optical system and a second imaging optical system separately. In this case, for example, the lens switching unit 154 moves the optical axis of the imaging optical system used for imaging out of the first imaging optical system and the second imaging optical system so as to be parallel to the observation axis. Thus, the imaging optical system is switched.

Although the illumination unit 155 according to this embodiment is described as being disposed on the support unit 165, it is only necessary that the light emitting unit of the illumination optical system 156 be disposed on the support unit 165. For example, the light source 157 may be used for observation. It may be placed anywhere on the device 100. Note that, for example, the intensity of illumination may be changed depending on the type of observation in order to reduce damage to the observation target such as the cell 324. Moreover, as control of illumination light, there can be a control method of intermittent illumination such that the sample 300 is illuminated only at the moment of photographing, and a control method of increasing or decreasing the number of lighting illuminations.

In the count scan process and the 3D scan process, the case where the image acquisition unit 150 starts moving in the Y direction from the start position has been described as an example. However, the present invention is not limited to this. For example, scanning starts from the start position in the X direction. May be.

<Advantages of observation system>
As described above, the observation apparatus 100 according to the present embodiment causes the moving mechanism 160 to change the position of the image acquisition unit 150 in the X direction and the Y direction while maintaining the optical axis of the imaging optical system 152 in parallel with the observation axis. The sample 300 is repeatedly photographed to acquire a plurality of images.

At this time, the observation apparatus 100 according to the present embodiment uses the second imaging optical system, which is a subject-side telecentric optical system, to acquire a second image suitable for synthesis of a wide range of high-pixel images. Note that the second image or the high pixel image is suitable for acquiring the shape and number of the subject of interest such as a cell. On the other hand, the observation apparatus 100 according to the present embodiment uses a first imaging optical system that is a subject-side non-telecentric optical system, thereby causing parallax from a plurality of first images acquired at different imaging positions. A first image movement amount ΔX including the image movement amount is acquired, and depth information of a subject of interest such as a cell is acquired based on the first image movement amount ΔX.

Further, the lens switching unit 154 according to the present embodiment switches the imaging optical system 152 according to the type of observation. Thus, the observation apparatus 100 according to the present embodiment includes the lens switching unit 154 that switches between the first imaging optical system and the second imaging optical system, and thus is suitable for, for example, acquisition of depth information and cell counting. Observations with different required optical characteristics, such as image acquisition, can be realized.

Therefore, if the observation system 1 according to the present embodiment is used, the user can accurately project the position of the cell in the optical axis direction, the three-dimensional model of the cell, the shape of the cell, etc. only by specifying the desired observation method. Images, cell count results, etc. can be acquired.

[Second Embodiment]
A second embodiment of the present invention will be described. Here, differences from the first embodiment will be described, and the same portions will be denoted by the same reference numerals and description thereof will be omitted.

In the first embodiment, in the observation apparatus that performs observation or the like by switching between the first imaging optical system and the second imaging optical system according to the purpose, the cell 324 is used when the first imaging optical system is used. The case where the depth information of the subject of interest such as is acquired has been described as an example. When the first imaging optical system is used, a light beam that can reach the imaging element 153 has an inclination between the cell 324 and the objective lens 152b with respect to the optical axis of the first imaging optical system. Therefore, the first image includes information related to the side surface of the subject that exists at a position other than the optical axis. Therefore, in the present embodiment, the observation system 1 that generates a side observation image as if the side surface of the cell 324 was imaged based on the first image acquired by using the first imaging optical system will be described. do.

<Configuration of observation system>
The observation apparatus 100 according to the present embodiment further has functions as a stereoscopic image generation unit and a side information processing unit. The stereoscopic image generation unit generates a stereoscopic image of the subject of interest based on, for example, a plurality of first images and depth information. The side information processing unit acquires information related to various side observations such as image data and position information obtained by the side observation processing, and generates a side observation image based on the first image and the depth information. . Note that the functions as the stereoscopic image generation unit and the side information processing unit can be realized by the observation-side control circuit 110 and / or the image processing circuit 120, for example. Note that the functions as the stereoscopic image generation unit and the side information processing unit may be realized by the controller-side control circuit 210, respectively.

(About side observation of a cell using the first imaging optical system)
Here, in the case where the first imaging optical system is used, between the principal ray radiated from an area on the cell 324 that is separated from the optical axis of the first imaging optical system by a predetermined threshold or more and the optical axis. Next, side observation of the cell 324 using the presence of inclination (parallax) will be described.

FIG. 15 shows a schematic diagram of an example of the positional relationship between the imaging unit 151 including the first imaging optical system according to the present embodiment and the sample 300. The state shown in FIG. 15 is a state where the point P1 on the cell 324 is moved by the third X movement amount ΔX3 in the X + direction from the state where it is located on the optical axis of the first imaging optical system. That is, when the image position of the point P1 is a corresponding point, the position of the corresponding point is in a state of moving from the point U1 to the point U1 ′. Based on the first image movement amount ΔX (the distance between the point U1 and the point U1 ′) acquired from the plurality of first images in this manner, the same as in the first embodiment, on the cell 324. Depth information is acquired for each position. In addition, since acquisition of depth information based on the first image has been described in the first embodiment, in the following description, depth information is treated as known information.

In the state shown in FIG. 15, the light ray R10 emitted from the point P1 on the cell 324 is incident on the point U1 ′ on the image sensor 153, and the light ray R30 emitted from the point P3 is incident on the point U3 on the image sensor 153. To do. In this way, the area A1 on the cell is imaged as the area A1 ′ on the image sensor 153.

In the state shown in FIG. 15, for example, in the optical axis direction of the first imaging optical system, the area between the position of the point P1 and the position of the point P3 corresponds to the in-focus range (within the depth of focus). That is, it is assumed that the area A1 is in focus. In the side observation according to the present embodiment, the focused area A1 of the subject of interest is referred to as a specific range, and the region A1 ′ obtained by capturing the specific range of the subject of interest in the first image is referred to as a specific range image. I will call it.

The observation apparatus 100 according to the present embodiment causes the image processing circuit 120 to cut out the specific range image from each first image. Further, the image processing circuit 120 synthesizes the specific range images based on depth information about the subject included in each specific range image, for example, and acquires a side observation image. The side information processing unit included in the observation device 100 according to the present embodiment combines the plurality of specific range images acquired in this manner, thereby imaging the cell 324 from below, and the side information processing unit A side observation image such as a depth composite image obtained by imaging can be acquired.

Here, an example of a side observation image according to the present embodiment is shown in FIG. 16 as a schematic diagram. This is an effective use of the image in the periphery of the screen other than the image near the optical axis, and as shown in FIG. 16, the observation device 100 according to the present embodiment acquired from each first image, A plurality of specific range images including the first specific range image I10, the second specific range image I11, and the third specific range image I12 are synthesized. Here, the width W in the Z-axis direction (after image processing) in the side-view image of each specific range image to be synthesized is, for example, the optical axis between the points P1 and P3 when the region A1 ′ is the specific range image. It is determined based on the position of the direction (depth information). In this case, the width W is the difference between the depth Z3 and the depth Z1. Thus, the width W in the side-view observation image of each specific range image can be different. In this way, with the magnifying optical system, the side image is actively utilized to perform side observation, and it is more important and abundant than ever to inspect and observe the condition of the object with a simple configuration. It is possible to obtain image information (stereoscopic conditions such as shading, color, and structure). By providing a moving mechanism 160 that moves the imaging unit 151 that captures a sample (object) using a magnifying optical system and acquires a first image, a plurality of the first images acquired at different positions of the imaging unit 151 are provided. It is possible to obtain the first image, which can be used to join one image and obtain three-dimensional information. The three-dimensional information acquisition unit acquires the three-dimensional information of the sample 300 based on the information related to each imaging position at the time of movement. At least one of the plurality of first images is the first imaging optical system. The image obtained in (1) is an image other than on the optical axis of the optical system.

Note that the width W in the Z-axis direction of each image may be determined based on, for example, the first image movement amount ΔX or the angle of view θ, or may be determined based on the focal length and the width of the focusing range. May be.

In addition, although the case where the area near the optical axis is cut out as a specific range image has been described as an example, the present invention is not limited to this. The region on the first image cut out as the specific range image may be a region that is separated from the optical axis of the first imaging optical system by a predetermined threshold or more. On the other hand, it is preferable that the range is less affected by image distortion that may be present at the periphery of the image sensor 153.

For example, depending on the width in the X direction of the cells 324 included in the focus range that can vary depending on the degree of unevenness of the cells 324, the angle of view θ and the first image movement amount ΔX in each first image or Needless to say, the width of the region A1 ′ can also change. Further, the angle of view θ, the first image movement amount ΔX, or the width of the region A1 ′ corresponds to the resolution of the side observation image as shown in FIG. The third X movement amount ΔX3, which is the distance between the optical axes between the positions where each first image is acquired (the movement amount of the image acquisition unit 150), also depends on the required resolution of the side observation image. Can change. For this reason, it is preferable that the third X movement amount ΔX3 of the side observation according to the present embodiment is smaller than, for example, the second X movement amount ΔX2 in the 3D scan process described above in the first embodiment.

When there is an overlap in each specific range included between the plurality of first images, such as when the third X movement amount ΔX3 is set to be sufficiently small, each acquired first Of these images, a region that is focused on the cell 324 and is separated from the optical axis by a predetermined threshold or more may be cut out as a specific range image and image synthesis may be performed. In this case, for example, the image composition is performed using information on the imaging position acquired simultaneously with the image. Further, for each of a plurality of specific range images including overlapping specific ranges, for example, a score is obtained based on the presence or absence of overexposure or blackout, the degree of focus on the cell 324, etc., and is used for image synthesis according to the score. The specific range image to be selected may be selected.

As described above, in the side observation according to the present embodiment, the range focused on the subject of interest in the first image is cut out as the specific range image. In such side-viewing, for example, the observation apparatus 100 determines a region to be imaged of the subject of interest, divides it into a plurality of regions, and repeats imaging focused on the divided regions to obtain the first image. Obtaining and using the range including the region of the first image as the specific range image. In addition, for example, the observation apparatus 100 uses an area corresponding to a specific area that is a predetermined threshold or more away from the optical axis on the image sensor 153 as an AF area, and performs first focusing so as to focus on the AF area. The Z position of the imaging optical system is adjusted. In this way, the observation apparatus 100 may acquire the side observation image while fixing the region occupied by the specific range image in the first image. Further, for example, the observation apparatus 100 acquires the first image while moving the image acquisition unit 150 in each of the X direction, the Y direction, and the Z direction according to a preset movement pattern, as in 3D scanning. Then, the side observation image may be acquired by using the range focused from the acquired first image as the specific range image.

Although the case where the image acquisition unit 150 is moved in the X direction by the moving mechanism 160 has been described as an example, the present invention is not limited to this. The acquisition of the side observation image may be performed while being moved, for example, in the X direction and the Y direction according to the set or selected movement pattern.

As described above, the side surface information processing unit included in the observation apparatus 100 according to the present embodiment cannot be observed using the subject-side telecentric optical system such as the sample 300 observed from the X + direction in the state illustrated in FIG. A side observation image that captures the side surface of the cell 324 can be acquired.

<Operation of observation system>
An example of the side observation process according to the present embodiment is shown as a flowchart in FIG. 17, and the operation of the observation system 1 will be described with reference to this. The side observation process is performed, for example, after step S205 and before step S210 in the observation apparatus control process described above with reference to FIG. In the observation device control process, the side observation process receives a control signal instructing execution of the side observation process output by the controller 200 according to, for example, a user operation result in the same manner as the count scan process or the 3D scan process. It is started when it is determined. Note that various types of information such as scan patterns and determination conditions required in the side observation processing are recorded in the observation side recording circuit 130 as side observation processing information, for example. The side observation processing information includes a result acquired in the side observation processing.

In step S501, the observation-side control circuit 110 causes the lens switching unit 154 to set the imaging optical system 152 as the first imaging optical system in the same manner as in step S301 of the 3D scanning process. Thereafter, the process proceeds to step S502.

In step S502, the observation-side control circuit 110 performs pre-processing in the same manner as in step S302 of 3D scanning processing. In addition, the observation side control circuit 110 starts scanning for acquiring a side observation image. Thereafter, the process proceeds to step S503.

In steps S503 and S504, the observation-side control circuit 110 performs pre-processing, determination regarding NG determination conditions and retry determination conditions, warning processing as necessary, and the like in the same manner as in steps S303 and S304 of 3D scanning processing. I do. The process proceeds to step S505 when it is determined in step S503 that the NG determination condition and the retry determination condition are not satisfied, and in step S504 when it is determined that the NG determination condition or the retry determination condition is satisfied in step S503. After giving a warning if necessary, the process returns to step S502.

In step S505, the observation-side control circuit 110 determines whether or not the side observation image acquisition in the specific area is completed. If it is determined that the process has been completed, the process proceeds to step S508; otherwise, the process proceeds to step S506.

In step S506, as described above with reference to FIG. 15, the observation-side control circuit 110 performs AF on a region (specific range) in which the distance from the optical axis of the first imaging optical system is equal to or greater than a predetermined threshold. Thus, the first image focused on the specific range is acquired.

In step S507, the observation side control circuit 110 moves the image acquisition unit 150 to the moving mechanism 160 according to the scan pattern recorded as the side observation processing information. For example, the observation-side control circuit 110 moves in a direction corresponding to the inclination of the principal ray emitted from the specific range. Thereafter, the process returns to step S503.

In step S508, the observation-side control circuit 110 causes the image processing circuit 120 to set a specific range from each first image acquired in the side-view observation scan in the specific area as described above with reference to FIG. The captured area is cut out as a specific range image. The observation-side control circuit 110 causes the image processing circuit 120 to convert the plurality of specific range images into an appropriate width W based on the depth information, and to synthesize the converted images.

In step S509, the observation side control circuit 110 transmits the side observation image to, for example, a preset transmission destination. Note that the transmission destination may be determined based on a control signal output by the controller 200 in accordance with a user operation result in this step. Thereafter, the process ends.

<Advantages of Second Embodiment>
The observation system 1 according to the present embodiment has the following advantages in addition to the advantages obtained in the first embodiment.

The side information processing unit included in the observation apparatus 100 according to the present embodiment is an area on the cell 324 in an observation apparatus that performs observation or the like by switching between the first imaging optical system and the second imaging optical system according to the purpose. The side surface of the cell 324 is imaged using the fact that there is an inclination between the principal ray radiated from a region separated from the optical axis of the first imaging optical system by a predetermined threshold or more and the optical axis. Acquired side observation image.

Therefore, if the user uses the observation device 100 according to the present embodiment, the first image including the side surface of the cell 324 that cannot be obtained by the observation device having telecentricity on the subject side, and the image based on the first image. Obtained side view images.

Furthermore, the observation apparatus 100 according to the present embodiment and the observation apparatus 100 according to the first embodiment can be combined. For example, the stereoscopic image generation unit performs a process of three-dimensionalizing the specific range image acquired as described in the present embodiment based on the depth information regarding each corresponding point included in the specific range image. The observation device 100 acquires the stereoscopic image of the cell 324 by synthesizing the three-dimensional specific range image acquired in this way at the corresponding position of the stereoscopic model of the cell 324 described in the first embodiment. it can.

<Modification>
In the first embodiment and the second embodiment, the objective lens 152b and the imaging lens 152c are illustrated as positive lenses for simplicity, but the present invention is not limited to this. For example, the objective lens may be a lens group having a negative refractive power in order to reduce the first image movement amount ΔX between the first images to be acquired and to increase the parallax. Needless to say, a plurality of lenses can be used according to the required performance.

Note that the observation apparatus 100 according to the first embodiment and the second embodiment may perform the AF operation based on the depth information acquired from the parallax image when the first imaging optical system is used.

In the first embodiment and the second embodiment, it is assumed that the observation apparatus 100 is used in an incubator, and emphasizes the use focused on cell observation. Needless to say, the present invention can be generalized as an observation apparatus for enlarging and confirming details, which can realize acquisition of size and the like and acquisition of depth information related to an observation object.

The sample 300 can be put in and out of, for example, an incubator, a clean bench or the like while being placed on the upper surface of the observation apparatus 100. In such a case, the cell 324 will be affected by a temperature change and may receive a heat shock. In addition, contamination may occur with taking in and out. For example, the present technology can appropriately acquire the shape and size of an observation object such as a cell or a cell group and depth information such as unevenness of the observation object such as a cell or a cell group. In the first embodiment and the second embodiment, the user is warned of the occurrence of observation failure according to the result of the determination using the NG determination condition and the retry determination condition for the image acquired by the observation apparatus 100. However, the application of the present technology is not limited to this. The user can be warned even when an abnormality of the observation object, contamination, or the like is detected. Moreover, this technique evaluates the state of the culture medium based on the result of image analysis.

In the first and second embodiments, the observation-side control circuit 110, the image processing circuit 120, the observation-side recording circuit 130, and the observation-side communication device 140 are arranged inside the housing 101 as a circuit group 104. Although the case where it is provided has been described as an example, it is not limited thereto. For example, one or more of these functions may be provided in the image acquisition unit 150. For example, the function as the observation-side communication device 140 may be provided in both the image acquisition unit 150 and the circuit group 104. In addition, one or more functions of the observation side control circuit 110, the image processing circuit 120, and the observation side recording circuit 130 may be provided in the controller 200. That is, for example, some or all of the above-described various determinations, image processing, and the like may be performed by the controller 200.

Further, among the units included in the controller 200, some elements such as the input / output device 270 may be included in the observation device 100. Furthermore, a configuration in which the observation apparatus 100 and the controller 200 are incorporated in one housing is also conceivable. The observation system 1 in which the observation apparatus 100 and the controller 200 are integrated can be used when the user himself enters a use environment such as a temperature-controlled room.

In addition, the observation system 1 records and learns observation results such as image analysis, usage of the observation system 1 including the usage frequency of the user, incubator settings, and the like. Artificial intelligence (AI) that presents parameters and the like may be included. Further, the AI may be built inside the observation system 1, for example, in a DSP or the like, or may be built on the Internet and outside the observation system 1. The observation system 1 including such an AI can determine, for example, the state of the cell, the type, the state of the medium, the presence or absence of foreign matter, etc. with respect to the acquired image by referring to a database prepared on the server.

The present technology is also effective when applied to an imaging apparatus that is used at a position where the imaging optical system and the user are separated, such as a surveillance camera or an endoscope. The user can acquire observation results such as images suitable for the purpose of use without having to replace the imaging device according to the purpose of imaging.

In addition, this invention is not limited to the said embodiment, In the implementation stage, it can change variously in the range which does not deviate from the summary. Further, the embodiments may be implemented in combination as appropriate, and in that case, the combined effect can be obtained. Furthermore, the present invention includes various inventions, and various inventions can be extracted by combinations selected from a plurality of disclosed constituent elements. For example, even if several constituent requirements are deleted from all the constituent requirements shown in the embodiment, if the problem can be solved and an effect can be obtained, the configuration from which the constituent requirements are deleted can be extracted as an invention. *

Claims

And a first imaging optical system including a magnifying optical system and a first imaging optical system which is a subject-side non-telecentric optical system in which an angle between any principal ray on the subject side and the optical axis is 6 ° or more. An imaging unit that images a sample using a system to obtain a first image;
A moving mechanism for changing a relative position between the sample and the imaging unit;
Three-dimensional information for acquiring three-dimensional information about the sample based on a plurality of the first images acquired at different positions of the imaging unit and information on each imaging position at the time of acquisition of the first image. An observation device comprising: an acquisition unit.
2. The at least one of the plurality of first images is an image obtained by the first imaging optical system other than an image on an optical axis of the first imaging optical system. Observation device.
A corresponding point acquisition unit that acquires corresponding points based on subject images included in each of the plurality of first images acquired at different imaging positions;
The observation according to claim 1, wherein the three-dimensional information acquisition unit acquires the three-dimensional information based on a change amount on the imaging surface of the position of the corresponding point and an interval between the imaging positions. apparatus.
The observation apparatus according to any one of claims 1 to 3, further comprising a three-dimensional model generation unit that generates a three-dimensional model of a subject included in the sample based on the three-dimensional information.
Any one of Claims 1 thru | or 3 further provided with the stereo image production | generation part which produces | generates the stereo image of the to-be-photographed object contained in the said sample based on the said stereo information and the said 1st image corresponding to the said stereo information. The observation apparatus according to item.
For each of the plurality of first images, a region focused on the subject is cut out as a specific range image, and the plurality of the specific range images are synthesized based on the stereoscopic information to generate a side-view image of the subject. The observation apparatus according to any one of claims 1 to 3, further comprising a side surface information processing unit.
The imaging unit further includes a second imaging optical system which is a subject-side telecentric optical system in which an angle formed by any principal ray on the subject side and the optical axis is 4 ° or less,
A lens switching unit that switches between the first imaging optical system and the second imaging optical system;
The imaging unit further acquires a second image by imaging the sample using the second imaging optical system.
The observation apparatus according to any one of claims 1 to 6.
The lens switching unit controls a position in an optical axis direction of a lens included in the imaging optical system, and the imaging optical system is the first imaging optical system or the second imaging optical system. The observation apparatus described in 1.
The optical axis of the first imaging optical system is parallel to the observation axis;
The moving mechanism moves the imaging unit while maintaining the optical axis of the first imaging optical system in parallel with the observation axis,
The three-dimensional information is information related to the position of the subject included in the sample in the direction of the observation axis.
The observation apparatus according to any one of claims 1 to 8.
An observation apparatus according to any one of claims 1 to 9,
An observation system comprising: a controller that acquires a user operation result, outputs the operation result to the observation device, and acquires an observation result of the observation device.
Using an imaging unit including a magnifying optical system and a first imaging optical system that is a subject-side non-telecentric optical system in which an angle formed between any principal ray on the subject side and the optical axis is 6 ° or more, Obtaining a first image by imaging a sample using a first imaging optical system;
Changing the relative position of the sample and the imaging unit;
Obtaining three-dimensional information about the sample based on a plurality of the first images acquired at different positions of the imaging unit and information on each imaging position at the time of acquisition of the first image A method for controlling an observation apparatus, comprising:
The optical axis of the first imaging optical system is parallel to the observation axis;
The change of the relative position includes moving the imaging unit while maintaining the optical axis of the first imaging optical system parallel to the observation axis,
The method for controlling an observation apparatus according to claim 11, wherein the acquisition of the three-dimensional information includes acquiring information related to a position in the direction of the observation axis for a subject included in the sample.