WO2018139025A1

WO2018139025A1 - Observation device

Info

Publication number: WO2018139025A1
Application number: PCT/JP2017/041964
Authority: WO
Inventors: 真央佐々木; 高橋　進
Original assignee: オリンパス株式会社
Priority date: 2017-01-24
Filing date: 2017-11-22
Publication date: 2018-08-02

Abstract

An observation device comprises: an illumination unit arranged inside a cylindrical sample with a center axis and outputting illumination light toward a cylindrical illumination region centered at a prescribed optical axis; an imaging unit arranged near the illumination unit inside the sample, capturing a cylindrical capture region in the sample centered at the prescribed optical axis, and outputting the captured signal; an image generation unit for generating an annular image centered at the prescribed optical axis on the basis of the captured signal; and an image processing unit for calculating a prescribed offset for the optical axis relative to the center axis on the basis of the inner diameter of the sample and a value acquired from the annular image, and generating a locally developed image by correcting the imaging magnification along a direction parallel to the center axis in an image acquired by developing the annular image in accordance with the offset.

Description

Observation device

The present invention relates to an observation apparatus.

2. Description of the Related Art Conventionally, a technique for developing a developed image by developing an image obtained by imaging the entire inner circumference of a cylindrical subject is known.

Specifically, for example, in Japanese Patent No. 25662059, an image projected on a convex mirror provided in the center of a horizontal shaft having a substantially circular cross section is captured to obtain a basic image, and the basic image is sequentially acquired. A configuration is disclosed in which a developed image of the inner wall surface of the horizontal shaft is generated by decomposing into a plurality of concentric annular images having different radii, developing the plurality of annular images into a linear image, and sequentially arranging them. ing.

Here, according to the configuration disclosed in Japanese Patent No. 25662059, the central axis of the imaging device provided with the convex mirror and the central axis of the horizontal shaft imaged by the imaging device by components such as a suspension arm In a state in which are substantially matched, a basic image used for generating a developed image is acquired.

That is, according to Japanese Patent No. 25662059, there is a problem that an accurate developed image cannot be generated unless a basic image is acquired using a centering mechanism constituted by components such as a suspension arm. ing.

The present invention has been made in view of the above-described circumstances, and an object thereof is to provide an observation apparatus capable of generating an accurate developed image without using a centering mechanism.

The observation apparatus according to one aspect of the present invention is arranged inside a cylindrical subject having a central axis and emits illumination light to a cylindrical illumination region centered on a predetermined optical axis. An illuminating unit that is configured and an imaging signal that is provided in the vicinity of the illuminating unit, is disposed inside the subject, and images a subject in a cylindrical imaging region centered on the predetermined optical axis. An image generating unit configured to generate an annular image having the predetermined optical axis as a center of the image based on the imaging signal, and an image of the subject. An offset amount of the predetermined optical axis with respect to the central axis is calculated based on an inner diameter and a value acquired from the annular image, and the central axis in an image obtained by developing the annular image The imaging magnification in the direction parallel to Having an image processing unit configured to generate a local expansion image by performing the magnification correction process for correcting in accordance with the serial offset.

The figure which shows the external appearance structure of the endoscope apparatus which concerns on embodiment. The figure for demonstrating a structure at the time of attaching an optical adapter to the front-end | tip part of the insertion part of an endoscope. The schematic diagram for demonstrating the illumination area | region LR and imaging region IR at the time of attaching an optical adapter to the front-end | tip part of the insertion part of an endoscope. The block diagram for demonstrating the structure of the endoscope apparatus which concerns on embodiment. The figure which shows an example of the state which inserted the insertion part of the endoscope into the inside of piping. The figure for demonstrating the area | region IRB and IRS which arise in the imaging area IR in the insertion state like FIG. The figure which shows an example of the endoscopic image produced | generated in the insertion state like FIG. The figure which shows an example of the display image produced | generated according to operation | movement of the endoscope apparatus which concerns on embodiment. The figure which shows an example of the display image produced | generated according to operation | movement of the endoscope apparatus which concerns on embodiment. The flowchart for demonstrating an example of the operation | movement which concerns on the production | generation and display of the wide area expansion | deployment image performed in the endoscope apparatus which concerns on embodiment. The figure which shows an example of the endoscopic image used for the process which concerns on the production | generation concerning a local expansion | deployment image. The figure which shows an example of the basic expansion | deployment image produced | generated by expand | deploying the endoscopic image of FIG. The figure which shows an example of the combined image produced | generated using the basic expansion | deployment image of FIG. The figure which shows an example of the cut-out image produced | generated by cutting out a part of combined image of FIG. The figure for demonstrating the width WS of the offset direction of the shadow area | region contained in the endoscopic image used for the production | generation of a basic expansion | deployment image. The figure which shows the outline | summary of the parameter used in the process which concerns on the production | generation concerning a wide area expansion | deployment image. The figure which shows the example at the time of deform | transforming the cutout image of FIG. 14 by the magnification correction process. The figure which shows an example of the local expansion | deployment image produced | generated by excising a part of the cutout image after a deformation | transformation of FIG. The figure which shows an example of the wide area expansion | deployment image produced | generated by bonding a some local expansion | deployment image. The figure which shows an example of the display image produced | generated according to operation | movement of the endoscope apparatus which concerns on embodiment. The figure which shows an example of the display image produced | generated according to operation | movement of the endoscope apparatus which concerns on embodiment. The figure which shows an example of the display image produced | generated according to operation | movement of the endoscope apparatus which concerns on embodiment.

Hereinafter, embodiments of the present invention will be described with reference to the drawings.

1 to 22 relate to the embodiment of the present invention.

The endoscope apparatus 1 has a function as an observation apparatus for observing the inside of a cylindrical subject having a central axis such as a pipe. Specifically, the endoscope apparatus 1 includes, for example, an endoscope 2 and an apparatus main body 3 to which the endoscope 2 can be connected as shown in FIG. The apparatus body 3 is provided with a display unit 35 that can display an image or the like.

The endoscope 2 includes an insertion portion 5 formed to have an elongated shape that can be inserted into a cylindrical subject having a central axis, and an operation portion 6 provided on the proximal end side of the insertion portion 5. And a universal cord 7 extending from the operation unit 6. Further, the endoscope 2 is configured to be detachably connected to the apparatus main body 3 via the universal cord 7. A light guide 21 (not shown in FIG. 1) for transmitting illumination light supplied from the apparatus main body 3 is inserted into the endoscope 2.

The insertion portion 5 is configured by sequentially providing a distal end portion 11, a bending portion 12 formed to be bendable, and a long flexible tube portion 13 having flexibility from the distal end side.

The operation unit 6 is provided with a bending joystick 6a configured to perform an operation for bending the bending unit 12 in a desired direction. Although not shown, the operation unit 6 is provided with operation buttons corresponding to functions available in the endoscope apparatus 1 such as a freeze button, a bending lock button, and a recording instruction button. Yes.

The distal end portion 11 can be detachably attached with an optical adapter 40 including one or more optical members corresponding to an imaging region when the inside of the cylindrical subject is imaged using the endoscope 2. It is configured. In the present embodiment, it is assumed that an optical adapter 40 configured to have a cylindrical (360 degrees) imaging region centered on an optical axis AC described later is attached to the distal end portion 11. I do.

As shown in FIG. 2, the distal end portion of the light guide 21 inserted into the endoscope 2 and the illumination light incident through the distal end portion of the light guide 21 are diffused into the distal end portion 11 by the optical adapter 40. An illumination lens 22 that emits light to the element 47 is provided. Further, as shown in FIG. 2, an imaging lens 23 that forms an image of light incident through the lens unit 46 of the optical adapter 40 and the light imaged by the imaging lens 23 are imaged at the distal end portion 11. And an image pickup device 24 for outputting an image pickup signal. Further, an uneven portion 25 having a shape that can be engaged with the mounting member 49 of the optical adapter 40 is formed on the side surface of the distal end portion 11. FIG. 2 is a diagram for explaining a configuration when an optical adapter is attached to the distal end portion of the insertion portion of the endoscope.

In the present embodiment, the image sensor 24 may be configured to include an image sensor such as a CCD or a CMOS. Further, according to the present embodiment, as long as it can be engaged with the mounting member 49 of the optical adapter 40, the structure having a shape different from the uneven portion 25 is formed on the side surface of the tip portion 11. Also good.

As shown in FIG. 2, the optical adapter 40 includes a cover member 41, an observation window 42, a free-form surface lens 43, a frame member 44, a lens holder 45, a lens unit 46, a light diffusing element 47, and an illumination. A window 48 and a mounting member 49 are included.

The cover member 41 is formed of, for example, a disk-shaped member having a light shielding property. Further, the cover member 41 is fixedly disposed at a position so as to cover the front surface of the free-form surface lens 43.

The observation window 42 is formed of, for example, a cylindrical member having translucency such as a sapphire pipe, and transmits light incident from the outside of the optical adapter 40 so as to be emitted to the free-form surface lens 43. It is configured. The observation window 42 is a position between the cover member 41 and the frame member 44 and is fixedly disposed at a position that covers the lens side surface of the free-form surface lens 43. The observation window 42 is formed with an outer diameter smaller than the outer diameter of a flange 44f (described later) of the frame member 44.

The free-form surface lens 43 is a position between the cover member 41 and the frame member 44 in the optical adapter 40, and is fixed at a position where light emitted through the observation window 42 is incident on the inside from the lens side surface. Has been placed. The free-form lens 43 is formed to have optical characteristics such that light incident inside from the lens side surface is refracted or reflected by the front surface of the lens and emitted to the lens unit 46.

The frame member 44 is formed by combining, for example, a cylindrical member having a light shielding property and a flange 44 f that shields the front surface of the light diffusing element 47. Further, the frame member 44 is configured such that a part of the cylindrical portion 45a of the lens holder 45 can be fitted from the opening on the flange 44f side.

The lens holder 45 has a cylindrical portion 45a having an inner diameter capable of holding the lens unit 46, and an enlarged diameter portion 45b having an inner diameter larger than the inner diameter of the cylindrical portion 45a.

The cylindrical portion 45a is formed so as to hold the lens unit 46 in a state where the central axis of the optical adapter 40 and the central axis of the lens unit 46 are matched.

The enlarged diameter portion 45b is provided on the rear end side of the cylindrical portion 45a and has an inner diameter capable of accommodating a portion including the distal end surface of the distal end portion 11. In addition, the diameter-expanded portion 45b allows the light of the light diffusing element 47 to contact the light emitting surface of the lens unit 46 and the light incident surface of the imaging lens 23 when the optical adapter 40 is attached to the distal end portion 11. It is formed to have a shape that allows the entrance surface and the light exit surface of the illumination lens 22 to be in contact with each other. An attachment member 49 is provided on the rear end side of the enlarged diameter portion 45b.

The lens unit 46 includes one or more lenses. In addition, the lens unit 46 is configured to emit light incident through the free-form surface lens 43 to the imaging lens 23 when the optical adapter 40 is attached to the distal end portion 11.

In the present embodiment, for example, as shown in FIG. 2, when the optical adapter 40 is attached to the distal end portion 11, the central axis of the imaging lens 23, the central axis of the optical adapter 40, and a free-form surface In the following description, it is assumed that the central axis of the lens 43 and the central axis of the lens unit 46 are located on the same optical axis AC.

The light diffusing element 47 has, for example, a substantially cylindrical shape, and is fixedly disposed between the frame member 44 and the lens holder 45 in the optical adapter 40 in a state of being fitted into the cylindrical portion 45a. The light diffusing element 47 is configured to diffuse the illumination light incident through the light exit surface of the illumination lens 22 and emit it to the side when the optical adapter 40 is attached to the distal end portion 11. .

The illumination window 48 is formed of a cylindrical member having translucency, such as a sapphire pipe, for example, and transmits the illumination light diffused by the light diffusing element 47 to be emitted outside the optical adapter 40. Is configured to do. The illumination window 48 is a position between the frame member 44 and the lens holder 45 and is fixedly disposed at a position that covers the side surface of the light diffusing element 47. Moreover, the observation window 42 is formed to have the same outer diameter as the outer diameter of the flange 44 f of the frame member 44.

The mounting member 49 has, for example, a substantially cylindrical shape, and is formed by providing a groove that can be engaged with the concavo-convex portion 25 on the inner peripheral surface.

That is, the optical adapter 40 is configured to include a cylindrical (360 degrees) illumination area centered on the optical axis AC. Further, according to the configuration of the optical adapter 40 as described above, for example, as schematically shown in FIG. 3, the illumination region LR of the illumination light emitted to the outside of the optical adapter 40 through the illumination window 48 is provided. It becomes wider than the imaging region IR when imaging light incident on the inside of the optical adapter 40 through the observation window 42. Further, according to the configuration of the optical adapter 40 as described above, the illumination area LR and the imaging area IR are shifted along the optical axis AC. (See FIG. 3). In addition, according to the configuration of the optical adapter 40 as described above, a predetermined range in front of the distal end portion 11 becomes a non-imaging region that is out of the imaging region IR. FIG. 3 is a schematic diagram for explaining the illumination region LR and the imaging region IR when the optical adapter is attached to the distal end portion of the insertion portion of the endoscope. In addition, the illumination unit in the present embodiment includes a light diffusing element 47 and an illumination window 48. In addition, the imaging unit in the present embodiment is provided in the vicinity of the illumination unit, and includes the imaging device 24, the observation window 42, and the free-form surface lens 43.

As shown in FIG. 4, the apparatus main body 3 includes a light source unit 31, a light source drive unit 32, an image sensor drive unit 33, an image generation unit 34, a display unit 35, a storage unit 36, and an input I / F. An (interface) unit 37 and a CPU 38 are included. The apparatus body 3 is provided with a connection port (not shown) for connecting a portable external storage device 51 such as a USB memory. FIG. 4 is a block diagram for explaining the configuration of the endoscope apparatus according to the embodiment.

The light source unit 31 includes, for example, an LED or a lamp. The light source unit 31 is configured to be turned on or off in accordance with a light source drive signal output from the light source drive unit 32. The light source unit 31 is configured to supply, for example, white light having a light amount corresponding to a light source drive signal output from the light source drive unit 32 to the light guide 21 as illumination light.

The light source driving unit 32 includes, for example, a light source driving circuit. Further, the light source driving unit 32 is configured to generate and output a light source driving signal for driving the light source unit 31 in accordance with the control of the CPU 38.

The image sensor driving unit 33 includes, for example, an image sensor driving circuit. The image sensor driving unit 33 is configured to generate and output an image sensor driving signal for driving the image sensor 24 under the control of the CPU 38.

The image generation unit 34 is configured by an integrated circuit such as an FPGA (Field Programmable Gate Array). In addition, the image generation unit 34 performs predetermined signal processing on the imaging signal output from the imaging device 24 to generate an annular endoscope image with the optical axis AC as the center of the image, The generated endoscopic images are sequentially output to the CPU 38.

The display unit 35 includes, for example, a liquid crystal panel. The display unit 35 is configured to display the display image output from the CPU 38 on the display screen. The display unit 35 includes a touch panel 35a that detects a touch operation on a GUI (Graphical User Interface) button or the like displayed on the display screen and outputs an instruction corresponding to the detected touch operation to the CPU 38. Has been.

The storage unit 36 includes, for example, a storage circuit such as a memory. In addition, the storage unit 36 is used for the operation of the CPU 38 such as a program used for controlling each unit of the endoscope apparatus 1 and a program used for processing related to generation of a wide-area expanded image (described later). Various corresponding programs are stored. In addition, the storage unit 36 is configured to store a plurality of locally developed images (described later) generated in the course of processing related to the generation of the wide area developed image by the CPU 38. Further, the storage unit 36 is configured to be able to store information input in response to an operation of the input I / F unit 37. In addition, the storage unit 36 stores parameters used in processing related to generation of a wide area developed image.

The input I / F unit 37 is configured to include a switch or the like that can instruct the CPU 38 according to a user's input operation. In addition, the input I / F unit 37 is configured to be able to input information used in processing related to generation of a wide area developed image by the CPU 38 in accordance with a user operation.

The CPU 38 is configured to be able to control the light source driving unit 32 and the image sensor driving unit 33 based on an instruction made in response to an operation of the touch panel 35a or the input I / F unit 37. The CPU 38 includes a timer (not shown) for measuring time. The CPU 38 is configured to measure the insertion distance of the insertion portion 5 based on, for example, an output signal from an acceleration sensor (not shown) provided at the distal end portion 11. The CPU 38 is configured to generate a display image in which a GUI button or the like is superimposed on an image such as an endoscopic image output from the image generation unit 34 and output the display image to the display unit 35. . Further, the CPU 38 is configured to be able to perform an operation for storing information input in accordance with an operation of the input I / F unit 37 in the storage unit 36. Further, the CPU 38 has a function as an image processing unit, and generates a locally expanded image by expanding the annular endoscope images sequentially output from the image generating unit 34 one by one, and generates the generated local It is configured to be able to perform a process for generating a wide area developed image by pasting a plurality of developed images. Details of such processing will be described later. Further, the CPU 38 is configured to be able to store the locally developed image generated in the process of generating the wide area developed image in the storage unit 36. Further, the CPU 38 is configured to be able to perform an operation for storing the wide area expanded image generated as described above in the external storage device 51. Further, the CPU 38 encodes the endoscopic image output from the image generation unit 34 using a still image format such as JPEG and a moving image format such as MPEG4 and stores the encoded image in the external storage device 51. It is configured to be able to. Further, the CPU 38 reads an image stored in the external storage device 51 and generates a display image corresponding to the read image based on an instruction made in response to an operation of the touch panel 35 a or the input I / F unit 37. And can be output to the display unit 35. The CPU 38 is configured to perform predetermined image processing such as color space conversion, interlace / progressive conversion, and gamma correction on the display image output to the display unit 35.

Subsequently, specific operations and the like of the endoscope apparatus 1 of the present embodiment will be described.

The user inputs information used in the processing related to the generation of the wide area developed image by operating the input I / F unit 37 after connecting each unit of the endoscope apparatus 1 and turning on the power. Specifically, the user operates the input I / F unit 37, for example, the inner diameter Φ of a pipe that is a cylindrical subject having a central axis, and a cutout reference position used when generating a locally developed image. And the matching accuracy in the pasting of the locally developed images and the number m (1 ≦ m) of re-obtained endoscopic images when a locally developed image that is not suitable for the pasting is generated. That is, according to the user's operation, the inner diameter Φ of the pipe, the cut-out reference position used when generating the locally developed image, the matching accuracy in the joining of the locally developed images, and the local development that is not suitable for the joining. Information that can specify the number m of re-obtained endoscope images when an image is generated is stored in the storage unit 36.

Thereafter, the user inserts the insertion unit 5 into the pipe while confirming the endoscopic image displayed live on the display unit 35. In the following, as shown in FIG. 5, the case where the insertion portion 5 is inserted with the optical axis AC decentered by an offset amount Δ with respect to the central axis AS of the cylindrical pipe 101 is taken as an example. explain. That is, in the following description, it is assumed that the offset amount Δ corresponds to the eccentric amount of the optical axis AC with respect to the central axis AS. FIG. 5 is a diagram illustrating an example of a state where the insertion portion of the endoscope is inserted into the pipe.

Here, when the insertion portion 5 is inserted in a state where the optical axis AC is shifted from the central axis AS, illumination light emitted through the illumination window 48 reaches the inner peripheral surface 101a of the pipe 101. A region that is imaged in a state where the illumination light is present and a region that is imaged in a state where the illumination light does not reach each occur. That is, when the insertion portion 5 is inserted in a state where the optical axis AC is decentered with respect to the central axis AS, for example, as shown in FIG. 6, an imaging region when imaging the inner peripheral surface 101 a of the pipe 101. Within the IR, an area IRB belonging to the inside of the illumination area LR (with illumination light reaching) and an area IRS outside the illumination area LR (with no illumination light reaching) are respectively generated. FIG. 6 is a diagram for explaining regions IRB and IRS generated in the imaging region IR in the insertion state as shown in FIG.

Therefore, when the insertion portion 5 is inserted into the pipe 101 with the optical axis AC shifted from the central axis AS, for example, as shown in FIG. An imaging region is depicted in the center, and an illuminated region corresponding to the region IRB of the inner peripheral surface 101a and a shadow region corresponding to the region IRS of the inner peripheral surface 101a are included in the non-imaging region. An annular endoscopic image drawn around is generated by the image generation unit 34. FIG. 7 is a diagram showing an example of an endoscopic image generated in the insertion state as shown in FIG.

The CPU 38 performs an operation for generating a display image 301 as shown in FIG. 8 and outputting it to the display unit 35 using the endoscopic image output from the image generation unit 34, for example. FIG. 8 is a diagram illustrating an example of a display image generated according to the operation of the endoscope apparatus according to the embodiment.

The display image 301 includes an annular endoscope image 201 and a developed image acquisition start button 202. In the display image 301, the endoscopic image 201 is displayed live.

The developed image acquisition start button 202 is configured as a GUI button capable of giving an instruction to start acquisition of a locally developed image used for generating a wide-area developed image, for example, in response to a user's touch operation.

In a state where the insertion unit 5 is inserted into the pipe, the user presses a development image acquisition start button 202 displayed on the display unit 35 to start acquisition of a local development image used for generating a wide-area development image. Give instructions to do so.

The CPU 38 starts acquiring a locally developed image used for generating a wide area developed image based on an instruction made in response to an operation of the touch panel 35a, starts measuring the insertion distance of the insertion unit 5, and, for example, in FIG. An operation for generating a display image 302 as shown and outputting it to the display unit 35 is performed. FIG. 9 is a diagram illustrating an example of a display image generated according to the operation of the endoscope apparatus according to the embodiment.

The display image 302 includes an annular endoscope image 201, a developed image acquisition end button 203, and insertion distance information 204. In the display image 302, the endoscopic image 201 is displayed live.

The developed image acquisition end button 203 is configured as a GUI button capable of giving an instruction to end acquisition of a locally developed image used for generating a wide-area developed image, for example, in response to a user's touch operation.

The insertion distance information 204 includes a character string for notifying the user of the insertion distance of the insertion unit 5 measured by the CPU 38. The insertion distance information 204 is updated substantially in real time according to the advance / retreat movement of the insertion unit 5 during the period from when the developed image acquisition start button 202 is pressed to when the developed image acquisition end button 203 is pressed. Therefore, at the timing immediately after the developed image acquisition start button 202 is pressed, for example, a character for notifying that the insertion distance of the insertion unit 5 is 0, such as “insertion distance: 0 mm” in FIG. Insertion distance information 204 including a column is displayed on the display unit 35.

After the user presses the developed image acquisition start button 202, the user inserts the insertion unit 5 into the back side of the pipe while confirming the display image 302 displayed on the display unit 35.

Here, a specific example of an operation related to generation and display of a wide area developed image will be described with reference to FIG. FIG. 10 is a flowchart for explaining an example of an operation related to generation and display of a wide area developed image performed in the endoscope apparatus according to the embodiment.

The CPU 38 performs processing for developing one endoscopic image output from the image generating unit 34 and generating one locally developed image (step S1 in FIG. 10). Further, the CPU 38 adds information that can identify that the image is the first image of the wide area developed image to the locally developed image generated by the process of step S1 in FIG. After step S2), the process of step S3 in FIG.

Here, the process related to the generation of the locally developed image performed in step S1 and the like in FIG. 10 will be described by taking as an example the case where an endoscopic image as shown in FIG. 11 is used. For convenience of explanation, in the endoscopic image of FIG. 11, the illuminated region BR corresponding to the region IRB of FIG. 6 is generated in the offset direction that is the eccentric direction of the optical axis AC with respect to the central axis AS. In addition, it is divided into four regions: a dark region DR that occurs in a direction opposite to the offset direction, and intermediate regions MRA and MRB that occur in the middle of the bright region BR and the dark region DR. For the sake of simplicity of explanation and illustration, it is assumed that the shadow region generated in accordance with the region IRS in FIG. 6 is omitted in the endoscopic image in FIG. FIG. 11 is a diagram illustrating an example of an endoscopic image used for processing related to generation of a locally developed image.

In the polar coordinate system in which the center of the annular endoscope image output from the image generation unit 34 is set as the origin, the CPU 38 has coordinate values corresponding to each pixel position other than the non-imaging region included in the endoscope image. To get.

The CPU 38 acquires the coordinate value of the orthogonal coordinate system by applying the Jacobian matrix to the coordinate value of the polar coordinate system acquired as described above, and the endoscopic image according to the acquired coordinate value of the orthogonal coordinate system By rearranging the pixels included in the basic image, for example, as shown in FIG. 12, a basic expanded image IEA in which the endoscope image is expanded in a rectangular shape is generated. FIG. 12 is a diagram illustrating an example of a basic developed image generated by developing the endoscopic image of FIG.

The basic developed image IEA is an image in which the direction parallel to the optical axis AC in the cylindrical imaging region IR, that is, the direction parallel to the central axis AS of the pipe 101 is the vertical direction (hereinafter also referred to as the Y-axis direction). Is generated as The basic developed image IEA is generated as an image in which the circumferential direction of the cylindrical imaging region IR, that is, the circumferential direction of the inner peripheral surface 101a is the left-right direction (hereinafter also referred to as the X-axis direction). In addition, in the X-axis direction of the basic developed image IEA, the inner peripheral surface 101a imaged in the cylindrical imaging region IR is included for one round (360 degrees).

The CPU 38 generates, for example, a rectangular combined image IJA as shown in FIG. 13 by combining two basic expanded images IEA generated as described above in the X-axis direction. That is, in the X-axis direction of the combined image IJA, the inner peripheral surface 101a imaged in the cylindrical imaging region IR is included for two rounds (for 720 degrees). FIG. 13 is a diagram illustrating an example of a combined image generated using the basic development image of FIG.

The CPU 38 performs processing for generating a cutout image that is an image obtained by cutting out the combined image IJA for one round (for 360 degrees) based on the information of the cutout reference position stored in the storage unit 36. Specifically, for example, when the cutout reference position stored in the storage unit 36 is the dark region DR, the CPU 38 sets the combined image IJA to 1 so that the dark region DR is arranged at both ends in the X-axis direction. By cutting out the circumference (360 degrees), a rectangular cut-out image IEB as shown in FIG. 14 is generated. FIG. 14 is a diagram illustrating an example of a cutout image generated by cutting out a part of the combined image in FIG.

The CPU 38 estimates the direction in which the shadow area included in the endoscopic image used for generating the basic development image IEA is the maximum as the offset direction, and calculates the width WS of the shadow area in the offset direction (See FIG. 15). Further, the CPU 38 performs a process for calculating the offset amount Δ based on the width WS calculated as described above and the inner diameter Φ of the pipe 101 stored in the storage unit 36. FIG. 15 is a diagram for explaining the width WS in the offset direction of the shadow region included in the endoscopic image used for generating the basic development image.

In this embodiment, instead of the width WS in the offset direction of the shadow area, the CPU 38 sets the center pixel Dp in the dark area DR depicted in the endoscopic image used for generating the basic developed image IEA, for example. A ratio value RV (= Dpi / Bpi) obtained by dividing the luminance value Dpi by the luminance value Bpi of the central pixel Bp in the bright region BR depicted at a position facing the dark region DR of the endoscopic image. It may be used to calculate the offset amount Δ. In the present embodiment, the CPU 38 may calculate the offset amount Δ based on the width WS of the shadow region in the offset direction, the ratio value RV, and the inner diameter Φ of the pipe 101. Good.

Based on the outer diameter DA of the distal end portion 11, the outer diameter DB of the free curved surface lens 43, the inner diameter Φ of the pipe 101, and the offset amount Δ, the CPU 38 determines the outermost and inner peripheral surfaces 101 a of the free curved surface lens 43. A process for calculating the distance WD (see FIG. 16) between the entire circumference of the inner peripheral surface 101a is performed. It should be noted that the outer diameters DA and DB are known parameters used in processing related to the generation of the wide area developed image, and are stored in advance in the storage unit 36 before the processing is performed, for example. FIG. 16 is a diagram showing an outline of parameters used in the processing related to the generation of the wide area developed image.

Here, for example, when the distance WD in the direction of the azimuth angle θ about the optical axis AC is WD (θ), the imaging magnification β (θ when the subject located in the direction of the azimuth angle θ is imaged ) Can be expressed as the following formula (1). In the following formula (1), f represents the focal length of the free-form surface lens 43, and Enp in the following formula (1) represents the distance between the outermost periphery of the free-form surface lens 43 and the entrance pupil position (see FIG. 16). It shall represent. In addition, f and Enp in the following mathematical formula (1) are known parameters used in the process related to the generation of the wide area expanded image, and are stored in the storage unit 36 in advance before the process is performed, for example. To do.

β (θ) = f / (WD (θ) + Enp) (1)
Further, when taking into consideration that the azimuth angle θ and the position of the basic developed image IEA in the X-axis direction are in a corresponding relationship, by applying the distance WD (θ) to the above equation (1), the X of the cut-out image IEB An imaging magnification β (θ) at a position corresponding to the azimuth angle θ in the axial direction can be calculated.

The CPU 38, based on the imaging magnification β (θ) calculated using the above mathematical formula (1) and the predetermined reference magnification βth, the Y-axis direction of the position corresponding to the azimuth angle θ in the X-axis direction of the cut-out image IEB A magnification correction process is performed to correct the width of the image by βth / β (θ) times. According to such a magnification correction process, for example, a pincushion-shaped cut-out image IEC as shown in FIG. 17 is generated. FIG. 17 is a diagram illustrating an example when the cutout image of FIG. 14 is deformed by the magnification correction process.

That is, in the cutout image IEC, as shown in FIG. 17, the width of the dark region DR located at both ends (near θ = 180 degrees and −180 degrees) of the cutout image IEB in FIG. 14 is expanded in the Y-axis direction. The width of the bright area BR located in the center of the cutout image IEB (near θ = 0 degrees) is reduced in the Y-axis direction, and the widths of the intermediate areas MRA and MRB in the cutout image IEB are set to the imaging magnification β ( is generated as an image scaled in the Y-axis direction according to θ). Further, according to the cut-out image IEC in FIG. 17, the width WM in the Y-axis direction at the center position in the X-axis direction, which is a position corresponding to θ = 0 °, is the minimum width.

The CPU 38 generates a rectangular locally developed image IEL as shown in FIG. 18 by performing a process of cutting out the upper and lower portions of the cut-out image IEC in accordance with the width WM. FIG. 18 is a diagram illustrating an example of a locally developed image generated by excising a part of the deformed cutout image of FIG.

That is, according to the processing described above, the CPU 38 calculates the offset amount Δ based on the inner diameter Φ and the width WS and / or the ratio value RV, and further, in the Y-axis direction of the cutout image IEB. The locally developed image IEL is generated by performing a magnification correction process for correcting the imaging magnification according to the offset amount Δ.

The CPU 38 performs the same processing as step S1 in FIG. 10 to develop one endoscopic image output from the image generation unit 34 and generate one locally developed image (step in FIG. 10). S3).

The CPU 38 performs a determination process related to whether or not the locally developed image generated by the process of step S3 in FIG. 10 is an image suitable for pasting (step S4 in FIG. 10).

Specifically, the CPU 38 performs, for example, the feature points and feature amounts acquired from the locally developed image IEL1 stored in the storage unit 36 immediately before performing the process of step S3 of FIG. 10 and the process of step S3 of FIG. By comparing the feature points and feature quantities acquired from the generated locally expanded image IEL2, the Y-axis direction of the locally expanded image IEL2 overlaps the width WM of the locally expanded image IEL1 in the Y-axis direction. The ratio PA is calculated. Then, when the CPU 38 detects that the ratio PA calculated as described above does not belong to the threshold range TR, the CPU 38 obtains a determination result that the locally developed image IEL2 is an image that is not suitable for pasting. Further, when the CPU 38 detects that the ratio PA calculated as described above belongs to the threshold range TR, the CPU 38 obtains a determination result that the locally developed image IEL2 is an image suitable for pasting.

That is, the CPU 38 has a function as a determination unit that performs the determination process as described above.

The threshold range TR may be set as a range of 20% or more and less than 50%, for example. Further, the threshold range TR may be enlarged and reduced according to the matching accuracy stored in the storage unit 36. Specifically, for example, when the matching accuracy stored in the storage unit 36 is 100%, the threshold range TR may be set to only 20%.

When the CPU 38 obtains a determination result that the locally developed image generated by the process of step S3 of FIG. 10 is an image not suitable for pasting (S4: NO), the operation of step S5 of FIG. 10 described later is performed. I do. In addition, when the CPU 38 obtains a determination result that the locally developed image generated by the process of step S3 in FIG. 10 is an image suitable for pasting (S4: YES), the CPU 38 in step S8 of FIG. Perform the action.

CPU38 performs the operation | movement for producing | generating the character string etc. for making the request | requirement regarding the advance / retreat movement of the insertion part 5 with respect to a user, and making it display on the display part 35 (step S5 of FIG. 10).

Specifically, for example, when the ratio PA calculated by the process of step S4 in FIG. 10 is less than the lower limit of the threshold range TR, that is, the overlapping portion of the locally developed image IEL2 with respect to the locally developed image IEL1 is insufficient. If it is present or does not exist, an operation for generating a character string or the like for returning the position of the insertion unit 5 backward from the current position and displaying it on the display unit 35 is performed. Further, for example, when the ratio PA calculated by the process of step S4 in FIG. 10 exceeds the upper limit of the threshold range TR, that is, when the overlapping portion of the locally developed image IEL2 with respect to the locally developed image IEL1 is excessive. In addition, an operation for generating a character string or the like for increasing the insertion speed of the insertion unit 5 forward from the current speed and displaying it on the display unit 35 is performed.

After performing the operation of step S5 in FIG. 10, the CPU 38 performs the same process as step S1 in FIG. 10 continuously m times based on the number m of re-obtained endoscope images stored in the storage unit 36. Thus, m local development images corresponding to the m endoscopic images output from the image generation unit 34 are generated (step S6 in FIG. 10).

The CPU 38 performs a determination process similar to that in step S4 in FIG. 10 on each of the m locally developed images generated by the process in step S6 in FIG. It is determined whether there is an image suitable for (step S7 in FIG. 10).

When the CPU 38 obtains a determination result that there is no image suitable for pasting in the m locally developed images generated by the process of step S6 of FIG. 10 (S7: NO), the CPU 38 of FIG. The operations of S5 and S6 are performed again. When the CPU 38 obtains a determination result that an image suitable for pasting exists among the m locally developed images generated by the process of step S6 of FIG. 10 (S7: YES), the CPU 38 of FIG. The operation of step S8 is performed.

That is, the CPU 38 performs the operation relating to the generation of the locally developed image again when the determination condition of step S4 or step S7 in FIG. 10 is not satisfied.

If the CPU 38 detects that there are a plurality of locally developed images suitable for pasting in step S7 of FIG. 10, for example, the CPU 38 has a ratio PA closest to a predetermined value included in the threshold range TR. Assume that after selecting one locally developed image, the operation proceeds to step S8 in FIG.

The CPU 38 adds information that can identify that the image is the n (2 ≦ n) th image of the wide area developed image to the locally developed image that satisfies the determination condition of step S4 or step S7 in FIG. 36 (step S8 in FIG. 10), an operation for generating a character string or the like for requesting the user to maintain the insertion state of the insertion unit 5 and displaying it on the display unit 35 is performed (FIG. 10). Step S9). That is, in step S8 in FIG. 10, the CPU 38 performs an operation for storing the locally developed image satisfying the determination condition in step S4 or step S7 in FIG.

Specifically, in step S9 of FIG. 10, for example, the CPU 38 generates a character string or the like indicating that the insertion of the insertion unit 5 is continued while maintaining the current insertion speed (and insertion direction), and displays the display unit 35. Performs the operation to display on the screen.

CPU38 performs the determination process which concerns on whether acquisition of a local expansion | deployment image is complete | finished (step S10 of FIG. 10).

For example, the CPU 38 cannot detect an instruction in response to pressing of the developed image acquisition end button 203 and detects that the storage unit 36 has a free space capable of storing at least one locally developed image. If it is (S10: NO), the operation from step S3 in FIG. 10 is performed again.

That is, by repeatedly performing the operations from step S3 to step S10 in FIG. 10, a plurality of locally developed images used for generating a wide area developed image are sequentially stored in the storage unit 36 (in order from the first).

For example, the CPU 38 detects that an instruction in response to pressing of the developed image acquisition end button 203 has been performed, or that the storage unit 36 does not have a free space capable of storing at least one locally developed image. Is detected (S10: YES), the measurement of the insertion distance of the insertion unit 5 is stopped, and each locally developed image stored in the storage unit 36 is pasted to generate one wide-area developed image. Processing is performed (step S11 in FIG. 10). That is, when the CPU 38 ends the operation for storing the locally developed image satisfying the determination condition of step S4 or step S7 of FIG. 10 in the storage unit 36 in time series, each CPU stored in the storage unit 36 An operation for generating one wide area developed image by pasting the locally developed images and causing the display unit 35 to display the generated one wide area developed image is performed.

Specifically, the CPU 38 calculates, for example, a homography matrix of two numbers adjacent to each other with respect to the first to p (2 ≦ p) th locally developed images IEL stored in the storage unit 36. In addition, by performing a process of pasting the images together, a single wide area developed image IEW as shown in FIG. 19 is generated. FIG. 19 is a diagram illustrating an example of a wide area developed image generated by pasting together a plurality of locally developed images.

CPU38 performs the operation | movement for producing | generating the display image 303 as shown in FIG. 20, for example instead of the display image 302 shown in FIG. 9, and outputting it to the display part 35 (step S11 of FIG. 10). FIG. 20 is a diagram illustrating an example of a display image generated according to the operation of the endoscope apparatus according to the embodiment.

The display image 303 includes insertion distance information 204, a rectangular wide area developed image 205, and a live image display button 206. The insertion distance information 204 of the display image 303 includes a character string for notifying the user of the insertion distance of the insertion unit 5 at the timing when the measurement by the CPU 38 is stopped.

The live image display button 206 is configured, for example, as a GUI button that can issue an instruction to cancel the display of the wide-area developed image 205 and display the endoscopic image 201 live according to a user's touch operation. Yes.

The CPU 38 performs a determination process related to whether or not to end the display of the wide area developed image (step S12 in FIG. 10).

The CPU 38 stands by while performing an operation for outputting the display image 303 to the display unit 35 until an instruction corresponding to the pressing of the live image display button 206 is detected (S12: NO).

When the CPU 38 detects that an instruction corresponding to the pressing of the live image display button 206 has been performed (S12: YES), the CPU 38 stores the wide area developed image generated in step S11 of FIG. In addition, after performing an operation for displaying the display image 301 again instead of the display image 303, a series of processes relating to generation and display of the wide area expanded image is completed.

In the present embodiment, the CPU 38 is not limited to performing the operation for generating the wide area developed image and displaying it on the display unit 35 after the storage of the locally developed image in the storage unit 36 is completed. Alternatively, an operation for generating a wide area developed image and displaying it on the display unit 35 while storing the locally developed image in the storage unit 36 may be performed.

Specifically, for example, as shown in a display image 304a in FIG. 21, the CPU 38 acquires the latest local development image when the latest local development image satisfying the determination condition in step S4 or step S7 in FIG. 10 is acquired. While performing the operation for storing the image in the storage unit 36, one or more locally developed images already stored in the storage unit 36 and the latest locally developed image are pasted together to form a wide area developed image 211. An operation for generating and displaying the generated wide area developed image 211 on the display unit 35 may be performed. FIG. 21 is a diagram illustrating an example of a display image generated according to the operation of the endoscope apparatus according to the embodiment.

The display image 304a includes an endoscope image 201, a developed image acquisition end button 203, and insertion distance information 204 that are substantially the same as those included in the display image 302 of FIG. Further, according to the display image 304a, each time a locally developed image satisfying the determination condition of step S4 or step S7 in FIG. 10 is acquired, a wide area developed image 211 obtained by pasting the locally developed image is displayed on the display unit 35. Is done. In addition, according to the display image 304a, for example, as indicated by the alternate long and short dash line in FIG. 21, each time the locally developed image satisfying the determination condition in step S4 or step S7 in FIG. The area is gradually expanded.

Alternatively, for example, as shown in the display image 304b in FIG. 22, the CPU 38 stores the latest locally developed image when the latest locally developed image satisfying the determination condition in step S4 or step S7 in FIG. 10 is acquired. While performing the operation for storing in the unit 36, the operation for causing the display unit 35 to display the wide area expanded image 211 and the locally expanded image 221 corresponding to the latest locally expanded image together is performed. Also good. FIG. 22 is a diagram illustrating an example of a display image generated according to the operation of the endoscope apparatus according to the embodiment.

The display image 304b includes a developed image acquisition end button 203 and insertion distance information 204 that are substantially the same as those included in the display image 302 of FIG. Further, according to the display image 304b, each time the locally developed image 221 that satisfies the determination condition of step S4 or step S7 in FIG. 10 is acquired, the locally developed image 221 and the locally developed image 221 are pasted together. 211 are also displayed on the display unit 35. In addition, according to the display image 304b, for example, as indicated by a one-dot chain line in FIG. 22, each time a locally developed image satisfying the determination condition in step S4 or step S7 in FIG. The area is gradually expanded.

As described above, according to the present embodiment, the offset amount Δ is calculated based on the endoscopic image output from the image generation unit 34, and the magnification correction process according to the calculated offset amount Δ is performed. While performing, a locally developed image and a wide area developed image are generated. Therefore, according to the present embodiment, for example, without using a centering mechanism configured to be able to insert the insertion portion 5 into the pipe 101 while matching the optical axis AC and the center axis AS. Can be generated.

In addition, according to the present embodiment, since the developed image can be generated without providing the centering mechanism as described above in the insertion portion 5, for example, the inside of a small diameter pipe corresponding to the outer diameter of the insertion portion 5. The state can be confirmed with a wide area developed image.

In addition, according to the present embodiment, the operations of Step S5 to Step S7 in FIG. 10 are repeatedly performed, so that the nth locally developed image suitable for pasting to the (n−1) th locally developed image is reliably acquired. can do. Therefore, according to the present embodiment, for example, even when the insertion state of the insertion portion 5 inserted into the pipe 101 is temporarily disturbed, an accurate wide area developed image can be generated.

It should be noted that the present invention is not limited to the above-described embodiment, and various changes and applications can be made without departing from the spirit of the invention.

This application is filed on the basis of the priority claim of Japanese Patent Application No. 2017-10126 filed in Japan on January 24, 2017. The above disclosure is included in the present specification and claims. Shall be quoted.

Claims

An illumination unit that is arranged inside a cylindrical subject having a central axis and configured to emit illumination light to a cylindrical illumination region centered on a predetermined optical axis;
Provided in the vicinity of the illumination unit, arranged inside the subject, and configured to image a subject in a cylindrical imaging region centered on the predetermined optical axis and output an imaging signal An imaging unit,
An image generation unit configured to generate an annular image having the predetermined optical axis as a center of the image based on the imaging signal;
An image obtained by calculating an offset amount of the predetermined optical axis with respect to the central axis based on an inner diameter of the subject and a value acquired from the annular image, and further developing the annular image An image processing unit configured to generate a locally developed image by performing a magnification correction process for correcting the imaging magnification in a direction parallel to the central axis according to the offset amount;
An observation apparatus comprising:
The image processing unit is based on the inner diameter of the subject and the width of a shadow area drawn in the annular image when an area outside the illumination area in the imaging area is imaged. The observation apparatus according to claim 1, wherein an offset amount is calculated.
The image processing unit includes an inner diameter of the subject, a luminance value of a central pixel in a dark region depicted in the annular image, and a central pixel in a bright region depicted at a position facing the dark region The observation apparatus according to claim 1, wherein the offset amount is calculated based on a luminance value of the observation value.
A determination unit configured to perform determination processing related to whether or not the locally developed image is suitable for pasting;
The image processing unit performs an operation for storing the locally developed image, which is obtained as a result of determination as being suitable for pasting by the determination process, in a time series in a storage unit. The observation apparatus described in 1.
When the image processing unit finishes the operation for causing the storage unit to store the locally developed image that is determined to be suitable for pasting by the determination process in time series, the storage unit A single wide-area development image is generated by pasting each locally developed image stored in the image, and an operation for displaying the generated single wide-area development image on a display unit is performed. Item 5. The observation device according to Item 4.
The image processing unit is already stored in the storage unit while performing an operation for storing in the storage unit the latest locally developed image that has been determined to be suitable for pasting by the determination process. Generating one wide area developed image by combining one or more local developed images and the latest locally developed image, and displaying the generated one wide area developed image on the display unit The observation apparatus according to claim 4, wherein the observation apparatus performs an operation.
The said image processing part performs operation | movement which concerns on the production | generation of the said local expansion | deployment image again, when the determination result that it is not suitable for bonding is obtained by the said determination process. Observation device.