KR20230093891A - Apparatus for uniting underwater images and method therof - Google Patents

Abstract

An embodiment of the present invention relates to an apparatus and method for uniting an optical image captured using an RGB camera under water and a sonar image captured using a sonar camera and, more specifically, to an apparatus for uniting images and a method for uniting underwater images using the same to confirm a distance between the image of an underwater object that can be confirmed by an optical image and an underwater object that can be confirmed by an sonar image as an united image.

Description

Underwater image fusion device and underwater image fusion method using the same {APPARATUS FOR UNITING UNDERWATER IMAGES AND METHOD THEROF}

The present invention relates to an apparatus and method for fusing an optical image captured using an RGB camera and a sonar image captured using a sonar camera in water, and more particularly, to an image of an underwater object that can be confirmed with an optical image and a sonar image. The present invention relates to an underwater image convergence device and an underwater image convergence method using the same to confirm a distance to an underwater object that can be confirmed by sonar image as a single convergence image.

The contents described below are only described for the purpose of providing background information related to an embodiment of the present invention, and the contents described do not naturally constitute prior art.

Recently, there is an increasing demand for safely carrying out extreme underwater work by introducing an unmanned underwater vehicle instead of a person. When working underwater using an underwater drone, an image sensor that visualizes the underwater situation more accurately is one of the essential sensing technologies.

Among them, the most widely used underwater imaging sensors are optical cameras and multi-beam sonars.

An optical camera detects a light signal and provides an image in red, green, blue (RGB) tones, and a multi-beam sonar uses sound waves to express distance and direction information to a subject as an image.

However, in the case of underwater optical imaging, there is a limitation in that signal attenuation caused by water occurs differently depending on the wavelength of light, resulting in color casting of RGB colors. In addition, there is a problem that contrast and sharpness may be deteriorated due to light scattering and reflection by plankton and floating matter in the water.

Unlike this, multi-beam sonar has the advantage of being able to detect underwater even in situations where visibility is not secured due to the characteristics of sound waves that penetrate water well. However, there is a limit in that the quality of the sonar image is degraded due to low resolution and signal-to-noise ratio, and it is very difficult for non-experts to intuitively recognize the object of interest in the image and obtain necessary information.

Accordingly, there is a demand for an apparatus and method capable of more accurately visualizing an underwater environment.

The foregoing background art is technical information that the inventor possessed for derivation of the present invention or acquired during the derivation process of the present invention, and cannot necessarily be said to be known art disclosed to the general public prior to filing the present invention.

An embodiment of the present invention is to provide an underwater image fusion device capable of more accurately visualizing an underwater environment and an underwater image fusion method using the same.

In addition, an embodiment of the present invention is to provide an underwater image convergence device capable of improving the quality of an image captured underwater and an underwater image convergence method using the same.

The object of the present invention is not limited to the above-mentioned tasks, and other objects and advantages of the present invention not mentioned above can be understood by the following description and will be more clearly understood by the embodiments of the present invention. It will also be seen that the objects and advantages of the present invention may be realized by means of the instrumentalities and combinations indicated in the claims.

An underwater image fusion method according to an embodiment of the present invention collects at least one optical image and a sonar image of an underwater object for monitoring underwater information, corrects the color tone of the collected optical image, and corrects the corrected optical image. It can be performed through a process of generating a corrected sonar image by removing noise included in the sonar image, and then generating a fusion image in which the corrected optical image and the corrected sonar image are fused.

According to an embodiment of the present invention, after color correction of the optical image, the degree of improvement of the corrected optical image may be evaluated through a weighted sum of saturation, sharpness, and contrast of the corrected optical image. can

According to an embodiment of the present invention, when correcting the color tone of an optical image, the white balance of the optical image is set, the image of the optical image is optimized through image histogram equalization, and then the optimized optical image is unsharp masked ), it is possible to remove noise included in the optical image.

According to an embodiment of the present invention, when generating a sonar image, a region of interest may be set in the sonar image, and a contrast to noise ratio (CNR) within the region of interest may be measured.

에 의해 정의될 수 있다. According to an embodiment of the present invention, generating a sonar image includes applying a median filter and gamma correction to the sonar image, wherein the gamma correction comprises:

can be defined by

According to an embodiment of the present invention, when generating a fusion image, setting an arbitrary first point and a second point in the corrected optical image and the corrected sonar image, and matching the first point and the second point process can be performed.

According to an embodiment of the present invention, when setting a first point and a second point, image coordinates are extracted from the first point, the image coordinates are converted into a coordinate matrix, and then a transformation matrix for extracting the coordinate matrix is used. The coordinate matrix may be extracted by sequentially multiplying a preset reference matrix.

According to an embodiment of the present invention, a matrix in which a rotation matrix, which is a matrix that rotates the x, y, and z axes of the image coordinates, and a transition matrix representing the degree of movement of the x, y, and z axes of the image coordinates are listed is a matrix of the optical image. The conversion matrix may be extracted by sequentially multiplying a pixel size and an image parameter matrix, which is a matrix expressed by parameter values of the center point of the optical image.

According to an embodiment of the present invention, the first point designating the corner of the underwater object is set in the corrected optical image, and the second point designating the distance of the underwater object is set in the corrected sonar image. After that, the shape of the underwater object and the distance to the underwater object may be measured by matching the first point and the second point.

An underwater image convergence device according to an embodiment of the present invention collects at least one optical image of an underwater object and a sonar image for monitoring underwater information, corrects the color tone of the optical image, and adds the sonar image to the sonar image. Codes for generating a fusion image in which the optical image and the sonar image are fused may be stored when an interface and a contrast between the included underwater object and the periphery of the underwater object are improved.

Other aspects, features, and advantages other than those described above will become apparent from the following drawings, claims, and detailed description of the invention.

According to an embodiment of the present invention, an underwater object may be photographed underwater using an RGB optical camera, and at the same time, an underwater subject may be photographed using an underwater sonar. The captured optical image and the sonar image can be improved in color correction and contrast, and the two improved images can be fused.

The fusion image includes color tones and may include distance information between the fusion device and the underwater object. That is, the fusion image can intuitively recognize an underwater object and measure the distance to the underwater object through sound waves in the water. That is, there is an effect of simultaneously measuring and obtaining intuitive information of an underwater object and distance information to the underwater object.

The effects of the present invention are not limited to those mentioned above, and other effects not mentioned will be clearly understood by those skilled in the art from the description below.

1 is a diagram illustrating an underwater image fusion environment to which an underwater image fusion device according to an embodiment of the present invention is applied.
FIG. 2 is a perspective view illustrating the underwater image convergence device of FIG. 1 .
3 is a block diagram showing the configuration of an underwater video convergence device according to an embodiment of the present invention.
4 is a flowchart illustrating an underwater image fusion process according to an embodiment of the present invention.
5 is a diagram illustrating a fusion process of an optical image and a sonar image acquired through an underwater image convergence device according to an embodiment of the present invention.
6 to 8 are diagrams illustrating a process of correcting an optical image acquired through an underwater image convergence device according to an embodiment of the present invention.
9 is a diagram illustrating a process of correcting a sonar image acquired through an underwater image convergence device according to an embodiment of the present invention.

Hereinafter, the embodiments invented in this specification will be described in detail with reference to the accompanying drawings, but the same or similar components are assigned the same reference numerals regardless of reference numerals, and redundant description thereof will be omitted. The suffixes "module" and "unit" for components used in the following description are given or used together in consideration of ease of writing the specification, and do not have meanings or roles that are distinct from each other by themselves. In addition, when it is determined that detailed descriptions of related known technologies may obscure the gist of the embodiments disclosed in this specification in describing the embodiments disclosed herein, the detailed descriptions thereof will be omitted. In addition, the accompanying drawings are only for making it easy to understand the embodiments invented in this specification, and the technical ideas invented in this specification are not limited by the accompanying drawings, and are included in the spirit and technical scope of the present invention. It should be understood to include all modifications, equivalents or substitutes.

Terms including ordinal numbers, such as first and second, may be used to describe various components, but the components are not limited by the terms. These terms are only used for the purpose of distinguishing one component from another.

It is understood that when an element is referred to as being "connected" or "connected" to another element, it may be directly connected or connected to the other element, but other elements may exist in the middle. It should be. On the other hand, when an element is referred to as “directly connected” or “directly connected” to another element, it should be understood that no other element exists in the middle.

Expressions in the singular number include plural expressions unless the context clearly dictates otherwise.

In this application, terms such as "comprise" or "have" are intended to designate that there is a feature, number, step, operation, component, part, or combination thereof described in the specification, but one or more other features It should be understood that the presence or addition of numbers, steps, operations, components, parts, or combinations thereof is not precluded.

1 is a diagram illustrating an environment of an underwater image fusion system to which an underwater image fusion device according to an embodiment of the present invention is applied.

Referring to FIG. 1 , an underwater image fusion environment 10 to which an underwater image fusion device according to an embodiment of the present invention is applied captures an underwater object located in the water and intuitively identifies the object based on the captured video or image. At the same time, the distance to the underwater object and the direction of the underwater object can be measured.

To this end, in the underwater video convergence environment 10, the device 100 and the server 200 for underwater video convergence are communicatively connected through a network 300.

Specifically, the underwater image convergence device 100 is a device capable of simultaneously capturing an underwater object as an optical image and a sonar image in the water. An example including a multi-beam sonar 130 capable of capturing sonar images between the 120 and the pair of optical cameras 120 will be described.

The underwater image convergence device 100 may be installed at a predetermined distance from an underwater object. In particular, the optical camera 120 is installed to be rotatable in one direction or 360 degrees so that an underwater object can be photographed from various directions.

The underwater image convergence apparatus 100 may simultaneously acquire an optical image and a sonar image of an underwater object in a photographing mode. The acquired optical image and sonar image may be stored in the memory 130 included in the underwater image convergence device 100 or transmitted to the server 200 and stored in a memory (not shown) of the server 200.

The server 200 provides various services for monitoring intuitive information (eg, color tone) of an underwater object described in an embodiment of the present invention and information such as a distance to the underwater object or a direction in which the underwater object is located. In detail, the server 200 may obtain an optical image and a sonar image from an image or image of an underwater object captured by the underwater image convergence device 100 and fuse the obtained images to extract a fusion image.

In addition, the server 200 may calculate the optical characteristics of the underwater object and the distance to the underwater object based on the extracted fusion image, and may transmit the calculated result to a terminal (not shown) or a PC (not shown). Hereinafter, a technology for converging an optical image and a sonar image will be described later.

On the other hand, in an embodiment of the present invention, after transmitting the optical image and sonar image captured by the underwater image convergence device 100 to the server 200, the server 200 converts the optical image and sonar image into an image suitable for convergence. An example of fusing the corrected images after correction will be described, but it goes without saying that the original image may be corrected in the underwater image fusion device 100 and the corrected images may be fused.

Network 300 includes wired and wireless networks, such as, for example, local area networks (LANs), wide area networks (WANs), the Internet, intranets and extranets, and mobile networks, It may be any suitable communication network including, for example, cellular, 3G, LTE, 5G, WiFi networks, ad hoc networks, and combinations thereof.

Network 300 may include connections of network elements such as hubs, bridges, routers, switches, and gateways. Network 300 may include one or more connected networks, such as a multiple network environment, including a public network such as the Internet and a private network such as a secure corporate private network. Access to network 300 may be provided via one or more wired or wireless access networks.

2 is a perspective view showing the underwater image fusion device of FIG. 1, and FIG. 3 is a block diagram showing the configuration of the underwater image fusion device according to an embodiment of the present invention.

Referring to FIG. 2 , the underwater image convergence device 100 may include a frame 150, lighting 110, an optical camera 120, a multi-beam sonar 130, and the like.

Specifically, the frame 150 may be a component that forms the overall appearance of the underwater image convergence device 100. The shape of the frame 150 may be determined in the process of forming the underwater image convergence device 100, and the shape of the frame 150 may be applied differently depending on conditions.

At least one light 110 may be installed inside the frame 150 . The lighting 110 may be a device that radiates light toward an underwater object when the underwater image convergence device 100 captures the underwater object. These lights 110 may be formed of a waterproof structure to operate in the water, it is preferable to be composed of LED lights to operate with a minimum of power in the water.

In addition, the lighting 110 may be installed on both the left and right sides of the frame 150 (refer to the directional axis of FIG. 2 ) so that underwater photography can be performed even when the illumination is low.

The optical camera 120 and the multi-beam sonar 130 may be installed at the center of the frame 150, and at least one optical camera 120 may be installed. In the embodiment of the present invention, an example in which a pair of optical cameras 120 are installed and one multi-beam sonar 130 is installed between the pair of optical cameras 120 will be described, but the optical camera 120 And the installation number and installation location of the multi-beam sonar 130 may be changed according to conditions.

The optical camera 120 is a device capable of capturing RGB color images and images by detecting light signals in the water, and the multi-beam sonar 130 expresses distance and direction information to an underwater object using sound waves as an image. It is a device that can

As described above, since the optical camera 120 detects a light signal, the signal attenuation caused by water may cause a captured image or image of an underwater object to be captured differently from a shape of the actual underwater object according to the wavelength of light. In order to minimize such an imbalance, the optical camera 120 may be implemented to photograph an underwater object from various angles. To this end, a rotation axis (122_refer to FIG. 1 ) capable of rotating the optical camera 120 may be installed in the frame 150 , and the optical camera 120 may be installed on the rotation axis 122 . At this time, the rotating shaft 122 may rotate left, right, up, and down, or rotate only in any one direction of left, right, or up and down.

The multi-beam sonar 130 may be referred to as a detector capable of recognizing an underwater object and measuring a distance to the underwater object or a direction in which the underwater object is located even when the field of view is not secured underwater.

Meanwhile, in the underwater image fusion device 100, motors capable of driving the optical camera 120 and the multi-beam sonar 130 may be respectively installed. In addition, the optical camera 120 and the multi-beam sonar 130 can simultaneously photograph an underwater object, and since the optical image and the sonar image are simultaneously captured, the optical image and the sonar image are not calculated for a time difference when generating a fusion image. can be fused.

Meanwhile, referring to FIG. 3 , the underwater image convergence device 100 includes a communication unit 110, a data acquisition unit 120, a correction unit 140, a fusion unit 150, a memory 130 and a processor 160. include

The communication unit 110 is a component that can be connected to the server 200 through communication. Specifically, the communication unit 110 may transmit the optical image and the sonar image acquired by the underwater image convergence device 100 to the server 200, and the server 200 receives the convergence image obtained by fusing the optical image and the sonar image. can receive

The data acquisition unit 120 is a component that acquires an optical image and a sonar image acquired through the optical camera 120 and the multi-beam sonar 130 . The data acquisition unit 120 may simultaneously acquire an optical image and a sonar image. This is because when generating a fusion image, the color tone, distance, and direction of the underwater object can be more accurately extracted only when the fusion image is generated based on an image of the same underwater object photographed at the same time.

The correction unit 140 is a component that corrects the acquired optical image and sonar image before generating a fusion image. The correction unit 140 may include a first correction unit 142 for correcting an optical image and a second correction unit 144 for correcting a sonar image.

Each correction unit may correct an optical image and a sonar image according to conditions, and conditions of correction may be changed according to required conditions.

In an embodiment of the present invention, the convergence video is performed in the server 200, but may be performed in the underwater video convergence device 100 otherwise. Accordingly, the fusion unit 150 may be a component for fusing optical images and sonar images in the underwater image fusion device 100 .

The memory 130 may store images captured by the underwater image convergence device 100 and may store data supporting various functions performed by the server 200 .

The processor 160 may control general operations of the underwater video convergence device 100 in addition to operations related to the above-described application program. The processor 160 may provide or process appropriate information or functions to a user by processing signals, data, information, etc. input or output through the components described above or by running an application program stored in the memory 130.

4 is a flowchart illustrating an underwater image fusion process according to an embodiment of the present invention, and FIG. 5 is a diagram showing a fusion process between an optical image and a sonar image acquired through an underwater image fusion device according to an embodiment of the present invention. 6 to 8 are diagrams illustrating a process of correcting an optical image obtained through an underwater image fusion device according to an embodiment of the present invention, and FIG. 9 is a view showing an underwater image fusion device according to an embodiment of the present invention It is a diagram showing the correction process of the acquired sonar image.

Referring to the drawing, the optical camera 120 may capture an underwater object (S110). As described above, the optical camera 120 may capture a red, green, and blue (RGB) color image by detecting a light signal of an underwater object.

When an underwater object is photographed by the optical camera 120, a signal attenuation phenomenon caused by water may cause light wavelengths to be different, resulting in RGB color imbalance (color casting). For this problem, it is possible to improve a video or/and an image captured by the optical camera 120 (S120). Image enhancement can be performed through the Image Fusion technique.

First, tonal correction of the original image may be performed to improve the image. Color correction is to set the white balance of an optical image, and measures the image quality of an optical image taken underwater so that an optical image with clear color tone can be derived.

As shown in FIGS. 6 and 7 , the color tone of the original image captured by the optical camera 120 may be corrected (white balancing). Tonal correction can be calculated with the following formula.

,

,

는 각각 원 영상의 붉은 색, 파란 색과 녹색 채널의 평균값을 의미할 수 있다. Here, WB _r and WB _b mean the values of the red and blue channels of the image after tonal correction, and I _r , I _b , and I _g mean the values of the red, blue and green channels of the original image, respectively. can do. also,

,

,

may mean average values of red, blue, and green channels of the original image, respectively.

That is, referring to [Equation 1] and FIG. 7, the color tone can be corrected so that the values of the color and blue channels are as close as possible to the average values of green. Also, α may mean a weight of the correction value.

By correcting the color tone of the original data, the contrast of the original image can be improved (contrast limited adaptive histogram equalization (CLAHE)). Specifically, an optical video image may be optimized through image histogram equalization.

Thereafter, the optimized optical image may remove noise included in the optical image through unsharp masking. At this time, the process of unsharp masking to remove noise can be expressed by the following equation.

In [Equation 2], WB means an image after color correction, and G means a Gaussian filter. After color correction, if the low-frequency component is made stronger with a Gaussian filter, and then subtracted from the image after color correction, only the high-frequency component can be left, and sharpness can be enhanced by adding it to the image after color correction.

The image from which the noise has been removed is extracted using the Image Fusion technique with improved image quality. At this time, the Image Fusion technique may be implemented in the following equation.

Here, R _k means the image after CLAHE (k=1) and unsharp masking (k=2). ω _k means a weight that determines how much to fuse each pixel value of the image after CLAHE and unsharp masking. ω _k can be derived through the following equation.

Here, WLap is the Laplacian Contrast weight, which is the weight of each pixel determined by applying the Lapalcian filter to the luminance channel of the image, and Wsat is the Saturation weight, which is determined by the average difference between the luminance (Y) of the RGB color model and the YCbCr color model, and Wsal is the It can be determined by extracting the main features appearing in the CIELAB color model as a saliency weight.

Optical images with improved image quality can evaluate the degree of improvement. As shown in FIG. 5 , the degree of improvement can be referred to as confirming the degree of reduction of RGB color imbalance of the corrected optical image. This improvement evaluation can be extracted through a weighted sum of the chroma, sharpness, and contrast of the corrected optical image. Specifically, the degree of improvement can be derived through the following equation.

In this case, UICM may mean colorfulness, UISM may mean sharpness, and UIConM may mean contrast. In addition, each weight (a₁, a₂, a₃) can be applied differently depending on conditions.

As shown in FIG. 7 , it can be seen that, in the case of the optical image, after the image quality is improved compared to the original image, the saturation, sharpness, and contrast of the optical image are increased by the evaluation of the degree of improvement.

The optical camera 120 captures the underwater object, transmits sound waves from the multi-beam sonar 130 to the underwater object, and measures the distance and direction of the underwater object through a process in which the transmitted sound waves return to the underwater object. can do.

Since the distance and direction of the underwater object measured by the multi-beam sonar 130 are collected together with noise in the water, noise must be removed from the acquired sonar image (S130).

To this end, a region of interest (I) may be set in the sonar image. The region of interest (I) may be any one of an area where an underwater object can be located or an area in which image quality is to be improved according to a request of an operator.

When the region of interest (I) is set, a contrast to noise ratio (CNR) within the region of interest may be measured. The contrast-to-noise ratio can be implemented by the following equation.

Here, SA-SB can be referred to as the difference in signal strength of RoI ((Region of interest) where an underwater object is detected, and σA-σB can be referred to as the difference in standard deviation of noise being measured.

By measuring this contrast-to-noise ratio, the original sonar image can be extracted as a sonar image with improved quality, as shown in FIG. 6 .

Meanwhile, when improving a sonar image, a median filter and gamma correction may be applied to the sonar image. A median filter can be referred to as a filter that selects a medium-sized value after arranging the values of a given mask area in order of size.

For example, after arranging pixel values according to pixels in order of size, a middle pixel value is selected so that a pixel with severe noise can be extracted as a high-quality image.

Also, gamma correction may mean correcting a gradation (gradation) characteristic of a video/image. That is, gamma correction means that a light intensity signal can be nonlinearly transformed using a nonlinear transfer function in an image or the like.

Meanwhile, gamma correction according to an embodiment of the present invention may be implemented by the following equation.

In Equation 7, I(x,y) may mean a pixel value of an elemental sonar image at the position of coordinates (x,y), and J(x,y) may mean a pixel value of a sonar image after gamma correction. Also, γ may mean a correction coefficient.

By applying such gamma correction and media filter to the sonar image, noise included in the sonar image can be removed, and the shadow of the sonar image can be improved more brightly (refer to area 1 in FIG. 9).

In this way, when the optical image and the sonar image are collected and corrected through the underwater image convergence device 100, a fusion image in which the corrected optical image and the corrected sonar image are fused can be generated (S140).

The fusion image is an image in which an optical image obtained and calibrated at the same time is fused with a sonar image. In order to generate a fusion image, as shown in FIG. 5, an arbitrary first point P1 and a second point (P₂) can be set (refer to (a) and (b) of FIG. 5).

The first point P1 may be referred to as a position designating the corner of the underwater object in the corrected optical image, and the second point P2 may be referred to as a position designating the distance of the underwater object in the corrected sonar image.

After setting the first point (P1) and the second point (P2) in the optical image and the sonar image thus corrected, the first point (P1) and the second point (P2) may be matched. Referring to FIG. 5 , a first point designating a corner of a corrected optical image and a second point designating a corner of a corrected sonar image may be matched (A) (refer to (c) of FIG. 5 ).

When the first point and the second point are matched, a fusion image in which the optical image and the sonar image are combined may be generated (refer to (d) of FIG. 5 ).

As described above, the first point P1 is a position indicating the corner coordinates of the corrected optical image, and the second point P2 is a position indicating the corner coordinates of the corrected sonar image, so they contain different information. Therefore, when matching two edges, it should be converted to include color tone information and distance information.

To this end, when extracting the first point P1, the first point P1 may be extracted as image coordinates, and the extracted image coordinates may be converted into a coordinate matrix.

When extracting the coordinate matrix through the coordinates of the first point P1, it may be extracted by sequentially multiplying a transformation matrix capable of extracting the coordinate matrix by a preset reference matrix.

Here, the transformation matrix means a matrix capable of converting a point extracted from each image coordinate into a corresponding point of another image coordinate, and the reference coordinate refers to a coordinate that serves as a standard when converting between image coordinates (eg, Wx, Wy, Wz).

That is, in order to convert points extracted from one image into points of other image coordinates, coordinate conversion between coordinates may be performed by multiplying a conversion matrix by image coordinates to convert points in reference coordinates, and then multiplying them by another image coordinate conversion matrix.

When converting to a coordinate matrix, the transformation matrix may be a matrix obtained by sequentially multiplying a rotation matrix and a transition matrix by a 2X1 array matrix and an image parameter matrix. Specifically, the transformation matrix may be implemented through the following equation.

Here, : transformation matrix, : rotation matrix, : transition matrix, and : image parameter matrix.

Here, the rotation matrix is a 3x3 matrix, and may be a matrix that rotates the x, y, and z axes of image coordinates. In addition, the transition matrix is expressed as 3x1 and may be a matrix representing the degree of movement of the x, y, and z axes of image coordinates, and the image parameter matrix may be a matrix expressed by the pixel size of the image itself and the center point parameter values of the optical image. .

In this way, a 3D distance represented by the optical image may be extracted through a transformation matrix inferred from the optical image obtained through the optical camera 120 extracted through the matrix. In addition, the distance and direction of the underwater object may be extracted through the sonar image acquired through the multi-beam sonar 130 . By matching the distance and distance of the underwater object extracted through the extracted optical image with the distance to the underwater object extracted through the sonar image, the actual distance of the underwater object and the color tone of the underwater object can be extracted ((FIG. 5). d) Note).

As described above, it is possible to photograph an underwater object in the water through an RGB optical camera and simultaneously photograph the underwater object using an underwater sonar. The captured optical image and the sonar image can be improved in color correction and contrast, and the two improved images can be fused.

The fusion image includes color tones and may include distance information between the fusion device and the underwater object. That is, the fusion image can intuitively recognize an underwater object and measure the distance to the underwater object through sound waves in the water. That is, intuitive information of an underwater object and distance information to the underwater object may be simultaneously measured and acquired.

The description of the embodiments of the present invention described above is for illustrative purposes, and those skilled in the art can easily modify them into other specific forms without changing the technical spirit or essential features of the present invention. you will understand that Therefore, the embodiments described above should be understood as illustrative in all respects and not limiting. For example, each component described as a single type may be implemented in a distributed manner, and similarly, components described as distributed may be implemented in a combined form.

The scope of the present invention is indicated by the following claims rather than the above detailed description, and all changes or modifications derived from the meaning and scope of the claims and equivalent concepts thereof should be construed as being included in the scope of the present invention.

Claims (13)

A method in which at least a part of each step is performed by a processor and fuses images using sonar images and optical images,
collecting at least one optical image and sonar image of an underwater object for monitoring underwater information;
generating a corrected optical image by correcting color tone of the collected optical image;
generating a corrected sonar image by removing noise included in the sonar image; and
Generating a fusion image in which the corrected optical image and the corrected sonar image are fused,
Underwater image fusion method. According to claim 1,
After generating the corrected optical image,
Evaluating the degree of improvement of the corrected optical image through a weighted sum of saturation, sharpness, and contrast of the corrected optical image,
Underwater image fusion method. According to claim 1,
Generating the corrected optical image,
setting a white balance of the optical image;
optimizing an image of the optical image through image histogram equalization; and
Removing noise included in the optimized optical image through unsharp masking,
Underwater image fusion method. According to claim 1,
The step of generating the calibrated sonar image,
setting a region of interest in the sonar image; and
Measuring a contrast to noise ratio (CNR) in the region of interest,
Underwater image fusion method. According to claim 4,
The step of generating the calibrated sonar image,
Applying the sonar image to a median filter and gamma correction,
The gamma correction,

defined by
(Here, I(x,y) means the pixel value of the original SONA image at the position of coordinates (x,y), and J(x,y) means the pixel value of the SONA image after gamma correction. γ is correction factor),
Underwater image fusion method.
According to claim 1,
The generating step is
setting arbitrary first and second points in the corrected optical image and the corrected sonar image;
Including matching the first point and the second point,
Underwater image fusion method.
According to claim 6,
In the setting step,
extracting image coordinates from the first point;
converting the image coordinates into a coordinate matrix; and
Including the step of extracting the coordinate matrix by sequentially multiplying the transformation matrix for extracting the coordinate matrix by a preset reference matrix,
Underwater image fusion method.
According to claim 7,
The pixel size of the optical image and the center point of the optical image Further comprising extracting the transformation matrix by sequentially multiplying an image parameter matrix, which is a matrix represented by parameter values,
Underwater image fusion method.
According to claim 6,
setting the first point designating a corner of the underwater object in the corrected optical image;
setting the second point designating a distance of the underwater object in the calibrated sonar image; and
Measuring a shape of the underwater object and a distance to the underwater object by matching the first point and the second point,
Underwater image fusion method.
An apparatus for converging images using sonar images and optical images,
A memory electrically connected to the processor and storing at least one code executed by the processor;
When the memory is executed by the processor, the processor
After collecting at least one optical image of an underwater object for monitoring underwater information and a sonar image, color tone of the optical image is corrected, and an interface between the underwater object included in the sonar image and a peripheral portion of the underwater object And if the contrast is improved, storing codes for generating a fusion image in which the optical image and the sonar image are fused,
Underwater image fusion device.
According to claim 10,
The memory is the processor,
Further storing codes that cause the improvement of the corrected optical image to be evaluated through a weighted sum of saturation, sharpness, and contrast of the corrected optical image.
Underwater image fusion device.
According to claim 10,
The memory is the processor,
After setting the white balance of the optical image, optimizing the image of the optical image through image histogram equalization, and removing noise included in the optical image through unsharp masking of the optimized optical image to store more codes that cause
Underwater image fusion device.
According to claim 10,
The memory is the processor,
Further storing codes that cause setting a region of interest in the sonar image and measuring a contrast to noise ratio (CNR) in the region of interest,
Underwater image fusion device.

Images