KR102777985B1 - Apparatus and method for generating image - Google Patents

Abstract

An image generating device includes a camera capable of changing direction, a memory storing one or more commands, and a processor executing one or more commands. The processor controls the camera in a first direction to obtain a first image, controls the camera in a second direction different from the first direction to obtain a second image partially overlapping with the first image, analyzes a distortion area of the first image based on the first image and the second image, and generates a third image in which image correction corresponding to the analyzed distortion area of the first image is performed.

Description

{APPARATUS AND METHOD FOR GENERATING IMAGE}

It relates to an image generating device and method.

In the case of taking pictures while facing a light source, for example, when taking pictures using a camera in environments such as backlighting or night shooting, the captured image may contain image areas that are different from the actual appearance. When light from a light source enters the camera and is scattered or reflected inside the camera, an image containing image areas that are different from the actual appearance is created. In the case of taking pictures while facing a light source, the occurrence and occurrence pattern of image distortion may change depending on whether the shooting direction changes or the relative positions of the light source and the camera lens change.

The present invention provides an image generating device and an image generating method capable of correcting image distortion that occurs when shooting with a light source using a camera capable of changing direction.

An image generating device according to a first aspect includes a camera capable of changing direction, a memory storing one or more commands, and a processor executing the one or more commands, wherein the processor controls the camera in a first direction to obtain a first image, controls the camera in a second direction different from the first direction to obtain a second image partially overlapping with the first image, analyzes a distortion area of the first image based on the first image and the second image, and generates a third image by performing image correction corresponding to the analyzed distortion area on the first image.

An image generation method according to a second aspect includes a step of controlling a camera capable of changing direction in a first direction to acquire a first image, a step of controlling the camera in a second direction different from the first direction to acquire a second image partially overlapping with the first image, a step of analyzing a distortion area of the first image based on the first image and the second image, and a step of generating a third image in which image correction corresponding to the analyzed distortion area is performed on the first image.

A computer-readable recording medium according to a third aspect records a computer-executable program including commands for controlling a directional camera in a first direction to acquire a first image, commands for controlling the camera in a second direction different from the first direction to acquire a second image partially overlapping with the first image, commands for analyzing a distortion area of the first image based on the first image and the second image, and commands for generating a third image in which image correction corresponding to the analyzed distortion area is performed on the first image.

Figure 1 is a drawing showing an image generating device having a camera capable of changing direction.
Figure 2 is a drawing for explaining the configuration of an image generating device.
Figure 3 is a diagram for explaining the operation of the memory, processor, and camera of the image generation device.
Figure 4 is a drawing to explain a situation in which a lens flare occurs when light is incident from a light source and the change in the lens flare when the direction of the camera is changed by rotating the camera.
Figure 5 is a drawing for explaining the relationship between a first direction for acquiring a first image and a second direction for acquiring a second image.
Figure 6 is a drawing for explaining a process of analyzing a distortion area of a first image based on the first image and the second image.
Figure 7 is a drawing for explaining a process of performing image correction corresponding to a distortion area for the first image.
Figure 8 is a flowchart explaining the image generation method.
Figure 9 is a drawing for explaining an example of an image generating device.
Figure 10 is a drawing for explaining another example of an image generating device.

Hereinafter, embodiments of the present disclosure will be described in detail with reference to the drawings so that those skilled in the art can easily implement the present disclosure. The present disclosure may be implemented in various different forms and is not limited to the embodiments described herein.

When a part of the specification is said to "include" a component, this does not mean that other components are excluded, but rather that other components may be included, unless otherwise specifically stated. In addition, terms such as "part", "module", etc., used in the specification mean a unit that processes at least one function or operation, which may be implemented by hardware or software, or a combination of hardware and software.

Additionally, terms including ordinal numbers, such as "first" or "second," used herein may be used to describe various components, but the components should not be limited by the terms. The terms are used only for the purpose of distinguishing one component from another.

The present embodiments relate to an image generating device and method, and a detailed description of matters that are widely known to those skilled in the art to which the following embodiments pertain is omitted.

In the present disclosure, the term "image generating device" is a general term for an electronic device capable of generating an image, and may be an electronic device equipped with a camera function. For example, the image generating device may be a smart phone, an augmented reality device, a digital camera, etc.

"Distortion area" refers to a virtual area generated in an image acquired through a camera. A distortion phenomenon that occurs in an image when unintended external light is incident on the camera lens and is scattered or reflected is called "lens flare." Lens flare can be classified into veiling flare and ghosting flare. "Veiling flare" is a distortion phenomenon that creates an image with low contrast and unclearness by generating haze when external light is diffusely reflected or scattered. "Ghosting flare" is a distortion phenomenon that creates an image that includes virtual images such as polygonal shapes or light streaks when external light is clearly reflected.

FIG. 1 is a drawing showing an image generating device (1000) having a camera (1300) capable of changing direction.

Referring to FIG. 1, the image generating device (1000) may be equipped with a camera (1300) capable of changing direction. The fact that the camera (1300) can change direction means that the shooting direction can be changed. The camera (1300) is designed to have a structure capable of changing direction and can be mounted in a predetermined part of the image generating device (1000). The camera (1300) may be mounted in a predetermined part of the image generating device (1000) depending on the type and shape of the image generating device (1000). As illustrated in FIG. 1, when the image generating device (1000) is an augmented reality device in the form of glasses, the camera (1300) may be mounted in a built-in form in a bridge frame connecting the left-eye lens unit and the right-eye lens unit, or may be installed in the front part of the temple of the glasses, but is not limited thereto.

The image generating device (1000) can change the direction of the camera (1300) to various angles corresponding to the shooting direction. For example, the image generating device (1000) can control the camera (1300) in a first direction to obtain a first image, and control the camera (1300) in a second direction different from the first direction to obtain a second image that partially overlaps the first image. If the first image obtained in the first direction includes a light source, the second direction can be any direction that allows the light source included in the first image to be excluded from the second image. For example, if the first direction is a direction looking at a predetermined light source, the second direction can be a direction looking at a point that is an arbitrary distance away from the predetermined light source. The relationship between the first direction and the second direction is described in detail with reference to FIGS. 4 and 5.

For example, as illustrated in FIG. 1, when a light source is included in the left portion area of the first image, image distortion may occur in the right portion area of the first image due to light rays derived therefrom. In this case, the image generating device (1000) may obtain a second image that partially overlaps the right portion area of the first image in a second direction in which the direction of the camera (1300) is changed to the right from the first direction in which the first image was obtained. The first image and the second image may be used to generate a third image in which the distortion area of the first image is corrected. Hereinafter, a method in which the image generating device (1000) generates an image by correcting image distortion will be described in detail.

Figure 2 is a drawing for explaining the configuration of an image generating device (1000).

Referring to FIG. 2, the image generation device (1000) may include a memory (1100), a processor (1200), and a camera (1300). Those skilled in the art related to the present embodiment will appreciate that other general components may be included in addition to the components illustrated in FIG. 2.

The memory (1100) can store instructions executable by the processor (1200). The memory (1100) can store a program composed of instructions. The memory (1100) can include, for example, at least one type of hardware device among a RAM (Random Access Memory), a SRAM (Static Random Access Memory), a ROM (Read-Only Memory), a flash memory, an EEPROM (Electrically Erasable Programmable Read-Only Memory), a PROM (Programmable Read-Only Memory), a magnetic memory, a magnetic disk, and an optical disk.

The memory (1100) can store at least one software module including commands. Each software module is executed by the processor (1200) to cause the image generation device (1000) to perform a predetermined operation or function. For example, as shown in FIG. 2, the memory (1100) can store a shooting mode module, a distortion analysis module, and an image correction module, but is not limited thereto, and may store some of these or further include other software modules.

The processor (1200) can control operations or functions performed by the image generation device (1000) by executing commands or programmed software modules stored in the memory (1100). The processor (1200) can be composed of hardware components that perform arithmetic, logic, and input/output operations and signal processing.

The processor (1200) may include at least one processing module. For example, the processor may be configured with at least one hardware among a Central Processing Unit, a microprocessor, a Graphic Processing Unit, Application Specific Integrated Circuits (ASICs), Digital Signal Processors (DSPs), Digital Signal Processing Devices (DSPDs), Programmable Logic Devices (PLDs), and Field Programmable Gate Arrays (FPGAs), but is not limited thereto.

The camera (1300) can change direction. The camera (1300) has a structure that can change direction in the image generating device (1000), and can be a camera (1300) designed to face a direction at a predetermined angle as well as up and down or left and right. For example, the camera (1300) can be a so-called pan-tilt camera that can perform a pan operation that rotates the camera (1300) left and right while its position is fixed to capture an object, or a tilt operation that rotates the camera (1300) up and down while its position is fixed to capture an object. The camera (1300) can control the shooting direction at a predetermined angle by rotating about a plurality of axes. For example, the camera (1300) can rotate about a first axis corresponding to a horizontal direction, or can rotate about a second axis corresponding to a vertical direction. Alternatively, the camera (1300) may rotate about a first axis corresponding to a diagonal direction extending from the upper left to the lower right, or about a second axis corresponding to a diagonal direction extending from the upper right to the lower left. The camera (1300) may change direction to a preset angle depending on the actual shooting environment, the type of shooting function, and the environment setting information for shooting.

According to the above configuration, the processor (1200) can generate an image by executing at least one of a shooting mode module, a distortion analysis module, and an image correction module stored in the memory (1100).

The processor (1200) can execute commands stored in the memory (1100) to control the camera (1300) in a first direction to obtain a first image, and control the camera (1300) in a second direction different from the first direction to obtain a second image that partially overlaps the first image. The processor (1200) can change the direction of the camera (1300) in a shooting mode.

The processor (1200) may execute commands stored in the memory (1100) to analyze a distortion area of the first image based on the first image and the second image. In order to analyze the distortion area of the first image, the processor (1200) may warp the second image with respect to the first image and determine a matching area between the first image and the warped second image. The processor (1200) may segment a distortion candidate area in the first image based on a difference in pixel values between the matching areas. The processor (1200) may determine the distortion area based on whether the segmented distortion candidate area satisfies a condition for determining a lens flare. For example, the processor (1200) may determine the segmented distortion candidate area as a distortion area based on a degree of brightness change in the segmented distortion candidate area or a degree of continuity of pixel values at a boundary between the segmented distortion candidate area and a surrounding area. As another example, the process may determine the segmented distortion candidate area as a distortion area based on the similarity between the segmented distortion candidate area and a template image of a lens flare, or the similarity between the segmented distortion candidate area and a matching candidate area that is an area having a light amount greater than a predetermined standard among the first image and the second image.

The processor (1200) may execute commands stored in the memory (1100) to generate a third image on which image correction is performed corresponding to the analyzed distortion area of the first image. The processor (1200) may extract an area corresponding to the analyzed distortion area of the first image from the second image, and perform image correction by replacing the analyzed distortion area of the first image with the area extracted from the second image. To this end, the processor (1200) may adjust an exposure value of the second image according to an exposure value of the first image, and then extract an area corresponding to the analyzed distortion area of the first image from the second image. The processor (1200) may perform image correction by replacing the analyzed distortion area of the first image with the area extracted from the second image, and then performing blending processing on the boundary of the replaced area.

FIG. 3 is a drawing for explaining the operation of the memory (1100), processor (1200), and camera (1300) of the image generation device (1000).

The overlapping description of the memory (1100) and processor (1200) described in Fig. 2 is omitted below. The camera (1300) may include a camera driving module (1310) and an image processing module (1330).

The camera driving module (1310) may include a lens module including lenses, and an AF (Auto Focus) actuator. The lens module has a structure in which a plurality of lenses are arranged within a barrel, and may allow light incident from the outside to pass through the arranged lenses. The AF actuator may move the lenses to an optimal focus position to obtain a clear image. For example, the AF actuator may be a VCM (Voice Coil Motor) type. The VCM type determines the positions of the lenses and focuses by moving the lens module back and forth using a coil and an electromagnet.

The image processing module (1330) may include an image sensor and an image signal processor. The image sensor may be a CMOS (Complementary Metal Oxide Semiconductor) sensor or a CCD (Charge-Coupled Device) sensor, and may convert an optical signal into an electrical signal. The image signal processor may convert an electrical signal converted by the image sensor into an image signal.

According to an example of the present disclosure, the image sensor of the image processing module (1330) may be connected to the lens module of the camera driving module (1310). The image sensor may be connected to the lower portion of the lens module. An axis passing vertically through the center of the image sensor may be the same axis as an optical axis passing through the centers of the lenses. Since the image sensor is connected to the lens module, when the lens module moves by a predetermined angle based on the center point on the optical axis of the lenses, the image sensor may also move by a predetermined angle.

For example, if an AF actuator in the form of wrapping a lens module has two pairs of bearings, such as thrust bearings, at a portion that physically contacts the lens module, the camera driving module (1310) may further include a rotation module that virtually connects each pair of facing bearings to operate the image sensor of the image processing module (1330) and the lens module of the camera driving module (1310) connected thereto together on two axes, thereby controlling the direction of the camera (1300) to be at a predetermined angle. As another example, in order to perform a pan operation or a tilt operation of the camera (1300), the camera driving module (1310) may further include a rotation module that rotates the camera (1300) about a first axis corresponding to the vertical direction to perform a pan operation, or about a second axis corresponding to the horizontal direction to perform a tilt operation, thereby controlling the direction of the camera (1300) to be at a predetermined angle.

The processor (1200) can load and execute commands or software modules stored in the memory (1100). As illustrated in FIG. 3, the processor (1200) has at least one processing hardware module to execute a shooting mode module, a distortion analysis module, and an image correction module. For example, the processor (1200) has a CPU and a GPU dedicated to image processing to execute the shooting mode module in the CPU and the distortion analysis module and the image correction module in the GPU. As another example, the processor (1200) may be in the form of a single integrated processor that executes all of the shooting mode module, the distortion analysis module, and the image correction module, or may be in the form of individual processors for executing each module, respectively.

The processor (1200) may execute a shooting mode module to transmit a shooting control signal to the image processing module (1330). The image processing module (1330) may transmit a first image acquired according to the shooting control signal to the memory (1100). The memory (1100) may serve as a buffer that stores the first image acquired by the image processing module (1330). The first image stored in the memory (1100) may be used to correct the image by analyzing a distortion area.

The processor (1200) may execute a shooting mode module to transmit a driving control signal to the camera driving module (1310) to change the direction of the camera (1300) from a first direction in which a first image is acquired to a second direction different from the first direction. After the direction of the camera (1300) is changed to a predetermined angle by the camera driving module (1310), the processor (1200) may transmit a shooting control signal to the image processing module (1330). The image processing module (1330) may store the acquired second image in the memory (1100) according to the shooting control signal. The second image may be used to correct the image by analyzing a distortion area.

The processor (1200) can execute a distortion analysis module to load the first image and the second image stored in the memory (1100), and analyze the distortion area of the first image based on the first image and the second image. The processor (1200) can analyze the distortion area of the first image by checking and comparing overlapping portions in the first image and the second image. The processor (1200) can determine the portion to be corrected in the first image by analyzing the distortion area of the first image.

The processor (1200) may execute an image correction module to generate a third image by performing image correction corresponding to a distortion area of the first image on the first image. The processor (1200) may generate the third image based on the first image, the second image, and the distortion area of the first image. The processor (1200) may generate the third image by performing image correction corresponding to the distortion area of the first image on the first image using the second image. The third image may be an image in which the distortion area of the first image is corrected. The processor (1200) may store the generated third image in the memory (1100). Considering the limited storage space of the memory (1100), the first image and the second image may be deleted from the memory (1100) after the third image is generated, but may be stored in the memory (1100) depending on a setting option.

Meanwhile, the image generating device (1000) can acquire and utilize a plurality of images, including the second image, to generate an image that is more natural and has no distortion area throughout the entire image.

For example, the processor (1200) may adjust the shooting parameters when acquiring the second image in the second direction to acquire a fourth image, and generate a third image by performing image correction using the fourth image. That is, the processor (1200) may additionally acquire a fourth image by changing only the shooting parameters in the same shooting direction as the shooting direction in which the second image was acquired, and may use the fourth image when performing image correction corresponding to the distortion area of the first image. Accordingly, the processor (1200) may analyze the distortion area of the first image based on the first image and the second image, and generate the third image based on the analyzed distortion area of the first image, the first image, and the fourth image. When analyzing the distortion area of the first image, the second image captured in a different direction with the same shooting parameters as the first image is used, and when correcting the image of the first image, the fourth image captured in the same shooting direction with different shooting parameters from the second image used when analyzing the distortion area is used. When analyzing the distorted area of an image, it is advantageous to use an image in which all conditions except the shooting direction are the same, and when correcting an image, it may be advantageous to use an image captured with shooting parameters different from those of the first image containing the distorted area.

For another example, the processor (1200) can control the camera (1300) in a plurality of directions including the second direction to acquire a plurality of images including the second image, and analyze a distortion area of the first image based on the first image and the plurality of images. The processor (1200) can acquire a plurality of images while changing the direction of the camera (1300) along a predetermined trajectory centered on the first direction. By utilizing the plurality of images acquired in this manner, the distortion area can be analyzed for the entire area of the first image. The processor (1200) can utilize at least one of the plurality of images acquired by controlling the camera (1300) in a plurality of directions including the second direction to perform image correction corresponding to the analyzed distortion area of the first image.

When a photo is taken with an image correction function performed using an image generating device (1000) of this type, the user can access the third image stored in the memory (1100) after the photo is taken. In other words, the user can check the third image in which the distortion area of the first image is corrected instead of the first image taken directly by the user.

FIG. 4 is a drawing for explaining a situation in which a light beam from a light source is incident on a camera (1300) and a lens flare occurs, and a change in the lens flare when the direction of the camera (1300) is changed by rotating the camera (1300).

When an unintended bright light ray from a light source, rather than a regular light ray, is incident on the lens module of the camera (1300), the light may be scattered or reflected by the lenses constituting the lens module. Accordingly, as illustrated in FIG. 4, a lens flare occurs in which the scattered or reflected light is focused on an arbitrary point of the image plane of the image sensor. The point at which the lens flare occurs becomes a distorted area in the image generated by the image generating device (1000).

Referring to FIG. 4, when the camera (1300) is rotated by a predetermined angle about a center of rotation, which is a point on the optical axis, and the direction of the camera (1300) is changed, it can be seen that the relative position of the light source is changed, and the point or direction at which an unintended bright light from the light source is incident on the lens module is changed. As a result, it can be seen that the situation in which the lens flare occurred before the direction of the camera (1300) was changed is different from the situation in which the lens flare occurred before the direction was changed, and that the lens flare that occurred before the direction was changed does not occur, or the lens flare that is focused in a different pattern at a different point on the image plane of the image sensor occurs.

When the direction of the camera (1300) is changed due to the rotation of the camera (1300), a distortion area due to a lens flare generated in an image before the direction change may not be generated in an image after the direction change, or a different distortion area with a different pattern may be generated in a different area of the image. In other words, when the direction of the camera (1300) is changed due to the rotation of the camera (1300), a distortion area due to a lens flare generated in a first image acquired in a first direction may not be generated in a second image acquired in a second direction, or a different distortion area due to a lens flare with a different pattern may be generated in a different area that does not correspond to the distortion area of the first image. Therefore, when the direction of the camera (1300) is changed due to the rotation of the camera (1300), the camera (1300) can acquire a second image in which a distortion area due to a lens flare generated in the first image is not generated, and the processor (1200) can perform image correction for the distortion area of the first image using the second image.

Figure 5 is a drawing for explaining the relationship between a first direction for acquiring a first image and a second direction for acquiring a second image.

As described above, if there is a change in direction of the camera (1300) due to the rotation of the camera (1300), the same lens flare that occurred in the first image acquired in the first direction before the change in direction does not occur in the second image acquired in the second direction after the change in direction. Accordingly, if there are two images captured in different directions by the camera (1300), the area of distortion due to the lens flare that occurred in one image can be analyzed using the other image, and the corresponding area of distortion can be corrected.

FIG. 5 illustrates a second direction for acquiring a second image when a light source is included in a first image acquired in a first direction. As illustrated in FIG. 5, when a light source is included in a first image acquired in the first direction, the second direction may be any direction that allows the light source included in the first image to be excluded from the second image. However, the first image and the second image must have a relationship in which they partially overlap each other. Preferably, the second direction may be at least one direction that satisfies both a direction in which the light source in the first direction is not directly captured and a direction in which the first image and the second image are captured so that there is a region in which they partially overlap. FIG. 5 illustrates, as an example of a plurality of images (hereinafter, “second images”) including any second image corresponding to a plurality of directions (hereinafter, “second directions”) including any second direction, the second image is acquired from each of the upper, lower, left, and right sides of the first image corresponding to the first direction, but is not limited thereto. The image generating device (1000) can obtain second images corresponding to second directions, analyze a distortion area of the first image based on the first image and the second images, and correct the distortion area.

Since the second images are acquired while changing the second directions along a predetermined trajectory centered on the first direction, it can be seen that when the centers of the second images corresponding to the second directions shown in Fig. 5 are connected, a diamond-shaped trajectory centered on the first direction is formed. The shape of such a predetermined trajectory may vary depending on the interval at which the second images are captured and, consequently, the number of second images.

The processor (1200) of the image generating device (1000) may transmit a driving control signal to the camera driving module (1310) based on control values corresponding to preset angles to control the camera (1300) in a second direction different from the first direction. The control values corresponding to the preset angles may be obtained based on experimental statistics, based on the viewing angle of the camera (1300) and the area ratio of the area where the first image and the second image overlap with respect to the entire area of the first image. The memory (1100) may store the control values corresponding to the preset angles, and the processor (1200) may access the stored control values. Such control values may be included in the shooting mode module.

The processor (1200) of the image generation device (1000) may transmit a driving control signal to the camera driving module (1310) based on control values corresponding to angles calculated using a first image acquired in the first direction to control the camera (1300) in a second direction different from the first direction. The control values corresponding to the angles calculated using the first image may be obtained through calculation based on the field of view of the camera (1300) and the minimum angle that must be rotated to prevent a light source included in the first image from being directly photographed. The memory (1100) may store an operation code for obtaining the control values corresponding to the angles calculated using the first image acquired in the first direction, and the processor (1200) may access the stored operation code. Such an operation code may be included in the shooting mode module.

Meanwhile, the image generating device (1000) can further acquire a fourth image by adjusting the shooting parameters when acquiring the second image in the second direction. The fourth image acquired in this manner can be used to generate a third image on which image correction has been performed. For example, the image generating device (1000) can further acquire a fourth image having different shooting parameters from the second image in any second direction. In this manner, each of the fourth images acquired in the second directions can be images having the same shooting direction as each of the corresponding second images and only different shooting parameters. Since the second images are images captured in a different direction with the same shooting parameters as the first image, they are used to analyze a distortion area of the first image, and after the distortion area of the first image is analyzed, the fourth images having different shooting parameters from the second images used to analyze the distortion area of the first image can be used to perform image correction corresponding to the distortion area of the first image.

Figure 6 is a drawing for explaining a process of analyzing a distortion area of a first image based on the first image and the second image.

Referring to FIG. 6, the image generating device (1000) can analyze a distortion area of a first image based on a first image acquired in a first direction and a second image acquired in a second direction, the second image partially overlapping with the first image.

The image generating device (1000) can warp a second image with respect to a first image. Warping an image means causing a geometric deformation of an image by corresponding a pixel at a position (x, y) in one image to a pixel at a position (x', y') in another image. In order to convert one image into another image, a homography matrix representing a relationship between corresponding pixels of the two images can be used.

A warping function for obtaining a warped second image for matching with the first image can be obtained by obtaining a homography matrix by securing extrinsic parameters between the first image and the second image based on a rotation angle known in advance, or by directly calculating a homography matrix based on feature points between the first image and the second image. After warping the second image with respect to the first image according to the warping function, the warped second image and the first image can be matched.

The image generating device (1000) can determine a matching area between the first image and the warped second image. The matching area between the first image and the warped second image can be determined by calculating the relative position of the warped second image based on a rotation angle known in advance, or by determining the matching area according to the position of the point where the feature points between the first image and the warped second image match.

The image generating device (1000) can segment a distortion candidate area in the first image based on a difference in pixel values between matching areas. If the matching area of the first image includes a distortion area due to a lens flare, when the difference in pixel values between the corresponding matching areas in the first image and the warped second image is obtained, the difference in pixel values of pixels corresponding to the distortion area in the first image may be greater than or equal to a predetermined level from the difference in pixel values of surrounding pixels. In this way, pixels in the matching area in which the difference in pixel values between corresponding pixels is greater than or equal to a predetermined level from the difference in pixel values between corresponding surrounding pixels can be determined as a distortion candidate area and segmented.

When the exposure values of the first image and the warped second image are different, the exposure value of the first image and the exposure value of the warped second image are matched, and then the difference in pixel values between the corresponding matching areas is calculated, and pixels whose pixel value difference is greater than a predetermined level can be determined as distortion candidate areas. The exposure values of the two images can be matched based on the values of the exposure parameters in the environment in which the first image was photographed and the values of the exposure parameters in the environment in which the second image was photographed, which are recorded in the image generating device (1000). Alternatively, when the exposure values of the first image and the warped second image are different, the difference in pixel values between the corresponding matching areas in the first image and the warped second image is calculated, and pixels whose pixel value difference is greater than a predetermined range from the average value of the differences in pixel values of all pixels in the matching areas can be determined as distortion candidate areas. In this way, the areas determined as distortion candidate areas can be divided from the first image.

The image generating device (1000) can determine whether a segmented distortion candidate area is a distortion area based on whether the segmented distortion candidate area from the first image satisfies a condition for being determined as a lens flare.

For example, the image generating device (1000) can determine the divided distortion candidate region as a distortion region based on the degree of brightness change in the divided distortion candidate region. If the degree of brightness change in the divided distortion candidate region maintains a constant value or the brightness in the divided distortion candidate region shows a gradation, the image generating device (1000) can determine the divided distortion candidate region as a distortion region where a veiling flare has occurred.

For another example, the image generating device (1000) may determine the divided distortion candidate region as a distortion region based on the continuity of pixel values at the boundary between the divided distortion candidate region and the surrounding region. The image generating device (1000) may confirm the continuity of pixel values of the divided distortion candidate region and the surrounding region adjacent thereto according to at least one method such as color comparison of pixel values, color histogram comparison, edge detection, etc. If there is no continuity of pixel values of the divided distortion candidate region and the surrounding region adjacent thereto in the first image, the image generating device (1000) may determine the divided distortion candidate region as a distortion region in which ghosting flare has occurred. In addition, for the region of the second image corresponding to the divided distortion candidate region in the first image, the continuity of pixel values of the surrounding region adjacent thereto may be confirmed to confirm whether the distortion region determination for the divided distortion candidate region in the first image is appropriate.

As another example, the image generating device (1000) may determine the divided distortion candidate region as a distortion region based on the similarity between the divided distortion candidate region and the template image of the lens flare. If a matching or similar pattern is confirmed between the template image representing the distortion shape of the image according to a typical lens flare and the image of the divided distortion candidate region, the image generating device (1000) may determine the divided distortion candidate region as a distortion region.

For another example, the image generating device (1000) may determine the divided distortion candidate region as a distortion region based on the similarity between the divided distortion candidate region and the matching candidate region, which is an area having a light quantity greater than a predetermined standard among the first image and the second image. In the example described above, the template image of the lens flare represents the distortion shape of the image due to a general lens flare, and therefore may not reflect all of the various distortion shapes due to the lens flare that occur in various ways depending on the situation. The area having a light quantity greater than a predetermined standard among the first image and the second image is likely to be a distortion region of the image due to the lens flare, and therefore the image generating device (1000) may determine this as a matching candidate region. The image generating device (1000) may utilize the determined matching candidate region for the same purpose as the template image of the lens flare in the above example, and if a matching or similar pattern is confirmed between the image of the matching candidate region and the divided distortion candidate region, the divided distortion candidate region may be determined as a distortion region.

Figure 7 is a drawing for explaining a process of performing image correction corresponding to a distortion area for the first image.

Referring to FIG. 7, the image generating device (1000) can perform image correction by extracting an area corresponding to a distortion area analyzed in a first image from a second image and replacing the distortion area analyzed in the first image with an area extracted from the second image. As a result, it can be seen that the image generating device (1000) removes a distortion area from the first image, copies an area corresponding to the removed distortion area from the second image, and inserts it into the first image.

When an area of a second image corresponding to a distorted area of a first image is included in the first image by replacing the distorted area of the first image, the image generating device (1000) may adjust an exposure value of the second image according to an exposure value of the first image and then extract an area corresponding to the distorted area analyzed in the first image from the second image so as to maintain continuity with surrounding areas. In addition, the image generating device (1000) may perform blending processing on the boundary of the replaced area after replacing the distorted area analyzed in the first image with the area extracted from the second image. The blending processing may be performed by interpolating pixel values of pixels surrounding pixels at the boundary of the replaced area or by weighting and averaging pixel values of pixels surrounding the same.

Fig. 8 is a flow chart for explaining an image generation method. The content described above for the image generation device (1000) may be applied to the image generation method of the present disclosure even if the content is omitted below.

At step 810, the image generating device (1000) can control the direction-changing camera (1300) in a first direction to acquire a first image.

At step 820, the image generating device (1000) can control the camera (1300) in a second direction different from the first direction to obtain a second image that partially overlaps the first image. If the first image obtained in the first direction includes a light source, the second direction can be any direction that allows the light source included in the first image to be excluded from the second image.

Additionally, the image generating device (1000) may control the camera (1300) in a plurality of directions (i.e., second directions) including the second direction to obtain a plurality of images (i.e., second images) including the second image.

Meanwhile, the image generating device (1000) may further acquire a fourth image by adjusting the shooting parameters when acquiring the second image in the second direction. The fourth image may have the same shooting direction as the second image, but different shooting parameters, and may be preliminarily captured so that it can be used for image correction of the first image. When the image generating device (1000) acquires a plurality of second images, it may further acquire fourth images corresponding to the second images.

In step 830, the image generating device (1000) can analyze a distortion area of the first image based on the first image and the second image. The image generating device (1000) can warp the second image with respect to the first image and determine a matching area between the first image and the warped second image. The image generating device (1000) can segment a distortion candidate area in the first image based on a pixel difference between the matching areas. The image generating device (1000) can determine the distortion area based on whether the segmented distortion candidate area satisfies a condition for being determined as a lens flare. For example, the image generating device (1000) can determine the segmented distortion candidate area as a distortion area based on a degree of brightness change in the segmented distortion candidate area or a degree of continuity of pixel values at a boundary between the segmented distortion candidate area and a surrounding area. For another example, the image generating device (1000) may determine the segmented distortion candidate area as a distortion area based on the similarity between the segmented distortion candidate area and a template image of a lens flare, or the similarity between the segmented distortion candidate area and a matching candidate area, which is an area having a light amount greater than a predetermined standard among the first image and the second image.

Additionally, when the image generating device (1000) acquires second images corresponding to the second directions, it can analyze the distortion area of the first image based on the first image and the second images.

In step 840, the image generating device (1000) can generate a third image on which image correction is performed corresponding to the analyzed distortion area of the first image. The image generating device (1000) can perform image correction by extracting an area corresponding to the analyzed distortion area of the first image from the second image and replacing the analyzed distortion area of the first image with an area extracted from the second image. To this end, the image generating device (1000) can adjust an exposure value of the second image according to an exposure value of the first image and then extract an area corresponding to the analyzed distortion area of the first image from the second image. The image generating device (1000) can perform image correction by replacing the analyzed distortion area of the first image with an area extracted from the second image and then performing blending processing on the boundary of the replaced area. Accordingly, a third image on which image correction is performed corresponding to the analyzed distortion area of the first image can be generated.

In addition, the image generating device (1000) may obtain second images corresponding to the second directions, and, when analyzing a distortion area of the first image, use at least one of the second images to perform image correction corresponding to the analyzed distortion area of the first image. Meanwhile, when the image generating device (1000) obtains at least one fourth image together with at least one second image in the second direction, the image generating device (1000) may use at least one fourth image to generate a third image in which image correction corresponding to the analyzed distortion area of the first image is performed on the first image.

FIG. 9 is a drawing for explaining an example of an image generating device (1000).

FIG. 9 shows a case where the image generating device (1000) is a smart phone or a digital camera. The image generating device (1000) may further include a communication interface module (1400) and a display (1500) in addition to the memory (1100), processor (1200), and camera (1300) described above. In addition, the image generating device (1000) may include components such as a position sensor for detecting the position of the image generating device (1000) or a power supply for supplying power to the image generating device (1000), but a description thereof is omitted.

The communication interface module (1400) can perform wired and wireless communication with other devices or networks. To this end, the communication interface module (1400) can include a communication module that supports at least one of various wired and wireless communication methods. For example, a communication module that performs short-range communication such as Wi-Fi (Wireless Fidelity) and Bluetooth, or various types of mobile communication or ultra-wideband communication can be included. The communication interface module (1400) can be connected to a device located outside of the image generating device (1000), which is a smart phone, and can transmit an image acquired or generated by the image generating device (1000) to the device located outside.

The display (1500) includes an output unit that provides information or an image, and may further include an input unit that receives an input. The output unit may include a display panel and a controller that controls the display panel, and may be implemented in various ways such as an OLED (Organic Light Emitting Diodes) display, an AM-OLED (Active-Matrix Organic Light-Emitting Diode) display, an LCD (Liquid Crystal Display), etc. The input unit may receive various types of input from a user, and may include at least one of a touch panel, a keypad, and a pen recognition panel. The display (1500) may be provided in the form of a touch screen in which a display panel and a touch panel are combined, and may be implemented to be flexible or foldable.

FIG. 10 is a drawing for explaining another example of an image generating device (1000).

FIG. 10 shows a case where the image generating device (1000) is an augmented reality device. The image generating device (1000) may further include a communication interface module (1400), a display (1550), a display engine unit (1600), and a gaze tracking sensor (1700) in addition to the memory (1100), processor (1200), and camera (1300) described above. In addition, the device may include components such as a position sensor for detecting the position of the image generating device (1000) or a power supply unit for supplying power to the image generating device (1000), but a description thereof is omitted.

The communication interface module (1400) can perform wired and wireless communication with other devices or networks. To this end, the communication interface module (1400) can include a communication module that supports at least one of various wired and wireless communication methods. For example, a communication module that performs short-range communication such as Wi-Fi (Wireless Fidelity) and Bluetooth, or various types of mobile communication or ultra-wideband communication can be included. The communication interface module (1400) can be connected to a device located outside of the image generating device (1000), which is an augmented reality device, and can transmit an image acquired or generated by the image generating device (1000) to the device located outside.

The image generating device (1000), which is an augmented reality device, can provide a pop-up of a virtual image through a display (1550) and a display engine unit (1600). The virtual image is an image generated through an optical engine and can include both static images and dynamic images. The virtual image is observed together with a scene of the real world that a user sees through the augmented reality device, and can be an image that represents information about a real object in a scene of the real world, information about the operation of the image generating device (1000), which is an augmented reality device, or a control menu.

The display engine unit (1600) may include an optical engine that generates and projects a virtual image and a guide unit that guides light of the virtual image projected from the optical engine to the display (1550). The display (1550) may include a see-through light guide plate (waveguide) built into the left-eye lens unit and/or the right-eye lens unit of the image generating device (1000), which is an augmented reality device. The display (1550) may display a virtual image that represents information about an object or information about an operation of the image generating device (1000), or a control menu.

When a pop-up of a virtual image is displayed on the display (1550), a user wearing an image generating device (1000), which is an augmented reality device, can expose the user's hand to the camera (1300) to manipulate the pop-up of the virtual image, and execute the function of the image generating device (1000) by having the exposed hand select the function in the pop-up of the virtual image.

The gaze tracking sensor (1700) can detect gaze information such as the gaze direction of the user's eyes, the position of the pupil of the user's eyes, or the coordinates of the center point of the pupil. The processor (1200) can determine the form of eye movement based on the gaze information of the user detected by the gaze tracking sensor (1700). For example, the processor (1200) can determine various forms of gaze movement including fixation in which the user gazes at a certain location, pursuit in which the user chases a moving object, and saccade in which the user's gaze quickly moves from one gaze point to another, based on the gaze information obtained from the gaze tracking sensor (1700).

The processor (1200) of the image generation device (1000) can use the gaze tracking sensor (1700) to determine the user's gaze point or the user's gaze movement, and use this to control the image generation device (1000). The processor (1200) can control the direction of the camera (1300) according to the gaze point or the gaze movement determined by the gaze tracking sensor (1700), and obtain at least one image. For example, the user can wear the image generation device (1000), which is an augmented reality device, to obtain a first image in a first direction, and then control the direction of the camera (1300) according to the user's gaze point or the gaze movement, and obtain a second image in a second direction.

The image generating device (1000) described in the present disclosure may be implemented by hardware components, software components, and/or a combination of hardware components and software components. For example, the image generating device (1000) described in the disclosed embodiments may be implemented by using one or more general-purpose computers or special-purpose computers, such as processors, arithmetic logic units (ALUs), Application Specific Integrated Circuits (ASICs), Digital Signal Processors (DSPs), Digital Signal Processing Devices (DSPDs), Programmable Logic Devices (PLDs), microcomputers, microprocessors, or any other device capable of executing instructions and responding.

Software may include a computer program, code, instructions, or a combination of one or more of these, which may configure a processing device to perform a desired operation or may independently or collectively command the processing device.

The software may be implemented as a computer program including instructions stored on a computer-readable storage medium. Examples of the computer-readable storage medium include magnetic storage media (e.g., read-only memory (ROM), random-access memory (RAM), floppy disks, hard disks, etc.) and optical readable media (e.g., CD-ROMs, Digital Versatile Discs (DVDs)). The computer-readable storage medium may be distributed across network-connected computer systems so that the computer-readable code may be stored and executed in a distributed manner. The medium may be readable by a computer, stored in a memory, and executed by a processor.

A computer may include an image generating device (1000) according to the disclosed embodiments, as a device capable of calling a command stored from a storage medium and performing an operation according to the called command according to the disclosed embodiments.

A computer-readable storage medium may be provided in the form of a non-transitory storage medium. Here, 'non-transitory' means that the storage medium does not contain signals and is tangible, but does not distinguish whether data is stored semi-permanently or temporarily in the storage medium.

Additionally, the method according to the disclosed embodiments may be provided as a computer program product. The computer program product may be traded between sellers and buyers as a commodity.

The computer program product may include a software program, a computer-readable storage medium having the software program stored thereon. For example, the computer program product may include a product in the form of a software program (e.g., a downloadable application) distributed electronically by a manufacturer of the image generating device (1000) or through an electronic market (e.g., Google Play Store, App Store). For electronic distribution, at least a portion of the software program may be stored in a storage medium or temporarily generated. In this case, the storage medium may be a storage medium of a manufacturer's server, an electronic market's server, or an intermediary server that temporarily stores the software program.

The computer program product may include a storage medium of the server or a storage medium of the terminal in a system comprising a server and a terminal (e.g., an image generating device). Or, if there is a third device (e.g., a smart phone) that is communicatively connected to the server or the terminal, the computer program product may include a storage medium of the third device. Or, the computer program product may include a software program itself that is transmitted from the server to the terminal or the third device, or transmitted from the third device to the terminal.

In this case, one of the server, the terminal and the third device may execute the computer program product to perform the method according to the disclosed embodiments. Alternatively, two or more of the server, the terminal and the third device may execute the computer program product to perform the method according to the disclosed embodiments in a distributed manner.

For example, a server (e.g., a cloud server or an artificial intelligence server, etc.) may execute a computer program product stored on the server, thereby controlling a terminal in communication with the server to perform a method according to the disclosed embodiments.

As another example, a third device may execute a computer program product to control a terminal in communication with the third device to perform a method according to the disclosed embodiment.

When the third device executes the computer program product, the third device may download the computer program product from the server and execute the downloaded computer program product. Alternatively, the third device may execute the computer program product provided in a preloaded state to perform the method according to the disclosed embodiments.

Although the embodiments have been described above by way of limited examples and drawings, those skilled in the art can make various modifications and variations from the above description. For example, even if the described techniques are performed in a different order from the described method, or if the components of the described electronic devices, structures, circuits, etc. are combined or combined in a different form from the described method, or if they are replaced or substituted by other components or equivalents, appropriate results can be achieved.

Claims (20)

Camera with swivel rotation;
a memory for storing one or more instructions; and
A processor comprising one or more of the above instructions,
The above processor,
Controlling the camera in a first direction to obtain a first image, controlling the camera in a second direction different from the first direction to obtain a second image partially overlapping with the first image, analyzing a distortion area of the first image based on the first image and the second image, and generating a third image by performing image correction corresponding to the analyzed distortion area for the first image,
The above processor executes one or more of the above instructions,
An image generating device that warps the second image with respect to the first image, determines a matching area between the first image and the warped second image, segments a distortion candidate area in the first image based on a difference in pixel values between the matching areas, and determines the distortion area based on whether the segmented distortion candidate area satisfies a condition for being determined as a lens flare. delete In the first paragraph,
The above processor executes one or more of the above instructions,
An image generating device that determines the divided distortion candidate area as the distortion area based on the degree of change in brightness in the divided distortion candidate area or the degree of continuity of pixel values at the boundary between the divided distortion candidate area and the surrounding area. In the first paragraph,
The above processor executes one or more of the above instructions,
An image generating device that determines the divided distortion candidate area as the distortion area based on the similarity between the divided distortion candidate area and the template image of the lens flare or the similarity between the divided distortion candidate area and a matching candidate area that is an area having a light amount greater than a predetermined standard among the first image and the second image. In the first paragraph,
The above processor executes one or more of the above instructions,
An image generating device that performs image correction by extracting an area corresponding to the analyzed distortion area from the second image and replacing the analyzed distortion area in the first image with the area extracted from the second image. In clause 5,
The above processor executes one or more of the above instructions,
An image generating device that performs image correction by adjusting the exposure value of the second image according to the exposure value of the first image, extracting an area corresponding to the analyzed distortion area from the second image, replacing the analyzed distortion area in the first image with an area extracted from the second image, and then performing blending processing on the boundary of the replaced area. In the first paragraph,
An image generating device, wherein when a light source is included in the first image acquired in the first direction, the second direction is an arbitrary direction that causes the light source included in the first image to be excluded from the second image. In the first paragraph,
The above processor executes one or more of the above instructions,
An image generating device that acquires a fourth image by adjusting the shooting parameters when acquiring the second image in the second direction, and generates a third image by performing image correction using the fourth image. In the first paragraph,
The above processor executes one or more of the above instructions,
An image generating device that controls the camera in a plurality of directions including the second direction to obtain a plurality of images including the second image, and analyzes a distortion area of the first image based on the first image and the plurality of images. In Article 9,
The above processor executes one or more of the above instructions,
An image generating device that acquires the plurality of images while changing the direction of the camera along a predetermined trajectory centered on the first direction. delete delete delete delete delete delete delete delete delete delete

Images