KR101991666B1 - Method of generating sphere-shaped image, method of playing back sphere-shaped image, and apparatuses thereof - Google Patents

Abstract

According to an embodiment, an apparatus for generating a spherical image divides a surface of a spherical model into a mesh including a plurality of polygons, and generates a mesh based on target areas of polygons determined based on importance of polygons based on an input image. The image is deformed and a spherical image is generated based on the deformed mesh.

Description

METHODS OF GENERATING SPHERE-SHAPED IMAGE, METHOD OF PLAYING BACK SPHERE-SHAPED IMAGE, AND APPARATUSES THEREOF}

The embodiments below relate to a method for generating a spherical image, a method for reproducing a spherical image, and apparatuses thereof.

Equirectangular projection is a projection method in which lines of longitude are mapped to vertical lines at regular intervals, and circles of latitude are mapped to horizontal straight lines at regular intervals. An isotropic projection can be used to implement an image in two-dimensional space, but image distortion can occur. For example, as the north and south poles increase in the world map, and the land becomes wider toward the north and south, more and more image distortions occur, and the number of pixels used per unit area also increases. Can vary.

This problem can occur not only in isotropic projection, but also in all cases where the sphere surface is intended to be represented in a plane.

Embodiments to be described below propose a technique for optimizing a spherical image using a modified spring model.

According to one side, a method for generating a spherical image comprises the steps of dividing the surface of a sphere-shaped model (mesh) comprising a plurality of polygons (mesh); Determining importance of the polygons based on regions corresponding to the polygons in an input image; Determining target areas of the polygons based on the importance of the polygons; Deforming the mesh based on target areas of the polygons; And generating a spherical image based on the input image and the deformed mesh.

Determining importance of the polygons may include determining importance of pixels in the input image; Projecting the pixels onto the polygons; And corresponding to the polygons, determining the importance of the corresponding polygon based on the importance of at least one pixel projected onto the corresponding polygon.

The determining of the importance of the polygons may include an image gradient of the regions corresponding to the plurality of polygons, whether an edge is detected in the regions, and the number of feature points included in the regions. And determining the importance based on any one or a combination of objects detected in the regions.

The determining of the importance of the polygons may include determining the importance of the polygons based on a user's selection of the plurality of polygons.

The determining of the target areas may include determining a target area of the corresponding polygon to be wider than an area of the corresponding polygon when the importance of the polygon is greater than a preset criterion, corresponding to each of the plurality of polygons; And determining the target area of the corresponding polygon to be smaller than the area of the corresponding polygon when the importance of the polygon is less than or equal to a preset criterion.

The determining of the target areas corresponds to each of the plurality of polygons, and determining a target area of the corresponding polygon based on a ratio between the importance of the corresponding polygon and the sum of the importance of the plurality of polygons. It may include a step.

The determining of the target areas may include determining the target areas such that the sum of the target areas of the plurality of polygons corresponds to the total area of the surface of the spherical model.

Deforming the mesh may include deforming the mesh such that a target area of each of the polygons is proportional to the sum of the interior angles of the polygons.

Deforming the mesh may include determining deformation accelerations applied to vertices of each of the polygons using a spring model; And deforming the mesh using the deformation acceleration.

The determining of the deformation acceleration may correspond to each of the polygons, and the vertex of the corresponding polygon is proportional to the difference between the target area of the corresponding polygon and the actual area of the corresponding polygon multiplied by a spring constant. Determining the strain acceleration applied to the fields.

The determining of the deformation acceleration may include determining, according to each of the polygons, deformation angles of vertices of the corresponding polygon based on a difference between a target area of the corresponding polygon and a sum of the internal angles of the corresponding polygon; And determining the deformation acceleration based on a deformation angle of vertices of the corresponding polygon, a distance on a spherical surface of each of the vertices constituting the corresponding polygon, and a vector direction between the remaining vertices. It may include.

Deforming the mesh may include deforming the mesh by moving the vertices of the polygons in the mesh according to the deformation acceleration of the vertices of the polygons.

The method of generating a spherical image may include determining deformation accelerations applied to vertices of the deformed polygon based on a difference between an area of the deformed polygon and a target area of the deformed polygon by moving the vertices; And moving vertices of the deformed polygon in the deformed mesh based on the deformed acceleration determined in correspondence to the deformed polygon.

Determining the deformation acceleration applied to the vertices of the deformed polygon and moving the vertices of the deformed polygon may be repeatedly performed until a predetermined condition related to the deformation acceleration is satisfied.

Deforming the mesh may include determining whether the corresponding polygon is flipped from the mesh; And when it is determined that the corresponding polygon is inverted, the force applied to the vertices of the corresponding polygon is multiplied by a spring constant multiplied by the sum between the target area of the corresponding polygon and the actual area of the corresponding polygon. Determining a proportional value.

The generating of the spherical image may include fitting to the deformed mesh by non-uniformly sampling the regions included in the input image.

According to one side, a method for reproducing a spherical image comprises the steps of acquiring a spherical image generated by being unevenly sampled; Acquiring information about the deformed mesh corresponding to the spherical image; Mapping the spherical image to a spherical model based on the information about the deformed mesh; And playing the mapped spherical image.

The information about the deformed mesh may include location information where vertices of a plurality of polygons included in the deformed mesh are moved.

According to one side, the apparatus for generating a spherical image divides the surface of the sphere-shaped model (mesh) into a mesh including a plurality of polygons, and the area corresponding to the polygons in the input image Determine the importance of the polygons, determine the target areas of the polygons, modify the mesh based on the target areas of the polygons, and determine the input image and the At least one processor for generating a spherical image based on the deformed mesh.

1 is a flowchart illustrating a method of generating a spherical image according to an embodiment.
FIG. 2 is a diagram for describing a method of dividing a surface of a spherical model into a mesh according to one embodiment; FIG.
3 is a flowchart illustrating a method of determining importance of each of polygons according to an embodiment.
4 is a view for explaining a method of deforming a mesh using a spring model according to an embodiment.
5 is a diagram for describing a method of calculating a vector on a spherical model and moving a point on a spherical surface in a vector direction according to an embodiment.
6 is a view for explaining a process of deforming the mesh in accordance with one embodiment.
7 is a flowchart illustrating a method of playing a spherical image, according to an exemplary embodiment.
8 is a block diagram of an apparatus for generating a spherical image or reproducing a spherical image, according to an exemplary embodiment.

Specific structural or functional descriptions disclosed herein are illustrated for the purpose of describing embodiments only in accordance with a technical concept, and the embodiments may be embodied in various other forms and limited to the embodiments described herein. It doesn't work.

Terms such as first or second may be used to describe various components, but these terms should be understood only for the purpose of distinguishing one component from another. For example, the first component may be referred to as a second component, and similarly, the second component may also be referred to as a first component.

When a component is referred to as being "connected" or "connected" to another component, it may be directly connected to or connected to that other component, but it may be understood that other components may be present in between. Should be. On the other hand, when a component is said to be "directly connected" or "directly connected" to another component, it should be understood that there is no other component in between. Expressions describing the relationship between the components, such as "between liver" and "immediately between" or "neighboring to" and "direct neighboring to" should be interpreted as well.

Singular expressions include plural expressions unless the context clearly indicates otherwise. As used herein, the terms "comprise" or "having" are intended to designate that the stated feature, number, step, operation, component, part, or combination thereof is present, but one or more other features or numbers, It is to be understood that it does not exclude in advance the possibility of the presence or addition of steps, actions, components, parts or combinations thereof.

Unless defined otherwise, all terms used herein, including technical or scientific terms, have the same meaning as commonly understood by one of ordinary skill in the art. Terms such as those defined in the commonly used dictionaries should be construed as having meanings consistent with the meanings in the context of the related art, and are not construed in ideal or excessively formal meanings unless expressly defined herein. Do not.

1 is a flowchart illustrating a method of generating a spherical image, according to an exemplary embodiment. Referring to FIG. 1, an apparatus for generating a spherical image (hereinafter, referred to as a “generating apparatus”) may include a mesh including a plurality of polygons of a surface of a sphere-shaped model. Divided by 110. In this case, the area of the plurality of divided polygons may be the same. A method of dividing the surface of the spherical model into a mesh by the generating apparatus will be described in detail with reference to FIG. 2. The generating device may be implemented by one or more software modules, one or more hardware modules, or various combinations thereof.

The generating device determines the importance of the polygons based on the regions corresponding to the polygons in the input image (120). The generating apparatus may include, for example, an image gradient of an area corresponding to each of the plurality of polygons in the input image, whether an edge is detected in each area, the number of feature points included in each area, and each The importance of each of the polygons may be determined based on whether or not an object (eg, a person, an animal, a car, etc.) is detected in the area. Alternatively, the generating device may determine the importance level based on the user's selection of the polygons. The importance of each of the polygons may be determined by a value between 0 and 1, for example.

The generating device determines 130 target areas of the polygons based on the importance of each of the polygons. For example, when the importance of a particular polygon is greater than a preset criterion, the generation apparatus may determine a target area of the corresponding polygon to be wider than the current area of the polygons. When the importance of the polygon is smaller than the preset criterion, the generating device may determine the target area of the polygon to be smaller than the current area of the polygons. In an embodiment, the polygons having a higher importance may have a larger area in the sampling process, that is, the polygons having a higher importance may have a higher resolution by determining a target area of the polygons based on the importance of each polygon. .

The generating apparatus may determine the target area of the corresponding polygon based on, for example, a ratio between the importance of the specific polygon among the plurality of polygons and the sum of the importance of the plurality of polygons. In addition, the generating apparatus may determine the target area such that the sum of the target areas of the plurality of polygons corresponds to the total area of the surface of the spherical model.

The method of generating the importance of each of the polygons and the target areas of the polygons will be described in detail with reference to FIG. 3.

The generating device deforms the mesh 140 based on the target areas of the polygons. The generating device may deform the mesh such that the target area of each polygon is proportional to the sum of the interior angles of each polygon. A method of deforming the mesh by the generating apparatus will be described in detail with reference to FIG. 4.

The generating device generates a spherical image based on the input image and the deformed mesh (150). The generating apparatus may fit the deformed mesh by, for example, non-uniformly sampling the regions included in the input image. Non-uniform sampling may be performed based on the area of polygons included in the deformed mesh. For example, the non-uniform sampling may be performed at high resolution when the area of the polygon is large and at low resolution when the area of the polygon is narrow.

FIG. 2 is a diagram for describing a method of dividing a surface of a spherical model into a mesh, according to an exemplary embodiment. Referring to FIG. 2, an icosahedron 210, a geodesic polyhedron 230, and a spherical model 250 are shown.

In an exemplary embodiment, the geodetic polyhedron 230 may be generated by dividing the icosahedron 210 having the same size of each face among various types of models for representing a sphere. The generating apparatus according to the exemplary embodiment may divide the surface of the spherical model 250 into a mesh including a plurality of polygons by using the mesh of the geodetic polyhedron 230. In one embodiment, the reason for generating the spherical model 250 using the geodetic polyhedron 230 is to remove the directivity of the model as much as possible, and to equalize the size of each face of the polyhedron.

Vertices included in the geodetic polyhedron 230 may be adjacent to five or six triangles. For example, the vertices of icosahedron 210 may be adjacent to five triangles, and other vertices may be adjacent to six triangles. As will be described in detail below, each vertex may be subjected to deformation acceleration according to the spring model, and the magnitude and direction of the deformation acceleration may be determined by adjacent triangles.

The spherical model 250 may be referred to as a 'sphere-shaped template'. The radius of spherical model 250 may be, for example.

3 is a flowchart illustrating a method of determining importance of polygons according to an embodiment. Referring to FIG. 3, the generating apparatus according to an embodiment may determine the importance of pixels in an input image (310). For example, when the image gradient of the pixels in the input image is greater than or equal to a predetermined criterion, the generation device may determine a higher importance of the pixels. When the image gradient of the pixels in the input image is smaller than a predetermined criterion, the generation device may determine the importance of the pixels lower. For example, when the pixels in the input image correspond to edges, the generation device may determine a high importance of the pixels. When the pixels in the input image do not correspond to the edge, the generation device may determine the importance of the pixels lower. Or, for example, when the pixels in the input image correspond to an object (eg, a person, an object, etc.), the generation device may determine a high importance of the pixels. The importance of the pixels may have a value between 0 and 1.

The generating device may project the pixels onto the polygons formed on the surface of the spherical model (320). For example, the generating apparatus may find the three-dimensional pixel p 'projected on the polygon by projecting the two-dimensional pixel p onto the polygons formed on the surface of the spherical model.

In operation 330, the generating apparatus may determine the importance W _f of the corresponding polygon based on the importance value of the at least one pixel p projected on the corresponding polygon. The generating apparatus adds the importance of all the pixels included in the corresponding polygon to the importance of the corresponding polygon (W _f ), and adds the importance (W _f ) of each polygon to the number of pixels (C _f ) corresponding to each of the polygons. By dividing, we can finally determine the importance of the polygon (W _f ). In this case, the generating apparatus reduces the importance of the polygon by a predetermined ratio (for example, 0.8) and adds a lower limit (for example, 0.2) so that the importance of each polygon is not too small. , May have a value of 0.2 to 1.

After the importance of each of the polygons is determined, the generating device knows the total area of the surface of the spherical model so that the sum of the target areas of each polygon corresponds to the total area of the surface of the spherical model. (TA _f ) can be determined. The target area TA _f of each of the polygons may be obtained through, for example, Equation 1 below.

The generating apparatus determines a value obtained by multiplying the ratio between the importance W _f of the corresponding polygons among the plurality of polygons by the sum of the importances of the plurality of polygons by 4 * π as the target area TA _f of the corresponding polygon. Can be.

For example, when the input image is a 2D image, the above-described process may be represented by a pseudo code as shown in Table 1 below.

4 is a diagram for describing a method of deforming a mesh using a spring model, according to an exemplary embodiment. Referring to FIG. 4, a polygon with three vertices V, V _1, V ₂ is shown.

According to an embodiment, the generating apparatus determines a deformation acceleration (Accel _v ) applied to vertices V, V _{1, and} V ₂ of a corresponding polygon among the polygons using a spring model, and uses the mesh to generate deformation meshes using the deformation acceleration. Can be modified. The operation of the spring model used in one embodiment may be represented by a water code, for example, as shown in Table 2 below.

The generating device initializes the deformation acceleration (Accel _v ) of all the vertices of the polygons formed on the surface of the spherical model to (0,0,0), and then applies the force applied to each of the vertices of the corresponding polygon among the plurality of polygons. You can get the force.

In this case, the force applied to each of the vertices of the polygon is the spring constant (K) * (−) in the difference between the target area TA _f of the corresponding polygon and the actual area A _f of the corresponding polygon. The strain acceleration can be determined to be proportional to the product of 1). The spring constant (K) may be, for example, 50, but is not limited thereto. In addition, the spring constant K may be adjusted to various values.

For example, if the polygon is a triangle, the generating device determines the vertices V, V _{1 and} V ₂ of the polygon based on the difference between the target area TA _{f of} the polygon and the sum of the interior angles of the polygon. Can determine the force applied to each. In this case, a force applied to the vertices V, V _{1, and} V ₂ of the polygon may be expressed by, for example, Equation 2 below. In addition, a method of calculating an interior angle for one vertex of a polygon (triangle) on the surface of a spherical model will be described with reference to FIG. 5 (a) below.

Here, N may correspond to the number of vertices of the polygon.

에 의해 구할 수 있다. For example, if the polygon is a triangle as shown in Fig. 4, since N = 3, the force applied to the vertices of the triangle is

Can be obtained by

If the polygon is a triangle, a corresponding polygon may be flipped in the mesh. In this case, the generating device may determine whether the corresponding polygon is flipped in the mesh. For example, the generating apparatus may determine that the polygon corresponding to the case where the cross product direction of the vectors of two adjacent sides in a specific polygon is different from the direction of the cross product of two adjacent sides in another polygon is turned upside down in the mesh. .

When it is determined that the corresponding polygon is inverted, the generating device may calculate the force by reversing the sign of the actual area A _f and inverting the sign of the entire right term in Equation 2. For example, the generating device is a value proportional to the force exerted on the vertices of the corresponding polygon by the sum of the sum of the target area of the corresponding polygon and the actual area of the corresponding polygon multiplied by a spring constant (K), In other words, we can determine K * (TA _f + A _f ) / 6.

The generating device may determine deformation acceleration for each vertex of each of the polygons, and deform the mesh using the change acceleration. The generating device determines the angle of deformation of the vertices of the corresponding polygon (that is, the force), the spherical distance between each of the vertices constituting the corresponding polygon and each of the remaining vertices (GCD), and the remainder. The deformation acceleration can be determined based on the vector direction between the vertices.

For example, the vertex of the triangle in Figure 4 (V, V _1, V ₂₎ of the vertices V ₁ and a vertex V ₂ deulyigo vertex connected to the vertex V, V ₁₂ and V ₂₁ is about the two lines connected to the vertex V Let's call the tangent lines.

와 같이 구할 수 있다. 또한, 꼭지점 V₂에 대한 변형 가속도(Accel_v2)는

와 같이 구할 수 있다. 이때, 해당하는 다각형의 꼭지점들(V, V_1, V₂)의 변형 가속도(Accel_v)에 기초하여 해당하는 다각형의 꼭지점들(V, V_1, V₂)의 구형 모델 표면상(구면상)의 이동 속도(Vel_v)가 산출될 수 있다. The strain acceleration (Accel _v1 ) for vertex V ₁ is

It can be obtained as In addition, the strain acceleration (Accel _v2 ) for vertex V ₂ is

It can be obtained as At this time, in the vertices of the polygon that (V, V _1, V ₂₎ of the deformation acceleration (Accel _v) the vertices of the polygon corresponding to the basis of the (V, V _1, V ₂₎ onto a spherical model surface (sphere surface The moving speed Vel _{v of} ) may be calculated.

The generating device may deform the mesh by moving the vertices of the polygon in the mesh according to the deformation acceleration (Accel _v ) of the vertices of the polygon. The deformation of the mesh may be performed through a plurality of iterations, and when the generating device moves vertices of a specific polygon in individual iterations, the area of the corresponding polygon may be changed. In this case, the deformation acceleration (ie, the applied force) may vary according to the difference between the changed area of the polygon and the target area.

The generation device may determine the deformation acceleration applied to the vertices of the deformed polygon based on the difference between the area of the deformed polygon and the target area of the deformed polygon by moving the vertices as the iteration proceeds. The generating device may move the vertices in the next iteration based on the deformation acceleration determined corresponding to the deformed polygon.

For convenience of description, the deformation acceleration applied by a single triangle has been described in FIG. 4, but the deformation acceleration applied by a plurality of adjacent triangles is determined at each vertex. For example, in the case of a vertex of five adjacent triangles, the force may be determined by ten deformation accelerations determined by a total of five triangles. Also, in the case of a vertex having six adjacent triangles, the force may be determined by 12 deformation accelerations determined by a total of six triangles.

When calculating the deformation acceleration (Accel _v ) of vertices (V, V _1, V ₂ ) of a polygon, multiply the distance (GCD) on the spherical model surface (spherical) between each vertex The reason is that the longer the length of one side, the more the vertex needs to move to achieve the desired angle change. A method of obtaining a spherical distance GCD between one of the vertices constituting the corresponding polygon and each of the remaining vertices will be described with reference to FIG. 5 (b) below.

The generating device may be performed repeatedly, for example, until a predetermined condition related to the deformation acceleration is satisfied. The predetermined condition associated with the deformation acceleration is, for example, that the maximum value of the moving velocity Vel _v on the spherical model surface of each vertex updated at every step is equal to or less than the threshold value (eg 0.002), and the acceleration acceleration Accel _v The maximum of may be equal to or less than a threshold (eg, 0.0005), and may be a condition that the corresponding polygon does not flip over in the mesh. The predetermined condition associated with the deformation acceleration may change according to the needs of the user.

According to an embodiment, the generating device may move the vertices of the polygon according to the value of DAMPING_CONSTATN multiplied by the moving speed Vel _v on the sphere surface of each vertex updated for each polygon. In this case, DAMPING_CONSTATN is a constant used for damping in the spring model. The DAMPING_CONSTATN may be a value between 0 and 1 (eg, 0.8).

When the spring model according to one embodiment operates as described, for example, in Table 2 above, all movements of the vertices of the polygons are made only on the surface of the spherical model, and of each vertex on the surface of the spherical model. The strain acceleration (Accel _v ) and the movement speed (Vel _v ) may be tangent to the surface of the spherical model.

5 is a diagram for describing a method of calculating a vector on a spherical model and moving a point on a spherical surface in a vector direction according to an exemplary embodiment. Referring to Figure 5 (a), any one of a polygon on the surface of the sphere model (e.g., triangular) configuration the vertex (tangent) units from P ₁ in contact with the P ₁ toward the P _2, which vector is shown (P ₁ ≠ P ₂ ). In this case, a unit vector tangent to the vertex P ₁ may be expressed as Equation 3 below.

Here, norm means normalize.

In FIG. 5 (a), the cabinet for the vertex P ₂ in the spherical triangle formed by P ₁ , P _{2, and} P ₃ may be obtained through Equation 4 below.

를 이용하여

방향으로

만큼 구면 상에서 움직인 결과가 도시된다. 이때, 단위 구(unit sphere)에서는 구 중심에서 이루는 각의 크기가 곧 구형 모델 표면상(구면상)의 거리(GCD)이므로, P'이 P와 벡터의 크기만큼 각도 차이가 난다면 구 위에서 그 길이만큼 떨어져 있다는 것을 나타낸다. Referring to Figure 5 (b), vertex P on the sphere

Using

In the direction

The result of moving on the sphere is shown. In this case, in the unit sphere, the angle formed from the center of the sphere is the distance (GCD) on the surface of the sphere model (spherical). Indicates that they are separated by length.

는 아래의 [수학식 5]에 의해 구할 수 있다. At this time, in Figure 5 (b)

Can be obtained by Equation 5 below.

In addition, the vertex P 'moved on the spherical surface can be obtained by Equation 6 below.

According to an embodiment, the generating apparatus determines a vector direction and a speed using the above Equations 3 to 5, and uses the Equation 6 to determine each vertex according to the determined speed. Can be moved in the determined vector direction.

In addition, the generating apparatus transforms the unit vector to fit the spherical surface through the method shown in FIGS. 5 (a) and 5 (b), thereby providing a cabinet for one vertex of the polygon and one vertex and the other vertices. The distance GCD on the spherical model surface (spherical) between each can be calculated.

6 is a diagram for describing a process of deforming a mesh, according to an exemplary embodiment. Referring to FIG. 6, there are shown images 610, 620, 630, and 640 in which a surface of a spherical model is sequentially optimized with a deformed mesh, according to an exemplary embodiment.

Image 610 represents an input importance image. In the image 610, the center white square area is an area of greater importance than the black area. The image 620 is an image showing a target area. The middle white portion of the image 620 represents a target area corresponding to an area of high importance in the image 610.

The image 630 represents an initial mesh and corresponds to an image in which polygons dividing the surface of the spherical model into uniform sizes are unfolded in a plane. Image 640 represents a mesh deformed based on the target area.

The generating apparatus according to an embodiment may deform the initial mesh of the spherical model as shown in the image 640 based on the target area shown in the image 620. It can be seen that the area of the unit polygon corresponding to the target area of high importance in the image 640 is larger than that of the unit polygon in the initial mesh.

As described above, the generating apparatus determines the deformation acceleration applied to the vertices of the corresponding polygon among the polygons included in the initial mesh of the image 630 using the spring model, and uses the deformation acceleration to display the image 630. By modifying the initial mesh shown in FIG. 1, the mesh may be transformed into the mesh shown in the image 640.

7 is a flowchart illustrating a method of playing a spherical image, according to an exemplary embodiment. Referring to FIG. 7, in operation 710, an apparatus for reproducing a spherical image (hereinafter, referred to as a “reproducing apparatus”) acquires a spherical image generated by being unevenly sampled.

The playback device obtains information about the deformed mesh in response to the spherical image (720). Here, the information about the deformed mesh may include, for example, location information of vertices of a plurality of polygons included in the deformed mesh.

The playback apparatus maps the spherical image to the spherical model based on the information about the deformed mesh (730). The reproducing apparatus may map vertices included in the mesh to vertices included in the spherical model, for example, based on the information about the actual area of the polygons and / or the information about the target area of the polygons. The spherical model can be, for example, a viewing sphere that includes a plurality of polygons that divide the spherical surface evenly. The viewing sphere can be set to have the same resolution as that of the final mesh.

For example, when the reproducing apparatus reproduces a 360-degree image, each frame image may be texture mapped to the viewing sphere based on the information about the deformed mesh. In this case, in order to restore the rendered image having a non-uniform appearance to the original image, the UV position of each vertex in the viewing sphere may be determined according to the UV information of the spherical model.

The reproduction device reproduces the mapped spherical image (740).

8 is a block diagram of an apparatus for generating a spherical image or reproducing a spherical image, according to an exemplary embodiment. 9, an apparatus 900 according to an embodiment includes a communication interface 910 and a processor 930. The device 900 may further include a memory 950 and a display device 970. The communication interface 910, the processor 930, the memory 950, and the display device 970 may communicate with each other via the communication bus 905.

The communication interface 910 may acquire an input image. For example, the input image may be captured or photographed through a photographing device (not shown) such as a camera or an image sensor included in the device 900, or may be an image captured from the outside of the device 900.

The processor 930 divides the surface of the spherical model into a mesh including a plurality of polygons. The processor 930 determines the importance of the polygons based on regions corresponding to the polygons in the input image. The processor 930 determines target areas of the polygons based on the importance of the polygons, and deforms the mesh based on the target areas of the polygons. The processor 930 generates a spherical image based on the input image and the deformed mesh.

The memory 950 may store an old model. In addition, the memory 950 may store a spherical image generated by the processor 930.

Alternatively, the processor 930 obtains a spherical image generated by being nonuniformly sampled, and obtains information about a deformed mesh corresponding to the spherical image. In this case, the information about the spherical image and the deformed mesh may be stored in the memory 950. The processor 930 maps the spherical image to the spherical model based on the information about the deformed mesh, and reproduces the mapped spherical image.

The processor 930 may play the mapped spherical image through, for example, the display apparatus 970.

In addition, the processor 930 may perform an algorithm corresponding to at least one method or at least one method described above with reference to FIGS. 1 to 8. The processor 930 may be a data processing apparatus implemented in hardware having a circuit having a physical structure for executing desired operations. For example, desired operations may include code or instructions included in a program. For example, data processing devices implemented in hardware may include a microprocessor, a central processing unit, a processor core, a multi-core processor, and a multiprocessor. , An application-specific integrated circuit (ASIC), and a field programmable gate array (FPGA).

The processor 930 may execute a program and control the apparatus 900. Program code executed by the processor 930 may be stored in the memory 950.

The memory 950 may store various information generated in the process of the processor 930 described above. In addition, the memory 950 may store various data, programs, and the like. The memory 950 may include a volatile memory or a nonvolatile memory. The memory 950 may include a mass storage medium such as a hard disk to store various data.

The display device 970 may display a spherical image mapped to the spherical model by the processor 930.

The embodiments described above may be implemented as hardware components, software components, and / or combinations of hardware components and software components. For example, the apparatus, methods and components described in the embodiments may include, for example, a processor, a controller, a central processing unit (CPU), a graphics processing unit (GPU), an ALU (for example). arithmetic logic units, digital signal processors, microcomputers, field programmable gate arrays (FPGAs), programmable logic units (PLUs), microprocessors, application specific integrated circuits (ASICS), or instructions ( Like any other device capable of executing and responding to instructions, it may be implemented using one or more general purpose or special purpose computers.

The method according to the embodiment may be embodied in the form of program instructions that can be executed by various computer means and recorded in a computer readable medium. The computer readable medium may include program instructions, data files, data structures, etc. alone or in combination. The program instructions recorded on the media may be those specially designed and constructed for the purposes of the embodiments, or they may be of the kind well-known and available to those having skill in the computer software arts. Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks, and magnetic tape, optical media such as CD-ROMs, DVDs, and magnetic disks, such as floppy disks. Magneto-optical media, and hardware devices specifically configured to store and execute program instructions, such as ROM, RAM, flash memory, and the like. Examples of program instructions include not only machine code generated by a compiler, but also high-level language code that can be executed by a computer using an interpreter or the like. The hardware device described above may be configured to operate as one or more software modules to perform the operations of the embodiments, and vice versa.

Although the embodiments have been described with reference to the accompanying drawings as described above, various modifications and variations are possible to those skilled in the art from the above description. For example, the described techniques may be performed in a different order than the described method, and / or components of the described systems, structures, devices, circuits, etc. may be combined or combined in a different form than the described method, or other components. Or even if replaced or substituted by equivalents, an appropriate result can be achieved. Therefore, other implementations, other embodiments, and equivalents to the claims are within the scope of the claims that follow.

Claims (20)

Dividing a surface of a sphere-shaped model into a mesh including a plurality of polygons;
Determining importance of the polygons based on regions corresponding to the polygons in an input image;
Determining target areas of the polygons based on the importance of the polygons;
Deforming the mesh based on target areas of the polygons; And
Generating a spherical image based on the input image and the deformed mesh;
Including,
Determining the target areas
Determining the target areas such that the sum of target areas of the plurality of polygons is equal to the total area of the surface of the spherical model
Including, the method for generating a spherical image. The method of claim 1,
Determining the importance of the polygons
Determining importance of pixels in the input image;
Projecting the pixels onto the polygons; And
Corresponding to each of the polygons, determining the importance of the corresponding polygon based on the importance of at least one pixel projected onto the corresponding polygon
Including, the method for generating a spherical image. The method of claim 1,
Determining the importance of the polygons
An image gradient of the areas corresponding to the plurality of polygons, whether an edge is detected in the areas, the number of feature points included in the areas, and an object of the objects in the areas. Determining the importance based on any one of detection or a combination thereof
Including, the method for generating a spherical image. The method of claim 1,
Determining the importance of the polygons
Determining the importance based on a user's selection of the plurality of polygons
Including, the method for generating a spherical image. Dividing a surface of a sphere-shaped model into a mesh including a plurality of polygons;
Determining importance of the polygons based on regions corresponding to the polygons in an input image;
Determining target areas of the polygons based on the importance of the polygons;
Deforming the mesh based on target areas of the polygons; And
Generating a spherical image based on the input image and the deformed mesh;
Including,
Determining the target areas
Corresponding to each of the plurality of polygons,
If the importance of the polygon is greater than a preset criterion, determining the target area of the corresponding polygon to be wider than the area of the corresponding polygon; And
If the importance of the polygon is less than the preset criterion, determining the target area of the polygon is smaller than the area of the polygon.
Including, the method for generating a spherical image. Dividing a surface of a sphere-shaped model into a mesh including a plurality of polygons;
Determining importance of the polygons based on regions corresponding to the polygons in an input image;
Determining target areas of the polygons based on the importance of the polygons;
Deforming the mesh based on target areas of the polygons; And
Generating a spherical image based on the input image and the deformed mesh;
Including,
Determining the target areas
Corresponding to each of the plurality of polygons,
Determining a target area of the corresponding polygon based on a ratio between the importance of the corresponding polygon and the sum of the importance of the plurality of polygons
Including, the method for generating a spherical image. delete Dividing a surface of a sphere-shaped model into a mesh including a plurality of polygons;
Determining importance of the polygons based on regions corresponding to the polygons in an input image;
Determining target areas of the polygons based on the importance of the polygons;
Deforming the mesh based on target areas of the polygons; And
Generating a spherical image based on the input image and the deformed mesh;
Including,
Deforming the mesh
Deforming the mesh such that the sum of the interior angles of each of the polygons is proportional to a target area of each of the polygons
Including, the method for generating a spherical image. The method of claim 1,
Deforming the mesh
Determining a strain acceleration applied to vertices of each of the polygons using a spring model; And
Deforming the mesh using the deformation acceleration
Including, the method for generating a spherical image. Dividing a surface of a sphere-shaped model into a mesh including a plurality of polygons;
Determining importance of the polygons based on regions corresponding to the polygons in an input image;
Determining target areas of the polygons based on the importance of the polygons;
Deforming the mesh based on target areas of the polygons; And
Generating a spherical image based on the input image and the deformed mesh;
Including,
Deforming the mesh
Determining a strain acceleration applied to vertices of each of the polygons using a spring model; And
Deforming the mesh using the deformation acceleration
Including;
Determining the deformation acceleration
Corresponding to each of the polygons, the deformation acceleration applied to the vertices of the corresponding polygon is determined to be proportional to the difference between the target area of the corresponding polygon and the actual area of the corresponding polygon multiplied by a spring constant. Steps to
Including, the method for generating a spherical image. Dividing a surface of a sphere-shaped model into a mesh including a plurality of polygons;
Determining importance of the polygons based on regions corresponding to the polygons in an input image;
Determining target areas of the polygons based on the importance of the polygons;
Deforming the mesh based on target areas of the polygons; And
Generating a spherical image based on the input image and the deformed mesh;
Including,
Deforming the mesh
Determining a strain acceleration applied to vertices of each of the polygons using a spring model; And
Deforming the mesh using the deformation acceleration
Including;
Determining the deformation acceleration
Corresponding to each of the polygons,
Determining deformation angles of vertices of the corresponding polygon based on a difference between a target area of the corresponding polygon and the sum of the interior angles of the corresponding polygon; And
Determining the deformation acceleration based on a deformation angle of vertices of the corresponding polygon, a distance on a sphere between each one of the vertices constituting the corresponding polygon, and a vector direction between the remaining vertices
Including, the method for generating a spherical image. The method of claim 1,
Deforming the mesh
Deforming the mesh by moving the vertices of the polygons in the mesh according to the deformation acceleration of the vertices of the polygons
Including, the method for generating a spherical image. Dividing a surface of a sphere-shaped model into a mesh including a plurality of polygons;
Determining importance of the polygons based on regions corresponding to the polygons in an input image;
Determining target areas of the polygons based on the importance of the polygons;
Deforming the mesh based on target areas of the polygons; And
Generating a spherical image based on the input image and the deformed mesh;
Including,
Deforming the mesh
Deforming the mesh by moving the vertices of the polygons in the mesh according to the deformation acceleration of the vertices of the polygons;
Determining deformation accelerations applied to vertices of the deformed polygon based on the difference between the area of the deformed polygon and the target area of the deformed polygon by moving the vertices; And
Moving vertices of the deformed polygon in the deformed mesh based on the deformed acceleration determined corresponding to the deformed polygon
Including, the method for generating a spherical image. Dividing a surface of a sphere-shaped model into a mesh including a plurality of polygons;
Determining importance of the polygons based on regions corresponding to the polygons in an input image;
Determining target areas of the polygons based on the importance of the polygons;
Deforming the mesh based on target areas of the polygons; And
Generating a spherical image based on the input image and the deformed mesh;
Including,
Deforming the mesh
Deforming the mesh by moving the vertices of the polygons in the mesh according to the deformation acceleration of the vertices of the polygons;
Determining deformation accelerations applied to vertices of the deformed polygon based on the difference between the area of the deformed polygon and the target area of the deformed polygon by moving the vertices; And
Moving vertices of the deformed polygon in the deformed mesh based on the deformed acceleration determined corresponding to the deformed polygon
Including;
Determining a deformation acceleration applied to the vertices of the deformed polygon and moving the vertices of the deformed polygon
Repeatedly performing until a predetermined condition related to the deformation acceleration is satisfied. Dividing a surface of a sphere-shaped model into a mesh including a plurality of polygons;
Determining importance of the polygons based on regions corresponding to the polygons in an input image;
Determining target areas of the polygons based on the importance of the polygons;
Deforming the mesh based on target areas of the polygons; And
Generating a spherical image based on the input image and the deformed mesh;
Including,
Deforming the mesh
Corresponding to each of the polygons, determining whether the corresponding polygon is flipped in the mesh; And
If the corresponding polygon is determined to be inverted, the force applied to the vertices of the corresponding polygon is proportional to the sum of the sum of the target area of the corresponding polygon and the actual area of the corresponding polygon multiplied by a spring constant. To determine the value
Including, the method for generating a spherical image. The method of claim 1,
Generating the spherical image
Fitting to the deformed mesh by non-uniformly sampling regions included in the input image
Including, the method for generating a spherical image. Obtaining a spherical image generated by being unevenly sampled;
Acquiring information about the deformed mesh corresponding to the spherical image;
Mapping the spherical image to a spherical model based on the information about the deformed mesh; And
Playing the mapped spherical image
Including,
The deformed mesh is
Corresponding to each of the plurality of polygons included in the mesh, it is determined whether the corresponding polygon is inverted in the mesh, and if it is determined that the corresponding polygon is inverted, the force applied to the vertices of the corresponding polygon is determined. And determining the sum of the sum between the target area of the corresponding polygon and the actual area of the corresponding polygon as a value proportional to the product of the spring constant. The method of claim 17,
The information about the deformed mesh is
And the position information of the vertices of the plurality of polygons included in the deformed mesh. 19. A computer program stored in a medium in combination with hardware to carry out the method of any one of claims 1 to 6 and 8 to 18. Divide the surface of a sphere-shaped model into a mesh comprising a plurality of polygons,
Determining the importance of the polygons based on regions corresponding to the polygons in an input image,
Based on the importance of the polygons, determining target areas of the polygons,
Modifying the mesh based on target areas of the polygons,
Generating a spherical image based on the input image and the deformed mesh
At least one processor
Including,
The at least one processor determines whether a polygon corresponding to each of the polygons is inverted in the mesh, and if it is determined that the corresponding polygon is inverted, the force applied to the vertices of the corresponding polygon is determined. And determining the sum of the target area of the polygon and the actual area of the corresponding polygon as a value proportional to the product of a spring constant.

Images