CN114125273A - Face focusing method and device and electronic equipment - Google Patents

Abstract

The present application discloses a face focusing method, device and electronic device, which belong to the technical field of photography. Wherein, the face focusing method provided by the present application includes: inputting a target image into a target parameter detection model to obtain a target focal length output by the target parameter detection model; focusing on a face based on the target focal length; wherein, the target parameter detection model is Based on 2D face image samples, joint 3D face deformation model and camera model training; 3D face deformation model is used to obtain 3D face based on input facial feature parameters; camera model is used based on input pose parameters and the three-dimensional face to obtain the restored two-dimensional face image; among them, the face feature parameters and the pose parameters are the parameters output by the initial parameter detection model based on the two-dimensional face image sample; the pose parameters are the parameters that characterize the shooting posture of the camera ; The face feature parameter is the parameter that characterizes the face contour.

Description

Face focusing method, device and electronic device

technical field

The present application belongs to the technical field of photography, and in particular relates to a face focusing method, device and electronic device.

Background technique

In real life, when a user uses an electronic device to take a picture, if he needs to take a portrait, he usually needs to focus on the face.

In the related art, when focusing on the face, first perform face detection, and obtain the region of interest according to the face detection frame; then adjust the region of interest according to the pose of the face and the position of the face in the image; Count the phase images in the adjusted region of interest, and calculate the in-focus position based on the Phase Detection Auto Focus (PDAF) technology and the statistical phase images; finally, move the position of the motor according to the in-focus position to complete the face of autofocus.

However, the adjusted region of interest contains not only face information, but also background information. The background and the face are not on the same plane. If the focus position of the face is calculated based on the background information and the face information, the focus accuracy of the face will be affected. poor.

SUMMARY OF THE INVENTION

The purpose of the embodiments of the present application is to provide a face focusing method, device and electronic device, which can solve the problem of poor face focusing accuracy.

In a first aspect, an embodiment of the present application provides a method for focusing on a human face, the method comprising:

Input the target image into the target parameter detection model to obtain the target focal length output by the target parameter detection model;

focusing on the human face based on the target focal length;

The target parameter detection model is obtained by training based on a 2D face image sample, a joint 3D face deformation model and a camera model; the 3D face deformation model is used to obtain a 3D face based on the input facial feature parameters ; The camera model is used to obtain the restored two-dimensional face image based on the input pose parameters and the three-dimensional face;

Wherein, the facial feature parameter and the pose parameter are the parameters output by the initial parameter detection model based on the two-dimensional face image sample; the pose parameter is a parameter representing the shooting pose of the camera; the face feature The parameters are parameters that characterize the contour of the face.

In a second aspect, an embodiment of the present application provides a device for focusing on a human face, the device comprising:

a first detection module, configured to input the target image into the target parameter detection model to obtain the target focal length output by the target parameter detection model;

a focusing module for focusing on the face based on the target focal length;

The target parameter detection model is obtained by training based on a 2D face image sample, a joint 3D face deformation model and a camera model; the 3D face deformation model is used to obtain a 3D face based on the input facial feature parameters ; The camera model is used to obtain the restored two-dimensional face image based on the input pose parameters and the three-dimensional face;

Wherein, the facial feature parameter and the pose parameter are the parameters output by the initial parameter detection model based on the two-dimensional face image sample; the pose parameter is a parameter representing the shooting pose of the camera; the face feature The parameters are parameters that characterize the contour of the face.

In a third aspect, embodiments of the present application provide an electronic device, the electronic device includes a processor, a memory, and a program or instruction stored on the memory and executable on the processor, the program or instruction being The processor implements the steps of the method according to the first aspect when executed.

In a fourth aspect, an embodiment of the present application provides a readable storage medium, where a program or an instruction is stored on the readable storage medium, and when the program or instruction is executed by a processor, the steps of the method according to the first aspect are implemented .

In a fifth aspect, an embodiment of the present application provides a chip, the chip includes a processor and a communication interface, the communication interface is coupled to the processor, and the processor is configured to run a program or an instruction, and implement the first aspect the method described.

In the embodiment of the present application, the face is focused by the target focal length obtained by extracting the feature of the target image based on the target parameter detection model. Involving background information, the calculation of target focal length based on face information is realized, thereby improving the accuracy of face focusing.

Description of drawings

1 is one of the schematic flow charts of the method for focusing on a human face provided by an embodiment of the present application;

2 is a schematic diagram of a system for monocular 3D face reconstruction provided by an embodiment of the present application;

3 is a schematic structural diagram of a deep convolutional neural network provided by an embodiment of the present application;

4 is the second schematic flowchart of the method for focusing on a face provided by an embodiment of the present application;

5 is a third schematic flowchart of a method for focusing on a face provided by an embodiment of the present application;

6a is one of the schematic diagrams of the focusing principle of the PDAF technology provided by the embodiment of the present application;

6b is the second schematic diagram of the focusing principle of the PDAF technology provided by the embodiment of the present application;

7a is the third schematic diagram of the focusing principle of the PDAF technology provided by the embodiment of the present application;

7b is the fourth schematic diagram of the focusing principle of the PDAF technology provided by the embodiment of the present application;

Fig. 8a is the fifth schematic diagram of the focusing principle of the PDAF technology provided by the embodiment of the present application;

Fig. 8b is the sixth schematic diagram of the focusing principle of the PDAF technology provided by the embodiment of the present application;

9 is a fourth schematic flowchart of a method for focusing on a face provided by an embodiment of the present application;

10 is a schematic flowchart of a photographing method provided by an embodiment of the present application;

11 is a schematic structural diagram of a face focusing device provided by an embodiment of the present application;

12 is a schematic structural diagram of an electronic device provided by an embodiment of the present application;

FIG. 13 is a schematic diagram of hardware of an electronic device provided by an embodiment of the present application.

Detailed ways

The technical solutions in the embodiments of the present application will be clearly described below with reference to the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, but not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by those of ordinary skill in the art fall within the protection scope of this application.

The terms "first", "second" and the like in the description and claims of the present application are used to distinguish similar objects, and are not used to describe a specific order or sequence. It is to be understood that the data so used are interchangeable under appropriate circumstances so that the embodiments of the present application can be practiced in sequences other than those illustrated or described herein, and distinguish between "first", "second", etc. The objects are usually of one type, and the number of objects is not limited. For example, the first object may be one or more than one. In addition, "and/or" in the description and claims indicates at least one of the connected objects, and the character "/" generally indicates that the associated objects are in an "or" relationship.

The following describes in detail the face focusing method, device, and electronic device provided by the embodiments of the present application through specific embodiments and application scenarios with reference to the accompanying drawings.

In the face focusing method provided by the embodiments of the present application, the execution subject of the face focusing method may be an electronic device or a functional module or functional entity capable of implementing the face focusing method in the electronic device, and the electronic device mentioned in the embodiments of the present application Including but not limited to mobile phones, tablet computers, computers, cameras, wearable devices, etc. The following describes the face focusing method provided by the embodiment of the present application by taking an electronic device as an execution subject as an example.

FIG. 1 is one of the schematic flowcharts of the method for focusing on a human face provided by an embodiment of the present application. As shown in FIG. 1 , the method for focusing on a human face includes step 101 and step 102:

Step 101: Input the target image into the target parameter detection model to obtain the target focal length output by the target parameter detection model.

The target parameter detection model is obtained by training based on a two-dimensional face image sample, a joint three-dimensional face deformable model (3D Morphable Model, 3DMM) and a camera model.

The 3D face deformation model is used to obtain a 3D face based on the input facial feature parameters; the camera model is used to obtain a restored 2D face image based on the input pose parameters and the 3D face; the face feature parameters and pose The parameters are the parameters output by the initial parameter detection model based on the two-dimensional face image sample; the pose parameters are the parameters that characterize the shooting posture of the camera; the face feature parameters are the parameters that characterize the contour of the face.

Optionally, the 3D face deformation model and the camera model are included in the monocular 3D face reconstruction technology, and the monocular 3D face reconstruction technology reconstructs its corresponding 3D face from a single 2D face image. In the real world, the face information is three-dimensional, and the two-dimensional face image is the projection of the three-dimensional face to the camera plane. During the projection process, the depth information is lost. The monocular three-dimensional face reconstruction technology can be based on the single image. The two-dimensional face image restores the depth information of the face, and reconstructs the three-dimensional face corresponding to the two-dimensional face image.

Among them, the 3D face deformation model is a 3D face standard model composed of vertices and triangular patches. The positions of all vertices determine the shape of the face, and the colors of all vertices determine the texture of the face; triangular patches describe the vertices. Topological relationship with vertices; the camera model is to project a 3D face into a 2D face image.

2 is a schematic diagram of a system for monocular 3D face reconstruction provided by an embodiment of the present application. As shown in FIG. 2 , the monocular 3D face reconstruction technology mainly includes a 3D face deformation model, a camera model and parameter optimization. The model inputs the 3D face reconstructed based on the 2D face image sample to the camera model, and the camera model projects the 3D face into the restored 2D face image, and then uses the restored 2D face image and the input Two-dimensional face image samples for parameter optimization.

Optionally, after training to obtain the target parameter detection model, the obtained target image is input into the pre-trained target parameter detection model, and the convolution layer of the target parameter detection model extracts two-dimensional image features from the target image, and finally The target focal length is output through the fully connected layer of the target parameter detection model.

It should be noted that in addition to outputting the target focal length, the target parameter detection model can also output target rotation parameters, target translation parameters, target shape parameters, and target texture parameters, and output target focal length, target rotation parameters, target translation parameters, and target shape. The parameters and target texture parameters can be applied to scenes such as face beautification and intelligent face pinching, which are not limited in this application.

Step 102 , focus on the face based on the target focal length.

Optionally, after the target focal length is obtained, the target focal length can be converted into a corresponding target focal image distance, and the face is focused based on the target focal image distance.

In the face focusing method provided by the embodiment of the present application, the face is focused by a target focal length obtained by feature extraction of a target image based on a target parameter detection model, and the target parameter detection model is obtained by training based on face feature parameters and pose parameters. It does not involve background information, and realizes the calculation of target focal length based on face information, thereby improving the accuracy of face focusing.

Optionally, the pose parameters include focal length samples, rotation parameter samples and translation parameter samples; the face feature parameters include shape parameter samples and texture parameter samples.

In one embodiment, the rotation parameter samples include three-dimensional rotation parameters, the translation parameter samples include three-dimensional translation parameters, the shape parameter samples include 20-dimensional shape parameters, and the texture parameter samples include 20-dimensional texture parameters, as long as the three-dimensional rotation is estimated. parameters, three-dimensional translation parameters, 20-dimensional shape parameters, 20-dimensional texture parameters, and the above-mentioned target focal length can realize the process of reconstructing the three-dimensional face and projecting the three-dimensional face into the restored two-dimensional face image, and then The optimization of the model parameters of the initial parameter detection model is realized.

In the face focusing method provided by the embodiment of the present application, the pose parameters include rotation parameter samples, translation parameter samples, and focal length samples, and the face feature parameters include shape parameter samples and texture parameter samples, and these parameters involved can correctly reflect the image The main features of , which can improve the accuracy of the target parameter detection model obtained by the final training.

The following describes the training and optimization process of the target parameter detection model:

In one embodiment, a data set of a target number (eg, 200) three-dimensional faces is collected, and a face shape base vector S _i and a face texture base vector T _i are extracted from the data set by principal component analysis. At this time, the shape and texture of each three-dimensional face can be linearly represented by the face shape basis vector S in the following formula (1) and the face texture basis vector T in the formula (2):

表示平均三维人脸形状，

表示平均三维人脸纹理，α_i表示第i个人脸形状基向量S_i对应的参数，β_i表示第i个人脸纹理基向量T_i对应的参数，i为正整数。in,

represents the average 3D face shape,

represents the average three-dimensional face texture, α _i represents the parameter corresponding to the ith face shape base vector S _i , β _i represents the parameter corresponding to the ith face texture base vector T _i , and i is a positive integer.

From formula (1) and formula (2), it can be concluded that as long as all α _i and β _i are estimated, the three-dimensional face corresponding to the two-dimensional face image can be reconstructed. For example, a 20-dimensional α parameter and a 20-dimensional β parameter are estimated to reconstruct a 3D face corresponding to a 2D face image.

Further, in order to evaluate the quality of the reconstructed three-dimensional face, it is necessary to project the reconstructed three-dimensional face through the camera model to obtain a restored two-dimensional face image.

Assuming that in the three-dimensional space, the coordinates of the jth vertex of the three-dimensional face are (x _j , y _j , z _j ), after the camera model projection, in the two-dimensional space, the coordinates of the jth vertex are (u _j , v _j ), specifically using the following formula (3) for projection.

Among them, dx represents the length unit occupied by a pixel in the x-axis direction, dy represents the length unit occupied by a pixel in the y-axis direction, and u ₀ represents the center pixel coordinate of the two-dimensional face image and the two-dimensional face image origin pixel coordinates. The number of horizontal pixels that differ between the two, v ₀ represents the number of vertical pixels that differ between the center pixel coordinates of the two-dimensional face image and the pixel coordinates of the origin of the two-dimensional face image, Z represents the manually set imaging plane position, and R represents the three-dimensional The rotation parameter, t represents the three-dimensional translation parameter, and f represents the focal length.

From formula (3), it can be concluded that as long as the three-dimensional rotation parameter R, the three-dimensional translation parameter t and the focal length f are estimated, the reconstructed three-dimensional face can be reprojected into a two-dimensional face image.

It can be understood that the parameters that need to be estimated in the three-dimensional face reconstruction process are: the α parameter of the first quantitative dimension, the β parameter of the second quantitative dimension, the three-dimensional rotation parameter R, the three-dimensional translation parameter t and the focal length f. Take the 20-dimensional alpha parameter and the 20-dimensional beta parameter as an example.

In one embodiment, the specific training steps of the target parameter detection model include:

Input the two-dimensional face image sample into the initial parameter detection model to obtain the face feature parameters and pose parameters output by the initial parameter detection model; input the face feature parameters into the three-dimensional face deformation model to obtain the three-dimensional face deformation The 3D face output by the model; the pose parameters and 3D face are input into the camera model to obtain the restored 2D face image output by the camera model; based on the restored 2D face image and 2D face image The sample determines the target parameter detection model.

Wherein, the initial parameter detection model may be a deep convolutional neural network constructed, and a deep convolutional neural network is used to perform model training and estimation of the above-mentioned parameters. FIG. 3 is a schematic structural diagram of a deep convolutional neural network provided by the embodiment of the present application. , as shown in Figure 3, the deep convolutional neural network sequentially includes a convolutional layer, a normalization layer, an activation unit layer, an average pooling layer and a fully connected layer.

The two-dimensional face image samples are input into the deep convolutional neural network, and the two-dimensional image features are extracted from the two-dimensional face image samples by the convolutional layer of the deep convolutional neural network; and the extracted two-dimensional image features are input Then, the normalized two-dimensional image features are input to the activation unit layer for nonlinear transformation; The two-dimensional image features are input to the average pooling layer, and the average pooling layer is used to reduce the size of the input two-dimensional image feature map; finally, the reduced two-dimensional image feature map is input to the fully connected layer, and the fully connected layer is used for All the previous local features are reassembled through the weight matrix, and finally the three-dimensional rotation parameters, three-dimensional translation parameters, focal length, 20-dimensional shape parameters and 20-dimensional texture parameters are output.

After obtaining the 3D rotation parameters, 3D translation parameters, focal length, 20D shape parameters and 20D texture parameters, the 20D shape parameters and 20D texture parameters can be input into the 3D face deformation model for reconstruction 3D face; the reconstructed 3D face can be reprojected into a restored 2D face image based on the 3D rotation parameters, the 3D translation parameters, and the focal length.

It should be noted that when the target image is an image directly collected in the current scene, the two-dimensional face image sample is also an image collected directly; when the target image is an image collected based on the initial focus position determined by PDAF technology Below, the two-dimensional face image sample is also an image collected based on the initial focus position determined by PDAF technology.

It should be noted that, the reconstructed three-dimensional face can also realize augmented reality (AR) and virtual reality (VR), etc., which are not limited in this application.

In the face focusing method provided by the embodiment of the present application, the target parameter detection model is determined based on the face feature parameters and pose parameters output by the initial parameter detection model, and the face feature parameters and pose parameters can correctly reflect the main features of the image, thereby enabling Improve the accuracy of the target parameter detection model.

Further, determining the target parameter detection model based on the restored two-dimensional face image and the two-dimensional face image sample can be implemented in the following ways:

The loss function is determined based on the similarity between the restored two-dimensional face image and the two-dimensional face image samples; the model parameters of the initial parameter detection model are optimized based on the loss function until the convergence conditions are met, and the target parameter detection model is obtained.

In one embodiment, in order to optimize the model parameters of the model, a loss function based on the restored two-dimensional face image and two-dimensional face image samples is established, and the loss function reflects the restored two-dimensional face image. The similarity between the two-dimensional face image sample and the two-dimensional face image sample is used to optimize the model parameters of the model until the convergence conditions are met, and the final optimized target parameter detection model is obtained.

In one embodiment, the loss function Loss _rec constructed by the following formula (4) is used to optimize the model parameters of the deep convolutional neural network.

Loss _rec =∑|I _input -I _rec | (4)

Among them, I _input represents the input two-dimensional face image sample, and I _rec represents the restored two-dimensional face image.

In the face focusing method provided by the embodiments of the present application, the model parameters of the model are optimized through a loss function, and a final optimized target parameter detection model is obtained, which improves the accuracy of the target parameter detection model.

Optionally, FIG. 4 is a second schematic flowchart of a method for focusing on a face provided by an embodiment of the present application. As shown in FIG. 4 , before step 101 in FIG. 1 is executed, the method further includes the following steps:

Step 103: Obtain a target image when a human face is detected and auto-focusing.

Optionally, when starting to focus, it is determined whether there is a human face in the current scene according to the result returned by the face detection algorithm; if there is a return value from the face detection algorithm, it is determined that there is a human face in the current scene, which is a portrait scene; If the face detection algorithm does not have a return value, then it is determined that there is no face in the current scene, and it is not a portrait scene. At this time, the focusing strategy corresponding to other scenes is called to focus.

Further, when it is determined to be a portrait scene, it is detected whether a user operation is received, and the user operation may be a focus-related operation performed by the user on the screen. When a user operation is received, it means that the user wishes to focus on the user. On the imaging object where the operation position is located, call the touch focus strategy to focus at this time.

In the portrait scene, when no user operation is received, it can be determined that the user wants the electronic device to perform automatic focusing, and there is a portrait in the current scene, the user usually wants to be able to focus on the face, so that the part of the face is clearly imaged, so At this time, the face auto-focusing strategy is triggered, that is, the electronic device starts to perform the step of acquiring the target image.

It should be noted that the target image obtained by the electronic device can be the image directly collected in the current scene, or the image collected based on the initial focus position determined by the PDAF technology. PD) The focusing method for calculating the in-focus position by the pixel points, which is not limited in this application.

In the face focusing method provided by the embodiments of the present application, the target image is acquired only when a human face is detected and the focus is automatic, so as to avoid false triggering of the face focusing method.

Optionally, FIG. 5 is a third schematic flowchart of a method for focusing on a face provided by an embodiment of the present application. As shown in FIG. 5 , the implementation of step 103 in FIG. 4 may include the following steps:

Step 1031: Determine the initial focus position based on the target face detection area, the first phase map and the second phase map.

The target face detection area is an auto-focus area, and the first phase map and the second phase map are obtained according to PD pixels on the image sensor.

In one embodiment, after the face auto-focus strategy is triggered, the input required by the face auto-focus strategy needs to be obtained, mainly including the target face detection area output by the face detection algorithm, and the target face detection area can be the target person. The area in the face detection frame, that is, the auto-focus area; and the first phase map and the second phase map obtained according to the PD pixels on the image sensor, the first phase map can be the left phase map, and the second phase map can be the right phase map picture.

When the target face detection area, the first phase map and the second phase map are obtained, the initial focus position is determined based on the PDAF technology, the target face detection area, the first phase map and the second phase map.

Among them, the focusing principle of PDAF technology is: the first phase map and the second phase map are obtained respectively through the PD pixels on the image sensor. When the phase difference between the first phase map and the second phase map is 0, it is just In-focus position; when there is a phase difference between the first phase image and the second phase image, it is in the defocused position. Therefore, the current position can be directly calculated from the phase difference between the first phase image and the second phase image. distance of the location.

Fig. 6a is one of the schematic diagrams of the focusing principle of the PDAF technology provided by the embodiment of the present application, and Fig. 6b is the second schematic diagram of the focusing principle of the PDAF technology provided by the embodiment of the present application. As shown in Figs. 6a and 6b, the first phase There is a phase difference between the image and the second phase image, and they are in the defocused position.

Fig. 7a is the third schematic diagram of the focusing principle of the PDAF technology provided by the embodiment of the present application, and Fig. 7b is the fourth schematic diagram of the focusing principle of the PDAF technology provided by the embodiment of the present application. As shown in Figs. 7a and 7b, the first phase The image coincides with the second phase image and is in focus position.

Fig. 8a is the fifth schematic diagram of the focusing principle of the PDAF technology provided by the embodiment of the present application, and Fig. 8b is the sixth schematic diagram of the focusing principle of the PDAF technology provided by the embodiment of the present application. As shown in Figs. 8a and 8b, the first phase There is also a phase difference between the image and the second phase image, and they are in an out-of-focus position.

Specifically, the calculation process for determining the initial focus position based on the first phase map and the second phase map is as follows:

计算每个shift_k下，移动后的

和第二相位图PD_right在目标人脸检测区域内的相似度SAD_k，具体相似度SAD_k可采用以下公式(5)表示：Determine the initial in-focus position by traversing, assuming that the first phase image is PD _left , the second phase image is PD _right , and the Shift parameter represents the number of pixels moved by the first phase image PD _left , for example, Shift=1 represents the first phase The overall picture PD _left moves one pixel in the direction of the second phase picture PD _right , and Shift=-1 means that the first phase picture PD _left overall moves one pixel in the direction away from the second phase picture PD _right . In practical applications, set The specific value of the Shift parameter, assuming that the Shift parameter is an integer from -16 to 16, you need to traverse the Shift parameter from -16 to 16, and move the first phase map PD _left by shift _k pixel values to obtain

Calculate each shift _k , after the move

The similarity SAD _k with the second phase map PD _right in the target face detection area, and the specific similarity SAD _k can be expressed by the following formula (5):

Among them, ROI represents the target face detection area, that is, the region of interest (Region of Interest, ROI), SAD _k represents the similarity corresponding to the k-th Shift, shift _k represents the k-th Shift, l belongs to the ROI, and l represents The lth pixel within the ROI area.

It can be understood that the smaller the SAD value, the closer the corresponding two phase maps are in the target face detection area, and the closer the focus position is.

和

以及

对应的

对应的

以及

对应的

然后基于

和

拟合一条以下公式(6)所示的二次曲线：Optionally, select the three minimum values from all SAD values obtained

and

as well as

corresponding

corresponding

as well as

corresponding

then based on

and

Fit a quadratic curve shown in the following equation (6):

SAD=a*shift ² +b*shift+c (6)

where a, b, and c are all quadratic curve parameters.

和

代入公式(6)中，得到以下公式(7)；将

和

代入公式(6)中，得到以下公式(8)；将

和

代入公式(6)中，得到以下公式(9)：Will

and

Substituting into formula (6), the following formula (7) is obtained;

and

Substituting into formula (6), the following formula (8) is obtained;

and

Substituting into formula (6), the following formula (9) is obtained:

Based on formula (7), formula (8) and formula (9), the parameters a, b, and c of the quadratic curve can be calculated. It can be known that the SAD value of the position of the symmetry axis of the quadratic curve shown in formula (6) is the smallest , here is the initial focus position, so take the shift value of the position of the symmetry axis of the quadratic curve shown in formula (6) as the phase difference value between the first phase map and the second phase map, the specific phase difference value pd is expressed by the following formula (10):

Step 1032: Capture an image of the target object based on the initial focus position to obtain a target image.

In one embodiment, when the initial focus position is obtained, the phase difference value pd is converted into a motor Code value based on the following formula (11), and the position of the motor is moved based on the motor Code value, thereby completing the initial focus, and the focus at this time is: The position has been aligned near the portrait, so the imaging plane can image the human face more clearly to obtain the target image.

Code=pd*DCC (11)

Among them, Code is the motor stroke, and the Defocus Conversion Coefficient (DCC) represents the coefficient constant for converting the phase difference into the motor Code, which is calibrated by the module factory.

In the face focusing method provided by the embodiments of the present application, the initial focusing position is determined based on the PDAF technology, and focusing is performed again on the basis of the initial focusing position. The two-stage focusing can make the final target focal length more accurate and improve the Accuracy of face focus.

FIG. 9 is a fourth schematic flowchart of a method for focusing on a face provided by an embodiment of the present application. As shown in FIG. 9 , the implementation of step 102 in FIG. 1 may include the following steps:

1021. Based on the target focal length, the lens focal length, and the target image distance, determine the target focus image distance.

Among them, the target image distance is the distance from the imaging plane to the lens.

In one embodiment, the imaging Gaussian formula shown in the following formula (12) is used to calculate the target focusing image distance:

Among them, f represents the target focal length, Z represents the target image distance, which is the manually set imaging plane position, u represents the object distance, f _camera represents the lens focal length, and v represents the target focal length.

In the above formula (12), the object distance u of the face does not change during imaging, the lens focal length f _camera of the camera is a known parameter, Z is also a known parameter, and the target focal length f is calculated using the above step 102, so by formula ( 12) The target focus image distance v can be obtained by solving, that is, the target focus image distance v can be obtained by the following formula (13):

Step 1022: Focus on the human face based on the target focusing image distance.

Optionally, after the target focus image distance is determined, the target focus image distance is determined as the final focus position, and the human face is focused based on the target focus image distance.

Further, the target focus image distance can be converted into a new motor Code based on the following formula (14), that is, the motor corresponding to the far focus image distance v _inf and the far focus image distance v _inf programmed by the module factory in the electronic device. code _inf , near focus image distance v _micr and motor code _micr corresponding to near focus image distance v _micr , convert the target focus image distance _v into the required new motor code _v through linear interpolation, and finally move the motor to The target is focused on the image distance v, and the final focus of the face image is completed.

In the face focusing method provided by the embodiment of the present application, the final target focusing image distance is calculated based on the target focal length, the lens focal length and the target image distance, and the rapid conversion from the target focal length to the target focusing image distance is realized.

The face focusing method provided by the embodiments of the present application, on the basis of PDAF technology, optimizes the initial focusing position based on the target parameter detection model, and solves the interference of background information in the region of interest on the calculation of the focusing position; PDAF technology is difficult to deal with the problem of low-detail texture of faces, which improves the accuracy of face focusing, and brings users a faster and more accurate focusing experience in portrait shooting scenes.

FIG. 10 is a schematic flowchart of a photographing method provided by an embodiment of the present application. As shown in FIG. 10 , the photographing method includes step 1001 , step 1002 , step 1003 , step 1004 , and step 1005 :

Step 1001 , when a human face is detected and auto-focusing, acquire a target image.

Step 1002: Input the target image into the target parameter detection model to obtain the target focal length output by the target parameter detection model.

Among them, the target parameter detection model is trained based on the 2D face image samples, the joint 3D face deformation model and the camera model; the 3D face deformation model is used to obtain the 3D face based on the input facial feature parameters; the camera model uses It is based on the input pose parameters and 3D face to obtain the restored 2D face image; the face feature parameters and pose parameters are the parameters output by the initial parameter detection model based on the 2D face image samples; the pose parameters represent the The parameters of the camera's shooting posture; the face feature parameters are the parameters that characterize the contour of the face.

Step 1003 , focus on the human face based on the target focal length.

Step 1004: Receive a first input from the user.

Wherein, the first input is that the user performs a shooting operation, and the first input can be expressed in at least one of the following ways:

First, the first input can be expressed as a touch operation, including but not limited to a click operation or a pressing operation, that is, clicking on the screen to trigger the shooting of the target object.

In this implementation manner, receiving the user's first input may be expressed as receiving a user's click operation on the photographing interface of the electronic device, or the like.

Second, the first input can be represented as a physical key input.

In this embodiment, the body of the electronic device is provided with a corresponding physical button to receive the first input from the user, which can be expressed as receiving the first input of the user pressing the corresponding physical button; the first input can also be simultaneous pressing The combined operation of multiple physical buttons.

Third, the first input can be expressed as a voice input.

In this implementation manner, the electronic device may receive user voices such as "small V, small V, start shooting", etc., where small V is the wake-up word of the electronic device.

Step 1005: In response to the first input, photograph the target object based on the target focal length to obtain the target photographed image.

In one embodiment, when the first input of the user is detected, based on the analysis of the first input that the user needs to shoot at this time, a shooting instruction is sent to the image sensor, and the image sensor starts imaging when receiving the shooting instruction to obtain the target. Take an image.

It can be understood that in the case where the user's first input is not received, it means that the user is not satisfied with the current focusing effect. At this time, it is necessary to constantly judge whether the user has a new focusing command. If the user clicks the screen, it means that the user It is desirable to focus on the imaging object where the user is operating; or the electronic device re-uses the face detection algorithm to determine whether the user has changed the shooting scene at this time, and if it is determined that the user has changed the shooting scene, switch to the focusing strategy corresponding to the shooting scene to focus.

In the shooting method provided by the embodiment of the present application, the face is focused by using the target focal length obtained by feature extraction of the target image based on the target parameter detection model, and the target parameter detection model is obtained by training based on the face feature parameters and the pose parameters, It does not involve background information, realizes the calculation of the target focal length based on the face information, and improves the accuracy of the face focus; when the target object is photographed based on the target focal length, it can improve the clarity of the captured image, which means that the quality of the captured image.

It should be noted that, in the face focusing method provided by the embodiments of the present application, the execution subject may be a face focusing device, or a control module in the face focusing device for executing the face focusing method. In this embodiment of the present application, the method for performing human face focusing by a face focusing device is taken as an example to describe the device for human face focusing provided by the embodiments of the present application.

The embodiment of the present application also provides a face focusing device. FIG. 11 is a schematic structural diagram of a face focusing device provided by an embodiment of the present application. As shown in FIG. 11 , the device includes a first detection module 1101 and a focusing module 1102; wherein,

The first detection module 1101 is used to input the target image into the target parameter detection model to obtain the target focal length output by the target parameter detection model;

A focusing module 1102, configured to focus on the human face based on the target focal length;

Among them, the target parameter detection model is trained based on the 2D face image samples, the joint 3D face deformation model and the camera model; the 3D face deformation model is used to obtain the 3D face based on the input facial feature parameters; the camera model uses Obtain the restored 2D face image based on the input pose parameters and 3D face;

Among them, the face feature parameters and pose parameters are the parameters output by the initial parameter detection model based on the two-dimensional face image sample; the pose parameters are the parameters that characterize the camera's shooting posture; the face feature parameters are the parameters that characterize the face contour.

The face focusing device provided by the embodiment of the present application focuses on the face through the target focal length obtained by extracting the target image based on the target parameter detection model. The target parameter detection model is obtained by training based on the face feature parameters and the pose parameters. It does not involve background information, and realizes the calculation of target focal length based on face information, thereby improving the accuracy of face focusing.

Optionally, the device further includes:

The second detection module is used to input the two-dimensional face image sample into the initial parameter detection model, and obtain the facial feature parameters and pose parameters output by the initial parameter detection model;

The deformation module is used to input the facial feature parameters into the 3D face deformation model to obtain the 3D face output by the 3D face deformation model;

The restoration module is used to input the pose parameters and the three-dimensional face into the camera model, and obtain the restored two-dimensional face image output by the camera model;

The determining module is used for determining the target parameter detection model based on the restored two-dimensional face image and the two-dimensional face image samples.

Optionally, the determining module is also used to:

Determine the loss function based on the similarity between the restored two-dimensional face image and the two-dimensional face image samples;

Based on the loss function, the model parameters of the initial parameter detection model are optimized until the convergence conditions are met, and the target parameter detection model is obtained.

Optionally, the pose parameters include focal length samples, rotation parameter samples and translation parameter samples; the face feature parameters include shape parameter samples and texture parameter samples.

Optionally, the device further includes:

The acquisition module is used to acquire the target image when the face is detected and auto-focusing.

Optionally, the acquisition module is also used to:

The initial in-focus position is determined based on the target face detection area, the first phase map and the second phase map, the target face detection area is the auto-focus area, and the first phase map and the second phase map are obtained according to the PD pixels on the image sensor of;

Based on the initial in-focus position, image acquisition of the target object is performed to obtain the target image.

Optionally, the focusing module 1102 is also used for:

Based on the target focal length, the lens focal length and the target image distance, the target focal image distance is determined; wherein, the target image distance is the distance from the imaging plane to the lens;

Focuses on faces based on the target focusing distance.

The face focusing device in this embodiment of the present application may be a device, or may be a component, an integrated circuit, or a chip in a terminal. The apparatus may be a mobile electronic device or a non-mobile electronic device. Exemplarily, the mobile electronic device may be a mobile phone, a tablet computer, a notebook computer, a palmtop computer, an in-vehicle electronic device, a wearable device, an ultra-mobile personal computer (UMPC), a netbook, or a personal digital assistant (personal digital assistant). , PDA), etc., the non-mobile electronic device may be a server, a network attached storage (NAS), a personal computer (personal computer, PC), a television (television, TV), a teller machine or a self-service machine, etc. The embodiments of the present application There is no specific limitation.

The face focusing device in the embodiment of the present application may be a device with an operating system. The operating system may be an Android (Android) operating system, an IOS operating system, or other possible operating systems, which are not specifically limited in the embodiments of the present application.

The face focusing device provided in the embodiments of the present application can implement the various processes implemented by the method embodiments in FIG. 1 to FIG. 10 , and can achieve the same technical effect. To avoid repetition, details are not described here.

Optionally, as shown in FIG. 12 , an embodiment of the present application further provides an electronic device 1200, including a processor 1201, a memory 1202, a program or instruction stored in the memory 1202 and executable on the processor 1201, When the program or instruction is executed by the processor 1201, each process of the above-mentioned embodiment of the method for focusing on a face can be achieved, and the same technical effect can be achieved. In order to avoid repetition, details are not repeated here.

It should be noted that the electronic devices in the embodiments of the present application include the aforementioned mobile electronic devices and non-mobile electronic devices.

FIG. 13 is a schematic diagram of a hardware structure of an electronic device implementing an embodiment of the present application.

The electronic device 1300 includes but is not limited to: a radio frequency unit 1301, a network module 1302, an audio output unit 1303, an input unit 1304, a sensor 1305, a display unit 1306, a user input unit 1307, an interface unit 1308, a memory 1309, a processor 1310 and other components .

Those skilled in the art can understand that the electronic device 1300 may also include a power source (such as a battery) for supplying power to various components, and the power source may be logically connected to the processor 1310 through a power management system, so as to manage charging, discharging, and power management through the power management system. consumption management and other functions. The structure of the electronic device shown in FIG. 13 does not constitute a limitation on the electronic device. The electronic device may include more or less components than the one shown, or combine some components, or arrange different components, which will not be repeated here. .

The processor 1310 is configured to input the target image into the target parameter detection model to obtain the target focal length output by the target parameter detection model;

The processor 1310 is further configured to focus on the human face based on the target focal length;

Among them, the target parameter detection model is trained based on the 2D face image samples, the joint 3D face deformation model and the camera model; the 3D face deformation model is used to obtain the 3D face based on the input facial feature parameters; the camera model uses Obtain the restored 2D face image based on the input pose parameters and 3D face;

Among them, the face feature parameters and pose parameters are the parameters output by the initial parameter detection model based on the two-dimensional face image sample; the pose parameters are the parameters that characterize the camera's shooting posture; the face feature parameters are the parameters that characterize the face contour.

The electronic device provided by the embodiment of the present application focuses on the human face by using the target focal length obtained by extracting the feature of the target image based on the target parameter detection model, and the target parameter detection model is obtained by training based on the facial feature parameters and the pose parameters, The background information is not involved, and the calculation of the target focal length based on the face information is realized, thereby improving the accuracy of the face focusing.

Optionally, the processor 1310 is further configured to input the two-dimensional face image sample into the initial parameter detection model to obtain the facial feature parameters and pose parameters output by the initial parameter detection model;

Input the facial feature parameters into the 3D face deformation model to obtain the 3D face output by the 3D face deformation model;

Input the pose parameters and the three-dimensional face into the camera model, and obtain the restored two-dimensional face image output by the camera model;

The target parameter detection model is determined based on the restored 2D face image and the 2D face image samples.

In the electronic device provided by the embodiment of the present application, the target parameter detection model is determined based on the facial feature parameters and pose parameters output by the initial parameter detection model, and the facial feature parameters and pose parameters can correctly reflect the main features of the image, thereby improving the target Parameter detection model accuracy.

Optionally, the processor 1310 is further configured to determine a loss function based on the similarity between the restored two-dimensional face image and the two-dimensional face image sample;

Based on the loss function, the model parameters of the initial parameter detection model are optimized until the convergence conditions are met, and the target parameter detection model is obtained.

In the electronic device provided by the embodiment of the present application, the model parameters of the model are optimized through the loss function, and the final optimized target parameter detection model is obtained, which improves the accuracy of the target parameter detection model.

Optionally, the pose parameters include focal length samples, rotation parameter samples and translation parameter samples; the face feature parameters include shape parameter samples and texture parameter samples.

In the electronic device provided by the embodiment of the present application, the pose parameters include rotation parameter samples, translation parameter samples, and focal length samples, and the face feature parameters include shape parameter samples and texture parameter samples, and these involved parameters can correctly reflect the main characteristics of the image features, so as to improve the accuracy of the target parameter detection model obtained by the final training.

Optionally, the processor 1310 is further configured to acquire the target image in the case of detecting a human face and automatically focusing.

The electronic device provided by the embodiment of the present application acquires a target image only when a human face is detected and automatically focuses, so as to avoid false triggering of the human face focusing method.

Optionally, the processor 1310 is further configured to determine the initial focus position based on the target face detection area, the first phase map and the second phase map, where the target face detection area is an auto-focus area, the first phase map and the second phase map are The phase map is obtained from the PD pixels on the image sensor;

Based on the initial in-focus position, image acquisition of the target object is performed to obtain the target image.

The electronic device provided by the embodiment of the present application determines the initial focus position based on the PDAF technology, and performs focus search again on the basis of the initial focus position. Accuracy of face focus.

Optionally, the processor 1310 is further configured to determine the target focus image distance based on the target focal length, the lens focal length and the target image distance; wherein, the target image distance is the distance from the imaging plane to the lens;

Focuses on faces based on the target focusing distance.

The electronic device provided by the embodiment of the present application calculates the final target focal length based on the target focal length, the lens focal length and the target image distance, and realizes the rapid conversion from the target focal length to the target focal length.

It should be understood that, in this embodiment of the present application, the input unit 1304 may include a graphics processing unit (Graphics Processing Unit, GPU) 13041 and a microphone 13042. camera) to process the image data of still pictures or videos. The display unit 1306 may include a display panel 13061, which may be configured in the form of a liquid crystal display, an organic light emitting diode, or the like. The user input unit 1307 includes a touch panel 13071 and other input devices 13072 . The touch panel 13071 is also called a touch screen. The touch panel 13071 may include two parts, a touch detection device and a touch controller. Other input devices 13072 may include, but are not limited to, physical keyboards, function keys (such as volume control keys, switch keys, etc.), trackballs, mice, and joysticks, which will not be described herein again. Memory 1309 may be used to store software programs as well as various data, including but not limited to application programs and operating systems. The processor 1310 may integrate an application processor and a modem processor, wherein the application processor mainly processes the operating system, user interface, and application programs, and the like, and the modem processor mainly processes wireless communication. It can be understood that, the above-mentioned modulation and demodulation processor may not be integrated into the processor 1310.

Embodiments of the present application further provide a readable storage medium, where a program or an instruction is stored on the readable storage medium, and when the program or instruction is executed by a processor, each process of the above-mentioned embodiment of the method for focusing on a face can be realized, and can achieve The same technical effect, in order to avoid repetition, will not be repeated here.

Wherein, the processor is the processor in the electronic device described in the foregoing embodiments. The readable storage medium includes a computer-readable storage medium, such as a computer read-only memory (Read-Only Memory, ROM), a random access memory (Random Access Memory, RAM), a magnetic disk or an optical disk, and the like.

An embodiment of the present application further provides a chip, where the chip includes a processor and a communication interface, the communication interface is coupled to the processor, and the processor is used for running a program or an instruction to implement the above-mentioned embodiment of the face focusing method and can achieve the same technical effect, in order to avoid repetition, it will not be repeated here.

It should be understood that the chip mentioned in the embodiments of the present application may also be referred to as a system-on-chip, a system-on-chip, a system-on-a-chip, or a system-on-a-chip, or the like.

It should be noted that, herein, the terms "comprising", "comprising" or any other variation thereof are intended to encompass non-exclusive inclusion, such that a process, method, article or device comprising a series of elements includes not only those elements, It also includes other elements not expressly listed or inherent to such a process, method, article or apparatus. Without further limitation, an element qualified by the phrase "comprising a..." does not preclude the presence of additional identical elements in a process, method, article or apparatus that includes the element. Furthermore, it should be noted that the scope of the methods and apparatus in the embodiments of the present application is not limited to performing the functions in the order shown or discussed, but may also include performing the functions in a substantially simultaneous manner or in the reverse order depending on the functions involved. To perform functions, for example, the described methods may be performed in an order different from that described, and various steps may also be added, omitted, or combined. Additionally, features described with reference to some examples may be combined in other examples.

From the description of the above embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus a necessary general hardware platform, and of course hardware can also be used, but in many cases the former is better implementation. Based on this understanding, the technical solutions of the present application can be embodied in the form of computer software products that are essentially or contribute to the prior art, and the computer software products are stored in a storage medium (such as ROM/RAM, magnetic disk , CD), including several instructions to make a terminal (which may be a mobile phone, a computer, a server, or a network device, etc.) execute the methods described in the various embodiments of the present application.

The embodiments of the present application have been described above in conjunction with the accompanying drawings, but the present application is not limited to the above-mentioned specific embodiments, which are merely illustrative rather than restrictive. Under the inspiration of this application, without departing from the scope of protection of the purpose of this application and the claims, many forms can be made, which all fall within the protection of this application.

Claims (15)

1. A face focusing method is characterized by comprising the following steps:

inputting a target image into a target parameter detection model to obtain a target focal length output by the target parameter detection model;

focusing the human face based on the target focal length;

the target parameter detection model is obtained based on two-dimensional face image samples, a combined three-dimensional face deformation model and a camera model; the three-dimensional face deformation model is used for obtaining a three-dimensional face based on the input face characteristic parameters; the camera model is used for obtaining a restored two-dimensional face image based on the input pose parameters and the three-dimensional face;

the face feature parameters and the pose parameters are parameters output by an initial parameter detection model based on the two-dimensional face image samples; the pose parameters are parameters representing shooting postures of the camera; the face characteristic parameters are parameters for representing the face contour.

2. The method of claim 1, wherein before the target image is input into the target parameter detection model and the target focal length output by the target parameter detection model is obtained, the method further comprises:

inputting the two-dimensional face image sample into the initial parameter detection model to obtain the face characteristic parameters and the pose parameters output by the initial parameter detection model;

inputting the face characteristic parameters into the three-dimensional face deformation model to obtain the three-dimensional face output by the three-dimensional face deformation model;

inputting the pose parameters and the three-dimensional face into the camera model to obtain the restored two-dimensional face image output by the camera model;

and determining the target parameter detection model based on the restored two-dimensional face image and the two-dimensional face image sample.

3. The method of claim 2, wherein the determining the target parameter detection model based on the restored two-dimensional face image and the two-dimensional face image sample comprises:

determining a loss function based on the similarity of the restored two-dimensional face image and the two-dimensional face image sample;

and optimizing the model parameters of the initial parameter detection model based on the loss function until a convergence condition is met to obtain the target parameter detection model.

4. The face focusing method according to claim 1, wherein the pose parameters comprise a focal length sample, a rotation parameter sample and a translation parameter sample; the face feature parameters comprise shape parameter samples and texture parameter samples.

5. The method of claim 1, wherein before the inputting the target image into the target parameter detection model and obtaining the target focal length output by the target parameter detection model, the method further comprises:

and under the condition that the human face is detected and the automatic focusing is carried out, acquiring the target image.

6. The face focusing method of claim 5, wherein the acquiring the target image comprises:

determining an initial focusing position based on a target face detection area, a first phase diagram and a second phase diagram, wherein the target face detection area is an automatic focusing area, and the first phase diagram and the second phase diagram are obtained according to PD pixels on an image sensor;

and acquiring an image of the target object based on the initial focusing position to obtain the target image.

7. The method according to any one of claims 1 to 6, wherein the focusing the human face based on the target focal distance comprises:

determining a target focusing image distance based on the target focal length, the lens focal length and the target image distance; the target image distance is the distance from an imaging plane to a lens;

and focusing the human face based on the target focusing image distance.

8. A face focusing device, comprising:

the first detection module is used for inputting a target image into a target parameter detection model to obtain a target focal length output by the target parameter detection model;

the focusing module is used for focusing the human face based on the target focal length;

the target parameter detection model is obtained based on two-dimensional face image samples, a combined three-dimensional face deformation model and a camera model; the three-dimensional face deformation model is used for obtaining a three-dimensional face based on the input face characteristic parameters; the camera model is used for obtaining a restored two-dimensional face image based on the input pose parameters and the three-dimensional face;

the face feature parameters and the pose parameters are parameters output by an initial parameter detection model based on the two-dimensional face image samples; the pose parameters are parameters representing shooting postures of the camera; the face characteristic parameters are parameters for representing the face contour.

9. The device of claim 8, further comprising:

the second detection module is used for inputting the two-dimensional face image sample into the initial parameter detection model to obtain the face characteristic parameters and the pose parameters output by the initial parameter detection model;

the deformation module is used for inputting the face characteristic parameters into the three-dimensional face deformation model to obtain the three-dimensional face output by the three-dimensional face deformation model;

the restoration module is used for inputting the pose parameters and the three-dimensional face into the camera model to obtain a restored two-dimensional face image output by the camera model;

and the determining module is used for determining the target parameter detection model based on the restored two-dimensional face image and the two-dimensional face image sample.

10. The face focusing device of claim 9, wherein the determining module is further configured to:

determining a loss function based on the similarity of the restored two-dimensional face image and the two-dimensional face image sample;

and optimizing the model parameters of the initial parameter detection model based on the loss function until a convergence condition is met to obtain the target parameter detection model.

11. The face focusing device of claim 8, wherein the pose parameters comprise a focus parameter sample, a rotation parameter sample, and a translation parameter sample; the face feature parameters comprise shape parameter samples and texture parameter samples.

12. The device of claim 8, further comprising:

and the acquisition module is used for acquiring the target image under the conditions of face detection and automatic focusing.

13. The device of claim 12, wherein the obtaining module is further configured to:

determining an initial focusing position based on a target face detection area, a first phase diagram and a second phase diagram, wherein the target face detection area is an automatic focusing area, and the first phase diagram and the second phase diagram are obtained according to PD pixels on an image sensor;

and acquiring an image of the target object based on the initial focusing position to obtain the target image.

14. The device as claimed in any one of claims 8-13, wherein the focusing module is further configured to:

determining a target focusing image distance based on the target focal length, the lens focal length and the target image distance; the target image distance is the distance from an imaging plane to a lens;

and focusing the human face based on the target focusing image distance.

15. An electronic device comprising a processor, a memory, and a program or instructions stored on the memory and executable on the processor, the program or instructions when executed by the processor implementing the steps of the face focusing method of any one of claims 1-7.

Images