KR102502686B1 - Control method of electronic apparatus, server and system for non-face-to-face identification using speaker authentication and facial authentication - Google Patents

Abstract

Obtaining, by the electronic device, a first user image generated by photographing a user to be authenticated; obtaining, by the server, a face matching value between a face represented by a user face image of a second user image of a user whose pre-authentication is completed and a face represented by a user face image in the first user image; obtaining, by the server, a facial liveness value of a user face image in the first user image; obtaining, by the electronic device, first user voice data generated by recording a voice of a user who is the target of the user authentication; obtaining, by the server, a speaker matching value between a speaker of the second user voice data and a speaker of the first user voice data of the user for whom the pre-authentication has been completed; obtaining, by the server, a voice liveness value of the first user voice data; and performing, by the server, user authentication based on the facial matching value, the facial liveness value, the speaker matching value, and the voice liveness value.

Description

Control method of a system including an electronic device and a server that performs non-face-to-face authentication using speaker authentication and face authentication AUTHENTICATION}

The present invention relates to a control method for a system including an electronic device and a server that performs non-face-to-face authentication using speaker authentication and face authentication, and more particularly, to a user based on a speaker authentication result and a face authentication result. It relates to a control method of a system including an electronic device and a server that perform authentication for authentication.

Biometric authentication refers to a technology for identifying and authenticating a user based on body information that cannot be imitated by others. Among various biometric authentication technologies, research on voice recognition technology is currently being actively conducted. Voice recognition technology is largely divided into 'voice recognition' and 'speaker authentication'. Speech recognition is to understand the 'content' spoken by an unspecified number of people regardless of who is speaking, whereas speaker authentication is to distinguish 'who' said the story.

As an example of speaker authentication technology, there is a 'voice authentication service'. If the subject can be accurately and quickly identified as 'who' by voice alone, existing methods required for personal authentication in various fields, for example, by reducing cumbersome steps such as password input after login and public certificate authentication, etc. It can provide user convenience.

At this time, the speaker authentication technology registers the user's voice for the first time, and then compares the voice spoken by the user with the registered voice every time an authentication request is made, and performs authentication based on whether or not they match. When a user registers a voice, feature points may be extracted from voice data in units of several seconds (eg, 10 sec). Characteristic points may be extracted in various types such as intonation and speaking speed, and users may be identified by a combination of these characteristics points.

However, when a registered user registers or authenticates a voice, a third party located nearby may record the registered user's voice without permission and attempt speaker authentication with the recorded file, so the security of the makeup authentication technology may be problematic. can If this situation occurs, enormous damage will occur to the user, and the reliability of speaker authentication will inevitably be lowered. That is, the effectiveness of the speaker authentication technology is reduced, and forgery or alteration of voice authentication data may frequently occur.

Korean Patent Publication No. 10-2022-0081476

An object of the present invention is to provide a method for controlling a system including an electronic device and a server that authenticates a user based on a speaker authentication result and a face authentication result.

The objects of the present invention are not limited to the above-mentioned objects, and other objects and advantages of the present invention not mentioned above can be understood by the following description and will be more clearly understood by the embodiments of the present invention. It will also be readily apparent that the objects and advantages of the present invention may be realized by means of the instrumentalities and combinations indicated in the claims.

In the control method of a system including an electronic device and a server for performing non-face-to-face authentication using speaker authentication and face authentication according to an aspect of the present invention for solving the above problems, the electronic device performs user authentication. acquiring a first user image generated by photographing a user who is a target of the; obtaining, by the server, a face matching value between a face represented by a user face image of a second user image of a user whose pre-authentication is completed and a face represented by a user face image in the first user image; obtaining, by the server, a facial liveness value of a user face image in the first user image; obtaining, by the electronic device, first user voice data generated by recording a voice of a user who is the target of the user authentication; obtaining, by the server, a speaker matching value between a speaker of the second user voice data and a speaker of the first user voice data of the user for whom the pre-authentication has been completed; obtaining, by the server, a voice liveness value of the first user voice data; and performing, by the server, user authentication based on the facial matching value, the facial liveness value, the speaker matching value, and the voice liveness value.

Preferably, the performing of the user authentication may include determining, by the server, whether a face represented by a user face image in the first user image is the face of the user based on the face matching value and the face liveness value. performing facial authentication; and performing, by the server, speaker authentication for determining whether a speaker of the first user's voice data is the user based on the speaker matching value and the voice liveness value.

Preferably, the performing of the face authentication may include: determining, by the server, whether the face match value exceeds a reference face match value; determining, by the server, whether the facial liveness value exceeds a reference facial liveness value; and if the face match value exceeds the reference face match value and the face liveness value exceeds the reference face liveness value, the server determines that the face represented by the user's face image in the first user image is the user's face. Determining that the face of the; may include.

Preferably, the performing of the speaker authentication may include: determining, by the server, whether the speaker matching value exceeds a reference speaker matching value; determining, by the server, whether the voice liveness value exceeds a reference voice liveness value; and determining, by the server, that the speaker of the first user's voice data is the user when the speaker matching value exceeds the reference speaker matching value and the voice liveness value exceeds the reference voice liveness value. step; may be included.

Preferably, in the performing of the user authentication, the server determines, as a result of the face authentication, that a face represented by a user's face image in the first user image is the face of the user, and as a result of the speaker authentication, the first user If it is determined that the speaker of the first user's voice data is the user, the user may be authenticated.

The control method of the system may include extracting, by the electronic device or the server, ID information by recognizing text from the ID image; obtaining, by the electronic device, location information of a user attempting the user authentication; and determining, by the server, country information corresponding to the location of the address indicated by the address information among the ID information, and determining whether the location of the user indicated by the location information is included in the country indicated by the country information. can include

In the method for controlling a server that performs non-face-to-face identity authentication using speaker authentication and face authentication according to an aspect of the present invention for solving the above problems, the server controls a second user image of a user whose pre-authentication has been completed. obtaining a facial matching value between a face represented by the user's face image and a face represented by a user's face image in a first user image generated by photographing a user subject to user authentication; obtaining, by the server, a facial liveness value of a user face image in the first user image; obtaining, by the server, a speaker match value between a speaker of the second user voice data of the user whose pre-authentication is completed and a speaker of the first user voice data generated by recording the voice of the user subject to the user authentication; obtaining, by the server, a voice liveness value of the first user voice data; and performing, by the server, user authentication based on the facial matching value, the facial liveness value, the speaker matching value, and the voice liveness value.

Other specific details of the invention are included in the detailed description and drawings.

A control method of a system including an electronic device and a server according to the present invention performs authentication on a user based on a speaker authentication result and a face authentication result, thereby enabling an authentication process with improved authentication accuracy.

The effects of the present invention are not limited to the effects mentioned above, and other effects not mentioned will be clearly understood by those skilled in the art from the description below.

1 is a configuration diagram of a system according to an embodiment of the present disclosure.
2 is a configuration diagram of an electronic device and a server included in a system according to an embodiment of the present disclosure.
3 is a flowchart of a user authentication process according to an embodiment of the present disclosure.
4 is an exemplary view of how an electronic device acquires an ID image according to an embodiment of the present disclosure.
5 and 6 are exemplary diagrams of obtaining a user image by an electronic device according to an embodiment of the present disclosure.
7 is a diagram for explaining a process in which a server obtains a face liveness value for a face image using an artificial intelligence model according to an embodiment of the present disclosure.
8 is a diagram for explaining a process in which a server obtains a facial matching value between faces represented by two facial images by using an artificial intelligence model according to an embodiment of the present disclosure.
9 and 10 are exemplary diagrams of obtaining user voice data by an electronic device according to an embodiment of the present disclosure.
11 is a diagram for explaining a process in which a server obtains a voice liveness value for user voice data using an artificial intelligence model according to an embodiment of the present disclosure.
12 is a diagram for explaining a process in which a server obtains a speaker matching value between speakers of each of two voices using an artificial intelligence model according to an embodiment of the present disclosure.

Advantages and features of the present invention, and methods of achieving them, will become clear with reference to the detailed description of the following embodiments taken in conjunction with the accompanying drawings. However, the present invention is not limited to the embodiments disclosed below, but may be implemented in various different forms, only these embodiments are intended to complete the disclosure of the present invention, and are common in the art to which the present invention belongs. It is provided to fully inform the person skilled in the art of the scope of the invention, and the invention is only defined by the scope of the claims.

Terminology used herein is for describing the embodiments and is not intended to limit the present invention. In this specification, singular forms also include plural forms unless specifically stated otherwise in a phrase. As used herein, "comprises" and/or "comprising" does not exclude the presence or addition of one or more other elements other than the recited elements. Like reference numerals throughout the specification refer to like elements, and “and/or” includes each and every combination of one or more of the recited elements. Although "first", "second", etc. are used to describe various components, these components are not limited by these terms, of course. These terms are only used to distinguish one component from another. Accordingly, it goes without saying that the first element mentioned below may also be the second element within the technical spirit of the present invention.

Unless otherwise defined, all terms (including technical and scientific terms) used in this specification may be used with meanings commonly understood by those skilled in the art to which the present invention belongs. In addition, terms defined in commonly used dictionaries are not interpreted ideally or excessively unless explicitly specifically defined.

The term "unit" or "module" used in the specification means a hardware component such as software, FPGA or ASIC, and "unit" or "module" performs certain roles. However, "unit" or "module" is not meant to be limited to software or hardware. A “unit” or “module” may be configured to reside in an addressable storage medium and may be configured to reproduce one or more processors. Thus, as an example, a “unit” or “module” may refer to components such as software components, object-oriented software components, class components, and task components, processes, functions, properties, procedures, subroutines, segments of program code, drivers, firmware, microcode, circuitry, data, databases, data structures, tables, arrays and variables. Functions provided within components and "units" or "modules" may be combined into smaller numbers of components and "units" or "modules" or may be combined into additional components and "units" or "modules". can be further separated.

The spatially relative terms "below", "beneath", "lower", "above", "upper", etc. It can be used to easily describe a component's correlation with other components. Spatially relative terms should be understood as including different orientations of elements in use or operation in addition to the orientations shown in the drawings. For example, if you flip a component that is shown in a drawing, a component described as "below" or "beneath" another component will be placed "above" the other component. can Thus, the exemplary term “below” may include directions of both below and above. Components may also be oriented in other orientations, and thus spatially relative terms may be interpreted according to orientation.

In this specification, a computer means any kind of hardware device including at least one processor, and may be understood as encompassing a software configuration operating in a corresponding hardware device according to an embodiment. For example, a computer may be understood as including a smartphone, a tablet PC, a desktop computer, a laptop computer, and user clients and applications running on each device, but is not limited thereto.

Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings.

1 is a configuration diagram of a system according to an embodiment of the present disclosure.

As shown in FIG. 1 , a system according to an aspect of the present invention may include an electronic device 100 and a server 200 .

The electronic device 100 and the server 200 transmit and receive information on generated values, information on judgment results, data, and signals, and perform user authentication for users.

Here, user authentication may be a process of determining whether a user attempting to log in to an account is a user permitted to log in to the account.

The electronic device 100 may be implemented as a smart phone, tablet PC, TV, console device, set-top box, or other control device.

The server 200 performs user authentication on the user based on information on the generated value received from the electronic device 100, information on the determination result, data, and signals, and the user authentication result is transmitted to the electronic device 100. can be sent to

2 is a configuration diagram of an electronic device and a server included in a system according to an embodiment of the present disclosure.

Referring to FIG. 2 , the electronic device 100 may include a memory 110, a camera 120, a microphone 130, a display 140, a processor 150, and a communication unit 160.

The memory 110 may store various programs and data necessary for the operation of the electronic device 100 . The memory 110 may be implemented as a non-volatile memory 110, a volatile memory 110, a flash-memory, a hard disk drive (HDD), or a solid state drive (SSD).

The processor 150 may control overall operations of the electronic device 100 using various programs stored in the memory 110 . The processor 150 may include a RAM, a ROM, a graphic processing unit, a main CPU, first through n interfaces, and a bus. At this time, the RAM, ROM, graphic processing unit, main CPU, first to n interfaces, etc. may be connected to each other through a bus.

RAM stores O/S and application programs. Specifically, when the electronic device 100 is booted, O/S is stored in RAM, and various application data selected by the user may be stored in RAM.

The ROM stores instruction sets for system booting. When a turn-on command is input and power is supplied, the main CPU copies the O/S stored in the memory 110 to the RAM according to the command stored in the ROM, and executes the O/S to boot the system. When booting is completed, the main CPU copies various application programs stored in the memory 110 to RAM, and executes the application programs copied to RAM to perform various operations.

The graphic processing unit uses a calculation unit (not shown) and a rendering unit (not shown) to create a screen including various objects such as items, images, and text. Here, the calculation unit may be configured to calculate attribute values such as coordinate values, shape, size, color, etc. of each object to be displayed according to the layout of the screen by using a control command received from the input unit. And, the rendering unit may be configured to generate screens of various layouts including objects based on the attribute values calculated by the calculation unit. The screen created by the rendering unit may be displayed within the display area of the display 140 .

The main CPU accesses the memory 110 and performs booting using the OS stored in the memory 110 . And, the main CPU performs various operations using various programs, contents, data, etc. stored in the memory 110 .

The first through n interfaces are connected to the various components described above. One of the first through n interfaces may be a network interface connected to an external device through a network.

Meanwhile, the processor 150 may include one or more cores (not shown) and a graphic processing unit (not shown) and/or a connection path (eg, a bus) for transmitting and receiving signals to and from other components. can

Processor 150 according to one embodiment performs a method described in connection with the present invention by executing one or more instructions stored in memory 110 .

For example, the processor 150 acquires new training data by executing one or more instructions stored in the memory 110, performs a test on the acquired new training data using the learned model, and the test result, First training data in which the labeled information is obtained with accuracy equal to or higher than a predetermined first reference value is extracted, the extracted first training data is deleted from the new training data, and learning is performed using the new training data from which the extracted training data is deleted. The model can be re-trained.

Meanwhile, the processor 150 includes RAM (Random Access Memory, not shown) and ROM (Read-Only Memory) that temporarily and/or permanently store signals (or data) processed in the processor 150. , not shown) may be further included. In addition, the processor 150 may be implemented in the form of a system on chip (SoC) including at least one of a graphic processing unit, RAM, and ROM.

The memory 110 may store programs (one or more instructions) for processing and control of the processor 150 . Programs stored in the memory 110 may be divided into a plurality of modules according to functions.

The camera 120 is configured to photograph at least one subject, and may generate a user image including the face of the user by photographing the user and generate an ID image by photographing an identification card.

The microphone 130 is a component for recording sound, and may generate user voice data by recording a voice spoken by a user.

The display 140 is a configuration for visually outputting various information, and may be implemented as a Liquid Crystal Display (LCD), Plasma Display Panel (PDP), Organic Light Emitting Diodes (OLED), Transparent OLED (TOLED), Micro LED, etc. However, it is not limited thereto, and other types of conventionally known displays may be included.

As an example, the display 140 may be implemented in the form of a touch screen capable of detecting a user's touch manipulation, or may be implemented as a flexible display capable of being folded or bent.

The communication unit 160 may communicate with the server 200 . In particular, the communication unit 160 may include various communication chips such as a Wi-Fi chip, a Bluetooth chip, a wireless communication chip, an NFC chip, and a low energy Bluetooth chip (BLE chip). At this time, the Wi-Fi chip, the Bluetooth chip, and the NFC chip perform communication in a LAN method, a WiFi method, a Bluetooth method, and an NFC method, respectively. In the case of using a Wi-Fi chip or a Bluetooth chip, various connection information such as an SSID and a session key is first transmitted and received, and various information can be transmitted and received after communication is connected using this. The wireless communication chip refers to a chip that performs communication according to various communication standards such as IEEE, ZigBee, 3rd Generation (3G), 3rd Generation Partnership Project (3GPP), Long Term Evolution (LTE), and 5th Generation (5G).

Referring to FIG. 2 , the server 200 may include a memory 210 , a processor 220 and a communication unit 230 .

The memory 210 may store various programs and data necessary for the operation of the server 200 . The memory 210 may be implemented as a non-volatile memory 210, a volatile memory 210, a flash-memory, a hard disk drive (HDD), or a solid state drive (SSD).

The processor 220 may control overall operations of the server 200 using various programs stored in the memory 210 . The processor 220 may include a RAM, a ROM, a graphic processing unit, a main CPU, first through n interfaces, and a bus. At this time, the RAM, ROM, graphic processing unit, main CPU, first to n interfaces, etc. may be connected to each other through a bus.

RAM stores O/S and application programs. Specifically, when the server 200 is booted, O/S is stored in RAM, and various application data selected by the user may be stored in RAM.

The ROM stores instruction sets for system booting. When a turn-on command is input and power is supplied, the main CPU copies the O/S stored in the memory 210 to the RAM according to the command stored in the ROM, and executes the O/S to boot the system. When booting is completed, the main CPU copies various application programs stored in the memory 210 to RAM, and executes the application programs copied to RAM to perform various operations.

The main CPU accesses the memory 210 and performs booting using the OS stored in the memory 210 . In addition, the main CPU performs various operations using various programs, contents, data, etc. stored in the memory 210 .

The first through n interfaces are connected to the various components described above. One of the first through n interfaces may be a network interface connected to an external device through a network.

Meanwhile, furthermore, the processor 220 may control the artificial intelligence model. In this case, of course, the processor 220 may include a dedicated graphics processor (eg, GPU) for controlling the artificial intelligence model.

Meanwhile, the artificial intelligence model according to the present invention may be a model based on supervised learning or unsupervised learning. Furthermore, the artificial intelligence model according to the present invention may include a support vector machine (SVM), a decision tree, a neural network, and the like, and methodologies to which they are applied.

As an embodiment, the artificial intelligence model according to the present invention may be an artificial intelligence model based on convolutional deep neural networks (CNN) learned by inputting training data. However, it is not limited thereto, and it goes without saying that various artificial intelligence models can be applied to the present invention. For example, models such as a deep neural network (DNN), a recurrent neural network (RNN), and a bidirectional recurrent deep neural network (BRDNN) may be used as an artificial intelligence model, but are not limited thereto.

At this time, convolutional deep neural networks (CNNs) are a type of multilayer perceptrons designed to use a minimum of preprocessing. A convolutional neural network consists of one or several convolutional layers and general artificial neural network layers placed on top of them, and additionally utilizes weights and pooling layers. Thanks to this structure, convolutional neural networks can fully utilize input data with a two-dimensional structure. Also, convolutional neural networks can be trained via standard back-propagation. Convolutional neural networks are easier to train than other feedforward artificial neural network techniques and have the advantage of using fewer parameters.

In addition, deep neural networks (DNNs) are artificial neural networks (ANNs) consisting of a plurality of hidden layers between an input layer and an output layer.

At this time, the structure of the deep neural network may be composed of a perceptron. Perceptron consists of several inputs, one processor, and one output value. The processor multiplies each of a plurality of input values by a weight, and then sums all the input values multiplied by the weight. Then, the processor substitutes the summed value into the activation function and outputs one output value. If a specific value is desired as an output value of the activation function, a weight value multiplied with each input value may be modified, and an output value may be recalculated using the corrected weight value. At this time, each perceptron may use a different activation function. In addition, each perceptron accepts the outputs passed from the previous layer as inputs, and then obtains the outputs using the activation function. The obtained output is passed as input to the next layer. Through the process as described above, several output values can finally be obtained.

Recurrent Neural Network (RNN) refers to a neural network in which the connections between units constituting an artificial neural network constitute a directed cycle. Unlike feed-forward neural networks, recurrent neural networks can utilize the memory inside the neural network to process arbitrary inputs.

Deep Belief Networks (DBN) is a generative graphical model used in machine learning. In deep learning, it means a deep neural network consisting of multiple layers of latent variables. There are connections between layers, but there is no connection between units within a layer.

Due to the characteristics of a generative model, the deep trust neural network can be used for prior learning, and after learning initial weights through prior learning, the weights can be fine-tuned through backpropagation or other discrimination algorithms. This characteristic is very useful when there is little training data, because the smaller the training data, the stronger the influence of the initial weight values on the resulting model. The pre-learned initial value of the weight becomes closer to the optimal weight than the arbitrarily set initial value of the weight, which enables the performance and speed of the fine-tuning step to be improved.

The above-described artificial intelligence and its learning method are described for illustrative purposes, and the artificial intelligence and its learning method used in the above-described embodiments are not limited. For example, all kinds of artificial intelligence technologies and learning methods that can be applied by a person skilled in the art to solve the same problem can be used to implement the system according to the disclosed embodiment.

Meanwhile, the processor 220 may include one or more cores (not shown) and a graphic processing unit (not shown) and/or a connection path (eg, a bus) for transmitting and receiving signals to and from other components. can

Processor 220 according to one embodiment performs a method described in connection with the present invention by executing one or more instructions stored in memory 210 .

For example, the processor 220 acquires new training data by executing one or more instructions stored in the memory 210, performs a test on the acquired new training data using the learned model, and the test result, First training data in which the labeled information is obtained with accuracy equal to or higher than a predetermined first reference value is extracted, the extracted first training data is deleted from the new training data, and learning is performed using the new training data from which the extracted training data is deleted. The model can be re-trained.

Meanwhile, the processor 220 includes RAM (Random Access Memory, not shown) and ROM (Read-Only Memory) that temporarily and/or permanently store signals (or data) processed in the processor 220. , not shown) may be further included. In addition, the processor 220 may be implemented in the form of a system on chip (SoC) including at least one of a graphics processing unit, RAM, and ROM.

The memory 210 may store programs (one or more instructions) for processing and control of the processor 220 . Programs stored in the memory 210 may be divided into a plurality of modules according to functions.

The communication unit 230 may communicate with the electronic device 100 . In particular, the communication unit 230 may include various communication chips such as a Wi-Fi chip, a Bluetooth chip, a wireless communication chip, an NFC chip, and a low energy Bluetooth chip (BLE chip). At this time, the Wi-Fi chip, the Bluetooth chip, and the NFC chip perform communication in a LAN method, a WiFi method, a Bluetooth method, and an NFC method, respectively. In the case of using a Wi-Fi chip or a Bluetooth chip, various connection information such as an SSID and a session key is first transmitted and received, and various information can be transmitted and received after communication is connected using this. The wireless communication chip refers to a chip that performs communication according to various communication standards such as IEEE, ZigBee, 3rd Generation (3G), 3rd Generation Partnership Project (3GPP), Long Term Evolution (LTE), and 5th Generation (5G).

3 is a flowchart of a user authentication process according to an embodiment of the present disclosure.

As shown in FIG. 3, in a system for performing non-face-to-face identity authentication using speaker authentication and face authentication according to an aspect of the present invention, the server 200 generates a user image of a user for whom pre-authentication has been completed. Face template data is registered (S1), voice template data generated from user voice data of a user whose pre-authentication is completed is registered (S2), face authentication is performed using the face template data (S3), and voice template data The speaker authentication may be performed using (S4), and the user authentication may be performed (S5) based on the face authentication result and the speaker authentication result.

Specifically, a system for performing non-face-to-face identity authentication using speaker authentication and face authentication according to an aspect of the present invention first acquires a user image of a user for whom pre-authentication has been completed, and a face template created from the obtained user image. Data may be registered in the server 200 (S1).

In addition, a system for performing non-face-to-face identity authentication using speaker authentication and facial authentication according to an aspect of the present invention obtains user voice data of a user for whom pre-authentication has been completed, and a voice template generated from the obtained user voice data. Data may be registered in the server 200 (S2).

Thereafter, the system for performing non-face-to-face authentication using speaker authentication and face authentication according to an aspect of the present invention performs face authentication based on face template data registered in advance and face template data obtained for user authentication. (S3) Yes.

Subsequently, the system for non-face-to-face authentication using speaker authentication and face authentication according to an aspect of the present invention performs speaker authentication based on pre-registered voice template data and voice template data obtained for user authentication. (S4) Yes.

At this time, the system for performing non-face-to-face authentication using speaker authentication and face authentication according to an aspect of the present invention performs step S4 after face authentication is completed in step S3 or step S4 after speaker authentication is completed in step S4. S3 may be performed, or steps S3 and S4 may be performed simultaneously.

A system for performing non-face-to-face authentication using speaker authentication and face authentication according to an aspect of the present invention may complete user authentication when face authentication and speaker authentication are both completed.

Hereinafter, each of the above-described processes of pre-authentication, face authentication, and speaker authentication will be described, and user authentication including pre-authentication, face authentication, and speaker authentication will be described.

First, the above-described pre-authentication will be described.

The pre-authentication may be to determine whether the corresponding identity is a user by using information representing the identity. Here, the information representing the identity may be information obtained from an ID image obtained by photographing the ID card or information on the owner and account number of the bank account.

For example, the pre-authentication may be authentication of an ID card that determines whether an identity indicated by an ID card is a user, or account authentication that determines whether an owner of a specific bank account is a user.

A system for performing non-face-to-face identity authentication using speaker authentication and face authentication according to an aspect of the present invention may directly perform the above-described pre-authentication for a user or receive a result of the pre-authentication from an external server.

That is, in the system for performing non-face-to-face identity authentication using speaker authentication and face authentication according to an aspect of the present invention, in obtaining a user image and user voice data of a user for which pre-authentication has been completed, the user is identified as a first step. The above-described pre-authentication may be performed directly or the result of the pre-authentication may be received from an external server to acquire a user image and user voice data from a user who has completed pre-authentication.

Hereinafter, a process in which the system for performing non-face-to-face authentication using speaker authentication and face authentication according to an aspect of the present invention directly performs the above-described pre-authentication using an ID card image will be described.

A system for performing non-face-to-face authentication using speaker authentication and facial authentication according to an aspect of the present invention checks the authenticity of a corresponding ID card, and when the authenticity of the ID card is confirmed, a user who wants to perform pre-authentication Liveness can be determined for the user's face image.

At this time, the system for performing non-face-to-face identity authentication using speaker authentication and face authentication according to an aspect of the present invention determines that the user's facial image acquired in the pre-authentication process has liveness, the facial image of the identity card in the identity card image. It is possible to determine whether there is a match between the face represented by each of the user's face images.

Finally, in the system for performing non-face-to-face authentication using speaker authentication and face authentication according to one aspect of the present invention, if it is determined that the facial image of the ID card in the ID image and the face represented by each user's facial image match, Authentication can be completed.

Hereinafter, the process of confirming the authenticity of the above-described identification card will be described in detail.

4 is an exemplary view of how an electronic device acquires an ID image according to an embodiment of the present disclosure.

The electronic device 100 may allow the user to photograph the identification card (ID) by outputting an ID photographing UI on the display to allow the user to photograph the identification card (ID).

At this time, the electronic device 100 may acquire an ID image through an auto scan function that acquires the ID card image as an ID image when the ID card is recognized from an image captured by the camera.

According to another embodiment, the electronic device 100 may display a photographing button on the display, and when the photographing button is selected and input by the user, an ID image may be obtained using a camera at the time when the photographing button is selected and inputted.

Thereafter, the electronic device 100 may generate ID information by recognizing the text ID-C from the ID image ID-I generated by photographing the ID card ID.

To this end, the electronic device 100 may recognize the character ID-C using a character reading module.

Here, the ID information may include at least one of ID type information, ID unique information (eg, resident registration number), ID valid date information, ID card issuance date information, and name information and address information of an identity indicated by the ID card.

Meanwhile, the electronic device 100 may recognize the outline of the ID card from the ID image ID-I, and blind-process some of the above-described ID information from the ID image ID-I.

Here, the identification card may be any one of a real resident registration card, a driver's license, and a passport.

Meanwhile, the ID may be any one of a mobile driver's license, a mobile resident registration card, and an electronic passport. In this case, the electronic device 100 may acquire ID information from a mobile driver's license, a mobile resident registration card, or an electronic passport through an electronic method without photographing the ID card.

The electronic device 100 may transmit the generated ID information to the server 200 .

Meanwhile, instead of the electronic device 100, the server 200 may generate ID information through the above-described text recognition.

To this end, the electronic device 100 transmits the ID image (ID-I) generated by photographing the ID card (ID) to the server 200, and the server 200 transmits the character (ID) from the ID image (ID-I). -C) can be recognized to generate ID information.

To this end, the server 200 may recognize a character ID-C using a character reading module.

The server 200 may request confirmation of authenticity of the ID card to the server of the verification agency based on the ID information.

To this end, the server 200 may transmit an authenticity confirmation request signal together with ID information to the confirmation authority server.

Thereafter, the server 200 may receive ID authenticity information indicating authenticity of the ID card from the verification authority server in response to the request for verifying the authenticity of the ID card.

Here, the verification authority server may be a server operated by an institution that issued an identification card to provide a verification result on whether the identification card is genuine or not.

The server 200 may transmit the received ID authenticity information to the electronic device 100 .

The electronic device 100 may output a message informing that the ID is fake when the received ID authenticity information indicates that the ID is genuine.

In addition, when the received authenticity information of the ID card indicates that the ID card is genuine, the electronic device 100 may acquire or create images, data, and values required in a process of determining liveness of a subsequent user's face image.

Conversely, if the received authenticity information of the ID indicates that the ID is fake, the electronic device 100 may output a message informing that the ID is fake and a message requesting a photograph of the real ID.

In addition, if the received authenticity information of the ID card indicates that the ID card is fake, the electronic device 100 may transmit the ID information to the server 200 by re-performing the photo ID and generating the ID information.

5 and 6 are exemplary diagrams of obtaining a user image by an electronic device according to an embodiment of the present disclosure.

The electronic device 100 may output a user photographing UI on the display to allow the user U to photograph himself.

In this case, the electronic device 100 may determine passive liveness by photographing a user taking a specific pose and facial expression without inducing the user to take a specific pose and facial expression.

Thereafter, the electronic device 100 extracts a plurality of second facial feature points from the user facial image U-FI in the second user image U-I generated by photographing the user for pre-authentication, and extracts a plurality of second facial feature points. Second facial feature point data composed of feature points may be generated.

Here, the plurality of second facial feature points may be information about facial features represented by the second user facial image U-FI. For example, the plurality of second facial feature points include coordinate information of each facial detail (eye corners, mouth corners, nose tip, etc.), color values of unit images at positions corresponding to each coordinate information, and units of positions corresponding to each coordinate information. It may include the exposure value of the image, the change value of each coordinate information, the depth value of the image corresponding to each coordinate information (the distance between the camera and the object), the change value of the depth value, the change value of the color value, and the change value of the exposure value. .

To this end, the electronic device 100 may extract 10,000 or more facial feature points from the user's facial image U-FI by using a previously learned 3D facial geometric model.

The electronic device 100 may generate second facial template data by encrypting and tokenizing the second facial feature point data.

At this time, the electronic device 100 generates second facial template data by encrypting the second facial feature point data using the encryption processing module, or tokenizes the second facial feature point data using the tokenization processing module to create the second facial feature point data. Template data is generated, or the second facial feature point data is encrypted using the encryption processing module, and then the encrypted second facial feature point data is tokenized using the tokenization processing module to generate the second facial template data, or the second facial feature point data is tokenized. After tokenizing the second facial feature data using the processing module, second facial template data may be generated by encrypting the tokenized second facial feature data using the encryption processing module.

Then, the electronic device 100 may transmit second face template data to the server 200 .

7 is a diagram for explaining a process in which a server obtains a face liveness value for a face image using an artificial intelligence model according to an embodiment of the present disclosure.

Referring further to FIG. 7 , when the server 200 receives the second facial template data U-FT from the electronic device 100, the second artificial intelligence model M2 has the second facial template data U-FT. ), the facial liveness value of the user face image U-FI in the second user image U-I may be obtained.

Here, the liveness of an image indicates whether the image is obtained from a real object, and the liveness value may be a value indicating a probability that the image is obtained from a real object.

Accordingly, the face liveness value may be a value indicating a probability that a face image corresponding to the corresponding face template data is acquired from an actual target (user).

For example, the closer the face liveness value is to 100%, the higher the probability that the face image corresponding to the corresponding face template data is obtained from a real object (user).

Conversely, the closer the face liveness value is to 0%, the higher the probability that the face image corresponding to the face template data was obtained from a photo or video of the user's face, and a mask in the shape of the user's face, rather than the actual subject (user). can represent

Meanwhile, the second artificial intelligence model M2 may be an artificial intelligence model trained to determine a face liveness value of a face image.

To this end, the server 200 may include an artificial intelligence training module that trains a second artificial intelligence model that outputs facial liveness values.

Specifically, the server 200 may train the second artificial intelligence model M2 using training data composed of face template data generated from a face image and facial liveness values predetermined for the face template data.

Through this, the server 200 inputs the second facial template data U-FT as input data to the second artificial intelligence model M2, and the facial liveness value as output data of the second artificial intelligence model M2. can be obtained.

When the obtained facial liveness value exceeds the reference facial liveness value, the server 200 determines that the user's facial image U-FI in the second user image U-I is an image taken from an actual user, and determines that the second user's face image U-FI is an image taken from a real user. It may be determined that the user face image U-FI in the image U-I has liveness.

Conversely, if the obtained facial liveness value does not exceed the reference facial liveness value, the server 200 determines that the user's facial image U-FI in the second user image U-I is not an image taken from an actual user. Accordingly, it may be determined that the user face image U-FI in the second user image U-I does not have liveness.

The server 200 may transmit to the electronic device 100 a result of determining whether the user face image U-FI in the second user image U-I has liveness according to the face liveness value.

When the result of determining whether the user's face image (U-FI) in the second user image (U-I) has liveness indicates that the electronic device 100 has liveness, the electronic device 100 determines the user's face image in the second user image (U-I) ( U-FI) may output a message indicating that it has liveness.

In addition, when the result of determining whether the user's face image U-FI in the second user image U-I has liveness indicates that it has liveness, the electronic device 100 determines whether or not the user's face image U-FI in the second user image U-I has liveness. In the process of determining whether the faces represented by the user images match, required images, data, and values may be obtained or generated.

Conversely, when the result of determining whether the user's face image U-FI in the second user image U-I has liveness indicates that it does not have liveness, the electronic device 100 determines whether or not the user's face image U-FI in the second user image U-I has liveness. A message indicating that the facial image U-FI does not have liveness may be output.

In addition, when the result of determining whether the user's face image U-FI in the second user image U-I has liveness indicates that it does not have liveness, the electronic device 100 sends a message requesting rephotographing of the user's face. output, the second user image U-I including the user face image U-FI is recaptured, and the second face template data U-FT is regenerated and transmitted to the server 200. .

Meanwhile, referring to FIG. 4 again, the electronic device 100, as a result of verifying the authenticity of the above-described ID card, identifies the ID face image (ID-FI) in the ID image ID-I of the ID card ID confirmed to be genuine. A plurality of first facial feature points may be extracted from, and first facial feature point data composed of the plurality of first facial feature points may be generated.

Here, the plurality of first facial feature points may be information about facial features indicated by the ID facial image (ID-FI). For example, the plurality of first facial feature points include coordinate information of each facial detail (eye corners, mouth corners, nose tip, etc.), color values of unit images at positions corresponding to each coordinate information, and unit positions corresponding to each coordinate information. It may include the exposure value of the image, the change value of each coordinate information, the depth value (distance between the camera and the object) of the image corresponding to each coordinate information, the change value of the depth value, the change value of the color value, and the change value of the exposure value. .

To this end, the electronic device 100 may extract 10,000 or more facial feature points from the ID facial image (ID-FI) using a 3D facial geometric model learned in advance.

The electronic device 100 may generate first facial template data by encrypting and tokenizing the first facial feature point data.

At this time, the electronic device 100 generates first facial template data by encrypting the first facial feature point data using an encryption processing module, or tokenizes the first facial feature point data using a tokenization processing module to create the first facial feature point data. Template data is generated, or the first facial feature point data is encrypted using the encryption processing module, and then the encrypted first facial feature point data is tokenized using the tokenization processing module to generate the first facial template data, or the first facial feature point data is tokenized. After tokenizing the first facial feature data using the processing module, first facial template data may be generated by encrypting the tokenized first facial feature data using the encryption processing module.

Then, the electronic device 100 may transmit the first face template data to the server 200 .

8 is a diagram for explaining a process in which a server obtains a facial matching value between faces represented by two facial images by using an artificial intelligence model according to an embodiment of the present disclosure.

When the server 200 receives the first face template data ID-FT from the electronic device 100, the second face template data determined to have liveness with the received first face template data ID-FT. (U-FT) is input into the first artificial intelligence model (M1) to determine the facial matching value between the face represented by the ID card facial image (ID-FI) and the second face represented by the second user facial image (U-FI). can be obtained

Here, the face match indicates whether the faces represented by the two face images are the same face, and the face match value may be a value indicating the degree to which the faces represented by the two face images are identical.

For example, the closer the face match value is to 100%, the higher the probability that the faces represented by the two face images are the same face.

Conversely, the closer the face match value is to 0%, the higher the probability that the faces represented by the two face images are unequal faces.

Meanwhile, the first artificial intelligence model M1 may be an artificial intelligence model trained to determine a face matching value of a face image.

To this end, the server 200 may include an artificial intelligence training module that trains a first artificial intelligence model that outputs a face matching value.

Specifically, the server 200 may train the first artificial intelligence model M1 using face template data generated from two face images and training data composed of predetermined face match values for the two face images.

Through this, the server 200 inputs the first facial template data (ID-FT) and the second facial template data (U-FT) as input data to the first artificial intelligence model (M1), and the first artificial intelligence model (M1). As the output data of (M1), a face match value can be obtained.

The server 200 determines that the face represented by the ID facial image (ID-FI) and the face represented by the user facial image (U-FI) are the same, that is, match when the obtained facial matching value exceeds the reference facial matching value. can

Conversely, the server 200 determines that if the obtained facial matching value does not exceed the reference facial matching value, the face represented by the ID facial image (ID-FI) and the face represented by the user facial image (U-FI) are not the same, that is, can be judged to be inconsistent.

The server 200 may transmit to the electronic device 100 a result of determining whether a face represented by an identification card facial image (ID-FI) matches a face represented by a user facial image (U-FI) according to a face matching value. there is.

The electronic device 100 may output as a message a result of determining whether the face represented by the ID facial image (ID-FI) and the face represented by the user's facial image (U-FI) match according to the facial matching value.

The server 200 may perform pre-authentication to determine whether the identity of the ID card is a user based on the face matching value and the face liveness value.

In addition, the server 200 may perform pre-authentication to determine whether the identity of the ID card is a user based on authenticity information of the ID card.

Specifically, the server 200 confirms that the photographed ID is an ID issued by an ID issuing institution and is not a fake ID, determines that the user's face image of the second user image has liveness, and determines that the face of the ID card is the face of the ID. If it is determined that the face of the user's face image of the second user image matches the face of the user, the identity of the ID card is determined to be the user, and pre-authentication may be performed.

The server 200 transmits the performed pre-authentication result to the electronic device 100, and the electronic device 100 may output the received pre-authentication result as a message.

In addition, the server 200 may transmit the result of the pre-authentication to the authentication request server. The authentication request server may be a server operated by a subject who has requested pre-authentication for a user. For example, the authentication request server may be a financial company server.

In the above, the process of directly performing the above-described pre-authentication using an ID image in the system for performing non-face-to-face authentication using speaker authentication and face authentication according to an aspect of the present invention has been described.

As described above, the system for performing non-face-to-face identity authentication using speaker authentication and facial authentication according to an aspect of the present invention may directly perform pre-authentication or receive a pre-authentication result from an external server.

Describing step S1, when the pre-authentication of the user is completed through the above-described process or the completion of the pre-authentication is confirmed, the server 200 generates second face template data from the second user image of the pre-authenticated user. can be registered to the user's account.

That is, when the pre-authentication is completed by performing the pre-authentication, the server 200 may register the second face template data generated in the pre-authentication process to the user's account.

According to another embodiment, the server 200 receives a result of the pre-authentication from an external server without directly performing the pre-authentication, and when the completion of the pre-authentication for the user is confirmed, the server 200 sends the electronic device 100 The electronic device 100, which transmits a generation request signal for requesting generation of second face template data and receives the generation request signal, performs the same process as the process of generating second face template data in the above-described pre-authentication process, After generating the second face template data, the second template data is transmitted to the server 200, and the server 200 may receive the second face template data and register it in the user's account.

Subsequently, step S2 will be described. When the pre-authentication of the user is completed through the above-described process or the completion of the pre-authentication is confirmed, the server 200 obtains the second voice template from the second user voice data of the pre-authenticated user. Data can be registered to the user's account.

9 and 10 are exemplary diagrams of obtaining user voice data by an electronic device according to an embodiment of the present disclosure.

After the pre-authentication is completed, the electronic device 100 may output a recording UI on the display to allow the user U to record his/her own voice by outputting a recording UI on the display.

In this case, the electronic device 100 may induce the user to utter a specific keyword (KEY) to record the user's voice, or utter a certain keyword other than a specific keyword to record the user's voice.

Thereafter, the electronic device 100 extracts a plurality of second voice feature points from the second user voice data generated by recording the voice of the user whose pre-authentication has been completed, and generates second voice feature point data composed of the plurality of second voice feature points. can create

Here, the plurality of second voice feature points may be information about voice characteristics represented by the second user voice data. For example, the plurality of second voice feature points may include voice frequency, voice strength, repetition frequency of a specific keyword, voice pitch, frequency change pattern, and voice pitch change pattern.

The electronic device 100 may generate second voice template data by encrypting and tokenizing the second voice feature point data.

At this time, the electronic device 100 generates second voice template data by encrypting the second voice feature point data using the encryption processing module, or tokenizes the second voice feature point data using the tokenization processing module to generate the second voice feature point data. Template data is generated, or the second voice feature point data is encrypted using the encryption processing module, and then the encrypted second voice feature point data is tokenized using the tokenization processing module to generate the second voice template data, or the second voice feature point data is tokenized. After tokenizing the second voice feature point data using the processing module, second voice template data may be generated by encrypting the tokenized second voice feature point data using the encryption processing module.

Then, the electronic device 100 may transmit second voice template data to the server 200 .

As described above, when the pre-authentication of the user is completed or the completion of the pre-authentication is confirmed through the above-described process, the server 200 obtains information from the second user voice data of the pre-authenticated user from the electronic device 100. 2 Voice template data can be created and registered in the user's account.

Describing step S3, the electronic device 100 acquires a first user image by photographing a user who is subject to user authentication in order to perform face authentication on a user for whom pre-authentication has been completed or completion of pre-authentication has been confirmed. can do.

In this case, the second user image acquired during the pre-authentication process or after the completion of the pre-authentication is confirmed and the first user image obtained for user authentication may be different images at different photographing times, but the first user image and the first user image may be different from each other. Both of the two user images are images obtained by photographing the user, and the process of generating and acquiring them may be the same.

In order to obtain the first user image, the electronic device 100 may output a user photographing UI to the display to allow the user U to photograph himself.

In this case, the electronic device 100 may determine passive liveness by photographing a user taking a specific pose and facial expression without inducing the user to take a specific pose and facial expression.

Thereafter, the electronic device 100 extracts a plurality of third facial feature points from the user facial image U-FI in the first user image U-I generated by photographing the user for user authentication, and extracts a plurality of third facial feature points. Third facial feature point data composed of feature points may be generated.

Here, the plurality of third facial feature points may be information about facial features represented by the first user facial image U-FI. For example, the plurality of third facial feature points include coordinate information of each facial detail (corner of the eye, corner of the mouth, tip of the nose, etc.), a change value of each coordinate information, and a depth value of the image corresponding to each coordinate information (the distance between the camera and the object). ) and a change value of the depth value.

To this end, the electronic device 100 may extract 10,000 or more facial feature points from the user's facial image U-FI by using a previously learned 3D facial geometric model.

The electronic device 100 may generate third facial template data by encrypting and tokenizing the third facial feature point data.

In this case, the electronic device 100 generates third facial template data by encrypting the third facial feature point data using an encryption processing module, or tokenizes the third facial feature point data using a tokenization processing module to create a third facial feature point data. Template data is generated, or the third facial feature point data is encrypted using the encryption processing module, and then the encrypted third facial feature point data is tokenized using the tokenization processing module to generate the third facial template data, or the third facial feature point data is tokenized. After tokenizing the third facial feature data using the processing module, third facial template data may be generated by encrypting the tokenized third facial feature data using the encryption processing module.

Then, the electronic device 100 may transmit third face template data to the server 200 .

Then, upon receiving the third facial template data (U-FT) from the electronic device 100, the server 200 inputs the third facial template data (U-FT) to the second artificial intelligence model (M2), A facial liveness value of the user face image U-FI in the first user image U-I may be obtained.

When the obtained facial liveness value exceeds the reference facial liveness value, the server 200 determines that the user facial image U-FI in the first user image U-I is an image taken from an actual user, and determines that the first user face image U-FI is an image taken from a real user. It may be determined that the user face image U-FI in the image U-I has liveness.

Conversely, if the obtained facial liveness value does not exceed the reference facial liveness value, the server 200 determines that the user facial image U-FI in the first user image U-I is not an image taken from an actual user. Accordingly, it may be determined that the user face image U-FI in the first user image U-I does not have liveness.

The server 200 may transmit to the electronic device 100 a result of determining whether the user face image U-FI in the first user image U-I has liveness according to the face liveness value.

The electronic device 100 determines whether the user's face image (U-FI) in the first user image (U-I) has liveness, if the determination result indicates that the user's face image (U-FI) has liveness, the user's face image in the first user image (U-I) ( U-FI) may output a message indicating that it has liveness.

In addition, when the result of determining whether the user face image U-FI in the first user image U-I has liveness indicates that it has liveness, the electronic device 100 determines whether the first user image U-I and the subsequent first user image U-I have liveness. In a process of determining whether the faces represented by each of the second user images U-I match, required images, data, and values may be obtained or generated.

Conversely, the electronic device 100 determines whether the user's facial image U-FI in the first user image U-I has liveness or not, when the result of determining whether or not the user's face image U-FI in the first user image U-I has no liveness indicates that the user in the first user image U-I has no liveness. A message indicating that the facial image U-FI does not have liveness may be output.

In addition, the electronic device 100 sends a message requesting rephotographing of the user's face when a result of determining whether the user's face image U-FI in the first user image U-I has liveness indicates that the user's face image U-FI does not have liveness. output, the first user image U-I including the user face image U-FI is recaptured, and the third face template data U-FT is generated again and transmitted to the server 200. .

When the server 200 receives the third facial template data U-FT from the electronic device 100, the server 200 receives the third facial template data U-FT and the second facial template data U-FT registered in the user account (U-FT). -FT) may be input to the first artificial intelligence model M1 to obtain a face matching value between faces represented by each user face image in each of the first user image and the second user image.

When the obtained facial matching value exceeds the reference facial matching value, the server 200 determines that the faces represented by each of the user face images in each of the first user image and the second user image are the same, that is, they match, and completes face authentication. can be dealt with

Conversely, if the obtained face matching value does not exceed the reference face matching value, the server 200 determines that the faces represented by the respective user face images in the first user image and the second user image are not the same, that is, do not match. can

The server 200 determines whether the face liveness value of the first user image exceeds the reference face liveness value and the face match value between the faces of the first user image and the second user image exceeds the reference face match value; When the face is authenticated, face authentication may be completed.

A result of facial authentication may be transmitted to the electronic device 100 .

The electronic device 100 may output a result of facial authentication as a message.

Describing step S4, the electronic device 100 acquires first user voice data of a user subject to user authentication in order to perform speaker authentication on a user whose pre-authentication has been completed or has been confirmed. can

To this end, the electronic device 100 may output a recording UI on the display to allow the user U to record his or her own voice, thereby enabling the user U to record his or her own voice.

In this case, the electronic device 100 may induce the user to utter a specific keyword (KEY) to record the user's voice, or utter a certain keyword other than a specific keyword to record the user's voice.

Thereafter, the electronic device 100 extracts a plurality of first voice feature points from first user voice data generated by recording the voice of a user whose pre-authentication has been completed, and extracts first voice feature point data composed of the plurality of first voice feature points. can create

Here, the plurality of first voice feature points may be information about voice characteristics represented by the first user voice data. For example, the plurality of first voice feature points may include voice frequency, voice strength, repetition frequency of a specific keyword, voice pitch, frequency change pattern, and voice pitch change pattern.

The electronic device 100 may generate first voice template data by encrypting and tokenizing the first voice feature point data.

At this time, the electronic device 100 generates first voice template data by encrypting the first voice feature point data using an encryption processing module, or tokenizes the first voice feature point data using a tokenization processing module to generate the first voice feature point data. Template data is generated, or the first voice feature point data is encrypted using the encryption processing module, and then the encrypted first voice feature point data is tokenized using the tokenization processing module to generate the first voice template data, or the first voice template data is tokenized. First voice template data may be generated by tokenizing the first voice feature point data using the processing module and then encrypting the tokenized first voice feature point data using the encryption processing module.

Then, the electronic device 100 may transmit the first voice template data to the server 200 .

11 is a diagram for explaining a process in which a server obtains a voice liveness value for user voice data using an artificial intelligence model according to an embodiment of the present disclosure.

11, when receiving the first voice template data VT1 from the electronic device 100, the server 200 inputs the first voice template data VT1 to the fourth artificial intelligence model M4, , a voice liveness value of the first user voice data may be obtained.

Here, the liveness of the voice data indicates whether the voice data is obtained from a real object, and the liveness value may be a value indicating a probability that the voice data is obtained from the real object.

Accordingly, the voice liveness value may be a value representing a probability that voice data corresponding to the corresponding voice template data is acquired from an actual subject (user).

For example, the closer the voice liveness value is to 100%, the higher the probability that voice data corresponding to the corresponding voice template data is acquired from a real object (user).

Conversely, the closer the voice liveness value is to 0%, the higher the probability that the voice data corresponding to the corresponding voice template data is obtained from a pre-recorded image or sound rather than a real object (user).

Meanwhile, the fourth artificial intelligence model M4 may be an artificial intelligence model trained to determine a voice liveness value of user voice data.

To this end, the server 200 may include an artificial intelligence training module for training a fourth artificial intelligence model that outputs a voice liveness value.

Specifically, the server 200 may train the fourth artificial intelligence model M4 using training data composed of voice template data generated from user voice data and voice liveness values predetermined for the voice template data.

Through this, the server 200 inputs the first voice template data VT1 as input data to the fourth artificial intelligence model M4, and obtains a voice liveness value as output data of the fourth artificial intelligence model M4. can do.

If the obtained voice liveness value exceeds the reference voice liveness value, the server 200 determines that the first user voice data is voice data generated by recording from a real user, and determines that the first user voice data has liveness. can judge

Conversely, if the obtained voice liveness value does not exceed the reference voice liveness value, the server 200 determines that the first user voice data is not voice data generated by recording from a real user, and determines that the first user voice data is live. It can be judged that it does not have varnish.

The server 200 may transmit to the electronic device 100 a result of determining whether the first user's voice data has liveness according to the voice liveness value.

When the result of determining whether the first user voice data has liveness indicates that the first user voice data has liveness, the electronic device 100 may output a message informing that the first user voice data has liveness.

In addition, when the result of determining whether the first user's voice data has liveness indicates that it has liveness, the electronic device 100 may obtain or generate data and values required in the speaker authentication process.

Conversely, if the result of determining whether the first user's voice data has liveness indicates that it does not have liveness, the electronic device 100 may output a message informing that the first user's voice data does not have liveness.

In addition, if the result of determining whether the first user's voice data has liveness indicates that it does not have liveness, the electronic device 100 outputs a message requesting re-recording of the user's voice and re-records the user's voice; Generation of the first voice template data VT1 may be performed again and transmitted to the server 200 .

12 is a diagram for explaining a process in which a server obtains a speaker matching value between speakers of each of two voices using an artificial intelligence model according to an embodiment of the present disclosure.

In step S4, when the server 200 receives the first voice template data VT1 from the electronic device 100, the received first voice template data VT1 and the second voice template registered in the user's account. By inputting the data VT2 to the third artificial intelligence model M3, a speaker matching value between a speaker of the first user voice data and a speaker of the second user voice data may be obtained.

Here, speaker matching indicates whether the speakers of each of the two user voice data are the same speaker. The speaker matching value may be a value indicating the degree to which the speakers of the two user voice data are identical, and speaker authentication is completed. It may mean that the speaker corresponding to the second voice template data VT2 registered in and the speaker corresponding to the first voice template data VT1 are determined to be the same.

For example, the closer the speaker matching value is to 100%, the higher the probability that the speakers of the two user voice data are the same speaker.

Conversely, the closer the speaker matching value is to 0%, the higher the probability that the speakers of the two user voice data are different speakers.

Meanwhile, the third artificial intelligence model M3 may be an artificial intelligence model trained to determine a speaker matching value between two user voice data.

To this end, the server 200 may include an artificial intelligence training module that trains a third artificial intelligence model that outputs speaker matching values.

Specifically, the server 200 may train the third artificial intelligence model M3 using speaker template data generated from two user voice data and learning data composed of speaker matching values predetermined for the two user voice data. there is.

Through this, the server 200 inputs the first voice template data VT1 and the second voice template data VT2 as input data to the third artificial intelligence model M3, and the third artificial intelligence model M3 A speaker matching value may be obtained as output data.

If the acquired speaker matching value exceeds the reference speaker matching value, the server 200 may determine that the speaker of the first user voice data and the speaker of the second user voice data are the same, that is, they match.

Conversely, if the acquired speaker matching value does not exceed the reference speaker matching value, the server 200 may determine that the speaker of the first user voice data and the speaker of the second user voice data are not the same, that is, they do not match.

When the voice liveness value exceeds the reference voice liveness value and the speaker matching value exceeds the reference speaker matching value, the server 200 may conclude that the speaker is authenticated and complete speaker authentication.

The server 200 may transmit a speaker authentication result to the electronic device 100 .

The electronic device 100 may output a speaker authentication result as a message.

Describing step S5, the server 200 may complete user authentication when face authentication and speaker authentication are both completed.

The server 200 may transmit the user authentication completion process to the electronic device 100 and the authentication request server. The authentication request server may be a server operated by a subject who has requested user authentication for a user. For example, the authentication request server may be a financial company server.

Meanwhile, the electronic device 100 according to another embodiment may acquire location information of a user attempting user authentication and transmit the obtained location information of the user to the server 200 .

Thereafter, the server 100 according to another embodiment may check country information corresponding to the location of the address indicated by the address information among ID information.

Then, the server 100 according to another embodiment may determine whether the location of the user indicated by the location information is included in the country indicated by the country information.

To this end, the server 100 according to another embodiment may pre-obtain reference data for area information for each country and address information for each country.

The server 100 according to another embodiment may perform the above-described face authentication, voice authentication, and user authentication only when the user's location indicated by the location information is included in the country indicated by the country information.

Accordingly, after the server 100 according to another embodiment registers face template data (step S1) and voice template data (step S2), the location of the user indicated by the location information is the country indicated by the country information. It can be determined whether or not it is included.

Thereafter, the server 100 according to another embodiment may not perform the above-described face authentication, voice authentication, and user authentication when the user's location indicated by the location information is not included in the country indicated by the country information.

Conversely, the server 100 according to another embodiment may perform the above-described face authentication, voice authentication, and user authentication when the user's location indicated by the location information is included in the country indicated by the country information.

Meanwhile, the server 100 according to another embodiment may obtain a reference face matching value, a reference facial liveness value, a reference speaker matching value and a reference speaker matching value based on a separation distance between the location of the address indicated by the address information and the location of the user indicated by the location information. One or more of the reference voice liveness values may be calibrated.

Specifically, the server 100 according to another embodiment may correct one or more of the reference facial matching value, the reference facial liveness value, the reference speaker matching value, and the reference voice liveness value to increase as the separation distance increases.

Through this, when the location of the address is far from the location of the user, it is possible to improve security by increasing the difficulty of authentication.

Steps of a method or algorithm described in connection with an embodiment of the present invention may be implemented directly in hardware, implemented in a software module executed by hardware, or implemented by a combination thereof. A software module may include random access memory (RAM), read only memory (ROM), erasable programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), flash memory, hard disk, removable disk, CD-ROM, or It may reside in any form of computer readable recording medium well known in the art to which the present invention pertains.

Also, different embodiments of the present invention may complement each other or be combined.

Components of the present invention may be implemented as a program (or application) to be executed in combination with a computer, which is hardware, and stored in a medium. Components of the present invention may be implemented as software programming or software elements; similarly, embodiments may include various algorithms implemented as data structures, processes, routines, or combinations of other programming constructs, such as C, C++ , Java (Java), assembler (assembler), may be implemented in a programming or scripting language such as Python (Python). Functional aspects may be implemented in an algorithm running on one or more processors.

Although the embodiments of the present invention have been described with reference to the accompanying drawings, those skilled in the art to which the present invention pertains can be implemented in other specific forms without changing the technical spirit or essential features of the present invention. you will be able to understand Therefore, it should be understood that the embodiments described above are illustrative in all respects and not restrictive.

100: electronic device
200: server

Claims (7)

In the control method of a system including an electronic device and a server,
obtaining, by the electronic device, a first user image generated by photographing a user subject to user authentication;
obtaining, by the server, a face matching value between a face represented by a user face image of a second user image of a user whose pre-authentication is completed and a face represented by a user face image in the first user image;
obtaining, by the server, a facial liveness value of a user face image in the first user image;
The server extracts a plurality of second voice feature points from second user voice data generated by recording the voice of the user for whom the pre-authentication has been completed, and encrypts the second voice feature point data including the plurality of second voice feature points. and tokenizing to generate second voice template data;
obtaining, by the electronic device, first user voice data generated by recording a voice of a user who is the target of the user authentication;
The server extracts a plurality of first voice feature points from the first user voice data, encrypts and tokenizes the first voice feature point data including the plurality of first voice feature points, and generates first voice template data. step;
The server inputs the first voice template data and the second voice template data to a third artificial intelligence model trained to determine a speaker matching value between two user voice data, acquiring a speaker matching value between a speaker of the user voice data and a speaker of the first user voice data;
obtaining, by the server, a voice liveness value of the first user voice data by inputting the first voice template data into a fourth artificial intelligence model trained to determine a voice liveness value of the user voice data; and
Performing, by the server, user authentication based on the face match value, the face liveness value, the speaker match value, and the voice liveness value;
The first voice feature point is
The frequency of the user's voice, the intensity of the voice, the repetition frequency of a specific keyword, the pitch of the voice, the frequency change pattern, and the change pattern of the voice pitch † F of the user subject to the user authentication,
The second voice feature point is
The user's voice frequency, voice strength, repetition frequency of a specific keyword, voice pitch, frequency change pattern, voice pitch † F change pattern of the user for whom the pre-authentication has been completed,
The control method of the system is
learning, by the server, the third artificial intelligence model using voice template data generated from two user voice data and training data consisting of speaker matching values predetermined for the two user voice data; and
Further comprising, by the server, learning the third artificial intelligence model using training data composed of voice template data from user voice data and a voice liveness value predetermined for the voice template data;
Performing the user authentication step,
performing, by the server, face authentication to determine whether a face represented by a user's face image in the first user image is the face of the user based on the face matching value and the face liveness value; and
and performing, by the server, speaker authentication to determine whether a speaker of the first user's voice data is the user based on the speaker matching value and the voice liveness value.
delete According to claim 1,
The step of performing the face authentication,
determining, by the server, whether the face match value exceeds a reference face match value;
determining, by the server, whether the facial liveness value exceeds a reference facial liveness value; and
When the face match value exceeds the reference face match value and the face liveness value exceeds the reference face liveness value, the server determines that the face represented by the user face image in the first user image is the user's face. Determining that it is a face; including, the control method of the system.
◈Claim 4 was abandoned when the registration fee was paid.◈ According to claim 1,
The step of performing the speaker authentication,
determining, by the server, whether the speaker agreement value exceeds a reference speaker agreement value;
determining, by the server, whether the voice liveness value exceeds a reference voice liveness value; and
determining, by the server, that the speaker of the first user's voice data is the user, when the speaker matching value exceeds the reference speaker matching value and the voice liveness value exceeds the reference voice liveness value; ; Including, the control method of the system.
◈Claim 5 was abandoned when the registration fee was paid.◈ According to claim 1,
Performing the user authentication step,
As a result of the face authentication, the server determines that the face represented by the user's face image in the first user image is the face of the user, and as a result of the speaker authentication, it is determined that the speaker of the first user's voice data is the user. , the system control method of authenticating the user.
According to claim 3,
extracting, by the electronic device or the server, identification information by recognizing text from the user's identification card image;
obtaining, by the electronic device, location information of a user attempting the user authentication;
The server checks country information corresponding to the location of the address indicated by the address information among the identification information, and determines whether the user's location indicated by the location information is included in the country indicated by the country information. , how to control the system.
In the server control method,
The server sets a facial matching value between a face represented by a user face image of a second user image of a user whose pre-authentication is completed and a face represented by a user face image in a first user image generated by photographing a user subject to user authentication. obtaining;
obtaining, by the server, a facial liveness value of a user face image in the first user image;
The server extracts a plurality of second voice feature points from second user voice data generated by recording the voice of the user for whom the pre-authentication is completed, and encrypts the first voice feature point data including the plurality of second voice feature points. and tokenizing to generate second voice template data;
The server extracts a plurality of first voice feature points from first user voice data generated by recording the voice of the user subject to user authentication, and first voice feature point data including the plurality of first voice feature points Encrypting and tokenizing to generate first voice template data;
The server inputs the first voice template data and the second voice template data to a third artificial intelligence model trained to determine a speaker matching value between two user voice data, acquiring a speaker matching value between a speaker of the user voice data and a speaker of the first user voice data;
obtaining, by the server, a voice liveness value of the first user voice data by inputting the first voice template data into a fourth artificial intelligence model trained to determine a voice liveness value of the user voice data; and
Performing, by the server, user authentication based on the face match value, the face liveness value, the speaker match value, and the voice liveness value;
The first voice feature point is
The frequency of the user's voice, the intensity of the voice, the repetition frequency of a specific keyword, the pitch of the voice, the frequency change pattern, and the voice pitch † F change pattern of the user who is the target of the user authentication,
The second voice feature point is
The user's voice frequency, voice strength, repetition frequency of a specific keyword, voice pitch, frequency change pattern, voice pitch † F change pattern of the user for whom the pre-authentication has been completed,
The server control method
learning, by the server, the third artificial intelligence model using voice template data generated from two user voice data and training data consisting of speaker matching values predetermined for the two user voice data; and
Further comprising, by the server, learning the third artificial intelligence model using training data composed of voice template data from user voice data and a voice liveness value predetermined for the voice template data;
Performing the user authentication step,
performing, by the server, face authentication to determine whether a face represented by a user's face image in the first user image is the face of the user based on the face matching value and the face liveness value; and
and performing, by the server, speaker authentication to determine whether a speaker of the first user's voice data is the user based on the speaker matching value and the voice liveness value.

Images