RU2768797C1 - Method and system for determining synthetically modified face images on video - Google Patents

Abstract

FIELD: computer engineering.

SUBSTANCE: invention relates to computer engineering for determining synthetically altered images of faces on video. Technical result is achieved by a method in which: an image is obtained from a video; identifying images of faces on said image; calculating a vector representation of the geometric characteristics of the detected face images; using frame-by-frame video analysis, calculating the spatial-temporal significance of each face image of each person on said image and a vector representation of the temporal characteristic of the face image; calculating a vector of estimates of the probability of synthetic changes for images of human faces; calculating an overall estimate of the probability of synthetic changes; generating a final assessment of the presence of a synthetic face image change on the video; forming an integral estimate of the presence of a synthetically changed face image on the video at least one final estimate of the model and generating a notification on the presence of the synthetically changed face in the video.

EFFECT: high accuracy and efficiency of detecting synthetic change in images of human faces in video.

20 cl, 10 dwg, 2 tbl

Description

FIELD OF TECHNOLOGY

[0001] The present technical solution relates to the field of computer technology used in the field of data processing, in particular to a method and system for determining synthetically modified images of faces in a video.

BACKGROUND OF THE INVENTION

[0002] To date, the use of technologies for the formation of synthetic images superimposed on images of the faces of real people is based, as a rule, on the use of machine learning algorithms, for example, artificial neural networks (ANNs). Such approaches are aimed at imposing digital masks that mimic human faces. An example of such technology is the DeepFake technique, based on artificial intelligence and used for image synthesis (see https://ru.wikipedia.org/wiki/Deepfake).

[0003] There is a known method for recognizing synthetically modified images of people's faces, in particular DeepFake images (Tolosana et al. DeepFakes Evolution: Analysis of Facial Regions and Fake Detection Performance // Biometrics and Data Pattern Analytics - BiDA Lab, Universidad Autonoma de Madrid. 2020), which is based on the analysis of the segments that form the face images. The analysis is carried out using an ANN trained on real and synthetic images of people's faces, in particular, celebrities, which can be used to detect fake (fake) videos. The method makes it possible to analyze facial segments, on the basis of which a classification of the corresponding image is issued as containing synthetic changes or not.

[0004] The disadvantage of this approach is low efficiency due to the lack of application of an integral estimate, which would be formed based on the geometric parameters of the face image, and on the basis of the spatio-temporal characteristics of the person's face in the video. Another drawback is the lack of processing of several people in some solutions if there are several people on the video. In other well-known open solutions (https://www.kaggle.eom/c/deepfake-detection-challenge, https://ai.facebook.com/datasets/dfdc/), such processing is carried out by independently evaluating each face image, each person, on each analyzed frame of the video and further averaging all such estimates. All such solutions show low efficiency when processing videos with several people.

SUMMARY OF THE INVENTION

[0005] The claimed method and system is aimed at solving the technical problem of efficiently and accurately detecting synthetic changes in facial images in video.

[0006] The technical result is to increase the accuracy and efficiency of detecting a synthetic change in images of people's faces in a video.

[0007] In a first preferred embodiment of the invention, a computer-implemented method for determining synthetically modified images of faces in a video, performed using a processor, is provided, in which:

a) obtaining at least one image from the video;

b) detecting images of faces in said image;

c) calculating a vector representation of the geometric characteristics of the detected face images using at least a face reference point comparison algorithm to determine images of at least one person's face;

d) using frame-by-frame video analysis, the spatio-temporal significance of each image of the face of each person in said image is calculated, which is defined as a vector representation of the spatial characteristic of the face, characterizing the size of the face area in relation to the frame, and a vector representation of the temporal characteristic of the face image, characterizing the display time the analyzed image of the face on the video frames;

e) calculate the vector of estimates of the probability of synthetic changes for images of human faces, characterizing the presence of synthetic changes in the images of faces of this person in each frame

f) calculating an overall synthetic change probability score based on vector representations of the spatial, temporal, and synthetic change score vectors for each person's face images in the video;

g) forming a final assessment of the presence in the video of a synthetic image change of at least one face;

h) forming an integral estimate of the presence of a synthetically modified face image in the video according to at least one final assessment of the model and generating a notification about the presence of a synthetically modified face in the video.

[0008] In one of the particular implementations of the method, steps c)-h) are performed by a machine learning model or ensemble of models, while the machine learning model or ensemble of models is trained on a data set containing synthesized images of people's faces.

[0009] In another particular implementation of the method, the machine learning model uses an automatic markup correction function that corrects the incorrect markup of each face on frames by comparing the images of faces on the synthesized video with their images on the original video.

[0010] In another particular implementation of the method, faces are compared based on the value of the vector proximity of the reference points that form the geometric characteristics of the original face image and the synthesized image based on it.

[0011] In another particular implementation of the method, comparison of faces is carried out by analyzing the coordinates of the regions of the original face image and the synthesized face image.

[0012] In another particular implementation of the method, the spatiotemporal significance is calculated as a general matrix based on the values of vector representations, and the assessment of the presence of synthetic changes in the images of the faces of an individual is formed by a machine learning model using the obtained general matrix.

[0013] In another particular implementation of the method, an ensemble of machine learning models consists of a group of models, each of which is trained to identify a specific synthetic imaging algorithm.

[0014] In another particular implementation of the method, it contains an integral classifier that receives as input estimates generated using models included in the ensemble.

[0015] In another particular implementation of the method, the overall score is calculated using an integral classifier.

[0016] In another particular implementation of the method, an algorithm for generating a synthetic face image in the analyzed video stream is additionally defined.

[0017] In another particular implementation of the method, the video is an online video conference.

[0018] In another particular implementation of the method, when a synthetically modified face image is determined, a notification is generated in its display area.

[0019] In another particular implementation of the method, when determining a synthetically modified face image, the connection with this user is blocked.

[0020] In another particular implementation of the method, the analyzed image is obtained from a biometric identification or biometric authentication system.

[0021] In another particular implementation of the method, when determining a synthetically modified face image, access or the requested action is blocked from the user.

[0022] In another particular implementation of the method, when determining a synthetically modified face image, user authentication data is additionally requested, selected from the group: login, code, password, two-factor authentication, or combinations thereof.

[0023] In another particular implementation of the method, a signal is generated in the form of a quantitative estimate of the probability of the presence of a synthetically modified face image.

[0024] In another particular implementation of the method, images are obtained from a video media space monitoring and social media and media analysis system that performs content verification on social media and media.

[0025] In another particular implementation of the method, when a synthetically modified face image is determined, a notification is generated to inform the person who has been exposed to the creation of the modified face image.

[0026] In a second preferred embodiment of the invention, a system is provided for detecting synthetically modified face images in video, comprising at least one processor and at least one memory storing machine-readable instructions that, when executed by the processor, implement the above method.

BRIEF DESCRIPTION OF THE DRAWINGS

[0027] FIG. 1 illustrates a block diagram of the implementation of the claimed method.

[0028] FIG. 2 illustrates an example of generating a vector representation of images of faces in a video.

[0029] FIG. 3A-3B illustrate an example of generating vector representations of space-time characteristics.

характеристики изображений лиц для изображений лиц каждого человека на видео.[0030] FIG. 4 illustrates a block diagram of generating the synthetic face image score vector, the face image spatial characteristic vector, and the vector

face image characteristics for the face images of each person in the video.

характеристик, и общей оценки синтетических изображений лиц.[0031] FIG. 5 illustrates a block diagram of the independent formation of the final spatial and

characteristics, and overall evaluation of synthetic face images.

характеристик, при их независимом формировании от общей оценки синтетических изображений лиц, для исключения лиц людей из расчета оценки синтетических изменений в видео.[0032] FIG. 6 illustrates a flowchart for processing the resulting spatial and

characteristics, when they are formed independently from the overall assessment of synthetic images of faces, to exclude people's faces from the calculation of the assessment of synthetic changes in the video.

[0033] FIG. 7 illustrates a flowchart for generating a notification with an integrated assessment of the presence of synthetic images of people's faces in a video, a notification about a probable algorithm for generating synthetic data, when using a set of ensembles of trained machine learning models, when the models of each ensemble are trained on a data set with one specific algorithm for generating synthetic face changes, and at least the models of one ensemble are trained on a dataset with several algorithms for generating synthetic face changes.

[0034] FIG. 8 illustrates a flowchart when a notification is generated by an integrated classifier based on the scores of several trained machine learning models or their ensembles, and there are several people on the video.

[0035] FIG. 9 illustrates a general view of the computing device.

IMPLEMENTATION OF THE INVENTION

[0036] In this decision, the term "synthetically modified face image" hereinafter in the text will be understood as any type of digital imaging that imitates the face or part of the face of another person, including by applying digital masks, distorting/changing parts of the face, etc. .P. A synthetically modified face image should be understood as fully generated images, for example, masks using DeepFake technology, superimposed on the face of a real person in the frame while maintaining the mimic activity of the image, and the formation of a partial change in individual parts of the face (eyes, nose, lips, ears etc.).

[0037] As shown in FIG. 1, the implementation of the claimed method (100) for determining synthetically modified images of faces in a video consists in the execution by a computing computer device, in particular, using one or more processors in an automated mode, of a software algorithm presented as a sequence of steps (101) - (107), ensuring the performance of material actions in the form of processing electronic signals generated when the processor of the computing device performs its functions in order to implement the execution of data processing as part of the execution of the method (100).

[0038] At the first stage (101), one or more images obtained from the video are received and stored in the memory of the computing device that performs the method (100). In the present application materials, the term “video” will mean a video image, a video stream (for example, from an IP camera, an electronic device camera, a virtual camera, from an Internet application), an ordered sequence of frames (images), subsampling of frames, including up to and including up to one image.

[0039] At step (102), the received images are analyzed for the presence of images of faces to determine the presence of its synthetic change. Subsequent analysis of the obtained images can be performed using one or more (ensemble) machine learning models that are trained to detect and classify face images.

[0040] When detecting a synthetic change in face images in a video, various machine learning models can be used, for example, neural network architectures, such as fully connected neural networks, CNN (convolutional networks), RNN (recurrent networks), Transformer (transformer networks), CapsNet ( capsule networks) and their combinations.

[0041] In their training, the networks may detect one or more features of synthetically modified face images, in particular:

- anatomical proportion of the face and head;

- anatomical feature of the location of parts of the face;

- proportions of parts of the face;

- plasticity and relief of mimic diversity;

- features of plastic parts of the face: eyebrows, eyes, nose, ears, lips, skin;

- general characteristics of the muscles of the face and neck;

- structure and distribution of muscles into groups (facial, chewing, suboccipital and others), location;

- unnaturalness of shadows, light, glare, penumbra, reflexes of illumination and the environment of the details of the face and the surrounding space;

- temperature distribution by face elements;

- blurring, smoothing when rendering elements of the face, head and other elements of the image;

- sharpening (sharpness) and artificial enhancement of features when rendering elements of the face, head and other image elements;

- graphic artifacts left by generation algorithms and/or their specific implementations in software when creating synthetic images.

[0042] It is also possible to use pre-trained neural networks with or without further training. In the case of using architectures with convolutional networks, such pre-trained models can be used as: AlexNet, VGG, NASNet-A, DenseNet, DenseNet-B, DenseNet-BC, Inception, Xception, GoogleNet, PReLU-net, BN-inception, AmoebaNet, SENet, ResNet-18, ResNet-34, ResNet-50, ResNet-101, ResNet-152, XResNet, Squeeze-and-Excitation ResNet (SE-ResNet), EfficientNet-B0, EfficientNet EfficientNet-B1, EfficientNet-B2, EfficientNet-B3 , EfficientNet-B4, EfficientNet-B5, EfficientNet-B6, EfficientNet-B7, YOLO and inherited from them.

[0043] The machine learning model was trained with at least one of the following steps:

- Obtaining classified (labeled, with affixed classes) data in one or more formats: video stream, video file, video frames (frame);

- Selection of frames in case of receiving a video stream or video file;

- Detection of face(s) on frames. Cutting them out of the frame with some neighborhood around the face and getting arrays of face data;

- For the data of the class "Synthetically modified image" in the case of the presence of the original image frame - the frame from which such a modified image was formed, checking the correctness of the affixed class;

- For each face, its data array (pixel values, bmp-map) is transformed according to the preprocessing algorithm (data standardization, image scaling, etc.);

- Data augmentation;

- Formation of a data package and its submission for training a neural network;

- Calculate the value of the objective function and backpropagate the error of the data packet for network training. The following can be used as quality indicators: LogLoss, accuracy, precision (accuracy), recall (completeness), F-measure, AUC-ROC, AUC-PR, Gini coefficient / index (Gini coefficient), confusion matrix (error matrix).

[0044] One or more of the following algorithms can be used as a training algorithm for a machine learning model: Adagrad (Adaptive gradient algorithm), RMS (Root mean square), RMSProp (Root mean square propagation), Rprop (Resilient backpropagation algorithm), SGD (Stochastic Gradient Descent), BGD (Batch Gradient Descent), MBGD (Mini-batch Gradient Descent), Momentum, Nesterov Momentum, NAG (Nesterov Accelerated Gradient), FussySGD, SGDNesterov (SGD + Nesterov Momentum), AdaDelta, Adam (Adaptive Moment Estimation) , AMSGrad, AdamW, ASGD (Averaged Stochastic Gradient Descent), LBFGS (L-BFGS algorithm - Broyden-Fletcher-Goldfarb-Shanno algorithm with limited memory usage), as well as second-order optimizers such as: Newton method, Quasi-Newton method, Gauss-Newton algorithm, Conjugate gradient method, Levenberg-Marquardt algorithm.

[0045] At least one of the following functions is used as an objective function when training a machine learning model: L1Loss, MSELoss, CrossEntropyLoss, CTCLoss, NLLLoss, PoissonNLLLoss, GaussianNLLLoss, KLDivLoss, BCELoss, BCEWithLogitsLoss, MarginRankingLoss, HingeEmbeddingLoss, MultiLoginLabLoss , SoftMarginLoss, MultiLabelSoftMarginLoss, CosineEmbeddingLoss, MultiMarginLoss, TripletMarginLoss, TripletMarginWithDistanceLoss.

[0046] When training a machine learning model, a labeling self-check (automatic labeling correction) stage can be used, which checks each face in the image (frame) that is marked as containing a synthetic change, that it really contains signs of such a change.

[0047] This check is implemented if the source video (frames, images) is available. The source video (frames, images) is a real (unmodified by the introduction of a synthetic change) video from which a synthetically modified video of the training (additionally and test) set was formed. This feature is implemented as follows and may include the following steps:

- The face detection algorithm on the image on the frame of the synthetically modified video detects a face. A part of the image with a face and some neighborhood around it is cut out. The size of the neighborhood may vary.

- All faces are detected in the corresponding frame of the original video. The face with the closest characteristics to the face from the previous step is selected. As a measure of proximity, depending on the face detection algorithm used, proximity at one or more points (a set of points) of a face is used:

- nose;

- nostrils;

- hair lines;

- lines of vegetation on the face (beard, mustache);

- mouth;

- lips (upper and lower);

- forehead;

- eye;

- pupils

- ears;

- eyebrows;

- age;

- heads;

- cheekbones;

- chin;

- nasolabial triangle;

- coordinates of the face rectangle.

[0048] As an algorithm for detecting people's faces, approaches such as: adapted enhancement and the Viola-Jones method based on it, MTCNN, Elastic graph matching method, DeepFace Facebook, hidden Markov models (HMM, HMM), Principal component analysis and data matrix decomposition algorithms (RSA, SVD, LDA), Active Appearance Models (AAM), Active Shape Models (ASM), FERET (face recognition technology), SURF, NeoFace, SHORE, ROI, Template Matching Methods, DPM (Deformable Part Model), Artificial Neural Networks (Neural network: Multilayer Perceptrons), Factor Analysis (FA), Linear Discriminant Analysis, Support Vector Machines (SVM), Naive Bayesian classifier, Hidden Markov model, Distribution-based method, Mixture of PCA, Mixture of factor analyzers), Sparse network of winnows (SNoW)).

[0049] Proximity is understood as the minimum distance for numerical data according to the Bray-Curtis metric, Canberra, Ruzicka, Kulczynski, Jaccard, Euclidean distance, Manhattan metric, Penrose size distance, Penrose shape distance, Lorentzian distance, Hellinger distance, Minkowski distance measure p, Mahalanobis distance, statistical distance, correlation similarities and distances (Pearson correlation, Orchini similarity, normalized dot product) or otherwise. When calculating proximity, the coordinates of points on the frame of the synthetically modified video and the coordinates of such points on the frame of the original video are taken for calculation, then the closest face image is selected, as faces with the minimum distances between the points used.

[0050] In one of the particular examples of implementation, it is also possible to select (obtain coordinates) a face area on a frame of a synthetically modified video, after which an area with the same coordinates is cut out on the frame of the original video. In another particular implementation example, the reverse type of processing can be performed - a face is detected on the frame of the original video, and an area with the same coordinates is cut out on the frames of the synthetically modified video. As a result of the performed operations, two images are obtained, which are an area with a face from the frame of the original video and a face with a frame of synthetically modified video.

[0051] The resulting pair of images is compared with each other according to a given metric to assess the level of image distortion. As such a metric can be used:

- Peak signal-to-noise ratio (PSNR - peak signal-to-noise ratio). https://ru.wikipedia.org/wiki/Peak_signal_to_noise_ratio;

- Mean square error (MSE - mean square error). https://en.wikipedia.org/wiki/Standard_deviation;

- The square root of the root-mean-square error function (RMSE - root-mean-square error). https://ru.wikipedia.org/wiki/Peak_signal_to_noise_ratio;

- Relative mean squared deviation (RMD - Root mean squared deviation);

- Standard deviation (RMS - Root Mean Squared);

- Index of structural similarity (SSIM - structure similarity). https://ru.wikipedia.org/wiki/SSIM;

- Structural differences (DSSIM - structural dissimilarity). https://ru.wikipedia.org/wiki/SSIM;

- Signal-to-noise ratio (SNR; SNR - signal-to-noise ratio). https://ru.wikipedia.org/wiki/Signal/noise ratio/;

- Absolute difference between pixels and indicators inherited from it (average, relative, etc.).

[0052] In this case, if color images are analyzed (with several components per pixel), similar ones are applied with further weighted averaging for each of the components. For example, for an RGB image, PSNR or MSE is calculated over all three components (and divided by three times the image size). For a good quality synthetic image and a good video quality (no interference from noise), it is preferable to use "PSNR". If only part of the face is to be overlaid with the synthetic image, it is preferable to use PSNR. If the video is noisy or grainy, then it is preferable to use DSSIM or SSIM. In the presence of a lot of interference, it is preferable to apply SNR. If the video quality is extremely low quality, for example, with a high compression ratio, then it is preferable to use MSE or RMD. If the dimensions of the face in relation to the frame are small, then the absolute difference between the pixels is applied.

[0053] According to the applied metric, a boundary value is selected, and if the value of the metric between two received images is greater than this boundary value, then the face on the frame is accepted as synthetically modified. If the value is less than or equal, then, despite the marking of this image as synthetically modified, then this face image is taken as real.

[0054] When performing the transformation of face data arrays, such elements as: data normalization, data standardization, size reduction to a given one, image scaling algorithms can be used.

[0055] Data augmentation for training one or more machine learning models may be performed using at least one of the following approaches: image scaling (increase, decrease); image cropping; dimming the entire image, individual channels of the image; clarification of the entire image, individual channels of the image; increase in contrast; color transformations: repositioning (mixing) of color channels, amplification, reduction of one or more color channels, obtaining an image in grayscale, obtaining a monochrome image, deleting a color channel; shifts and decentering of the image; rotation of the image at different angles in different directions, rotation of the image or its part; slopes, distortions of the image; mirror image along an arbitrary axis, line; additional lines or geometric objects in the image: with transparency of their color, without transparency, colored objects; gray objects (from white to black), including the removal of a part of the image (placing a black object on the image) at the geometric or semantic positions of the image; adding any background to the image; glare and darkening of parts of the image; defocus (blur) of the image or its parts; increasing graininess, sharpness (sharpness) of the image; compression and stretching along axes, lines; image noise over the entire image or part of it, the placement of white or other noise; adding one or more elements of Gaussian noise (Blur), spotted noise; superposition (overlay) of two or more images from the training set (parts of images) with different weights; elastic transformation of the image (Elastic Transform); grid distortion of the image (GridDistortion); compression of image data by various image processing algorithms with some quality (for example, compression of the original bmp image according to the JPEG standard of some quality, and then getting a bmp image from it again); isotropic, affine and other transformations (https://github.com/albumentations-team/albumentations).

[0056] In this case, all of the above are applicable in various types of graphic representation or its channels: RGB, sRGB, RGBA, ProPhoto, CMYK, XYZ, LMS, HKS, HSV, HSB, HSL, AHSL, RYB, LAB, NCS, RAL, YUV . YCbCr. YPbPr, YDbDr, YIQ, PMS (Panton), Munsella. These methods of augmentation can be applied to one image, in any sequence, with or without the possibility of application.

[0057] Using the trained model or algorithm for detecting people's faces or at step (102), faces are highlighted. At step (103), the face images extracted at step (102) are processed to determine which face images belong to one person. To do this, at step (103), a vector representation of the geometric characteristics of face images is calculated. In the general case, this is done using the face reference point comparison algorithm. Using the definition of geometric characteristics, images of faces belonging directly to the same person are determined. The formation of this vector makes it possible to estimate the probability of having a face of a real person. The algorithm of work can be carried out as follows. The j-th face is highlighted on the i-th frame. This j-oe face is searched for in subsequent frames.

[0058] In one of the particular examples of the implementation of the invention, the search is carried out by highlighting the closest face image in space among all detected faces on the i + 1st frame. As a measure of proximity, depending on the face detection algorithm used, proximity is used (numerical data according to the Bray-Curtis metric, Canberra, Ruzic, Kulchinsky, Jaccard, Euclidean distance, Manhattan metric, Penrose size distance, Penrose shape distance, Lorentz distance, distance Hellinger distance, Minkowski distance p, Mahalanobis distance, statistical distance, correlation similarities and distances - Pearson correlation, Orchini similarity, normalized scalar product, or other) at one or more points of the face (reference points of the face): nose, nostrils, hairline, vegetation lines on the face (beard, mustache), mouth, lips (upper and lower), forehead, eyes, pupils, ears, eyebrows, eyelids, head, cheekbones, chin, nasolabial triangle, face rectangle coordinates.

[0059] The distance between the corresponding reference points of the j-th face in the i-th frame and the points of each face in the i+1-th frame is calculated. Then a face is selected from the i+1-th frame with the smallest distances by reference points. In another particular embodiment of the invention, on the i+1st frame, a face with the closest characteristics between the given reference points (the relative position of the points) is searched for. In this case, the geometric characteristics (dimensions) of the location of the reference points of the j-th face image are considered, and on the i + 1 frame, the face images with the most similar geometric characteristics are searched. In yet another embodiment, for each face, a certain spatial neighborhood (location area) on the frame is allocated and it is checked whether there is any image of the face in the i + 1st frame. The implementation of approaches in the implementation of the claimed method (100) does not limit other possible ways to search for a face image in frames.

[0060] Next, at step (104), for each detected face image in the frames, an estimate of the probability of its synthetic change is calculated using the trained machine learning model for detecting and classifying synthetic changes. This rating is added to the vector of ratings for face images of the j-th person. If the image of the j-th face is not found on the next frame (or series of frames) of the ordered sequence of frames, then the formation of the vector of estimates can be completed. An example of the formation of a rating vector for the image of a person's face on video frames is shown in Fig. 2. In another implementation, the formation of a rating vector for face images of the j-th person occurs throughout the video, and is not completed if a face image is not detected on the next frame.

характеристик (пространственного вектора и

вектора) могут проводится последовательно, как это представлено на Фиг. 4, или параллельно, независимо друг от друга. Описание изобретения не ограничивает порядок и способ расчета данных векторов, а описывает их применение для повышения качества выявления синтетических изменений изображений лиц на видео.[0061] Next, at step (104), for each specific image of a person's face, its spatial and temporal significance is determined, which is defined as a vector representation of the spatial characteristics of a person's face, characterizing the size of the face area in relation to the frame, and a vector representation of the temporal characteristic of the face image, characterizing display time of the analyzed face image on video frames. On FIG. 4 shows a diagram of step 104. Calculations of the vector of estimates of synthetic changes in the images of the face of the j-th person in the video, which consists of estimates of changes in the face image in each analyzed frame, calculation of the vector of spatial and

characteristics (space vector and

vector) can be carried out sequentially, as shown in FIG. 4, or in parallel, independently of each other. The description of the invention does not limit the order and method of calculating these vectors, but describes their use to improve the quality of detecting synthetic changes in face images in video.

[0062] In FIG. 3A-3B show an example of calculating the spatial and temporal significance vectors and the vector of synthetic change estimates. In the presented example, for each frame (K1)-(K6) of the video obtained at step (101), the vector of estimates of the synthetically modified face, the vector of the spatial distribution of faces on the frames, and the temporal characteristic of the face on the frames are calculated. The spatial characteristic can be calculated based on the occupied fraction of the face area from the frame size. For example, the rectangle in which the face is inscribed in the frame has the following coordinates: X1=100, Y1=50 - upper left corner; Х2=300, Y2=150 - lower right corner. The area of such a rectangle is 200*100=20000. The video was obtained at a resolution of 1280×1920 pixels and its area is 2457600. The proportion of the face area in the frame will be 20000/2457600=0.8%. The time characteristic for each face can be calculated as a scalar value, for example, the time it is displayed on the video. In another implementation, a vector may be formed in which 1 is assigned if the person is present in the frame, or 0 if he is not in the frame. Spatial and temporal significance can be represented, in particular, as a common matrix based on the values of the generated vector representations.

распределения, пространственного распределения и вектора оценок вероятности, что изображение лица на кадре было подвержено синтетическим изменениям.[0063] At step (105), an overall score of synthetic changes in images of human faces on video is generated based on the vectors obtained at steps (103)-(104). That is, the calculation of the synthetic image change probability estimate for each person's face in the video is performed based on the vectors

distribution, spatial distribution, and vector of probability estimates that the face image in the frame was subject to synthetic changes.

распределения, вектор оценок синтетических изменений изображения лица объединяются в общую двумерную матрицу, представленную в Таблице 1 для примера на Фиг. 3А. Полученная матрица подается на вход модели машинного обучения для формирования общей оценки синтетического изменения лица j-го человека на видео. Данная модель может представлять собой рекуррентную нейронную сеть, сверточную нейронную сеть, полносвязанную нейронную сеть. Подобное объединение рекомендуется использовать для случая, когда человек присутствует на разных временных отрезках видео, а не только в одной последовательной серии кадров.[0064] A separate machine learning model can be used to generate an overall estimate of synthetic changes in the face images of the j-th person. To form the mentioned general estimate, the obtained vectors of spatial and

distributions, the vector of estimates of synthetic changes in the face image are combined into a common two-dimensional matrix presented in Table 1 for an example in Fig. 3A. The resulting matrix is fed to the input of the machine learning model to form an overall estimate of the synthetic change in the face of the j-th person in the video. This model can be a recurrent neural network, a convolutional neural network, a fully connected neural network. Such a combination is recommended for the case when a person is present at different time intervals of the video, and not only in one consecutive series of frames.

[0065] In another particular embodiment of the invention, a vector of spatial distribution of a human face image and a vector of estimates of the presence of synthetic changes are combined into a common two-dimensional matrix. But their unification occurs only in frames on which there is a face of a given person. An example is shown in Table 2 for FIG. 3A. Such a combination is recommended for the case when a person is present in one consecutive series of frames.

One of the options for generating a notification of the presence of changes in the video, in the method of calculating the overall score of synthetic changes in the images of the faces of the j-th person at step 105 using a trained model that uses a matrix of combining spatio-temporal representation vectors and a score vector, is shown in Figure 8.

и пространственного распределения. Пример данной схемы приведен на Фиг. 5. В этом случае общая оценка синтетического изменения лица j-го человека строится только на векторе оценок синтетических изменений. Для формирования общей оценки может использоваться отдельная модель машинного обучения или отдельный алгоритм.[0066] In another particular embodiment of the invention, at step (105), the score vector characterizing that the face image in the frame was subject to synthetic changes is analyzed separately from the vectors

and spatial distribution. An example of this circuit is shown in Fig. 5. In this case, the overall assessment of the synthetic change in the face of the j-th person is based only on the vector of estimates of synthetic changes. A separate machine learning model or a separate algorithm can be used to generate the overall score.

[0067] In one of the particular embodiments of the invention shown in FIG. 5, the score vector is input to the separately trained model. This model can be a recurrent neural network, a convolutional neural network, a fully connected neural network. In such cases, a vector of a certain length can be used. If the resulting vector of face image ratings is less than the specified length of the vector, then such a vector is supplemented with values, for example, 0.5 from a certain end. If the vector is larger than the specified length, then it is cut off from a certain end.

[0068] In another particular embodiment of the invention, the number of ratings at predetermined intervals or the frequency of rating intervals is counted. For example, intervals are taken with a step of 0.1: [0-0.1; 0.1-0.2; 0.2-0.3; 0.3-0.4; 0.4-0.5; 0.5-0.6; 0.6-0.7; 0.7-0.8; 0.8-0.9; 0.9-1] and the frequency of estimates from the vector in these intervals is calculated. The obtained values are fed into a machine learning model, for example, support vectors (SVM), K-neighbors (K-nearest neighbor), linear (non-linear) regression, classification trees model. The description of the invention does not limit the form of the machine learning model, but describes its application to the received score vector.

[0069] In another particular embodiment of the invention, an overall score of synthetic changes in human face images is obtained by averaging a vector of scores, or is obtained by extracting the maximum value, either over the entire vector or over a part of it.

характеристики изображений лица человека. Общая пространственная характеристика рассчитывается как средняя по пространственному вектору данного лица. Общая

характеристика получается как длина вектора

характеристики по отношению к длине видео, то есть, является долей времени присутствия данного человека на видео от всего времени на видео. В другом частном варианте для расчета общей пространственной характеристики выбирается максимальное значение или минимальное.[0070] In one of the particular embodiments of the invention shown in FIG. 5, general spatial and

characteristics of human face images. The general spatial characteristic is calculated as an average over the spatial vector of a given person. General

the characteristic is obtained as the length of the vector

characteristics in relation to the length of the video, that is, is the proportion of the time the presence of a given person on the video from the total time on the video. In another particular variant, the maximum value or the minimum value is selected to calculate the overall spatial characteristic.

[0071] In step (106), a final synthetic face change presence score is calculated for the entire video. This estimate is built using each overall estimate of synthetic changes in images of people's faces. In other words, in steps (104)-(105), we obtain synthetic change scores for each person in the video (an individual person is highlighted in step 103), and in step (106), from the scores for people, we calculate a score for the video. This stage of the cumulative analysis of the ratings of all people in the video allows you to improve the quality of the invention compared to existing ones. For example, if there are a lot of people on the video and for all of them we have a high synthetic change score, then the video under study is probably very compressed, and we have a false positive decision of the models when analyzing face images. The cumulative analysis of the ratings in step (106) would then generate a final rating for the video as "video without synthetic changes".

[0072] In one of the private embodiments of the invention, shown in Fig. 1, using the total scores of the faces of all people to form the final score of the video can be applied as follows:

- The weighted average value of the estimates of the people's faces used is determined. In one particular embodiment of the invention, the weights for ratings can be the product of the average size of the images of the faces of a given person and the share of the presence time on the video.

- Calculates a simple average of the estimates of synthetic changes in the images of the used people's faces.

- The maximum rating is formed among the ratings of the people's faces used.

[0073] In another particular embodiment of the invention, a trained model may be used for the above example. This model can represent a support vector machine (SVM), K-neighbors (K-nearest neighbor), linear (non-linear) regression, a classification tree model, one or more neural networks. Such a model can take as input a vector (vector representation of data) that characterizes the amount of use of intervals of estimates of synthetic changes by faces.

характеристики изображений лица человека. На этапе (106) эти характеристики сравниваются с соответствующими граничными значениями. Если по итоговой характеристике размер изображения лица или время его присутствия меньше граничного значения, то оценка данного лица человека не учитывается при расчете вероятности синтетического изменения видео (оценка None). Схема данного примера приведена на Фиг. 6. Оставшиеся оценки синтетических изменений лиц людей анализируются далее выше описанными способами.[0074] In another particular embodiment of the invention, an example of the steps of which are shown in Figure 5, general spatial and

characteristics of human face images. At step (106) these characteristics are compared with the corresponding boundary values. If, according to the final characteristic, the size of the face image or the time of its presence is less than the boundary value, then the assessment of this person's face is not taken into account when calculating the probability of a synthetic change in the video (None score). The schematic of this example is shown in Fig. 6. The remaining estimates of synthetic changes in people's faces are analyzed further by the methods described above.

[0075] In another particular embodiment of the invention, two-dimensional matrices of vector representations of the spatiotemporal distribution and estimates of synthetic changes in the faces of various people, the formation of which is described above, are fed to the input of stage (106), where the formation of the final assessment of the presence of synthetic changes in the faces of people in the video is performed. This step can also be performed using a single machine learning model or an ensemble of trained models.

[0076] At step (107), an integral assessment of the presence of a synthetically modified face image on the video is formed based on the final assessment of the presence of synthetic changes in people's faces in the video. For this, at least one final assessment of the presence of synthetic changes in faces in the video is used, which is formed according to a separate model for classifying and detecting synthetic changes in faces. Upon completion of step (107), a notification about the presence of a synthetically modified face in the video is generated.

[0077] The notification may be displayed directly in the graphical user interface, for example, during an online conference (Zoom, Skype, MS Teams). Also, the notification may be displayed directly in the synthetic face change detection area, for example, in the area depicting a person's face. An additional effect of the application of the invention may be its use in biometric control systems, for example, when receiving services (for example, banking services) or access (access control system, turnstile with a biometric sensor). When a synthetically modified face image is detected, access or the requested action is blocked by the user. In this case, user authentication data may be additionally requested, selected from the group: login, code, password, two-factor authentication, or combinations thereof.

[0078] The claimed solution can be used in systems for monitoring the media space and analyzing social media and the media, to identify publicly known people (first persons of the state, media personalities, famous people, etc.), on which they can be compromised. Such systems will be the source of the received video for its subsequent analysis, and if synthetic changes in the images of the faces of such people are detected, they or the relevant service may be notified of the falsely generated information. For this type of notification, information about the time of the detected event, the source of the event can also be stored.

[0079] In one particular embodiment of the invention, several models for detecting synthetic changes in face images are used, each of which, at least one model, is trained on a different algorithm for generating synthetic changes.

[0080] In another particular embodiment, an ensemble of models is trained for each algorithm for generating synthetic changes. Estimates from several models in this ensemble are averaged.

[0081] For the final classification, the obtained scores are processed by an integral classifier, which makes it possible to reveal hidden relationships between model predictions for various algorithms for generating synthetic changes. This quality makes it possible to achieve a super-additive effect (synergistic) and improve the quality of video detection with the presence of synthetic changes in face images. The general scheme is shown in Fig. 7. A more detailed diagram is shown in FIG. 8.

[0082] In another particular embodiment of the invention, the integral classifier generates not only an integral assessment of the presence of synthetic changes in people's faces in the video, but also the most probable algorithm by which these synthetic changes were created. This example is shown in Fig. 7.

[0083] In FIG. 9 shows a general view of a computing device (600) suitable for implementing the claimed solution. Device (600) may be, for example, a computer, server, or other type of computing device that can be used to implement the claimed technical solution. Including being part of a cloud computing platform.

[0084] In general, the computing device (600) comprises one or more processors (601) connected by a common information exchange bus, memory means such as RAM (602) and ROM (603), input/output interfaces (604), input devices / output (605), and a device for networking (606).

[0085] The processor (601) (or multiple processors, multi-core processor) may be selected from a variety of devices currently widely used, such as Intel™, AMD™, Apple™, Samsung Exynos™, MediaTEK™, Qualcomm Snapdragon™, and etc. The processor (601) can also be a graphics processor such as Nvidia, AMD, Graphcore, etc.

[0086] RAM (602) is a random access memory and is designed to store machine-readable instructions executable by the processor (601) to perform the necessary data logical processing operations. RAM (602) typically contains the executable instructions of the operating system and associated software components (applications, program modules, etc.).

[0087] A ROM (603) is one or more persistent storage devices such as a hard disk drive (HDD), a solid state drive (SSD), flash memory (EEPROM, NAND, etc.), optical storage media ( CD-R/RW, DVD-R/RW, BlueRay Disc, MD), etc.

[0088] Various types of I/O interfaces (604) are used to organize the operation of device components (600) and organize the operation of external connected devices. The choice of appropriate interfaces depends on the particular design of the computing device, which can be, but not limited to: PCI, AGP, PS/2, IrDa, FireWire, LPT, COM, SATA, IDE, Lightning, USB (2.0, 3.0, 3.1, micro, mini, type C), TRS/Audio jack (2.5, 3.5, 6.35), HDMI, DVI, VGA, Display Port, RJ45, RS232, etc.

[0089] To ensure user interaction with the computing device (600), various means (605) of I/O information are used, for example, a keyboard, a display (monitor), a touch screen, a touchpad, a joystick, a mouse, a light pen, a stylus, touch panel, trackball, speakers, microphone, augmented reality, optical sensors, tablet, indicator lights, projector, camera, biometric identification tools (retinal scanner, fingerprint scanner, voice recognition module), etc.

[0090] The networking means (606) enables data to be transmitted by the device (600) via an internal or external computer network, such as an Intranet, Internet, LAN, and the like. As one or more means (606) can be used, but not limited to: Ethernet card, GSM modem, GPRS modem, LTE modem, 5G modem, satellite communication module, NFC module, Bluetooth and / or BLE module, Wi-Fi module and others

[0091] Additionally, satellite navigation tools in the device (600) can also be used, for example, GPS, GLONASS, BeiDou, Galileo.

[0092] The submitted application materials disclose preferred examples of the implementation of the technical solution and should not be construed as limiting other, particular examples of its implementation that do not go beyond the scope of the requested legal protection, which are obvious to specialists in the relevant field of technology.

Claims (28)

1. A computer-implemented method for determining synthetically modified images of faces in a video, which is performed using a processor and contains the following steps: a) obtaining at least one image from the video; b) detecting images of faces in said image; c) calculating a vector representation of the geometric characteristics of the detected face images using at least a face reference point comparison algorithm to determine images of at least one person's face; d) using frame-by-frame video analysis, the spatio-temporal significance of each image of the face of each person in said image is calculated, which is defined as a vector representation of the spatial characteristic of the face, characterizing the size of the face area in relation to the frame, and a vector representation of the temporal characteristic of the face image, characterizing the display time the analyzed image of the face on the video frames; e) calculating a vector of estimates of the probability of synthetic changes for images of human faces, characterizing the presence of synthetic changes in the images of faces of this person in each frame; f) calculating an overall synthetic change probability score based on vector representations of the spatial, temporal, and synthetic change score vectors for each person's face images in the video; g) forming a final assessment of the presence in the video of a synthetic image change of at least one face; h) forming an integral estimate of the presence of a synthetically modified face image in the video according to at least one final assessment of the model and generating a notification about the presence of a synthetically modified face in the video. 2. The method according to claim 1, characterized in that steps c)-h) are performed by a machine learning model or ensemble of models, while the machine learning model or ensemble of models is trained on a data set containing synthesized images of people's faces. 3. The method according to claim 2, characterized in that the machine learning model uses the automatic markup correction function, which corrects the incorrect markup of each face in the frames by comparing the images of faces on the synthesized video with their images on the original video. 4. The method according to claim 3, characterized in that the faces are compared based on the value of the vector proximity of the reference points that form the geometric characteristics of the original face image and the synthesized image based on it. 5. The method according to claim 3, characterized in that the face comparison is carried out by analyzing the coordinates of the areas of the original face image and the synthesized face image. 6. The method according to claim 1, characterized in that the spatiotemporal significance is calculated as a general matrix based on the values of vector representations, and the assessment of the presence of synthetic changes in the images of the faces of an individual is formed by a machine learning model based on the obtained general matrix. 7. The method according to claim 2, characterized in that the ensemble of machine learning models consists of a group of models, each of which is trained to identify a specific algorithm for generating synthetic images. 8. The method according to claim 7, characterized in that it contains an integral classifier that receives as input estimates generated using models included in the ensemble. 9. The method according to claim 8, characterized in that the final grade is calculated using an integral classifier. 10. The method according to claim 9, characterized in that the algorithm for generating a synthetic face image in the analyzed video stream is additionally determined. 11. The method of claim. 1, characterized in that the video is an online video conference. 12. The method according to claim 11, characterized in that when a synthetically modified face image is detected, a notification is generated in its display area. 13. The method according to p. 11, characterized in that when determining a synthetically modified face image, the connection with this user is blocked. 14. The method according to claim 1, characterized in that the analyzed image is obtained from a biometric identification or biometric authentication system. 15. The method according to claim 14, characterized in that when determining a synthetically modified face image, access or the requested action is blocked from the user. 16. The method according to claim 14, characterized in that, when determining a synthetically modified face image, they additionally request user authentication data selected from the group: login, code, password, two-factor authentication, or combinations thereof. 17. The method according to claim 14, characterized in that a signal is generated in the form of a quantitative estimate of the probability of the presence of a synthetically modified face image. 18. The method according to claim. 1, characterized in that the images are obtained from a video system for monitoring the media space and analyzing social media and media, which performs content verification in social media and media. 19. The method according to claim 18, characterized in that when the synthetically modified face image is determined, a notification is generated to inform the person who was subject to the creation of the modified face image. 20. A system for determining synthetically modified images of faces in a video, containing at least one processor and at least one memory storing machine-readable instructions that, when executed by the processor, implement the method according to any one of paragraphs. 1-19.

Images