CN118474327A - Video editing method, device, equipment and storage medium - Google Patents

Abstract

The application discloses a video editing method, a device, equipment and a storage medium, relating to the field of computer vision, wherein the method comprises the following steps: acquiring surrounding video of an inserted target, wherein the surrounding video comprises video frames of the inserted target under different visual angles; inputting the surrounding video into a preset three-dimensional reconstruction model to obtain a target three-dimensional model, wherein the preset three-dimensional reconstruction model is obtained by training each video frame obtained by segmenting the surrounding video; obtaining video reconstruction information according to optical flow information of a video to be inserted in combination with a preset optimization algorithm, wherein the video reconstruction information comprises camera inner and outer parameter information and dense point cloud information; and implanting the target three-dimensional model into a user selected area to be inserted into the video according to the camera inside-outside parameter information and the dense point cloud information to obtain the edited video. Because the user only needs to provide the surrounding video of the insertion target, the video editing process from the three-dimensional model production to the video insertion integration can be realized, and compared with the prior art, the video editing speed is greatly improved.

Description

Video editing method, device, equipment and storage medium

Technical Field

The present application relates to the field of computer vision technology, and in particular to a video editing method, apparatus, device and storage medium.

Background Art

With the development of video generation technology, video editing methods that insert three-dimensional models into existing videos have been applied in film and television production, advertising production, virtual reality and other fields. Users can perform three-dimensional reconstruction of the insertion target and then insert the generated three-dimensional model into the video to be inserted to achieve a real fusion effect.

However, traditional methods of object 3D reconstruction often rely on professional scanning equipment and special shooting environments. The reconstruction cost is high and time-consuming, and the implantation of 3D models into existing videos requires the use of multiple post-production software. Therefore, the video editing process of generating 3D models for real objects and implanting 3D models into existing videos is cumbersome and inefficient, and difficult for general users.

The above contents are only used to assist in understanding the technical solution of the present application and do not constitute an admission that the above contents are prior art.

Summary of the invention

The main purpose of this application is to provide a video editing method, which aims to solve the technical problem that the existing video editing method of inserting a three-dimensional model into an existing video is cumbersome and inefficient, and is relatively difficult for general users.

To achieve the above object, the present application proposes a video editing method, which includes:

Acquire a surround video of the inserted target, wherein the surround video includes video frames of the inserted target at different viewing angles;

Inputting the surround video into a preset 3D reconstruction model to obtain a target 3D model, wherein the preset 3D reconstruction model is trained by each video frame obtained by segmenting the surround video;

Obtaining video reconstruction information based on the optical flow information of the video to be inserted and a preset optimization algorithm, wherein the video reconstruction information includes camera internal and external parameter information and dense point cloud information;

The target three-dimensional model is implanted into a user-selected area in the video to be inserted according to the camera internal and external parameter information and the dense point cloud information to obtain an edited video.

In one embodiment, the preset 3D reconstruction model includes a geometric coding perceptron and a texture coding perceptron; the training step of the preset 3D reconstruction model includes:

Initialize a tetrahedral mesh, input the three-dimensional coordinates of the tetrahedral mesh into the geometric encoding perceptron, obtain corresponding geometric information, and convert the target mesh according to the geometric information;

Inputting the three-dimensional coordinates of the target grid into the texture encoding perceptron to obtain texture information;

Inputting the environment lighting map, the camera pose information of the surround video, the target mesh and the texture information into a preset differentiable renderer to obtain a current rendering image;

Perform loss calculation according to the current rendering image and the foreground image of the inserted target at different viewing angles to obtain a loss gradient, and optimize the geometry encoding perceptron, the texture encoding perceptron and the ambient lighting map through the loss gradient based on a gradient feedback mechanism;

Based on the optimization goal of minimizing empirical risk, a preset three-dimensional reconstruction model is obtained.

In one embodiment, before the steps of initializing the tetrahedral mesh, inputting the three-dimensional coordinates of the tetrahedral mesh into the geometric encoding perceptron, obtaining corresponding geometric information, and converting the target mesh according to the geometric information, the steps include:

When receiving a click instruction from the user, generating a mask area corresponding to the insertion target in the current video frame of the surround video according to the click instruction;

Segmentation is performed in other video frames obtained by segmenting the surround video according to the mask area to obtain foreground images of the inserted target at different viewing angles.

In one embodiment, the step of obtaining video reconstruction information based on the optical flow information of the video to be inserted in combination with a preset optimization algorithm includes:

Inputting a plurality of video frames corresponding to the video to be inserted into the optical flow prediction model to determine the optical flow displacement value of each of the video frames;

When the optical flow displacement value is greater than a preset threshold, adding the video frame corresponding to the optical flow displacement value to a key frame set;

A preset optimization algorithm is used in the key frame set to determine the video reconstruction information to obtain camera internal and external parameter information and dense point cloud information.

In one embodiment, the step of determining video reconstruction information using a preset optimization algorithm for each video frame in the key frame set to obtain camera internal and external parameter information and dense point cloud information includes:

Dividing the key frame set into an online optimization set and an offline optimization set;

In the online optimization set, a preset optimization algorithm is used to determine the relative position and depth map of the camera and the camera intrinsic parameters corresponding to each video frame;

In the offline optimization set, a preset optimization algorithm is used to update the camera relative pose, the depth map and the camera intrinsic parameters to obtain camera internal and external parameter information and dense point cloud information.

In one embodiment, the step of implanting the target three-dimensional model into the user-selected area in the video to be inserted according to the camera internal and external parameter information and the dense point cloud information to obtain the edited video includes:

Upon receiving a selection instruction from a user, generating a user-selected area in the video to be inserted;

Back-projecting the pixel coordinates of the area selected by the user into the camera coordinate system corresponding to the video to be inserted according to the camera internal and external parameter information and the dense point cloud information to obtain the three-dimensional point cloud coordinates of the area selected by the user;

Determine the three-dimensional plane normal vector corresponding to the video to be inserted, and pre-process the target three-dimensional model in combination with the three-dimensional point cloud coordinates of the area selected by the user;

Different viewing angle pictures corresponding to the preprocessed target three-dimensional model are rendered according to the internal and external parameter information of the video, and the edited video is synthesized according to each of the viewing angle pictures and a number of video frames corresponding to the video to be inserted.

In one embodiment, the step of determining the three-dimensional plane normal vector corresponding to the video to be inserted, and preprocessing the target three-dimensional model in combination with the three-dimensional point cloud coordinates of the user-selected area includes:

Based on a random sampling consensus algorithm, a three-dimensional plane normal vector is determined according to a camera coordinate system corresponding to the video to be inserted;

The target three-dimensional model is preprocessed based on the three-dimensional plane normal vector and combined with the three-dimensional point cloud coordinates of the user-selected area.

In addition, to achieve the above-mentioned purpose, the present application also proposes a video editing device, which includes: a model generation module, a video reconstruction module and a model implantation module;

The model generation module is used to obtain a surround video of the inserted target, wherein the surround video includes video frames of the inserted target at different viewing angles;

The model generation module is further used to input the surround video into a preset 3D reconstruction model to obtain a target 3D model, wherein the preset 3D reconstruction model is trained by each video frame obtained by segmenting the surround video;

The video reconstruction module is used to obtain video reconstruction information according to the optical flow information of the video to be inserted in combination with a preset optimization algorithm, wherein the video reconstruction information includes camera internal and external parameter information and dense point cloud information;

The model implantation module is used to implant the target three-dimensional model into the user-selected area in the video to be inserted according to the camera internal and external parameter information and the dense point cloud information to obtain the edited video.

In addition, to achieve the above objectives, the present application also proposes a video editing device, which includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program is configured to implement the steps of the video editing method described above.

In addition, to achieve the above-mentioned purpose, the present application also proposes a storage medium, which is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, the steps of the video editing method described above are implemented.

The present application provides a video editing method, which includes obtaining a surround video of an inserted target, wherein the surround video includes video frames of the inserted target at different viewing angles; inputting the surround video into a preset 3D reconstruction model to obtain a target 3D model, wherein the preset 3D reconstruction model is obtained by training each video frame obtained by segmenting the surround video; obtaining video reconstruction information based on the optical flow information of the video to be inserted in combination with a preset optimization algorithm, wherein the video reconstruction information includes camera internal and external parameter information and dense point cloud information; and implanting the target 3D model into a user-selected area in the video to be inserted based on the camera internal and external parameter information and the dense point cloud information to obtain an edited video. Since the user in the present application only needs to provide a surround video of the inserted target, the target 3D model can be obtained from the preset 3D reconstruction model, which simplifies the traditional 3D reconstruction process and can reduce the cost of 3D reconstruction. Furthermore, the insertion of the target 3D model is realized in the user-selected area of the video to be inserted based on the video reconstruction information, thereby realizing a video editing process that integrates 3D model production and video insertion, which greatly improves the speed of video editing compared to the existing methods.

BRIEF DESCRIPTION OF THE DRAWINGS

The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

FIG1 is a schematic diagram of a flow chart of a video editing method according to an embodiment of the present invention;

FIG2 is a schematic diagram of a flow chart of a second embodiment of a video editing method of the present application;

FIG3 is a flow chart of a third embodiment of the video editing method of the present application;

FIG4 is a schematic diagram of a module structure provided by Embodiment 1 of the video editing device of the present application;

FIG. 5 is a schematic diagram of the device structure of the hardware operating environment involved in the video editing method in an embodiment of the present application.

The purpose, features and advantages of this application will be further described in conjunction with the embodiments and with reference to the accompanying drawings.

DETAILED DESCRIPTION

It should be understood that the specific embodiments described herein are only used to explain the technical solutions of the present application and are not used to limit the present application.

In order to better understand the technical solution of the present application, a detailed description will be given below in conjunction with the accompanying drawings and specific implementation methods.

The main solution of the embodiment of the present application is: obtaining a surround video of the inserted target, the surround video containing video frames of the inserted target at different perspectives; inputting the surround video into a preset 3D reconstruction model to obtain a target 3D model, the preset 3D reconstruction model is trained by each video frame obtained by segmenting the surround video; obtaining video reconstruction information based on the optical flow information of the video to be inserted in combination with a preset optimization algorithm, the video reconstruction information including camera internal and external parameter information and dense point cloud information; rendering the target 3D model to the user-selected area in the video to be inserted based on the camera internal and external parameter information and the dense point cloud information to obtain an edited video.

The video editing process of generating 3D models for real objects and inserting them into existing videos is difficult and costly for ordinary people. Reconstructing the precise geometry and surface material of real objects often requires professional 3D scanning equipment and special shooting environments. Inserting a 3D model into an existing video usually requires a long time for professional film and television post-production engineers, such as first using Boujou to restore the sparse point cloud and camera internal and external parameters of the video, and then manually aligning the object placement plane with the target plane in the video in 3D rendering software such as 3ds Max and Blender, and using the camera parameters restored by Boujou to render 3D pictures from different perspectives frame by frame. In addition to video post-production software, current AR augmented reality technology SDKs such as Android's ARcore and Google's ARKit need to first use the phone's precise camera internal parameters and auxiliary sensors such as IMU and radar to complete the camera pose recovery and scene geometry reconstruction before rendering the 3D object into the real-time video stream. The overall process is cumbersome and inefficient.

In this application, the user only needs to provide a surround video of the insertion target, and the target 3D model can be obtained from the preset 3D reconstruction model, which simplifies the traditional 3D reconstruction process and can reduce the 3D reconstruction cost. Then, the target 3D model can be inserted into the user-selected area of the video to be inserted according to the video reconstruction information, thereby realizing an integrated video editing process from 3D model production to video insertion, which greatly improves the speed of video editing compared to the existing method.

It should be noted that the execution subject of this embodiment can be a computing service device with interactive display, data processing, network communication and program running, such as a tablet computer, a personal computer, a mobile phone, etc., or other electronic devices that can achieve the same or similar functions. This embodiment is not limited to this. The following uses a video editing device (hereinafter referred to as "editing device") as an example to illustrate this embodiment and the following embodiments.

Based on this, the embodiment of the present application provides a video editing method, refer to Figure 1, which is a flow chart of the video editing method embodiment 1 of the present application. The video editing method includes steps S10 to S40:

Step S10: Acquire a surround video of the inserted target, wherein the surround video includes video frames of the inserted target at different viewing angles.

It should be noted that the inserted target is the target object to be reconstructed into a three-dimensional model. The user can first shoot a video around the inserted target so that the captured surround video includes different perspectives of the inserted target, and then upload the surround video to the editing device.

Step S20: inputting the surround video into a preset 3D reconstruction model to obtain a target 3D model, wherein the preset 3D reconstruction model is trained by each video frame obtained by segmenting the surround video.

It should be understood that the preset 3D reconstruction model can reconstruct the 3D mesh model and surface material of the inserted target through the surround video shot by the user. The preset 3D model can be a de-rendering model based on a neural network, which can include a geometry encoding perceptron for extracting the 3D mesh model corresponding to the inserted target, and a texture encoding perceptron for obtaining the surface material of the inserted target.

It should also be noted that the video frames containing the inserted target at different viewing angles can be used as supervision data for training the preset 3D reconstruction model. The model training process is specifically described below. The model training process includes steps A01 to A05:

Step A01: Initialize a tetrahedral mesh, input the three-dimensional coordinates of the tetrahedral mesh into the geometric encoding perceptron, obtain corresponding geometric information, and convert the geometric information to obtain a target mesh.

It should be noted that the Signed Distance Function (SDF) is a mathematical representation method for representing the shape of a geometric object. It defines a function that returns a signed distance value from the surface of a body for each point in space. This value can be positive, negative or zero, and is used to represent the positional relationship of the points. The geometric encoding perceptron can be a multi-layer perceptron (MLP). In the geometric encoding perceptron, the MLP network structure can be used to learn and approximate the SDF, thereby realizing the representation and generation of the geometric shape of a three-dimensional object.

It should be understood that the geometric information may be the SDF value corresponding to each vertex of the tetrahedral mesh, and the target mesh is a textureless mesh obtained by converting the SDF.

In the specific implementation, we can first initialize a tetrahedral mesh with a fixed resolution, then input the vertex coordinates of the tetrahedral mesh into the geometric encoding perceptron, convert them into corresponding SDF values, and then use Marching tetrahedrons (an isosurface extraction method that can extract a mesh model based on the SDF values of coordinates in space) to obtain a texture-free mesh.

Step A02: Input the three-dimensional coordinates of the target grid into the texture coding perceptron to obtain texture information.

It should be noted that the texture coding perceptron can use the MLP grid structure to encode all material parameters, learn the automatic texture parameterization process of the grid surface, and obtain the pixel value corresponding to each grid vertex.

In the specific implementation, the texture encoding perceptron first takes the mesh vertices of the mesh as input, outputs the basic color, reflection coefficient and normal coefficient, and determines the pixel values corresponding to each mesh vertex of the target mesh; then, it converts the color map, reflection map and normal map corresponding to the mesh through UV mapping to obtain the texture information.

Step A03: Input the environment lighting map, the camera pose information of the surround video, the target mesh and the texture information into a preset differentiable renderer to obtain a current rendering image.

It should be noted that, while initializing the tetrahedral mesh, a fixed-size ambient lighting (High Dynamic Range, HDR) map may also be initialized for subsequent learning of ambient lighting from image observations.

It should be understood that a common camera pose estimation algorithm, such as the colmap algorithm, may be used to perform Structure From Motion (SFM) recovery on each of the surround videos to obtain the camera pose information of the surround video.

It should also be noted that the preset differentiable renderer can adopt the diffrast differentiable renderer, which can render a four-channel RGBA image corresponding to the camera pose from the input mesh model with material, environment HDR and camera pose information.

Step A04: Perform loss calculation based on the current rendering and the foreground image of the inserted target at different viewing angles to obtain a loss gradient, and optimize the geometry encoding perceptron, the texture encoding perceptron and the ambient lighting map through the loss gradient based on a gradient feedback mechanism.

It should be noted that, in order to obtain the foreground image of the inserted target at different viewing angles, before step A04, a background removal process of the surrounding video may be included, and the steps of the process include steps A001 to A002:

Step A001: upon receiving a click instruction from the user, generating a mask area corresponding to the insertion target in the current video frame of the surround video according to the click instruction.

Step A002: Segment the other video frames obtained by segmenting the surround video according to the mask area to obtain foreground images of the inserted target at different viewing angles.

It should be understood that a video marker segmentation model such as Track Anything can be used to implement video background removal in surround video. Specifically, the user can click on the uploaded surround video in the interactive interface of the editing device, select the insertion target to be 3D modeled in the current video frame, segment the insertion target by the video marker segmentation model and generate the corresponding mask area, and then automatically track the insertion target in other video frames using the frame tracking target technology to obtain the foreground four-channel image segmented from each frame of the surround video, that is, the foreground image under different viewing angles.

It should be noted that the foreground image under different viewing angles can be used as the ground truth and the four-channel RGBA image corresponding to the camera pose rendered by the differentiable renderer is used for loss calculation. The obtained loss gradient is back-propagated to the geometry encoding perceptron, texture encoding perceptron and initialized environment HDR map for parameter update optimization.

Step A05: Based on the optimization goal of minimizing the empirical risk, a preset three-dimensional reconstruction model is obtained.

It should be noted that the model training process can be regarded as an optimization task for the target shape representation of a triangular mesh with geometric shape, texture material and lighting. The above loss calculation formula can be:

L _total ＝l ₁ (S,R)+L _light +L _mat

Where l ₁ (S, R) is the l ₁ loss of the current rendering and the foreground image R. _{The L light} regularization term constrains the brightness value ci of each channel of the ambient HDR image _to not differ too much, which is closer to the environment of most white light in the real world. The L _mat regularization term constrains the base colors of adjacent mesh vertices (k _d (x _surf ) and k _d (x _surf +ε)) to not differ too much, making the texture smoother.

In the specific implementation, based on the optimization goal of minimizing empirical risk, the preset differentiable renderer diffrast is used to perform the above optimization tasks, and the relevant parameters of the geometric shape, texture material and lighting are updated through the back propagation of the loss gradient to realize the optimization process of the geometric coding perceptron, texture coding perceptron and the initialized environment HDR map, and finally a preset 3D reconstruction model with better effect is obtained, and then the target 3D model corresponding to the inserted target is constructed through the preset 3D reconstruction model.

Step S30: Obtain video reconstruction information according to the optical flow information of the video to be inserted in combination with a preset optimization algorithm, wherein the video reconstruction information includes camera internal and external parameter information and dense point cloud information.

It should be noted that the optical flow information can be used to describe the pixel motion in the video to be inserted. According to the optical flow information, the depth information corresponding to each video frame in the video to be inserted can be obtained, and then the dense point cloud of the video scene to be inserted can be reconstructed.

It should be understood that the preset optimization algorithm can adopt a self-calibration bundle adjustment method, and its optimization task is to minimize the reprojection error. It can be solved by, for example, the gradient descent method, the Newton method, or the Gauss-Newton method to achieve the solution of the camera internal and external parameter information to be inserted into the video.

Step S40: implanting the target three-dimensional model into the user-selected area in the video to be inserted according to the camera internal and external parameter information and the dense point cloud information to obtain an edited video.

It should be understood that the user can select the video to be inserted in the interactive interface of the editing device to determine the insertion position of the target three-dimensional model.

In a specific implementation, after obtaining the camera's internal and external parameter information and dense point cloud information, the coordinate point information selected by the user in the current frame of the video to be inserted can be received to generate a user-selected area, and then based on the user-selected area, the corresponding perspective images of the target three-dimensional model in each video frame of the video to be inserted are rendered, and multi-frame image rendering is completed and the final result video is synthesized to obtain the edited video.

In this embodiment, the user only needs to provide a surround video of the insertion target, and click to select the target to be reconstructed in the surround video, and the target three-dimensional model can be obtained from the preset three-dimensional reconstruction model, which simplifies the traditional three-dimensional reconstruction process and can reduce the three-dimensional reconstruction cost. Then, the target three-dimensional model can be inserted into the user-selected area of the video to be inserted according to the video reconstruction information, realizing a video editing process from three-dimensional model production to video insertion, which greatly improves the video editing speed compared to the existing method.

Based on the first embodiment of the present application, in the second embodiment of the present application, the same or similar contents as those in the first embodiment can be referred to the above description, and will not be described in detail later. On this basis, please refer to FIG. 2, which is a flow chart of the second embodiment of the video editing method of the present application, step S30, specifically including steps S301 to S303:

Step S301: inputting a plurality of video frames corresponding to the video to be inserted into an optical flow prediction model to determine the optical flow displacement value of each of the video frames.

It should be noted that the optical flow prediction model can use RAFT (Recurrent All-Pairs Field Transforms, a deep neural network for optical flow estimation). RAFT uses a neural network structure and deep learning technology to learn pixel-level motion information from image sequences. Compared with traditional optical flow algorithms, RAFT can better handle complex motion situations. It uses a recurrent neural network (RNN) to process image sequence information, and introduces technologies such as local block decoupling attention mechanism and deformable convolution, which is conducive to improving the accuracy and stability of optical flow estimation.

In a specific implementation, the video to be inserted is input into RAFT, and the optical flow displacement value of each video frame relative to the previous video frame can be obtained, thereby realizing the calculation of the optical flow information of each video frame of the video to be inserted.

Step S302: when the optical flow displacement value is greater than a preset threshold, adding the video frame corresponding to the optical flow displacement value to a key frame set.

In a specific implementation, an optical flow threshold can be set, and video frames whose optical flow displacement values are higher than the optical flow threshold are added to the key frame set, and video frames whose optical flow displacement values are not higher than the optical flow threshold are skipped. After traversing each video frame, the key frame set corresponding to the video to be inserted is finally obtained.

Step S303: using a preset optimization algorithm in the key frame set to determine video reconstruction information, and obtaining camera internal and external parameter information and dense point cloud information.

It should be noted that, in order to improve the optimization efficiency, the optimization task of obtaining the above video reconstruction information can be divided into an online optimization part and an offline optimization part. Therefore, step S303 also includes steps S3031 to S3033:

Step S3031: Divide the key frame set into an online optimization set and an offline optimization set.

It should be noted that the online optimization set may be a set to be optimized that only selects the last 25 frames in the key frame set as the online local estimation task; the offline optimization set corresponds to the offline global optimization task and may include all the key frames in the key frame set.

Step S3032: using a preset optimization algorithm in the online optimization set to determine the relative camera pose, depth map, and camera intrinsic parameters corresponding to each video frame.

Step S3033: using a preset optimization algorithm in the offline optimization set to update the camera relative pose, the depth map and the camera intrinsic parameters to obtain camera intrinsic and extrinsic parameter information and dense point cloud information.

It should be noted that the online local estimation task can be performed synchronously when each video frame to be inserted into the video is input. After reading the entire video frame sequence and completing the online local estimation, the camera relative pose, depth map and camera intrinsic parameters corresponding to each video frame can be obtained, and then the offline global optimization task can be performed to achieve parameter fine-tuning and updating.

It should be understood that the set to be optimized includes the online optimization set and the offline optimization set, and the corresponding optimization tasks are the online local estimation task and the offline global optimization task. When performing the optimization task, for any two frames i and j in the set to be optimized whose optical flow displacement is within the specified range, the camera internal and external parameters and the dense point cloud can be estimated by minimizing the reprojection error through the self-calibration bundle adjustment method.

The task function of minimizing the reprojection error is:

Where P is the set to be optimized. _{G ij} is the relative camera pose between the i-th frame and the j-th frame. is the pixel coordinates in the i-th frame estimated by optical flow in the j-th frame. _{z i} is the depth map corresponding to the i-th frame. θ is the camera internal parameter. ∑ij is the estimated confidence weight. The larger ∑ij is, the more The more accurate the estimation is. The three-dimensional coordinates in the camera coordinate system can be projected to the pixel coordinate system according to the intrinsic parameters, and the pixel coordinates can be back-projected to the camera coordinate system according to the camera intrinsic parameters and their corresponding depth values. Since minimizing the reprojection error is a nonlinear optimization problem, ^the Gauss-Newton method can be used to iteratively optimize the relative pose G _ij , the key frame depth z _i and the camera intrinsic parameter θ.

In the specific implementation, after completing the above optimization tasks, the relative camera pose corresponding to each video frame, that is, the camera extrinsic parameters, the depth value of each video frame (dense point cloud information) and the camera intrinsic parameters corresponding to the video frame sequence can be restored to complete the video reconstruction of the video to be inserted.

This embodiment adopts the self-calibration bundle adjustment method combined with the optical flow algorithm RAFT in the video reconstruction process of the video to be inserted, which can restore the camera internal and external parameters of any continuous video. It is suitable for any video to be inserted with unknown camera parameters, and reconstructs a dense point cloud of the corresponding video scene, thereby enhancing the ability to locate weak texture areas and repeated texture areas in the video scene, and solving the problem that the sparse point cloud of the scene restored by traditional camera tracking software cannot locate weak texture areas and repeated texture areas in the video.

Based on the first embodiment and/or the second embodiment of the present application, in the third embodiment of the present application, the same or similar contents as those in the first embodiment and the second embodiment can be referred to the above description, and will not be described in detail later. On this basis, please refer to FIG. 3, which is a flowchart of the third embodiment of the video editing method of the present application, step S40, including steps S401 to S404:

Step S401: upon receiving a selection instruction from the user, generating a user-selected area in the video to be inserted.

It should be noted that the user can click on the interactive interface of the editing device to determine the insertion position of the target 3D model in the video to be inserted. Specifically, the user can click on the first frame of the video to be inserted to select four coordinate points to generate a user-selected area.

Step S402: back-projecting the pixel coordinates of the area selected by the user into the camera coordinate system corresponding to the video to be inserted according to the camera internal and external parameter information and the dense point cloud information to obtain the three-dimensional point cloud coordinates of the area selected by the user.

It should be noted that the pixel coordinates _ur of the user selected area can be back-projected to the three-dimensional point cloud coordinates under the camera coordinates according to the back-projection formula π ^-1 ( _ur , _zr , θ), where _zr is the dense point cloud depth value of the video to be inserted, to determine the three-dimensional point cloud coordinates of the user selected area in the camera coordinate system of the video to be inserted.

Step S403: Determine the three-dimensional plane normal vector corresponding to the video to be inserted, and pre-process the target three-dimensional model in combination with the three-dimensional point cloud coordinates of the area selected by the user.

It should be understood that the three-dimensional plane normal vector can be determined based on the camera coordinate system corresponding to the video to be inserted based on a random sampling consensus algorithm; based on the three-dimensional plane normal vector, the target three-dimensional model is preprocessed in combination with the three-dimensional point cloud coordinates of the user-selected area.

It should be noted that the RANdom SAmple Consensus (RANSAC) algorithm is an iterative optimization algorithm based on random sampling and consistency check. The normal vector can be estimated by fitting a plane model. The RANSAC algorithm can also filter out some pixels that are mistakenly selected by the user and noise values of the depth estimation.

It should also be noted that, considering that the plane normal vector estimated by the RANSAC algorithm may be upward or downward, additional constraints can be added to prevent the subsequent insertion of the target 3D model from being upside down. Specifically, for the estimated plane normal vector The world coordinate system coordinate c of the camera position and the coordinate p of any point on the plane need to satisfy If the condition is not met, the normal vector Do the reverse.

It should be understood that after obtaining the 3D plane normal vector, when preprocessing the target 3D model, the editing device can rotate the y-axis direction of the target 3D model to be in the same direction as the 3D plane normal vector to achieve a placement visual effect. Furthermore, in order to fit the bottom surface of the target 3D model to the user-selected area as closely as possible, the bottom surface center of the target 3D model can be moved to the plane center of the user-selected area based on the 3D point cloud coordinates of the user-selected area, and the size of the target 3D model can be adjusted according to the ratio of the distance of the user-selected area extending on the x-axis and z-axis to the width of the target 3D model in the x-axis and z-axis directions.

Step S404: Rendering preprocessed images of different viewing angles corresponding to the target three-dimensional model according to the internal and external parameter information of the video, and synthesizing the edited video according to each of the viewing angle images and a number of video frames corresponding to the video to be inserted.

It should be noted that after completing the preprocessing of the target three-dimensional model, the perspective pictures corresponding to each video frame to be inserted into the video can be rendered according to the camera intrinsic parameters and camera extrinsic parameters obtained in the above video reconstruction process. The different perspective pictures are RGBA images, and the RGBA images are consistent with the video frame pixels of the inserted video.

It should also be noted that, since the target 3D model is placed in the user-selected area when the first frame of the video to be inserted is initialized during the preprocessing process, the 2D results obtained by rendering the target 3D model in the RGBA images of different viewing angles are all located in the user-selected area.

Furthermore, considering that only the camera extrinsic parameters of the key frames are restored in the above video reconstruction process, for the non-key frames to be inserted into the video, their approximate camera extrinsic parameters can be obtained by linear interpolation of adjacent key frames.

It should also be noted that since the rendering in each video frame to be inserted into the video is relatively independent, the pytorch tensor-based renderer pytorch3d can be used to render multiple frames of images in parallel and synthesize the final result video, where the synthesis process of each rendered RGBA image and the corresponding video frame can also be implemented in parallel.

In a specific implementation, the camera intrinsic parameters corresponding to the video to be inserted and the camera intrinsic parameters corresponding to each video frame to be inserted can be used to render the RGBA image corresponding to each video frame, and the RGBA image and the video frame can be synthesized to obtain the result video frame, thereby obtaining the edited video.

Furthermore, the interactive interface of the editing device can also provide the user with a fine-tuning interface for the size and orientation of the target three-dimensional model, so that the user can further adjust the insertion effect of the target three-dimensional model.

In this embodiment, the two-dimensional pixel coordinates of the user-selected area can be back-projected into the world coordinate system and the three-dimensional plane parameters can be estimated, and the initial size of the target three-dimensional model can be adjusted according to the size of the user-selected area. The user only needs to generate the selected area by clicking on the first frame of the video to be inserted, without setting the camera and plane parameters in the three-dimensional rendering space as in traditional post-production software, and without manually aligning the plane to be video and the three-dimensional plane, providing a user-friendly video editing method that can meet the video editing needs of general users to insert a three-dimensional model into an existing video.

This embodiment also provides a video editing device. Referring to FIG. 4 , FIG. 4 is a schematic diagram of a module structure provided in Embodiment 1 of the video editing device of this application. The video editing device includes: a model generation module 401, a video reconstruction module 402, and a model implantation module 403.

The model generation module 401 is used to obtain a surround video of the inserted target, wherein the surround video includes video frames of the inserted target at different viewing angles;

The model generation module 401 is further used to input the surround video into a preset 3D reconstruction model to obtain a target 3D model, wherein the preset 3D reconstruction model is trained by each video frame obtained by segmenting the surround video, and the preset 3D reconstruction model includes a geometric coding perceptron and a texture coding perceptron;

The video reconstruction module 402 is used to obtain video reconstruction information according to the optical flow information of the video to be inserted in combination with a preset optimization algorithm, wherein the video reconstruction information includes camera internal and external parameter information and dense point cloud information;

The model implantation module 403 is used to implant the target three-dimensional model into the user-selected area in the video to be inserted according to the camera internal and external parameter information and the dense point cloud information to obtain the edited video.

Furthermore, the model generation module 401 is also used to initialize a tetrahedral mesh, input the three-dimensional coordinates of the tetrahedral mesh into the geometry coding perceptron to obtain corresponding geometry information, and convert the geometry information to obtain a target mesh; input the three-dimensional coordinates of the target mesh into the texture coding perceptron to obtain texture information; input the ambient lighting map, the camera pose information of the surround video, the target mesh and the texture information into a preset differentiable renderer to obtain a current rendering; perform loss calculation based on the current rendering and the foreground image of the inserted target at different viewing angles to obtain a loss gradient, and optimize the geometry coding perceptron, the texture coding perceptron and the ambient lighting map through the loss gradient based on a gradient feedback mechanism; obtain a preset three-dimensional reconstruction model based on the optimization goal of minimizing empirical risk.

Furthermore, the model generation module 401 is also used to generate a mask area corresponding to the insertion target in the current video frame of the surround video according to the click instruction when receiving a user's click instruction; and to segment other video frames obtained by segmenting the surround video according to the mask area to obtain foreground images of the insertion target at different perspectives.

Furthermore, the video reconstruction module 402 is also used to input several video frames corresponding to the video to be inserted into the optical flow prediction model to determine the optical flow displacement value of each of the video frames; when the optical flow displacement value is greater than a preset threshold, the video frame corresponding to the optical flow displacement value is added to a key frame set; a preset optimization algorithm is used in the key frame set to determine the video reconstruction information to obtain the camera internal and external parameter information and dense point cloud information.

Furthermore, the video reconstruction module 402 is also used to divide the key frame set into an online optimization set and an offline optimization set; in the online optimization set, a preset optimization algorithm is used to determine the camera relative pose, depth map and camera intrinsic parameters corresponding to each video frame; in the offline optimization set, a preset optimization algorithm is used to update the camera relative pose, the depth map and the camera intrinsic parameters to obtain camera internal and external parameter information and dense point cloud information.

Furthermore, the model implantation module 403 is also used to back-project the pixel coordinates of the user-selected area into the camera coordinate system corresponding to the video to be inserted according to the camera internal and external parameter information and the dense point cloud information, so as to obtain the three-dimensional point cloud coordinates of the user-selected area; determine the three-dimensional plane normal vector corresponding to the video to be inserted, and pre-process the target three-dimensional model in combination with the three-dimensional point cloud coordinates of the user-selected area; render different perspective images corresponding to the pre-processed target three-dimensional model according to the video internal and external parameter information, and synthesize the edited video according to each of the perspective images and several video frames corresponding to the video to be inserted.

Furthermore, the model implantation module 403 is also used to determine the three-dimensional plane normal vector based on the camera coordinate system corresponding to the video to be inserted based on a random sampling consensus algorithm; based on the three-dimensional plane normal vector, the target three-dimensional model is pre-processed in combination with the three-dimensional point cloud coordinates of the user-selected area.

In this embodiment, the user only needs to provide a surround video of the inserted target, and the model generation module can obtain the target 3D model corresponding to the inserted target, and the video reconstruction module restores the video reconstruction information including the camera internal and external parameters of the video to be inserted and the dense point cloud information. Finally, the target 3D model is inserted into the user-selected area according to the video reconstruction information in the model implantation module, and the edited video is rendered and synthesized. It can realize the video editing process from 3D model production to video insertion, meet the video editing needs of general users to insert 3D models into existing videos, and lower the threshold of video editing and increase the speed of video editing compared to the existing methods.

The present application provides a video editing device, which includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the video editing method in the above-mentioned embodiment one.

Reference is made to FIG5 , which is a schematic diagram of the device structure of the hardware operating environment involved in the video editing method in the embodiment of the present application. The video editing device in the embodiment of the present application may include but is not limited to mobile terminals such as mobile phones, laptop computers, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (Portable Application Descriptions), PMPs (Portable Media Players), vehicle-mounted terminals (such as vehicle-mounted navigation terminals), etc., and fixed terminals such as digital TVs, desktop computers, etc. The video editing device shown in FIG5 is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

As shown in FIG5 , the video editing device may include a processing device 1001 (e.g., a central processing unit, a graphics processor, etc.), which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM: Read Only Memory) 1002 or a program loaded from a storage device 1003 to a random access memory (RAM: Random Access Memory) 1004. In RAM 1004, various programs and data required for the operation of the video editing device are also stored. The processing device 1001, ROM 1002, and RAM 1004 are connected to each other via a bus 1005. An input/output (I/O) interface 1006 is also connected to the bus. Generally, the following systems can be connected to the I/O interface 1006: an input device 1007 including, for example, a touch screen, a touch pad, a keyboard, a mouse, an image sensor, a microphone, an accelerometer, a gyroscope, etc.; an output device 1008 including, for example, a liquid crystal display (LCD: Liquid Crystal Display), a speaker, a vibrator, etc.; a storage device 1003 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 1009. Communication device 1009 can allow the video editing device to communicate wirelessly or wired with other devices to exchange data. Although the video editing device with various systems is shown in the figure, it should be understood that it is not required to implement or have all the systems shown. More or fewer systems can be implemented or have alternatively.

In particular, according to the embodiments disclosed in the present application, the process described above with reference to the flowchart can be implemented as a computer software program. For example, the embodiments disclosed in the present application include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication device, or installed from a storage device 1003, or installed from a ROM 1002. When the computer program is executed by the processing device 1001, the above-mentioned functions defined in the method of the embodiment disclosed in the present application are executed.

The video editing device provided by the present application adopts the video editing method in the above embodiment to solve the technical problems of video editing. Compared with the prior art, the beneficial effects of the video editing device provided by the present application are the same as the beneficial effects of the video editing method provided by the above embodiment, and the other technical features in the video editing device are the same as the features disclosed in the method of the previous embodiment, which will not be repeated here.

It should be understood that the various parts disclosed in this application can be implemented by hardware, software, firmware or a combination thereof. In the description of the above embodiments, specific features, structures, materials or characteristics can be combined in any one or more embodiments or examples in a suitable manner.

The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

The present application provides a computer-readable storage medium having computer-readable program instructions (ie, computer programs) stored thereon, wherein the computer-readable program instructions are used to execute the video editing method in the above-mentioned embodiment.

The computer-readable storage medium may be included in the video editing device; or may exist independently without being assembled into the video editing device.

The computer-readable storage medium carries one or more programs. When the one or more programs are executed by a video editing device, the video editing device: obtains a surround video of an inserted target, wherein the surround video contains video frames of the inserted target at different viewing angles; inputs the surround video into a preset three-dimensional reconstruction model to obtain a target three-dimensional model, wherein the preset three-dimensional reconstruction model is trained by each video frame obtained by segmenting the surround video; obtains video reconstruction information based on optical flow information of the video to be inserted in combination with a preset optimization algorithm, wherein the video reconstruction information includes camera internal and external parameter information and dense point cloud information; and implants the target three-dimensional model into a user-selected area in the video to be inserted based on the camera internal and external parameter information and the dense point cloud information to obtain an edited video.

The readable storage medium provided in the present application is a computer-readable storage medium, which stores computer-readable program instructions (i.e., computer programs) for executing the above-mentioned video editing method, and can solve the technical problems of video editing. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided in the present application are the same as the beneficial effects of the video editing method provided in the above-mentioned embodiment, and will not be described in detail here.

The above descriptions are only some embodiments of the present application, and are not intended to limit the patent scope of the present application. All equivalent structural changes made using the contents of the present application specification and drawings under the technical concept of the present application, or direct/indirect applications in other related technical fields are included in the patent protection scope of the present application.

Claims (10)

1.A method of video editing, the method comprising:

acquiring surrounding video of an insertion target, wherein the surrounding video comprises video frames of the insertion target under different visual angles;

inputting the surrounding video into a preset three-dimensional reconstruction model to obtain a target three-dimensional model, wherein the preset three-dimensional reconstruction model is obtained by training each video frame obtained by dividing the surrounding video;

obtaining video reconstruction information according to optical flow information of a video to be inserted in combination with a preset optimization algorithm, wherein the video reconstruction information comprises camera inside and outside parameter information and dense point cloud information;

and implanting the target three-dimensional model into a user selected area in the video to be inserted according to the camera inside-outside parameter information and the dense point cloud information to obtain an edited video.

2. The method of claim 1, wherein the pre-set three-dimensional reconstruction model includes a geometrically encoded perceptron and a texture encoded perceptron; the training step of the preset three-dimensional reconstruction model comprises the following steps:

initializing a tetrahedral grid, inputting the three-dimensional coordinates of the tetrahedral grid to the geometric coding perceptron to obtain corresponding geometric information, and converting the geometric information to obtain a target grid;

inputting the three-dimensional coordinates of the target grid to the texture coding perceptron to obtain texture information;

Inputting the ambient lighting map, the camera pose information of the surrounding video, the target grid and the texture information to a preset micro-renderer to obtain a current rendering map;

Performing loss calculation according to the current rendering map and foreground images of the insertion target under different view angles to obtain a loss gradient, and optimizing the geometric coding perceptron, the texture coding perceptron and the ambient lighting map through the loss gradient based on a gradient return mechanism;

based on an optimization target for minimizing experience risks, a preset three-dimensional reconstruction model is obtained.

3. The method of claim 2, wherein the initializing the tetrahedral mesh, inputting three-dimensional coordinates of the tetrahedral mesh to the geometric coding perceptron, obtaining corresponding geometric information, and converting the geometric information to obtain the target mesh, comprises:

When a click command of a user is received, generating a mask area corresponding to the insertion target in a current video frame of the surrounding video according to the click command;

And dividing the other video frames obtained by dividing the surrounding video according to the mask area to obtain foreground images of the insertion target under different visual angles.

4. The method of claim 1, wherein the step of obtaining the video reconstruction information according to the optical flow information of the video to be inserted in combination with a preset optimization algorithm comprises:

inputting a plurality of video frames corresponding to the video to be inserted into an optical flow prediction model, and determining an optical flow displacement value of each video frame;

when the optical flow displacement value is larger than a preset threshold value, adding a video frame corresponding to the optical flow displacement value to a key frame set;

and determining video reconstruction information in the key frame set by adopting a preset optimization algorithm to obtain camera inside and outside parameter information and dense point cloud information.

5. The method of claim 4, wherein the step of determining video reconstruction information for each video frame in the keyframe set by using a preset optimization algorithm to obtain camera inside-outside parameter information and dense point cloud information comprises:

dividing the keyframe set into an online optimization set and an offline optimization set;

determining the relative pose, depth map and camera internal parameters of the cameras corresponding to each video frame by adopting a preset optimization algorithm in the online optimization set;

and updating the relative pose of the camera, the depth map and the camera internal parameters in the offline optimization set by adopting a preset optimization algorithm to obtain camera internal and external parameter information and dense point cloud information.

6. The method of claim 1, wherein the step of implanting the target three-dimensional model into the user-selected area in the video to be inserted according to the camera inside-outside parameter information and the dense point cloud information to obtain the edited video comprises:

generating a user selected area in the video to be inserted when receiving a selection instruction of a user;

Back projecting the pixel coordinates of the user selected area into a camera coordinate system corresponding to the video to be inserted according to the camera inside and outside parameter information and the dense point cloud information to obtain three-dimensional point cloud coordinates of the user selected area;

Determining a three-dimensional plane normal vector corresponding to the video to be inserted, and preprocessing the target three-dimensional model by combining the three-dimensional point cloud coordinates of the user selected area;

Rendering different view angle pictures corresponding to the preprocessed target three-dimensional model according to the video inside and outside parameter information, and synthesizing an edited video according to each view angle picture and a plurality of video frames corresponding to the video to be inserted.

7. The method of claim 6, wherein the step of determining the normal vector of the three-dimensional plane corresponding to the video to be inserted and preprocessing the target three-dimensional model in combination with the three-dimensional point cloud coordinates of the user-selected area comprises:

Based on a random sampling coincidence algorithm, determining a three-dimensional plane normal vector according to a camera coordinate system corresponding to the video to be inserted;

and preprocessing the target three-dimensional model by combining the three-dimensional point cloud coordinates of the user selected area based on the three-dimensional plane normal vector.

8. A video editing apparatus, the apparatus comprising: the system comprises a model generation module, a video reconstruction module and a model implantation module;

The model generation module is used for acquiring surrounding video of an insertion target, wherein the surrounding video comprises video frames of the insertion target under different visual angles;

The model generation module is further used for inputting the surrounding video into a preset three-dimensional reconstruction model to obtain a target three-dimensional model, wherein the preset three-dimensional reconstruction model is obtained by training each video frame obtained by segmenting the surrounding video;

the video reconstruction module is used for obtaining video reconstruction information according to optical flow information of a video to be inserted in combination with a preset optimization algorithm, wherein the video reconstruction information comprises camera inner and outer parameter information and dense point cloud information;

The model implantation module is used for rendering the target three-dimensional model to a user selected area in the video to be inserted according to the camera inside-outside parameter information and the dense point cloud information to obtain an edited video.

9. A video editing apparatus, the apparatus comprising: memory, a processor and a computer program stored on the memory and executable on the processor, the computer program being configured to implement the steps of the video editing method of any of claims 1 to 7.

10. A storage medium, characterized in that the storage medium is a computer-readable storage medium, on which a computer program is stored, which computer program, when being executed by a processor, realizes the steps of the video editing method according to any of claims 1 to 7.