CN110225285B - Audio and video communication method and device, computer device and readable storage medium - Google Patents

Abstract

The present invention provides an audio and video communication method, comprising: when performing audio and video communication with an external device, acquiring audio and video data to be transmitted, and extracting audio and video related parameters from the audio and video data to be transmitted; The scene recognition model, identifies the current scene of the user according to the acquired audio and video related parameters; determines the processing method for the audio and video data to be transmitted according to the current scene of the user; The audio and video data to be transmitted is processed, and the processed audio and video data is transmitted to the external device. The present invention also provides a device, a computer device, and a readable storage medium for implementing the audio and video communication method. The present invention can solve the technical problem of poor user experience in audio and video communication.

Description

Audio and video communication method, device, computer device, and readable storage medium

technical field

The present invention relates to the field of computer technology, and in particular, to an audio and video communication method, device, computer device, and readable storage medium.

Background technique

In the process of audio and video communication, the current environment of the user has a great influence on the audio and video communication experience. For example, a noisy environment can make it difficult for the communicating party to hear the user's speech.

SUMMARY OF THE INVENTION

In view of the above content, it is necessary to propose an audio and video communication method, device, computer device, and readable storage medium, so as to solve the technical problem of poor user experience in audio and video communication.

A first aspect of the present invention provides an audio and video communication method, the method comprising:

When performing audio and video communication with an external device, obtain audio and video data to be transmitted, and extract audio and video related parameters from the audio and video data to be transmitted;

Call the scene recognition model generated by pre-training, and identify the current scene of the user according to the acquired audio and video related parameters;

Determine the processing method for the audio and video data to be transmitted according to the current scene of the user; and

The audio and video data to be transmitted is processed according to the determined processing mode, and the processed audio and video data is transmitted to the external device.

Preferably, the method for training the scene recognition model comprises:

Acquire a preset number of audio and video-related parameters corresponding to different scenarios, and label the audio and video-related parameters corresponding to each scene with categories, so that the audio and video-related parameters corresponding to each of the scenarios carry category labels ;

The audio and video related parameters corresponding to the different scenes are randomly divided into a training set of a first preset ratio and a verification set of a second preset ratio, and the scene recognition model is trained by using the training set, and the verification set is used to train the scene recognition model. set the accuracy of the scene recognition model after training; and

If the accuracy rate is greater than or equal to the preset accuracy rate, end the training; if the accuracy rate is less than the preset accuracy rate, increase the number of samples to retrain the scene recognition model until the accuracy rate is greater than Or equal to the preset accuracy.

Preferably, the determining of the processing method for the audio and video data to be transmitted according to the current scene of the user includes:

When the current scene where the user is located is outdoors, it is determined that the processing mode for the audio and video data to be transmitted is the first mode, wherein the first mode means that the processing of the audio and video data to be transmitted at least includes: Noise reduction processing;

When the scene where the user is currently located is indoors, it is determined that the processing method for the audio and video data to be transmitted is the second method, wherein the second method refers to processing the to-be-transmitted data according to the indoor area and the material of the indoor walls audio and video data.

Preferably, the processing of the audio and video data to be transmitted according to the indoor area and the material of the indoor walls includes the steps of:

Estimate the size of the indoor area;

Cut out a frame of an image including a wall from the audio and video data to be transmitted;

Use an image recognition algorithm to match the captured image of the wall with a plurality of pre-stored images of different materials to determine the material of the wall; determine the sound absorption coefficient according to the material of the wall;

multiplying the indoor area by the determined sound absorption coefficient to estimate the sound absorption; and

The audio and video data to be transmitted is processed according to the estimated sound absorption, wherein, when the estimated sound absorption is greater than a preset sound absorption value, the audio and video data to be transmitted is processed at least De-reverberation processing is included. When the estimated sound absorption amount is less than or equal to the preset sound absorption amount value, the processing of the audio and video data to be transmitted does not include de-reverberation processing.

Preferably, the estimating the size of the indoor area includes:

intercepting a frame of an image including the user's avatar from the audio and video data;

calculating the total number of first pixels included in the avatar of the user, and calculating the total number of second pixels included in the captured image;

The size of the indoor area is estimated according to the ratio between the total number of the first pixel points and the total number of the second pixel points, wherein the size of the indoor area is equal to a preset value divided by the ratio.

Preferably, after the audio and video data to be transmitted is processed according to the determined processing method, the method further includes:

Determine whether there are multiple portraits in the video images included in the audio and video data to be transmitted;

When it is determined that there are multiple portraits in the video image, identify the portrait in the video image facing the camera, and when there are no multiple portraits in the video image, do not identify the portrait in the video image facing the camera ;and

In the video image, other portraits except the portrait facing the lens are blurred.

Preferably, after the audio and video data to be transmitted is processed according to the determined processing method, the method further includes:

Obtain the average brightness of the video image in the audio-video data to be transmitted;

determining whether the average brightness of the video image is less than a preset brightness threshold; and

When the average brightness of the video image is less than the preset brightness threshold, the brightness enhancement is performed on the video image, and when the average brightness of the video image is greater than or equal to the preset brightness threshold, the brightness enhancement process is not performed on the video image.

A second aspect of the present invention provides a computer device, the computer device includes a memory and a processor, the memory is configured to store at least one instruction, and the processor is configured to implement the audio-video communication method when executing the at least one instruction .

A third aspect of the present invention provides a computer-readable storage medium, where the computer-readable storage medium stores at least one instruction, and when the at least one instruction is executed by a processor, implements the audio-video communication method.

A fourth aspect of the present invention provides an audio and video communication device, the device comprising:

an acquisition module for acquiring audio and video data to be transmitted when performing audio and video communication with an external device, and extracting audio and video related parameters from the audio and video data to be transmitted;

The execution module is used to call the scene recognition model generated by pre-training, and identify the current scene of the user according to the acquired audio and video related parameters;

The execution module is further configured to determine a processing method for the audio and video data to be transmitted according to the current scene of the user; and

The execution module is further configured to process the audio and video data to be transmitted according to the determined processing mode, and transmit the processed audio and video data to the external device.

The audio and video communication method, device, computer device, and readable storage medium described in the embodiments of the present invention obtain audio and video data to be transmitted when the computer device performs audio and video communication with an external device, and obtain audio and video data to be transmitted from the to-be-transmitted audio and video data. The audio and video related parameters are extracted from the audio and video data; the scene recognition model generated by pre-training is called, and the user's current scene is identified according to the acquired audio and video related parameters; and processing the audio and video data to be transmitted according to the determined processing mode, and transmitting the processed audio and video data to the external device, which can improve the user's audio and video communication experience.

Description of drawings

In order to explain the embodiments of the present invention or the technical solutions in the prior art more clearly, the following briefly introduces the accompanying drawings that need to be used in the description of the embodiments or the prior art. Obviously, the accompanying drawings in the following description are only It is an embodiment of the present invention. For those of ordinary skill in the art, other drawings can also be obtained according to the provided drawings without creative work.

FIG. 1 is a flowchart of an audio and video communication method provided by Embodiment 1 of the present invention.

FIG. 2 is a structural diagram of an audio and video communication apparatus according to Embodiment 2 of the present invention.

FIG. 3 is a schematic diagram of a computer apparatus according to Embodiment 3 of the present invention.

The following specific embodiments will further illustrate the present invention in conjunction with the above drawings.

Detailed ways

In order to more clearly understand the above objects, features and advantages of the present invention, the present invention will be described in detail below with reference to the accompanying drawings and specific embodiments. It should be noted that the embodiments of the present invention and the features in the embodiments may be combined with each other under the condition of no conflict.

In the following description, many specific details are set forth in order to facilitate a full understanding of the present invention, and the described embodiments are only some, but not all, embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present invention.

Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention belongs. The terms used herein in the description of the present invention are for the purpose of describing specific embodiments only, and are not intended to limit the present invention.

Example 1

FIG. 1 is a flowchart of an audio and video communication method provided by Embodiment 1 of the present invention.

In this embodiment, the audio and video communication method can be applied to a computer device. For a computer device that needs to perform audio and video communication, the function for audio and video communication provided by the method of the present invention can be directly integrated on the computer device. , or run on a computer device in the form of a software development kit (Software Development Kit, SDK).

As shown in FIG. 1 , the audio and video communication method specifically includes the following steps. According to different requirements, the order of the steps in the flowchart can be changed, and some steps can be omitted.

Step S1, when the computer device performs audio and video communication with an external device, acquire audio and video data to be transmitted, and extract audio and video related parameters from the audio and video data to be transmitted.

In this embodiment, the audio and video related parameters include, but are not limited to, audio frequency spectrum features, volume, frequency distribution, portraits included in the video image and the number of portraits, ground, and background.

In one embodiment, the audio and video data refers to audio data collected by a microphone and video data captured by a camera synchronization.

In one embodiment, the audio data may be firstly windowed and framed. For example, the audio data can be divided into multiple frames with a frame length of 10-30ms (milliseconds) by using a Hanning window, and the frame shift can be 10ms, so that the audio data can be divided into multiple frames. After windowing and framing the audio data, fast Fourier transform is then performed on the audio data after windowing and framing, thereby obtaining the frequency spectrum of the audio data. Then, the spectral features corresponding to the audio data are extracted according to the frequency spectrum of the audio data.

In one embodiment, the volume level included in the audio and video related parameters may refer to the average volume of the volume.

In one embodiment, an image recognition algorithm can be used to identify the human figures included in the video image and the number, ground, and background of the human figures from the audio and video data.

In one embodiment, the microphone and the camera may be built into the computer device, or externally connected to the computer device in a wired/wireless manner.

For example, a USB data cable can be used to establish a communication connection between the microphone and the camera and the computer device.

In one embodiment, the computer device and the external device may be a smartphone, a tablet computer, a laptop computer, a desktop computer, a smart TV, or the like.

In one embodiment, the computer apparatus and external device may be communicatively connected via any conventional wired network and/or wireless network. The wired network may be any type of traditional wired communication, such as the Internet, a local area network. The wireless network may be any type of conventional wireless communication, such as radio, Wireless Fidelity (WIFI), cellular, satellite, broadcast, and the like. Wireless communication technologies may include, but are not limited to, Global System for Mobile Communications (GSM), General Packet Radio Service (GPRS), Code Division Multiple Access (CDMA), broadband Code Division Multiple Access (W-CDMA), CDMA2000, IMT Single Carrier, Enhanced Data Rates for GSM Evolution (EDGE), Long-Term Evolution (LTE), Advanced Long Term Evolution, Time-Division LTE (TD-LTE), High Performance Radio Local Area Network (HiperLAN), High Performance Radio Wide Area Network (HiperWAN), local Local Multipoint Distribution Service (LMDS), Worldwide Interoperability for Microwave Access (WiMAX), ZigBee, Bluetooth, Flash Orthogonal Frequency-Division Multiplexing, Flash-OFDM), High Capacity Spatial Division Multiple Access (HC-SDMA), Universal Mobile Telecommunications System (UMTS), Universal Mobile Telecommunications System Time Division Duplex (UMTS) Time-Division Duplexing, UMTS-TDD), Evolved High Speed Packet Access (HSPA+), Time Division Synchronous Code Division Multiple Access (TD-SCDMA), Evolution Data Optimization (Evolution -Data Optimized, EV-DO ), Digital Enhanced Cordless Telecommunications (DECT) and others.

Step S2, calling the scene recognition model generated by pre-training, and identifying the scene where the user is currently located according to the acquired audio and video related parameters.

Specifically, the acquired audio and video related parameters are input into the scene recognition model generated by the pre-training to obtain the scene where the user is currently located.

In this embodiment, the scene can be divided into indoor and outdoor. Different scenarios correspond to different audio and video related parameters.

Preferably, the method for training the scene recognition model includes:

1) Acquire a preset number of audio and video related parameters corresponding to the different scenes, and label the audio and video related parameters corresponding to each kind of scene, so that the audio and video related parameters corresponding to each kind of scene are related. Parameters carry category labels.

For example, a total of 1000 records of audio and video related parameters corresponding to the room are selected, and the 1000 records are marked as "1" respectively, that is, "1" is used as the label. Similarly, select 1000 audio and video related parameters corresponding to the outdoors, and mark the 1000 records as "2", that is, use "2" as the label.

2) The audio and video related parameters corresponding to the different scenes are respectively randomly divided into a training set of a first preset ratio and a verification set of a second preset ratio, and the scene recognition model is trained by using the training set, and the The validation set verifies the accuracy of the trained scene recognition model.

For example, training samples corresponding to different scenes (that is, audio and video related parameters) may be distributed to different folders first. For example, the training samples corresponding to the indoor are distributed to the first folder, and the training samples corresponding to the outdoor are distributed to the second folder. Then, the training samples of the first preset ratio (for example, 70%) are respectively extracted from different folders as the total training samples to train the scene recognition model, and the remaining second preset ratios ( For example, 30%) of the training samples are used as the total test samples to verify the accuracy of the trained scene recognition model.

3) If the accuracy rate is greater than or equal to the preset accuracy rate, end the training, and use the trained scene recognition model as a classifier to identify the current environment of the user; if the accuracy rate is less than the preset accuracy rate When the accuracy rate is set, the number of samples is increased to retrain the scene recognition model until the accuracy rate is greater than or equal to the preset accuracy rate.

Step S3: Determine a processing method for the audio and video data to be transmitted according to the scene where the user is currently located, wherein different scenes correspond to different processing methods.

In this embodiment, the determining of the processing method for the audio and video data to be transmitted according to the current scene of the user includes:

When the scene where the user is currently located is outdoors, determining that the processing mode for the audio and video data to be transmitted is the first mode; and

When the scene where the user is currently located is indoors, it is determined that the processing mode for the audio and video data to be transmitted is the second mode.

In one embodiment, the first manner means that the processing of the audio and video data to be transmitted at least includes noise reduction processing. In one embodiment, speech enhancement may also be further included.

In one embodiment, the second manner refers to processing the audio and video data to be transmitted according to the indoor area and the material of the indoor wall.

In one embodiment, the processing of the audio and video data to be transmitted according to the indoor area and the material of the indoor walls includes steps (a1)-(a4):

(a1) Estimate the size of the indoor area.

In one embodiment, the estimating the size of the indoor area includes steps (a11)-(a13):

(a11) intercepting a frame of an image including the user's avatar from the audio and video data;

(a12) Calculate the total number of pixels included in the user's avatar (for convenience of description, referred to as "the first total number of pixels"), and calculate the total number of pixels included in the captured image (for convenience of description, referred to as "the total number of pixels") is "the total number of second pixels");

(a13) Estimate the size of the indoor area according to the ratio between the total number of the first pixel points and the total number of the second pixel points.

In one embodiment, the size of the indoor area is equal to a predetermined value divided by the ratio.

(a2) Determine the material of the indoor wall, and determine the sound absorption coefficient according to the material of the wall.

Specifically, the determining the material of the indoor wall includes steps (a21)-(a22):

(a21) Cut out a frame of an image including a wall from the audio and video data.

In one embodiment, the image including the wall can be intercepted from the audio and video data according to a user's operation.

(a22) Using an image recognition algorithm to match the captured image with a plurality of pre-stored images of different materials to determine the material of the wall.

Specifically, when the similarity between the captured image and the pre-stored image of a certain material is greater than a preset similarity value, it is determined that the material of the wall is the certain material.

Different materials correspond to different sound absorption coefficients. Therefore, once the material of the wall is determined, the sound absorption coefficient can be determined.

(a3) The amount of sound absorption is estimated by multiplying the indoor area by the determined sound absorption coefficient.

(a4) Process the audio and video data to be transmitted according to the estimated sound absorption.

In one embodiment, when the estimated sound absorption amount is greater than a preset sound absorption amount value, the processing of the audio and video data to be transmitted at least includes de-reverberation processing. When the estimated sound absorption amount is less than or equal to the preset sound absorption amount value, the processing of the audio and video data to be transmitted may not include de-reverberation processing.

In one embodiment, the processing of the audio and video data to be transmitted according to the estimated sound absorption may further include echo cancellation and speech enhancement.

Step S4: Process the audio and video data to be transmitted according to the determined processing mode, and transmit the processed audio and video data to the external device.

For example, assuming that the determined processing method is the first method, at least noise reduction processing is performed on the audio and video data to be transmitted.

In one embodiment, regardless of whether the audio and video data to be transmitted is processed in the first manner or the second manner, the process further includes processing the audio and video data to be transmitted as follows, including step (b1 )-(b3):

(b1) determine whether there are multiple portraits in the video images included in the audio and video data to be transmitted (for example, the number of portraits is greater than or equal to 2);

(b2) When it is determined that there are multiple portraits in the video image, identify the portraits in the video image that are facing the lens, and when there are no multiple portraits in the video image, do not face recognition;

(b3) Perform bokeh processing on other portraits in the video image except the portrait facing the lens, so as to highlight the portrait facing the lens.

In one embodiment, whether the audio and video data to be transmitted is processed in the first manner or the second manner, the process further includes processing the audio and video data to be transmitted as follows, including step (c1 )-(c3):

(c1) Obtain the average brightness of video images in the audio and video data to be transmitted.

Specifically, the average brightness of the video image can be obtained through an image brightness detection algorithm.

Specifically, in the embodiment of the present application, the specific implementation process of acquiring the average brightness of the video image may include: acquiring the resolution of the video image, determining a corresponding sampling interval according to the resolution, and performing a sampling interval on the video image according to the sampling interval. The brightness of the pixels in is sampled to generate the average brightness.

In one embodiment, the image brightness detection algorithm may include an averaging algorithm, a histogram algorithm, and the like.

In one embodiment, a corresponding brightness detection algorithm may be selected according to the scene where the user is currently located to obtain the average brightness of the video image.

In one embodiment, taking the averaging algorithm as an example, the sampling calculation can be performed according to the resolution of the video image.

For example, the resolution of the video image may be acquired first, and then the corresponding sampling interval may be determined according to the size of the resolution of the video image. For example, when the resolution of the video image is smaller than the preset resolution, the sampling interval is determined to be 1, that is, the entire video image is calculated; when the resolution of the video image is 1 to 4 times the preset resolution, the sampling interval is determined to be 1. The sampling interval between the horizontal and vertical directions is 2, that is, one pixel is selected for every two pixels in the video image; when the resolution of the video image is 4 to 8 times the preset resolution, determine the horizontal and vertical The sampling interval in the direction is 4, that is, one pixel is selected for every four pixels in the video image; when the resolution of the video image is greater than 8 times the preset resolution, the sampling interval in the horizontal and vertical directions is determined as 8, that is, select one pixel for every eight pixels in the video image. By analogy, the sampling interval is determined for larger resolution video images. After the sampling interval is determined, calculate the luminance value of the pixel points sampled according to the sampling interval in the video image, add the luminance values of all the sampled pixel points and average them, and use the calculated value as the entire video image. average brightness.

(c2) Determine whether the average brightness of the video image is less than a preset brightness threshold.

Wherein, the preset brightness threshold can be selected according to the scene where the user is currently located, that is, different thresholds need to be used for different scenes.

In one embodiment, the brightness threshold value corresponding to the scene where the user is currently located is outdoor is greater than the brightness threshold value corresponding to when the scene where the user is currently located is indoors.

(c3) If the average brightness of the video image is less than the preset brightness threshold, perform brightness enhancement on the video image. If the average brightness of the video image is greater than or equal to the preset brightness threshold, no brightness enhancement processing is performed on the video image.

In one embodiment, when the average brightness of the video image is less than the preset brightness threshold, a linear brightness enhancement algorithm may be used to enhance the brightness of the video image.

To sum up, the audio and video communication method described in the embodiments of the present invention obtains audio and video data to be transmitted when a computer device performs audio and video communication with an external device, and extracts audio and video data from the audio and video data to be transmitted. Audio and video related parameters; call a scene recognition model generated by pre-training, identify the current scene of the user according to the acquired audio and video related parameters; determine the processing method for the audio and video data to be transmitted according to the current scene of the user; and The audio and video data to be transmitted is processed according to the determined processing method, and the processed audio and video data is transmitted to the external device, which can improve the user's audio and video communication experience.

The above-mentioned Fig. 1 introduces the audio-video communication method of the present invention in detail, and below in conjunction with Fig. 2 and Fig. 3, the functional modules of the software device for realizing the audio-video communication method and the hardware device architecture for realizing the audio-video communication method are introduced. .

It should be understood that the embodiments are only used for illustration, and are not limited by this structure in the scope of the patent application.

Embodiment 2

Referring to FIG. 2 , it is a structural diagram of an audio and video communication apparatus provided by Embodiment 2 of the present invention.

In some embodiments, the audiovisual communication device 30 runs on a computer device. The computer apparatus is connected to an external device through a network. The audio and video communication device 30 may include a plurality of functional modules composed of program code segments. The program codes of each program segment in the audio-video communication device 30 may be stored in the memory of the computer device and executed by the at least one processor to realize (see description in FIG. 2 for details) audio-video communication functions.

In this embodiment, the audio and video communication apparatus 30 may be divided into multiple functional modules according to the functions performed by the audio and video communication apparatus 30 . The functional modules may include: an acquisition module 301 and an execution module 302 . The modules referred to in the present invention refer to a series of computer program segments that can be executed by at least one processor and can perform fixed functions, and are stored in a memory. In this embodiment, the functions of each module will be described in detail in subsequent embodiments.

The acquisition module 301 acquires audio and video data to be transmitted when the computer device performs audio and video communication with an external device, and extracts audio and video related parameters from the audio and video data to be transmitted.

In this embodiment, the audio and video related parameters include, but are not limited to, audio frequency spectrum features, volume, frequency distribution, portraits included in the video image and the number of portraits, ground, and background.

In one embodiment, the audio and video data refers to audio data collected by a microphone and video data captured by a camera synchronization.

In one embodiment, the audio data may be firstly windowed and framed. For example, the audio data can be divided into multiple frames with a frame length of 10-30ms (milliseconds) by using a Hanning window, and the frame shift can be 10ms, so that the audio data can be divided into multiple frames. After windowing and framing the audio data, fast Fourier transform is then performed on the audio data after windowing and framing, thereby obtaining the frequency spectrum of the audio data. Then, the spectral features corresponding to the audio data are extracted according to the frequency spectrum of the audio data.

In one embodiment, the volume level included in the audio and video related parameters may refer to the average volume of the volume.

In one embodiment, an image recognition algorithm can be used to identify the human figures included in the video image and the number, ground, and background of the human figures from the audio and video data.

In one embodiment, the microphone and the camera may be built into the computer device, or externally connected to the computer device in a wired/wireless manner.

For example, a USB data cable can be used to establish a communication connection between the microphone and the camera and the computer device.

In one embodiment, the computer device and the external device may be a smartphone, a tablet computer, a laptop computer, a desktop computer, a smart TV, or the like.

In one embodiment, the computer apparatus and external device may be communicatively connected via any conventional wired network and/or wireless network. The wired network may be any type of traditional wired communication, such as the Internet, a local area network. The wireless network may be any type of conventional wireless communication, such as radio, Wireless Fidelity (WIFI), cellular, satellite, broadcast, and the like. Wireless communication technologies may include, but are not limited to, Global System for Mobile Communications (GSM), General Packet Radio Service (GPRS), Code Division Multiple Access (CDMA), broadband Code Division Multiple Access (W-CDMA), CDMA2000, IMT Single Carrier, Enhanced Data Rates for GSM Evolution (EDGE), Long-Term Evolution (LTE), Advanced Long Term Evolution, Time-Division LTE (TD-LTE), High Performance Radio Local Area Network (HiperLAN), High Performance Radio Wide Area Network (HiperWAN), local Local Multipoint Distribution Service (LMDS), Worldwide Interoperability for Microwave Access (WiMAX), ZigBee, Bluetooth, Flash Orthogonal Frequency-Division Multiplexing, Flash-OFDM), High Capacity Spatial Division Multiple Access (HC-SDMA), Universal Mobile Telecommunications System (UMTS), Universal Mobile Telecommunications System Time Division Duplex (UMTS) Time-Division Duplexing, UMTS-TDD), Evolved High Speed Packet Access (HSPA+), Time Division Synchronous Code Division Multiple Access (TD-SCDMA), Evolution Data Optimization (Evolution -Data Optimized, EV-DO ), Digital Enhanced Cordless Telecommunications (DECT) and others.

The execution module 302 invokes the scene recognition model generated by the pre-training, and identifies the scene where the user is currently located according to the acquired audio and video related parameters.

Specifically, the execution module 302 inputs the acquired audio and video related parameters into the scene recognition model generated by the pre-training to obtain the scene where the user is currently located.

In this embodiment, the scene can be divided into indoor and outdoor. Different scenarios correspond to different audio and video related parameters.

Preferably, the method for training the scene recognition model includes:

1) Acquire a preset number of audio and video related parameters corresponding to the different scenes, and label the audio and video related parameters corresponding to each kind of scene, so that the audio and video related parameters corresponding to each kind of scene are related. Parameters carry category labels.

For example, a total of 1000 records of audio and video related parameters corresponding to the room are selected, and the 1000 records are marked as "1" respectively, that is, "1" is used as the label. Similarly, select 1000 audio and video related parameters corresponding to the outdoors, and mark the 1000 records as "2", that is, use "2" as the label.

2) The audio and video related parameters corresponding to the different scenes are respectively randomly divided into a training set of a first preset ratio and a verification set of a second preset ratio, and the scene recognition model is trained by using the training set, and the The validation set verifies the accuracy of the trained scene recognition model.

For example, training samples corresponding to different scenes (that is, audio and video related parameters) may be distributed to different folders first. For example, the training samples corresponding to the indoor are distributed to the first folder, and the training samples corresponding to the outdoor are distributed to the second folder. Then, the training samples of the first preset ratio (for example, 70%) are respectively extracted from different folders as the total training samples to train the scene recognition model, and the remaining second preset ratios ( For example, 30%) of the training samples are used as the total test samples to verify the accuracy of the trained scene recognition model.

3) If the accuracy rate is greater than or equal to the preset accuracy rate, end the training, and use the trained scene recognition model as a classifier to identify the current environment of the user; if the accuracy rate is less than the preset accuracy rate When the accuracy rate is set, the number of samples is increased to retrain the scene recognition model until the accuracy rate is greater than or equal to the preset accuracy rate.

The execution module 302 determines a processing method for the audio and video data to be transmitted according to the scene where the user is currently located, wherein different scenarios correspond to different processing methods.

In this embodiment, the determining of the processing method for the audio and video data to be transmitted according to the current scene of the user includes:

When the scene where the user is currently located is outdoors, determining that the processing mode for the audio and video data to be transmitted is the first mode; and

When the scene where the user is currently located is indoors, it is determined that the processing mode for the audio and video data to be transmitted is the second mode.

In one embodiment, the first manner means that the processing of the audio and video data to be transmitted at least includes noise reduction processing. In one embodiment, speech enhancement may also be further included.

In one embodiment, the second manner refers to processing the audio and video data to be transmitted according to the indoor area and the material of the indoor wall.

In one embodiment, the processing of the audio and video data to be transmitted according to the indoor area and the material of the indoor walls includes steps (a1)-(a4):

(a1) Estimate the size of the indoor area.

In one embodiment, the estimating the size of the indoor area includes steps (a11)-(a13):

(a11) intercepting a frame of an image including the user's avatar from the audio and video data;

(a12) Calculate the total number of pixels included in the user's avatar (for convenience of description, referred to as "the first total number of pixels"), and calculate the total number of pixels included in the captured image (for convenience of description, referred to as "the total number of pixels") is "the total number of second pixels");

(a13) Estimate the size of the indoor area according to the ratio between the total number of the first pixel points and the total number of the second pixel points.

In one embodiment, the size of the indoor area is equal to a predetermined value divided by the ratio.

(a2) Determine the material of the indoor wall, and determine the sound absorption coefficient according to the material of the wall.

Specifically, the determining the material of the indoor wall includes steps (a21)-(a22):

(a21) Cut out a frame of an image including a wall from the audio and video data.

In one embodiment, the image including the wall can be intercepted from the audio and video data according to a user's operation.

(a22) Using an image recognition algorithm to match the captured image with a plurality of pre-stored images of different materials to determine the material of the wall.

Specifically, when the similarity between the captured image and the pre-stored image of a certain material is greater than a preset similarity value, it is determined that the material of the wall is the certain material.

Different materials correspond to different sound absorption coefficients. Therefore, once the material of the wall is determined, the sound absorption coefficient can be determined.

(a3) The amount of sound absorption is estimated by multiplying the indoor area by the determined sound absorption coefficient.

(a4) Process the audio and video data to be transmitted according to the estimated sound absorption.

In one embodiment, when the estimated sound absorption amount is greater than a preset sound absorption amount value, the processing of the audio and video data to be transmitted at least includes de-reverberation processing. When the estimated sound absorption amount is less than or equal to the preset sound absorption amount value, the processing of the audio and video data to be transmitted may not include de-reverberation processing.

In one embodiment, the processing of the audio and video data to be transmitted according to the estimated sound absorption may further include echo cancellation and speech enhancement.

The execution module 302 processes the audio and video data to be transmitted according to the determined processing mode, and transmits the processed audio and video data to the external device.

For example, assuming that the determined processing method is the first method, at least noise reduction processing is performed on the audio and video data to be transmitted.

In one embodiment, regardless of whether the audio and video data to be transmitted is processed in the first manner or the second manner, the process further includes processing the audio and video data to be transmitted as follows, including step (b1 )-(b3):

(b1) determine whether there are multiple portraits in the video images included in the audio and video data to be transmitted (for example, the number of portraits is greater than or equal to 2);

(b2) When it is determined that there are multiple portraits in the video image, identify the portraits in the video image that are facing the lens, and when there are no multiple portraits in the video image, do not face recognition;

(b3) Perform bokeh processing on other portraits in the video image except the portrait facing the lens, so as to highlight the portrait facing the lens.

In one embodiment, whether the audio and video data to be transmitted is processed in the first manner or the second manner, the process further includes processing the audio and video data to be transmitted as follows, including step (c1 )-(c3):

(c1) Obtain the average brightness of video images in the audio and video data to be transmitted.

Specifically, the average brightness of the video image can be obtained through an image brightness detection algorithm.

Specifically, in the embodiment of the present application, the specific implementation process of acquiring the average brightness of the video image may include: acquiring the resolution of the video image, determining a corresponding sampling interval according to the resolution, and performing a sampling interval on the video image according to the sampling interval. The brightness of the pixels in is sampled to generate the average brightness.

In one embodiment, the image brightness detection algorithm may include an averaging algorithm, a histogram algorithm, and the like.

In one embodiment, a corresponding brightness detection algorithm may be selected according to the scene where the user is currently located to obtain the average brightness of the video image.

In one embodiment, taking the averaging algorithm as an example, the sampling calculation can be performed according to the resolution of the video image.

For example, the resolution of the video image may be acquired first, and then the corresponding sampling interval may be determined according to the size of the resolution of the video image. For example, when the resolution of the video image is smaller than the preset resolution, the sampling interval is determined to be 1, that is, the entire video image is calculated; when the resolution of the video image is 1 to 4 times the preset resolution, the sampling interval is determined to be 1. The sampling interval between the horizontal and vertical directions is 2, that is, one pixel is selected for every two pixels in the video image; when the resolution of the video image is 4 to 8 times the preset resolution, determine the horizontal and vertical The sampling interval in the direction is 4, that is, one pixel is selected for every four pixels in the video image; when the resolution of the video image is greater than 8 times the preset resolution, the sampling interval in the horizontal and vertical directions is determined as 8, that is, select one pixel for every eight pixels in the video image. By analogy, the sampling interval is determined for larger resolution video images. After the sampling interval is determined, calculate the brightness value of the pixel points sampled according to the sampling interval in the video image, add the brightness values of all the sampled pixel points and average them, and use the calculated value as the entire video image. average brightness.

(c2) Determine whether the average brightness of the video image is less than a preset brightness threshold.

Wherein, the preset brightness threshold can be selected according to the scene where the user is currently located, that is, different thresholds need to be used for different scenes.

In one embodiment, the brightness threshold value corresponding to the scene where the user is currently located is outdoor is greater than the brightness threshold value corresponding to when the scene where the user is currently located is indoors.

(c3) If the average brightness of the video image is less than the preset brightness threshold, perform brightness enhancement on the video image. If the average brightness of the video image is greater than or equal to the preset brightness threshold, no brightness enhancement processing is performed on the video image.

In one embodiment, when the average brightness of the video image is less than the preset brightness threshold, a linear brightness enhancement algorithm may be used to enhance the brightness of the video image.

To sum up, the audio and video communication device described in the embodiment of the present invention acquires audio and video data to be transmitted when a computer device performs audio and video communication with an external device, and extracts audio and video data from the audio and video data to be transmitted. Audio and video related parameters; call the scene recognition model generated by pre-training, identify the current scene of the user according to the acquired audio and video related parameters; determine the processing method for the audio and video data to be transmitted according to the current scene of the user, wherein , different scenarios correspond to different processing methods; and the audio and video data to be transmitted is processed according to the determined processing method, and the processed audio and video data is transmitted to the external device, which can improve the user experience. Audio and video communication experience.

Embodiment 3

Referring to FIG. 3 , it is a schematic structural diagram of a computer apparatus according to Embodiment 3 of the present invention. In a preferred embodiment of the present invention, the computer device 3 includes a memory 31 , at least one processor 32 , and at least one communication bus 33 . Those skilled in the art should understand that the structure of the computer device shown in FIG. 3 does not constitute a limitation of the embodiments of the present invention, and may be a bus-type structure or a star-shaped structure, and the computer device 3 may also include a ratio diagram more or less other hardware or software, or a different arrangement of components is shown.

In some embodiments, the computer device 3 includes a terminal capable of automatically performing numerical calculations and/or information processing according to pre-set or stored instructions, and its hardware includes but is not limited to microprocessors, application-specific integrated circuits, Programmable gate arrays, digital processors and embedded devices, etc.

It should be noted that the computer device 3 is only an example. If other existing or future electronic products can be adapted to the present invention, they should also be included in the protection scope of the present invention, and are incorporated herein by reference. .

In some embodiments, the memory 31 is used for storing program codes and various data, such as the audio and video communication device 30 installed in the computer device 3 , and realizes high-speed and automatic operation during the operation of the computer device 3 . Complete program or data access. The memory 31 includes a read-only memory (Read-Only Memory, ROM), a random access memory (Random Access Memory, RAM), a programmable read-only memory (Programmable Read-Only Memory, PROM), an erasable programmable read-only memory. (Erasable Programmable Read-Only Memory, EPROM), one-time programmable read-only memory (One-time Programmable Read-Only Memory, OTPROM), electronically erasable rewritable read-only memory (Electrically-Erasable Programmable Read-Only Memory, EEPROM) ), Compact Disc Read-Only Memory (CD-ROM) or other optical disk storage, magnetic disk storage, magnetic tape storage, or any other computer-readable storage medium that can be used to carry or store data.

In some embodiments, the at least one processor 32 may be composed of integrated circuits, for example, may be composed of a single packaged integrated circuit, or may be composed of multiple integrated circuits packaged with the same function or different functions, including one Or a combination of multiple central processing units (Central Processing Units, CPUs), microprocessors, digital processing chips, graphics processors, and various control chips. The at least one processor 32 is the control core (Control Unit) of the computer device 3, and uses various interfaces and lines to connect various components of the entire computer device 3, by running or executing the program stored in the memory 31 or modules, and call the data stored in the memory 31 to perform various functions of the computer device 3 and process data, such as the function of performing audio and video communication.

In some embodiments, the at least one communication bus 33 is configured to enable connection communication between the memory 31 and the at least one processor 32 and the like.

Although not shown, the computer device 3 may also include a power source (such as a battery) for supplying power to various components. Preferably, the power source may be logically connected to the at least one processor 32 through a power management device, so as to be implemented by the power management device Manage charging, discharging, and power management functions. The power source may also include one or more DC or AC power sources, recharging devices, power failure detection circuits, power converters or inverters, power status indicators, and any other components. The computer device 3 may further include a variety of sensors, Bluetooth modules, Wi-Fi modules, etc., which will not be repeated here.

It should be understood that the embodiments are only used for illustration, and are not limited by this structure in the scope of the patent application.

The above-mentioned integrated units implemented in the form of software functional modules may be stored in a computer-readable storage medium. The above-mentioned software function modules are stored in a storage medium, and include several instructions for causing a computer device (which may be a server, a personal computer, etc.) or a processor (processor) to execute parts of the methods described in the various embodiments of the present invention.

In a further embodiment, with reference to FIG. 2 , the at least one processor 32 can execute the operating device of the computer device 3 and various installed applications (such as the audio and video communication device 30 ), program codes, etc. , for example, the various modules above.

Program codes are stored in the memory 31, and the at least one processor 32 can call the program codes stored in the memory 31 to perform related functions. For example, each module described in FIG. 2 is a program code stored in the memory 31 and executed by the at least one processor 32, so as to realize the function of each module and achieve the purpose of audio and video communication.

In one embodiment of the present invention, the memory 31 stores one or more instructions (ie, at least one instruction), and the one or more instructions are executed by the at least one processor 32 to achieve the purpose of audio and video communication .

Specifically, for the specific implementation method of the above instruction by the at least one processor 32, reference may be made to the description of the relevant steps in the embodiment corresponding to FIG. 1 , and details are not described herein.

In the several embodiments provided by the present invention, it should be understood that the disclosed apparatus and method may be implemented in other manners. For example, the apparatus embodiments described above are only illustrative. For example, the division of the modules is only a logical function division, and there may be other division manners in actual implementation.

The modules described as separate components may or may not be physically separated, and components shown as modules may or may not be physical units, that is, may be located in one place, or may be distributed to multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution in this embodiment.

In addition, each functional module in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically alone, or two or more units may be integrated into one unit. The above-mentioned integrated units can be implemented in the form of hardware, or can be implemented in the form of hardware plus software function modules.

It will be apparent to those skilled in the art that the present invention is not limited to the details of the above-described exemplary embodiments, but that the present invention may be embodied in other specific forms without departing from the spirit or essential characteristics of the invention. Therefore, the embodiments are to be regarded in all respects as illustrative and not restrictive, and the scope of the invention is to be defined by the appended claims rather than the foregoing description, which are therefore intended to fall within the scope of the claims. All changes within the meaning and range of the equivalents of , are included in the present invention. Any reference signs in the claims shall not be construed as limiting the involved claim. Furthermore, it is clear that the word "comprising" does not exclude other units or, and the singular does not exclude the plural. Several units or means recited in the device claims can also be realized by one unit or means by means of software or hardware. The terms first, second, etc. are used to denote names and do not denote any particular order.

Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit them. Although the present invention has been described in detail with reference to the preferred embodiments, those of ordinary skill in the art should understand that the technical solutions of the present invention can be Modifications or equivalent substitutions can be made without departing from the spirit and scope of the technical solutions of the present invention.

Claims (9)

1. an audio-video communication method, is characterized in that, described method comprises: When performing audio and video communication with an external device, obtain audio and video data to be transmitted, and extract audio and video related parameters from the audio and video data to be transmitted; Call the scene recognition model generated by pre-training, and identify the current scene of the user according to the acquired audio and video related parameters; Determine the processing method for the audio and video data to be transmitted according to the current scene where the user is located, and when the current scene where the user is located is outdoors, determine that the processing method for the audio and video data to be transmitted is the first method, wherein, The first method means that the processing of the audio and video data to be transmitted at least includes noise reduction processing; when the current scene where the user is located is indoors, it is determined that the processing method of the audio and video data to be transmitted is the second. method, wherein the second method refers to processing the audio and video data to be transmitted according to the indoor area and the material of the indoor wall; and The audio and video data to be transmitted is processed according to the determined processing mode, and the processed audio and video data is transmitted to the external device. 2. The audio-video communication method according to claim 1, wherein the method for training the scene recognition model comprises: Acquire a preset number of audio and video-related parameters corresponding to different scenarios, and label the audio and video-related parameters corresponding to each scene with categories, so that the audio and video-related parameters corresponding to each of the scenarios carry category labels ; The audio and video related parameters corresponding to the different scenes are randomly divided into a training set of a first preset ratio and a verification set of a second preset ratio, and the scene recognition model is trained by using the training set, and the verification set is used to train the scene recognition model. set the accuracy of the scene recognition model after training; and If the accuracy rate is greater than or equal to the preset accuracy rate, end the training; if the accuracy rate is less than the preset accuracy rate, increase the number of samples to retrain the scene recognition model until the accuracy rate is greater than Or equal to the preset accuracy. 3. The audio and video communication method according to claim 1, wherein the processing of the audio and video data to be transmitted according to the indoor area and the material of the indoor wall comprises the steps: Estimate the size of the indoor area; Cut out a frame of an image including a wall from the audio and video data to be transmitted; Use an image recognition algorithm to match the captured image of the wall with a plurality of pre-stored images of different materials to determine the material of the wall; determine the sound absorption coefficient according to the material of the wall; multiplying the indoor area by the determined sound absorption coefficient to estimate the sound absorption; and The audio and video data to be transmitted is processed according to the estimated sound absorption, wherein, when the estimated sound absorption is greater than a preset sound absorption value, the audio and video data to be transmitted is processed at least De-reverberation processing is included. When the estimated sound absorption amount is less than or equal to the preset sound absorption amount value, the processing of the audio and video data to be transmitted does not include de-reverberation processing. 4. The audio-video communication method according to claim 3, wherein the estimating the size of the indoor area comprises: intercepting a frame of an image including the user's avatar from the audio and video data; calculating the total number of first pixels included in the avatar of the user, and calculating the total number of second pixels included in the captured image; The size of the indoor area is estimated according to the ratio between the total number of the first pixel points and the total number of the second pixel points, wherein the size of the indoor area is equal to a preset value divided by the ratio. 5. The audio and video communication method according to claim 1, wherein after the audio and video data to be transmitted is processed according to the determined processing mode, the method further comprises: Determine whether there are multiple portraits in the video images included in the audio and video data to be transmitted; When it is determined that there are multiple portraits in the video image, identify the portrait in the video image facing the camera, and when there are no multiple portraits in the video image, do not identify the portrait in the video image facing the camera ;and In the video image, other portraits except the portrait facing the lens are blurred. 6. The audio and video communication method according to claim 1, wherein after the audio and video data to be transmitted is processed according to the determined processing mode, the method further comprises: Obtain the average brightness of the video image in the audio-video data to be transmitted; determining whether the average brightness of the video image is less than a preset brightness threshold; and When the average brightness of the video image is less than the preset brightness threshold, the brightness enhancement is performed on the video image, and when the average brightness of the video image is greater than or equal to the preset brightness threshold, the brightness enhancement process is not performed on the video image. 7. A computer device, characterized in that the computer device comprises a memory and a processor, the memory is used to store at least one instruction, and the processor is used to implement the at least one instruction when executing the at least one instruction as claimed in claims 1 to 6 The audio and video communication method described in any one of the above. 8. A computer-readable storage medium, characterized in that the computer-readable storage medium stores at least one instruction, and when the at least one instruction is executed by a processor, the implementation of any one of claims 1 to 6 is implemented Audio and video communication methods. 9. An audio and video communication device, wherein the device comprises: an acquisition module for acquiring audio and video data to be transmitted when performing audio and video communication with an external device, and extracting audio and video related parameters from the audio and video data to be transmitted; The execution module is used to call the scene recognition model generated by pre-training, and identify the current scene of the user according to the acquired audio and video related parameters; The execution module is further configured to determine the processing mode of the audio and video data to be transmitted according to the current scene of the user, and determine the processing of the audio and video data to be transmitted when the current scene of the user is outdoors The method is the first method, wherein the first method means that the processing of the audio and video data to be transmitted at least includes noise reduction processing; when the scene where the user is currently located is indoors, it is determined to The processing method of the video data is the second method, wherein the second method refers to processing the audio and video data to be transmitted according to the indoor area and the material of the indoor wall; and The execution module is further configured to process the audio and video data to be transmitted according to the determined processing mode, and transmit the processed audio and video data to the external device.

Images