CN107135207A - Method and system for live streaming of panoramic conference - Google Patents

Abstract

The present invention proposes a panoramic conference live broadcast method and system, wherein the panoramic conference live broadcast method includes: respectively acquiring the panoramic image of the conference scene, the speaker's video image and the speaker's speech content; receiving the user's current visual center; according to the current visual The center locates the scene in the panoramic image to obtain the regional scene video; combines the regional scene video, the speaker's video image and the speech content to obtain the combined audio and video information and send it. The present invention obtains the user's visual center information through the VR device, and only transmits the video information of the venue scene that conforms to the user's visual center to the user; according to the characteristics of the conference live broadcast, the video information of the speaker and the speech content is combined into the video of the venue scene, and the combination is The bandwidth occupied by one channel of video transmission is close to the bandwidth occupied by the existing video conference live broadcast system, which meets the requirements of real-time transmission under low bandwidth conditions.

Description

Method and system for live streaming of panoramic conference

technical field

The present invention relates to the technical field of network applications, in particular to a method and system for live broadcasting of panoramic conferences.

Background technique

With the continuous development of Internet technology, the network has become the fastest-growing and increasingly dominant media. With the improvement of people's requirements for obtaining information, people are not limited to only understand text information through the Internet, and more users hope to obtain audio and video information through the Internet to realize news conferences, sports games, live teaching exchanges, business promotion, remote communication, etc. On-site live broadcast of conferences, school opening ceremonies, celebrations, wedding celebrations, etc.

Web conference live broadcast means that people can watch the live audio and video of the remote conference site through the network, such as the speaker, the content of the speech, and so on. The core idea of web conference live broadcast is to use the existing network conditions of LAN, MAN and even the Internet to realize real-time transmission of audio and video signals, and to achieve smooth viewing at the remote end. By setting up an independent signal acquisition device (audio + video) on site, import it into the broadcasting terminal (directing equipment or platform), upload it to the server through the network, and publish it to the website for people to watch. It can be used for live webcasting of government affairs open meetings, mass hearings, court trial live broadcasts, civil servant examination training, product launches, corporate annual meetings, industry annual meetings, exhibition live broadcasts and other applications that are difficult to live broadcast by TV media.

The existing network video conference live broadcast system represented by "Sequoia Live Broadcast System" can realize the storage and sharing of speakers' audio and video materials through the live broadcast server, and users can join the conference to watch the live broadcast completely without barriers by logging in through the webpage. Online questioning and other ways to achieve interaction with the speaker.

Chinese patent CN 201610166645 "Panoramic live broadcast method and device" proposes a panoramic live broadcast method and device, which can synthesize each frame of the captured picture of the live broadcast camera from different angles of view into a panoramic picture and select the picture corresponding to the perspective of the new live picture as the New live screen.

However, the existing network video conference live broadcast system represented by the "Sequoia Tree Live Broadcast System" can only provide videos of local areas in the venue during use. The switch makes it almost impossible for users outside the venue to watch the live broadcast through the Internet. Especially in large-scale events, concerts and other occasions that need to mobilize the enthusiasm of the audience, the viewing experience of users outside the venue is greatly reduced. Although the panorama live broadcast system proposed by the Chinese patent CN201610166645 "Panoramic live broadcast method and device" can broadcast the panoramic information of the venue, the panoramic live broadcast in this invention cannot present the details of a certain viewpoint, and the user can only see the scene at a close distance , unable to clearly see the speaker and the content of the speech at a certain distance.

Contents of the invention

In view of the above problems, the present invention provides a panoramic conference live broadcast method and system.

The panorama conference live broadcast method provided by the present invention comprises the following steps:

Obtain the panoramic image of the conference scene, the video image of the speaker and the content of the speaker's speech respectively;

Receive the user's current visual center;

Performing scene positioning in the panoramic image according to the current visual center to obtain a regional scene video;

Combine the scene video of the area, the video image of the speaker and the speech content to obtain the combined audio and video information and send it.

The present invention also provides a panoramic conference live broadcast system, including:

The first acquisition module, the user respectively acquires the panoramic image of the meeting scene, the video image of the speaker and the speech content of the speaker;

A receiving module, configured to receive the user's current visual center;

A scene positioning module, configured to perform scene positioning in the panoramic image according to the current visual center to obtain regional scene videos;

The combination module is used to combine the scene video of the area, the video image of the speaker and the speech content to obtain and send the combined audio and video information.

The beneficial effects of the present invention are as follows:

In the embodiment of the present invention, after obtaining the lecturer, speech content, and panoramic video information of the venue, in order to meet the requirements of real-time transmission under low bandwidth, and also to meet the requirements of further "immersive" immersive participation of remote participants According to the requirements, the user's visual center information is obtained through the VR device, and only the video information of the venue scene that conforms to the user's visual center is scaled and delivered to the user; according to the characteristics of the conference live broadcast, the video information of the speaker and the speech content are combined into the video of the venue scene , combined into one channel of video, after compression, the bandwidth occupied in transmission is close to that occupied by the existing video conference live broadcast system, which meets the requirements of real-time transmission under low bandwidth conditions.

Compared with the existing network live broadcast system, the technical solution proposed by the present invention introduces the panoramic information of the venue for the defect of insufficient user immersion; compared with the existing panoramic live broadcast system, the technical solution proposed by the present invention aims at details Insufficient description ability combines the information of the speaker and the content of the speech, and transmits the qualified venue scene in real time according to the user's visual center and zoom information, so that users watching the conference live broadcast can have an "immersive" experience under low bandwidth conditions The whole process of the meeting can realize the purpose of real virtual participation.

Description of drawings

Fig. 1 is the flow chart of the panorama conference live broadcasting method of the first method embodiment of the present invention;

Fig. 2 is the flow chart of the panorama conference reception method of the second method embodiment of the present invention;

FIG. 3 is a schematic structural diagram of a panoramic conference live broadcast system according to a first device embodiment of the present invention;

4 is a schematic structural diagram of a VR device in a second device embodiment of the present invention;

Fig. 5 is a schematic diagram of the system structure of Example 1 of the present invention;

6 is a schematic diagram of the process of scene positioning and zooming in Example 1 of the present invention;

Fig. 7 is a schematic diagram of combining the lecturer's video, the voice and video of the speech content, and the scene video of the meeting place in Embodiment 1 of the present invention.

detailed description

Exemplary embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although exemplary embodiments of the present disclosure are shown in the drawings, it should be understood that the present disclosure may be embodied in various forms and should not be limited by the embodiments set forth herein. Rather, these embodiments are provided for more thorough understanding of the present disclosure and to fully convey the scope of the present disclosure to those skilled in the art.

In order to solve the deficiencies of insufficient immersion for users of panoramic live broadcasting in the prior art and insufficient ability to describe panoramic video details, the present invention provides a method and system for panoramic conference live broadcasting, a receiving method and VR equipment. The following describes the present invention in conjunction with the accompanying drawings and embodiments For further details. It should be understood that the specific embodiments described here are only used to explain the present invention, not to limit the present invention.

According to the first method embodiment of the present invention, a method for live broadcasting of a panoramic conference is provided. FIG. 1 is a flow chart of the method for live broadcasting of a panoramic conference according to the first method embodiment of the present invention. As shown in FIG. 1 , according to the first method of the present invention The panoramic conference live broadcast method of the embodiment includes the following processing:

Step 101: Obtain the panoramic image of the meeting scene, the video image of the speaker and the speech content of the speaker respectively.

Specifically, the method for acquiring the panorama image of the meeting scene is: acquiring the panorama of the meeting place through a panorama camera device (a camera module composed of multiple cameras that simultaneously shoots). The method for acquiring the video image of the speaker is: the acquisition of the close-up video of the speaker in the venue is achieved by shooting with a camera.

The speaker's speech content includes voice speech information and graphic speech information. Among them, the method for acquiring the voice speech content is: collecting the speech voice of the speaker (simultaneous interpretation voice is required in certain scenarios). The method for acquiring the graphic speech information is as follows: acquiring the content of the graphic speech (generally PPT) by collecting the output of the speaker's computer graphics card.

Step 102: Receive the user's current visual center.

Step 103: Perform scene positioning in the panoramic image according to the current visual center to obtain regional scene videos.

Specifically, performing scene positioning in the panoramic image according to the current visual center in step 103 includes the following steps:

Calculate the relative position of the current visual center and the previous visual center;

In the panoramic image, the center of vision at the last moment is moved to the relative position to obtain the position of the center of vision at present;

Clipping is performed with the current visual center point as the center according to preset conditions to obtain the regional scene video.

Step 104: Combine the scene video of the area, the video image of the speaker and the content of the speech to obtain and send the combined audio and video information.

Specifically, step 104 combines the scene video of the area to which it belongs, the video information of the speaker and the speech content, and according to whether the graphic speech information of the speaker at the current moment has changed compared with the graphic speech information of the speaker at the previous moment, including The following two processing methods.

The first processing method is: when the graphic speech information of the speaker at the current moment changes, the graphic speech information of the speaker at the current moment, the voice speech information of the speaker at the current moment, and the video information of the speaker , regional scene video to combine.

The second processing method is: when the graphic speech information of the speaker at the current moment has not changed, the graphic speech information of the speaker at the previous moment, the voice speech information of the speaker at the current moment, and the video information of the speaker , regional scene video to combine.

The panorama conference live broadcasting method of the first method embodiment of the present invention also includes the step of obtaining a zooming instruction, and further, taking the position of the current visual center point as the center and cutting according to preset conditions to obtain the regional scene video, including the following steps : Taking the position of the current center of vision as the center, zooming the panoramic image according to the zoom command, and then cropping according to preset conditions to obtain a regional scene video. Specifically, the preset condition may be a preset resolution.

The panoramic conference live broadcast method in the second method embodiment of the present invention further includes: receiving comment information and feeding it back to the speaker, so as to realize interaction with the speaker.

According to the second method embodiment of the present invention, a method for receiving a panoramic conference is provided. FIG. 2 is a flow chart of the method for receiving a panoramic conference according to the second method embodiment of the present invention. As shown in FIG. 2 , according to the second method of the present invention The panoramic conference receiving method of the embodiment includes the following processing:

Step 201: Receive and display the combined video and voice information.

Step 202: Obtain and send the user's current visual center.

The method for receiving a panorama meeting in the second method embodiment of the present invention further includes: obtaining and sending a user's zoom instruction.

Corresponding to the first method embodiment of the present invention, a panoramic conference live broadcast system is provided. FIG. 3 is a schematic structural diagram of the panoramic conference live broadcast system according to the first device embodiment of the present invention. As shown in FIG. 3 , according to the present invention The panorama conference live broadcast system of an embodiment of the device includes: a first acquisition module 30, a receiving module 32, a scene positioning module 34, and a combination module 36; the following describes each module in the embodiment of the present invention in detail.

In the first acquiring module 30, the user respectively acquires the panoramic image of the conference scene, the video image of the speaker, and the speech content of the speaker.

The receiving module 32 is configured to receive the user's current visual center.

The scene positioning module 34 is configured to perform scene positioning in the panoramic image according to the current visual center to obtain regional scene videos.

Specifically, the scene positioning module 34 is specifically used for:

Calculate the relative position of the current visual center and the visual center at the previous moment; change the relative position of the visual center at the previous moment in the panoramic image to obtain the position of the current visual center; take the position of the current visual central point as the center Cut according to the preset conditions to obtain the regional scene video.

The combination module 36 is used to combine the scene video of the area, the video image of the speaker and the speech content to obtain and send the combined audio and video information.

In the panoramic conference live broadcast system of the first device embodiment of the present invention, the first acquiring module is also used to: acquire zooming instructions;

Further, the scene positioning module is further configured to zoom the panoramic image according to the zoom command and then crop it according to preset conditions with the current center of vision as the center, so as to obtain the regional scene video. Specifically, the preset condition may be a preset resolution.

Specifically, the speech includes voice speech information and graphic speech information, and the panoramic conference live broadcast system also includes a static frame judgment module:

The static frame judging module is used to judge whether the graphic speech information of the speaker at the current moment changes;

Further, the combination module is also used to combine the graphic speech information of the speaker at the current moment, the voice speech information of the speaker at the current moment, the video information of the speaker, Combine regional scene videos; when the graphic speech information does not change, combine the graphic speech information of the speaker at the previous moment, the voice speech information of the speaker at the current moment, the video information of the speaker, and the regional scene video .

In the panoramic conference live broadcast system of the first device embodiment of the present invention, a feedback module is also included:

The feedback module is used to feed back the obtained comment information to the speaker, so as to realize the interaction with the speaker.

Corresponding to the second method embodiment of the present invention, a VR device is provided. FIG. 4 is a schematic structural diagram of the VR device in the second device embodiment of the present invention. As shown in FIG. 4 , according to the second device embodiment of the present invention The VR device includes: a display module 40 and a second acquisition module 42; each module of the embodiment of the present invention will be described in detail below.

The display module 40 is configured to receive and display the combined video and audio information.

The second acquiring module 42 is configured to acquire and send the user's current visual center.

The second acquiring module of the VR device in the second apparatus embodiment of the present invention is further configured to acquire a user's scaling instruction.

The VR (Virtual Reality) device described in the present invention is a device capable of displaying VR, such as VR glasses.

In order to illustrate the above method embodiment and device embodiment of the present invention in detail, Example 1 is given. Fig. 5 is a schematic diagram of the system structure of Example 1 of the present invention. As shown in Fig. 5, the virtual conference room system proposed by the present invention mainly includes the following modules:

Module 1: Video Acquisition

This module achieves the acquisition of close-up videos of speakers in the venue through camera shooting.

Module Two: Content Acquisition

This module realizes the acquisition of speech content (usually PPT) by collecting the output of the speaker's computer graphics card.

Module 3: Speech Acquisition

This module collects the voice of the speaker (simultaneous interpretation is required in certain scenarios).

Module Four: Panorama Acquisition

This module obtains the panorama of the venue through panoramic camera equipment (a camera module composed of multiple cameras that simultaneously shoots).

Module Five: Still Frame Judgment

This module judges whether the content of the speech PPT changes (whether the page is switched or whether a new control is loaded).

Module 6: Real-time Stitching

This module performs real-time splicing processing on the panoramic information of the venue obtained by Module 3 to generate a real-time panoramic video of the venue.

Module 7: Scene Positioning and Scaling

This module processes the user's visual center and zoom commands fed back by the user through the VR device, and cuts out the corresponding scene area from the real-time panoramic video of the venue according to the user's visual center and whether to zoom. Fig. 6 is a schematic diagram of the process of scene positioning and zooming in the present invention. See Figure 6 for the specific implementation process.

Module Eight: Video Composition

This module combines the video of the speaker, the voice and video of the speech content, and the scene video of the venue to form a video. The specific implementation process is shown in Figure 7.

Module Nine: Compression

This module uses a video compression algorithm (such as H.264 or H.265 algorithm) to compress the video and voice information obtained in module eight.

Module 10: Encrypted Transmission

This module encrypts and transmits the video and voice information compressed by module 9 through the network.

Module Eleven: Decryption Distribution

This module receives the video and voice information transmitted by module 10, decrypts and distributes the content, and distributes the video and voice information to each user.

Module 12: VR Display

This module uses VR glasses or similar devices with VR display capabilities to display the video of the venue scene combined by Module 8 to the user.

Module Thirteen: Reviews

This module obtains user comments through web page comments and messages.

Module Fourteen: Obtaining Comments

This module feeds back the comments submitted by the user to the speaker to realize the real-time interaction between the user and the speaker.

Module 15: Visual Center Acquisition

This module obtains the visual center of the user's current viewing through the display device used in module 12. The implementation method can be obtained through the pose sensor of the VR device or the pointer position of the operating device capable of displaying the panorama of the venue such as a PC.

Module 16: Acquisition of Scaling Instructions

This module obtains the zoom command issued by the user through the display device used in module 12. The implementation method can be obtained through the operating device that comes with the VR device or a PC that is capable of displaying the panoramic device of the venue.

According to the system structure diagram shown in Fig. 5, the realization process of the technical solution of the present invention is as follows:

Step 1: Module 1 obtains the video of the speaker in the venue, and transmits the digital video information to Module 8;

Step 2: Module 2 obtains the lecture content (usually PPT) in the speaker's computer, converts the information output from the computer to the screen into a video, and passes the video information to Module 5;

Step 3: Module 3 obtains the content of the speaker's voice speech (simultaneous voice interpretation is also required in certain scenarios), and transmits the voice information to Module 8;

Step 4: Module 4 obtains the panorama information of the venue, and transmits multiple sets of video information obtained at the same time to Module 6;

Step 5: After Module 5 receives the video information transmitted by Module 2, it judges whether the video content has changed. If it changes, it transmits the received changed video frame information to Module 8. If it does not change, it does not transmit it to Module 8. any information;

Step 6: After module 6 receives the multi-channel video information that can cover the panoramic view of the venue delivered by module 4, it performs real-time synchronous splicing of the multi-channel video information, and passes the spliced panoramic video of the venue to module 7;

Step 7: Module 7 receives the spliced conference venue panoramic video information delivered by Module 6, and simultaneously receives the visual center information and zoom information of the user's VR device fed back by Module 15 and Module 16, and cuts out the corresponding information from the conference venue panoramic video information. The scene video information of the user's visual center and zoom scale is passed to module 8;

Step 8: Module 8 receives the speaker video information transmitted by module 1, the venue voice information transmitted by module 3, the speech content information transmitted by module 5, and the venue scene information transmitted by module 7, and combines the speaker video with the venue scene information as the background Information and speech content information and venue voice information, and pass the combined video information to module nine;

Step 9: After module nine receives the combined video information delivered by module eight, it compresses the video information, and transmits the compressed video and voice information to module ten;

Step ten: After module ten receives the compressed video and voice information delivered by module nine, it encrypts and transmits the video and voice information through the Internet, and receives it by module eleven;

Step 11: After module 11 receives the encrypted video and voice information transmitted by module 10, it first decrypts the video and voice information, and then distributes the video and voice information (including the speaker, speech content and venue scene) to users, and module 10 Two receive;

Step 12: Module 12 receives the audio and video information of the venue scene delivered by Module 11, displays it to the user through VR equipment (VR helmet, VR glasses, etc.), and obtains the user's current visual center information, and passes it to Module 15 to obtain The zoom control information of the scene video by the user is passed to module 16;

Step 13: Module 15 acquires the user's current visual center information transmitted by Module 12, and transmits it to Module 7;

Step 14: Module 16 obtains the user's zoom control information on the scene video, and transmits it to Module 7;

Step 15: Module 13 receives user comment information (text or voice comment information) and submits it to Module 14;

Step 16: Module 14 receives the user comment information submitted by Module 13 and displays it to the speaker.

After the present invention obtains the lecturer, speech content, and panoramic video information of the venue, in order to meet the requirements of real-time transmission under low bandwidth, and also to meet the needs of remote participants for further "immersive" immersive participation, Obtain the user's visual center information and zoom control information through the VR device, and only the video information of the venue scene that conforms to the user's visual center is scaled and delivered to the user; according to the characteristics of the conference live broadcast, the video information of the speaker and the speech content is combined into the venue scene In the video, one channel of video is combined. After compression, the bandwidth occupied in transmission is close to the bandwidth occupied by the existing video conference live broadcast system, which meets the requirements of real-time transmission under low bandwidth conditions.

Compared with the existing network live broadcast system, the technical solution proposed by the present invention introduces the panoramic information of the venue for the defect of insufficient user immersion; compared with the existing panoramic live broadcast system, the technical solution proposed by the present invention aims at the panoramic video details Insufficient description ability combined speaker and speech content information, and real-time transmission of qualified venue scenes according to the user's visual center and zoom information, so that users watching the conference live broadcast can have an "immersive" experience under low bandwidth conditions The whole process of the meeting can realize the purpose of real virtual participation.

In summary, the present invention establishes a system that enables participants who cannot attend the meeting to watch the meeting in person. By adding the panoramic video of the venue, the purpose of "immersive" participation of the participants is realized, and the presentation of the details of the meeting is realized by combining the video information of the speaker and the content of the speech. Cutting, data compression and other methods have realized the real-time conference panorama live broadcast under the existing bandwidth.

The above description is only an embodiment of the present invention, and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and changes. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present invention shall be included within the scope of the claims of the present invention.

Claims (10)

1. a kind of panorama meeting live broadcasting method, it is characterised in that including：

Panoramic picture, the video image of speechmaker and the speech content of speechmaker of conference scenario are obtained respectively；

Receive the Current vision center of user；

Scene positioning is carried out in the panoramic picture according to the Current vision center, region scene video is obtained；

The region scene video, the video image of speechmaker and speech content are combined, the voice after being combined is regarded Frequency information is simultaneously sent.

2. panorama meeting live broadcasting method as claimed in claim 1, it is characterised in that according to the Current vision center described Scene positioning is carried out in panoramic picture, including：

Calculate Current vision center and the relative position of last moment optic centre；

Last moment optic centre is moved into the relative position in the panoramic picture, the position of Current vision central point is obtained Put；

Cut centered on the Current vision center position according to default condition, obtain region scene video.

3. panorama meeting live broadcasting method as claimed in claim 2, it is characterised in that also include：

Obtain scaling instruction；

It is described to be cut centered on the Current vision center position according to default condition, obtain region scene visual Frequently, including：

Centered on the Current vision center position, according to it is described scaling instruction the panoramic picture is zoomed in and out after by Cut according to default condition, obtain region scene video.

4. panorama meeting live broadcasting method as claimed in claim 1, it is characterised in that include voice speech letter inside the speech Breath and picture and text speech information, affiliated area scene video, the video information of speechmaker and speech content are combined, including：

Judge whether the picture and text speech information of current time speechmaker is sent out compared with the picture and text speech information of last moment speechmaker Changing；

When the picture and text speech information of the current time speechmaker changes, the picture and text of the current time speechmaker are drilled Say that information, the voice speech information of current time speechmaker, the video information of speechmaker, region scene video are combined；

When the picture and text speech information of the current time speechmaker does not change, the picture and text of last moment speechmaker are given a lecture Information, the voice speech information of current time speechmaker, the video information of speechmaker, region scene video are combined.

5. panorama meeting live broadcasting method as claimed in claim 1, it is characterised in that also include：

Receive comment information and feed back to speechmaker, to realize the interaction with speechmaker.

6. a kind of panorama meeting live broadcast system, it is characterised in that including：

First acquisition module, user obtains drilling for the panoramic picture of conference scenario, the video image of speechmaker and speechmaker respectively Say content；

Receiving module, the Current vision center for receiving user；

Scene locating module, for carrying out scene positioning in the panoramic picture according to the Current vision center, obtains area Domain scene video；

Composite module, for affiliated area scene video, the video image of speechmaker and speech content to be combined, obtains group Voice and video information after conjunction is simultaneously sent.

7. panorama meeting live broadcast system as claimed in claim 6, it is characterised in that the scene locating module specifically for：

Calculate Current vision center and the relative position of last moment optic centre；

Last moment optic centre is changed into the relative position in the panoramic picture, the position of Current vision central point is obtained Put；

Cut centered on the Current vision center position according to default condition, obtain region scene video.

8. panorama meeting live broadcast system as claimed in claim 7, it is characterised in that first acquisition module, is additionally operable to：Obtain Scaling is taken to instruct；

The scene locating module, is additionally operable to centered on the Current vision center position, according to the scaling instruction pair The panoramic picture is cut after zooming in and out according to default condition, obtains region scene video.

9. panorama meeting live broadcast system as claimed in claim 6, it is characterised in that include voice speech letter inside the speech Breath and picture and text speech information, the live system of panorama meeting also include frozen frame judge module：

The frozen frame judge module, for judging whether the picture and text speech information of current time speechmaker changes；

The composite module, is additionally operable to when picture and text speech information changes, by the figure of the current time speechmaker Text speech information, voice speech information, the video information of speechmaker, the region scene video carry out group of current time speechmaker Close；When picture and text speech information does not change, the picture and text speech information of last moment speechmaker, current time are given a lecture The voice speech information of people, the video information of speechmaker, region scene video are combined.

10. panorama meeting live broadcast system as claimed in claim 6, it is characterised in that also including feedback module：

The feedback module, for the comment information of acquisition to be fed back into speechmaker, to realize the interaction with speechmaker.