CN110266984B - A PTZ camera intelligent analysis teaching recording and broadcasting integrated machine - Google Patents

Abstract

The invention discloses a PTZ camera intelligent analysis teaching recording and broadcasting integrated machine, comprising a PTZ and a camera unit mounted on the PTZ; the camera unit comprises: a camera for collecting audio and video signals of a teaching scene; a host processing module, It is used to receive audio and video signals, encode and encapsulate the audio and video signals. On the one hand, the packaged audio and video are stored locally, and on the other hand, the audio and video are released in the form of streaming media through the 4G network module; the video analysis DSP module is used for Receive the video signal, classify and identify the target in the video signal, output the motion control signal to the pan-tilt control module according to the result of the classification and identification, and output the camera focus signal to the camera control module, so as to track the target in real time; The integration with the PTZ, the identification and analysis of specific target behaviors in classroom scenarios, combined with the real-time transmission of 4G networks, will help to further improve the intelligence of the teaching classroom.

Description

A PTZ camera intelligent analysis teaching recording and broadcasting integrated machine

technical field

The invention relates to the field of intelligent teaching, in particular to a pan-tilt camera intelligent analysis and teaching recording and broadcasting integrated machine.

technical background

With the continuous development of computer vision and network streaming media technology, coupled with the continuous maturity of cloud computing, mobile Internet and big data analysis technologies, its deep integration with modern education has spawned a new generation of intelligent teaching recording and broadcasting systems.

The intelligent recording and broadcasting system generally adopts the separation method of the camera and the recording and broadcasting host, that is, the front end uses an independent camera to obtain video data, and inputs the video into the recording and broadcasting host through SDI and HDMI. After the recording and broadcasting host processes the video data, a On the one hand, the video is stored, and on the other hand, the network is broadcast live through WiFi or Ethernet. Most of the live broadcasts in this mode are limited to local area network live broadcasts.

The camera is separated from the recording and broadcasting host, which results in a large space occupation and an insufficiently compact structure; the network live broadcast mode based on WiFi or Ethernet has greatly limited its applicable scenarios; in the past, the recording and broadcasting host was only responsible for recording and live broadcasting, and did not integrate Corresponding intelligent recognition algorithms will take extra time in post-processing.

SUMMARY OF THE INVENTION

In view of the above problems, the purpose of the present invention is to provide a PTZ camera intelligent analysis teaching recording and broadcasting integrated machine, which adopts an integrated solution, which can realize the real-time live broadcast of the classroom and the classroom on-demand in the wide area network, and at the same time can also Realize automatic analysis of goals and behaviors in classroom scenarios and complete real-time tracking of teachers and students' movements.

The concrete technical scheme of the present invention is as follows:

A PTZ camera intelligent analysis teaching recording and broadcasting integrated machine, comprising a PTZ and a camera unit mounted on the PTZ;

The camera unit includes a camera, a host processing module 4G network module, a video analysis DSP module, a camera control module, and a PTZ control module;

The camera is used to collect video signals and audio signals of the teaching scene;

The host processing module is used to receive video signals and audio signals, encode the video and audio, and encapsulate the encapsulated audio and video. ;

The video analysis DSP module is used to receive the video signal, classify and identify the target in the video signal, and output the motion control signal to the pan-tilt control module according to the result of the classification and identification, and output the camera focus signal to the camera control module, so as to Targets are tracked in real time;

The camera control module is used to control the focus of the camera;

The PTZ control module is used to control the movement of the PTZ.

Further, in the video analysis DSP module, the classification and recognition of the target includes the identification of the teacher and the student and the recognition of the specific behavior of the teacher and the student in the classroom scene.

Further, for teachers, their specific behaviors include teacher writing on the blackboard, teachers ask questions, and teachers linger; for students, their specific behaviors include students raising their hands, standing up, and sitting down.

Further, the real-time tracking of the target includes the tracking of the teacher's behavior or the tracking of the student's behavior; the tracking of the teacher's behavior mainly includes the tracking movement of the pan-tilt camera when the teacher is wandering; the tracking of the student's behavior mainly includes when the student is standing. A close-up of the camera's zoom lens and the camera's zoom lens release while the student is seated.

Further, the classification and identification of the target in the video signal is realized by training a corresponding classifier, and the specific method is as follows:

S1: Train the classifier:

S11: Obtain the image training sample data set in the classroom scene. The data set includes various action information that needs to be identified. Action information: teacher writing on the blackboard, teacher asking questions, teacher wandering, student raising hand, student standing up, and student sitting down;

S12: Extract sample features;

S13: Make a sample training set, and correspond the extracted features of the samples to the labels;

S14: establish a classifier model;

S15: Perform classifier training on the sample training set, and the termination condition is that the predetermined precision is reached or the predetermined number of training times is reached;

S2: Use the trained classifier to classify and recognize classroom actions:

S21: obtain real-time video, and obtain the moving target area;

S22: Feature extraction is performed on the moving target area;

S23: Classify the extracted features using the classifier trained in step S1 to obtain a classification result, and correspondingly, a corresponding behavior action category can be obtained.

Further, in the S12, SIFT features, HOG features and LBP features are used to extract the sample features.

Further, in S14, the classifier model adopts SVM classifier, K-nearest neighbor classifier or Bayesian classifier.

Further, the camera unit further includes a local data transmission module, and the audio and video packaged by the host processing module is directly transmitted to the local display through the local data transmission module.

Further, the 4G network module can directly access the Internet, and users can directly access real-time streaming media through the external network.

Further, the pan/tilt is a suspended integrated pan/tilt, and its horizontal movement adopts a worm gear transmission mode, and its pitch direction movement adopts a planetary gear transmission mode.

With respect to the prior art, the beneficial effects of the present invention are as follows:

1. The integrated recording and broadcasting host solution is adopted, which solves the large space occupied by the previous recording and broadcasting host system and the cumbersome installation and configuration process.

2. The 4G network is used for live streaming media transmission, which is more convenient for users to access anytime, anywhere.

3. It is equipped with a dedicated teacher and student analysis camera in the classroom scene to intelligently track the behavior of teachers and students, which greatly improves the intelligence of the education classroom.

Description of drawings

Figure 1 is the overall structure diagram of the integrated PTZ;

Figure 2 is a schematic diagram of the dismantling of the integrated gimbal;

Figure 3 is a schematic diagram of the deceleration structure of the integrated gimbal;

Fig. 4 is the concrete structural block diagram of the main board of the integrated video recording head;

Fig. 5 is the structural representation of the main board of the integrated recording and broadcasting camera;

Fig. 6 is the functional flow chart of the teaching recording and broadcasting machine;

in:

1: Camera unit 2: Gimbal

3: Tilt motion output end cap 4: Right end cap

5: Right swing arm 6: Gimbal tee

7: Connector 8: Bearing

9: PTZ bracket 10: Worm gear shaft

11: Gimbal gear box 12: Left swing arm

13: Power input for pitching motion 14: Power input for horizontal motion

15: PTO for pitching motion 16: PTO for horizontal motion

Detailed ways

The present invention will be described in more detail below in conjunction with the accompanying drawings:

As shown in FIG. 1 , a PTZ camera intelligent analysis teaching recording and broadcasting integrated machine includes a PTZ 2 and a camera unit 1 mounted on the PTZ 2 .

According to the present invention of the above-mentioned scheme, the described pan/tilt 2 is a suspended integrated pan/tilt. As shown in FIG. 2 , its horizontal movement adopts a worm gear and worm drive mode, the worm gear is fixed on the worm gear shaft 10, and the worm gear is relative to the worm gear. Do a rotary motion, that is, the horizontal rotation of the gimbal, wherein the worm gear shaft 10 on the gimbal bracket 9 and the bearing 8 adopt an interference fit, the bearing 8 and the connecting piece 7 also adopt an interference fit, and the connecting piece 7 and the gimbal tee 6 Thread fit is adopted; the movement in the pitching direction of the gimbal adopts planetary gear transmission, the planetary gear and the pitching movement output end cover 3 are meshed with internal and external gears, the position of the central axis of the planetary pinion remains unchanged, and the pitching movement output end cover 3 is relative to its central axis. The rotation movement drives the left swing arm 12 and the right swing arm 5 to swing, thereby driving the camera unit 1 to perform a pitching motion. The right end cover 4 is used to assist in positioning the pitching motion output end cover 3 and bear a certain load.

According to the present invention of the above scheme, the pan-tilt gear box 11 is used as the core component of the integrated pan-tilt head. The specific structure is shown in FIG. 3 . As the power output 15 for pitching motion, it drives the camera unit 1 to do pitching motion; the eccentric position motor acts as the power input 14 for horizontal motion, and drives the worm to rotate as the power output 16 for horizontal motion. 1 horizontal movement; worm gear transmission and planetary gear transmission have the advantages of large transmission ratio and compact structure, which are very suitable for the application scene of the integrated pan/tilt.

According to the present invention of the above scheme, the specific structural block diagram of the main board of the camera unit is shown in FIG. 4 , and the main board includes an audio signal interface, a video signal interface, a host processing module, a hard disk storage interface, a 4G network module, and a local data transmission module. , video analysis DSP module, camera control module and PTZ control module; the video signal and audio signal are respectively obtained from the camera and transmitted to the host processing module, the host processing module encodes the audio and video, and the encoded audio and video are processed by the host The processing module continues to encapsulate it. On the one hand, the encapsulated video is stored locally through the hard disk storage interface, on the other hand, the network stream is pushed out through the 4G network module, and on the other hand, it can be directly passed through the local data transmission module without delay and loss. The video signal is simultaneously transmitted to the video analysis DSP module to classify and identify the target in the video signal. According to the result of the classification and identification, the motion control signal is output to the PTZ control module, and the camera focus signal is output to the camera control module. , so as to track the target in real time; the camera control module is used to control the focus of the camera; the PTZ control module is used to control the movement of the PTZ.

According to the present invention of the above scheme, a schematic diagram of the main board structure of the integrated recording and broadcasting camera according to the preferred embodiment is provided, as shown in FIG. , its main processing chip is preferably HiSilicon HI3531A chip; for 4G network module for 4G network function, its chip is preferably MT7620 series chip, which provides data through SIM card to achieve 4G network transmission; video analysis for video analysis function and motion tracking function DSP module, its chip is preferably TMS320 series chip; the PTZ control module and camera control module used for PTZ control function, its chip is preferably STM32 series chip, through two channels of RS232 to control the related motion of PTZ and camera respectively; local data transmission module On the one hand, data can be transmitted directly through the HDMI interface, on the other hand, it can be converted from parallel output to serial output through the GV7600 chip, and then transmitted through the SDI interface. In addition, the camera motherboard provides SATA interface and SD card interface, For local video storage.

According to the present invention of the above scheme, the network live broadcast function accesses the Internet based on the 4G network module, and the user can directly access the real-time streaming media of the recording and broadcasting host through the external network, and the recording and broadcasting host adopts the RTMP real-time streaming media protocol to transmit audio and video real-time streams, The basic process is as follows:

S01: The recording and broadcasting host obtains the real-time audio and video stream through the video processing module, and encodes it into a specific format, the audio is encoded in the AAC format, and the video is encoded in the H.264 format.

S02: Encapsulate the encoded audio and video data according to the FLV format, so that it conforms to the RTMP transmission standard.

S03: Use the LibRTMP streaming media push framework to push the real-time stream to the RTMP streaming media server.

S04: RTMP streaming media player plays real-time streaming media.

According to the present invention of the above scheme, the video analysis DSP module classifies and recognizes the target in the video signal.

The classification and identification of the target includes the identification of teachers and students and the identification of specific behaviors of teachers and students in classroom scenarios.

For teachers, their specific behaviors include teacher writing on the blackboard, teachers ask questions, and teachers wander; for students, their specific behaviors include students raising their hands, standing up, and sitting down.

For the behavior analysis of teachers and students in classroom scenarios, the classification and recognition of behaviors are realized by training corresponding classifiers. The specific methods are as follows:

S1: Train the classifier:

S11: Acquire a large number of image training sample data sets in the classroom scene, which include various action information that needs to be identified, such as teacher writing on the blackboard, teacher asking questions, teacher wandering, students raising their hands, students standing up, and students sitting down;

S12: Extract sample features, such as using SIFT features, HOG features, and LBP features.

S13: Create a sample training set, and map the extracted features of the samples to labels.

S14: Build a classifier model, such as SVM classifier, K-nearest neighbor classifier or Bayesian classifier.

S15: Perform classifier training on the sample training set, and the termination condition is reaching a predetermined precision or reaching a predetermined number of training times.

S2: Classroom action action recognition using the trained classifier:

S21: Obtain a real-time video, and use a frame difference method or a background method to obtain a moving target area.

S22: Perform feature extraction on the moving target area, such as SIFT features, HOG features, and LBP features, consistent with the sample production method.

S23: Use the classifier to classify the extracted features to obtain a classification result, and correspondingly, the corresponding behavior action category can be obtained.

According to the present invention of the above scheme, the motion tracking function can be used for the tracking of the teacher's behavior or the tracking of the student's behavior; the tracking of the teacher's behavior mainly includes the tracking motion of the PTZ camera when the teacher is wandering; the tracking of the student's behavior , mainly including the close-up of the zoom lens of the camera when the student is standing and the release of the zoom lens of the camera when the student is sitting down. Motion tracking is realized based on behavior recognition. First, the corresponding behavior actions are obtained according to the video analysis DSP module, and the corresponding tracking type is executed according to the specific action type. If the action type is teacher wandering, the PTZ control module is activated to control the PTZ. Movement to achieve the effect of motion tracking; if the action type is the student standing up, control the PTZ control module and the camera control module at the same time, first the PTZ control module controls the PTZ to move until the corresponding student is in the middle of the image, and then the camera control module controls Zoom lens close-up, zoom in to achieve the effect of close-up of students.

According to the present invention of the above scheme, the flow chart of the realization of each functional module of the teaching recording and broadcasting integrated machine is shown in FIG. 6 , when the integrated machine system is turned on, recording or on-demand can be selected; when the system chooses recording, on the one hand The audio and video collected by the camera are encoded by the main processing unit, and then the audio and video are encapsulated into flv format video, which are respectively used for local storage, network live broadcast and local playback. On the other hand, the video signal collected by the camera is used for video motion analysis and analysis. The result is used for motion tracking; when the system selects on-demand, it accesses the local storage through the 4G network, selects the recording record to be watched, downloads the on-demand video to the local, and completes the playback.

The above descriptions are only preferred embodiments of the present invention, and are not intended to limit the scope of the present invention. Any equivalent structure or equivalent process transformation made by using the contents of the description and drawings of the present invention, or directly or indirectly applied to other related All technical fields are similarly included in the scope of patent protection of the present invention.

Claims (9)

1. A cloud deck camera shooting intelligent analysis teaching recording and broadcasting integrated machine is characterized by comprising a cloud deck and a camera unit carried on the cloud deck;

the camera unit comprises a camera, a host processing module 4G network module, a video analysis DSP module, a camera control module and a holder control module;

the camera is used for collecting video signals and audio signals of a teaching scene;

the host processing module is used for receiving the video signal and the audio signal, coding and packaging the video and the audio, and the packaged audio and video are stored locally on the one hand and are pushed out in a streaming media form through the 4G network module on the other hand;

the video analysis DSP module is used for receiving the video signals, classifying and identifying targets in the video signals, outputting a motion control signal to the holder control module and outputting a camera focusing signal to the camera control module according to the classification and identification results, and accordingly tracking the targets in real time;

the camera control module is used for controlling the focusing of the camera;

the holder control module is used for controlling the movement of the holder;

the classification and identification of the target in the video signal are realized by training a corresponding classifier, and the specific method is as follows:

s1: training a classifier:

s11: acquiring a picture training sample data set in a classroom scene, wherein the data set comprises various action information needing to be identified: teacher writing on blackboard, teacher asking questions, teacher loitering, student lifting hands, student standing up, student sitting down;

s12: extracting sample characteristics;

s13: making a sample training set, and corresponding the extracted features of the sample to the label;

s14: establishing a classifier model;

s15: carrying out classifier training on the sample training set, wherein the termination condition is that the preset precision is reached or the preset training times are reached;

s2: using the trained classifier to classify and recognize the classroom behavior and actions:

s21: acquiring a real-time video and a moving target area;

s22: extracting the characteristics of the moving target area;

s23: and classifying the extracted features by using the classifier obtained by training in the step S1 to obtain a classification result, and correspondingly obtaining the corresponding behavior and action category.

2. The pan-tilt-zoom-camera intelligent analysis teaching recording and broadcasting all-in-one machine as claimed in claim 1, wherein in the video analysis DSP module, the classification and identification of the targets comprises identification of teachers and students and identification of specific behaviors of teachers and students in classroom scenes.

3. The intelligent cloud deck camera analysis, teaching and recording integrated machine as claimed in claim 2, wherein specific behaviors of the teacher comprise teacher writing, teacher asking questions and teacher wandering; specific behaviors for a student include the student lifting his hands, standing up and sitting down.

4. The pan-tilt-zoom-camera intelligent analysis teaching recording and broadcasting all-in-one machine according to claim 3, wherein the real-time tracking of the target comprises the tracking of teacher behaviors or the tracking of student behaviors; the teacher behavior tracking mainly comprises the tracking motion of a pan-tilt camera when the teacher wanders; the tracking of the student behaviors mainly comprises the close-up of a zoom lens of a camera when a student stands and the release of the zoom lens of the camera when the student sits down.

5. The pan-tilt-zoom-camera intelligent analysis teaching and recording all-in-one machine according to claim 1, wherein in the step S12, the SIFT feature, the HOG feature and the LBP feature are used for extracting the sample features.

6. The tripod head camera shooting intelligent analysis teaching recording and broadcasting all-in-one machine as claimed in claim 1, wherein in the S14, an SVM classifier, a K neighbor classifier or a bayesian classifier is adopted as a classifier model.

7. The intelligent analysis teaching recording and broadcasting integrated machine with the pan-tilt camera shooting function as claimed in claim 1, wherein the camera unit further comprises a local data transmission module, and the audio and video packaged by the host processing module is directly transmitted to a local display through the local data transmission module.

8. The pan-tilt camera shooting intelligent analysis teaching recording and broadcasting all-in-one machine as claimed in claim 1, wherein the 4G network module can directly access the Internet, and a user directly accesses real-time streaming media through an external network.

9. The integrated machine of claim 1, wherein the cradle head is a suspended integrated cradle head, the horizontal movement of which adopts a worm and gear transmission mode, and the pitch movement of which adopts a planet gear transmission mode.

Images