+

WO2013039062A1 - Dispositif d'analyse faciale, procédé d'analyse faciale et support à mémoire - Google Patents

Dispositif d'analyse faciale, procédé d'analyse faciale et support à mémoire Download PDF

Info

Publication number
WO2013039062A1
WO2013039062A1 PCT/JP2012/073185 JP2012073185W WO2013039062A1 WO 2013039062 A1 WO2013039062 A1 WO 2013039062A1 JP 2012073185 W JP2012073185 W JP 2012073185W WO 2013039062 A1 WO2013039062 A1 WO 2013039062A1
Authority
WO
WIPO (PCT)
Prior art keywords
face
analysis
unit
student
lecture
Prior art date
Application number
PCT/JP2012/073185
Other languages
English (en)
Japanese (ja)
Inventor
史雄 仲矢
昌直 片桐
Original Assignee
国立大学法人大阪教育大学
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 国立大学法人大阪教育大学 filed Critical 国立大学法人大阪教育大学
Publication of WO2013039062A1 publication Critical patent/WO2013039062A1/fr

Links

Images

Classifications

    • G—PHYSICS
    • G09—EDUCATION; CRYPTOGRAPHY; DISPLAY; ADVERTISING; SEALS
    • G09B—EDUCATIONAL OR DEMONSTRATION APPLIANCES; APPLIANCES FOR TEACHING, OR COMMUNICATING WITH, THE BLIND, DEAF OR MUTE; MODELS; PLANETARIA; GLOBES; MAPS; DIAGRAMS
    • G09B5/00—Electrically-operated educational appliances
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06Q—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
    • G06Q50/00—Information and communication technology [ICT] specially adapted for implementation of business processes of specific business sectors, e.g. utilities or tourism
    • G06Q50/10—Services
    • G06Q50/20—Education
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V20/00—Scenes; Scene-specific elements
    • G06V20/50—Context or environment of the image
    • G06V20/52—Surveillance or monitoring of activities, e.g. for recognising suspicious objects
    • G06V20/53—Recognition of crowd images, e.g. recognition of crowd congestion
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
    • G06V40/10—Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
    • G06V40/16—Human faces, e.g. facial parts, sketches or expressions

Definitions

  • the present invention relates to a face analysis device that recognizes the faces of one or more students in a lecture and analyzes the faces in the lecture.
  • a web server having a number confirmation program for acquiring classroom information for each person, and a mail server for sending an e-mail to an administrator who manages the classroom when the class exists by the number confirmation program
  • a classroom management system characterized by comprising (see, for example, Patent Document 1).
  • JP 2009-129117 A (first page, FIG. 1 etc.)
  • the face analysis apparatus of the present invention recognizes a student's face from a student image storage unit that stores a student image, which is a moving image obtained by capturing one or more students' faces during a lecture, and a student image.
  • the face analysis apparatus includes a face analysis unit that performs analysis on the recognized face, and an output unit that outputs information related to the analysis result by the face analysis unit.
  • the face analysis device of the present invention further includes a statistical analysis unit that performs a statistical analysis specified in advance using the analysis result acquired by the face analysis unit in the face analysis device, and the output unit performs statistical analysis. It is a face analysis device that outputs a result.
  • the face analysis device of the present invention includes a statistical analysis unit that performs a statistical analysis specified in advance using the analysis result acquired by the face analysis unit in the face analysis device, and a statistical analysis result of the statistical analysis unit.
  • a lecture evaluation unit that evaluates a lecture is further provided, and the output unit is a face analysis device that outputs an evaluation result of the lecture evaluation unit.
  • the face analysis device of the present invention is the face analysis device, wherein the face analysis device includes at least one of a lecturer image that is a moving image of a lecturer during a lecture or lecturer voice information that is voice information of a lecturer during a lecture.
  • a lecturer information storage unit for storing lecturer information and a change detection unit for detecting changes in the lecturer information, and the face analysis unit is a class acquired immediately after the change detection unit detects a change.
  • the face analysis apparatus acquires a face analysis result for the raw image, and the output unit outputs information related to the face analysis result for the student raw image acquired immediately after the change detection unit detects the change.
  • the face analysis device of the present invention is a related statistics for performing statistical analysis specified in advance using the analysis result for the student raw image acquired immediately after the change detection unit detects the change in the face analysis device.
  • the face analysis apparatus further includes an analysis unit, and the output unit outputs a result of statistical analysis of the related statistical analysis unit.
  • the face analysis device of the present invention is a related statistics for performing statistical analysis specified in advance using the analysis result for the student raw image acquired immediately after the change detection unit detects the change in the face analysis device.
  • An analysis unit and a related lecture evaluation unit that evaluates a lecture using the statistical analysis result of the related statistical analysis unit are further provided, and the output unit is a face analysis device that outputs an evaluation result of the related lecture evaluation unit.
  • the face analysis unit acquires the analysis result of the face of the student along the time series, and the output unit displays the analysis result of the face of the face analysis unit. It is a face analyzer that outputs in time series.
  • the face analysis device of the present invention further includes a designation receiving unit that receives designation for the analysis result of the face at one time point that is output by the output unit in the face analysis device, and the output unit is designated by the designation receiving unit. It is a face analysis device that outputs a student raw image at a time corresponding to a received analysis result.
  • the display mode is such that the student's face cannot be identified by the student.
  • This is a face analysis device that outputs the trained raw images.
  • Block diagram of face analysis apparatus Flow chart explaining the operation A diagram schematically showing part of the frame image of the student image Schematic diagram showing a frame image with the same face recognized
  • the figure which shows the same face analysis result management table Figure showing the statistical analysis result management table Figure showing the display example Figure showing the display example Figure showing the display example Block diagram of face analysis apparatus according to Embodiment 2 of the present invention Flow chart explaining the operation Figure showing a part of the lecturer information Figure showing the same time analysis result management table Figure showing the statistical analysis result management table
  • the figure which shows an example of the external appearance of the computer system in each embodiment of this invention The figure which shows an example of a structure of the computer system
  • FIG. 1 is a block diagram of face analysis apparatus 1 in the present embodiment.
  • the face analysis apparatus 1 includes a first photographing unit 11, a student raw image storage unit 12, a face analysis unit 13, a statistical analysis unit 14, a lecture evaluation unit 15, a designation receiving unit 16, and an output unit 17.
  • the first photographing unit 11 photographs the faces of one or more students during the lecture and acquires images of the faces of the one or more students during the lecture.
  • the face image of one or more students means, for example, an image including at least one student's face. It is preferable that the 1st imaging
  • the first photographing unit 11 performs photographing from the front side of one or more students and acquires a moving image.
  • Lecture is a concept including classes in elementary and junior high schools, high schools, universities, cram schools, prep schools, and the like.
  • the student is, for example, a student taking a lecture. A student may be considered a student.
  • the first imaging unit 11 captures a moving image. Note that a plurality of still images taken continuously at a predetermined or indefinite timing designated in advance may be considered as moving images here. The same applies to the following.
  • the first imaging unit 11 accumulates the captured moving images in, for example, the student raw image storage unit 12.
  • the first imaging unit 11 acquires a moving image associated with a time series. For example, a moving image composed of frame images associated with so-called time codes is acquired. A frame image may be considered as each still image constituting a moving image.
  • photography part 11 is realizable with a camera, for example.
  • the student raw image storage unit 12 stores student raw images.
  • the student image is a moving image obtained by photographing one or more students' faces during a lecture.
  • the student raw image storage unit 12 stores a student raw image captured by the first imaging unit 11.
  • the student image is preferably an image obtained by photographing a plurality of students' faces.
  • the student image may be composed of a plurality of moving images obtained by dividing a plurality of students into a plurality of areas.
  • Two or more frame images and still images constituting the student image are usually associated with a time series, for example.
  • each frame image or still image constituting the student raw image is associated with a time code indicating the shooting time, for example.
  • the file format of the student image and the compression method are not limited.
  • the size and aspect ratio of the image do not matter.
  • the student raw image is a moving image in a format such as MPEG2, MPEG4, motion JPEG, AVCHD, or DV.
  • the time here may be an absolute time such as a standard time or a relative time such as an elapsed time from the start of imaging.
  • the storage here is a concept including temporary storage.
  • the student raw image storage unit 12 is preferably a non-volatile recording medium, but can also be realized by a volatile recording medium. The same applies to other storage units.
  • the face analysis unit 13 recognizes one or more students' faces from the student raw images stored in the student raw image storage unit 12 and analyzes the recognized faces. For example, the face analysis unit 13 analyzes the face of each student recognized from the student image, for example, and shows the analysis results for the analysis items such as facial expressions, face orientation, line-of-sight direction, and blinking. Get information.
  • the information indicating the analysis result of the analysis item related to blinking is, for example, information such as blinking, closing eyes, or opening eyes.
  • the analysis result acquired here may be considered as a value indicating a feature amount for each analysis item, for example. This value may be a numerical value, a character string indicating the analysis result, or the like.
  • the face analysis result may be, for example, information indicating facial expression, face orientation, line of sight, and blinking obtained by analysis.
  • the facial expression is, for example, an expression such as laughter, anger, or sadness.
  • the face analysis unit 13 detects, for example, one or more faces for consecutive frame images before or after one time point (for example, one time) of the student raw image, and faces detected at substantially the same location. By comparing these images with each other between the frame images, information indicating the facial expression of the student and the blinking state at one time point may be acquired. Further, information indicating the face direction, the line of sight, etc. may be acquired by detecting one or more faces in each frame image at one time of the student image and analyzing the face image. Face detection technology, facial expression recognition technology, etc.
  • the face analysis unit 13 uses, for example, the student image, and acquires the student's face analysis results at one or more time points, preferably at a plurality of time points.
  • the time point here may be considered to mean a frame image at one time point of the student image.
  • the face analysis unit 13 recognizes one or more students' faces for each of the frame images at a plurality of time points constituting the student image, and for each recognized student's face, for each frame image Get analysis results.
  • the frame image before or after each frame may be used as appropriate. The same applies to the following.
  • the plurality of time points are, for example, a plurality of times at predetermined time intervals within a time when the student image is taken, or a plurality of times and frame images specified by the user.
  • the face analysis unit 13 may sequentially acquire the analysis results for every n seconds (n is an integer equal to or greater than 1) for the student image that is a moving image.
  • the statistical analysis unit 14 performs statistical analysis designated in advance using the analysis result acquired by the face analysis unit 13.
  • the statistical analysis unit 14 may perform any statistical analysis.
  • the statistical analysis unit 14 may individually perform statistical analysis for each of one or more time points of the student image, or may comprehensively analyze the entire student image corresponding to one lecture or a part thereof.
  • Statistical analysis may be performed.
  • the statistical analysis unit 14 may perform statistical analysis on the analysis result for each analysis item acquired by the face analysis unit 13, or use the analysis result for each analysis item acquired by the face analysis unit 13. Then, a comprehensive analysis result may be acquired and statistical analysis may be performed on the comprehensive analysis result.
  • the statistical analysis unit 14 aggregates the number of faces detected for each analysis item for each analysis result acquired by the face analysis unit 13 at each of one or more time points of the moving image.
  • the analysis items include, for example, facial expressions, directions, line-of-sight directions, blinks, and the like.
  • the statistical analysis unit 14 may acquire, as a statistical analysis result, information that associates the tabulation result or a ratio of the tabulation result with respect to the entire detected face with information that identifies each time point. .
  • the result of counting the number of faces may be normalized as appropriate. The same applies to the following.
  • the information for specifying each time point is, for example, the time corresponding to each time point, the elapsed time from the start of the lecture, the frame number, or the like.
  • the statistical analysis unit 14 uses, for example, one face or a part of the one lecture to detect faces detected by analysis items at one or more time points of the student raw images obtained in one lecture.
  • the total value or the ratio of the total value to the number of detected faces may be acquired as a statistical analysis result.
  • the statistical analysis unit 14 converts the analysis result for each analysis item for each of the plurality of faces recognized for each of one or more time points of the student image into numerical values using a conversion table, a conversion formula, or the like prepared in advance. Convert.
  • the statistical analysis part 14 may acquire the average value, dispersion
  • a numerical value used in face analysis may be used.
  • the analysis result for each analysis item is, for example, an expression, a face direction, a line-of-sight direction, or the like.
  • the statistical analysis unit 14 may acquire, as a statistical analysis result, information indicating a counting result or a time point at which the ratio is high from a counting result of the number of faces for each analysis item of the analysis result acquired for each time point or a ratio thereof. good.
  • the information indicating the time when the ratio is high is, for example, the time when the ratio is high, the elapsed time from the start of the lecture, or the like.
  • the statistical analysis unit 14 extracts, for example, face analysis results over a plurality of points in time for each student face recognized by the face analysis unit 13 as a statistical analysis result, or arranges them in time series. Or you may.
  • the analysis results of student faces over a plurality of time points are, for example, analysis results acquired at different time points of the student image captured for one lecture.
  • facial analysis results for one or more students at two or more points in time may be acquired from the student image, and the analysis results arranged in time series for each student may be acquired.
  • the face analysis unit 13 may acquire information in which each analysis result is associated with information indicating a time point when each analysis result is acquired.
  • the information indicating the time is, for example, time information or information such as a frame number.
  • the statistical analysis unit 14 determines the number of appearances of the analysis result for each analysis item for each face recognized by the face analysis unit 13 for the analysis results obtained by the face analysis unit 13 for a plurality of time points of the student image. Tally. And the statistical analysis part 14 may acquire the information which matched the total result and the information which identifies the recognized face as a statistical result. Thereby, the statistical analysis part 14 can acquire the statistical result which shows the change of the analysis result of the time series of each student's face recognized by the student image.
  • the information for identifying the face is, for example, identification information assigned to the recognized face.
  • the face analysis unit 13 may perform cluster analysis or multivariate analysis by using the analysis result value for each analysis item as a multivariate.
  • the statistical analysis unit 14 adds a value obtained by converting the detection result of the analysis item for each face into a numerical value or a value obtained by weighting and adding the analysis item for each comprehensive face for each recognized face. It may be acquired as a feature value that is an analysis result, and an average value, variance, deviation value, or the like may be acquired using this feature value.
  • the analysis result acquired by the face analysis unit 13 may be the analysis result acquired by the face analysis unit 13 accumulated in a storage medium (not shown) by the output unit 17 described later.
  • the lecture evaluation unit 15 evaluates the lecture using the statistical analysis result of the statistical analysis unit 14.
  • the lecture evaluation performed here is, for example, an evaluation related to the lecture.
  • the evaluation of the lecture performed by the lecture evaluation unit 15 is, for example, an evaluation of how to advance the lecture or the content of the lecture.
  • the evaluation of the lecture may be an evaluation for a student who takes a lecture or an evaluation for a lecturer who gives a lecture.
  • the evaluation of a lecture is an evaluation for determining whether a lecture is good or bad, an evaluation for determining whether a lecture is appropriate, or a numerical value or an index indicating the good or appropriate of a lecture.
  • the evaluation of the lecture may be an evaluation of whether the lecture is easy to understand, whether the lecture is interesting, and whether it is easy to concentrate on the content of the lecture.
  • the lecture evaluation unit 15 evaluates a lecture by using the result of statistical analysis acquired by the statistical analysis unit 14 for one entire lecture, for example. For example, it is determined whether or not the statistical analysis result for one entire lecture satisfies a predesignated condition. If the condition is satisfied, an evaluation result prepared in advance in association with the condition is acquired. In addition, also when not satisfy
  • This condition is, for example, a condition that the value of an analysis item designated in advance prepared for each analysis item exceeds a threshold value or does not exceed a threshold value.
  • this condition may be a condition such that all the conditions individually prepared for each of the plurality of classification items are satisfied, not satisfied, or only a part thereof is satisfied.
  • the statistical analysis result for one entire lecture is, for example, the number of detected faces for each analysis item aggregated in one entire lecture.
  • the lecture evaluation unit 15 determines, for example, whether or not the result of the statistical analysis acquired by the statistical analysis unit 14 satisfies a predetermined condition for each time point of the student image. May be performed. For example, it is determined whether the number of detected faces for each analysis item acquired by the statistical analysis unit 14 for each time point of the student raw image exceeds a threshold value specified in advance for each analysis item, and the number of detections of 1 or more is determined. If the threshold value is exceeded, it may be determined that the condition is satisfied, and an evaluation result associated with the condition may be acquired. Note that the face detection count information for each analysis item may be a normalized value.
  • the lecture evaluation unit 15 may acquire the evaluation result of one lecture using the evaluation result of the lecture at each time point of the raw image of the one lecture acquired as described above. For example, when all or some of the evaluation results of lectures acquired for a plurality of time points of a student image of a single lecture satisfy a predetermined condition, the lecture evaluation unit 15 associates with the condition. The obtained evaluation result may be acquired as the evaluation result of one lecture. For example, if the evaluation results of lectures acquired for a plurality of points in time include a predetermined number of evaluation results equal to or greater than a predetermined threshold value, the lecture evaluation unit 15 satisfies the condition Judgment may be made and an evaluation result of “quality lecture” associated with this condition may be acquired. Alternatively, the lecture evaluation unit 15 may acquire the evaluation results of the entire plurality of lectures using the evaluation results of the plurality of lectures. In addition, said evaluation result designated previously is an evaluation result that a student's interest is maintained, for example.
  • the lecture evaluation unit 15 may appropriately change depending on the purpose of evaluation of lectures, etc., what kind of student analysis results are used for the evaluation of lectures.
  • the result of statistical analysis used for evaluation is, for example, the result of statistical analysis acquired by the statistical analysis unit 14 from a student image taken in one lecture.
  • the result of the statistical analysis acquired by the statistical analysis unit 14 from the student images taken in one or more lectures conducted on is used for evaluation.
  • the result of statistical analysis acquired by the statistical analysis unit 14 from student images taken in one or more lectures conducted by one lecturer is evaluated. Used.
  • the lecture evaluation unit 15 preferably uses, for evaluation, the results of statistical analysis acquired by the statistical analysis unit 14 from student images taken in a plurality of lectures.
  • the statistical analysis result of the statistical analysis unit 14 used by the lecture evaluation unit 15 for the evaluation of the lecture is a statistical analysis result acquired by the statistical analysis unit 14 accumulated in a storage medium (not shown) by the output unit 17 to be lectured. Also good.
  • the designation receiving unit 16 designates the analysis result of the face at one time point output by the output unit 17 described later. Accept. For example, the designation receiving unit 16 receives one or more designations of the face analysis results output by the output unit 17 in time series.
  • the output here is display, storage in a storage medium (not shown), or the like.
  • the designation of the analysis result may be designation of analysis result identification information, an image showing the analysis result, or the like. Accepting here means accepting information input from input devices such as a keyboard, mouse, touch panel, receiving information sent via a wired or wireless communication line, recording on an optical disc, magnetic disc, semiconductor memory, etc. It is a concept including reception of information read from a medium.
  • the input means for designating may be anything such as a numeric keypad, keyboard, mouse or menu screen.
  • the designation receiving unit 16 can be realized by a device driver for input means such as a numeric keypad or a keyboard, control software for a menu screen, and the like.
  • the output unit 17 outputs information related to the analysis result by the face analysis unit 13.
  • the information related to the analysis result by the face analysis unit 13 is, for example, information on the analysis result acquired by the face analysis unit 13.
  • the information related to the analysis result by the face analysis unit 13 is information having the analysis result information and information at the time of acquisition of the frame image or the like that is the acquisition target of the analysis result in association with each other. Also good.
  • the information related to the analysis result by the face analysis unit 13 may be information acquired using the analysis result acquired by the face analysis unit 13, for example.
  • the information related to the analysis result by the face analysis unit 13 is the result of the statistical analysis acquired by the statistical analysis unit 14.
  • information related to the analysis result by the face analysis unit 13 is the evaluation result of the lecture evaluation unit 15.
  • the output unit 17 outputs the face analysis results of the face analysis unit 13 along the time series when the face analysis unit 13 acquires a plurality of analysis results of the face of the student along the time series. It may be. Outputting the analysis results in time series means, for example, outputting the analysis results arranged in time series. For example, the output unit 17 may arrange and output the analysis results in accordance with the order of the time points of the student raw images from which the analysis results are acquired. Outputting the analysis results in association with the time is easy to grasp the time-series order, and it is easy to rearrange the time-series order. May be. For example, the output unit 17 may output the analysis result and information identifying the time point of the student raw image from which the analysis result is acquired in association with each other.
  • the information for identifying the time point of the student raw image is, for example, time information associated with the frame image of the student raw image that is the acquisition target of the analysis result, identification information of the frame image, or the like.
  • the output is a display
  • the identification information of each analysis result, the link button or anchor associated with the analysis result, and the like are displayed in association with the time axis. It may be thought of as output.
  • the output unit 17 receives the student at the time corresponding to the analysis result received by the designation receiving unit 16. Output an image.
  • the output unit 17 reads out the frame image associated with the time point corresponding to the analysis result received by the designation receiving unit 16 from the student raw image storage unit 12 and outputs the frame image.
  • the output unit 17 reads out the moving images of the period before and after the time corresponding to the analysis result received by the designation receiving unit 16 from the student raw image, and the student at the time corresponding to the analysis result received the designation. It may be output as an image.
  • the output unit 17 may output the student raw image in a display mode that prevents the student from being identified.
  • the display mode in which the student cannot be identified is, for example, a student image such as a monochrome image, an image such as an icon prepared in advance, a pattern image, etc., on the face part of the student in the student image. An image irrelevant to each individual face is displayed, or the face portion of the student's face is masked to hide the face.
  • the display mode in which the student cannot be identified is, for example, performing mosaic processing or blurring processing on the image of the student's face.
  • the face portion of the student included in the student image may be detected using information indicating the face area recognized by the face analysis unit 13 when the face analysis is performed, for example.
  • the face analyzing unit 13 may detect the face in the student raw image by recognizing the face in the same manner as the process for analyzing the face. . By doing in this way, it becomes possible to protect each student's privacy by not distinguishing each student's face in a student image.
  • the information indicating the face area is, for example, the coordinates of the pixels of the face outline or the coordinates of the pixels constituting the face.
  • Output here means display on a display, projection using a projector, printing on a printer, transmission to an external device, storage on a recording medium, processing results to other processing devices or other programs, etc. It is a concept that includes delivery.
  • the output described here is a concept including temporary storage.
  • the output unit 17 may or may not include an output device such as a display.
  • the output unit 17 can be realized by output device driver software, or output device driver software and an output device. This also applies to the following output units.
  • the operation of the face analysis apparatus 1 will be described using the flowchart of FIG.
  • the student raw images taken for one lecture are stored in the student raw image storage unit 12 in advance.
  • this student raw image is a moving image constituting one file.
  • a time code is associated with each frame image of the student image.
  • Step S201 The face analysis apparatus 1 determines whether or not an instruction to analyze one student raw image stored in the student raw image storage unit 12 is received via a reception unit (not shown) or the like. If an instruction has been accepted, the process proceeds to step S202. If not, the process returns to step S201.
  • Step S202 The face analysis unit 13 assigns 1 to the counter n.
  • Step S203 The face analysis unit 13 determines whether or not a frame image associated with the n-th time to be analyzed exists in the student raw image to be analyzed in the student raw image.
  • the frame image here may not be a continuous frame image. For example, when face analysis is sequentially performed at intervals of 5 seconds from the start of shooting, it is determined whether there is a frame image associated with a time code n ⁇ 5 seconds after the start of shooting. If there is a frame image, the process proceeds to step S204; otherwise, the process proceeds to step S210.
  • the face analysis unit 13 sequentially recognizes one or more faces in the frame image associated with the nth time. For example, the face analysis unit 13 assigns face identification information to information indicating the contours of one or more recognized faces, and temporarily stores them in a storage medium (not shown).
  • Step S205 The face analysis unit 13 analyzes each face recognized in step S204. For example, for each face recognized in step S204, an analysis is performed on the analysis item designated in advance, and information indicating the analysis result is acquired.
  • face analysis for example, a frame image before or after the frame image associated with the nth time may be used.
  • Step S206 The face analysis unit 13 associates the identification information of each face, the analysis result for each face, and the time corresponding to the frame image in the analysis mode, and accumulates them in a storage medium (not shown). . As a result, the analysis results are output (in particular, accumulated here) along the time series. This accumulation may be performed by the output unit 17.
  • the face analysis unit 13 performs a face detection process on the frame image associated with the nth time, and performs an analysis as shown in step S205 for each face each time a face is detected. The process of accumulating the analysis results may be repeated until no new face can be detected.
  • the statistical analysis unit 14 performs a statistical analysis using the analysis result of the face analysis unit 13 with respect to the frame image associated with the nth time. For example, for the analysis result of each face acquired in step S206, the number of faces having the same analysis result is totaled. For example, this total value is a statistical analysis result.
  • the same analysis result means that the analysis item and the analysis value obtained for the analysis item are the same.
  • a part of the same analysis result may be preferentially aggregated. Alternatively, this one face may be counted redundantly in the aggregation of the respective analysis results.
  • the statistical analysis is performed for each frame image. However, using the analysis result of the face analysis unit 13 for the plurality of frame images, the same statistical analysis as described above is performed for each of the plurality of frame images immediately before step S210. May be performed.
  • Step S208 The statistical analysis unit 14 stores the statistical analysis result acquired in Step S207 in a storage medium (not shown) in association with the nth time. This accumulation may be performed by the output unit 17.
  • Step S209 The face analysis unit 13 increments the value of the counter n by 1. Then, the process returns to step S203.
  • Step S210 The lecture evaluation unit 15 uses the statistical analysis result accumulated in step S208 to evaluate the lecture corresponding to the one student raw image described above. For example, if there is a statistical analysis result that satisfies a previously specified condition among the statistical analysis results accumulated in step S208, information indicating the evaluation of a lecture that is associated with this condition in advance is acquired.
  • the statistical analysis result corresponding to each time is the total number of faces of students who are facing sideways, and the predesignated condition is that the number of faces of students who are facing sideways is designated in advance. It is assumed that more than a predetermined number of statistical analysis results are detected, and the evaluation of the lecture corresponding to this condition is an evaluation that “the concentration is not maintained”.
  • step S208 If the statistical analysis result accumulated in step S208 satisfies this condition, the evaluation result of the lecture “Concentration is not maintained” is obtained.
  • a plurality of different evaluations may be performed.
  • the statistical analysis unit 14 may perform a process of counting statistical analysis results in which the number of faces of students facing sideways is equal to or greater than a predetermined number.
  • the lecture evaluation unit 15 stores the evaluation result in a storage medium (not shown) in association with the identification information of the lecture, the student image, etc. This accumulation may be performed by the output unit 17.
  • Step S212 The output unit 17 displays the statistical analysis result accumulated in Step S208 and the evaluation result accumulated in Step S211.
  • Step S213 The output unit 17 determines whether to display the analysis result acquired by the face analysis unit 13 accumulated in Step S206. For example, the output unit 17 determines whether an instruction to display the analysis result is received via a reception unit (not shown). If the analysis result is displayed, the process proceeds to step S214. If not, the process proceeds to step S219.
  • Step S214 The output unit 17 displays the analysis result accumulated in Step S206.
  • Step S215 The designation receiving unit 16 determines whether or not designation of one analysis result has been received via a not-shown receiving unit or the like.
  • the one analysis result here is, for example, an analysis result associated with one time. If accepted, the process proceeds to step S216. If not accepted, the process proceeds to step S220.
  • Step S216 The output unit 17 acquires a frame image at the same time as the time corresponding to the analysis result designated in Step S215 from the student raw image.
  • Step S217) The output unit 17 displays the frame image acquired in Step S216.
  • the output unit 17 may continuously reproduce the frame images after the frame image acquired in step S216 of the student raw images. That is, the output unit 17 may reproduce a moving image from the frame image acquired in step S216.
  • the frame image acquired by the output unit 17 in step S216 is a frame image one frame before or one frame after the time corresponding to the analysis result specified in step S215.
  • it may be a frame image of the time before and after the time specified in advance with respect to the time corresponding to the analysis result specified in step S215.
  • Step S2128 The output unit 17 determines whether or not to end the display of the frame image. For example, the output unit 17 determines to end the display when an operation for ending the display is received via a reception unit (not illustrated) or the like. If it is determined that the display is to be terminated, the display is terminated and the process returns to step S215. If it is determined that the display is not to be terminated, the process returns to step S218.
  • Step S219) The output unit 17 determines whether or not to end the display of the evaluation result and the statistical analysis result. For example, the output unit 17 determines to end the display when an operation for ending the display is received via a reception unit (not illustrated) or the like. If it is determined to end the display, the display ends and the process returns to step S212. If it is determined not to end the display, the process returns to step S215.
  • Step S220 The output unit 17 determines whether or not to end the display of the analysis result acquired by the face analysis unit 13. For example, the output unit 17 determines to end the display when an operation for ending the display is received via a reception unit (not illustrated) or the like. If it is determined to end the display, the display ends and the process returns to step S201. If it is determined not to end the display, the process returns to step S213.
  • the process is terminated by turning off the power or interrupting the termination of the process.
  • FIG. 3 is a diagram schematically showing a part of the frame image constituting the student raw image stored in the student raw image storage unit 12. It is assumed that this student image is a moving image obtained by photographing the entire student participating in one lecture using the first photographing unit 11 from the lecturer side. This student raw image is composed of one file.
  • a time code associated with each frame image is shown below each frame image. The time code represents the time at which each frame image was taken as “hour: minute: second.frame”.
  • a user operates an input device (not shown) such as a mouse or a keyboard to give an instruction to analyze the student raw image shown in FIG.
  • an input device such as a mouse or a keyboard to give an instruction to analyze the student raw image shown in FIG.
  • face analysis processing is set in advance for frame images at intervals of 1 second.
  • the face analysis unit 13 reads out a frame image associated with the time code “10: 30: 30: 01”, which is the first frame image of the student raw image. Then, processing for recognizing a face is performed in the read frame image.
  • the first frame image is usually the frame image at the start of shooting.
  • FIG. 4 is a schematic diagram showing a frame image in a state where the face is recognized by the face analysis unit 13.
  • regions 401 to 430 surrounded by dotted lines are recognized facial regions.
  • the facial expression and orientation, line-of-sight direction, and blinking situation shown in this figure are schematic and do not necessarily accurately represent the facial expression and direction, line-of-sight direction, and blinking situation, etc. Shall. The same applies to the following.
  • the face analysis unit 13 associates information indicating the contours of the recognized face regions 401 to 430 with face IDs, which are identification information for identifying faces, and accumulates them in a storage medium (not shown).
  • the information indicating the contour is, for example, a coordinate group of pixels constituting the contour in the frame image.
  • the areas 401 to 430 are associated with “ID 401” to “ID 430” as face IDs, respectively.
  • the face analysis unit 13 sequentially analyzes each face recognized above. First, an analysis is performed on the face with the face ID “ID 401” (that is, the face in the region 401).
  • ID 401 that is, the face in the region 401.
  • an analysis item whether or not the facial expression is a smiling face an analysis item whether or not the gaze direction is facing the front (the lecturer side), and the face direction is the front (the lecturer side)
  • these analysis processes are well-known techniques, specific description is abbreviate
  • one or more frame images before or after the frame image to be analyzed are read from the student raw image, and the image in the area 401 of the read frame image or its frame image is read. Peripheral images may be used.
  • the face analysis unit 13 stores the analysis result for the face with the face ID “ID401” in a storage medium (not shown) in association with the time code “10: 30: 10: 01” and the face ID “ID401”. .
  • the face analysis unit 13 performs the same processing for the faces whose face IDs are “ID402” to “ID430”.
  • FIG. 5 is a face analysis result management table for managing information indicating the analysis result of the face analysis acquired by the face analysis unit 13 and accumulated in a storage medium (not shown).
  • the face analysis result management table includes items of “time”, “face ID”, “expression”, “gaze direction”, “face direction”, and “blink”.
  • “Time” is the time of the frame image to be analyzed of the student image, and here is the time code of the frame image.
  • “Face ID” is the face ID described above.
  • “Expression” is the analysis result of the facial expression of the student.
  • the value “Smile” indicates that the face is laughing, and the blank indicates that the face is not laughing.
  • “Gaze direction” is the analysis result of the gaze direction of the student's face, where the value “front” indicates that the gaze direction is facing the front and the blank indicates that it is not facing the front .
  • “Face direction” is the analysis result of the student's face direction, and the value “front” indicates that the face is facing the front, and the blank indicates that the face is not facing.
  • “Blink” is the analysis result of whether or not the student has closed eyes, the value “Closed” indicates that the eyes are closed, and the blank indicates that the eyes are open .
  • the statistical analysis unit 14 performs statistical analysis using the face analysis result by the face analysis unit 13 with respect to the frame image associated with the time code “10: 30: 30: 01” as illustrated in FIG. .
  • the statistical analysis unit 14 has the number of faces whose “expression” is “smile” and the “gaze direction” is “front” for the frame image whose time code is “10: 30: 00: 00”.
  • the number of faces, the number of faces whose “face direction” is “front”, and the number of faces whose “blink” is “closed” are respectively tabulated. That is, the statistical analysis unit 14 performs aggregation for each analysis item. Then, these total results are divided by the total number of recognized faces “30” to obtain the ratio of appearance of faces for each analysis item.
  • the statistical analysis unit 14 may calculate variance or the like instead of calculating the ratio. Then, the statistical analysis unit 14 stores the acquired ratio for each analysis item as a statistical analysis result in a storage medium (not shown) in association with the time code “10: 30: 30: 01”.
  • the face analysis unit 13 further reads out a frame image associated with the time code “10: 30: 01: 01” corresponding to the time after 1 second of the frame image read out above, and performs the same as described above. Then, the process of recognizing the face is performed, the same analysis as described above is performed on the recognized face, and the analysis result is added to a storage medium (not shown). As a result, an analysis result record in which “time” is associated with “10: 30: 01: 01” is added to the face analysis result management table shown in FIG.
  • the same face ID as that of the face recognized immediately before is assigned to the face in the recognized face that is substantially in the same position as the face recognized immediately before.
  • a face in substantially the same position is, for example, a face in which the coordinates of pixels included in the region match a predetermined number or more (for example, half or more).
  • the statistical analysis unit 14 performs statistical analysis for each analysis item in the same manner as described above, and accumulates the statistical analysis results.
  • the face analysis unit 13 and the statistical analysis unit 14 sequentially repeat the above processing until there is no frame image after one second in the student raw images.
  • FIG. 6 is a diagram showing a statistical analysis result management table for managing the statistical analysis results acquired and accumulated by the statistical analysis unit 14.
  • the statistical analysis result management table has items of “time”, “expression (smile)”, “line of sight (front), face direction (front)”, and “blink (closed)”.
  • “Time” is the time of the frame image to be analyzed, and here is the time code of the frame image.
  • “Expression (smile)” is the ratio (%) of faces whose analysis item “expression” has a value of “smile” in the face analysis results.
  • the “line of sight (front)” is the ratio (%) of faces whose analysis item “line of sight” is “front” in the face analysis results.
  • “Face direction (front)” is the ratio (%) of faces whose analysis item “face direction” is “front” in the face analysis results.
  • “Blink (closed)” is the ratio (%) of faces whose analysis item “blink” is “closed” in the face analysis results.
  • the lecture evaluation unit 15 evaluates the lecture corresponding to the student raw image using the statistical analysis result acquired by the statistical analysis unit 14.
  • the time when the ratio of ““ line of sight (front) ”is“ 70% or more ”is“ 80% ”of the time when the face analysis is performed is stored in advance in a storage medium (not shown).
  • the value “lecture with high concentration” is stored in a storage medium (not shown) in association with this condition as a value indicating the evaluation of the lecture when this condition is satisfied.
  • the time here may be considered as each frame image to be analyzed, or may be considered as each record of the statistical analysis result management table shown in FIG.
  • the lecture evaluation unit 15 first reads out the above conditions. Next, in the statistical analysis result management table shown in FIG. 6, the number of records whose “line of sight (front)” ratio is “70% or more” is counted. For example, assume that the count number is “1752”. Also, the lecture evaluation unit 15 acquires the total number of records in the statistical analysis result management table shown in FIG. For example, it is assumed that the total number of records is “2386”. The lecture evaluation unit 15 divides the count number “1752” of the record whose “line of sight (front)” is “70% or more” by the total number of records “2382” to obtain the ratio of “line of sight (front)”. “73.6 (%)” is acquired.
  • the acquired ratio is “70% or more”.
  • the evaluation result “lecture with high concentration” is acquired.
  • an evaluation result “lecture with a low degree of concentration” may be acquired.
  • the acquired evaluation result is stored in a storage medium (not shown) in association with the file name of the student raw image, for example.
  • the output unit 17 displays the statistical analysis result acquired by the statistical analysis unit 14 and the evaluation result acquired by the lecture evaluation unit 15 on a monitor or the like.
  • FIG. 7 is a diagram showing a display example of a statistical analysis result and a lecture evaluation result.
  • a scroll bar or the like may be shown so that a portion that cannot be entered by operating the scroll bar may be displayed as appropriate. The same applies to the following.
  • the output unit 17 is managed by the face analysis result management table shown in FIG.
  • the face analysis result along the time series acquired by the face analysis unit 13 is displayed on a monitor or the like.
  • the output unit 17 displays the analysis results side by side along a time series.
  • FIG. 8 is a diagram showing a display example of the face analysis result.
  • the character string of “time” of each analysis result includes, for example, a determination area for determining whether or not designation for each analysis result is accepted. It shall be provided. Note that the determination area is not shown.
  • the output unit 17 reads the student raw image at the time corresponding to the analysis result from the student raw image storage unit 12.
  • the frame image associated with the same time as the time “10: 30: 02.01” of the analysis result that received the designation is read from the student image.
  • the output unit 17 displays the read frame image on the monitor. Note that when outputting the frame image, the output unit 17 may output a frame image in a display mode in which the student's face portion of the frame image cannot be identified by the student.
  • the output unit 17 recognizes the contours of the face regions 401 to 430 that the face analysis unit 13 recognizes in the face recognition process as shown in FIG. Is read, and the face portion in the frame image whose display mode is to be changed is detected.
  • the face part newly detected with respect to the frame image output by the output unit 17 by the face analysis unit 13 may be the display mode change target by the output unit 17.
  • FIG. 9 is a diagram showing a display example of a frame image corresponding to the analysis result of the face whose designation is accepted by the designation receiving unit 16.
  • the face of one or more students in the lecture is automatically recognized and analyzed, the analysis result is statistically analyzed, and the evaluation of the lecture is performed from the result of the statistical analysis. And conduct appropriate analysis on lectures.
  • the statistical analysis unit 14 performs the statistical analysis of the face analysis results for each frame image.
  • the facial expression, the line-of-sight direction, Information indicating changes in analysis results such as face direction and blinking may be acquired as information indicating the results of statistical analysis.
  • a result obtained by performing Fourier transform or the like may be acquired as a result of statistical analysis.
  • the predesignated time is, for example, a time that is equal to or greater than the maximum time during which the eyelid is considered to be closed during normal blinking. Note that the above-specified time is, for example, one minute.
  • the face analysis apparatus 2 according to the second embodiment is the same as the face analysis apparatus 1 according to the first embodiment described above. It is used for statistical analysis of results and evaluation of lectures.
  • FIG. 10 is a block diagram of face analysis device 2 in the present embodiment.
  • the face analysis device 2 includes a first photographing unit 11, a student raw image storage unit 12, a second photographing unit 21, a voice acquisition unit 22, a lecturer information storage unit 23, a change detection unit 24, a face analysis unit 25, and a related statistical analysis unit. 26, a related lecture evaluation unit 27, and an output unit 28.
  • the second photographing unit 21 photographs the lecturer during the lecture and acquires a moving image of the lecturer during the lecture.
  • a lecturer is a person who gives a lecture. There is usually one instructor, but there may be multiple instructors.
  • Shooting the instructor means, for example, obtaining a moving image that shows the instructor's movement.
  • the second photographing unit 21 acquires a moving image obtained by photographing the entire lecturer or the upper body.
  • the second image capturing unit 21 and the moving image acquired by the second image capturing unit 21 have the same configuration as the first image capturing unit 11 described above except that the object to be imaged is different. Is omitted.
  • the second photographing unit 21 accumulates the acquired moving image in the instructor information storage unit 23 as instructor information described later or as a part thereof.
  • the voice acquisition unit 22 acquires voice information of the lecturer during the lecture.
  • the voice acquisition unit 22 acquires information in which a voice is associated with a timing at which the voice is acquired.
  • the timing at which the sound is acquired is, for example, time.
  • the voice acquisition unit 22 accumulates the acquired voice information in the lecturer information storage unit 23 as a part of lecturer information described later.
  • the file format, compression method, etc. of lecturer audio information are not limited.
  • either the audio acquisition unit 22 or the second imaging unit 21 has instructor information associated with the audio acquired by the audio acquisition unit 22 and the moving image acquired by the second imaging unit 21 so as to be synchronized. You may make it accumulate
  • the voice acquisition unit 22 is realized by a microphone, for example.
  • the second imaging unit 21 and the audio acquisition unit 22 may be an imaging unit (camera unit) and an audio acquisition unit (microphone) of one video camera.
  • the lecturer information storage unit 23 stores lecturer information.
  • Instructor information is information including at least one of instructor image or instructor audio information.
  • the instructor image is a moving image of the instructor being lectured. For example, it is a moving image acquired by the second imaging unit 21.
  • the lecturer audio image is information on the audio of the lecturer who is giving a lecture.
  • the lecturer image is the same information as the student image, except that the shooting target is different.
  • the lecturer sound image is, for example, information associated with a time series.
  • the lecturer image may be associated with a time code or the like, and the lecturer voice information may be stored in the lecturer information storage unit 23 in synchronization with the lecturer image.
  • the storage is a concept including temporary storage.
  • the change detection unit 24 detects changes in instructor information. For example, the change detection unit 24 detects a predetermined change designated in advance from the lecturer information. The change detection unit 24 may acquire information specifying the time point when the change is detected as a detection result. The information specifying the time point when the change is detected is, for example, the time when the change is detected.
  • the detection of the conversion of the lecturer information is either the detection of the change in the lecturer image included in the lecturer information, the detection of the change in the lecturer audio information included in the lecturer information, or both. For example, the change detection unit 24 detects a change that satisfies a predetermined condition.
  • the predesignated condition is, for example, the presence or absence of a motion or a motion greater than the size of a predesignated motion.
  • the predesignated condition is, for example, voice output of a level specified in advance or higher.
  • the change detection unit 24 may acquire information indicating the content (for example, type) of the detected change or information indicating the amount of change as a detection result.
  • the change detection unit 24 performs motion vector detection by comparing the pixels of the frame image with respect to the lecturer image included in the lecturer information, and the presence or absence of motion or a motion vector greater than or equal to a predetermined threshold value. Detection is performed. When a motion that is equal to or greater than a pre-specified threshold is detected, it is determined that a motion has been detected.
  • the movement above the threshold value specified in advance is, for example, a change in distance above the threshold value.
  • the instructor image included in the instructor information like the face analysis unit 25, detects the instructor's face, detects the instructor's hand, etc., and performs motion vector detection of the detected portion, thereby moving The detection of the presence or absence of movement and the movement more than the threshold value designated beforehand is performed.
  • the change detection unit 24 detects a change in the lecturer information based on, for example, a change in voice level in the lecturer voice information included in the lecturer information. For example, the change detection unit 24 determines that a change in the lecturer information is detected when an output having a level equal to or higher than a predetermined threshold value is detected in the lecturer voice information. Moreover, you may detect the change of lecturer information, when the value of SN ratio of lecturer audio
  • the face analysis unit 25 recognizes the student's face from the student image and analyzes the recognized face.
  • the face analysis unit 23 acquires the analysis result of the face for the student raw image acquired immediately after the change detection unit 24 detects the change.
  • the term “immediately after here” has the same meaning as just described in the first embodiment.
  • the face recognition processing and analysis processing performed by the face analysis unit 25 on the student raw images are the same as those of the face analysis device 13 of the first embodiment.
  • the face analysis unit 25 performs one or more classes on a student raw image acquired immediately after a student raw image (specifically, a frame image) is detected immediately after a change detection unit 24 described later detects a change in lecturer information.
  • the point of acquiring the raw face analysis result is different from the face analysis unit 13 of the first embodiment.
  • acquiring the student's face analysis result for the student image acquired immediately after the change is detected means that the face analysis unit 25 has detected a change in the lecturer information by the change detection unit 24 described later. Analysis of the face may be performed on the student raw image acquired immediately after the time and the analysis result may be acquired, or the face analysis unit 25 may acquire the face acquired at a plurality of times different from the student raw image. From the analysis result, the analysis result acquired immediately after the change is detected in the lecturer information may be acquired, or a process corresponding to this may be performed.
  • the term “immediately after the change is detected” refers to, for example, a point in time that is a predetermined time from the point in time when the change is detected. Immediately after the change is detected, it may be considered that the time is a predetermined time or the number of frames. This elapsed time is for considering the time lag between the change of the teacher's movement and the change of the voice and the change of the student's facial expression corresponding to the change. This elapsed time is, for example, a time of 5 seconds or less. This elapsed time may be acquired based on, for example, experimental results in advance. However, this elapsed time may be the time when the change is detected, or may be considered as the next time after the change is detected.
  • the next time may be the time of the next frame image, for example.
  • the time immediately after the change is detected may be considered as the time around the time when the change is detected.
  • the time here may be considered as the number of frames.
  • immediately after the time when the change is detected it may be considered as a period of a predesignated length immediately after the time when the change is detected.
  • the face analysis unit 25 acquires the analysis result of the face performed on the frame image at the time when a predetermined number of seconds have elapsed from the time when the change detection unit 24 detects the change in the lecturer information.
  • frame images before and after that may be used as appropriate.
  • the related statistical analysis unit 26 performs statistical analysis specified in advance using the analysis result of the face analysis unit 25 with respect to the student raw image acquired immediately after the change detection unit 24 detects the change.
  • the statistical analysis designated in advance may be any statistical analysis.
  • the statistical analysis designated in advance by the related statistical analysis unit 26 is the same statistical analysis as the statistical analysis performed by the statistical analysis unit 14, and the analysis result used for the statistical analysis is changed by the change detection unit 24.
  • This is an analysis result acquired by the face analysis unit 25 with respect to the student raw image acquired immediately after the detection.
  • the statistical analysis designated in advance may be statistical analysis in which information indicating the change target detected by the change detection unit 24, information indicating the content of the change, and the analysis result of the face analysis unit 25 are associated with each other. Good.
  • the change target is a target for which the change detection unit 24 has detected a change, such as a lecturer image or lecturer audio information.
  • the information indicating the content of the change is information indicating the content of the change detected by the change detecting unit 24.
  • voice more than the level designated beforehand may be sufficient.
  • the result of the analysis of the face analysis unit 25 with respect to the frame image of the student raw image acquired immediately after each of the plurality of time points when the change is detected is further tabulated according to the analysis item as in the statistical analysis unit 14 described above.
  • the data may be aggregated according to the change target detected by the change detection unit 24 and the content of the change.
  • the related lecture evaluation unit 27 uses the statistical analysis result of the related statistical analysis unit 26 to evaluate the lecture.
  • the lecture evaluation performed here is, for example, an evaluation related to the lecture. Similar to the evaluation performed by the lecture evaluation unit 15, the evaluation performed by the related lecture evaluation unit 27 is, for example, an evaluation of how to proceed with the lecture and the content of the lecture.
  • the related lecture evaluation unit 27 uses the result of the statistical analysis performed by the related statistical analysis unit 26 included in the result of the statistical analysis acquired by the related statistical analysis unit 26 (hereinafter referred to as the related statistical analysis result). An evaluation result may be acquired. Further, the related lecture evaluation unit 27 evaluates the lecture based on, for example, a combination of the content of the change detected by the change detection unit 24 included in the related statistical analysis result and the result of the statistical analysis performed by the related statistical analysis unit 26. Results may be obtained.
  • the related lecture evaluation unit 27 determines whether or not the related statistical analysis result acquired by the related statistical analysis unit 26 from the student image obtained by photographing one lecture satisfies a predetermined condition, for example. If the condition is satisfied, the evaluation result of the lecture prepared in advance associated with this condition is acquired. Even when the condition is not satisfied, the related lecture evaluation unit 27 may acquire an evaluation result prepared in advance for the case where the condition is not satisfied.
  • This condition is, for example, a condition that the value of the analysis item designated in advance exceeds or does not exceed the threshold value prepared for each analysis item. Alternatively, this condition may be a condition such that all the conditions individually prepared for a plurality of classification items are satisfied, or not satisfied, or only a part thereof is satisfied.
  • the related lecture evaluation unit 27 has a condition in which a combination of the content of the change detected by the change detection unit 24 included in the related statistical analysis result and the result of the statistical analysis performed by the related statistical analysis unit 26 is specified in advance. If the condition is satisfied, the evaluation result of the lecture prepared in advance associated with the condition is acquired. For example, the number of laughing faces detected from each student image immediately after one or more time points when the level of the lecturer audio information included in the lecturer information is equal to or higher than a predetermined threshold is calculated. When the number is equal to or greater than a predetermined threshold value, the related lecture evaluation unit 27 may acquire an evaluation result of “lecture with sustained interest” associated with this condition in advance. In this case, for example, since it is considered that there are many students laughing at the words spoken by the lecturer, it is considered that the students are listening to the lecture with interest.
  • the related lecture evaluation unit 27 evaluates the lecture using the result of statistical analysis acquired by the related statistical analysis unit 26 for each student raw image at each time point when the change detection unit 24 detects the change. Also good.
  • the related lecture evaluation unit 27 specifies in advance the number of face detections for each analysis item acquired by the related statistical analysis unit 26 at each time point when a change designated in advance in the lecturer information is detected. It is determined whether or not the set threshold value is exceeded. And when it exceeds the threshold value designated beforehand, the related lecture evaluation part 27 may acquire the evaluation result matched with this condition. Needless to say, the number of detected faces for each analysis item may be a normalized value.
  • the related lecture evaluation unit 27 may acquire the evaluation result of one lecture by using the evaluation result of the lecture at each time point of the raw image of the one lecture acquired as described above. For example, when all or some of the evaluation results of lectures acquired for a plurality of time points of a student image of one lecture satisfy a predetermined condition, the evaluation results associated with this condition are It may be acquired as an evaluation result of one lecture. For example, if the evaluation results of lectures acquired for each of a plurality of time points include a predetermined number of evaluation results equal to or greater than a predetermined threshold, it is determined that the condition is satisfied. An evaluation result of “good quality lecture” associated with “” may be acquired. Alternatively, evaluation results of a plurality of lectures as a whole may be acquired using evaluation results of a plurality of lectures. The evaluation result designated in advance is, for example, an evaluation result that the student's interest is maintained.
  • the lecture evaluation unit 15 it may be changed as appropriate according to the purpose of the lecture evaluation, etc., as with the lecture evaluation unit 15, what kind of student analysis statistical analysis results are used by the related lecture evaluation unit 27. .
  • the statistical analysis result of the related statistical analysis unit 26 used by the related lecture evaluation unit 27 for the evaluation of the lecture is acquired by the related statistical analysis unit 26, and the output unit 28 described later accumulates in a storage medium (not shown). It may be the result of analysis.
  • the output unit 28 outputs information related to the analysis result by the face analysis unit 25.
  • the output unit 28 outputs information related to the analysis result of the face analysis unit 25 for the student raw image acquired immediately after the change detection unit 24 detects the change.
  • the information related to the analysis result here is, for example, information on the analysis result acquired by the face analysis unit 25 with respect to the student raw image immediately after the change detection unit 24 detects the change.
  • the information related to the analysis result includes the analysis result information acquired by the face analysis unit 25 for the student raw image immediately after the change detection unit 24 detects the change, and the change corresponding to the analysis result information.
  • It may be information having at least one of information indicating the content of the change in the instructor information detected by the detection unit 24 or information at the time of acquisition such as a frame image that is the acquisition target of the analysis result.
  • the information related to the analysis result by the face analysis unit 25 here is information acquired using the analysis result acquired by the face analysis unit 25 immediately after the change detection unit 24 detects the change. good.
  • the output unit 28 may output the result of statistical analysis acquired by the related statistical analysis unit 26 as information related to the analysis result here.
  • the output unit 28 may output the evaluation result acquired by the related lecture evaluation unit 27 as information related to the analysis result by the face analysis unit 25.
  • the output unit 28 may further output information indicating the content of the change in the instructor information detected by the change detection unit 24 as information related to the analysis result.
  • the output unit 28 of the face analysis unit 25 You may make it output the analysis result of a face along a time series.
  • the student image taken in advance for one lecture and the lecturer information including the lecturer image and the lecturer audio information are stored in advance in the student image storage unit 12 and the lecturer information storage unit 23, respectively. It shall be. It is assumed that a time code is associated with each frame image of the student raw image and the lecturer image. In the lecturer information, it is assumed that the lecturer image and the lecturer audio image are synchronized.
  • Step S1101 The face analysis apparatus 2 determines whether or not an instruction to analyze one student raw image stored in the student raw image storage unit 12 is received via a reception unit (not shown) or the like. If an instruction is accepted, the process proceeds to step S1102, and if not accepted, the process returns to step S1101.
  • the change detection unit 17 performs a process of detecting a change location from the lecturer image. For example, the change detection unit 17 detects changes in order from the top frame image of the lecturer image.
  • Step S1103 The change detection unit 17 determines whether or not one change point has been detected. If detected, the process proceeds to step S1104. If not detected, the process proceeds to step S1105.
  • Step S1104 The change detection unit 17 associates the content of the detected change with the time corresponding to the frame image in which the change is detected, and accumulates it in a storage medium (not shown). Then, the process returns to step S1102.
  • Step S1105 The change detection unit 17 determines whether or not there is a remaining frame image that has not been subjected to change detection processing in the lecturer image. If there is, the process returns to step S1102, and if not, the process proceeds to step S1106.
  • the change detection unit 17 detects the location of change from the lecturer voice information. For example, the change detection unit 17 performs a process of detecting, from the beginning of the lecturer voice information, an output portion having a level higher than a predetermined level as a conversion portion.
  • Step S1107 The change detection unit 17 determines whether or not a change point is detected in the lecturer voice information. If a change location is detected, the process proceeds to step S1108. If not detected, the process proceeds to step S1109.
  • the change detection unit 17 associates information indicating the content of the change with the time when the change is detected, and stores the information in a storage medium (not shown).
  • the information indicating the content of the change is, for example, information indicating that a sound having a level exceeding the threshold is detected, or information indicating the degree of change in the sound.
  • Step S1109 The change detection unit 17 determines whether there is any remaining lecturer audio information for which no change is detected. If there is a remainder, the process returns to step S1106, and if there is no remainder, the process proceeds to step S1110.
  • Step S1110 The face analysis unit 25 substitutes 1 for the counter m.
  • Step S1111 The face analysis unit 25 determines whether or not there is an mth time among the times when the changes accumulated in Step S1104 and Step S1108 are detected. If there is an mth time, the process proceeds to step S1112; otherwise, the process proceeds to step S1118.
  • the face analysis unit 25 sequentially recognizes one or more faces in the frame image of the student raw image corresponding to the mth time. For example, the face analysis unit 25 assigns face identification information to information indicating the outline of one or more recognized faces, and temporarily stores the information in a storage medium (not shown).
  • Step S1113 The face analysis unit 25 analyzes each face recognized in step S1112. For example, for each face recognized in step S1112, an analysis is performed on an analysis item designated in advance, and information indicating the analysis result is acquired.
  • face analysis for example, a frame image before or after the frame image associated with the mth time may be used.
  • Step S1114 The face analysis unit 25 associates the identification information of each face, the analysis result for each face, and the time corresponding to the frame image in the analysis mode, and accumulates them in a storage medium (not shown). . As a result, the analysis results are output (in particular, accumulated here) along the time series. This accumulation may be performed by the output unit 28. Note that the face analysis unit 25 performs a face detection process on the frame image associated with the nth time, and performs an analysis as shown in step S1113 for each face each time a face is detected. The process of accumulating the analysis results may be repeated until no new face can be detected.
  • the related statistical analysis unit 26 performs statistical analysis using the analysis result of the face analysis unit 25 with respect to the frame image associated with the mth time. For example, for the analysis results of each face acquired in step S1113, the number of faces having the same analysis result is totaled. For example, information in which the total value is associated with information indicating the content of the change accumulated in association with the time corresponding to the frame image is an example of a statistical analysis result here.
  • the same analysis result means that the analysis item and the analysis value obtained for the analysis item are the same.
  • a part of the same analysis result may be preferentially aggregated.
  • the statistical analysis unit 20 stores the related statistical analysis result, which is the result of the statistical analysis acquired in Step S1107, in a storage medium or the like not shown in association with the mth time. This accumulation may be performed by the output unit 28.
  • Step S1117 The face analysis unit 25 increments the value of the counter m by 1. Then, the process returns to step S1111.
  • Step S1118 The related lecture evaluation unit 27 evaluates the lecture using the related statistical analysis result accumulated in step S1116. For example, if there is a statistical analysis result that satisfies a predesignated condition among the related statistical analysis results accumulated in step S1116, information indicating the evaluation of the lecture that is associated in advance with this condition is acquired.
  • each related statistical analysis result obtained corresponding to the time when a change was detected in instructor information correlates the total number of faces of laughing students with the content of changes detected in instructor information
  • the change detected for the instructor information is the predesignated condition
  • the instructor audio information is at or above the predesignated level
  • the number of faces of laughing students is predesignated
  • It is a condition that statistical analysis results that are greater than or equal to a number are detected in advance, and the evaluation of a lecture corresponding to this condition is an evaluation of “fun lecture”.
  • the related lecture evaluation unit 27 may obtain information indicating a plurality of different evaluations.
  • Step S1119) The lecture evaluation unit 21 stores the evaluation result acquired in Step S1118 in a storage medium (not shown) in association with the identification information of the lecture, the student image, etc. This accumulation may be performed by the output unit 28.
  • Step S1120 The output unit 28 displays the statistical analysis result accumulated in Step S1116 and the evaluation result accumulated in Step S1119.
  • Step S1119 The output unit 28 determines whether or not to end the display of the evaluation result and the statistical analysis result. For example, the output unit 28 determines to end the display when an operation for ending the display is received via an unillustrated receiving unit or the like. If it is determined to end the display, the display ends and the process returns to step S1101. If it is determined not to end the display, the process returns to step S1121.
  • the student raw image storage unit 12 stores student raw images similar to those described with reference to FIG. 3 in the specific example of the first embodiment.
  • FIG. 12 is a diagram showing a part of the lecturer image 121 and the lecturer audio information 122 which are the lecturer information stored in the lecturer information storage unit 23.
  • the lecturer image 121 and the lecturer voice information 122 are the lecturer's moving image and voice information respectively acquired by the second imaging unit 21 and the voice acquisition unit 22 when the student raw image shown in FIG. 3 is shot.
  • a time code is shown below each frame image of the lecturer image 121.
  • the lecturer image 121 and the lecturer audio information 122 are assumed to be synchronized.
  • an input device such as a mouse or a keyboard to give an instruction to analyze the student raw image shown in FIG.
  • the change detection unit 24 reads out the lecturer image and the lecturer audio information stored in the lecturer information storage unit 23 from the head, and detects a change for each.
  • the frame images are sequentially read out, and motion detection is performed using the difference between the previous and subsequent frame images.
  • the change detection unit 24 determines whether or not an object movement (for example, a distance or the like) that is equal to or greater than a movement designated in advance in the lecturer image is detected.
  • the change detection unit 24 determines that a change has been detected.
  • the lecturer voice information it is judged in order from the beginning of the lecturer voice information whether or not there is voice information of a level specified in advance, and in some cases, it is judged that a change has been detected.
  • the face analysis unit 25 associates the change with the frame image or updated voice information of the part of the lecturer image where the change is detected.
  • Read time code from instructor information.
  • a frame image associated with a time obtained by adding a predetermined time to the time indicated by the read time code is read from the student raw image shown in FIG.
  • face recognition and analysis for each recognized face are performed.
  • the pre-designated time is preferably a time suitable for determining whether, for example, the student's face change or the like is due to the instructor's action or the voice uttered by the instructor. Then, for example, it is preset to about 1 to 5 seconds.
  • the reason for adding the time is that there is a time lag from when a change occurs in the teacher's action or voice until a change or reaction occurs in the student's face or the like.
  • the time obtained by adding the predesignated time is “10: 35: 38.21”.
  • the information indicating the change target detected by the change detection unit 24, the analysis result of each face of the face analysis unit 25, and the time code “10:35:38. 21 ” is stored in association with a storage medium (not shown).
  • the information indicating the change target is, for example, information indicating whether the change detected by the change detection unit 24 is a change for the lecturer image or a change for the lecturer audio information.
  • FIG. 13 is a change time analysis result management table for managing the face analysis results accumulated by the change detection unit 24.
  • This change-time analysis result management table is referred to as “change target” which is information indicating the content of the change detected by the change detection unit 24 in the face analysis result management table shown in FIG. 5 in the specific example of the first embodiment. An item is added.
  • “change target” indicates that the change detected by the change detection unit 24 is a change in the instructor image
  • “instructor voice information” indicates the change detected by the change detection unit 24. Indicates a change in the lecturer image.
  • the related statistical analysis unit 26 statistically analyzes the analysis result of the face analysis unit 25.
  • statistical analysis is performed on the analysis result associated with the frame image whose time code is “10: 35: 38.21” and the “instructor image” which is the value of “change target”. I do.
  • the analysis result of the frame image with the time code “10: 35: 38.21” the number of faces whose “expression” is “smile” and “line-of-sight direction” is “front”
  • the number of faces, the number of faces whose “face direction” is “front”, and the number of faces whose “blink” is “closed” are respectively tabulated. That is, the related statistical analysis unit 26 performs aggregation for each analysis item.
  • the related statistical analysis unit 26 may calculate variance or the like instead of calculating the ratio. And the related statistical analysis part 26 respond
  • it is stored in a storage medium (not shown).
  • the analysis result to be subjected to statistical analysis is associated with “change object” “instructor image”
  • the statistical analysis result is associated with “change object” “instructor voice information”. Accumulated.
  • the change detection unit 24 repeats the process of detecting changes for the remaining lecturer images and lecturer audio information, and the face analysis unit 25 performs the change detection for each time a change is detected.
  • the face analysis is performed on the frame image of the student image corresponding to the time obtained by adding the time specified in advance, and the related statistical analysis unit 26 performs the statistical analysis.
  • FIG. 14 is a diagram showing a statistical analysis result management table for managing the results of statistical analysis accumulated by the related statistical analysis unit 26.
  • the related statistical analysis result management table is obtained by adding the item “change target” as shown in FIG. 13 to the statistical analysis result management table shown in FIG.
  • the related lecture evaluation unit 27 evaluates the lecture corresponding to the student image using the statistical analysis result acquired by the related statistical analysis unit 26 as shown in FIG.
  • a condition for evaluating a lecture “the number of times when“ change target ”is“ lecturer voice information ”and the ratio of“ expression (smile) ”is“ 60% or more ”
  • the condition “is over 40” is stored in advance in a storage medium (not shown), and the value indicating the evaluation of the lecture when this condition is satisfied is called “lecture concentrated on the lecturer's story”. It is assumed that the value is stored in a storage medium (not shown) in association with this condition.
  • the time may be considered as each frame image to be analyzed, or may be considered as each record in the related statistical analysis result management table shown in FIG.
  • the related lecture evaluation unit 27 reads the above conditions. Next, in the related statistical analysis result management table shown in FIG. 14, records whose “change target” is “lecturer voice information” and whose “expression (smile)” ratio is “60% or more” are recorded. Count the number. For example, assume that the count number is “52”. Then, it is determined whether or not the acquired count number is “40 or more”. Here, since it is 40 or more, the related lecture evaluation unit 27 acquires the evaluation result “lecture concentrated on the lecturer's story”. When the number is less than 40, the related lecture evaluation unit 27 may acquire an evaluation result “lecture not concentrated on the lecturer's story”. The acquired evaluation result is stored in a storage medium (not shown) in association with the file name of the student raw image, for example.
  • the output unit 28 displays the statistical analysis result acquired by the related statistical analysis unit 26 and the evaluation result acquired by the related lecture evaluation unit 27 on a monitor or the like as in FIG.
  • the output unit 28 receives an instruction to display the analysis result of the face acquired by the face analysis unit 25, the output unit 28 acquired by the face analysis unit 25 managed by the change analysis result management table shown in FIG. As shown in FIG. 8, the face analysis result is displayed on a monitor or the like in time series. For example, the output unit 17 displays the analysis results side by side along a time series.
  • a specification for the analysis result of the face at one point output by the output unit 28 is received, and when the specification is received, a frame image corresponding to the specified analysis result is displayed.
  • the output unit 28 may read out from the student image and display it.
  • each process may be realized by centralized processing by a single device (system), or by distributed processing by a plurality of devices. May be.
  • the face analysis device may be a stand-alone device or a server device in a server / client system.
  • the output unit or the reception unit receives input or outputs a screen via a communication line.
  • the constituent elements such as the face analysis unit 13, the statistical analysis unit 14, the lecture evaluation unit 15, the change detection unit 24, the face analysis unit 25, the related statistical analysis unit 26, the related lecture evaluation unit 27, etc.
  • each component can be realized by a program execution unit such as a CPU reading and executing a software program recorded on a recording medium such as a hard disk or a semiconductor memory.
  • the software that realizes the face analysis apparatus in each of the above embodiments is a program as described below.
  • this program uses a computer that can access a student image storage unit in which a student image, which is a moving image obtained by capturing one or more students' faces during a lecture, from a student image to a student's face.
  • the functions realized by the program do not include functions that can only be realized by hardware.
  • functions that can be realized only by hardware such as a modem and an interface card in an acquisition unit that acquires information, an output unit that outputs information, and the like are not included in the functions realized by the program.
  • the computer that executes this program may be singular or plural. That is, centralized processing may be performed, or distributed processing may be performed.
  • FIG. 15 is a schematic diagram showing an example of the appearance of a computer that executes the program and realizes the face analysis apparatus according to the embodiment.
  • the above-described embodiment can be realized by computer hardware and a computer program executed on the computer hardware.
  • a computer system 900 includes a computer 901 including a CD-ROM (Compact Disk Only Memory) drive 905, an FD (Floppy (registered trademark) Disk) drive 906, a keyboard 902, a mouse 903, a monitor 904, Is provided.
  • a computer 901 including a CD-ROM (Compact Disk Only Memory) drive 905, an FD (Floppy (registered trademark) Disk) drive 906, a keyboard 902, a mouse 903, a monitor 904, Is provided.
  • FIG. 16 is a diagram showing an internal configuration of the computer system 900.
  • a computer 901 in addition to the CD-ROM drive 905 and the FD drive 906, a computer 901 is connected to an MPU (Micro Processing Unit) 911, a ROM 912 for storing a program such as a bootup program, and the MPU 911.
  • MPU Micro Processing Unit
  • ROM Read Only Memory
  • a RAM Random Access Memory
  • a hard disk 914 that stores application programs, system programs, and data
  • an MPU 911 and a ROM 912 are interconnected.
  • a bus 915 The computer 901 may include a network card (not shown) that provides connection to the LAN.
  • a program for causing the computer system 900 to execute the functions of the face analysis apparatus and the like according to the above embodiment is stored in the CD-ROM 921 or the FD 922, inserted into the CD-ROM drive 905 or the FD drive 906, and stored in the hard disk 914. May be forwarded. Instead, the program may be transmitted to the computer 901 via a network (not shown) and stored in the hard disk 914. The program is loaded into the RAM 913 when executed. The program may be loaded directly from the CD-ROM 921, the FD 922, or the network.
  • the program does not necessarily include an operating system (OS) or a third-party program that causes the computer 901 to execute the functions of the face analysis apparatus according to the above embodiment.
  • the program may include only a part of an instruction that calls an appropriate function (module) in a controlled manner and obtains a desired result. How the computer system 900 operates is well known and will not be described in detail.
  • the face analysis apparatus is suitable as an apparatus for analyzing a lecture, and is particularly useful as an apparatus for performing an analysis on a lecture using an image obtained by photographing a student's face. .

Landscapes

  • Engineering & Computer Science (AREA)
  • Business, Economics & Management (AREA)
  • Physics & Mathematics (AREA)
  • Theoretical Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Educational Administration (AREA)
  • Educational Technology (AREA)
  • Multimedia (AREA)
  • Tourism & Hospitality (AREA)
  • General Health & Medical Sciences (AREA)
  • Health & Medical Sciences (AREA)
  • Human Resources & Organizations (AREA)
  • Oral & Maxillofacial Surgery (AREA)
  • Economics (AREA)
  • Human Computer Interaction (AREA)
  • Marketing (AREA)
  • Primary Health Care (AREA)
  • Strategic Management (AREA)
  • General Business, Economics & Management (AREA)
  • Image Processing (AREA)
  • Management, Administration, Business Operations System, And Electronic Commerce (AREA)
  • Image Analysis (AREA)

Abstract

L'invention vise à fournir un dispositif d'analyse faciale qui permet de réaliser une analyse appropriée portant sur un cours à l'aide d'informations telles qu'une image d'étudiants pendant ce cours. Pour cela, cette invention comporte : une unité de stockage d'image d'étudiant(s) (12) servant à stocker une image d'étudiant(s), c'est-à-dire une image vidéo du visage d'un étudiant ou des visages de plusieurs étudiants pendant un cours ; une unité d'analyse faciale (13) destinée à percevoir les visages du ou des étudiants dans l'image d'étudiant(s) et à analyser le ou les visages perçus ; et une unité de sortie (17) conçue pour émettre des informations relatives au résultat de l'analyse effectuée par ladite unité d'analyse faciale (13). Une analyse portant sur le cours est ainsi réalisée.
PCT/JP2012/073185 2011-09-15 2012-09-11 Dispositif d'analyse faciale, procédé d'analyse faciale et support à mémoire WO2013039062A1 (fr)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
JP2011201450A JP5441071B2 (ja) 2011-09-15 2011-09-15 顔分析装置、顔分析方法、及びプログラム
JP2011-201450 2011-09-15

Publications (1)

Publication Number Publication Date
WO2013039062A1 true WO2013039062A1 (fr) 2013-03-21

Family

ID=47883291

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/JP2012/073185 WO2013039062A1 (fr) 2011-09-15 2012-09-11 Dispositif d'analyse faciale, procédé d'analyse faciale et support à mémoire

Country Status (2)

Country Link
JP (1) JP5441071B2 (fr)
WO (1) WO2013039062A1 (fr)

Cited By (10)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2017018012A1 (fr) * 2015-07-28 2017-02-02 ソニー株式会社 Système de traitement d'informations, procédé de traitement d'informations, et support d'informations
WO2018097177A1 (fr) * 2016-11-24 2018-05-31 株式会社ガイア・システム・ソリューション Système de mesure d'engagement
CN109889916A (zh) * 2019-01-07 2019-06-14 北京汉博信息技术有限公司 录播数据的应用系统
CN110633615A (zh) * 2018-06-20 2019-12-31 矢崎总业株式会社 乘车人数监视系统
CN110827491A (zh) * 2019-09-26 2020-02-21 天津市华软创新科技有限公司 一种学校学生行为大数据分析系统
CN111931585A (zh) * 2020-07-14 2020-11-13 东云睿连(武汉)计算技术有限公司 一种课堂专注度的检测方法及装置
CN112232166A (zh) * 2020-10-10 2021-01-15 中国平安人寿保险股份有限公司 基于人工智能的讲师动态评价方法、装置、计算机设备
CN114724229A (zh) * 2022-05-23 2022-07-08 北京英华在线科技有限公司 在线教育平台用的学习状态检测系统及方法
CN116798103A (zh) * 2023-08-29 2023-09-22 广州诚踏信息科技有限公司 基于人工智能的人脸图像处理方法及系统
CN117079338A (zh) * 2023-10-17 2023-11-17 山东惠通科技有限公司 基于人工智能的教学质量监测方法及系统

Families Citing this family (13)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP6672592B2 (ja) * 2015-02-19 2020-03-25 富士通株式会社 発表者選択支援プログラム、発表者選択支援方法、および情報処理装置
JP2016218766A (ja) * 2015-05-21 2016-12-22 富士ゼロックス株式会社 情報処理装置及び情報処理プログラム
JP6451608B2 (ja) * 2015-11-30 2019-01-16 京セラドキュメントソリューションズ株式会社 講義確認システム
JP6512082B2 (ja) * 2015-11-30 2019-05-15 京セラドキュメントソリューションズ株式会社 講義確認システム
JP6859640B2 (ja) * 2016-09-21 2021-04-14 富士ゼロックス株式会社 情報処理装置、評価システムおよびプログラム
JP6897095B2 (ja) * 2016-12-27 2021-06-30 富士通株式会社 画像処理プログラム、画像処理装置及び画像処理方法
JP7069550B2 (ja) * 2017-03-16 2022-05-18 日本電気株式会社 講義映像分析装置、講義映像分析システム、方法およびプログラム
US10930169B2 (en) * 2017-05-04 2021-02-23 International Business Machines Corporation Computationally derived assessment in childhood education systems
TWI684159B (zh) * 2018-03-20 2020-02-01 麥奇數位股份有限公司 用於互動式線上教學的即時監控方法
JP7136216B2 (ja) * 2018-09-14 2022-09-13 日本電気株式会社 授業支援システム、判定装置、授業支援方法及びプログラム
JP6850507B1 (ja) * 2020-03-31 2021-03-31 株式会社オープンソース活用研究所 セミナー運営システム、セミナー運営方法、及びサーバ装置
KR102737980B1 (ko) * 2021-12-10 2024-12-04 (주)유시스 지능형 실시간 비대면 영상 교육 시스템
WO2023203960A1 (fr) * 2022-04-19 2023-10-26 Ｍｅｔａｃｌａｓｓｒｏｏｍ株式会社 Dispositif de traitement d'informations, procédé de traitement d'informations, programme, et système de traitement d'informations

Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2006330464A (ja) * 2005-05-27 2006-12-07 Fujifilm Holdings Corp 受講状態判別装置および方法並びにプログラム
JP2007102344A (ja) * 2005-09-30 2007-04-19 Fujifilm Corp 自動評価装置、プログラムおよび方法
JP2009258175A (ja) * 2008-04-11 2009-11-05 Yamaha Corp 講義システムおよび集計システム

Family Cites Families (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2002023716A (ja) * 2000-07-05 2002-01-25 Pfu Ltd プレゼンテーションシステムおよび記録媒体
JP2003037826A (ja) * 2001-07-23 2003-02-07 Alpine Electronics Inc 代理画像表示装置およびテレビ電話装置
JP2007213282A (ja) * 2006-02-09 2007-08-23 Seiko Epson Corp 講演者支援装置および講演者支援方法
WO2008129356A2 (fr) * 2006-03-13 2008-10-30 Imotions-Emotion Technology A/S Système de détection et d'affichage d'attention visuelle et de réponse émotionnelle

Patent Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2006330464A (ja) * 2005-05-27 2006-12-07 Fujifilm Holdings Corp 受講状態判別装置および方法並びにプログラム
JP2007102344A (ja) * 2005-09-30 2007-04-19 Fujifilm Corp 自動評価装置、プログラムおよび方法
JP2009258175A (ja) * 2008-04-11 2009-11-05 Yamaha Corp 講義システムおよび集計システム

Non-Patent Citations (1)

* Cited by examiner, † Cited by third party
Title
EIJI WATANABE ET AL.: "Analysis of behaviors by lecturer and students in lecture based on image processing", IEICE TECHNICAL REPORT, vol. 110, no. 383, 21 January 2011 (2011-01-21) *

Cited By (19)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US10607063B2 (en) 2015-07-28 2020-03-31 Sony Corporation Information processing system, information processing method, and recording medium for evaluating a target based on observers
CN107851324A (zh) * 2015-07-28 2018-03-27 索尼公司 信息处理系统、信息处理方法和记录介质
US20190095695A1 (en) * 2015-07-28 2019-03-28 Sony Corporation Information processing system, information processing method, and recording medium
WO2017018012A1 (fr) * 2015-07-28 2017-02-02 ソニー株式会社 Système de traitement d'informations, procédé de traitement d'informations, et support d'informations
WO2018097177A1 (fr) * 2016-11-24 2018-05-31 株式会社ガイア・システム・ソリューション Système de mesure d'engagement
CN110633615B (zh) * 2018-06-20 2023-04-25 矢崎总业株式会社 乘车人数监视系统
CN110633615A (zh) * 2018-06-20 2019-12-31 矢崎总业株式会社 乘车人数监视系统
CN109889916B (zh) * 2019-01-07 2023-04-07 北京汉博信息技术有限公司 录播数据的应用系统
CN109889916A (zh) * 2019-01-07 2019-06-14 北京汉博信息技术有限公司 录播数据的应用系统
CN110827491A (zh) * 2019-09-26 2020-02-21 天津市华软创新科技有限公司 一种学校学生行为大数据分析系统
CN111931585A (zh) * 2020-07-14 2020-11-13 东云睿连(武汉)计算技术有限公司 一种课堂专注度的检测方法及装置
CN112232166A (zh) * 2020-10-10 2021-01-15 中国平安人寿保险股份有限公司 基于人工智能的讲师动态评价方法、装置、计算机设备
CN112232166B (zh) * 2020-10-10 2023-12-01 中国平安人寿保险股份有限公司 基于人工智能的讲师动态评价方法、装置、计算机设备
CN114724229A (zh) * 2022-05-23 2022-07-08 北京英华在线科技有限公司 在线教育平台用的学习状态检测系统及方法
CN114724229B (zh) * 2022-05-23 2022-09-02 北京英华在线科技有限公司 在线教育平台用的学习状态检测系统及方法
CN116798103A (zh) * 2023-08-29 2023-09-22 广州诚踏信息科技有限公司 基于人工智能的人脸图像处理方法及系统
CN116798103B (zh) * 2023-08-29 2023-12-01 广州诚踏信息科技有限公司 基于人工智能的人脸图像处理方法及系统
CN117079338A (zh) * 2023-10-17 2023-11-17 山东惠通科技有限公司 基于人工智能的教学质量监测方法及系统
CN117079338B (zh) * 2023-10-17 2024-01-09 山东惠通科技有限公司 基于人工智能的教学质量监测方法及系统

Also Published As

Publication number Publication date
JP5441071B2 (ja) 2014-03-12
JP2013061906A (ja) 2013-04-04

Similar Documents

Publication Publication Date Title
JP5441071B2 (ja) 顔分析装置、顔分析方法、及びプログラム
CN106605218B (zh) 收集和处理与基于网络的内容的交互过程中的计算机用户数据的方法
US20140075465A1 (en) Time varying evaluation of multimedia content
CN111193890B (zh) 会议记录解析装置、方法和会议记录播放系统
WO2008045153A3 (fr) Interaction basée sur la reconnaissance faciale des participants à une conférence
JP2006085440A (ja) 情報処理システム及び情報処理方法、並びにコンピュータ・プログラム
WO2017018012A1 (fr) Système de traitement d'informations, procédé de traitement d'informations, et support d'informations
JP6859640B2 (ja) 情報処理装置、評価システムおよびプログラム
CN104092957A (zh) 一种融合人像和语音的屏幕视频生成方法
KR102263154B1 (ko) 스마트 미러 기반 얼굴 감성 표현 시스템 및 그 운용방법
CN105561536A (zh) 一种具有健身动作矫正功能的人机交互系统
Yu et al. Capture, recognition, and visualization of human semantic interactions in meetings
CN111770300B (zh) 一种会议信息处理的方法及虚拟现实头戴设备
JP2009267621A (ja) 通信装置
CN109949392A (zh) 动画发送系统
JP2011164681A (ja) 文字入力装置、文字入力方法、文字入力プログラムおよびそれを記録したコンピュータ読み取り可能な記録媒体
JP6819194B2 (ja) 情報処理システム、情報処理装置およびプログラム
WO2020137536A1 (fr) Dispositif d'authentification de personne, procédé de commande et programme
KR101838792B1 (ko) 콘텐츠에 대한 사용자의 감정을 공유하는 방법 및 장치
US20220335275A1 (en) Multimodal, dynamic, privacy preserving age and attribute estimation and learning methods and systems
WO2020144835A1 (fr) Dispositif de traitement d'informations, et procédé de traitement d'informations
JP7206741B2 (ja) 健康状態判定システム、健康状態判定装置、サーバ、健康状態判定方法、及びプログラム
JP2009098901A (ja) 表情検出方法及び装置及びプログラム
Suryani et al. Multi-modal asian conversation mobile video dataset for recognition task
JP6859641B2 (ja) 評価システム、情報処理装置およびプログラム

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 12831355

Country of ref document: EP

Kind code of ref document: A1

DPE1 Request for preliminary examination filed after expiration of 19th month from priority date (pct application filed from 20040101)
NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 12831355

Country of ref document: EP

Kind code of ref document: A1

点击 这是indexloc提供的php浏览器服务，不要输入任何密码和下载