JP2012212325A - Visual axis measuring system, method and program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To improve accuracy of an estimation or measurement of a visual axis by collating with a face image of an entire frame and correcting a personal parameter and a frame parameter.SOLUTION: In Step S105, a server 12 acquires position data of feature points of a face and, in Step S107, acquires data of a posture and position of the face. In Step S111, the server sets the personal parameter and the frame parameter for each frame image based on the position data of feature points and the face position and posture data, and, in Step S115, calculates a score value representing goodness of fit of the personal parameter and the frame parameter to each frame face image, and integrates the score values on the entire frame image. In Step S121, each parameter is corrected till the integrated score value becomes less than a predetermined threshold. The server measures the visual axis direction as a three-dimensional straight line connecting the eyeball center to the iris center, based on the corrected personal parameter and frame parameter (S123).

Description

  The present invention relates to a line-of-sight measurement system, method, and program, and in particular, a line-of-sight measurement system that estimates or measures the line-of-sight direction of an eye included in the face image, for example, by processing a number of frame face images captured by a monocular camera, It relates to a method and a program.

Patent Document 1 discloses a gaze estimation system proposed by the present applicant. In this background art, the line-of-sight direction of the subject can be estimated by processing the face image signal of the subject from the monocular camera.
JP 2008-102902 A [G06T 7/60 A61B 3/113]

  In the background art of Patent Document 1, the gaze direction is estimated by real-time processing of a face image signal from a camera, and processing is started from a state in which the arrangement of parts such as eyes and nose in the subject's face is not known at all. Therefore, there was a limit to the improvement of accuracy by sequential learning.

  Therefore, a main object of the present invention is to provide a novel gaze measurement system, method and program.

  Another object of the present invention is to provide a line-of-sight measurement system, method, and program capable of improving accuracy.

  The present invention employs the following configuration in order to solve the above problems. Note that reference numerals in parentheses, supplementary explanations, and the like indicate correspondence with embodiments to be described later in order to help understanding of the present invention, and do not limit the present invention.

  A first invention is a line-of-sight measurement system that measures a line-of-sight direction for each frame face image based on a human eyeball position and iris position acquired from the frame face image, and uses the standard model for each frame face image. Feature point data acquisition means for acquiring feature point position data including the iris position of the face image, face data acquisition means for acquiring face position and orientation data for each frame face image using the feature point position data, Parameter setting means for setting individual parameters and frame parameters for each frame image based on feature point position data and face position and orientation data, and fitness calculation for calculating the fitness of individual parameters and frame parameters for all frame face images Means for correcting personal parameters and frame parameters until the matching level reaches a predetermined threshold And a means for measuring a gaze direction based on the modified personal parameter and frame parameters, a gaze tracking system.

  In the first invention, for example, the feature point data acquisition means (12,261, S105) formed by the computer (server 12) uses the standard model to position the feature points including the iris position of the face image for each frame face image. Get the data. Here, the standard model is, for example, coordinate data indicating the position of the three-dimensional coordinates of predetermined feature points (for example, the eyes of the eyes, the corners of the eyes, and the corners of the eyes) constructed according to the knowledge of anatomy, and those Data of the position of the eyeball and the radius of the eyeball that can be estimated anatomically with respect to these feature points. The feature point data acquisition means (12,261, S105) applies such a standard model to the frame face image and detects the feature point of the subject. The face data acquisition means (12,262, S107) uses the position data of the feature points, for example, QR-decomposes the projection matrix (P) related to a plurality of feature points, and performs face position and posture for each frame face image. Get the data. The parameter setting means (12, S111, S121) sets personal parameters and frame parameters for each frame image based on the feature point position data and the face position and orientation data. As an example, the personal parameter corresponds to the above-described standard model unique to each subject and is common to all frames, and is a set of three-dimensional coordinates of the six feature points, the position of the eyeball, and the radius (r) of the eyeball. As an example, the frame parameter is a parameter specific to each frame, and includes a face position and posture, an iris (pupil) position, and an iris (pupil) radius. The goodness-of-fit calculation means (12,263, S115) calculates score values indicating the goodness of the individual parameters and frame parameters set by the parameter setting means for each frame face image, and accumulates the score values for all frame images. That is, the fitness level calculation means (12,263, S115) calculates the integrated score value as the fitness level. The correction means (12, 264, S117, S121) corrects the personal parameters and the frame parameters until the matching level reaches a predetermined threshold. That is, the integrated score value is calculated each time the personal parameter and the frame parameter are corrected by the correcting means, and the correction and the integrated score value calculation are repeated until the score value falls below a predetermined threshold value. Then, the gaze direction measuring means (12,265, S123) measures the gaze direction as, for example, a three-dimensional straight line connecting the eyeball center and the iris center based on the corrected personal parameter and frame parameter.

  According to the first invention, since the personal parameters and the frame parameters are corrected so as to be suitable for the full-frame face image, the accuracy of the line-of-sight measurement is improved.

  The second invention is dependent on the first invention, and the fitness calculation means compares the iris projection image generated based on the personal parameter and the frame parameter using the standard model and the iris in each frame face image. This is a line-of-sight measurement system that calculates the sum of all frame face images of the above error as fitness.

  According to the second aspect of the invention, since the error between the generated projection image of the iris and the iris of the actual frame face image is integrated for all frames to obtain the fitness, the accuracy of the gaze direction to be estimated or measured can be expected to improve. .

  The third invention is dependent on the first invention, and the fitness calculation means includes a projected image of predetermined feature points of the face including the iris generated based on the personal parameters and the frame parameters using the standard model, This is a line-of-sight measurement system that calculates the total of all frame face images of errors in comparison with their feature points in the frame face image as the fitness.

  According to the third aspect of the invention, the sum of errors (distances) between the projected images of predetermined feature points of the face including the iris and those feature points in the actual frame face image is integrated for all frames to obtain the fitness. According to these inventions, the fitness can be accurately calculated, and as a result, it is expected that the accuracy of the gaze direction to be estimated or measured is improved.

  A fourth invention is a line-of-sight measurement system according to any one of the first to third inventions, further comprising update means for updating the standard model based on the corrected parameter.

  According to the fourth invention, since the standard model is updated with the corrected personal parameter, the range of subjects who can detect feature points using the standard model is expanded. That is, a standard model that can be applied to more types of subjects is obtained.

  A fifth invention is a line-of-sight measurement method for measuring a line-of-sight direction for each frame face image based on a human eyeball position and an iris position acquired from the frame face image, and uses the standard model for each frame face image. A feature point data acquisition step for acquiring feature point position data including the iris position of the face image, a face data acquisition step for acquiring face position and orientation data for each frame face image using the feature point position data, Parameter setting step for setting individual parameters and frame parameters for each frame image based on the feature point position data and face position and posture data, and fitness calculation for calculating the suitability of individual parameters and frame parameters for all frame face images Step, modify personal parameters and frame parameters until the matching level reaches a predetermined threshold Comprising the step of measuring a gaze direction based on personal parameters and frame parameters were fixed steps, and modifications that are line-of-sight measurement method.

  In the fifth invention, the same effect as in the first invention can be expected.

  A sixth invention is a line-of-sight measurement program executed by a computer of a line-of-sight measurement system that measures a line-of-sight direction for each frame face image based on a human eyeball position and an iris position acquired from the frame face image. The feature point data acquisition means for acquiring the feature point position data including the iris position of the face image for each frame face image using the standard model, the face point for each frame face image using the feature point position data Face data acquisition means for acquiring position and orientation data, parameter setting means for setting individual parameters and frame parameters for each frame image based on the position data of feature points and the position and orientation data of faces, all of the personal parameters and frame parameters A fitness calculation means for calculating the fitness for the frame face image, and the fitness is a predetermined threshold Made up to function as a means for measuring a gaze direction based on personal parameters and correcting means corrects the frame parameters, and modified personal parameter and frame parameters, a line-of-sight measurement program.

  In the sixth invention, the same effect as in the first invention can be expected.

  According to the present invention, since the personal parameters and the frame parameters are corrected by collating with the face images of all frames, the accuracy of eye gaze estimation or measurement can be improved.

  The above object, other objects, features, and advantages of the present invention will become more apparent from the following detailed description of embodiments with reference to the drawings.

FIG. 1 is a block diagram showing a visual line measuring system according to an embodiment of the present invention. FIG. 2 is an illustrative view showing a memory map of the memory of the server in FIG. 1 embodiment. FIG. 3 is a flowchart showing the processing operation of eye gaze measurement executed by the server. FIG. 4 is an illustrative view showing one example of a face image of a subject for performing a gaze measurement in the gaze measurement system of the FIG. 1 embodiment. FIG. 5 is a conceptual diagram for explaining a filter for detecting an eyebrow candidate region. FIG. 6 is a conceptual diagram showing another configuration of the six-divided rectangular filter. FIG. 7 is an illustrative view for explaining modeling by SVM using an image region centered on the eyebrows. FIG. 8 is an illustrative view showing an example of a face detection result. FIG. 9 is a conceptual diagram illustrating a model for determining the line-of-sight direction. FIG. 10 is a conceptual diagram showing the relationship between the subject's iris center, eyeball center, and projection point.

  The line-of-sight measurement system 10 according to an embodiment of the present invention includes a server 12 that is accessed from a client 14 via a network 16. The client 14 is provided with a camera 18 including a solid-state imaging device such as a CCD or a CMOS sensor. The camera 18 captures the face of the subject 20 and sends a face image (moving image) signal of the subject to the client 14. The client 14 uses a series of face image signals of the subject 20 acquired by the camera 18 as a series of frame face image data, and estimates the direction of the subject's line of sight 20 from the frame face image to be measured. Send to server 12. However, transmission of the frame face image data from the client 14 to the server 12 is performed via the network 16, and for example, a method of taking in the server 12 from a recording medium (not shown) on which the frame face image data is recorded is considered. It is done.

  The server 12 stores the input image data input as described above in an input image storage device 22 which is a storage device such as a hard disk or a semiconductor memory.

  The line-of-sight data (line-of-sight data) measured by the server 12 is sent to the client 14 via the network 16 as necessary.

  The line-of-sight measurement in the server 12 uses the line-of-sight estimation method already proposed by the present inventors in Patent Document 1 (Japanese Patent Laid-Open No. 2008-102902) cited as background art. In simple terms, the background gaze measurement technology calculates the relative relationship between the facial feature point and the eyeball center from the relationship between the facial feature point of the subject and the iris center, and then based on that relative relationship, The eyeball center position is estimated from the feature point group obtained in the image, and the gaze direction is estimated from the position and the iris center position.

  FIG. 2 shows a memory map of the memory 24 of the server 12 in the embodiment of FIG. 1, in which a program storage area 26 and a data storage area 28 are formed. In the program storage area 28, a face detection / feature point extraction program 261 for detecting the face of the subject from the input face image and extracting feature points of the face, which will be described in detail later, and the subject for the camera 16 from the face image A face position / posture detection program 262 for detecting the position and posture of the face. As facial feature points, in the embodiment, a total of six points including the eyes and corners of the left and right eyes of the subject and both ends (mouth corners) of the mouth are used. These feature points are expressed as two-dimensional coordinates.

  The program storage area 26 is further based on a score integration program 263 for score integration to be described later and the result of the score integration processing in order to determine the adaptability of the personal parameters and frame parameters (to be described later) to the input face image. A parameter correction program 264 for correcting personal parameters and frame parameters is included.

  The program storage area 26 further updates the standard model based on the result of parameter correction, and the line-of-sight estimation program 265 that estimates the line-of-sight direction based on the eyeball center included in the corrected personal parameter and the iris center included in the frame parameter. The standard model update program 266 is included.

  The data storage area 28 includes a frame face image data storage area 281 for storing data of each frame face image obtained by disassembling a series of frame face images, and the input image storage device 22 ( The face image data read from FIG. 1) and decomposed into each frame is stored.

  The data storage area 28 also includes a standard model storage area 282 that stores standard model data used in the above face detection / feature point extraction program 261 and face position / posture estimation program 262. Here, the standard model is coordinate data indicating the position of the three-dimensional coordinates of the above-mentioned six feature points of the human face constructed according to anatomical knowledge, and can be estimated anatomically with respect to these feature points. It is a database including data on the position of the eyeball and the radius (r) of the eyeball.

  The feature point data and the face position and orientation data detected by the face detection / feature point extraction program 261 and the face position / posture estimation program 262 together with the image of the eye region as shown in FIG. The point, face position, and posture data storage area 283 is stored.

  The data storage area 28 further includes a personal parameter storage area 284 and a frame parameter storage area 285. The personal parameters are a set of data sets corresponding to the above-mentioned standard model unique to each subject and common to all frames, including the three-dimensional coordinates of the six feature points, the position of the eyeball, and the radius (r) of the eyeball. That is. On the other hand, the frame parameter is a parameter specific to each frame, and is a data set including a face position and posture, an iris (pupil) position, and an iris (pupil) radius. Therefore, the frame parameter storage area 285 has a storage location corresponding to the number of frames (K) of a series of face images.

  The line-of-sight data storage area 286 formed in the data storage area 28 stores data indicating the line-of-sight direction estimated or measured by the above-described line-of-sight estimation program, for example, an angle in a turning direction (an angle in a horizontal plane) and an elevation direction. This is an area for storing data for each frame as angle (angle in the vertical plane) data. This line-of-sight data storage area also includes a storage location corresponding to the number of frames (K) of a series of face images.

  FIG. 3 is a flowchart showing the operation of the embodiment in FIG. 1 executed by the server 12. In the first step S101, the server 12 is input from, for example, the client computer 14 and stored in the input image storage device 22 (FIG. 1). The series of frame face image data that has been processed is decomposed into face image data for each frame and stored in the frame face image data storage area 281 of the memory 24 (FIG. 2).

  Then, a counter (not shown) that is formed in an appropriate area of the memory 26 and counts the number of frames is incremented (step S103). In order to process the first frame, “1” is set in this counter, and is sequentially incremented for each frame. Thereafter, steps S105 to S111 are repeatedly executed until it is determined in step S113 that the processing up to the last frame (K) has been completed.

In step S105, according to the face detection / feature point extraction program 261 shown in FIG. 2, the face of the subject is detected from the frame face image at that time, and then feature points are extracted.
(Face detection)
As a premise of the operation of the gaze direction estimation process, first, for example, a face detection process is executed using a six-divided rectangular filter.

  When processing the face image, the server 12 scans the screen with a rectangular filter whose width is the width of the face and whose length is about half that of the face. The rectangle is divided into, for example, 3 × 2, and the average brightness of each divided region is calculated, and when the relative brightness relationship is satisfied, the center of the rectangle is set as a candidate for the eyebrows.

  When consecutive pixels become the eyebrow candidate, only the center candidate of the frame surrounding it is left as the eyebrow candidate. By comparing the remaining eyebrow candidates with the standard model and performing template matching or the like, the false eyebrow candidates are discarded from the eyebrow candidates obtained by the above-described procedure, and the true eyebrow candidates are extracted. This will be described in more detail below.

  FIG. 4 is a conceptual diagram for explaining a filter for detecting an eyebrow candidate region. FIG. 4A shows the above described 3 × 2 rectangular filter (hereinafter referred to as “6-divided rectangular filter”). ").

  The six-divided rectangular filter is a filter that extracts facial features such as (1) nose muscles are brighter than both eye regions and (2) eye regions are darker than the cheeks, and obtains the position between the eyebrows. For example, a rectangular frame of horizontal i pixels and vertical j pixels (i, j: natural number) is provided centering on one point (x, y). Then, as shown in FIG. 4A, this rectangular frame is divided into three equal parts horizontally and two equal parts vertically and divided into six blocks S1-S6.

  When such a 6-divided rectangular filter is applied to both eye regions and cheeks of a face image, the result is as shown in FIG.

  However, although the 6-divided filter in FIG. 4 is obtained by equally dividing each rectangular area, this filter may be modified as shown in FIG.

  Considering that the nose muscle portion is usually narrower than the eye region, it is more desirable that the width w2 of the blocks S2 and S5 is narrower than the width w1 of the blocks S1, S3, S4 and S6. Preferably, the width w2 can be half of the width w1. FIG. 10 shows the configuration of a six-divided rectangular filter in such a case. Further, the vertical width h1 of the blocks S1, S2 and S3 and the vertical width h2 of the blocks S4, S5 and S6 are not necessarily the same.

  In the 6-divided rectangular filter shown in FIG. 5, for each block Si (1 ≦ i ≦ 6), the average value “bar Si” of the pixel luminance (“−” (bar) is added above Si). .

  Assuming that one eye and eyebrows exist in the block S1 and another eye and eyebrows exist in the block S3, the following relational expressions (1) and (2) hold.

  Therefore, a point satisfying these relationships is extracted as an eyebrow candidate (face candidate).

  For the process of calculating the sum of pixels in a rectangular frame, a known document (P. Viola and M. Jones, “Rapid Object Detection using a Boosted Cascade of Simple Features,” Proc. Of IEEEConf. CVPR, 1, pp.511). -518, 2001), it is possible to incorporate a high-speed calculation method using an integral image. By using an integral image, it can be executed at high speed regardless of the size of the filter. By applying this method to a multi-resolution image, face candidates can be extracted even when the size of the face on the image changes.

  For the eyebrow candidate (face candidate) obtained in this way, the true eyebrow position (true face region) can be specified by template matching with the standard model described above.

Note that a face area can be determined by applying verification processing using a face model by a support vector machine (SVM) to the obtained face candidates. In order to avoid a reduction in recognition rate due to differences in hairstyles, presence or absence of wrinkles, and changes in facial expressions, for example, as shown in FIG. For the determination of the true face area by SVM, refer to S. Kawato, N. Tetsutani and K. Hosaka: “Scale-adaptive face detection and tracking in real time with ssr fi1ters and support vector machine”, IEICE Trans. on Info. and Sys., E88-D, 12, pp. 2857-2863 (2005). Real-time face detection is possible by combining high-speed candidate extraction with a six-divided rectangular filter and processing by SVM.
(Feature point detection)
Subsequently, the extraction of feature points at the eyes, mouth and iris (pupil), for example, the feature points at both ends of the eye and at both ends of the mouth, is realized by searching for similar points using template images around each feature point prepared in advance. it can. By converting the template image into a low-dimensional vector using a spatial filter such as a Gabor filter, it is possible to perform a collation process that is robust against an illumination change and efficient.

  As for the positions of both eyes, since the pattern between the eyebrows is searched for by detecting the face area described above, the positions of both eyes can be roughly estimated by searching again for the dark areas on both sides of the eyebrows. However, it is necessary to extract the iris center more accurately in order to estimate the gaze direction. Here, for the peripheral region of the eye obtained above, iris edge candidates are extracted by Laplacian, and the Hough transform of the circle is applied to detect the projection position of the iris and the center of the iris.

  FIG. 8 is a diagram illustrating an example of a face detection result. In the detected face, the iris center, nose tip and mouth are also detected. For example, as the feature points, the right and left eye corners, the corners of the eyes, both ends of the mouth, and the like can be used.

The feature point position data detected in this way is stored for each frame in the feature point, face position, and posture storage area 283 of the memory 24 (FIG. 2).
(Principle of gaze estimation)
In the gaze estimation, the gaze direction is given as a three-dimensional straight line connecting the eyeball center and the iris center.

  FIG. 9 is a conceptual diagram illustrating a model for determining the line-of-sight direction. When the eyeball radius on the image is r and the distance between the eyeball center and the iris center on the image is d, the angle θ formed by the line-of-sight direction and the camera optical axis is expressed by the following equation (3).

As will be described in detail later, in the estimation of the line-of-sight direction in this embodiment, the estimation process of the relative relationship between the eyeball center and the face feature point and the projection position estimation of the eyeball center are performed. Therefore, in step S107, the face position and posture are estimated.
(Face position / posture estimation)
2D observation position xj (k) (bold) = [xj (k), yj (k)] t and v) 3D position sj (bold) = [Xj, Yj, In the range of Zj] t (j = 1,..., M), when attention is paid to the m feature points observed among the M feature points, the following relationship is obtained.

  However, the matrix P (k) is a 2 × 3 matrix. The matrix S (k) of the second term on the right side is a partial matrix consisting of only elements corresponding to the observed feature points in the matrix S. As described above, it is assumed that the camera and the face are sufficiently separated from each other and an orthogonal projection is assumed. Here, if four or more feature points are observed, the matrix P (k) can be calculated as follows.

  The projection positions xr (i) (bold) and xl (i) (bold) at the center of the eyeball in the image frame Ik can be calculated as follows using the matrix P (k) (step S210).

  Therefore, the line of sight can be estimated by using the iris center projection position extracted as the feature point in the image frame Ik and the eyeball center projection position.

  By decomposing the matrix P by QR decomposition, the face posture R and the face position can be calculated as follows.

  However, r1 and r2 are 1 × 3 vectors, respectively. Such detection of face posture R is described in literature: L.L. Quan: “Self-calibration of an affine camera from multiple views”, Int’l Journal of Computer Vision, 19, pp. 93-105 (1996).

  The obtained r is regarded as a true value, and the scale s and the translation vectors vx and vy that minimize the projection error are obtained by the least square method according to the equations (10), (11), and (12). The scale s in Expression (12) indicates the size of the face, and the translation vectors vx and vy indicate the position of the face.

  In this way, in step S107, the position and posture of the subject's face in each frame image are estimated.

  The face position and orientation data detected in this way is stored for each frame in the feature point, face position and orientation storage area 283 of the memory 24 (FIG. 2).

  In subsequent step S109, the server 12 extracts image data of the eye area as shown in FIG. 7 from the input image being processed at that time, and stores it in the feature point, face position, and posture storage area 283 of the memory 24. Along with the feature point, face position, and posture data, each frame is stored.

  In the next step S111, the server 12 sets initial parameters of the frame in accordance with the feature point, face position, and posture data obtained above. As described above, the personal parameter is the eyeball position obtained from the relative positional relationship with the feature point, and the eyeball radius obtained anatomically with the eyeball position as the center. However, personal parameters do not change from frame to frame, but are common to all frames. The initial values of the personal parameters set in step S111 are stored in the personal parameter storage area 284 (FIG. 2). The frame parameters change from frame to frame, and the initial values of the face position, posture, iris (pupil) position, and iris (pupil) radius in each frame are stored in the frame parameter storage area 285 (FIG. 2). Store it in a memory location.

  After setting the initial parameters in this way, the server 12 accumulates scores for all frames in step S115. “Score” is, for example, a projection image (computer graphics image) of an iris generated based on personal parameters (eyeball radius, eyeball position) and frame parameters (iris radius and iris position) using standard model data, This is an error in comparison with the iris in each actual frame face image. As an actual face image, in the embodiment, the image of the eye region as shown in FIG. 7 stored for each frame together with the feature point, face position, and posture data in the feature point, face position, and posture storage region 283 in step S109. Is used. For this error, the difference (distance) and size (area) between the projected image of the iris and the iris in the frame face image are calculated as the number of pixels on the image. Alternatively, the sum of the deviation (distance) of the projection positions of predetermined feature points of the face including the iris and the observation positions of those feature points in the actual frame face image may be calculated as an error. The error (score) for each frame is totaled for all K frames. That is, the scores are integrated for all frames. The score indicates the degree (fitness) that the personal parameter and the frame parameter set in step S111 functioning as the parameter setting means are adapted to each frame face image. Therefore, the integrated score obtained in step S115. In other words, the value is the degree of suitability of the personal parameter and the frame parameter set by the initial parameter setting means with respect to the all-frame face image.

  Next, in step S117, the score value integrated in step S115 is compared with a predetermined threshold value, and whether or not the score value is smaller than the threshold value, that is, the personal parameters common to all frames set in step S111 and for each frame. It is determined whether or not the frame parameters are sufficiently adapted to all the actual frame images. In step S117, that is, it is determined whether or not the parameter falls within an allowable range when the gaze direction is estimated using such parameters.

  If “NO” is determined in step S117, that is, if it is determined that the integrated score value is equal to or greater than the threshold value, in the next step S119, the server 12 performs step S121 based on the score value obtained in step S115. It is determined whether or not the number of times the personal parameter or frame parameter has been corrected (the number of repetitions) exceeds a predetermined specified value.

If “NO” in the step S119, in the next step S121, the server 12 modifies the personal parameter and the frame parameter by using an optimization method so that the score value becomes the smallest. Although various correction methods are conceivable, in the embodiment, the steepest descent method is used as an example. The steepest descent method is a method of obtaining a numerical value of a parameter that minimizes the error by correcting the parameter in a direction in which the slope of the error curved surface descends most steeply. However, another method may be employed to optimize error correction.

  In this way, steps S115 to S121 are repeatedly executed, and the personal parameters and the frame parameters are actually corrected to be below the threshold value so that the score value is ideally zero (0). Note that the personal parameters and the frame parameters (initial parameters) set in step S111 are corrected in step S121 based on the comparison with the face images of all frames. However, the parameter modified in step S121 is further modified in step S121, and in this sense, step S121 also functions as a parameter setting unit, similar to the previous step S111.

  The personal parameters modified in step S121 and the frame parameters for each frame face image are stored in the personal parameter storage area 284 and the frame parameter storage area 285 shown in FIG.

  Then, if “YES” is determined in step S117 or “YES” is determined in step S119, the server 12 subsequently proceeds based on the personal parameters and each frame parameter corrected and stored in step S121. In step S123, the gaze direction is estimated for each frame face image as a three-dimensional straight line (FIG. 10) connecting the eyeball position and the iris position, and the gaze direction data is stored in the gaze data storage area 285 (FIG. 2). This line-of-sight direction data is returned to the client 14 as needed as data indicating the line-of-sight direction of the subject for each frame of the series of face image signals input previously.

  Finally, in step S125, the server 12 updates the standard model using the personal parameters corrected in step S121. If the corresponding parameter of the standard model is an average value, the average value is recalculated by adding the corresponding parameter of the personal parameter corrected in step S121. For example, if one parameter of the standard model is an average value of corresponding parameters of N subjects, in this step S125, an average value of N + 1 people is calculated, and the parameter is updated with the result. When updating the variance value of the parameter of the standard model, the variance value is changed in consideration of weighting, for example, based on the parameter value corrected in step S121.

  By updating the standard model with the modified personal parameters, the range of subjects that can detect feature points using the standard model is expanded. That is, a standard model that can be applied to more types of subjects is obtained.

  As described above, according to this embodiment, in step S121, the parameter is corrected so as to minimize the score value (sum of errors between the setting parameter and the actual face image) accumulated in all frames. The accuracy of the gaze direction estimated or measured by is improved.

DESCRIPTION OF SYMBOLS 10 ... Eye-gaze measurement system 12 ... Server 14 ... Client 16 ... Network 18 ... Camera 20 ... Test subject 22 ... Input image storage device 24 ... Memory

Claims (6)

A line-of-sight measurement system that measures a line-of-sight direction for each frame face image based on a human eyeball position and an iris position acquired from a frame face image,
Feature point data acquisition means for acquiring feature point position data including the iris position of the face image for each frame face image using a standard model;
Face data acquisition means for acquiring face position and posture data for each frame face image using the feature point position data;
Parameter setting means for setting individual parameters and frame parameters for each frame image based on the position data of the feature points and the position and orientation data of the face;
Goodness-of-fit calculation means for calculating goodness of the personal parameters and frame parameters to all frame face images;
A line-of-sight measurement system comprising: correction means for correcting the personal parameter and frame parameter until the fitness level reaches a predetermined threshold; and means for measuring the line-of-sight direction based on the corrected personal parameter and frame parameter.   The goodness-of-fit calculation means calculates the total of the projection image of the iris generated based on the personal parameter and the frame parameter using the standard model and the total frame face image of the error in comparison with the iris in each frame face image. The line-of-sight measurement system according to claim 1, which is calculated as follows.   The fitness calculation means includes an error of comparison between a projected image of predetermined feature points of a face including an iris generated based on personal parameters and frame parameters using a standard model, and those feature points in each frame face image The line-of-sight measurement system according to claim 1, wherein a total of all frame face images is calculated as the fitness.   The line-of-sight measurement system according to claim 1, further comprising updating means for updating the standard model based on the corrected personal parameter. A gaze measurement method for measuring a gaze direction for each frame face image based on a human eyeball position and an iris position acquired from a frame face image,
A feature point data acquisition step for acquiring feature point position data including the iris position of the face image for each frame face image using a standard model;
A face data acquisition step of acquiring face position and posture data for each frame face image using the feature point position data;
A parameter setting step for setting personal parameters and frame parameters for each frame image based on the position data of the feature points and the position and orientation data of the face;
A fitness calculation step of calculating the fitness of the personal parameters and the frame parameters with respect to all frame face images;
A line-of-sight measurement method, comprising: a correction step of correcting the personal parameter and the frame parameter until the fitness level reaches a predetermined threshold; and a step of measuring the line-of-sight direction based on the corrected personal parameter and the frame parameter. A line-of-sight measurement program executed by a computer of a line-of-sight measurement system that measures a line-of-sight direction for each frame face image based on a human eyeball position and iris position acquired from a frame face image, the computer comprising:
Feature point data acquisition means for acquiring feature point position data including the iris position of the face image for each frame face image using a standard model;
Face data acquisition means for acquiring face position and posture data for each frame face image using the feature point position data;
Parameter setting means for setting individual parameters and frame parameters for each frame image based on the position data of the feature points and the position and orientation data of the face;
Goodness-of-fit calculation means for calculating goodness of the personal parameters and frame parameters to all frame face images;
A line-of-sight measurement program that functions as a correction unit that corrects the personal parameter and the frame parameter until the fitness level reaches a predetermined threshold, and a unit that measures the line-of-sight direction based on the corrected personal parameter and the frame parameter.