KR101968523B1 - Mobile terminal using eye tracking function and event indication method thereof - Google Patents

Abstract

The present invention relates to a mobile terminal having an eye tracking function for displaying an event of a mobile terminal in cooperation with a user's eyes and an event displaying method thereof, The display position of the event content is determined in a direction different from the direction of the user's gaze on the content, and the content of the event is displayed on a predetermined window at a determined position.

Description

BACKGROUND OF THE INVENTION 1. Field of the Invention [0001] The present invention relates to a mobile terminal having an eye tracking function,

The present invention relates to a mobile terminal having an eye tracking function for displaying an event of a mobile terminal in cooperation with a user's gaze and an event display method thereof.

The mobile terminal may be configured to perform various functions. Examples of such various functions include a data and voice communication function, a function of photographing a video or a moving image through a camera, a voice storage function, a music file playback function through a speaker system, and an image or video display function. Some mobile terminals include additional functions to execute games, and some other mobile terminals are also implemented as multimedia devices. Moreover, recent mobile terminals can receive a broadcast or multicast signal to view a video or television program.

Further, efforts for supporting and increasing the functions of the mobile terminal continue. Such efforts include not only changes and improvements in structural components that form the mobile terminal, but also improvements in software or hardware. Among them, the touch function of the mobile terminal allows a user who is unfamiliar with the button / key input to conveniently perform the operation of the terminal by using the touch screen. In recent years, not only simple input but also user interface (UI) Is becoming an important function of

Generally, retina, fingerprint, voice, and face have individual characteristics of a living body.

Accordingly, various biometric technologies such as fingerprint recognition, iris recognition, facial recognition, voice recognition, and eye recognition, which are applied to the characteristics of individual living bodies as described above, have recently been installed in mobile terminals.

Among the various biometric technologies, Eye Tracking technology detects eye movements through a front camera and an eye tracking sensor, and then performs a specific operation desired by a user according to the detected movement (eye line) of the eyeball . For example, when the user looks at the screen to use the mobile terminal, the user can automatically enter the sleep mode when the power is turned on or when the user eyes the screen. In addition, the user may blink the eyes or perform the scroll or page turn according to the left and right movements of the eyes.

However, the current eye tracking technology is used only in some mobile terminals, and is used for some limited operation control. Accordingly, it is necessary to implement various operation controls of the mobile terminal using the above-described eye tracking technology and a related user interface in proportion to interest in the current eye tracking technology.

It is an object of the present invention to provide a mobile terminal capable of effectively displaying an event using an eye tracking technique and an event display method thereof.

It is another object of the present invention to provide a mobile terminal and a method of displaying an event of the mobile terminal capable of varying a display position of an event according to an eye position on the screen.

According to another aspect of the present invention, there is provided a method of displaying events in a mobile terminal using eye tracking, the method comprising: displaying content; Detecting a user's gaze on the content; Determining a display position of a content of the event according to a user's gaze detected by the eye tracking sensor when a predetermined event occurs; And displaying a window for displaying the contents of the event at the determined location.

The content includes text, a photograph, a moving picture, a web page, and a menu screen of the mobile terminal.

The event is a notification event indicating the state of the mobile terminal, including an error occurrence, an alarm, a schedule, a missed call, a message reception, and a USB cable connection.

The window for displaying the contents of the event is displayed in the opposite direction to the user's line of sight around the horizontal axis of the center.

The window for displaying the contents of the event is displayed in a transparent state when the user's gaze stays on the content, and is displayed darkly when the user's gaze is moved to the transparent window.

Checking the display cancellation condition of the window and releasing the display of the window when the condition is satisfied.

The display cancellation condition of the window includes completion of confirmation of the event contents, completion of response to the event contents, window closing by the user, and change of the user's gaze.

If there are a plurality of the predetermined events, a window for displaying the contents of each event is displayed according to the user's selection or in the order of occurrence according to the user's line of sight.

According to an aspect of the present invention, there is provided a mobile terminal having an eye tracking function, including: a display unit displaying content; An eye tracking sensor for sensing a user's gaze on the content; And a control unit for determining a display position of the contents of the event according to a line of sight of the user detected by the eye tracking sensor and displaying a window for displaying the contents of the event at the corresponding position.

The content includes text, a photograph, a moving picture, a web page, and a menu screen of the mobile terminal.

The event is a notification event indicating the state of the mobile terminal, including an error occurrence, an alarm, a schedule, a missed call, a message reception, and a USB cable connection.

The control unit displays a window in a direction opposite to the user's line of sight centered on the horizontal axis of the center.

The controller displays the window in a transparent state when the user's gaze stays on the content, and darkly when the user's gaze moves to the transparent window.

The control unit checks the display cancellation condition of the window and cancels the display of the window when the condition is satisfied.

The display cancellation condition of the window includes completion of confirmation of the event contents, completion of response to the event contents, window closing by the user, and change of the user's gaze.

The control unit displays a window for displaying the contents of each event in accordance with the user's selection or in the order of occurrence in accordance with the user's eyes when there are a plurality of predetermined events.

According to an embodiment of the present invention, when a predetermined event is generated while a user is viewing content on a screen, a window is displayed at a corresponding location after determining the location of the window to display the contents of the event according to the user's gaze, By performing the related operation, the user can conveniently confirm the contents of the event by eye tracking or perform an operation related thereto.

1 is a block diagram of a mobile terminal according to an embodiment of the present invention;
2 is a block diagram of a wireless communication system in which a mobile terminal may operate in accordance with an embodiment of the present invention.
FIG. 3A and FIG. 3B are views illustrating an event display method of a mobile terminal according to the present invention, in which one event is displayed using eye tracking; FIG.
Figures 4A-4D illustrate various embodiments for unmarking a message window.
5 is a flowchart illustrating an event display method using eye tracking of a mobile terminal according to an embodiment of the present invention.
6 is a view showing an example of displaying a message informing a reservation function according to a line of sight of a user in the present invention.
7A and 7B illustrate an example of displaying a plurality of events using eye tracking in a mobile terminal according to the present invention.
8 is a flowchart showing an event display method using eye tracking according to an embodiment of the present invention.
FIG. 9 is a flowchart illustrating an event display method using eye tracking when a plurality of events are generated in FIG. 8; FIG.

Hereinafter, a mobile terminal related to the present invention will be described in detail with reference to the drawings. The suffix " module " and " part " for components used in the following description are given merely for convenience of description, and do not give special significance or role in themselves. Accordingly, it should be noted that the above-mentioned " module " and " part "

The terminal may be implemented in various forms. For example, the terminal described in this specification may include a mobile terminal such as a mobile phone, a smart phone, a notebook computer, a digital broadcasting terminal, a PDA (Personal Digital Assistants), a PMP (Portable Multimedia Player) And fixed terminals such as a digital TV, a desktop computer, and the like. In the following description, it is assumed that the terminal is a mobile terminal. However, it will be readily apparent to those skilled in the art that the configuration according to the following description may be applied to the fixed terminal, except for components specifically configured for mobile use.

1 is a block diagram of a mobile terminal according to an embodiment of the present invention.

The mobile terminal 100 includes a wireless communication unit 110, an audio / video input unit 120, a user input unit 130, a sensing unit 140, an output unit 150, a memory 160, A controller 170, a controller 180, a power supply 190, and the like. 1 shows a mobile terminal having various components. However, not all illustrated components are required. A mobile terminal may be implemented by more components than the illustrated components, or a mobile terminal may be implemented by fewer components.

Hereinafter, the components will be described in order.

The wireless communication unit 110 may include one or more components that allow wireless communication between the mobile terminal 100 and the wireless communication system or wireless communication between the mobile terminal 100 and a network on which the mobile terminal 100 is located. For example, the wireless communication unit 110 may include a broadcast receiving module 111, a mobile communication module 112, a wireless Internet module 113, a short range communication module 114, and a location information module 115 .

The broadcast receiving module 111 receives broadcast signals and / or broadcast-related information from an external broadcast management server through a broadcast channel. The broadcast channel may include a satellite channel and a terrestrial channel. The broadcast management server may refer to a server for generating and transmitting broadcast signals and / or broadcast related information, or a server for receiving broadcast signals and / or broadcast related information generated by the broadcast management server and transmitting the generated broadcast signals and / or broadcast related information. The broadcast-related information may refer to a broadcast channel, a broadcast program, or information related to a broadcast service provider. The broadcast signal may include a TV broadcast signal, a radio broadcast signal, a data broadcast signal, and a broadcast signal in which a data broadcast signal is combined with a TV broadcast signal or a radio broadcast signal.

Meanwhile, the broadcast related information may be provided through a mobile communication network, and in this case, it may be received by the mobile communication module 112.

The broadcast-related information may exist in various forms. For example, an EPG (Electronic Program Guide) of DMB (Digital Multimedia Broadcasting) or an ESG (Electronic Service Guide) of Digital Video Broadcast-Handheld (DVB-H).

The broadcast receiving module 111 receives broadcasting signals using various broadcasting systems. In particular, the broadcasting receiving module 111 may be a Digital Multimedia Broadcasting-Terrestrial (DMB-T), a Digital Multimedia Broadcasting-Satellite (DMB-S) Only Digital Broadcast-Handheld (DVB-H), Integrated Services Digital Broadcast-Terrestrial (ISDB-T), and the like. Of course, the broadcast receiving module 111 is configured to be suitable for all broadcasting systems that provide broadcasting signals as well as the digital broadcasting system described above.

The broadcast signal and / or broadcast related information received through the broadcast receiving module 111 may be stored in the memory 160.

In addition, the mobile communication module 112 transmits and receives radio signals to at least one of a base station, an external terminal, and a server on a mobile communication network. Here, the wireless signal may include various types of data depending on a voice call signal, a video call signal, or a text / multimedia message transmission / reception.

The wireless Internet module 113 is a module for wireless Internet access, and the wireless Internet module 113 can be built in or externally. WLAN (Wi-Fi), Wibro (Wireless broadband), Wimax (World Interoperability for Microwave Access), HSDPA (High Speed Downlink Packet Access) and the like can be used as wireless Internet technologies.

The short-range communication module 114 refers to a module for short-range communication. Bluetooth, radio frequency identification (RFID), infrared data association (IrDA), ultra wideband (UWB), ZigBee, and the like can be used as the short distance communication technology.

The location information module 115 is a module for confirming or obtaining the location of the mobile terminal. A typical example of the position information module 115 is a Global Position System (GPS) module. According to the present technology, the GPS module calculates distance information and accurate time information from three or more satellites, and then applies trigonometry to the calculated information to obtain three-dimensional current position information according to latitude, longitude, and altitude It can be calculated accurately. At present, a method of calculating position and time information using three satellites and correcting an error of the calculated position and time information using another satellite is widely used. In addition, the GPS module can calculate speed information by continuously calculating the current position in real time.

An audio / video (A / V) input unit 120 is for inputting an audio signal or a video signal, and may include a camera 121 and a microphone 122. The camera 121 processes an image frame such as a still image or a moving image obtained by the image sensor in the video communication mode or the photographing mode. Then, the processed image frame can be displayed on the display module 151.

The image frame processed by the camera 121 may be stored in the memory 160 or transmitted to the outside through the wireless communication unit 110. [ The camera 121 may be equipped with two or more cameras according to the configuration of the terminal.

The microphone 122 receives an external sound signal through a microphone in a communication mode, a recording mode, a voice recognition mode, or the like, and processes it as electrical voice data. The processed voice data can be converted into a form that can be transmitted to the mobile communication base station through the mobile communication module 112 when the voice data is in the call mode, and output. The microphone 122 may be implemented with various noise reduction algorithms for eliminating noise generated in the process of receiving an external sound signal.

The user input unit 130 generates input data for a user to control the operation of the terminal. The user input unit 130 may include a key pad, a dome switch, a touch pad (static / static), a jog wheel, a jog switch, and the like. Particularly, when the touch pad has a mutual layer structure with the display module 151 described later, it can be called a touch screen.

The sensing unit 140 senses the current state of the mobile terminal 100 such as the open / close state of the mobile terminal 100, the position of the mobile terminal 100, the presence or absence of user contact, the orientation of the mobile terminal, And generates a sensing signal for controlling the operation of the mobile terminal 100. For example, when the mobile terminal 100 is in the form of a slide phone, it is possible to sense whether the slide phone is opened or closed. Also, it is responsible for a sensing function related to whether or not the power supply unit 190 is powered on, whether the interface unit 170 is connected to an external device, and the like. Meanwhile, the sensing unit 140 may include a proximity sensor 141. This will be discussed later in connection with touch screens.

The sensing unit 140 includes a geomagnetic sensor for calculating a moving direction when a user moves, a gyro sensor for calculating a rotating direction, and an acceleration sensor.

The interface unit 170 serves as an interface with all external devices connected to the mobile terminal 100. For example, a wired / wireless headset port, an external charger port, a wired / wireless data port, a memory card port, a port for connecting a device having an identification module, an audio I / O port, Video input / output (I / O) ports, earphone ports, and the like.

Here, the identification module is a chip that stores various information for authenticating the use right of the mobile terminal 100, and includes a user identification module (UIM), a subscriber identity module (SIM) ), A Universal Subscriber Identity Module (" USIM "), and the like. In addition, an apparatus having an identification module (hereinafter referred to as 'identification device') can be manufactured in a smart card format. Accordingly, the identification device can be connected to the terminal 100 through the port. The interface unit 170 receives data from an external device or receives power from the external device to transfer the data to each component in the mobile terminal 100 or to transmit data in the mobile terminal 100 to an external device.

The interface unit 170 may be a path through which power from the cradle is supplied to the mobile terminal 100 when the mobile terminal 100 is connected to an external cradle, Various command signals may be transmitted to the mobile terminal. The various command signals or the power source input from the cradle may be operated as a signal for recognizing that the mobile terminal is correctly mounted on the cradle.

The output unit 150 is for outputting an audio signal, a video signal, or an alarm signal. The output unit 150 may include a display module 151, an audio output module 152, and an alarm unit 153.

The display module 151 displays and outputs information processed by the mobile terminal 100. For example, when the mobile terminal is in the call mode, a UI (User Interface) or a GUI (Graphic User Interface) associated with a call is displayed.

Meanwhile, as described above, when the display module 151 and the touch pad have a mutual layer structure to constitute a touch screen, the display module 151 can be used as an input device in addition to the output device. The display module 151 may be a liquid crystal display, a thin film transistor-liquid crystal display, an organic light-emitting diode, a flexible display, a three-dimensional display 3D display). Some of these displays can be configured to be transparent so that they can be viewed from outside. This may be referred to as a transparent display. A typical example of the transparent display is a transparent organic light emitting diode (TOLED) or the like. In addition, there may be two or more display modules 151 according to the embodiment of the mobile terminal 100. For example, the mobile terminal 100 may be provided with an external display module (not shown) and an internal display module (not shown) at the same time. The touch screen may be configured to detect not only the touch input position and area but also the touch input pressure.

The audio output module 152 outputs audio data received from the wireless communication unit 110 or stored in the memory 160 in a call signal reception mode, a call mode or a recording mode, a voice recognition mode, a broadcast reception mode, The sound output module 152 outputs an acoustic signal related to functions (e.g., call signal reception sound, message reception sound, etc.) performed in the mobile terminal 100. [ The sound output module 152 may include a speaker, a buzzer, and the like.

The alarm unit 153 outputs a signal for notifying the occurrence of an event of the mobile terminal 100. Examples of events that occur in the mobile terminal include call signal reception, message reception, key signal input, touch input, and the like. The alarm unit 153 may output a signal for informing occurrence of an event in a form other than an audio signal or a video signal. For example, it is possible to output a signal in a vibration mode. When a call signal is received or a message is received, the alarm unit 153 can output a vibration to notify it. Alternatively, when the key signal is input, the alarm unit 153 can output the vibration by the feedback to the key signal input. The user can recognize the occurrence of an event through the vibration output as described above. Of course, a signal for notifying the occurrence of an event may also be output through the display module 151 or the sound output module 152.

The memory 160 may store a program for processing and controlling the control unit 180 and may store a function for temporarily storing input / output data (e.g., a phone book, a message, a still image, . In addition, the memory 160 may store data on vibration and sound of various patterns outputted when a touch is input on the touch screen.

The memory 160 may be a flash memory type, a hard disk type, a multimedia card micro type, a card type memory (for example, SD or XD memory), a RAM (Random Access Memory) SRAM (Static Random Access Memory), ROM (Read Only Memory), EEPROM (Electrically Erasable Programmable Read-Only Memory), PROM (Programmable Read- And an optical disc. Also, the mobile terminal 100 may operate a web storage for performing a storage function of the memory 160 on the Internet.

The control unit 180 typically controls the overall operation of the mobile terminal. For example, voice communication, data communication, video communication, and the like. In addition, the control unit 180 may include a multimedia module 181 for multimedia playback. The multimedia module 181 may be implemented in the control unit 180 or may be implemented separately from the control unit 180. [

The controller 180 may perform a pattern recognition process for recognizing handwriting input or drawing input performed on the touch screen as characters and images, respectively.

The power supply unit 190 receives external power and internal power under the control of the control unit 180 as a battery, and supplies power necessary for operation of the respective components.

The various embodiments described herein may be embodied in a recording medium readable by a computer or similar device using, for example, software, hardware, or a combination thereof.

According to a hardware implementation, the embodiments described herein may be implemented as application specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays May be implemented using at least one of processors, controllers, micro-controllers, microprocessors, and electrical units for performing functions. In some cases, And may be implemented by the control unit 180.

According to a software implementation, embodiments such as procedures or functions may be implemented with separate software modules that perform at least one function or operation. The software code may be implemented by a software application written in a suitable programming language. In addition, the software codes may be stored in the memory 160 and executed by the control unit 180. [

The terminal 100 shown in Fig. 1 is configured to be operable in a communication system capable of transmitting data through a frame or packet, including a wired / wireless communication system and a satellite-based communication system .

Hereinafter, a communication system capable of operating a terminal according to the present invention will be described with reference to FIG.

The communication system may use different air interfaces and / or physical layers. For example, wireless interfaces that can be used by communication systems include, but are not limited to, Frequency Division Multiple Access ('FDMA'), Time Division Multiple Access ('TDMA'), Code Division Multiple Access (CDMA), Universal Mobile Telecommunications Systems (UMTS) (especially Long Term Evolution (LTE)), Global System for Mobile Communications (GSM) . Hereinafter, for the sake of convenience of description, the description will be limited to CDMA. However, the present invention is applicable to all communication systems including CDMA wireless communication systems.

2, the CDMA wireless communication system includes a plurality of terminals 100, a plurality of base stations (BSs) 270, and base station controllers (BSCs) 275 , And a mobile switching center ('MSC') 280. MSC 280 is configured to be coupled to a Public Switched Telephone Network (" PSTN ") 290, and is also configured to be coupled to BSCs 275. BSCs 275 may be coupled in pairs with BS 270 through a backhaul line. The backhaul line may be provided according to at least one of E1 / T1, ATM, IP, PPP, Frame Relay, HDSL, ADSL or xDSL. Thus, a plurality of BSCs 275 may be included in the system shown in FIG.

Each BS 270 may comprise at least one sector, and each sector may comprise an omnidirectional antenna or an antenna pointing to a particular direction of radial from BS 270. [ In addition, each sector may include two or more antennas of various types. Each BS 270 may be configured to support a plurality of frequency assignments and each of the plurality of frequency assignments has a specific spectrum (e.g., 1.25 MHz, 5 MHz, etc.).

The intersection of sector and frequency assignment may be referred to as a CDMA channel. BS 270 may be referred to as a Base Station Transceiver Subsystem (BTSs). In this case, the word " base station " may be referred to as a combination of one BSC 275 and at least one BS 270. [ The base station may also indicate a " cell site ". Alternatively, each of the plurality of sectors for a particular BS 270 may be referred to as a plurality of cell sites.

2, a broadcasting transmitter (BT) 295 transmits a broadcasting signal to the terminals 100 operating in the system. The broadcast receiving module 111 shown in FIG. 1 is provided in the terminal 100 to receive a broadcast signal transmitted by the BT 295.

In addition, FIG. 2 illustrates several Global Positioning System ('GPS') satellites 300. The satellites 300 help to locate at least one of the plurality of terminals 100. Although two satellites are shown in FIG. 2, useful location information may be obtained by two or more satellites. The location information module 115 shown in FIG. 1 cooperates with satellites 300 to obtain desired location information. Here, you can track your location using all of the technologies that can track your location, as well as your GPS tracking technology. Also, at least one of the GPS satellites 300 may optionally or additionally be responsible for satellite DMB transmission.

Among the typical operations of a wireless communication system, the BS 270 receives a reverse link signal from the various terminals 100. At this time, the terminals 100 are connecting a call, transmitting or receiving a message, or performing another communication operation. Each of the reverse link signals received by the particular base station 270 is processed by the particular base station 270. The data resulting from the processing is transmitted to the connected BSC 275. The BSC 275 provides call resource allocation and mobility management functions, including the organization of soft handoffs between the base stations 270. The BSC 275 also transmits the received data to the MSC 280 and the MSC 280 provides additional transmission services for connection with the PSTN 290. [ Similarly, the PSTN 290 connects to the MSC 280, the MSC 280 connects to the BSCs 275, and the BSCs 275 communicate with the BSs 270 ).

The control unit 180 of the mobile terminal frequently reports that a state of the mobile terminal has occurred (e.g., an error occurs, a specific event (e.g., an alarm, a schedule, a missed call, a message, a USB cable connection or the like) . The notification is for the user to receive questions and work orders for the next operation in addition to informing the user of the current state.

The notification is a kind of event generated in the mobile terminal and is displayed in the indication area in the indication area where the current time, remaining battery level, network status, and the like are displayed. The notification icon continues to be displayed until the user confirms it. However, in order to view detailed information of the notification icon, it has been troublesome to expand the indication area by dragging it down.

When an event occurs in the mobile terminal, the user may be viewing the predetermined content on the screen or may be executing the application. If a user views a predetermined content on the screen or displays a menu in order to check a message while the application is running, or if a message indicating the occurrence of an event is displayed in the center of the screen, the content is displayed or the execution of the application is interrupted There are disadvantages.

Accordingly, the present invention uses an eye tracking function to vary an indication position of an event generated in a mobile terminal according to a position of an eye gazing at a screen, that is, a change in a line of sight of a user, We propose a method to check the contents of the event conveniently while viewing the contents and perform additional operations related to the contents.

The eye tracking according to the present invention is performed by a front camera and an eye tracking sensor. The eye tracking sensor is provided in the sensing unit 140 of FIG. 1 to detect a position of an eye gazing at a screen, Eye line). The detected movement of eyes (eye line) is input to the control unit 180 and used for various control operations of the mobile terminal.

The event generated in the mobile terminal may be generated in the mobile terminal such as a text message reception, an error occurrence message of the mobile terminal, a reservation function notification (e.g., an alarm, a schedule) and a power level indication Includes all events. In particular, the event is an event that is displayed to a user when it occurs.

The content includes a text (e.g., a text message, an e-book), a photograph, a video (movie, music, game), a web page and a menu screen of a mobile terminal (e.g., a phone book).

Such additional operations include storage, transmission, menu retrieval, and the like.

The present invention can temporarily stop the display or playback of the content when the content displayed on the current screen is a moving image when the event generated in the mobile terminal is displayed at an appropriate position on the screen in accordance with the change of the user's gaze.

In addition, the present invention can use the eye tracking function to 'blink' eye for 'select' and 'clear' for a specific function. The 'select' and 'deselect' are determined by the number of blinks of the eye.

Message window occurrence and display

FIGS. 3A and 3B show an example of a method of displaying an event of a mobile terminal according to the present invention, in which an event is processed according to a user's gaze when one event is generated. 3A and 3B show an example of displaying a received text message.

As shown in FIG. 3A, when a message (e.g., a kakao chat message) is received while the user is reading the text displayed on the screen, a notification icon, that is, a message icon 50, is displayed in the indication area. The text is assumed to be an article written on a specific website, for example.

When the message icon 50 is displayed, the eye tracking sensor senses a position on the screen on which the user's eyes are directed, and the controller 180 displays the message window 51 according to the user's gaze detected by the eye tracking sensor. As shown in FIG.

The display position of the message window 51 is determined in a direction opposite to the user's line of sight centering on the horizontal axis of the center of the screen. The message window 51 is displayed at a position distant from the user's line of sight. For example, if the user's line of sight is located at the top of the text, the message window 51 is displayed at the bottom.

When the display position (point) of the message window 51 is determined, the controller 180 displays a message window 51 of a predetermined size at the corresponding position, and when the user's eyes are moved to the transparent message window 51 (If the user views the transparent message window), the message window 51 is displayed in dark.

In addition, the control unit 180 emphatically displays the certain portion 52 of the text that the user has read, distinguishing it from other portions so that the user can easily find the portion of the text 52 that is being read again. The predetermined portion specifies a range of text and can be arbitrarily set in a menu, and the highlighting includes highlight, color indication, and bold. The highlighting 52 is performed when the user's gaze is changed. The size and position of the message window 51 may be adjusted by a touch & drag operation.

Accordingly, the user can confirm the contents of the message displayed in the message window 51, and can confirm the message reception history or create the response message by selecting the message window 51. [ The control unit 180 regards the message window 51 as being selected when the user directly touches the message window 51 or when the eye tracking sensor detects blinking of the eyes, And related menus (modification, deletion, transmission).

3B is an example of displaying a message window 51 at the upper part of the center of the screen when a message is received while the user is reading the lower part of the text and the line of sight is positioned at the lower end of the screen. The operation of FIG. 3B is the same as that of FIG. 3A, except that the message window 51 is displayed at the top of the screen.

Dismiss message window

4A to 4D are various embodiments for canceling the display of the message window 51. Fig.

In one embodiment, the display of the message window 51 displayed on the screen is released upon receipt of a received message acknowledgment or an acknowledgment message. As shown in FIG. 4A, when the user B confirms that the user B has read the message from the user A based on the eye line sensed by the eye tracking sensor, or when the user B transmits a response message, The control unit 180 cancels (closes) the display of the message window 51 currently displayed on the screen. At this time, the emphasis display 52 is automatically released when the user's gaze is moved to a previously read portion.

In another embodiment, the user can close the message window 51 with his / her finger. 4B, if the user touches the upper end of the message window 51 and drags the finger downward, the control unit 180 cancels the display of the message window 51 currently displayed on the screen do.

As another embodiment, the display of the message window 51 can be released according to the change of the user's line of sight. 4C, when the user's gaze is moved from the message window 51 to the highlight 52, the control unit 180 releases the highlight 52 and displays a message window 51 ) Is also released.

In another embodiment, the display of the message window 51 may be canceled according to the retention time of the line of sight. As shown in FIG. 4D, when the user looks at another place (except for the low tide display part) or looks outside the screen in a state where the user's eyes are staying in the message window 51, The display of the message window 51 displayed on the current screen is canceled if it continues for a predetermined time (critical time).

In each of the above embodiments, the display of the message window 51 may be released immediately, or may be released after being transparent. 4A to 4D can be set as a message window display cancel condition.

As described above, when a predetermined event is generated while the user is viewing predetermined content displayed on the screen, the event is displayed in a direction opposite to the direction of the user's eyes and then processed according to the line of sight, Or perform an operation associated therewith.

FIG. 5 illustrates an example of displaying a received text message while viewing a photograph, as an example of a method of displaying an event of a mobile terminal according to an exemplary embodiment of the present invention.

5, when a text message is received while the user is viewing a picture, the controller 180 performs the same method as in FIG. 3A, and when the user's eyes are not moved first, That is, the message window 51 is displayed transparently at the bottom of the screen, and when the user's gaze is moved to the transparent message window 51, the transparent message window 51 is displayed in dark and then the text message is displayed. When the user's gaze is moved to the message window 51, the previously viewed photo is highlighted 52 to allow the user to view it.

In FIG. 5, a photograph is described as an example, but the same applies to the case where different contents are displayed on the screen. The content includes a text (e.g., a text message, an e-book), a photograph, a video (movie, music, game), a web page and a menu screen of a mobile terminal (e.g., a phone book).

In particular, if a message is received when a moving picture is displayed on the screen, the control unit 180 displays the message window 51 in the opposite direction to the line of sight. If the moving picture includes the caption, the message window 51 is displayed Display. In another embodiment, when a message is received upon playback of a moving image, the control unit 180 may suspend the playback of the moving image and display the message window 51 in the opposite direction to the line of sight. It can also be resumed.

The control unit 180 of the mobile terminal frequently reports that a state of the mobile terminal has occurred (e.g., an error occurs, a specific event (e.g., an alarm, a schedule, a missed call, a message, a USB cable connection or the like) . The notification is for the user to receive questions and work orders for the next operation in addition to informing the user of the current state.

The notification is a kind of event generated in the mobile terminal and is displayed in the indication area in the indication area where the current time, remaining battery level, network status, and the like are displayed. The notification icon continues to be displayed until the user confirms it. However, in order to view detailed information of the notification icon, it has been troublesome to expand the indication area by dragging it down.

Therefore, the display of the notification message can be displayed and confirmed by the method described in FIGS. 3A and 3B. In particular, when a notification message is displayed to a user at a specific time such as a reservation function (alarm, schedule) Do not display it in the center, but display it in the direction opposite to the user's gaze as described above.

6 is an example of displaying a message informing the reservation function according to the user's gaze in the present invention.

As shown in FIG. 6, when the user is reading the text, the control unit 180 checks the user's current gaze on the current screen of the user and displays the notification window 53 in the direction opposite to the user's gaze , And the user is presented with the message "There is a meeting at 5 o'clock" on the schedule content.

7A and 7B illustrate another example of a method of displaying an event of a mobile terminal according to the present invention, in which an event is processed according to a user's gaze when two or more events are generated.

As shown in FIG. 7A, when two or more events occur while the user is reading the text displayed on the screen, a notification icon corresponding to the event is displayed in the indication area. For convenience of explanation, the present invention assumes that a message and a general message are received via an instant messenger (e.g., KakaoTalk), and the text is an article written on a specific web site.

When the KakaoTalk message and the general message are received, an icon 50 indicating a KakaoTalk message and an icon 54 indicating a general message are displayed in the indication area.

When the two message icons 50 and 54 are displayed, the eye tracking sensor senses a position (line of sight) on the screen of the user's eyes, and the controller 180 controls the direction of the user's eyes That is, the message icons 50 and 54 are enlarged and displayed at the lower end of the screen.

In this state, when the user changes the line of sight and looks at the message icon 50, the controller 180 displays a highlight 52 on a predetermined portion of the previously viewed content. The control unit 180 regards the message icon 50 as being selected and displays a message window 51 at the lower end of the screen to display a message icon 50 ) Is displayed. In another embodiment, the message icon 50 may be selected by a user's direct touch or blinking in the viewing state.

The user can confirm the contents of the message displayed in the message window 51 and select the message window 51 to confirm the message reception history or to create a response message. When the user confirms the contents of the message in the message window 51 or completes the transmission of the response message, the control unit 180 cancels the display of the message window 51. [

When the display of the message window 51 is released, only the unselected message icon 54 is displayed on the screen as shown in FIG. 7B. The control unit 180 regards the corresponding message icon 54 as being selected and displays another message window 51 at the lower end of the screen to display a message icon 54). In another embodiment, the message icon 54 may be selected by a user's direct touch or blinking in a viewing state.

After confirming the contents of the message displayed in the message window 51, the user releases the display of the message window 51 according to the method described in Figs. 4A to 4D.

As another embodiment, the present invention may display the message window 51 according to the line of sight of the user in the order generated when two or more events are generated. In this case, the generation and release of the message window 51 are the same as those described in Figs. 4A to 4D and 7A, and therefore, detailed description thereof will be omitted.

8 is a flowchart illustrating an event display method using eye tracking according to an embodiment of the present invention.

As shown in FIG. 8, the user selects the eye tracking function in the menu, and then displays the content to be viewed on the screen (S10, S11). Since the eye tracking function may be selected after the contents are displayed on the screen, the steps (S10) and (S11) The execution order may be changed.

When the eye tracking function is activated, the eye tracking sensor continuously senses a position (line of sight) on the content screen on which the user's eyes are directed, and the controller 180 controls the eyeball of the content displayed on the screen according to the line of sight detected by the tracking sensor And controls an operation (e.g., page turning) (S12).

If an event is generated while the user is viewing the content displayed on the screen (S13), the controller 180 determines whether the event is one or more (S14). The generated event is an event generated at the same time or within the time when the control unit 180 checks the occurrence of an event.

If there is one event generated as a result of the check, the control unit 180 determines the display position of the window for displaying the contents of the event based on the sight line of the user detected by the tracking sensor, and displays the contents of the event immediately at the determined position (S15). ≪ / RTI > If the event is a message reception event, the window is a message window.

The display position of the window 51 is determined in the direction opposite to the user's line of sight about the horizontal axis of the center of the screen. The window 51 is displayed in a position far away from the user's line of sight. For example, if the user's line of sight is located at the top of the text, the window 51 is displayed at the bottom.

When the display position (point) of the window 51 is determined, the controller 180 displays the window 51 of a predetermined size at the corresponding position transparently. When the user's eyes are moved to the transparent window 51 So that the window 51 is displayed in a dark state.

Thereafter, the control unit 180 checks whether the window display cancellation condition is satisfied (S16). As shown in FIGS. 4A to 4D, when the user confirms the contents displayed in the window 51, After shifting the visual line, the control unit 180 regards the window display cancellation condition as satisfied and releases the display of the window 51 (S17).

9 is a flowchart illustrating an event display method using eye tracking when a plurality of events are generated in FIG.

In FIG. 8, if a plurality of events are generated as a result of the check in step S14, the controller 180 displays a mode event icon on the content screen. When a specific event icon is selected by the user, the control unit 180 displays a window 51 for displaying the selected event icon in a direction opposite to the sight line of the detected user, as shown in FIG.

The control unit 180 checks whether the window display cancellation condition is satisfied (S21). If the user confirms the contents displayed in the window 51 as shown in FIGS. 4A to 4D, The control unit 180 regards the window display cancellation condition as satisfied and displays the remaining event icons that are not selected on the contents screen (S22) (Fig. 7B).

If the unselected event icon is not the last event icon, the controller 180 performs the operation after step S18 to receive the user selection. If the unselected event icon is the last event icon, (S23, S24) again to display the selected event icon or the last icon in the direction opposite to the direction of the icon.

If the window display cancellation condition is satisfied, the controller 180 cancels the window 51 to display the selected event icon or the last icon and returns to the original content screen (S25, S26).

As described above, according to the present invention, when a predetermined event occurs in a state where a user is viewing contents on a screen, a position of a window to display contents of the event is determined according to a user's line of sight, The user can easily confirm the contents of the event by eye tracking or perform an operation related thereto.

Further, according to an embodiment of the present invention, the above-described method can be implemented as computer-readable code on a medium on which a program is recorded. The computer readable medium includes all kinds of recording devices in which data that can be read by a computer system is stored. Examples of the computer-readable medium include ROM, RAM, CD-ROM, magnetic tape, floppy disk, optical data storage, and the like, and also implemented in the form of a carrier wave (for example, transmission over the Internet) . Further, the computer may include a control unit of the terminal.

The above-described mobile terminal and its event display method are not limited to the configuration and method of the above-described embodiments, but the embodiments may be modified so that all or some of the embodiments are selectively As shown in FIG.

110: wireless communication unit 111: broadcast receiving module
112: mobile communication module 113: wireless internet module
140: sensing unit 150: output unit
151: display unit 160: memory
180:

Claims (16)

delete delete delete delete delete delete delete delete A display unit for displaying contents;
An eye tracking sensor for sensing a user's gaze staying on the content; And
(a) when a predetermined event occurs, determining a display position of an event content so that the direction is different from a line of sight of the user detected by the eye tracking sensor,
(b) a controller for displaying the event content at the determined display position,
Wherein,
Wherein the display unit displays the window displaying the event content in a transparent state when the user's eyesight stays in the content and darkens the window when the user's eyesight is moved to the transparent window. 10. The method of claim 9,
And a menu screen of text, a photograph, a moving picture, a web page, and a mobile terminal. 10. The method of claim 9,
Wherein the mobile terminal includes an error occurrence, an alarm, a schedule, a missed call, a message reception, and a USB cable connection as a notification event indicating the state of the mobile terminal. 10. The apparatus of claim 9, wherein the control unit
And displays the event contents in a direction opposite to a direction of the user's eyes, with the horizontal axis of the center being the center. delete 10. The apparatus of claim 9, wherein the control unit
And the display state of the window is checked. When the condition is satisfied, the display of the window is released. 10. The method according to claim 9,
The completion of the event content confirmation, the completion of the response to the event content, the window closing by the user, and the change of the user's gaze. 10. The apparatus of claim 9, wherein the control unit
And displays an icon of each event in a direction different from a direction of the user's eyes in the order of occurrence when a plurality of predetermined events occur, and executes the corresponding event when the user's eyes move to a predetermined icon.

Images