JPH11249772A - Virtual environment presentation device - Google Patents

Abstract

(57) [Summary] [Problem] To provide an environment in which the user can receive support through video while continuing to perform and work in a place of practice. SOLUTION: A liquid crystal goggle 12, a small video camera 1
4. A headset 10 integrating a headphone 16 and a microphone 18 is wirelessly connected to a computer main body (clant terminal) A, and the computer main body A is connected to a file server C of a WWW server B via the Internet. By inputting a code from the microphone 18 to the voice recognition module 30 as a voice, the image processing unit 38 and the voice synthesis module 3 transmit the video and voice information relating to the model performance and instructions from the file server C.
4, and outputs a video of the model performance to the liquid crystal goggles 12, and outputs voice instructions, sound effects, and the like from the headphones 16.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to the acquisition of various acts and techniques mainly performed by humans, such as medical treatment (emergency treatment, surgery, massage, nursing care, etc.), dance and gymnastics, and manuals. The present invention relates to a virtual environment presentation device suitable for learning practical skills that can be learned by the body at a place of practice that cannot be brought in, for example, at an educational site, or for supporting maintenance work of equipment requiring many drawings.

[0002]

2. Description of the Related Art When learning various acts, techniques, or practical skills (hereinafter, also simply referred to as acting) mainly performed by humans, such as medical practice, there is no room to refer to a manual in a practice place. Is one of the most efficient and essential.

[0003] That said, not everyone can always receive one-on-one education. Therefore, besides this one-on-one education, printed materials and videos are still used as instruction manuals and manuals, and furthermore, it is possible to learn using CD-ROMs and, more recently, the media of the Internet WWW (World Wide Web). I have.

[0004]

However, it is difficult to bring these manuals to the field of practice,
Even if they are brought in, it is difficult to refer to the manual at the same time while learning the performance, and there are various problems depending on the medium as follows.

[0005] (1) Printed matter: Although it has good portability, it cannot be referred to during performances or turned pages.

(2) Video: Although there is an advantage that the user can simultaneously perform while watching from a distance without operation, there is an advantage that it is not possible to go back, change the speed, or change the angle. Can not meet the need to see the spot in the blind spot. Also, when the model performance and the user face in opposite directions, the procedure must be constantly translated in the head in the opposite direction, which may be difficult for a beginner.

(3) CD-ROM: Although the problem of video is solved, mouse operation is required.

(4) WWW: There is an advantage that the contents which can be provided are expanded as compared with a CD-ROM, but similarly, mouse operation is required, and portability such as line connection is inferior.

SUMMARY OF THE INVENTION The present invention has been made to solve the above-mentioned conventional problems, and an environment in which it is possible to receive support through an online manual while continuing to perform and work in a practice place, that is, a walking dictionary environment. The task is to provide

[0010]

SUMMARY OF THE INVENTION The present invention relates to a virtual environment presentation apparatus, in which multimedia information can be presented and controlled by image recognition means and voice recognition means, and a hands-free apparatus for the apparatus. Remote interactive operation by, and enabled see-through viewing for information provided by the device, by having at least a video input means, a video output means, a voice input means, a user interface device having a voice output means,
This has solved the above-mentioned problem.

That is, according to the present invention, the video input means of the user interface device enables the user to present manual information supporting the performance and the like to the user as a video, and outputs the video to the audio output means or the video output means. For example, the user can switch to the desired information content in a hands-free manner, that is, hands-free with the user's voice or line of sight, for example, so that the user can continue performing The user can receive desired support information for assisting the user.

[0012]

Embodiments of the present invention will be described below in detail with reference to the drawings.

FIG. 1 is a block diagram conceptually showing an emergency medical support system (virtual environment presentation device) according to an embodiment of the present invention, and FIG. 2 is a block diagram more specifically showing the configuration of the system. It is. FIG. 3 is a front view showing a headset (user interface device) constituting the support system of the present embodiment together with a user wearing the headset, and FIG. 4 is a plan view thereof.

The emergency medical support system according to the present embodiment is intended for a home that realizes a virtual environment that enables hands-free learning of acting while viewing a model video provided as a manual on an electronic medium from a remote position. System.

The support system according to this embodiment is the same as that shown in FIG.
As shown in FIG. 2, a personal computer main body (information presenting device, client terminal) A capable of presenting and controlling multimedia information including a manual image for emergency medical support, and A headset (user interface device) 10 connected wirelessly.

In this embodiment, the multimedia information is provided from the file server (multimedia server) C of the WWW server B to the computer main unit A via the Internet.

The HTML (Hyper Text M) shown in FIG.
arkup Language) is a language used to create Internet homepages, and Java is a programming language developed by Sun Microsystems. Programs (applets) created in this language are executed when displaying Internet homepages. You. Also, VRML
(Virtual Reality Modeling Language) is a language capable of expressing three-dimensional graphics on the Internet, and is based on MIDI (Musical Instruments Digital).
(Interface) is a standard for exchanging music data with a computer.

The WWW is currently the most promising medium because it can provide abundant contents online and can technically provide functions such as backtracking, speed change, and angle change.

In this embodiment, the following five functions required as a user interface are realized so that the user can refer to the electronic manual while learning the performance and the like.

(1) A computer operation can be performed without using both hands (hands-free operation).

(2) Being able to move to the extent that it does not hinder the performance (portability and light weight).

(3) A model performance (virtual image) and a real image can be viewed simultaneously (see-through).

(4) The angle of the model performance should be as close as possible to the user's viewpoint (at least not in the opposite direction).

(5) The speed of the model performance is variable according to the skill of the user, and can be changed during the performance.

In the present embodiment, the above functions are realized by the following system configuration.

That is, the headset 10 is moved to the position shown in FIGS.
As shown in the figure, the liquid crystal goggles 12, the small video camera 1
4. a headphone 16 comprising left and right ear pads, a hands-free microphone 18 movable in the direction of the arrow,
A penlight 20 fixed on the liquid crystal goggles 12 and a telescopic headband 10A are integrated.

The liquid crystal goggles 12 are of a see-through type having an optical see-through function using a translucent liquid crystal display. These goggles 12 and headphones 1
Numerals 6 are connected to receivers (video receiving means, audio receiving means) 22 which are fixed to the user's waist by a belt via a goggle video cable 12C and a headphone audio cable 16C, respectively. And receiver 2
2 has a built-in battery, and in addition to supplying power to the video receiving means and the audio receiving means,
2 is supplied with necessary power via a goggle video cable 12C.

The small video camera 14 and the microphone 18 are connected to a transmitter (video transmission means, audio transmission means) 24 via a video camera cable 14C and a microphone audio cable 18C, respectively. The transmitter 24 also has a built-in battery. In addition to supplying power to the video transmission means and the audio transmission means, the video camera cable 1 is connected to the video camera 14.
The necessary power is supplied to the microphone 18 via the microphone audio cable 18C while supplying necessary power via the 4C.

In this embodiment, the computer main unit A (client terminal) comprises a voice recognition module 30, a status control unit 32, a voice synthesis module (or MIDI sound source) 34, a playback speed control as shown in FIG. An image processing unit 38 for performing rendering, video decoding / reproduction, and synthesis with a real background image, and an image recognition module 40 having a camera parameter extracting function.

The user attaches the headset 10 to his / her head, so that the user can view the manual provided on an electronic medium while performing the performance. The computer main body A is connected via a wireless analog line via the receiver 22 and the transmitter 24.

That is, to the liquid crystal goggles 12, a video signal transmitted from the image processing section 38 via the receiver 22 by infrared rays is input, and to the headphones 16, an audio signal is input from the voice synthesis module 34 in the same manner. It is supposed to be. On the other hand, a video signal transmitted by a radio wave in the VHF band is output from the video camera 14 to the image processing unit 38 and the image recognition module 40 via the transmitter 24, and the microphone 18 outputs the voice recognition module 3.
The audio signal is output in the same manner as 0.

As described above, here, the headset 10
In realizing two-way wireless transmission of video and audio between the computer and the computer main unit A, in order to avoid interference between the two, measures are being taken to share currently available two types of transmission devices in the infrared and VHF bands. .

The computer main unit A is connected to the WWW server B via the Internet digital line, and the status control unit 32 outputs the performance status to the file server C, and the image recognition module 40 Output viewpoint information, and conversely, file server C
From the voice information is input to the voice synthesis module 34,
Also, video information is input to the image processing unit 38.

In this embodiment, the function (1) is realized by operating a computer by voice input / recognition. Specifically, the microphone 18
Then, the voice of the user is input, and the voice recognition module 30 of the computer main body A recognizes the character string uttered by the user from the voice, and the status control unit 32 uses the file server C based on the character string. To the computer main unit A from the file server C, and the liquid crystal goggles 1
2. It is possible to switch the contents of the video / audio multimedia information provided to the headphones 16.

In realizing this function, HT
As for the ML anchor (underlined character string: a function of jumping to a predetermined page when clicked), a control program for jumping to a designated destination based on a character code obtained by voice recognition is added to an HTML sentence. The method was adopted.

FIG. 5 is a block diagram conceptually showing this. When the computer main unit A is connected to the server via the Internet, the main management program main unit described in the Java applet is started by the main management program starting HTML statement. Thereafter, by performing a key input to the menu button of the main body with a mouse or the like, the multimedia display program main body also described in the Java applet can be started by an HTML sentence for starting the multimedia display program.
Then, by inputting a key to the menu button of the display program body in this state, desired file information can be read from the file server C. Thus, at present, W can only be entered with a mouse click.
Where the menu for information transmission cannot be switched with the WW server B, it can be realized by key input.

Then, the voice command is transmitted to the microphone 1
When the input is made to the voice recognition module 30 from step 8, the module 30 converts the character code into a character code corresponding to a predetermined menu button, and executes an input process equivalent to a key input of the character code. That is, a desired anchor function for jumping to another HTML page by the voice command is realized.

In order to realize the function (2), as shown in FIGS. 3 and 4, the headset 10 having the liquid crystal goggles 12, the small video camera 14, the headphones 16 and the microphone 18 is provided to the user. The headset 10 and the computer main unit A which is a WWW terminal
The wireless connection between the headset 10 and the WWW server B is realized as a result.

In order to realize the function (3), an image processing section 38 converts a real image input by the video camera 14 and a model image provided by the computer main unit A by CG (computer graphics). Synthesized with
The composite image is provided to the liquid crystal goggles 12 with the see-through function closed. By doing so, it is easy to make the background part completely solid in the CG image. Therefore, if the real image can be embedded in the background part using a so-called chroma key synthesis method, both qualities can be improved. Because it can be prevented from deteriorating,
A clear model image can be provided. However, the model video becomes unclear, but the see-through function can be used as it is.

In order to realize the function (4), for example, four types of silhouettes of a subject (for example, a practitioner) in a line-of-sight image captured by the user with the video camera 14 as shown in FIG. May be automatically selected.

The video camera 14 inputs a visual field image of the user, and the image recognition module 10 of the computer main body A recognizes the user's line of sight based on the visual field image, and provides an image to be provided to the liquid crystal goggles 12. May be controlled so as to approach the angle of view.

More specifically, only the direction of the face of the subject is recognized from the user's visual field image by the moment method of image processing, and an image of the angle closest to the direction is provided. For this purpose, a model video is produced by three-dimensional CG, and a video rendered at several angles per scene is prepared, or a real video taken from several directions is prepared. It is difficult to recognize an accurate viewpoint with the current image processing. However, in the case of the three-dimensional CG method, there is a problem that the calculation takes a long time. For example, a method of rendering in real time may be adopted, such as by mounting an image, so that the angle can be arbitrarily changed. In addition, there is another method that is not suitable for this application because the movement range is largely restricted, but a position sensor is attached to goggles to completely follow the viewpoint.

In this embodiment, the user operates the liquid crystal goggles 1
Although the camera work of the image that can be seen through 2 is only the case of the CG image, the operator is displayed in a skeleton (skeleton) so that the operation hidden behind the operator can be seen through. There is also a function to display the operative subject (patient) in skeleton so that the operation can be seen anatomically. A specific example will be described later.

Similarly, only in the case of a CG image,
When multiple parts (head, hand, etc.) are moving in harmony,
You can stop and play the operation of all parts other than the selected part, or stop and play the operation of the selected part,
There is also a single part operation display function.

Further, when reproducing the exemplary video presented on the liquid crystal goggles 12, there is a function of expanding the time axis with respect to the designated area.

A moving image reproduced basically displays a time-varying motion as a continuous still image. Therefore, each of the continuously changing still images is stored in the database as frames 1 to 7 as shown in FIG.
.. Are stored as image data.

However, since only one occurrence occurs in the model performance, the start or end of a repetitive performance that is easily overlooked, and a skilled person is smoothly coupled to the repetitive performance, is easily overlooked, and is sometimes simplified or omitted. Figure 7 (A)
There is a case where the slow reproduction simply reducing the reproduction speed of the frame cannot respond. As shown in FIG. 3B, such a video is subdivided into two frames, as shown in FIG. .. Are prepared separately.

In this embodiment, in order to realize the function (5), the microphone 18 inputs a user's voice, and the voice is used by the voice recognition module 30 of the computer main unit A from the voice. A plurality of character strings uttered by the user are recognized, and based on the time interval between the character strings, the playback speed control unit 36 sends the signals from the speech synthesis module 34 or the image processing unit 38 of the computer A to the headphone 16 or The playback speed of time-series multimedia information such as support video and audio provided to the liquid crystal goggles 12 is controlled. Note that the character string referred to here also includes one character such as “A”, “I”, “U”, and the like.

That is, the frame rate for reproducing a moving image is made variable, and the BGM sound is described in MIDI. When the user speaks, for example, “1, 2, 3” as a plurality of character strings “1, 1, 2, 3” using the microphone 18 at predetermined intervals, the pronunciation of “1” and “3” starts. A time interval is measured, and control can be performed such that 30 frames of a moving image and 8 quarter notes are reproduced at the interval.

FIG. 8 conceptually shows this function. In this figure, the utterance of “1, 2, 3” is replaced with pulses 1 to 3 of the uppermost user input pulse, and the pulse interval of the system clock changes at the timing of pulse 3. I have. In synchronization with this, the interval at which the moving image overwrites the already-made frame data is controlled, and C
In the G image, the interval at which the frame drawn in the background is switched to the front is controlled, and in the MIDI, the interval of playing strong beats is controlled. Note that MIDI
In the performance interval control, the sound intensity can be modulated by the fluctuation component of the pulse amplitude.

An external synchronizing signal can also be input to the reproduction speed control unit 36, and a biological signal such as a heart rate of a user or a person facing the user is input to the speed control unit 36. The reproduction speed of the time-series multimedia information can be similarly controlled based on the biological signal.

Further, as shown in FIG. 1, the support system is also provided with a CRT monitor so that other people can see the image that the user is watching through the goggles.

Next, a method of using the emergency medical support system of the present embodiment will be described.

In recent years, with the aging society, an increase in lifestyle-related diseases (old adult diseases) cannot be avoided anymore. Among these lifestyle-related diseases, cardiovascular diseases such as angina pectoris and myocardial infarction
There is the danger of fighting for a moment due to a seizure. In such emergency care, lifesaving and first aid measures before being transported to the hospital are very important, and in an environment where there are no doctors around, no image inspection equipment, no therapeutic drugs,
The appropriate medical practices of the people around them will determine the fate of the patient. The support system according to the present embodiment exhibits particularly excellent effects in such an environment.

First, as shown in FIG. 3, the headset 10, which is a user interface device, is attached to the head of the user.

Next, the user inputs a code to the computer main unit A by key input or a voice command through the microphone 18, and connects the main unit A to the WWW server B.
And the file server C is set up so that information such as video and audio related to the model performance can be input.

Then, according to the main management program shown in FIG. 5, each of the still images (1) to (10) shown in the outline of the terminal control program in FIG. The goggles 12 can be sequentially displayed.

FIG. 10 shows the menu screen of the CRT monitor for convenience, but the liquid crystal goggles 12 displays the image displayed in the window of this screen (the image of (6) shown in FIG. 9 here). Is displayed, and instructions and questions as in the screen are transmitted to the user through the headphones 16.

The user operates in accordance with the instruction and answers the question with “yes” or “no” through the microphone 18 so that the menu images shown in FIG. 9 can be sequentially updated. .

Here, for convenience, it is assumed that the user has selected the heart massage mode (9) from the menu of FIG. When the user inputs a command indicating “video” shown on the screen of FIG. 10 through the microphone 18, the multimedia display program shown in FIG. 5 starts, and the necessary video and audio information is transmitted from the file server C. Is loaded, and an exemplary image of the heart massage is displayed on the liquid crystal goggles 12 as a moving image.

FIG. 11 shows one frame of the model image, that is, a frame in which the subject (patient) is displayed as a skeleton as described above, on the same monitor screen as in FIG. Actually, the user selects an operation menu displayed in characters on this screen by inputting a character code by voice, displays a support image in a desired state, and looks at the model performance while watching the model performance. You will learn how to massage.

Incidentally, the operation menu includes the speed (S
Peed), playback and stop can be selected by voice input, and four images of front, top, side, and enlargement corresponding to the camera work shown in FIG. You can select by angle or voice input.

Although illustration of a continuous frame image corresponding to the moving image image of the heart massage is omitted, another image display function of the emergency medical support system of the present embodiment will be described.

FIG. 12 shows one image of a scene in which a practitioner covers a patient who is lying on his back in the lateral direction and performs a heart massage with the operator covering the chest. The condition above the chest is difficult to understand. Therefore, as shown in FIG. 13 corresponding to this figure, the practitioner can be displayed as a skeleton.

FIG. 14 is a front view of a scene in which the practitioner is placing his hand on the patient's chest. This image shows the position of the practitioner's hand with respect to the patient's heart. It is unknown whether or not they are involved. Therefore, as shown in FIG. 15, the patient is displayed as a skeleton so that the relationship between the heart inside the ribs and the position of the operator's hand can be understood.

Such a skeleton display can be dealt with by, for example, separately preparing in advance a database in addition to the moving image frame by the four camera works, especially for a video that is a point in performance.

As described in detail above, according to the emergency medical support system of the present embodiment, when the user performs emergency medical treatment or practice, the user can operate the system without using both hands, and can perform the performance. You can move to the extent that there is no trouble,
The model image and the real image can be viewed at the same time, and the user can view the image in the direction desired by the user, and the speed of the model image can be changed during the performance according to the skill of the user. Therefore, even at home, it is possible to easily learn how to give emergency medical treatment.

Although the present invention has been specifically described above, the present invention is not limited to the above-described embodiments, and can be variously modified without departing from the gist thereof.

For example, in the above embodiment, the case where the virtual environment display device is an emergency medical support system has been described, but the present invention is not limited to this. Further, the specific external shape and configuration of the headset are not similarly limited.

In the above-described embodiment, the case where the WWW server is used as the server for the multi-information has been described. However, other networks may be used, and it is possible to directly communicate with the database without using the network. You may.

Although the case where the computer main body (terminal) is of a stationary type has been described, since the size of the computer has been reduced, the computer itself is worn, and a wireless connection is established between the main body and the data server. You may make it connect.

[0072]

As described above, according to the present invention,
It is possible to provide an environment in which the support can be received by video and audio while performing and working in a place of practice.

Further, according to the present invention, in the case of a practice where there is no instructor, if a forgotten matter or a document to be referred to comes out, it is possible to search or refer to the document without interrupting the work. As a result, operating in the Internet environment can provide the possibility of obtaining solutions even in difficult situations that cannot be handled by a single instructor.

Further, when it is necessary to refer to many drawings in a dim and narrow work site, the work has conventionally had to be performed while communicating with a person in charge of the design office by radio. According to this, the user can work while viewing the drawing virtually at the site, so that the work can be performed efficiently.

[Brief description of the drawings]

FIG. 1 is a block diagram showing a schematic configuration of an emergency medical support system according to an embodiment of the present invention.

FIG. 2 is a block diagram conceptually showing a specific configuration of the emergency medical support system.

FIG. 3 is a front view showing a headset constituting the emergency medical support system together with a mounted state.

FIG. 4 is a plan view of the headset as viewed from above.

FIG. 5 is a block diagram conceptually showing a method of switching WWW contents by voice.

FIG. 6 is an explanatory diagram showing types of camera work of reproduced video.

FIG. 7 is an explanatory diagram showing a relationship between a standard and a reproduced video sequence in which a time axis is enlarged.

FIG. 8 is an explanatory diagram for explaining data reproduction speed control;

FIG. 9 is an explanatory diagram showing an outline of a control program for emergency medical support.

FIG. 10 is an explanatory diagram showing a monitor screen corresponding to a virtual image.

FIG. 11 is an explanatory diagram showing a monitor screen corresponding to another virtual image.

FIG. 12 is an explanatory diagram showing one scene of a model video.

FIG. 13 is an explanatory diagram showing an example of a skeleton display of FIG. 12;

FIG. 14 is an explanatory diagram showing another scene of the exemplary video.

FIG. 15 is an explanatory diagram showing an example of a skeleton display of FIG. 14;

[Explanation of symbols]

 A: Computer body B: WWW server C: File server 10: Headset 12: Liquid crystal goggles 14: Small video camera 16: Headphones 18: Microphone 20: Penlight 22: Receiver 24 ... Transmitter 30: Voice recognition module 32 ... Status control unit 34 voice synthesis module 36 playback speed control unit 38 image processing unit 40 image recognition module

Continued on the front page (51) Int.Cl. ⁶ Identification code FI G10L 3/00 551 G06F 15/62 350K

Claims (11)

[Claims] 1. An information presenting apparatus configured so that presentation of multimedia information can be controlled by an image recognizing means and a voice recognizing means, a hands-free remote interactive operation on the apparatus, and a method for controlling information provided by the apparatus. A virtual environment presentation device, comprising: a user interface device having at least video input means, video output means, audio input means, and audio output means, which enables see-through viewing. 2. The image input device according to claim 1, wherein the image input means inputs a visual field image of the user, recognizes the user's line of sight by the image recognizing means of the information presenting device based on the visual field image, and outputs the video image. A virtual environment presentation device, wherein an angle of an image provided to a means is controlled so as to approach the line of sight. 3. The image processing device according to claim 1, wherein the image input by the image input device and the image provided by the information presenting device are synthesized, and the synthesized image is provided to the image output device. A virtual environment presentation device characterized by the following. 4. The apparatus according to claim 1, wherein the voice input means inputs a user's voice, and recognizes a character string uttered by the user from the voice by a voice recognition means of the information presenting device. A virtual environment presentation device, wherein the content of the multimedia information provided by the information presentation device is switched based on the information. 5. The character string according to claim 1, wherein the voice input means inputs a user's voice, and recognizes a plurality of character strings uttered by the user from the voice by a voice recognition means of the information presenting apparatus. A virtual environment presentation device, wherein a reproduction speed of time-series multimedia information provided by the information presentation device is controlled based on a time interval between the information presentation device and a character string. 6. The system according to claim 1, wherein a biological signal of a user or a person facing the user is input, and the reproduction speed of the time-series multimedia information provided by the information presentation device is determined based on the biological signal. A virtual environment presentation device characterized in that it is controlled. 7. The user interface device according to claim 1, wherein said user interface device having said video input means, video output means, audio input means and audio output means is constituted by a headset attachable to a user's head. A virtual environment presentation device characterized by the following. 8. The information processing apparatus according to claim 7, wherein the headset is capable of wirelessly transmitting information with the information presenting device via a video transmitting unit, a video receiving unit, a voice transmitting unit, and a voice receiving unit. A virtual environment presentation device characterized by the following. 9. The virtual environment presentation device according to claim 7, wherein the information presentation device is connected to a data server that provides multimedia information. 10. The information processing apparatus according to claim 9, wherein the headset is wire-connected to the information presentation device attached to a user's body, and the information presentation device is wirelessly connected to the data server. Virtual environment presentation device. 11. The system according to claim 9, wherein the information presentation device and the data server are respectively a WWW client and a WW connected to the Internet.
A virtual environment presentation device, being a W server.