+

WO2016006070A1 - Portable information terminal device and head-mount display linked thereto - Google Patents

Portable information terminal device and head-mount display linked thereto Download PDF

Info

Publication number
WO2016006070A1
WO2016006070A1 PCT/JP2014/068343 JP2014068343W WO2016006070A1 WO 2016006070 A1 WO2016006070 A1 WO 2016006070A1 JP 2014068343 W JP2014068343 W JP 2014068343W WO 2016006070 A1 WO2016006070 A1 WO 2016006070A1
Authority
WO
WIPO (PCT)
Prior art keywords
information terminal
portable information
display
image
hmd
Prior art date
Application number
PCT/JP2014/068343
Other languages
French (fr)
Japanese (ja)
Inventor
坂本　哲也
橋本　康宣
益岡　信夫
吉澤　和彦
Original Assignee
日立マクセル株式会社
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 日立マクセル株式会社 filed Critical 日立マクセル株式会社
Priority to PCT/JP2014/068343 priority Critical patent/WO2016006070A1/en
Publication of WO2016006070A1 publication Critical patent/WO2016006070A1/en

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
    • G06F3/048—Interaction techniques based on graphical user interfaces [GUI]
    • G06F3/0484—Interaction techniques based on graphical user interfaces [GUI] for the control of specific functions or operations, e.g. selecting or manipulating an object, an image or a displayed text element, setting a parameter value or selecting a range
    • G06F3/0485—Scrolling or panning

Definitions

  • the present invention relates to a portable information terminal device capable of displaying image information, such as a smartphone, and more particularly to a portable information terminal device capable of displaying image information in cooperation with a head-mounted display (HMD (Head Mounted Display)), and And a head-mounted display for that purpose.
  • HMD Head Mounted Display
  • a portable terminal that uses a head-mounted display as an external display device is already known, for example, from Patent Document 1 below.
  • a mobile terminal system includes a mobile terminal provided with a display device that performs display according to display information, and a virtual screen in a display area within the wearer's field of view based on display information acquired from the mobile terminal.
  • a head-mounted display, and the head-mounted display indicates a relative position of the mobile terminal with respect to the head-mounted display based on the display of the imaging unit and the display device imaged by the imaging unit.
  • a terminal posture acquisition unit that generates information.
  • the display area is set at a position overlapping the display screen of the display device based on the position information, and display according to the display information is performed on the virtual screen.
  • the virtual display is displayed on the HMD when looking at the screen of the portable terminal only when the user wants to see it.
  • the problem is that the line of sight is diverted from the terminal, which does not necessarily lead to an improvement in the operability of the portable terminal.
  • the present invention has been achieved in view of the above-described problems in the prior art, and in particular, the operability of the portable terminal can be improved in the portable information terminal device and the head mounted display linked therewith. It is an object of the present invention to provide a portable information terminal device and a head mounted display that cooperates with the portable information terminal device.
  • a portable information terminal capable of displaying an image in cooperation with a head mounted display, an image display unit for displaying an image, and the head
  • a data communication unit that performs data communication with the mount display; an operation input unit; and a control unit that displays an image on the image display unit based on control signals from the data communication unit and the operation input unit.
  • the control unit displays a part of the image displayed on the image display unit in cooperation with the head mounted display so as to be continuous with the image displayed on the head mounted display, and
  • the control unit selectively displays on the portable information terminal when the position of the portable information terminal is moved within the field of view of the head mounted display.
  • a mode for fixing the image the portable information terminal has a function of switching between a mode for fixing the image to be displayed by the head-mounted display is provided.
  • a head mounted display capable of displaying an image in cooperation with a portable information terminal, the human being wearing the head mounted display on the head
  • An image display unit that displays an image on the screen, a data communication unit that performs data communication with the portable information terminal, an imaging unit, and a control signal from the data communication unit and the imaging unit,
  • a control unit that displays an image, and the control unit displays a part of the image displayed in cooperation with the portable information terminal so as to be continuous with the image displayed on the portable information terminal
  • the control unit selectively fixes an image to be displayed on the portable information terminal when the position of the portable information terminal is moved within the field of view of the head mounted display.
  • Head-mounted display which provides the ability to switch between a mode for fixing the image to be displayed on the head mounted display is provided.
  • the portable information terminal device capable of improving the operability by the portable terminal and the head mounted display linked with the portable information terminal device are provided.
  • FIG. 1 It is a figure which shows the portable information terminal device which becomes one embodiment of this invention. It is a figure which shows an example of the internal structure of the said portable information terminal device. It is a figure which shows the external appearance structure of the head mounted display which displays image information in cooperation with the said portable information terminal device. It is a figure which shows an example of an internal structure of the portable information terminal of the said head mounted display. It is a figure which shows the example of a specific structure of the image display part of the said head mounted display, and its operation principle. It is a figure explaining the environment which uses the said portable information terminal device and a head mounted display. It is a figure explaining the display image by the said portable information terminal device and a head mounted display.
  • FIG. 1 shows the appearance of a portable information terminal device according to the present invention.
  • the portable information terminal device 100 is operated on a part (surface) of the casing.
  • This is a portable information terminal device in the form of a so-called smartphone equipped with a touch panel as the input unit 102 and a liquid crystal panel as the display unit.
  • reference numeral 110 in the drawing which will be described in detail later, is attached to a part of the casing of the portable information terminal device 100 (in this example, four corners outside the display screen on the main body surface).
  • a positioning marker is shown.
  • each of these positioning markers 110 is composed of an infrared light source.
  • FIG. 2 shows an example of the internal configuration of the portable information terminal device 100.
  • the portable information terminal device 100 for example, data that communicates with an external network or the like by radio waves.
  • a communication unit 101 for example, an operation input unit 102 such as a touch panel, on which operation input is performed by a user, a control unit 103 that inputs and outputs control signals and the like to and from the data communication unit 101 and the operation input unit 102,
  • Storage unit 104 including various memories for storing software executed in the computer, necessary data, and the like, and an image display unit configured to display various images in response to commands from the control unit 103, for example, a liquid crystal panel 105 or the like.
  • the data communication unit 101 transmits and receives cooperative display image data, a mobile terminal display area, and screen position information of the mobile information terminal 100 within the field of view of the HMD with the HMD described later.
  • the operation input unit 102 performs operations such as changing the clamp mode and the release mode, or changing the mobile terminal display area, and is specifically a touch panel or the like.
  • the storage unit 104 stores cooperative display image data, information on the mobile terminal display area, screen position information of the mobile information terminal 100 within the field of view of the HMD, and more specifically, RAM and storage. It is a device.
  • the image display unit 105 displays a cooperation display image or the like, and specifically, a liquid crystal panel, an OLED panel, or the like.
  • FIG. 3 is a so-called wearable display device, and shows a head-mounted display (HMD (Head-Mounted Display)) 20 that displays image information to the user in cooperation with the portable information terminal device 100.
  • HMD Head-Mounted Display
  • a glasses-type head mounted display is shown. That is, this glasses-type head-mounted display is provided with the portable information terminal 200 in a part of the glasses holding portion (vine).
  • Reference numeral 202 in the figure denotes a camera that constitutes the imaging unit 202 described later, and 205 denotes an image display unit that will be described later.
  • FIG. 4 shows an example of the internal configuration of the portable information terminal 200.
  • the mobile information terminal 200 includes a data communication unit 201, an imaging unit 202, a control unit 203, a storage unit 204, and an image display unit 205. .
  • the data communication unit 201 transmits and receives cooperative display image data, a portable terminal display area, and screen position information of the portable information terminal within the field of view of the HMD with the portable information terminal.
  • a communication unit or a direct communication unit with the HMD 20 such as WiFi Direct.
  • the communication method may be wired or wireless.
  • the imaging unit 202 recognizes the screen position of the portable information terminal within the field of view of the HMD 20, and is specifically a camera.
  • the control unit 203 controls linked display and the like.
  • the storage unit 204 stores cooperative display image data, information on the mobile terminal display area, screen position information of the mobile information terminal within the field of view of the HMD 20, and the like. is there.
  • the image display unit 205 displays a linked display image and the like, and specifically includes an image projection unit and an optical system that projects the image onto the user's pupil.
  • the optical system includes a mirror 211 and a half mirror 212, and the video light projected from the image projection unit 210 that generates an image to be displayed on the HMD 20 (see the arrow in the figure) is displayed by the user.
  • the image is only required to be projected onto the pupil 213 of the user, so the configuration is not limited to this.
  • the mirror just before the pupil is desirably a half mirror as indicated by reference numeral 212.
  • the portable information terminal device 100 and the HMD 20 described above send data from the server device 300 to a network (Internet) 400 or a wireless line (relay station) 500, and further to a router device. 600 and the like are connected to each other.
  • a network Internet
  • a wireless line relay station
  • FIG. 7 shows an image displayed to the user's eyes by the HMD 20 and the portable information terminal 100.
  • the display image of the HMD 20 is fixed, while the portable information terminal
  • the image displayed on 100 is adjusted to connect as part of the image displayed on the HMD 20.
  • ⁇ Clamp mode> In this mode, as shown in FIG. 8, the image displayed on the portable information terminal 100 is fixed, and the display image of the HMD 20 is adjusted to be connected to the image of the portable information terminal 100.
  • “to be connected” it is not always necessary to connect strictly, and there may be a slight gap or overlap.
  • the display of the HMD 20 and the portable information terminal 100 but here, as a preparation stage common to each case, it is information necessary for the display cooperation described above.
  • information on the positional relationship between the positioning marker, which is a reference for the screen position of the portable information terminal 100, and the display screen is transmitted from the portable information terminal 100 to the HMD 20.
  • the positioning marker 110 is composed of an infrared light source, it can be easily distinguished from other shapes, and the display screen position can be accurately recognized.
  • the imaging unit 202 of the HMD 20 can capture an infrared image separately from a visible light image.
  • the HMD 20 recognizes the position of the positioning marker with the camera (imaging unit 202), and recognizes the screen position of the portable information terminal 100 from the information.
  • the screen position of the portable information terminal 100 may be recognized using an image recognition unit without using the positioning marker.
  • a portion displayed on the portable information terminal 100 in the above-described cooperative display image as a whole is referred to as a “portable information terminal display area”.
  • the mobile information terminal 100 transmits the cooperation display image data to the HMD 20.
  • the clamp mode is changed to the release mode, the linked display image data has already been transmitted to the HMD 20 at the time of the change, and therefore the transmitted data is used.
  • the HMD 20 After entering this release mode, the HMD 20 recognizes the screen position of the portable information terminal 100, and the HMD 20 transmits information on the display area of the portable information terminal 100 to the portable information terminal 100. When this screen position is changed, new portable information terminal display area information is transmitted from the HMD to the portable information terminal 100. At this time, the mobile information terminal screen position information within the field of view of the HMD 20 may also be transmitted from the HMD 20 to the mobile information terminal 100.
  • the linked display image does not move in the field of view of the HMD 20 during the release mode, but the HMD 20 does not move the display area other than the display area of the portable information terminal 100 in the linked display image. Display only the part.
  • the portable information terminal 100 displays a portion of the portable information terminal display area. Note that both the HMD 20 and the portable information terminal 100 may display images other than the cooperative display images such as icons and operation buttons in an overlapping manner.
  • the mobile information terminal 100 does not display the cooperative display image
  • the HMD 20 side displays the cooperative display image including the screen area of the mobile information terminal 100. I do not care.
  • the screen of the portable information terminal 100 may be displayed in black, or an image other than the cooperative display image such as an icon or an operation button may be displayed.
  • the portable information terminal 100 transmits the cooperation display image data and information on the portable information terminal display area to the HMD 20.
  • the release mode is changed to the clamp mode, the linked display image data has already been transmitted to the HMD 20 at the time of the change, and therefore the transmitted data is used.
  • the information of the portable information terminal display area is already shared between the HMD 20 and the portable information terminal 100, the information is used.
  • the HMD 20 After entering the clamp mode, the HMD 20 recognizes the screen position of the portable information terminal 100 within the field of view of the HMD 20. This information may be transmitted from the HMD 20 to the portable information terminal 100 every time the position is changed.
  • the portable information terminal 100 displays an image in the portable information terminal display area in the same manner as described above.
  • the display position of the cooperative display image is adjusted according to the screen position of the portable information terminal 100 within the field of view of the HMD 20, and the portable information terminal display area is overlapped with the screen position of the portable information terminal 100. The part outside the information terminal display area is displayed. And when the position of the portable information terminal 100 moves within the field of view of the HMD 20, the display image of the HMD 20 is adjusted according to the new screen position of the portable information terminal 100.
  • the portable information terminal 100 when the position of the portable information terminal display area in the cooperative display image is changed by a swipe operation that slides the screen with a finger, etc., or pinch-in, pinch-out
  • the size of the portable information terminal display area is changed by the operation such as the above, information on the new portable information terminal display area is transmitted from the portable information terminal 100 to the HMD 20, and the HMD 20 The position is adjusted, and a portion outside the portable information terminal display area is displayed so that the new portable information terminal display area overlaps the screen position of the portable information terminal.
  • the boundary of a cooperation display image may be in the state which can be in the displayable area
  • new cooperative display image data may be transmitted from the portable information terminal 100 to the HMD 20 to expand the data range of the cooperative display image.
  • the positional information of the portable information terminal 100 within the field of view of the HMD 20 is necessary, the information is transmitted from the HMD 20 to the portable information terminal 100.
  • both the HMD 20 and the portable information terminal 100 may display images other than the cooperative display images such as icons and operation buttons in a superimposed manner.
  • FIG. 11 shows an example in which operation buttons are displayed on the portable information terminal 100.
  • the cooperative display image of the HMD 20 at that time is maintained, and the control in the release mode described in the case 1 is performed.
  • the cooperative display image may not be displayed on the portable information terminal 100, and the cooperative display image may be displayed on the HMD side including the screen area of the portable information terminal 100. .
  • the position and size of the cooperative display image are adjusted in accordance with the update of the display area of the portable information terminal 100 and the update of the position of the portable information terminal 100 within the HMD visual field.
  • the screen of the portable information terminal 100 may be displayed in black, or an image other than the cooperative display image such as an icon or an operation button may be displayed.
  • the cooperation display image data is transmitted from the HMD 20 to the portable information terminal 100.
  • the clamp mode is changed to the release mode, the cooperative display image data has already been transmitted to the portable information terminal 100 at the time of the change, and the data is used.
  • the HMD 20 After entering the release mode, the HMD 20 recognizes the screen position of the portable information terminal 100 and transmits information of the portable information terminal display area from the HMD 20 to the portable information terminal 100. When this screen position is changed, new portable information terminal display area information is transmitted from the HMD 20 to the portable information terminal 100. Note that the screen position information of the portable information terminal 100 within the field of view of the HMD 20 may be transmitted together from the HMD 20 to the portable information terminal 100.
  • the cooperative display image does not move within the field of view of the HMD 20, but the HMD 20 displays only the portion other than the portable information terminal display area in the cooperative display image.
  • the portable information terminal 100 displays a portion of the portable information terminal display area.
  • both the HMD 20 and the portable information terminal 100 may display images other than the cooperative display images such as icons and operation buttons in an overlapping manner.
  • the cooperative display image may be displayed on the HMD side including the screen area of the portable information terminal 100 without displaying the cooperative display image on the portable information terminal 100.
  • the screen of the portable information terminal 100 may be displayed in black, or an image other than the cooperative display image such as an icon or an operation button may be displayed.
  • the cooperative display image data and the information of the portable information terminal display area may not be transmitted from the HMD 20 to the portable information terminal 100.
  • the portable information terminal 100 performs an operation on the cooperation display image, the information needs to be transmitted.
  • the display image data is transmitted from the HMD 20 to the portable information terminal 100.
  • the mobile information terminal display area is determined from the screen position of the mobile information terminal 100 within the field of view of the HMD 20 at that time, and this information is transmitted from the HMD 20 to the mobile information terminal 100.
  • the release mode is changed to the clamp mode, the linked display image data has already been transmitted to the portable information terminal at the time of the change, so that data is used, and information on the portable information terminal display area is also used. Is shared between the HMD 20 and the portable information terminal 100, and uses that information.
  • the HMD 20 After entering the clamp mode, the HMD 20 recognizes the screen position of the portable information terminal 100 within the field of view of the HMD. This information may be transmitted from the HMD 20 to the portable information terminal 100 every time the position is changed.
  • the mobile information terminal 100 displays an image in the mobile information terminal display area.
  • the display position of the cooperative display image is adjusted according to the screen position of the portable information terminal 100 within the field of view of the HMD 20, and the portable information terminal display area overlaps the screen position of the portable information terminal 100. The part outside the terminal display area is displayed.
  • the display image of the HMD 20 is adjusted according to the screen position of the new portable information terminal.
  • the portable information terminal 100 when the position of the portable information terminal display area in the cooperative display image is changed by a swipe operation of sliding the screen with a finger, or the portable information terminal is displayed by an operation such as pinch-in and pinch-out.
  • the size of the area is changed, information on a new portable information terminal display area is transmitted from the portable information terminal 100 to the HMD 20, and the HMD 20 adjusts the size and position of the linked display image to create a new portable information terminal.
  • a portion outside the portable information terminal display area is displayed so that the information terminal display area overlaps the screen position of portable information terminal 100.
  • the boundary of a cooperation display image may be in the state which can enter into the displayable area
  • new cooperative display image data may be transmitted from the HMD 20 to the portable information terminal 100 to expand the data range of the cooperative display image.
  • both the HMD 20 and the portable information terminal 100 may display images other than the cooperative display images such as icons and operation buttons in an overlapping manner.
  • control units 103 and 203 incorporated in the HMD 20 or the portable information terminal 100.
  • a memory provided in a part of the control unit, or This is realized by executing software stored in the storage unit 104 or 204 in advance.
  • the cooperative display image is not displayed on the portable information terminal 100, and the cooperative display image is displayed on the HMD 20 side including the screen area of the portable information terminal 100. Absent. On the HMD 20 side, the position and size of the cooperative display image are adjusted in accordance with the update of the portable information terminal display area and the update of the position of the portable information terminal within the HMD visual field. In this case, the screen of the portable information terminal 100 may be displayed in black, or an image other than the cooperative display image such as an icon or an operation button may be displayed.
  • the cooperative display image is not displayed on the portable information terminal 100, the cooperative display image data and the information on the portable information terminal display area may not be transmitted from the HMD 20 to the portable information terminal 100.
  • the information terminal 100 performs an operation on the cooperation display image, the information needs to be transmitted.
  • the cooperation display image data is divided into the case of sending from the portable information terminal 100 to the HMD 20 and the case of sending from the HMD 20 to the portable information terminal 100.
  • the data may not be stored in the portable information terminal 100 or the HMD 20 but may be acquired via the Internet. Further, this includes a case where the acquisition destination via the Internet is transmitted from the portable information terminal 100 to the HMD 20 or from the HMD 20 to the portable information terminal 100 to transmit the cooperative display image data.
  • FIG. 1 In the above-described embodiment, an example in which a map is displayed as an example by the portable information terminal 100 and the HMD 20 has been described in detail.
  • the present invention is not limited to this example, and FIG.
  • a part of the menu image (for example, one page) is displayed on the display screen of the portable information terminal 100 and another page is displayed.
  • the display screen of the HMD 20 can be moved together with the display screen of the portable information terminal 100. It becomes possible. Alternatively, as shown in FIG. 14, a part of the display screen of the HMD 20 can be moved on the display screen of the portable information terminal 100.
  • a desktop display screen of a PC can be displayed on the display screen of the HMD 20.
  • a word processor window W is opened and a plurality of buttons B are provided.
  • FIGS. 16A and 16B the position of the window W on the desktop can be moved by performing a swipe operation on the portable information terminal 100.
  • this invention is not limited to the above-mentioned Example, Various modifications are included.
  • the above-described embodiments are described in detail for the entire system in order to explain the present invention in an easy-to-understand manner, and are not necessarily limited to those having all the configurations described.
  • a part of the configuration of one embodiment can be replaced with the configuration of another embodiment, and the configuration of another embodiment can be added to the configuration of one embodiment.
  • each of the above-described configurations, functions, processing units, processing means, and the like may be realized by hardware by designing a part or all of them with, for example, an integrated circuit.
  • Each of the above-described configurations, functions, and the like may be realized by software by interpreting and executing a program that realizes each function by the processor.
  • Information such as programs, tables, and files that realize each function can be stored in a memory, a hard disk, a recording device such as an SSD (Solid State Drive), or a recording medium such as an IC card, an SD card, or a DVD.
  • DESCRIPTION OF SYMBOLS 100 ... Portable information terminal device, 101 ... Data communication part, 102 ... Operation input part, 103 ... Control part, 104 ... Memory

Landscapes

  • Engineering & Computer Science (AREA)
  • General Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Human Computer Interaction (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Controls And Circuits For Display Device (AREA)
  • User Interface Of Digital Computer (AREA)

Abstract

The purpose of the present invention is to provide a portable information terminal device which can improve portable terminal operability, and a head-mount display linked to the portable information terminal device. Provided is a portable information terminal that can display images by linking to a head-mount display, and that comprises: an image display unit for displaying images; a data communication unit for performing data communication with the head-mount display; an operation input unit; and a control unit that displays images on the image display unit on the basis of control signals from the data communication unit and the operation input unit. The control unit links with the head-mount display so that a portion of the image displayed by the image display unit is displayed to be continuous with the image displayed by the head-mount display as a series of images. The control unit is provided with a function for switching selectively between a mode in which the image displayed by the portable information terminal is fixed and a mode in which the image displayed by the head-mount display is fixed, when the position of the portable information terminal is moved within the field of view of the head-mount display.

Description

携帯情報端末装置及びそれと連携するヘッドマウントディスプレイPortable information terminal device and head mounted display linked therewith
　本発明は、例えば、スマートフォン等、画像情報を表示可能な携帯情報端末装置に関し、特に、ヘッドマウントディスプレイ（ＨＭＤ(Head Mounted Display)）と連携して画像情報を表示可能な携帯情報端末装置、及び、そのためのヘッドマウントディスプレイに関する。 The present invention relates to a portable information terminal device capable of displaying image information, such as a smartphone, and more particularly to a portable information terminal device capable of displaying image information in cooperation with a head-mounted display (HMD (Head Mounted Display)), and And a head-mounted display for that purpose.
　ヘッドマウントディスプレイを外部表示装置として使用する携帯端末については、例えば、以下の特許文献１により既に知られている。 A portable terminal that uses a head-mounted display as an external display device is already known, for example, from Patent Document 1 below.
特開２０１１－１８６８５６号公報JP 2011-186856 A
　かかる従来技術によれば、携帯端末システムは、表示情報に応じて表示を行う表示装置を備える携帯端末と、携帯端末から取得した表示情報に基づいて、装着者の視野内の表示領域に仮想画面を表示するヘッドマウントディスプレイとを備えており、当該ヘッドマウントディスプレイは、撮像部と、当該撮像部によって撮像された表示装置の表示に基づいてヘッドマウントディスプレイに対する携帯端末の相対的な位置を示す位置情報を生成する端末姿勢取得部とを備えており、表示領域は位置情報に基づいて表示装置の表示画面に重なる位置に設定されると共に、仮想画面に表示情報に応じた表示が行われる。 According to such a conventional technique, a mobile terminal system includes a mobile terminal provided with a display device that performs display according to display information, and a virtual screen in a display area within the wearer's field of view based on display information acquired from the mobile terminal. A head-mounted display, and the head-mounted display indicates a relative position of the mobile terminal with respect to the head-mounted display based on the display of the imaging unit and the display device imaged by the imaging unit. And a terminal posture acquisition unit that generates information. The display area is set at a position overlapping the display screen of the display device based on the position information, and display according to the display information is performed on the virtual screen.
　しかしならが、上述した従来技術になる携帯端末システムでは、使用者がみたいときだけ、携帯端末の画面の方向を見ればＨＭＤに仮想ディスプレイが表示され、周囲の状況を確認する場合には、携帯端末から視線をそらすようにするというものであり、これでは、必ずしも、携帯端末による操作性を向上することには繋がらないといという問題点があった。 However, in the above-described portable terminal system, the virtual display is displayed on the HMD when looking at the screen of the portable terminal only when the user wants to see it. The problem is that the line of sight is diverted from the terminal, which does not necessarily lead to an improvement in the operability of the portable terminal.
　そこで、本発明は、上述した従来技術における問題点に鑑みて達成されたものであり、特に、携帯情報端末装置及びそれと連携するヘッドマウントディスプレイにおいて、携帯端末による操作性を向上することが可能な携帯情報端末装置と、それと連携するヘッドマウントディスプレイを提供することをその目的とする。 Therefore, the present invention has been achieved in view of the above-described problems in the prior art, and in particular, the operability of the portable terminal can be improved in the portable information terminal device and the head mounted display linked therewith. It is an object of the present invention to provide a portable information terminal device and a head mounted display that cooperates with the portable information terminal device.
　上記の目的を達成するために、本発明によれば、まず、ヘッドマウントディスプレイと連携して画像を表示することが可能な携帯情報端末であって、画像を表示する画像表示部と、前記ヘッドマウントディスプレイとのデータ通信を行うデータ通信部と、操作入力部と、前記データ通信部と前記操作入力部からの制御信号に基づいて、前記画像表示部に画像を表示する制御部とを備えており、前記制御部は、前記ヘッドマウントディスプレイと連携して前記画像表示部に表示する画像の一部を、前記ヘッドマウントディスプレイで表示される画像と一連の画像として連続するように表示し、かつ、前記制御部は、選択的に、前記ヘッドマウントディスプレイの視野内で当該携帯情報端末の位置を動かした時に、当該携帯情報端末で表示する画像を固定するモードと、前記ヘッドマウントディスプレイで表示する画像を固定するモードとの間で切り替える機能を備えている携帯情報端末が提供される。 In order to achieve the above object, according to the present invention, first, a portable information terminal capable of displaying an image in cooperation with a head mounted display, an image display unit for displaying an image, and the head A data communication unit that performs data communication with the mount display; an operation input unit; and a control unit that displays an image on the image display unit based on control signals from the data communication unit and the operation input unit. And the control unit displays a part of the image displayed on the image display unit in cooperation with the head mounted display so as to be continuous with the image displayed on the head mounted display, and The control unit selectively displays on the portable information terminal when the position of the portable information terminal is moved within the field of view of the head mounted display. A mode for fixing the image, the portable information terminal has a function of switching between a mode for fixing the image to be displayed by the head-mounted display is provided.
　また、本発明によれば、やはり上記の目的を達成するため、携帯情報端末と連携して画像を表示することが可能なヘッドマウントディスプレイであって、前記ヘッドマウントディスプレイを頭部に装着した人間に画像を表示する画像表示部と、前記携帯情報端末とのデータ通信を行うデータ通信部と、撮像部と、前記データ通信部と前記撮像部からの制御信号に基づいて、前記画像表示部に画像を表示する制御部とを備えており、前記制御部は、前記携帯情報端末と連携して表示する画像の一部を、前記携帯情報端末で表示される画像と連続するように表示し、かつ、前記制御部は、選択的に、当該ヘッドマウントディスプレイの視野内で前記携帯情報端末の位置を動かした時に、前記携帯情報端末で表示する画像を固定するモードと、当該ヘッドマウントディスプレイで表示する画像を固定するモードとの間で切り替える機能を備えているヘッドマウントディスプレイが提供される。 In addition, according to the present invention, in order to achieve the above object, a head mounted display capable of displaying an image in cooperation with a portable information terminal, the human being wearing the head mounted display on the head An image display unit that displays an image on the screen, a data communication unit that performs data communication with the portable information terminal, an imaging unit, and a control signal from the data communication unit and the imaging unit, A control unit that displays an image, and the control unit displays a part of the image displayed in cooperation with the portable information terminal so as to be continuous with the image displayed on the portable information terminal, In addition, the control unit selectively fixes an image to be displayed on the portable information terminal when the position of the portable information terminal is moved within the field of view of the head mounted display. Head-mounted display which provides the ability to switch between a mode for fixing the image to be displayed on the head mounted display is provided.
　上述した本発明によれば、携帯情報端末装置及びそれと連携するヘッドマウントディスプレイにおいて、携帯端末による操作性を向上することが可能な携帯情報端末装置と、それと連携するヘッドマウントディスプレイが提供される。 According to the present invention described above, in the portable information terminal device and the head mounted display linked with the portable information terminal device, the portable information terminal device capable of improving the operability by the portable terminal and the head mounted display linked with the portable information terminal device are provided.
本発明の一実施の形態になる携帯情報端末装置を示す図である。It is a figure which shows the portable information terminal device which becomes one embodiment of this invention. 上記携帯情報端末装置の内部構成の一例を示す図である。It is a figure which shows an example of the internal structure of the said portable information terminal device. 上記携帯情報端末装置と連携して画像情報を表示するヘッドマウントディスプレイの外観構成を示す図である。It is a figure which shows the external appearance structure of the head mounted display which displays image information in cooperation with the said portable information terminal device. 上記ヘッドマウントディスプレイの携帯情報端末の内部構成の一例を示す図である。It is a figure which shows an example of an internal structure of the portable information terminal of the said head mounted display. 上記ヘッドマウントディスプレイの画像表示部の具体的な構成の例とその動作原理を示す図である。It is a figure which shows the example of a specific structure of the image display part of the said head mounted display, and its operation principle. 上記携帯情報端末装置とヘッドマウントディスプレイを使用する環境を説明する図である。It is a figure explaining the environment which uses the said portable information terminal device and a head mounted display. 上記携帯情報端末装置とヘッドマウントディスプレイによる表示画像を説明する図である。It is a figure explaining the display image by the said portable information terminal device and a head mounted display. クランプモードにおける上記携帯情報端末装置とヘッドマウントディスプレイによる表示画像を示す図である。It is a figure which shows the display image by the said portable information terminal device and head mounted display in clamp mode. ケース１における上記携帯情報端末装置とヘッドマウントディスプレイによる表示画像の一例を示す図である。It is a figure which shows an example of the display image by the said portable information terminal device in case 1 and a head mounted display. ケース２における上記携帯情報端末装置とヘッドマウントディスプレイによる表示画像の一例を示す図である。It is a figure which shows an example of the display image by the said portable information terminal device in case 2, and a head mounted display. 変形例における上記携帯情報端末装置とヘッドマウントディスプレイによる表示画像の一例を示す図である。It is a figure which shows an example of the display image by the said portable information terminal device and head mounted display in a modification. 変形例における上記携帯情報端末装置とヘッドマウントディスプレイによる表示画像の他の例を示す図である。It is a figure which shows the other example of the display image by the said portable information terminal device and head mounted display in a modification. 変形例における上記携帯情報端末装置とヘッドマウントディスプレイによる表示画像の他の例を示す図である。It is a figure which shows the other example of the display image by the said portable information terminal device and head mounted display in a modification. 変形例における上記携帯情報端末装置とヘッドマウントディスプレイによる表示画像の他の例を示す図である。It is a figure which shows the other example of the display image by the said portable information terminal device and head mounted display in a modification. 変形例における上記携帯情報端末装置とヘッドマウントディスプレイによる表示画像の更に他の例を示す図である。It is a figure which shows the further another example of the display image by the said portable information terminal device and head mounted display in a modification. 変形例における上記携帯情報端末装置とヘッドマウントディスプレイによる表示画像の更に他の例を示す図である。It is a figure which shows the further another example of the display image by the said portable information terminal device and head mounted display in a modification.
　以下、本発明の実施の形態になる携帯情報端末装置について、添付の図面を参照しながら、詳細に説明する。 Hereinafter, a portable information terminal device according to an embodiment of the present invention will be described in detail with reference to the accompanying drawings.
　まず、添付の図１には、本発明になる携帯情報端末装置の外観が示されており、図からも明らかなように、携帯情報端末装置１００は、その筺体の一部（表面）に操作入力部１０２であるタッチパネルと表示部である液晶パネルを搭載した、所謂、スマートフォンと呼ばれる形態の携帯情報端末装置である。また、図中の参照符号１１０は、後にも詳細に説明するが、当該携帯情報端末装置１００の筺体の一部（本例では、本体主面における表示画面の外側の四隅）に取り付けられた、位置決めマーカを示している。なお、本例では、これらの位置決めマーカ１１０を、それぞれ、赤外光源で構成している。 First, the attached FIG. 1 shows the appearance of a portable information terminal device according to the present invention. As is apparent from the figure, the portable information terminal device 100 is operated on a part (surface) of the casing. This is a portable information terminal device in the form of a so-called smartphone equipped with a touch panel as the input unit 102 and a liquid crystal panel as the display unit. In addition, reference numeral 110 in the drawing, which will be described in detail later, is attached to a part of the casing of the portable information terminal device 100 (in this example, four corners outside the display screen on the main body surface). A positioning marker is shown. In this example, each of these positioning markers 110 is composed of an infrared light source.
　次に、図２は、上記携帯情報端末装置１００の内部構成の一例を示しており、当該携帯情報端末装置１００は、図にも示すように、例えば、電波により外部のネットワーク等と通信するデータ通信部１０１、ユーザによる操作入力が行われる、例えば、タッチパネル等の操作入力部１０２、これらデータ通信部１０１や操作入力部１０２との間で制御信号等を入出力する制御部１０３、当該制御部で実行されるソフトウェアや必要なデータ等を格納する各種のメモリを含む記憶部１０４、そして、例えば、液晶パネルなどにより構成され、上記制御部１０３からの指令により各種の画像を表示する画像表示部１０５などにより構成されている。 Next, FIG. 2 shows an example of the internal configuration of the portable information terminal device 100. The portable information terminal device 100, as shown in the figure, for example, data that communicates with an external network or the like by radio waves. A communication unit 101, for example, an operation input unit 102 such as a touch panel, on which operation input is performed by a user, a control unit 103 that inputs and outputs control signals and the like to and from the data communication unit 101 and the operation input unit 102, Storage unit 104 including various memories for storing software executed in the computer, necessary data, and the like, and an image display unit configured to display various images in response to commands from the control unit 103, for example, a liquid crystal panel 105 or the like.
　なお、データ通信部１０１は、後に説明するＨＭＤとの間で、連携表示画像データ、携帯端末表示領域、ＨＭＤの視野内における携帯情報端末１００の画面位置情報の送受信を行うものであり、具体的には、ＬＡＮ通信部あるいは、WiFi Direct等のＨＭＤとの直接通信部であり、有線・無線を問わない。 Note that the data communication unit 101 transmits and receives cooperative display image data, a mobile terminal display area, and screen position information of the mobile information terminal 100 within the field of view of the HMD with the HMD described later. Is a LAN communication unit or a direct communication unit with an HMD such as WiFi Direct, regardless of whether it is wired or wireless.
　また、操作入力部１０２は、クランプモードとリリースモードの変更、あるいは携帯端末表示領域の変更等の操作を行うものであり、具体的にはタッチパネル等である。そして、記憶部１０４は、連携表示画像データや、携帯端末表示領域の情報、またはＨＭＤの視野内における携帯情報端末１００の画面位置情報等を記憶するものであり、具体的には、ＲＡＭやストレージデバイスである。また、画像表示部１０５は、連携表示画像等の表示を行うものであり、具体的には液晶パネル、ＯＬＥＤパネル等である。 The operation input unit 102 performs operations such as changing the clamp mode and the release mode, or changing the mobile terminal display area, and is specifically a touch panel or the like. The storage unit 104 stores cooperative display image data, information on the mobile terminal display area, screen position information of the mobile information terminal 100 within the field of view of the HMD, and more specifically, RAM and storage. It is a device. The image display unit 105 displays a cooperation display image or the like, and specifically, a liquid crystal panel, an OLED panel, or the like.
　図３は、所謂、ウェアラブルな表示装置であり、上記携帯情報端末装置１００と連携して画像情報をユーザに対して表示するヘッドマウントディスプレイ（ＨＭＤ(Head Mounted Display)）２０を示している。ここでは、特に、メガネ型のヘッドマウントディスプレイを示す。即ち、このメガネ型のヘッドマウントディスプレイは、メガネの保持部（蔓）の一部に携帯情報端末２００を備えたものである。また、図中の符号２０２は、後にも説明する撮像部２０２を構成するカメラであり、２０５は、後にも説明する画像表示部を示す。 FIG. 3 is a so-called wearable display device, and shows a head-mounted display (HMD (Head-Mounted Display)) 20 that displays image information to the user in cooperation with the portable information terminal device 100. Here, in particular, a glasses-type head mounted display is shown. That is, this glasses-type head-mounted display is provided with the portable information terminal 200 in a part of the glasses holding portion (vine). Reference numeral 202 in the figure denotes a camera that constitutes the imaging unit 202 described later, and 205 denotes an image display unit that will be described later.
　図４は、上記携帯情報端末２００の内部構成の一例を示しており、本例では、データ通信部２０１、撮像部２０２、制御部２０３、記憶部２０４、そして、画像表示部２０５を備えている。 FIG. 4 shows an example of the internal configuration of the portable information terminal 200. In this example, the mobile information terminal 200 includes a data communication unit 201, an imaging unit 202, a control unit 203, a storage unit 204, and an image display unit 205. .
　データ通信部２０１は、携帯情報端末との間で、連携表示画像データ、携帯端末表示領域、ＨＭＤの視野内における携帯情報端末の画面位置情報の送受信を行うものであり、具体的には、ＬＡＮ通信部あるいは、WiFi Direct等のＨＭＤ２０との直接通信部である。なお、通信の方法としては、有線・無線を問わない。 The data communication unit 201 transmits and receives cooperative display image data, a portable terminal display area, and screen position information of the portable information terminal within the field of view of the HMD with the portable information terminal. A communication unit or a direct communication unit with the HMD 20 such as WiFi Direct. The communication method may be wired or wireless.
　撮像部２０２は、ＨＭＤ２０の視野内での携帯情報端末の画面位置を認識するものであり、具体的にはカメラである。 The imaging unit 202 recognizes the screen position of the portable information terminal within the field of view of the HMD 20, and is specifically a camera.
　制御部２０３は、連携表示等の制御をおこなうものである。 The control unit 203 controls linked display and the like.
　記憶部２０４は、連携表示画像データや、携帯端末表示領域の情報、または、ＨＭＤ２０の視野内における携帯情報端末の画面位置情報等を記憶するものであり、具体的には、ＲＡＭやストレージデバイスである。 The storage unit 204 stores cooperative display image data, information on the mobile terminal display area, screen position information of the mobile information terminal within the field of view of the HMD 20, and the like. is there.
　画像表示部２０５は、連携表示画像等の表示を行うものであり、具体的には画像投射部とその画像を、ユーザの瞳孔に投射する光学系からなる。 The image display unit 205 displays a linked display image and the like, and specifically includes an image projection unit and an optical system that projects the image onto the user's pupil.
　この画像表示部２０５の具体的な構成の一例と、その動作原理が、図５に示されている。即ち、この図５では、光学系はミラー２１１とハーフミラー２１２からなっており、ＨＭＤ２０に表示される画像を生成する画像投射部２１０から投射される映像光（図の矢印を参照）をユーザの瞳孔２１３に導く。但し、本発明では、画像がユーザの瞳孔２１３に投射されればよいので、構成はこれに限定されるものではない。しかしながら、以下に詳述するが、本発明では、ＨＭＤ２０の画像と携帯情報端末１００の画像とを重ねてみることから、符号２１２のように、瞳孔直前のミラーは、ハーフミラーが望ましい。 An example of a specific configuration of the image display unit 205 and its operating principle are shown in FIG. That is, in FIG. 5, the optical system includes a mirror 211 and a half mirror 212, and the video light projected from the image projection unit 210 that generates an image to be displayed on the HMD 20 (see the arrow in the figure) is displayed by the user. Guide to the pupil 213. However, in the present invention, the image is only required to be projected onto the pupil 213 of the user, so the configuration is not limited to this. However, as will be described in detail below, in the present invention, since the image of the HMD 20 and the image of the portable information terminal 100 are overlapped, the mirror just before the pupil is desirably a half mirror as indicated by reference numeral 212.
　そして、図６にも示すように、上述した携帯情報端末装置１００やＨＭＤ２０は、例えば、サーバ装置３００からのデータを、ネットワーク（インターネット）４００又は無線回線（中継ステーション）５００、更には、ルータ装置６００等を経由して、互いに接続されている。 As shown in FIG. 6, the portable information terminal device 100 and the HMD 20 described above, for example, send data from the server device 300 to a network (Internet) 400 or a wireless line (relay station) 500, and further to a router device. 600 and the like are connected to each other.
　更に、ＨＭＤ２０と携帯情報端末１００との間の表示の連携について、以下に、詳細に説明する。ここでは、２種類の連携表示モードがあり、表示連携の対象となる画像を「連携表示画像」と呼ぶ。 Furthermore, the display cooperation between the HMD 20 and the portable information terminal 100 will be described in detail below. Here, there are two types of cooperative display modes, and an image that is a target of display cooperation is referred to as a “cooperative display image”.
＜リリースモード＞
　図７には、上記のＨＭＤ２０と携帯情報端末１００とによって、利用者の目に対して表示される画像が示されており、このモードでは、ＨＭＤ２０の表示画像を固定し、他方、携帯情報端末１００に表示される画像は、ＨＭＤ２０に表示される画像の一部として接続するように調整される。 <Release mode>
FIG. 7 shows an image displayed to the user's eyes by the HMD 20 and the portable information terminal 100. In this mode, the display image of the HMD 20 is fixed, while the portable information terminal The image displayed on 100 is adjusted to connect as part of the image displayed on the HMD 20.
＜クランプモード＞
　このモードでは、図８にも示すように、携帯情報端末１００に表示される画像を固定し、ＨＭＤ２０の表示画像を、上記携帯情報端末１００の画像に接続するように調整される。なお、ここで、上記では「接続するように」としたが、しかしながら、必ずしも厳密に接続する必要はなく、多少の隙間や重なりがあっても構わない。 <Clamp mode>
In this mode, as shown in FIG. 8, the image displayed on the portable information terminal 100 is fixed, and the display image of the HMD 20 is adjusted to be connected to the image of the portable information terminal 100. Here, in the above description, “to be connected”, however, it is not always necessary to connect strictly, and there may be a slight gap or overlap.
　続いて、以下には、上記ＨＭＤ２０と携帯情報端末１００の表示について、複数のケースについて記載するが、ここでは、各ケースに共通の準備段階として、上述した表示連携に必要な情報である、即ち、携帯情報端末１００の画面位置の基準となる位置決めマーカと表示画面の位置関係の情報を、携帯情報端末１００からＨＭＤ２０に送信しておくこととする。その際、上述したように、位置決めマーカ１１０を赤外光源で構成すれば、他の形状と区別がし易く、表示画面位置の認識が正確に行える。その場合、ＨＭＤ２０の撮像部２０２は、赤外画像を可視光画像と区別して撮像出来るものであることが望ましい。 Subsequently, a plurality of cases will be described below regarding the display of the HMD 20 and the portable information terminal 100, but here, as a preparation stage common to each case, it is information necessary for the display cooperation described above. Suppose that information on the positional relationship between the positioning marker, which is a reference for the screen position of the portable information terminal 100, and the display screen is transmitted from the portable information terminal 100 to the HMD 20. At this time, as described above, if the positioning marker 110 is composed of an infrared light source, it can be easily distinguished from other shapes, and the display screen position can be accurately recognized. In that case, it is desirable that the imaging unit 202 of the HMD 20 can capture an infrared image separately from a visible light image.
　即ち、以下に述べる各ケースでは、上述した準備段階が終わっているものとする。さらに、表示連携を行っている期間は、ＨＭＤ２０においては、カメラ（撮像部２０２）にて位置決めマーカの位置を認識し、その情報から、携帯情報端末１００の画面位置を認識する。なお、この画面位置認識の変形例として、位置決めマーカによらず、例えば、画像認識手段を用いて携帯情報端末１００の画面位置を認識するものであっても構わない。また、さらに、上記の連携表示画像全体の内、携帯情報端末１００で表示する部分を「携帯情報端末表示領域」と呼ぶことにする。 That is, in each case described below, it is assumed that the above-described preparation stage has been completed. Furthermore, in the period during which display cooperation is performed, the HMD 20 recognizes the position of the positioning marker with the camera (imaging unit 202), and recognizes the screen position of the portable information terminal 100 from the information. As a modification of the screen position recognition, for example, the screen position of the portable information terminal 100 may be recognized using an image recognition unit without using the positioning marker. Furthermore, a portion displayed on the portable information terminal 100 in the above-described cooperative display image as a whole is referred to as a “portable information terminal display area”.
＜ケース１＞
　まず、連携表示画像データを、携帯情報端末１００からＨＭＤ２０に送る場合であり、かつ、上述した連携表示モードがリリースモードである場合について説明する。 <Case 1>
First, a case where the cooperative display image data is sent from the portable information terminal 100 to the HMD 20 and the above-described cooperative display mode is the release mode will be described.
　表示連携の開始時点でリリースモードの場合には、まず、携帯情報端末１００からＨＭＤ２０に対し、連携表示画像データを送信する。クランプモードからリリースモードに変更された場合には、その変更時点において、既に、ＨＭＤ２０に連携表示画像データが送信済であり、そのため、当該送信済のデータを使用する。 In the release mode at the start of display cooperation, first, the mobile information terminal 100 transmits the cooperation display image data to the HMD 20. When the clamp mode is changed to the release mode, the linked display image data has already been transmitted to the HMD 20 at the time of the change, and therefore the transmitted data is used.
　このリリースモードに入った後、ＨＭＤ２０では、携帯情報端末１００の画面位置を認識し、ＨＭＤ２０からは、携帯情報端末１００に対し、携帯情報端末１００の表示領域の情報を送信する。この画面位置が変更された場合は、新たな携帯情報端末表示領域情報を、ＨＭＤから、携帯情報端末１００に対して送信する。なお、このとき、ＨＭＤ２０から携帯情報端末１００に対して、ＨＭＤ２０の視野内における携帯情報端末画面位置情報をも併せて送信しても構わない。 After entering this release mode, the HMD 20 recognizes the screen position of the portable information terminal 100, and the HMD 20 transmits information on the display area of the portable information terminal 100 to the portable information terminal 100. When this screen position is changed, new portable information terminal display area information is transmitted from the HMD to the portable information terminal 100. At this time, the mobile information terminal screen position information within the field of view of the HMD 20 may also be transmitted from the HMD 20 to the mobile information terminal 100.
　このリリースモードに入っている期間は、図９にも示すように、ＨＭＤ２０の視野内では、連携表示画像は動かないが、ＨＭＤ２０では、連携表示画像の内、携帯情報端末１００の表示領域以外の部分のみ表示する。一方、携帯情報端末１００では、携帯情報端末表示領域の部分を表示する。なお、ＨＭＤ２０、携帯情報端末１００は、共に、アイコンや操作ボタン等、連携表示画像以外の画像を重ねて表示しても構わない。 As shown in FIG. 9, the linked display image does not move in the field of view of the HMD 20 during the release mode, but the HMD 20 does not move the display area other than the display area of the portable information terminal 100 in the linked display image. Display only the part. On the other hand, the portable information terminal 100 displays a portion of the portable information terminal display area. Note that both the HMD 20 and the portable information terminal 100 may display images other than the cooperative display images such as icons and operation buttons in an overlapping manner.
　その後、上記のリリースモードからクランプモードに変更した場合は、その時点における携帯情報端末１００の表示領域の情報を保持し、以下に述べるケース２のクランプモードの制御を行う。 After that, when the release mode is changed to the clamp mode, the information of the display area of the portable information terminal 100 at that time is held, and the clamp mode of case 2 described below is controlled.
　なお、このリリースモードの変形例としては、例えば、携帯情報端末１００では連携表示画像は表示せず、携帯情報端末１００の画面領域も含めて、ＨＭＤ２０側で連携表示画像を表示するようにしても構わない。この場合、携帯情報端末１００の画面は、黒表示でも構わないし、アイコンや操作ボタン等の連携表示画像以外の画像を表示しても構わない。 As a modified example of this release mode, for example, the mobile information terminal 100 does not display the cooperative display image, and the HMD 20 side displays the cooperative display image including the screen area of the mobile information terminal 100. I do not care. In this case, the screen of the portable information terminal 100 may be displayed in black, or an image other than the cooperative display image such as an icon or an operation button may be displayed.
＜ケース２＞
　連携表示画像データを携帯情報端末１００からＨＭＤ２０に送る場合であり、かつ、上記連携表示モードがクランプモードの場合である。 <Case 2>
This is a case where cooperative display image data is sent from the portable information terminal 100 to the HMD 20, and the cooperative display mode is the clamp mode.
　表示連携の開始時点でクランプモードの場合には、まず、携帯情報端末１００から、ＨＭＤ２０に対し、連携表示画像データと、携帯情報端末表示領域の情報を送信する。リリースモードからクランプモードに変更された場合は、当該変更時点では、既に、ＨＭＤ２０には連携表示画像データは送信済であり、そのため、当該送信済みのデータを使用する。また、携帯情報端末表示領域の情報は、ＨＭＤ２０と携帯情報端末１００において既に共有しているので、その情報を使用する。 In the clamp mode at the start of display cooperation, first, the portable information terminal 100 transmits the cooperation display image data and information on the portable information terminal display area to the HMD 20. When the release mode is changed to the clamp mode, the linked display image data has already been transmitted to the HMD 20 at the time of the change, and therefore the transmitted data is used. Moreover, since the information of the portable information terminal display area is already shared between the HMD 20 and the portable information terminal 100, the information is used.
　クランプモードに入った後、ＨＭＤ２０では、ＨＭＤ２０の視野内における携帯情報端末１００の画面位置を認識する。なお、この情報は、位置変更の都度、ＨＭＤ２０から携帯情報端末１００に送信しても構わない。 After entering the clamp mode, the HMD 20 recognizes the screen position of the portable information terminal 100 within the field of view of the HMD 20. This information may be transmitted from the HMD 20 to the portable information terminal 100 every time the position is changed.
　このクランプモードに入っている期間、携帯情報端末１００には、上記と同様に、携帯情報端末表示領域内の画像を表示する。 During the period of entering the clamp mode, the portable information terminal 100 displays an image in the portable information terminal display area in the same manner as described above.
　一方、ＨＭＤ２０では、連携表示画像の表示位置を、ＨＭＤ２０の視野内の携帯情報端末１００の画面位置に応じて調整し、携帯情報端末表示領域が携帯情報端末１００の画面位置に重なるように、携帯情報端末表示領域外の部分を表示する。そして、ＨＭＤ２０の視野内で携帯情報端末１００の位置が動いた場合は、新たな携帯情報端末１００の画面位置に応じてＨＭＤ２０の表示画像を調整する。 On the other hand, in the HMD 20, the display position of the cooperative display image is adjusted according to the screen position of the portable information terminal 100 within the field of view of the HMD 20, and the portable information terminal display area is overlapped with the screen position of the portable information terminal 100. The part outside the information terminal display area is displayed. And when the position of the portable information terminal 100 moves within the field of view of the HMD 20, the display image of the HMD 20 is adjusted according to the new screen position of the portable information terminal 100.
　また、図１０にも示すように、携帯情報端末１００において、指で画面をスライドするスワイプ動作等で、連携表示画像内の携帯情報端末表示領域の位置を変更した場合、あるいは、ピンチイン、ピンチアウト等の動作で携帯情報端末表示領域の大きさを変更した場合には、新たな携帯情報端末表示領域の情報を携帯情報端末１００からＨＭＤ２０に送信し、ＨＭＤ２０においては、連携表示画像の大きさ、位置を調整し、新しい携帯情報端末表示領域が携帯情報端末の画面位置に重なるように、携帯情報端末表示領域外の部分を表示する。 Also, as shown in FIG. 10, in the portable information terminal 100, when the position of the portable information terminal display area in the cooperative display image is changed by a swipe operation that slides the screen with a finger, etc., or pinch-in, pinch-out When the size of the portable information terminal display area is changed by the operation such as the above, information on the new portable information terminal display area is transmitted from the portable information terminal 100 to the HMD 20, and the HMD 20 The position is adjusted, and a portion outside the portable information terminal display area is displayed so that the new portable information terminal display area overlaps the screen position of the portable information terminal.
　また、携帯情報端末表示領域の位置や大きさの変更、ＨＭＤ２０視野内での携帯情報端末１００の位置の変更により、連携表示画像の境界が、ＨＭＤ２０の表示可能領域内に入る状態となることがある。この場合、新たな連携表示画像データを携帯情報端末１００からＨＭＤ２０に送信し、連携表示画像のデータ範囲を広げても構わない。なお、かかる制御を行う場合は、ＨＭＤ２０の視野内での携帯情報端末１００の位置情報が必要になるので、ＨＭＤ２０から携帯情報端末１００にその情報を送信する。 Moreover, the boundary of a cooperation display image may be in the state which can be in the displayable area | region of HMD20 by the change of the position and magnitude | size of a portable information terminal display area, and the change of the position of the portable information terminal 100 in HMD20 visual field. is there. In this case, new cooperative display image data may be transmitted from the portable information terminal 100 to the HMD 20 to expand the data range of the cooperative display image. In addition, when performing such control, since the positional information of the portable information terminal 100 within the field of view of the HMD 20 is necessary, the information is transmitted from the HMD 20 to the portable information terminal 100.
　なお、ＨＭＤ２０及び携帯情報端末１００は、共に、例えば、アイコンや操作ボタン等の連携表示画像以外の画像を重ねて表示しても構わない。変形例の一例として、図１１には、操作ボタンを携帯情報端末１００に表示している例を示す。 Note that both the HMD 20 and the portable information terminal 100 may display images other than the cooperative display images such as icons and operation buttons in a superimposed manner. As an example of a modification, FIG. 11 shows an example in which operation buttons are displayed on the portable information terminal 100.
　また、クランプモードからリリースモードに変更した場合は、その時点でのＨＭＤ２０の連携表示画像を維持し、上記ケース１で述べたリリースモードでの制御を行う。 Also, when changing from the clamp mode to the release mode, the cooperative display image of the HMD 20 at that time is maintained, and the control in the release mode described in the case 1 is performed.
　なお、このクランプモードの変形例としては、携帯情報端末１００では連携表示画像は表示せず、携帯情報端末１００の画面領域も含めて、ＨＭＤ側で連携表示画像を表示するようにしても構わない。ＨＭＤ側では、携帯情報端末１００の表示領域の更新、ＨＭＤ視野内での携帯情報端末１００の位置の更新に合わせて、連携表示画像の位置、大きさを調整する。この場合、携帯情報端末１００の画面は、黒表示でも構わないし、アイコンや操作ボタン等の連携表示画像以外の画像を表示しても構わない。 As a modification of the clamp mode, the cooperative display image may not be displayed on the portable information terminal 100, and the cooperative display image may be displayed on the HMD side including the screen area of the portable information terminal 100. . On the HMD side, the position and size of the cooperative display image are adjusted in accordance with the update of the display area of the portable information terminal 100 and the update of the position of the portable information terminal 100 within the HMD visual field. In this case, the screen of the portable information terminal 100 may be displayed in black, or an image other than the cooperative display image such as an icon or an operation button may be displayed.
＜ケース３＞
　連携表示画像データを、ＨＭＤ２０から携帯情報端末１００に送る場合であり、かつ、上記連携表示モードがリリースモードの場合である。 <Case 3>
This is a case where cooperative display image data is sent from the HMD 20 to the portable information terminal 100, and the cooperative display mode is a release mode.
　表示連携の開始時点でリリースモードの場合は、まず、ＨＭＤ２０から、携帯情報端末１００に対し、連携表示画像データを送信する。クランプモードからリリースモードに変更された場合は、当該変更時点では、既に、携帯情報端末１００には連携表示画像データが送信済であり、そのデータを使用する。 In the case of the release mode at the start of display cooperation, first, the cooperation display image data is transmitted from the HMD 20 to the portable information terminal 100. When the clamp mode is changed to the release mode, the cooperative display image data has already been transmitted to the portable information terminal 100 at the time of the change, and the data is used.
　リリースモードに入った後、ＨＭＤ２０では、携帯情報端末１００の画面位置を認識し、ＨＭＤ２０から携帯情報端末１００に対し、携帯情報端末表示領域の情報を送信する。この画面位置が変更された場合は、新たな携帯情報端末表示領域情報を、ＨＭＤ２０から携帯情報端末１００に対し送信する。なお、ＨＭＤ２０から携帯情報端末１００に対し、ＨＭＤ２０の視野内における携帯情報端末１００の画面位置情報を併せて送信しても構わない。 After entering the release mode, the HMD 20 recognizes the screen position of the portable information terminal 100 and transmits information of the portable information terminal display area from the HMD 20 to the portable information terminal 100. When this screen position is changed, new portable information terminal display area information is transmitted from the HMD 20 to the portable information terminal 100. Note that the screen position information of the portable information terminal 100 within the field of view of the HMD 20 may be transmitted together from the HMD 20 to the portable information terminal 100.
　このリリースモードに入っている期間、ＨＭＤ２０の視野内では、連携表示画像は動かないが、ＨＭＤ２０では、連携表示画像の内、携帯情報端末表示領域以外の部分のみを表示する。一方、携帯情報端末１００では、携帯情報端末表示領域の部分を表示する。 During the release mode, the cooperative display image does not move within the field of view of the HMD 20, but the HMD 20 displays only the portion other than the portable information terminal display area in the cooperative display image. On the other hand, the portable information terminal 100 displays a portion of the portable information terminal display area.
　なお、ＨＭＤ２０と携帯情報端末１００は、共に、アイコンや操作ボタン等、連携表示画像以外の画像を重ねて表示しても構わない。 Note that both the HMD 20 and the portable information terminal 100 may display images other than the cooperative display images such as icons and operation buttons in an overlapping manner.
　リリースモードからクランプモードに変更した場合は、その時点での携帯情報端末表示領域の情報を保持し、以下に述べるケース４のクランプモードの制御を行う。 When changing from the release mode to the clamp mode, the information of the portable information terminal display area at that time is held, and the clamp mode of case 4 described below is controlled.
　なお、このリリースモードの変形例として、携帯情報端末１００では連携表示画像は表示せず、携帯情報端末１００の画面領域も含めて、ＨＭＤ側で連携表示画像を表示するようにしても構わない。この場合、携帯情報端末１００の画面は、黒表示でも構わないし、アイコンや操作ボタン等の連携表示画像以外の画像を表示しても構わない。また、携帯情報端末１００で連携表示画像を表示しない場合には、連携表示画像データや、携帯情報端末表示領域の情報をＨＭＤ２０から携帯情報端末１００に送らなくても構わないが、しかしながら、その場合でも、携帯情報端末１００で、連携表示画像を対象とした操作を行う場合は、当該情報を送信する必要がある。 As a modified example of this release mode, the cooperative display image may be displayed on the HMD side including the screen area of the portable information terminal 100 without displaying the cooperative display image on the portable information terminal 100. In this case, the screen of the portable information terminal 100 may be displayed in black, or an image other than the cooperative display image such as an icon or an operation button may be displayed. Further, when the cooperative display image is not displayed on the portable information terminal 100, the cooperative display image data and the information of the portable information terminal display area may not be transmitted from the HMD 20 to the portable information terminal 100. However, in that case However, when the portable information terminal 100 performs an operation on the cooperation display image, the information needs to be transmitted.
＜ケース４＞
　連携表示画像データを、ＨＭＤ２０から携帯情報端末１００に送る場合であり、かつ、上記連携表示モードがクランプモードの場合である。 <Case 4>
This is the case where the cooperative display image data is sent from the HMD 20 to the portable information terminal 100, and the cooperative display mode is the clamp mode.
　表示連携の開始時点でクランプモードの場合は、まず、ＨＭＤ２０から、携帯情報端末１００に対し、連携表示画像データを送信する。そして、その時点でのＨＭＤ２０の視野内における携帯情報端末１００の画面位置から、携帯情報端末表示領域を決定し、この情報をＨＭＤ２０から携帯情報端末１００に送信する。リリースモードからクランプモードに変更された場合は、その変更時点で、既に、携帯情報端末には連携表示画像データが送信済なことから、そのデータを使用し、また、携帯情報端末表示領域の情報は、ＨＭＤ２０と携帯情報端末１００で共有しているので、その情報を使用する。 In the clamp mode at the start of display cooperation, first, the display image data is transmitted from the HMD 20 to the portable information terminal 100. Then, the mobile information terminal display area is determined from the screen position of the mobile information terminal 100 within the field of view of the HMD 20 at that time, and this information is transmitted from the HMD 20 to the mobile information terminal 100. When the release mode is changed to the clamp mode, the linked display image data has already been transmitted to the portable information terminal at the time of the change, so that data is used, and information on the portable information terminal display area is also used. Is shared between the HMD 20 and the portable information terminal 100, and uses that information.
　クランプモードに入った後、ＨＭＤ２０では、ＨＭＤの視野内における携帯情報端末１００の画面位置を認識する。なお、この情報は、位置変更の都度、ＨＭＤ２０から携帯情報端末１００に送信しても構わない。 After entering the clamp mode, the HMD 20 recognizes the screen position of the portable information terminal 100 within the field of view of the HMD. This information may be transmitted from the HMD 20 to the portable information terminal 100 every time the position is changed.
　クランプモードに入っている期間、携帯情報端末１００には携帯情報端末表示領域内の画像を表示する。一方、ＨＭＤ２０では、連携表示画像の表示位置をＨＭＤ２０の視野内の携帯情報端末１００の画面位置に応じて調整し、携帯情報端末表示領域が携帯情報端末１００の画面位置に重なるように、携帯情報端末表示領域外の部分を表示する。ＨＭＤ２０の視野内で携帯情報端末１００の位置が動いた場合は、新たな携帯情報端末の画面位置に応じて、ＨＭＤ２０の表示画像を調整する。 During the period of entering the clamp mode, the mobile information terminal 100 displays an image in the mobile information terminal display area. On the other hand, in the HMD 20, the display position of the cooperative display image is adjusted according to the screen position of the portable information terminal 100 within the field of view of the HMD 20, and the portable information terminal display area overlaps the screen position of the portable information terminal 100. The part outside the terminal display area is displayed. When the position of the portable information terminal 100 moves within the field of view of the HMD 20, the display image of the HMD 20 is adjusted according to the screen position of the new portable information terminal.
　また、携帯情報端末１００において、指で画面をスライドするスワイプ動作等で、連携表示画像内の携帯情報端末表示領域の位置を変更した場合、あるいは、ピンチイン、ピンチアウト等の動作で携帯情報端末表示領域の大きさを変更した場合には、新たな携帯情報端末表示領域の情報を、携帯情報端末１００からＨＭＤ２０に送信し、ＨＭＤ２０においては、連携表示画像の大きさ、位置を調整し、新しい携帯情報端末表示領域が携帯情報端末１００の画面位置に重なるように、携帯情報端末表示領域外の部分を表示する。 Further, in the portable information terminal 100, when the position of the portable information terminal display area in the cooperative display image is changed by a swipe operation of sliding the screen with a finger, or the portable information terminal is displayed by an operation such as pinch-in and pinch-out. When the size of the area is changed, information on a new portable information terminal display area is transmitted from the portable information terminal 100 to the HMD 20, and the HMD 20 adjusts the size and position of the linked display image to create a new portable information terminal. A portion outside the portable information terminal display area is displayed so that the information terminal display area overlaps the screen position of portable information terminal 100.
　また、携帯情報端末表示領域の位置や大きさの変更、ＨＭＤ２０視野内での携帯情報端末１００の位置の変更により、連携表示画像の境界が、ＨＭＤ２０の表示可能領域内に入る状態になることがある。この場合、新たな連携表示画像データをＨＭＤ２０から携帯情報端末１００に送信し、連携表示画像のデータ範囲を広げても構わない。 Moreover, the boundary of a cooperation display image may be in the state which can enter into the displayable area | region of HMD20 by the change of the position and magnitude | size of a portable information terminal display area, and the change of the position of the portable information terminal 100 in HMD20 visual field. is there. In this case, new cooperative display image data may be transmitted from the HMD 20 to the portable information terminal 100 to expand the data range of the cooperative display image.
　なお、ＨＭＤ２０と携帯情報端末１００は、共に、アイコンや操作ボタン等、連携表示画像以外の画像を重ねて表示しても構わない。 Note that both the HMD 20 and the portable information terminal 100 may display images other than the cooperative display images such as icons and operation buttons in an overlapping manner.
　また、クランプモードからリリースモードに変更した場合は、その時点でのＨＭＤ２０の連携表示画像を維持し、上記のケース３のリリースモードでの制御を行う。 Also, when changing from the clamp mode to the release mode, the cooperative display image of the HMD 20 at that time is maintained, and the control in the release mode of the above case 3 is performed.
　なお、上述した動作は、上記ＨＭＤ２０や携帯情報端末１００に内蔵された制御部１０３、２０３により、選択的に行われるものであり、例えば、当該制御部の一部に設けられたメモリ、又は、上記記憶部１０４や２０４に予め格納されたソフトウェアを実行することにより実現される。 The above-described operation is selectively performed by the control units 103 and 203 incorporated in the HMD 20 or the portable information terminal 100. For example, a memory provided in a part of the control unit, or This is realized by executing software stored in the storage unit 104 or 204 in advance.
　また、このクランプモードの変形例として、例えば、携帯情報端末１００では連携表示画像は表示せず、携帯情報端末１００の画面領域も含めて、ＨＭＤ２０側で連携表示画像を表示するようにしても構わない。ＨＭＤ２０側では、携帯情報端末表示領域の更新、ＨＭＤ視野内での携帯情報端末位置の更新に合わせて、連携表示画像の位置、大きさを調整する。この場合、携帯情報端末１００の画面は、黒表示でも構わないし、アイコンや操作ボタン等の連携表示画像以外の画像を表示しても構わない。携帯情報端末１００で連携表示画像を表示しない場合は、連携表示画像データや、携帯情報端末表示領域の情報をＨＭＤ２０から携帯情報端末１００に送らなくても構わないが、なお、その場合でも、携帯情報端末１００で連携表示画像を対象とした操作を行う場合は、当該情報を送信する必要がある。 Further, as a modification of the clamp mode, for example, the cooperative display image is not displayed on the portable information terminal 100, and the cooperative display image is displayed on the HMD 20 side including the screen area of the portable information terminal 100. Absent. On the HMD 20 side, the position and size of the cooperative display image are adjusted in accordance with the update of the portable information terminal display area and the update of the position of the portable information terminal within the HMD visual field. In this case, the screen of the portable information terminal 100 may be displayed in black, or an image other than the cooperative display image such as an icon or an operation button may be displayed. When the cooperative display image is not displayed on the portable information terminal 100, the cooperative display image data and the information on the portable information terminal display area may not be transmitted from the HMD 20 to the portable information terminal 100. When the information terminal 100 performs an operation on the cooperation display image, the information needs to be transmitted.
　以上に詳細に述べたケースでは、連携表示画像データを、携帯情報端末１００からＨＭＤ２０に送る場合と、ＨＭＤ２０から携帯情報端末１００に送る場合に分けて説明した。しかしながら、当該データが、携帯情報端末１００やＨＭＤ２０に記憶されていたものではなく、インターネット経由で取得したものでも構わない。さらに、インターネット経由の取得先を、携帯情報端末１００からＨＭＤ２０、あるいは、ＨＭＤ２０から携帯情報端末１００へ送信することをもって、連携表示画像データの送信とする場合もこれに含まれる。 In the case described in detail above, the cooperation display image data is divided into the case of sending from the portable information terminal 100 to the HMD 20 and the case of sending from the HMD 20 to the portable information terminal 100. However, the data may not be stored in the portable information terminal 100 or the HMD 20 but may be acquired via the Internet. Further, this includes a case where the acquisition destination via the Internet is transmitted from the portable information terminal 100 to the HMD 20 or from the HMD 20 to the portable information terminal 100 to transmit the cooperative display image data.
　上述した実施例では、携帯情報端末１００とＨＭＤ２０により、その一例として地図を表示する例について詳細に説明したが、しかしながら、本発明はこれにのみ限定されるものではなく、更には、図１２にも示すように、複数ページ画像で構成されるメニュー画像を有する携帯情報端末１００において、その一部（例えば、１ページ）のメニュー画像を、携帯情報端末１００の表示画面に表示させ、他のページの画像は、ＨＭＤ２０の表示画面に表示し、即ち、メガネ型表示デバイスでスマホの画面と並べて表示させることも可能である。この場合には、例えば、図１３に示すように、携帯情報端末１００の表示画面上でスワイプ操作することにより、携帯情報端末１００の表示画面と共に、ＨＭＤ２０の表示画面をも一緒に移動することが可能となる。又は、図１４にも示すように、ＨＭＤ２０の表示画面の一部を、携帯情報端末１００の表示画面上に移動させることも可能となる。 In the above-described embodiment, an example in which a map is displayed as an example by the portable information terminal 100 and the HMD 20 has been described in detail. However, the present invention is not limited to this example, and FIG. As also shown, in the portable information terminal 100 having a menu image composed of a plurality of page images, a part of the menu image (for example, one page) is displayed on the display screen of the portable information terminal 100 and another page is displayed. Can be displayed on the display screen of the HMD 20, that is, displayed side by side with the screen of the smartphone with a glasses-type display device. In this case, for example, as shown in FIG. 13, by swiping on the display screen of the portable information terminal 100, the display screen of the HMD 20 can be moved together with the display screen of the portable information terminal 100. It becomes possible. Alternatively, as shown in FIG. 14, a part of the display screen of the HMD 20 can be moved on the display screen of the portable information terminal 100.
　加えて、図１５にも示すように、ＨＭＤ２０の表示画面に、例えば、ＰＣのデスクトップ表示画面をすることも可能である。なお、この図では、デスクトップ表示画面には、例えば、ワードプロセッサのウィンドウＷが開くと共に、複数のボタンＢが設けられている。そして、図１６（Ａ）及び（Ｂ）にも示すように、携帯情報端末１００上でスワイプ操作することにより、デスクトップ上のウィンドウＷの位置を移動することが可能となっている。 In addition, as shown in FIG. 15, for example, a desktop display screen of a PC can be displayed on the display screen of the HMD 20. In this figure, on the desktop display screen, for example, a word processor window W is opened and a plurality of buttons B are provided. Then, as shown in FIGS. 16A and 16B, the position of the window W on the desktop can be moved by performing a swipe operation on the portable information terminal 100.
　なお、本発明は上記した実施例に限定されるものではなく、様々な変形例が含まれる。例えば、上記した実施例は本発明を分かりやすく説明するためにシステム全体を詳細に説明したものであり、必ずしも説明した全ての構成を備えるものに限定されるものではない。また、ある実施例の構成の一部を他の実施例の構成に置き換えることが可能であり、また、ある実施例の構成に他の実施例の構成を加えることも可能である。また、各実施例の構成の一部について、他の構成の追加・削除・置換をすることが可能である。 In addition, this invention is not limited to the above-mentioned Example, Various modifications are included. For example, the above-described embodiments are described in detail for the entire system in order to explain the present invention in an easy-to-understand manner, and are not necessarily limited to those having all the configurations described. Further, a part of the configuration of one embodiment can be replaced with the configuration of another embodiment, and the configuration of another embodiment can be added to the configuration of one embodiment. Further, it is possible to add, delete, and replace other configurations for a part of the configuration of each embodiment.
　また、上記の各構成、機能、処理部、処理手段等は、それらの一部又は全部を、例えば集積回路で設計する等によりハードウェアで実現してもよい。また、上記の各構成、機能等は、プロセッサがそれぞれの機能を実現するプログラムを解釈し、実行することによりソフトウェアで実現してもよい。各機能を実現するプログラム、テーブル、ファイル等の情報は、メモリや、ハードディスク、ＳＳＤ（Ｓｏｌｉｄ　Ｓｔａｔｅ　Ｄｒｉｖｅ）等の記録装置、または、ＩＣカード、ＳＤカード、ＤＶＤ等の記録媒体に置くことができる。 In addition, each of the above-described configurations, functions, processing units, processing means, and the like may be realized by hardware by designing a part or all of them with, for example, an integrated circuit. Each of the above-described configurations, functions, and the like may be realized by software by interpreting and executing a program that realizes each function by the processor. Information such as programs, tables, and files that realize each function can be stored in a memory, a hard disk, a recording device such as an SSD (Solid State Drive), or a recording medium such as an IC card, an SD card, or a DVD.
　１００…携帯情報端末装置、１０１…データ通信部、１０２…操作入力部、１０３…制御部、１０４…記憶部、１０５…画像表示部、１１０…位置決めマーカ、２０…ヘッドマウントディスプレイ（ＨＭＤ）、２００…携帯情報端末、２０１…データ通信部、２０２…撮像部、２０３…制御部、２０４…記憶部、２０５…画像表示部 DESCRIPTION OF SYMBOLS 100 ... Portable information terminal device, 101 ... Data communication part, 102 ... Operation input part, 103 ... Control part, 104 ... Memory | storage part, 105 ... Image display part, 110 ... Positioning marker, 20 ... Head mounted display (HMD), 200 DESCRIPTION OF SYMBOLS ... Portable information terminal, 201 ... Data communication part, 202 ... Imaging part, 203 ... Control part, 204 ... Memory | storage part, 205 ... Image display part

Claims (3)

  1. 　ヘッドマウントディスプレイと連携して画像を表示することが可能な携帯情報端末であって、
    　画像を表示する画像表示部と、
    　前記ヘッドマウントディスプレイとのデータ通信を行うデータ通信部と、
    　操作入力部と、
    　前記データ通信部と前記操作入力部からの制御信号に基づいて、前記画像表示部に画像を表示する制御部とを備えており、
    　前記制御部は、前記ヘッドマウントディスプレイと連携して前記画像表示部に表示する画像の一部を、前記ヘッドマウントディスプレイで表示される画像と一連の画像として連続するように表示し、かつ、
    　前記制御部は、選択的に、前記ヘッドマウントディスプレイの視野内で当該携帯情報端末の位置を動かした時に、当該携帯情報端末で表示する画像を固定するモードと、前記ヘッドマウントディスプレイで表示する画像を固定するモードとの間で切り替える機能を備えていることを特徴とする携帯情報端末。     A portable information terminal capable of displaying an image in cooperation with a head-mounted display,
    An image display unit for displaying an image;
    A data communication unit for performing data communication with the head mounted display;
    An operation input section;
    A control unit for displaying an image on the image display unit based on a control signal from the data communication unit and the operation input unit;
    The control unit displays a part of an image displayed on the image display unit in cooperation with the head mounted display so as to be continuous with an image displayed on the head mounted display, and
    The control unit selectively fixes an image displayed on the portable information terminal when the position of the portable information terminal is moved within the field of view of the head mounted display, and an image displayed on the head mounted display. A portable information terminal having a function of switching between a mode for fixing a mobile phone.
  2. 　前記請求項１に記載した携帯情報端末において、更に、前記制御部は、タッチ操作又はスワイプ操作により、一連の画像を動かすことを特徴とする携帯情報端末。 The portable information terminal according to claim 1, wherein the control unit moves a series of images by a touch operation or a swipe operation.
  3. 　携帯情報端末と連携して画像を表示することが可能なヘッドマウントディスプレイであって、
    　前記ヘッドマウントディスプレイを頭部に装着した人間に画像を表示する画像表示部と、
    　前記携帯情報端末とのデータ通信を行うデータ通信部と、
    　撮像部と、
    　前記データ通信部と前記撮像部からの制御信号に基づいて、前記画像表示部に画像を表示する制御部とを備えており、
    　前記制御部は、前記携帯情報端末と連携して表示する画像の一部を、前記携帯情報端末で表示される画像と連続するように表示し、かつ、
    　前記制御部は、選択的に、当該ヘッドマウントディスプレイの視野内で前記携帯情報端末の位置を動かした時に、前記携帯情報端末で表示する画像を固定するモードと、当該ヘッドマウントディスプレイで表示する画像を固定するモードとの間で切り替える機能を備えていることを特徴とするヘッドマウントディスプレイ。     A head mounted display capable of displaying an image in cooperation with a portable information terminal,
    An image display unit for displaying an image to a human wearing the head-mounted display on the head;
    A data communication unit for performing data communication with the portable information terminal;
    An imaging unit;
    A control unit for displaying an image on the image display unit based on a control signal from the data communication unit and the imaging unit;
    The control unit displays a part of an image displayed in cooperation with the portable information terminal so as to be continuous with an image displayed on the portable information terminal, and
    The control unit selectively fixes an image displayed on the portable information terminal when the position of the portable information terminal is moved within the field of view of the head mounted display, and an image displayed on the head mounted display. A head mounted display characterized by having a function of switching between modes for fixing the screen.
PCT/JP2014/068343 2014-07-09 2014-07-09 Portable information terminal device and head-mount display linked thereto WO2016006070A1 (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
PCT/JP2014/068343 WO2016006070A1 (en) 2014-07-09 2014-07-09 Portable information terminal device and head-mount display linked thereto

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
PCT/JP2014/068343 WO2016006070A1 (en) 2014-07-09 2014-07-09 Portable information terminal device and head-mount display linked thereto

Publications (1)

Publication Number Publication Date
WO2016006070A1 true WO2016006070A1 (en) 2016-01-14

Family

ID=55063745

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/JP2014/068343 WO2016006070A1 (en) 2014-07-09 2014-07-09 Portable information terminal device and head-mount display linked thereto

Country Status (1)

Country Link
WO (1) WO2016006070A1 (en)

Cited By (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2016130978A (en) * 2015-01-14 2016-07-21 コニカミノルタ株式会社 Operation display device and operation display system
WO2017156954A1 (en) * 2016-03-16 2017-09-21 北京小米移动软件有限公司 Video live broadcast method and device
WO2020003361A1 (en) * 2018-06-25 2020-01-02 マクセル株式会社 Head-mounted display, head-mounted display linking system, and method for same
TWI844970B (en) * 2021-12-14 2024-06-11 宏達國際電子股份有限公司 Method for operating mirrored content under mirror mode and computer readable storage medium

Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2008059511A (en) * 2006-09-04 2008-03-13 Brother Ind Ltd E-mail browsing system and e-mail browsing method using portable terminal device and head-mounted display
US20110077903A1 (en) * 2009-09-30 2011-03-31 Samsung Electronics Co., Ltd. Apparatus and method sensing motion
JP2011186856A (en) * 2010-03-09 2011-09-22 Nec Corp Mobile terminal to be used with head mounted display as external display device
JP2013161267A (en) * 2012-02-06 2013-08-19 Sony Corp Information processing device, information processing method and program

Patent Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2008059511A (en) * 2006-09-04 2008-03-13 Brother Ind Ltd E-mail browsing system and e-mail browsing method using portable terminal device and head-mounted display
US20110077903A1 (en) * 2009-09-30 2011-03-31 Samsung Electronics Co., Ltd. Apparatus and method sensing motion
JP2011186856A (en) * 2010-03-09 2011-09-22 Nec Corp Mobile terminal to be used with head mounted display as external display device
JP2013161267A (en) * 2012-02-06 2013-08-19 Sony Corp Information processing device, information processing method and program

Cited By (8)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2016130978A (en) * 2015-01-14 2016-07-21 コニカミノルタ株式会社 Operation display device and operation display system
WO2017156954A1 (en) * 2016-03-16 2017-09-21 北京小米移动软件有限公司 Video live broadcast method and device
WO2020003361A1 (en) * 2018-06-25 2020-01-02 マクセル株式会社 Head-mounted display, head-mounted display linking system, and method for same
US11378805B2 (en) 2018-06-25 2022-07-05 Maxell, Ltd. Head-mounted display, head-mounted display linking system, and method for same
US11567333B2 (en) 2018-06-25 2023-01-31 Maxell, Ltd. Head-mounted display, head-mounted display linking system, and method for same
US11921293B2 (en) 2018-06-25 2024-03-05 Maxell, Ltd. Head-mounted display, head-mounted display linking system, and method for same
US12169281B2 (en) 2018-06-25 2024-12-17 Maxell, Ltd. Head-mounted display, head-mounted display linking system, and method for same
TWI844970B (en) * 2021-12-14 2024-06-11 宏達國際電子股份有限公司 Method for operating mirrored content under mirror mode and computer readable storage medium

Similar Documents

Publication Publication Date Title
US10591729B2 (en) Wearable device
US9377873B2 (en) Flexible glass display apparatus and method for controlling the same
US9904360B2 (en) Head tracking based gesture control techniques for head mounted displays
JP6251957B2 (en) Display device, head-mounted display device, and display device control method
US9500867B2 (en) Head-tracking based selection technique for head mounted displays (HMD)
US20180217680A1 (en) Wearable device, control method, and control code
US20150054730A1 (en) Wristband type information processing apparatus and storage medium
KR20140115906A (en) Display device detecting gaze location and method for controlling thereof
CN104866262B (en) Wearable device
CN104793842B (en) Graphical user interface system, display processing device and input processing device
KR20170048069A (en) System and method for controlling the same
KR20160099399A (en) Watch type terminal
EP2978218A1 (en) Computer display device mounted on eyeglasses
WO2017018388A1 (en) Wearable device, control method, and control program
JP2015176588A (en) Display device, image display system and information processing method
JP2017508193A (en) User configurable speech commands
KR20160125674A (en) Mobile terminal and method for controlling the same
KR20170140702A (en) Mobile terminal
WO2016006070A1 (en) Portable information terminal device and head-mount display linked thereto
JP6740613B2 (en) Display device, display device control method, and program
US20160026244A1 (en) Gui device
JP6638392B2 (en) Display device, display system, display device control method, and program
KR101622695B1 (en) Mobile terminal and control method for the mobile terminal
JP6999822B2 (en) Terminal device and control method of terminal device
JP2016139174A (en) Head mounted display, head mounted display control method and control program

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 14897249

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

NENP Non-entry into the national phase

Ref country code: JP

122 Ep: pct application non-entry in european phase

Ref document number: 14897249

Country of ref document: EP

Kind code of ref document: A1

点击 这是indexloc提供的php浏览器服务，不要输入任何密码和下载