WO2018167844A1

WO2018167844A1 - Head-up display device and image display method thereof

Info

Publication number: WO2018167844A1
Application number: PCT/JP2017/010206
Authority: WO
Inventors: 望下田; 裕司藤田
Original assignee: マクセル株式会社
Priority date: 2017-03-14
Filing date: 2017-03-14
Publication date: 2018-09-20

Abstract

Provided are an excellent head-up display device and an image display method thereof in which a natural display suitable for a situation in a space can be achieved even when a display object is displayed three-dimensionally. This head-up display device, which displays a virtual image of a display object for a driver in a space in front of a vehicle by projecting an image of the display object in front of the vehicle, is provided with: a vehicle information acquisition unit for acquiring various kinds of vehicle information which can be detected by the vehicle; a control unit for controlling the displaying of the display object in the space on the basis of the vehicle information acquired by the vehicle information acquisition unit; and an image display device for forming the display object on the basis of an instruction from the control unit, wherein the vehicle information acquisition unit includes a space depth input means for inputting depth information on a front space of the vehicle, and the control unit has a function of controlling the displaying of a part of the display object displayed in the space on the basis of the depth information on the space.

Description

Head-up display device and image display method thereof

The present invention relates to a technology for a head-up display device, and more particularly to a technology effective when applied to a head-up display device capable of mapping and displaying a destination guidance object on a road using AR (Augmented Reality). is there.

In vehicles such as automobiles, information such as vehicle speed and engine speed is usually displayed on an instrument panel in the dashboard. A screen such as a car navigation is incorporated in the dashboard or displayed on a display installed on the dashboard. Since it is necessary to move the line of sight largely when the driver visually recognizes such information, as a technique for reducing the amount of movement of the line of sight, the front of information such as information on the vehicle speed and the direction of the traveling direction of the vehicle is used. 2. Description of the Related Art A head-up display (Head-Up Display, which may be referred to as “HUD” below) that projects and displays a virtual image on glass (wind shield) is known.

For example, in Japanese Patent Application Laid-Open No. 2007-55365 (Patent Document 1), when displaying a distance scale on a HUD, the travel point information that the host vehicle is currently traveling from the map data of the navigation device, and the host vehicle Information on the planned travel point where the vehicle will travel, and based on this information, obtain the inclination angle of the road on which the vehicle travels, and use the correction coefficient according to the inclination angle to display the distance scale from the ground. It is described that the image is corrected and displayed.

Japanese Patent Laid-Open No. 2006-7867 (Patent Document 2) describes the display position of the generated image according to the detected driving situation such as right / left turn, acceleration / deceleration, and the like when left turn is detected. It is described that control is performed such as shifting to the right direction and shifting to the right direction when a right turn is detected.

JP 2007-55365 A JP 2006-7867 A

HUD projects the image on a windshield or combiner to make the video image recognized by the driver as a virtual image outside the vehicle. On the other hand, a so-called AR function that makes it possible to show information related to an object or the like to the driver by displaying a virtual image superimposed on an actual scenery (space) outside the vehicle seen through a windshield or combiner. There is known a HUD that realizes (hereinafter, may be referred to as “AR-HUD”). Even in such an AR-HUD, it is necessary to make adjustments for maintaining the visibility and appropriateness of the display image in accordance with the traveling state of the vehicle.

In this regard, for example, by using the technology described in

Patent Documents

1 and 2 described above, the influence of the vehicle on the visibility and appropriateness of the display image (virtual image) is reduced or eliminated according to the driving situation. It is possible to make it. On the other hand, in these techniques, the display position, display content, and the like are adjusted in accordance with the driving situation for an image displayed in a virtual image display area in the HUD. However, in the above prior art, when an object such as a destination guide is displayed superimposed on a road, there is a problem that an unnatural display as described in detail in the following embodiments occurs.

Therefore, in the present invention, there is provided a method and an apparatus capable of superimposing and displaying a virtual image in conformity with an actual space according to the traveling state of the vehicle, for example, a destination guidance object ( An excellent head-up display device capable of natural display suitable for the foreground situation and an image display method therefor are provided, particularly when an object is displayed three-dimensionally in a display space as indicated by arrows). For the purpose. The above and other objects and novel features of the present invention will be apparent from the description of this specification and the accompanying drawings.

Of the inventions disclosed in this application, the outline of typical ones will be briefly described as follows.

A head-up display device according to an exemplary embodiment of the present invention projects a display object image onto a windshield or combiner of a vehicle, thereby providing a virtual image of the display object in a space in front of the vehicle to a driver. A head-up display device that displays various vehicle information that can be detected by the vehicle, and based on the vehicle information acquired by the vehicle information acquisition unit, in the space A control unit that controls display of the display object; and an image display device that forms the display object based on an instruction from the control unit. The vehicle information acquisition unit includes a front space of the vehicle. Spatial depth input means for inputting the depth information, and the control unit, based on the spatial depth information, A head-up display apparatus having a function of controlling a part of display of the display object to be displayed on the serial space.

In addition, the image display method of the head-up display device according to the representative embodiment of the present invention includes various types of vehicles that can be detected by the vehicle, including vehicle front image input means for inputting depth information of the front space of the vehicle. A vehicle information acquisition unit that acquires information, a control unit that controls display of a display object based on the vehicle information acquired by the vehicle information acquisition unit, and the display object that is formed based on an instruction from the control unit A head-up display device that displays a virtual image superimposed on the scenery ahead of the vehicle by projecting the display object onto a windshield or combiner of the vehicle An image display method according to claim 1, wherein a display of a part of the display object to be displayed in the space is input to the vehicle front image. An image display method of control based on the depth information of the space from the stage.

Among the inventions disclosed in the present application, effects obtained by typical ones will be briefly described as follows. In other words, according to the representative embodiment of the present invention, a natural display suitable for the foreground situation can be obtained even when an object is displayed three-dimensionally in the display space, such as a destination guidance object indicating the traveling direction of the vehicle. And an image display method therefor are provided.

It is the figure which showed the outline | summary about the example of the operation | movement concept of the head-up display apparatus in one embodiment of this invention. It is the functional block diagram which showed the outline | summary about the structural example of the whole head-up display apparatus which is one embodiment of this invention. It is the figure which showed the outline | summary about the example of the hardware constitutions which concern on acquisition of the vehicle information in one embodiment of this invention. It is the functional block diagram which showed the detail about the structural example of the head-up display apparatus in one embodiment of this invention. It is the flowchart which showed the outline | summary about the example of the initial stage operation in one embodiment of this invention. It is the flowchart which showed the outline | summary about the example of the normal operation | movement in one embodiment of this invention. It is the flowchart which showed the outline | summary about the example of the brightness level adjustment process in one embodiment of this invention. (A), (b) is a figure explaining the display which considered the obstruction and the display which considered the obstruction in one embodiment of this invention. (A), (b), (c) is a figure shown about Z buffer obtained from the viewpoint in 3D space. It is a figure which shows an example of the depth map which actually calculated the depth from the parallax map in one embodiment of this invention. It is the flowchart which showed the detail of the display image change and production | generation process of the system 1 in one embodiment of this invention. It is explanatory drawing which showed the display image change and production | generation process of the system 1 in one embodiment of this invention. It is the flowchart which showed the detail of the display image change and production | generation process of the system 2 in one embodiment of this invention. It is explanatory drawing which showed the detail of the display image change and production | generation process of the system 2 in one embodiment of this invention. (A), (b) is a figure which shows the result of the image display process using the drawing Z buffer and space Z buffer of this invention. It is explanatory drawing which showed the content of the function expansion 1 (interlocking with an audio | voice) in one embodiment of this invention. (A), (b) is explanatory drawing which showed the content of the function expansion 2 (ON / OFF function of a depth map) in one embodiment of this invention. It is explanatory drawing which showed the content of the function extension 3 (area definition function which hides an object / area | region which does not hide) in one embodiment of this invention. It is explanatory drawing which showed the display video change and production | generation process in the case of combining the function expansion 3 in one embodiment of this invention with said system 1. FIG. It is explanatory drawing which showed the display image change and production | generation process in the case of combining the function expansion 3 in one embodiment of this invention with said system 2. FIG. It is explanatory drawing which showed the content of the function expansion 5 (luminance change function) in one embodiment of this invention. It is explanatory drawing which showed the content of the function expansion 6 (chromaticity change function) in one embodiment of this invention. It is explanatory drawing which showed the content of the modification of the drawing essential area | region in one embodiment of this invention. It is the figure shown about the operation | movement concept of the display distance adjustment in other embodiment of this invention. It is the figure shown about the example of the hardware constitutions which concern on the display distance adjustment in other embodiment of this invention.

Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings. Note that components having the same function are denoted by the same reference symbols throughout the drawings for describing the embodiment, and the repetitive description thereof will be omitted. On the other hand, parts described with reference numerals in some drawings may be referred to with the same reference numerals although not illustrated again in the description of other drawings.

Example 1
Prior to the description of the image display method according to the present invention, first, an outline of a head-up display device to which the present invention is applied will be described below.

FIG. 1 is a diagram showing an outline of an example of an operation concept of a head-up display device according to an embodiment of the present invention (Example 1). In the AR-HUD 1 of the present embodiment, an image displayed on the image display device 30 such as a projector or an LCD (Liquid Crystal Display) is converted into a mirror 51 or a mirror 52 (for example, a free-form surface mirror or an optical axis asymmetric shape). And is projected onto the windshield 3 of the vehicle 2.

The driver 5 views the image projected as a virtual image in front of the transparent windshield 3 by viewing the image projected on the windshield 3. In the present embodiment, the display position of the virtual image seen by the driver 5 can be adjusted in the vertical direction by adjusting the position of the image projected onto the windshield 3 by adjusting the angle of the mirror 52. is there. Then, the AR function is realized by adjusting the display position and the display distance so that the virtual image is superimposed on the scenery outside the vehicle (roads, buildings, people, etc.).

Here, the member to be projected is not limited to the windshield 3 and may be another member such as a combiner as long as the image is projected. The video display device 30 is configured by, for example, a projector having a backlight, an LCD (Liquid Crystal Display), or the like. A self-luminous VFD (Vacuum Fluorescent Display) or the like may be used.

FIG. 2 is a functional block diagram showing an outline of an overall configuration example of the head-up display device according to the embodiment of the present invention. The AR-HUD 1 mounted on the vehicle 2 includes, for example, a vehicle information acquisition unit 10, a control unit 20, a video display device 30, a display distance adjustment mechanism 40, a mirror driving unit 50, a mirror 52, and a speaker 60. In the example of FIG. 2, the shape of the vehicle 2 is displayed like a passenger car. However, the shape is not limited to this, and can be applied as appropriate to general vehicles.

The vehicle information acquisition unit 10 includes information acquisition devices such as various sensors, which will be described later, installed in each part of the vehicle 2, detects various events occurring in the vehicle 2, and relates to the driving situation at predetermined intervals. The vehicle information 4 is acquired and output by detecting and acquiring values of various parameters. The vehicle information 4 includes, for example, speed information, gear information, steering wheel steering angle information, lamp lighting information, external light information, distance information, infrared information, engine ON / OFF information, and camera video information of the vehicle 2 as illustrated. (Inside / outside the vehicle), acceleration gyro information, GPS (Global Positioning System) information, navigation information, vehicle-to-vehicle communication information, road-to-vehicle communication information, and the like may be included.

The control unit 20 has a function of controlling the operation of the AR-HUD 1, and is implemented by, for example, a CPU (Central Processing Unit) and software executed thereby. It may be implemented by hardware such as a microcomputer or FPGA (Field Programmable Gate Gate Array). As shown in FIG. 2, the control unit 20 drives the video display device 30 to form an image to be displayed as a virtual image based on the vehicle information 4 acquired from the vehicle information acquisition unit 10, and this is mirrored. The light is projected onto the windshield 3 by being appropriately reflected by 52 or the like. Then, control such as adjusting the display position of the virtual image display area or adjusting the display distance of the virtual image is performed by a method described later.

As described above, the video display device 30 is a device configured by, for example, a projector or an LCD, and forms a video for displaying a virtual image based on an instruction from the control unit 20 and projects or displays the video. To do. The display distance adjustment mechanism 40 is a mechanism for adjusting the distance of the displayed virtual image from the driver 5 based on an instruction from the control unit 20.

The mirror driving unit 50 adjusts the angle of the mirror 52 based on an instruction from the control unit 20 and adjusts the position of the virtual image display area in the vertical direction. The adjustment of the position of the virtual image display area will be described later. The speaker 60 performs audio output related to the AR-HUD1. For example, voice guidance of the navigation system, voice output when notifying the driver 5 by the AR function, etc. can be performed.

FIG. 3 is a diagram showing an outline of an example of a hardware configuration related to acquisition of the vehicle information 4 in the head-up display device of the present embodiment. Here, a part of the hardware configuration of the vehicle information acquisition unit 10 and the control unit 20 will be mainly shown. The vehicle information 4 is acquired by an information acquisition device such as various sensors connected to the ECU 21 under the control of an ECU (Electronic Control Unit) 21, for example.

As these information acquisition devices, for example, a vehicle speed sensor 101, a shift position sensor 102, a steering wheel steering angle sensor 103, a headlight sensor 104, an illuminance sensor 105, a chromaticity sensor 106, a distance measuring sensor 107, an infrared sensor 108, an engine start sensor 109, acceleration sensor 110, gyro sensor 111, temperature sensor 112, road-to-vehicle communication wireless receiver 113, vehicle-to-vehicle communication wireless receiver 114, camera (inside the vehicle) 115, camera (outside the vehicle) 116, GPS receiver 117, and Each device includes a VICS (Vehicle Information and Communication System: a road traffic information communication system, registered trademark (hereinafter the same)) receiver 118 and the like. It is not always necessary to include all these devices, and other types of devices may be included. The vehicle information 4 that can be acquired by the provided device can be used as appropriate.

The vehicle speed sensor 101 acquires speed information of the vehicle 2. The shift position sensor 102 acquires current gear information of the vehicle 2. The steering wheel angle sensor 103 acquires steering wheel angle information. The headlight sensor 104 acquires lamp lighting information related to ON / OFF of the headlight. The illuminance sensor 105 and the chromaticity sensor 106 acquire external light information. The distance measuring sensor 107 acquires distance information between the vehicle 2 and an external object. The infrared sensor 108 acquires infrared information related to the presence / absence and distance of an object at a short distance of the vehicle 2. The engine start sensor 109 detects engine ON / OFF information.

The acceleration sensor 110 and the gyro sensor 111 acquire acceleration gyro information including acceleration and angular velocity as information on the posture and behavior of the vehicle 2. The temperature sensor 112 acquires temperature information inside and outside the vehicle. The road-to-vehicle communication wireless receiver 113 and the vehicle-to-vehicle communication wireless receiver 114 are respectively road-to-vehicle communication information received by road-to-vehicle communication between the vehicle 2 and roads, signs, signals, etc. The vehicle-to-vehicle communication information received by the vehicle-to-vehicle communication with another vehicle is acquired.

The camera (inside the vehicle) 115 and the camera (outside the vehicle) 116 respectively capture the moving image of the situation inside and outside the vehicle and acquire camera video information (inside / outside the vehicle). The camera (inside the vehicle) 115 captures, for example, the posture of the driver 5, the position of the eyes, the movement, and the like. By analyzing the obtained moving image, it is possible to grasp, for example, the fatigue status of the driver 5 and the position of the line of sight. In addition, the camera (outside the vehicle) 116 captures a situation around the vehicle 2 such as the front or rear. By analyzing the obtained video, for example, it is possible to grasp the presence or absence of moving objects such as other vehicles and people around the building, topography, road surface conditions (rain, snow, freezing, unevenness, etc.) It is.

The GPS receiver 117 and the VICS receiver 118 obtain GPS information obtained by receiving the GPS signal and VICS information obtained by receiving the VICS signal, respectively. It may be implemented as a part of a car navigation system that acquires and uses these pieces of information.

FIG. 4 is a functional block diagram showing details of a configuration example of the head-up display device of the present embodiment. The example of FIG. 4 shows a case where the video display device 30 is a projector, and the video display device 30 includes, for example, each unit such as a light source 31, an illumination optical system 32, and a display element 33. The light source 31 is a member that generates illumination light for projection. For example, a high-pressure mercury lamp, a xenon lamp, an LED (Light-Emitting-Diode) light source, a laser light source, or the like can be used. The illumination optical system 32 is an optical system that collects the illumination light generated by the light source 31 and irradiates the display element 33 with more uniform illumination light. The display element 33 is an element that generates an image to be projected. For example, a transmissive liquid crystal panel, a reflective liquid crystal panel, a DMD (Digital Micromirror Device) panel, or the like can be used.

More specifically, the control unit 20 includes an ECU 21, an audio output unit 22, a nonvolatile memory 23, a memory 24, a light source adjustment unit 25, a distortion correction unit 26, a display element drive unit 27, a display distance adjustment unit 28, and a mirror adjustment. Each part such as the part 29 is included. As shown in FIG. 3, the ECU 21 acquires the vehicle information 4 via the vehicle information acquisition unit 10, and records, stores, and reads out the acquired information in the nonvolatile memory 23 and the memory 24 as necessary. To do. The nonvolatile memory 23 may store setting information such as setting values and parameters for various controls. Further, the ECU 21 generates video data relating to a virtual image to be displayed as the AR-HUD 1 by executing a dedicated program.

The audio output unit 22 outputs audio information via the speaker 60 as necessary. The light source adjustment unit 25 adjusts the light emission amount of the light source 31 of the video display device 30. When there are a plurality of light sources 31, they may be controlled individually. When the video generated by the ECU 21 is projected onto the windshield 3 of the vehicle 2 by the video display device 30, the distortion correction unit 26 corrects the video distortion caused by the curvature of the windshield 3 by image processing. The display element drive unit 27 sends a drive signal corresponding to the video data corrected by the distortion correction unit 26 to the display element 33 to generate an image to be projected.

When the position of the virtual image display area itself needs to be adjusted, the mirror adjustment unit 29 changes the angle of the mirror 52 via the mirror driving unit 50 and moves the virtual image display area up and down.

<Processing in head-up display device>
FIG. 5 is a flowchart showing an outline of an example of the initial operation of the head-up display device of the present embodiment. When the power of the AR-HUD 1 is turned on by turning on the ignition switch in the vehicle 2 that is stopped (S01), the AR-HUD 1 starts with the vehicle information acquisition unit 10 based on an instruction from the control unit 20. Thus, vehicle information is acquired (S02). Then, the control unit 20 calculates a suitable brightness level based on external light information acquired by the illuminance sensor 105, the chromaticity sensor 106, and the like in the vehicle information 4 (S03), and the light source adjustment unit 25 calculates the light source 31. Is set so that the calculated brightness level is obtained (S04). For example, when the outside light is bright, the brightness level is set high, and when the outside light is dark, the brightness level is set low.

Thereafter, the ECU 21 determines and generates a video (for example, an initial image) to be displayed as a virtual image (S05), and performs a process of correcting the distortion by the distortion correction unit 26 for the generated video (S06). The display element 33 is driven and controlled by the element driving unit 27 to form a projected image (S07). Thereby, an image | video is projected on the windshield 3, and the driver | operator 5 comes to be able to visually recognize a virtual image.

When the start-up / start-up of each part including the above-described series of initial operations is completed in the entire AR-HUD 1, the HUD-ON signal is output. The control part 20 determines whether or not this signal has been received (S08). . If not received, the HUD-ON signal is further waited for a predetermined time (S09), and the HUD-ON signal waiting process (S09) is repeated until it is determined in step S11 that the HUD-ON signal has been received. If it is determined in step S11 that the HUD-ON signal has been received, normal operation of AR-HUD1 described later is started (S10), and a series of initial operations are terminated.

FIG. 6 is a flowchart showing an outline of an example of normal operation of the head-up display device of the present embodiment. Also in the normal operation, the basic processing flow is substantially the same as the initial operation shown in FIG. First, the AR-HUD 1 acquires vehicle information by the vehicle information acquisition unit 10 based on an instruction from the control unit 20 (S21). And the control part 20 performs a brightness level adjustment process based on the external light information acquired by the illumination intensity sensor 105, the chromaticity sensor 106, etc. among the vehicle information 4 (S22).

FIG. 7 is a flowchart showing an outline of an example of brightness level adjustment processing of the head-up display device of the present embodiment. When the brightness level adjustment process is started, first, a suitable brightness level is calculated based on the acquired outside light information (S221). Then, by comparing with the currently set brightness level, it is determined whether or not the brightness level needs to be changed (S222). If no change is necessary, the brightness level adjustment process is terminated. On the other hand, when the change is necessary, the light source adjustment unit 25 controls the light emission amount of the light source 31 to set the brightness level after the change (S223), and the brightness level adjustment process is ended. . In step S222, even when there is a difference between the preferred brightness level calculated in step S221 and the currently set brightness level, the brightness level is only when the difference is equal to or greater than a predetermined threshold. It may be determined that the change is necessary.

Returning to FIG. 6, the video displayed as a virtual image is changed as necessary from the current one based on the latest vehicle information 4 acquired in step S <b> 21, and the changed video is determined and generated. (S23). In addition, the pattern which changes a display content based on the vehicle information 4 can have many things according to the content of the acquired vehicle information 4, those combinations, etc. For example, when the speed information changes, the value of the speed display that is displayed at all times is changed, the guidance arrow graphic is displayed / erased based on the navigation information, and the arrow shape and display position are changed. There may be various patterns, such as when doing.

Thereafter, in the present embodiment, adjustment / correction processing is performed to maintain visibility, appropriateness of display contents, and the like according to the traveling state of the vehicle 2. First, when it is necessary to adjust the position of the virtual image display area itself, the mirror adjustment process is performed to change the angle of the mirror 52 via the mirror driving unit 50 and move the virtual image display area up and down (S24). . Thereafter, a vibration correction process for correcting the display position of the image in the display area with respect to the vibration of the vehicle 2 is performed (S25).

Thereafter, the distortion correction unit 26 performs distortion correction processing on the adjusted / corrected image (S26), and then the display element driving unit 27 drives and controls the display element 33 to form a projected image ( S27).

When the above-described series of normal operations are being performed, if the power is turned off when the vehicle 2 is stopped or the like, a HUD-OFF signal is output to the AR-HUD 1. It is determined whether or not this signal has been received (S28). If the HUD-OFF signal has not been received, the process returns to step S21, and a series of normal operations are repeated until the HUD-OFF signal is received. If it is determined that the HUD-OFF signal has been received, a series of normal operations is terminated.

As described above, in the head-up display device configured as described above, a part of the memory area for storing information used in various arithmetic processes executed in the ECU 21 that constitutes the control unit 20 shown in FIG. According to a drawing buffer (for example, a color buffer) constituting the AR function, it is possible to realize an AR function by appropriately superimposing a virtual image on a front landscape (hereinafter, “drawing space” or “space”).

<Drawing control using depth information>
On the other hand, in the above-described head-up display device, depending on an object that is drawn and displayed (hereinafter also referred to as a “drawing target” or a “display object”), a virtual image is naturally displayed with more reality (reality). There are cases where the object is expressed in a three-dimensional (three-dimensional) manner (3D expression). At this time, for example, as shown in FIG. 8A, the drawing target (in this example, a destination guidance arrow) is displayed as it is even though there is an obstacle such as a building in front of it. Indicates that the drawing target is displayed on an obstacle in the space (note the arrow shown in the broken-line rectangle in the figure). In other words, in this example, we are trying to show that the road between buildings is turning to the right, but in fact, as shown in the figure, the arrow is superimposed on the building in front of the point turning to the right. End up. Such a drawing target should originally be hidden by the building, but when displayed on an obstacle, it will indicate a right turn despite the presence of the building, which is unnatural, There is also a risk of losing the driver's sense of distance.

Therefore, in the present invention, in order to solve the above-described problem, as shown in FIG. 8B, when the obstacle is in front, the drawing target to be hidden by the obstacle is not displayed, or the The present invention provides a controllable video display method for suppressing display. According to this, drawing of a more natural and realistic object is realized, and the driver can correctly grasp the sense of distance. Hereinafter, a method for realizing this will be described in detail.

[Display Object Depth (Z Buffer for Drawing)]
In the embodiment described in detail below, the information indicating the depth of the drawing target is, for example, used when 3D representation (conversion) of the drawing target, which is a display object, and the depth information for each pixel. Z buffer (drawing Z buffer) having In the following, a case where drawing is inhibited (suppressed control) using the drawing Z buffer will be described.

[Get foreground distance information (depth map or space Z buffer)]
On the other hand, the distance (hereinafter referred to as “depth”) between an obstacle present in the foreground of the vehicle (hereinafter also simply referred to as “space”), which is a space in which the display object is displayed as a virtual image, in part of the vehicle (hereinafter referred to as “depth”) 3), the distance measuring sensor 106 and the camera (outside the vehicle) 115 shown in FIG. 3 are used, or other stereo cameras and color aperture imaging technology (for example, https://www.toshiba. co.jp/rdc/detail/1606_01.htm), a depth sensor using infrared rays, LiDAR, and the like. When a stereo camera is used, an image taken from two viewpoints has a shift (hereinafter referred to as “parallax”) according to the distance of the subject. Since there is an inversely proportional relationship between the amount of parallax and the depth, the depth can be obtained from the parallax. In addition, what imaged parallax is called a parallax map (disparity map).

In the present invention, these foreground distance information (depth map) is also used after being converted into the same format as the drawing Z buffer, and the foreground converted into this Z buffer format having depth information for each pixel. The distance information (depth map) is hereinafter also referred to as “space Z buffer”.

In the image display method of the present invention, when drawing a drawing target such as a destination guide as an arrow on the display element 33 (see FIG. 4) as a projection video generation element, the drawing Z buffer is used. In addition, the display object is drawn with reference to the space Z buffer storing the foreground distance information (depth map) as necessary.

At that time, it is determined by referring to the drawing Z buffer whether or not the display object that is the new drawing object is in front of the display object of the previous drawing. If it is, the drawing is not performed. In addition, in the present invention, it is determined whether or not the drawing target is in front of an obstacle in the foreground, and as a result, the memory area stores the foreground depth information, and has the depth information for each pixel. When it is determined that an obstacle is in front with reference to the space Z buffer, the display of the display object is controlled. Specifically, drawing is not performed, or suppression processing such as changing the color or expression is performed.

Here, FIG. 9 shows a Z buffer generally obtained from a viewpoint in 3D space. In the Z buffer, the pixel that draws the display object (in this case, a cube) that is the drawing target in the foreground has a smaller value. For example, it is shown brighter in this figure, and the larger the value, the deeper the position. Means that. For example, as shown in FIGS. 9A and 9B, when a new drawing object is drawn between two drawing objects on the left and right in the figure, the Z buffer after the drawing of the new object is As shown in FIG. That is, the newly drawn display object in the center of the figure is brighter (that is, the value is smaller) in the portion that overlaps the drawn display object (drawing target) in the upper left of the figure, and the lower right of the figure The portion that overlaps the drawn display object becomes darker (that is, the value is larger). In other words, in the Z buffer after the new display object is drawn, the pixel in the foreground is displayed with a smaller (brighter) value, whereas when the new display object is in the foreground, the depth of the new display object is increased. On the other hand, if the old display object is in front, the depth is not updated.

The space Z buffer described above can also be obtained by the same principle as described above, and an example of actually calculating the depth from the parallax map in this way (hereinafter referred to as “depth map (depth map)”). Is shown in FIG. As a method for obtaining a parallax map, there are a block matching method, a semi-global block matching method, and the like.

Subsequently, a display video change / generation process for changing / generating a video of a display object using the above-described Z buffer (depth information) (specifically, a drawing Z buffer and a space Z buffer) will be described below. .

[Method 1]
FIG. 11 and FIG. 12 show a display video change / generation process according to method 1 and an explanatory diagram thereof.

First, when the display video change / generation process is started in the flow shown in FIG. 11, it is determined whether or not the foreground depth map setting is selected, that is, “ON” (step S31). As a result, if the foreground depth map is not set (“No”), whether or not drawing of all images has been completed in the drawing buffer (for example, a color buffer) shown in FIG. Is determined (step S32). More specifically, the Z value of the drawing target pixel is acquired from the drawing Z buffer (step S33) and compared with the drawing Z value (step S34). If the result is “Yes”, the drawing target pixel is drawn in the drawing buffer (color buffer) (step S35). On the other hand, if “No”, no drawing is performed. These steps are repeated until all drawing is completed. This eliminates an unnatural overlap between a plurality of display objects.

On the other hand, when the foreground depth map is set (“Yes”), as described above, distance information is acquired from a depth map or the like obtained by photographing the front of the vehicle with a stereo camera or the like (step) S36). Thereafter, it is converted into a Z buffer format (space Z buffer), and the contents are registered as an initial value in the drawing required area of the drawing Z buffer (step S37). That is, as shown in FIG. 12, the depth map obtained from the foreground photographed by a stereo camera or the like is converted into the Z buffer format (foreground space Z buffer) and set in the drawing Z buffer.

Thereafter, it is determined whether or not all drawing of the video in the drawing buffer (color buffer) has been completed (step S38). If the result is “Yes”, the process ends. On the other hand, if the result is “No”, the Z value of the drawing target pixel is acquired from the set space Z buffer (step S39). The value in the Z buffer is compared with the Z value (drawing Z value) of the drawing target pixel (step S40).

As a result, if the drawing Z value <the Z buffer value, that is, “Yes”, the drawing object is positioned in front of the forward-facing obstacle, and the pixel is drawn in the drawing buffer (color buffer) as it is. On the other hand, in the case of “No”, the drawing target is drawn while being suppressed as being hidden by the foreground obstacle (step S42). As an example of this suppression processing, for example, when changing the luminance, the value (RGB) of each color is lowered, and when changing the chromaticity, the chromaticity is changed and the pixel is drawn (color buffer). ) To draw. Alternatively, it may be possible not to draw the pixels in the drawing buffer (color buffer). According to such processing, as shown in the lower part of FIG. 12, the drawing in the drawing buffer (color buffer) is completed, and the contents are displayed on the head-up display device.

[Method 2]
FIG. 13 and FIG. 14 show a display video change / generation process to be method 2 and an explanatory diagram thereof. In this method 2, comparison is made for each pixel, and if there is a hidden pixel, the color information of the corresponding pixel in the drawing buffer is set to “0” and deleted.

First, when the display video change / generation process is started in the flow shown in FIG. 13, it is determined whether or not all drawing of the video in the drawing buffer (color buffer) has been completed (step S41). If the result is “No”, the Z value of the drawing target pixel is obtained from the drawing Z buffer (step S42), and it is determined whether or not the drawing Z value <the drawing Z buffer value (step S43). Only when “Yes”, the drawing target pixel is drawn in the drawing buffer (color buffer) (step S44), and the same process is executed until all the drawing is completed.

Thereafter, it is determined whether or not the depth map setting is “ON” (step S45). If the result is “No”, the process is terminated.

On the other hand, when the depth map is set (“Yes”), the above-described distance information is acquired (step S46), converted into the Z buffer format, and used as the foreground space Z buffer (step S47). Thereafter, a comparison process is performed in the same manner as described above (step S48). In this comparison processing, for each pixel, the value of the drawing Z buffer is compared with the Z value derived from the depth map that is the space Z buffer, and if the value of the drawing Z buffer is larger (the space Z buffer (Z value <drawing Z buffer value), the same suppression processing as described above is executed for the pixel. Specifically, the luminance is changed by deleting the pixel or lowering the value (RGB) of each color. Or when changing chromaticity, chromaticity is changed. As a result, the drawing target is processed as if it is hidden by an obstacle in the foreground. As a result, as shown in the lower part of FIG. Drawing in the color buffer) is completed, and the contents are displayed on the head-up display device.

The image drawing is performed by using the 3D drawing library (for example, “OpenGL” or the like) supporting the Z buffer by the control unit 20 or the ECU 21 shown in FIGS. 2 and 3. .

FIG. 15 shows an example of the result of the image display process using the drawing Z buffer and the space Z buffer described above. As is apparent from the figure, when displaying a semi-ring-shaped display object on the foreground to alert the driver to the pedestrian, the entire semi-ring-shaped display object is usually left as it is. Is displayed. Therefore, it is also displayed on the guardrail located in front of it (see the broken line part in FIG. 15A), which gives the driver a sense of incongruity. On the other hand, according to the above-described drawing suppression using depth information, an obstacle (in this example, a guardrail) is present in front of the display object as shown in FIG. When the part is to be hidden by the guardrail, the part to be hidden is deleted (or the brightness or chromaticity is changed) and displayed. As a result, it is possible to display the display object by the head-up display device with a more natural appearance, which is solved by the driver feeling unnatural and losing a sense of distance. .

Subsequently, the details of the function expansion of the display image change / generation process described in detail above will be described below.

[Function expansion 1 (linked with audio)]
When the object is hidden, it is linked with voice notification (generation of complementary information by voice linkage).

As shown in FIG. 16, when an obstacle such as a person or a vehicle is present in front of the host vehicle at a corner or the like, most of the drawn object is hidden. If this occurs, there is a risk that the timing of turning at a corner will be missed. Therefore, when the object is hidden, a voice notification is always performed. In the example shown in the figure, an announcement is made that “turn right after 150 m”. In addition, when the display object is hidden, the synchronization with the sound is always ON, and it is possible to advance the sound output timing (compared to when it is not hidden) or increase the sound output frequency. It is. It should be noted that those skilled in the art can realize this function by driving the speaker 60 by the control unit 20 in FIG. 2 using video information from the camera (outside the vehicle) 115 shown in FIG. It will be clear if there is any.

[Function expansion 2 (depth map ON / OFF function)]
As shown in FIG. 17A, depending on the vehicle ahead, most of the drawn display object may be hidden. In such a case, the depth map derived from the foreground is provided with an ON / OFF function (addition of an ON / OFF function). Accordingly, it is preferable to hide the display when the distance is long, and to display the drawn display object when approaching, that is, to change the control according to the distance to the event. According to this, as shown in FIG. 17 (b), it is possible to display the display object regardless of the presence of the vehicle ahead. It could also be done.

In addition, as an implementation method, ON / OFF of the depth map is set to “ON” when there is a distance (time) to an event such as a corner, and the depth map is set to “OFF” (forced display) when approaching. Alternatively, the driver may be able to freely turn on / off on the setting screen or the like. Further, as a process when the depth map is set to “OFF”, in the case of the above-described method 1, the foreground depth map (space Z buffer) derived from the stereo camera must be the initial value of the drawing Z buffer. In the case of method 2, the foreground depth map (space Z buffer) derived from the stereo camera may not be compared with the drawing Z buffer.

[Function Extension 3 (Function to define object hidden / not hidden)]
This function extension 3 defines an area that hides / does not hide a display object when it overlaps with the foreground. For example, when a front car is detected by a front camera, that part is forcibly drawn. It is an area.

For example, as shown in FIG. 18, when an obstacle such as a car is present in front, a phenomenon occurs in which most of the drawn display object is hidden. In that case, an area that hides the display object and an area that is not hidden (hereinafter referred to as “drawing-required area”: the gray part in the figure) are defined, and ON / OFF of the display can be set for each area. To do. For example, what percentage or more of the display object may be displayed, or when 75% is displayed, it is displayed as it is, or when only 50% is displayed, 75% is displayed. Some major objects may be displayed on the screen. On the other hand, the region other than the gray portion at the center of the drawing, which is the drawing-required region described above, is a region that hides the display object when it overlaps the foreground.

FIG. 19 shows the contents of the display video change / generation process when this function extension 3 is combined with the above method 1. That is, the foreground depth map is converted into the Z buffer format (spatial Z buffer), and then the drawing-required area is initialized. The subsequent processing is the same as in the above-described method 1, and is described in detail here. Do not do.

FIG. 20 shows the contents of the display video change / generation process when this function extension 3 is combined with the above method 2. That is, after the depth map is converted into the Z buffer format (foreground space Z buffer), the drawing essential area is initialized. The subsequent processing is the same as in the above-described method 2, and will not be described in detail here.

[Function expansion 5 (brightness change function)]
This is a function for changing the brightness of an object hidden in the foreground. As shown in FIG. 21, even when an obstacle such as a car or a person is present in front, most of the drawn display object is hidden. However, this hidden part is drawn with its brightness lowered.

[Function expansion 6 (Chromaticity change function)]
This is a function for changing the chromaticity of an object hidden in the foreground. As shown in FIG. 22, when a car or a person is present in front, a phenomenon in which most of the drawn display object is hidden occurs. However, the hidden portion is drawn by changing the chromaticity of the object. This function may be combined with the brightness adjustment described above. Furthermore, the range for changing the chromaticity may be limited.

As a method for realizing the above-described function expansion 5 (brightness change function) and function expansion 6 (chromaticity change function), the driver can freely turn on / off the brightness and chromaticity change on the setting screen or the like. Alternatively, ON / OFF automatic control may be performed such that it is turned on when there is a distance (time) to an event such as a corner, and turned off when approaching. In the case of the above-described method 2, when the display object is in the back in the comparison between the space Z buffer that is the foreground depth map and the drawing Z buffer, For example, the luminance of the pixel is reduced to 50% (when the luminance is changed) or the chromaticity is changed (when the chromaticity is changed). On the other hand, in the case of the method 1, when there is no overlap between the display object and the obstacle, or when drawing in the order from the back to the front, the same as the above, the display object and the obstacle These extended functions are not applied when a display object that overlaps and is behind is drawn.

[Modification of drawing required area]
As described above, when an obstacle such as a person or a vehicle is present in the front, and as a result, a phenomenon in which most of the drawn display object is hidden occurs, the display object is displayed without being hidden. A modification in which the “drawing essential area” is realized without using the foreground space Z buffer described above will be described below.

As shown in FIG. 23, the front lane is recognized by the camera, and the own vehicle traveling scheduled area is directly obtained. Or you may acquire the own vehicle travel plan area | region from map information, such as car navigation. Note that drawing is required in the above area, and drawing is not performed when the 3D coordinates of the drawing scheduled object are within the drawing required area. On the other hand, if it is outside the drawing essential area, drawing is performed using the

above method

1 or 2.

As described in detail above, according to the image display method having the drawing suppression function applying depth information according to the present invention, the object displayed as a virtual image is more suitable for the space in which the virtual image is displayed. For example, when displaying an object displayed with a depth (depth) on the display screen, such as a destination guidance object (arrow) indicating the traveling direction of the vehicle. However, it is possible to perform natural display on the road, and an excellent head-up display device is realized.

(Example 2)
In the embodiment described in detail above, the object displayed by the head-up display device is described as being displayed as a virtual image after being converted as a stereoscopic image and drawn on the color buffer. However, the present invention can also be applied to the case where a display object to be displayed is displayed by adjusting the display distance by a head-up display device having a display distance adjustment mechanism. .

[Display distance adjustment mechanism]
Hereinafter, an embodiment provided with a mechanism for displaying an object displayed by the head-up display device by adjusting the display distance will be described. The outline of the head-up display device is the same as described above, and a description thereof will be omitted. Only the characteristic points will be described below.

FIG. 24 is a diagram showing an overview of an example of display distance adjustment using a movable diffusion plate in the head-up display device of the second embodiment. In this example, the image projected from the projector 30a is imaged by a movable diffuser plate (movable diffuser) 41c and then incident on the mirror 52 via the lens 42a.

As shown in FIG. 25, the movable diffusion plate 41c is further rotated by being controlled by the display distance adjustment unit 28 provided with the diffusion plate control unit 201 and the display distance adjustment mechanism 40 provided with the diffusion control mechanism 401. Can be made. The display position of the virtual image based on the image projected from the projector 30a is determined according to the inclination of the movable diffusion plate 41c and the lens 42a. Therefore, by rotating the movable diffusion plate 41c, the display distance of the virtual image can be changed by changing the focal length. Specifically, the display distance of the virtual image can be shortened by tilting the movable diffuser plate 41c to a position close to the lens 42a. Conversely, if the movable diffuser plate 41c is tilted away from the lens 42a, the display distance of the virtual image can be increased. For example, the display object is displayed with the distance set based on the Z value of the drawing Z buffer storing the depth. At that time, by drawing the display object with reference to the space Z buffer storing the distance information (depth map) of the foreground, the distance can be changed and displayed. It is possible to realize display of a display object using a virtual image.

At that time, according to the image display method for performing the drawing suppression applying the depth information described above, it is possible to execute the image display method so that the display object is not displayed overlapping the obstacle in the foreground. Display is possible.

In the above example, the example in which the movable diffuser is used as the mechanism for displaying the object by adjusting the display distance is described. However, the present invention is not limited to this example. For films, films with multiple mirrors, movable lenses, dimming mirrors, movable optical filters, comb optical filters, etc. Is available.

As described in detail above, according to the video display method having a drawing suppression function applying depth information according to the present invention, the object displayed as a virtual image is more suitable for the space where the virtual image is displayed. For example, an object displayed with a depth (depth) on the display screen, such as a destination guidance object indicating the traveling direction of the vehicle such as an arrow, can be displayed naturally. In the case of display, an excellent head-up display device that can be optimally displayed on the road is realized.

As mentioned above, the invention made by the present inventor has been specifically described based on the embodiments. However, the present invention is not limited to the above-described embodiments, and various modifications can be made without departing from the scope of the invention. Needless to say. For example, the above-described embodiment has been described in detail for easy understanding of the present invention, and is not necessarily limited to the one having all the configurations described. In addition, it is possible to add, delete, and replace other configurations for a part of the configuration of the above-described embodiment.

DESCRIPTION OF SYMBOLS 1 ... AR-HUD, 2 ... Vehicle, 3 ... Wind shield, 4 ... Vehicle information, 5 ... Driver, 6 ... Display area, 10 ... Vehicle information acquisition part, 20 ... Control part, 21 ... ECU, 22 ... Sound output , 23 ... Non-volatile memory, 24 ... Memory, 25 ... Light source adjustment part, 26 ... Distortion correction part, 27 ... Display element drive part, 28 ... Display distance adjustment part, 29 ... Mirror adjustment part, 30 ... Video display device, 30a ... projector, 30b ... LCD, 31 ... light source, 32 ... illumination optical system, 33 ... display element,
DESCRIPTION OF SYMBOLS 40 ... Display distance adjustment mechanism, 41c ... Movable diffuser plate, 42a ... Lens, 52 ... Mirror, 60 ... Speaker, 107 ... Ranging sensor, 108 ... Infrared sensor, 116 ... Camera (outside vehicle), 117 ... GPS receiver.

Claims

A head-up display device that displays a virtual image of the display object in a space in front of the vehicle to the driver by projecting an image of the display object in front of the vehicle,
A vehicle information acquisition unit that acquires various types of vehicle information that the vehicle can detect;
A control unit that controls display of the display object in the space based on the vehicle information acquired by the vehicle information acquisition unit;
An image display device that forms the display object based on an instruction from the control unit;
Have
The vehicle information acquisition unit includes spatial depth input means for inputting depth information of a front space of the vehicle,
The said control part is a head-up display apparatus which has a function which controls the display of a part of said display object displayed in the said space based on the depth information of the said space.
The head-up display device according to claim 1,
The control unit has a function of displaying the display object in a three-dimensional manner, and depth information in the case of inputting depth information for displaying the display object in a three-dimensional manner from the spatial depth input means. A head-up display device that controls display of a part of the display object in comparison with information.
The head-up display device according to claim 2,
The image display device includes a display element composed of a plurality of pixels,
The depth information for stereoscopically displaying the display object includes depth information of the display object corresponding to each pixel of the display element,
The depth information of the space ahead of the vehicle includes depth information of the space ahead of the vehicle corresponding to each pixel of the display element,
The control unit stores a memory region for drawing an image of the display object, a memory region for storing depth information of the display object, and depth information of the vehicle front space, together with a calculation processing unit that performs predetermined calculation processing. A head-up display device including a memory area.
The head-up display device according to claim 2,
The head-up display device further includes means for generating complementary information by sound when the control unit controls display of a part of the display object.
The head-up display device according to claim 2,
The display control of a part of the display object by the control unit includes at least one of deletion of a part of the display object, reduction in luminance, and change in chromaticity.
The head-up display device according to claim 1,
An area for controlling display of a part of the display object in the image display device can be set in advance.
The head-up display device according to claim 1, wherein the image display device includes a drawing essential region for displaying a part of the display object regardless of the depth information of the space from the space depth input unit. Head-up display device.
A vehicle information acquisition unit that acquires various types of vehicle information that can be detected by the vehicle, including a vehicle front image input unit that inputs depth information of a front space of the vehicle, and the vehicle information acquired by the vehicle information acquisition unit. A control unit that controls display of the display object based on the image display device, and a video display device that forms the display object based on an instruction from the control unit, and projects the display object onto the windshield of the vehicle. Thus, an image display method of a head-up display device that displays a virtual image superimposed on a landscape in front of the vehicle for a driver,
The image display method which controls a display of a part of said display object displayed in the said space based on the depth information of the said space from the said vehicle front image input means.
The image display method of the head-up display device according to claim 8,
The display object is stereoscopically displayed in the space, and the depth information when the display object is stereoscopically displayed is compared with the depth information of the space input from the spatial depth input means. An image display method for controlling display of a part of the display object.
The image display method of the head-up display device according to claim 9,
The display control of a part of the display object is an image display method in which suppression control including at least one of deletion of part of the display object, change of luminance, and change of chromaticity is performed.