JP7685981B2 - Image display device and image display method - Google Patents

Description

This disclosure relates to an image display device and an image display method.

An operator who is unfamiliar with remotely controlling the moving object being operated cannot get a feel for the body of the moving object, and therefore has difficulty remotely controlling the moving object using only images captured in real time by a camera mounted on the moving object. Therefore, a technology has been proposed that improves the operability of the moving object by superimposing an image of the moving object on an image that allows the current moving object to be viewed objectively from among past images captured by a camera (for example, Patent Document 1).

JP 2010-128935 A

However, when an operator remotely controls a moving object indoors where there are many obstacles, the image viewed from a viewpoint that is a certain distance away from the rear of the moving object may not be desirable for the operator.

For example, consider a case where an operator remotely controls a moving object to turn right, move to the back of an indoor corridor, and then return, while viewing an image seen from a viewpoint position a certain distance behind the moving object. In this case, there is a problem that the moving object cannot be displayed immediately after the moving object turns right due to obstacles such as walls or shelves that make up the corridor. In addition, when attempting to maintain a certain distance when the moving object returns, the viewpoint position changes significantly, making it difficult for the operator to grasp the situation of the moving object. Thus, in order to perform remote control indoors where there are many obstacles, it was necessary to optimize the objective viewpoint position that serves as the basis for displaying the moving object and its surroundings.

In particular, for mobile objects that perform tasks not only remotely but also autonomously, it is envisioned that the operator will remotely control the mobile object only when overcoming a situation in which the mobile object cannot perform tasks autonomously. In such operations, the operator does not monitor the mobile object until remote operation begins, so images viewed from an optimized, objective viewpoint are required to properly grasp the status of the mobile object when remote operation begins.

Therefore, this disclosure has been made in consideration of the above-mentioned problems, and aims to provide technology that can optimize the objective viewpoint position.

The image display device according to the present disclosure includes an acquisition unit that acquires a request for remote operation of a moving body, work information indicating a work including movement of the moving body, and a captured image captured from the moving body; a work position calculation unit that calculates the work position of the moving body when the request is acquired based on the request and the work information; a captureable area calculation unit that calculates an area in which the captured image including an image of the work position can be captured as a captureable area based on work environment information including map information indicating the work space of the moving body and obstacle information regarding obstacles in the work space and the work position; a first viewpoint position extraction unit that extracts a position included in the captureable area as a viewpoint position from among positions within a movement path in which the moving body has moved in the past based on the work information and the captureable area; a superimposed image generation unit that generates a superimposed image by superimposing a corresponding position image indicating the work position on the captured image captured at the viewpoint position based on the captured image, the work position, and the viewpoint position; and a display unit that displays the superimposed image.

According to the present disclosure, an area in which a captured image including an image of the work position can be captured is calculated as the captureable area, and a position included in the captureable area among positions within a moving path traveled by a moving body in the past is extracted as a viewpoint position based on the work information and the captureable area, and a superimposed image is generated in which a corresponding position image indicating the work position is superimposed on the captured image captured at the viewpoint position based on the captured image, the work position, and the viewpoint position. With this configuration, it is possible to optimize the objective viewpoint position.

1 is a block diagram showing a configuration of an image display device according to a first embodiment; 11 is a bird's-eye view for explaining an example of a method for calculating a photographable area. FIG. FIG. 11 is a bird's-eye view for explaining an example of a method for extracting a viewpoint position. FIG. 13 is a diagram showing a display example according to a modified example. FIG. 13 is a diagram showing a display example according to a modified example. FIG. 11 is a block diagram showing a configuration of an image display device according to a second embodiment. 13 is a diagram showing an example of a screen for generating a work situation reproduction image according to the second embodiment; FIG. 11 is a block diagram showing a configuration of an image display device according to a third embodiment. FIG. 13 is a block diagram showing a configuration of an image display device according to a fourth embodiment. FIG. 13 is a diagram showing a display example according to the fourth embodiment; FIG. 13 is a block diagram showing a configuration of an image display device according to a fifth embodiment. FIG. 13 is a block diagram showing a configuration of an image display device according to a sixth embodiment. FIG. 13 is a block diagram showing a configuration of an image display device according to a seventh embodiment. FIG. 13 is a block diagram showing a configuration of an image display device according to an eighth embodiment. FIG. 13 is a block diagram showing a hardware configuration of an image display device according to another modified example. FIG. 13 is a block diagram showing a hardware configuration of an image display device according to another modified example.

<First embodiment>
Conventionally, in a factory or other environment, an environment that is easy for mobile robots to work in has been created, and fully automated operation of mobile robots has been realized by employees following rules. Examples of an environment that is easy for mobile robots to work in include a consistent floor color and illuminance, wide aisles with few blind spots, and no obstacles in the aisles. Examples of rules that employees follow include avoiding mobile robots and not occupying the movement path of mobile robots.

Meanwhile, the autonomous task performance of mobile robots is improving due to improvements in the performance of devices such as batteries and sensors that make up mobile robots, the widespread use of safety sensors, and improvements in situational judgment capabilities due to advances in AI (artificial intelligence) technology. As a result, there is an increasing demand for autonomous mobile robots that can work autonomously not only in factories but also in human workspaces, and as a result, the use of autonomous mobile robots that share workspaces with people is on the rise.

However, it is difficult for an autonomous mobile robot to achieve fully automated operation in a workspace designed for humans. For example, commercial facilities or general offices visited by customers are decorated in a way that leaves a good impression on people, and people who share the environment with an autonomous mobile robot often do not pay special consideration to the autonomous mobile robot. For this reason, it is difficult to anticipate in advance all the situations and countermeasures that an autonomous mobile robot will encounter, to the point where it can operate fully automated in a human workspace.

However, in order to prioritize the safety of people sharing the workspace, autonomous mobile robots often stop operating when they are unable to confirm safety due to insufficient information from sensors, or when they determine that there is a person in an area where contact is possible. Furthermore, even in unexpected situations, humans can more flexibly grasp the situation from image information and the like than autonomous mobile robots can, and therefore can appropriately grasp the risk of contact accidents. In addition, humans can alert people in the surrounding area and predict the movements of people nearby from video footage.

Taking these factors into consideration, an operation has been proposed in which an operator remotely controls an autonomous mobile robot to return it to work if the autonomous mobile robot gets into a situation where it cannot perform a task autonomously. There are many situations in which an operator can remotely control an autonomous mobile robot to return it to work, which reduces the number of situations and countermeasures that must be anticipated in advance, and such an operation can expand the range in which autonomous mobile robots can be applied even within human workspaces.

In this type of operation, since the operator has few opportunities to remotely control an autonomous mobile robot, it is expected that one person will remotely control multiple autonomous mobile robots as multiple operation targets. Since the work environments are diverse, it is important for the operator to be able to easily grasp the status of multiple autonomous mobile robots in order to appropriately perform remote operation such as recovery work on the autonomous mobile robots.

If the recovery work is for an autonomous vehicle traveling on a public road, rather than indoor recovery work, the operator can limit attention to the road, making it easier for the operator to grasp the situation, shape, and movement characteristics of the autonomous vehicle. Therefore, the operator can easily grasp the situation of the autonomous vehicle with just first-person perspective information obtained from on-board sensors such as LiDER (Light Detection and Ranging) or an omnidirectional camera.

However, in an indoor environment, there are many different objects to pay attention to, and the shapes and movement characteristics of the objects being controlled vary widely, making it difficult for an operator to grasp the situation, shape, and movement characteristics of the object being controlled from a first-person perspective image captured by a camera mounted on the object being controlled. For this reason, in order for an operator to easily grasp the situation of the object being controlled, it is preferable to have an image that includes an image of the object being controlled and its surroundings, viewed from an objective viewpoint, rather than an image viewed from the object's own viewpoint, such as a first-person perspective.

It is possible to capture images from such objective viewpoints using a fixed camera installed on the ceiling of the work space. However, in places with many desks and shelves, such as an office environment, the desks and shelves create blind spots, and there are places where the situation on the floor cannot be grasped using such a fixed camera. Also, for example, while many surveillance cameras are installed in commercial facilities, in an office environment, surveillance cameras are installed only at entrances and exits, and there are few surveillance cameras inside the room.

Therefore, by processing the image from a camera mounted on an autonomous mobile robot and generating an image seen from an objective viewpoint from the image from that camera, it is possible to reduce blind spots in the work space. In addition, by using the image from a camera mounted on an autonomous mobile robot that has requested remote operation as the image to be processed, it is possible to increase the possibility of capturing an image of a point where the autonomous mobile robot is no longer able to perform its work autonomously.

In the first embodiment described below, an image showing the position of a remotely controllable mobile body as seen from an objective viewpoint can be generated and displayed from an image captured from the mobile body. In the following, the mobile body capable of image capture and remote control is described as an autonomous mobile robot, but this is not limited to this and may be, for example, other mobile bodies that operate by remote control or a ride-on mobile body. Even with these mobile bodies, as with the autonomous mobile robot described below, when an obstacle occurs to the movement of the mobile body, the operator can remotely control it while viewing an image from an objective viewpoint.

Figure 1 is a block diagram showing the configuration of an image display device according to the first embodiment. Task information of an autonomous mobile robot and an image captured by a camera are input to the image display device in Figure 1.

The work information is information indicating work including the movement of the autonomous mobile robot, and indicates work that the autonomous mobile robot has performed so far, and work that the autonomous mobile robot plans to perform in the future. The work information may include, for example, movement path information indicating positions that the autonomous mobile robot has performed so far in chronological order, information detected in chronological order by a sensor mounted on the autonomous mobile robot, and log information of drive information of a drive unit that drives the autonomous mobile robot. Furthermore, the work information may not only indicate the work of the autonomous mobile robot, but may also be information indicating the work of peripheral devices such as security cameras and automatic doors within the workspace of the autonomous mobile robot.

The captured images are images captured by an autonomous mobile robot, and are images captured by a camera mounted on the autonomous mobile robot. Hereinafter, a camera mounted on a moving body such as an autonomous mobile robot may be referred to as an "mounted camera". In this embodiment 1, the mounted camera is assumed to be an omnidirectional camera capable of capturing images in all directions of the autonomous mobile robot. The captured images are, for example, images captured in a time series, and may be images in a video format or images in a still image format. The captured images may also include distance information indicating the distance between the captured object and the mounted camera. In the following, for simplicity of explanation, the captured images will be described as still image format images that do not include distance information, captured by the mounted camera. The captured images may also be still image format images that can be extracted from video format images.

In the example of FIG. 1, the work environment information is information that changes little while the autonomous mobile robot is performing work, and may be input to the image display device in the same way as the work information and captured images, or may be stored in advance in the image display device. The work environment information includes map information showing the work space of the autonomous mobile robot, and obstacle information related to obstacles in the work space. The work environment information may include mobile object information such as the shape and model of the autonomous mobile robot, and the installation positions and shapes of desks, shelves, surveillance cameras, and automatic doors in the work space. The map information may include, for example, information on a three-dimensional space measured in advance, and information on a three-dimensional space calculated from time-series information of a distance sensor mounted on the autonomous mobile robot.

Although not shown in FIG. 1, information for remotely controlling the autonomous mobile robot is input to the autonomous mobile robot. The autonomous mobile robot is remotely controlled based on this information.

The image display device in FIG. 1 includes an acquisition unit 1, a work position calculation unit 2, a captureable area calculation unit 3, a first viewpoint position extraction unit (own aircraft viewpoint position extraction unit 4), a superimposed image generation unit 5, and a display unit 6. The components of the image display device are described in detail below.

<Acquisition part 1>
The acquisition unit 1 acquires a remote operation request, which is a request for remote operation of the autonomous mobile robot, work information, and captured images. The remote operation request is a request to an operator for remote operation to enable the autonomous mobile robot to resume an autonomous task when it is determined that it is difficult for the autonomous mobile robot to autonomously perform a task. When it is determined that it is difficult for the autonomous mobile robot to perform a task autonomously, for example, a management system that manages all of the autonomous mobile robots, or an autonomous mobile robot, requests the operator to remotely operate the autonomous mobile robot by the remote operation request and outputs the remote operation request to the acquisition unit 1.

The remote control request includes information specifying the autonomous mobile robot that requires remote control, but may also include other additional information. The additional information includes, for example, the time of the request, the work priority, the reason for the request, and a remote control proposal. The work priority is set to "high" when there is a high risk that the autonomous mobile robot will harm a person, such as a collision with a person or an accident in a situation where a person is nearby, is set to "medium" when the risk is low and the work is urgent, and is set to "low" when the risk is low and the work is not urgent. The reason for the request indicates, for example, an error code and the reason for the request for remote control, and the remote control proposal indicates a response proposal for the reason for the request.

The acquisition unit 1 not only acquires a remote control request, but also acquires information about the autonomous mobile robot that requested remote control. The information about the autonomous mobile robot includes work information about the autonomous mobile robot and captured images related to the work information. If work environment information is not stored in advance, the acquisition unit 1 acquires the work environment information. After acquiring a remote control request, etc., processing is transferred to the work position calculation unit 2.

<Working position calculation unit 2>
The work position calculation unit 2 calculates, based on the remote control request and the work information, the position where the autonomous mobile robot was working when the remote control request was acquired, as the work position.

In the present embodiment 1, first, the work position calculation unit 2 determines from the work information acquired by the acquisition unit 1 whether the work being performed by the autonomous mobile robot when the remote operation request was acquired was a specific work performed in the vicinity of a specific position or movement between specific positions. A specific work is a general work in which movement is not the objective, such as, for example, loading/unloading or transferring work at an item delivery point.

When it is determined that the work being performed is a specific work, the work position calculation unit 2 determines the location where a series of operations in the specific work is being performed as the work position. When it is determined that the work being performed is movement between specific positions, the work position calculation unit 2 determines the movement range for a predetermined period until the time of the remote operation request, or the movement range for a predetermined distance until the stop position after the time of the remote operation request, as the work position. The predetermined period is, for example, a specified time such as 10 seconds until the request time, or a specified time such as 5 seconds until the request time not including the time after the stop. The certain distance is, for example, a specified distance such as 5 m until the stop position after the request time, or a distance proportional to the size of the autonomous mobile robot. After the work position is calculated, the process is transferred to the imageable area calculation unit 3.

<Photographable area calculation unit 3>
The imageable area calculation unit 3 calculates an area in which an image including an image of the work position can be captured as an imageable area based on the work environment information and the work position calculated by the work position calculation unit 2.

Figure 2 is a bird's-eye view to explain an example of a method for calculating the imageable area. Figure 2 shows a scene in which remote control is requested while an autonomous mobile robot is transferring an object to a shelf. Figure 2 shows the autonomous mobile robot 21 that requested remote control, indicated by a black circle, the work position 22, indicated by hatching, an obstacle 23 such as a shelf, and the imageable area 24, surrounded by a dashed line.

In this embodiment 1, first, the photographable area calculation unit 3 combines the work position 22 calculated by the work position calculation unit 2 with an obstacle 23 such as a shelf indicated by the obstacle information included in the work environment information on a map of the map information included in the work environment information. Then, the photographable area calculation unit 3 calculates an area in which it is possible to take an image including an image of the work position 22 without being obstructed by the obstacle 23 as the photographable area 24. For example, the photographable area calculation unit 3 calculates an area that includes points from which a straight line can be drawn to each area of the work position 22 without hitting the obstacle 23 as the photographable area 24.

The method of calculating the photographable area 24 may be a geometric calculation method in which the work position 22 and obstacles 23 are projected onto the map of the map information as polyhedrons and their vertices are connected, or it may be an exploratory calculation method in which it is determined whether an appropriately extracted representative point is a point in the photographable area 24. Note that the method of calculating the photographable area 24 is not limited to these. In the example of FIG. 2, the photographable area calculation unit 3 calculates the photographable area 24 on a plane such as a bird's-eye view, but the photographable area 24 may also be calculated in a three-dimensional space. After the photographable area 24 is calculated, the process is transferred to the aircraft viewpoint position extraction unit 4.

<Own aircraft viewpoint position extraction unit 4>
The own aircraft viewpoint position extraction unit 4 extracts, based on the work information and the photographable area 24 calculated by the photographable area calculation unit 3, a position included in the photographable area 24 from among positions within a moving path traveled in the past by the autonomous mobile robot 21 as a viewpoint position. For example, when extracting the viewpoint position, the own aircraft viewpoint position extraction unit 4 uses the moving path information included in the work information. The viewpoint position is a position that becomes the viewpoint of the image displayed on the display unit 6. Note that the own aircraft viewpoint position extraction unit 4 may extract the viewpoint position by taking into consideration optical system information of the camera mounted on the autonomous mobile robot 21 as well.

Figure 3 is a bird's-eye view diagram for explaining an example of a method for extracting a viewpoint position. In this embodiment 1, first, the own aircraft viewpoint position extraction unit 4 superimposes a moving path 25 along which the autonomous mobile robot has moved so far onto a photographable area 24 based on moving path information, and extracts the portion of the moving path 25 that is included in the photographable area 24 as an extracted path. In the example of Figure 3, continuous extracted paths 25a and 25b are extracted.

Then, the own aircraft viewpoint position extraction unit 4 calculates an evaluation value of each position in the extraction route based on the distance from the work position 22, the distance from the boundary of the photographable area 24, and the time it took to pass through the extraction route. For example, the own aircraft viewpoint position extraction unit 4 superimposes the autonomous mobile robot 21 on the captured image at a position in the extraction route based on the movement route information, the optical system information of the camera mounted on the autonomous mobile robot 21, and information indicating the size of the autonomous mobile robot 21. Then, the own aircraft viewpoint position extraction unit 4 may increase the evaluation value of the position as the proportion of the autonomous mobile robot 21 in the captured image approaches a certain range. Also, for example, the own aircraft viewpoint position extraction unit 4 may decrease the evaluation value of the position as the distance from the boundary of the photographable area 24 of the position in the extraction route becomes smaller than a certain value. Also, for example, the own aircraft viewpoint position extraction unit 4 may decrease the evaluation value of the position as the period from the time the autonomous mobile robot 21 passed through the position in the extraction route to the current time is longer.

The position at which the evaluation value is calculated may be, for example, a position within each division of the extracted path, or a position at which some index for calculating the evaluation value, such as the shooting distance, takes a maximum value. Furthermore, the position at which the evaluation value is calculated is not limited to these.

Next, the aircraft's viewpoint position extraction unit 4 determines the position with the highest evaluation value for each extracted route as the viewpoint position candidate. The aircraft's viewpoint position extraction unit 4 then compares the evaluation values of the viewpoint position candidates and determines the candidate with the highest evaluation value as the viewpoint position.

In the example of FIG. 3, the aircraft viewpoint position extraction unit 4 calculates an evaluation value at each position in the extraction route 25a, and determines the position with the highest evaluation value as the viewpoint position candidate 26a. Similarly, the aircraft viewpoint position extraction unit 4 calculates an evaluation value at each position in the extraction route 25b, and determines the position with the highest evaluation value as the viewpoint position candidate 26b. The aircraft viewpoint position extraction unit 4 compares the evaluation values of the viewpoint position candidates 26a and 26b, and determines the candidate with the highest evaluation value as the viewpoint position. For example, although the image of the candidate 26b was taken more recently than that of the candidate 26a, if a certain distance can be secured from the work position 22 compared to the candidate 26a, the evaluation value of the candidate 26b will be higher than that of the candidate 26a, and the candidate 26b will be determined as the viewpoint position. After the viewpoint position is extracted, the process is transferred to the superimposed image generation unit 5.

<Superimposed Image Generator 5>
The superimposed image generating unit 5 generates a superimposed image by superimposing a corresponding position image indicating the work position 22 on the photographed image photographed at the viewpoint position, based on the photographed image, the work position 22 calculated by the work position calculating unit 2, and the viewpoint position extracted by the own aircraft viewpoint position extracting unit 4. In the present embodiment 1, a work image indicating the work being performed by the autonomous mobile robot 21 when the remote operation request was acquired by the acquiring unit 1 is used as the corresponding position image.

In the first embodiment, first, the superimposed image generating unit 5 extracts, from among the captured images acquired by the acquiring unit 1, an image captured when the autonomous mobile robot 21 passes the viewpoint position extracted by the own-machine viewpoint position extracting unit 4, as an objective viewpoint environment image. For example, when there are multiple captured images for the viewpoint position, such as when the autonomous mobile robot 21 passes or stops at the viewpoint position multiple times, the superimposed image generating unit 5 extracts the most recent captured image from among them as the objective viewpoint environment image. Also, for example, when there is no captured image captured when the viewpoint position is passed, the superimposed image generating unit 5 extracts, as the objective viewpoint environment image, an image captured around the time when the viewpoint position is passed and at the closest time to that time.

Then, based on the work information, the superimposed image generation unit 5 generates a work image that reproduces the work that the autonomous mobile robot 21 was performing when the remote operation request was acquired by the acquisition unit 1. The reproduced work is, for example, a performed work for which the work position was calculated by the work position calculation unit 2. The work image may include only the operation of the autonomous mobile robot 21, without including an image of the environment such as the background. The work image may be, for example, an image that causes a 3D model of the autonomous mobile robot 21 to operate.

Next, the superimposed image generating unit 5 generates a work situation reproduction image, which is a superimposed image, by superimposing the work image on the objective viewpoint environment image. A detailed explanation of the method of calibrating the coordinate system of each image between multiple images is omitted because it is widely known. After the work situation reproduction image is generated, processing is transferred to the information display operation unit.

In the above description, the superimposed image generating unit 5 uses the captured image taken at the viewpoint position as the objective viewpoint environment image as is, but this is not limited to this, as described in other embodiments. In addition, the superimposed image generating unit 5 may superimpose the name of an item shown in the captured image and additional information obtained by other functions onto the objective viewpoint environment image.

<Display section 6>
The display unit 6 displays the work situation reproduction image, which is a superimposed image. The display unit 6 may also display information useful for remote operation, such as a request state for remote operation, map information of the facility, information on the operation target, and images captured in the past over a long period of time. The display unit 6 may also switch the display information in response to an operation by an operator. The display unit 6 is configured to be able to communicate with the superimposed image generation unit 5 by wireless technology, such as a wireless LAN (Local Area Network), and is a display device that can be viewed by an operator located remotely from the autonomous mobile robot 21.

Summary of the First Embodiment
According to the image display device of the first embodiment as described above, it is possible to extract a position included in the imageable area 24 from among the movement path 25 traveled in the past by the autonomous mobile robot 21 as a viewpoint position, and generate and display a work situation reproduction image in which a corresponding position image indicating the work position 22 is superimposed on an image captured at the viewpoint position based on the captured image, the work position 22, and the viewpoint position. According to such a configuration, the operator can easily grasp the situation of the autonomous mobile robot 21 by checking the situation in which the autonomous mobile robot 21 can no longer continue the work from the work situation reproduction image viewed from an objective viewpoint position.

In particular, in indoor work where there are many obstacles, the display of the work position of the autonomous mobile robot 21 seen from an objective viewpoint position may be obstructed by obstacles or the like. In contrast, in this embodiment 1, the objective viewpoint position can be appropriated by the photographable area 24, so that the display of the work position of the autonomous mobile robot 21 seen from an objective viewpoint position can be prevented from being obstructed by obstacles or the like. In addition, it is possible to eliminate the need for the operator to search for a photographed image for generating a work situation reproduction image from among the photographed images acquired by the acquisition unit 1.

In addition, in this embodiment 1, the corresponding position image showing the work position 22 includes a work image showing the work being performed by the autonomous mobile robot 21 when the remote operation request was acquired. With this configuration, the operator can easily understand the work being performed by the autonomous mobile robot 21 that has fallen into a state where it is unable to perform the work autonomously.

<Modification>
In the first embodiment, the aircraft's viewpoint position extraction unit 4 not only extracts a viewpoint position such as determining a candidate viewpoint position, but also determines the viewpoint position itself, but this is not limited to the above. For example, the extraction of the viewpoint position in the aircraft's viewpoint position extraction unit 4 may include determining a candidate viewpoint position and calculating an evaluation value of the candidate, and the operator may determine a viewpoint position from among the candidate viewpoint positions based on the evaluation value. In this case, the aircraft's viewpoint position extraction unit 4 may display the candidate viewpoint positions and the evaluation values of the candidates to the operator.

Figure 4 is a diagram showing an example of the display of the own aircraft viewpoint position extraction unit 4 to the operator. In the example of Figure 4, the autonomous mobile robot 21, the work position 22, the obstacle 23, the photographable area 24, and the movement path 25 of the autonomous mobile robot 21 are displayed on a map. In addition, positions with high evaluation values among the extracted paths 25a, 25b in the photographable area 24 are displayed to the operator as viewpoint position candidates 26a, 26b. In such a case, the own aircraft viewpoint position extraction unit 4 may determine the candidate selected by the operator from candidates 26a, 26b as the viewpoint position.

In addition, the operator may determine an arbitrary point on the extracted route 25a, 25b as the viewpoint position, rather than determining the viewpoint position from among the candidates 26a, 26b. In this case, the own aircraft viewpoint position extraction unit 4 may highlight the extracted route 25a, 25b within the photographable area 24. The extracted route 25a, 25b may also be displayed so that the photographing time can be identified by the color or thickness of the line. For example, the longer the period from the time when the autonomous mobile robot 21 passed a position on the extracted route to the current time, the lighter the color of the line at that position may be, the different shades of red, blue, etc. of the line at that position may be, or the line at that position may be made thinner.

In addition, if the camera mounted on the autonomous mobile robot 21 is not an omnidirectional camera, the shooting ranges 27a, 27b of the mounted camera may be displayed for the candidates 26a, 26b, as shown in FIG. 5. With this configuration, even if the mounted camera is not an omnidirectional camera, an appropriate captured image can be used for the work situation reproduction image.

For example, extraction of the viewpoint position in the aircraft viewpoint position extraction unit 4 may include determining candidates for the viewpoint position and calculating an evaluation value thereof, and the superimposed image generation unit 5 may determine the viewpoint position from among the candidates for the viewpoint position based on the evaluation value.

<Embodiment 2>
In this second embodiment, the three-dimensional spatial texture image of the surrounding environment is updated using a time series of captured images, thereby generating a three-dimensional virtual environment image that can be used as an objective viewpoint environment image for the superimposed image generation unit 5 and that can be changed over time.

Figure 6 is a block diagram showing the configuration of an image display device according to the second embodiment. The configuration of the image display device in Figure 6 is the same as that of the image display device in Figure 1, with an environmental image generating unit 7 added.

In this second embodiment, the work environment information includes three-dimensional spatial position and orientation information for fixed objects in the work space, and the three-dimensional spatial position and orientation information indicates the three-dimensional objects of the fixed objects.

The environment image generating unit 7 generates a three-dimensional virtual environment image that can be changed over time by attaching the time-series captured images as textures to a three-dimensional object based on the time-series captured images and the position and orientation information included in the work environment information. An example of generating a virtual environment image is described below.

First, in this second embodiment, the environmental image generating unit 7 extracts, from the position and orientation information, position and orientation information indicating three-dimensional objects that can be photographed from the viewpoint position extracted by the aircraft viewpoint position extracting unit 4, as within-field-of-view object information. The within-field-of-view object information includes, for example, information about which surfaces of which three-dimensional objects can be photographed.

Then, the environmental image generating unit 7 chronologically determines whether the captured images taken up until the remote control request include any captured images that correspond to the three-dimensional object in the field of view object information.

When it is determined that a photographed image corresponding to a three-dimensional object is included, the environment image generating unit 7 generates a virtual environment image by attaching the photographed image as a texture to the three-dimensional object in chronological order. This generates a three-dimensional virtual environment image that can be changed in chronological order and is seen from the viewpoint position extracted by the player's viewpoint position extracting unit 4.

If it is not determined that the captured image corresponding to a three-dimensional object is included, the environmental image generating unit 7 extracts, from the captured images acquired by the acquiring unit 1, an image captured at the viewpoint position extracted by the aircraft viewpoint position extracting unit 4.

The superimposed image generating unit 5 generates a work situation reproduction image by using the virtual environment image generated by the environment image generating unit 7 or the extracted photographed image as the objective viewpoint environment image described in the first embodiment. In the first embodiment, a static photographed image taken at the viewpoint position is used as the objective viewpoint environment image, but in the second embodiment, a three-dimensional virtual environment image that can be changed in time series can be used as the objective viewpoint environment image. When a work image showing the work of the autonomous mobile robot 21 is superimposed on the objective viewpoint environment image made of a virtual environment image, the degree of reproduction of the situation of the autonomous mobile robot 21 can be improved by matching the time of the work image and the time of the objective viewpoint environment image.

The time of the work image and the time of the objective-viewpoint environment image may be made different. Figure 7 shows an example of a screen for generating a work situation reproduction image by making the time of the work image and the time of the objective-viewpoint environment image different. At the bottom of the screen in Figure 7, there is a seek bar for specifying the time of the work image of the autonomous mobile robot, the time of the viewpoint position that defines the three-dimensional object, and the time of the captured image that is pasted as a texture onto the three-dimensional object. In this example, the time of the viewpoint position and the time of the captured image are included in the time of the objective-viewpoint environment image.

When the operator operates the seek bar to specify a time, an objective viewpoint environment image is reproduced with a combination of the work image, viewpoint position, and time specified for the captured image. Note that various methods have already been proposed for combining 3D objects and captured images, and are not limited to the above.

Summary of the second embodiment
According to the second embodiment, a three-dimensional virtual environment image that can be changed in time series is generated based on the time-series captured images and the three-dimensional spatial position and orientation information of fixed objects in the working space. With this configuration, the operator can more easily grasp the situation of the autonomous mobile robot 21 that has fallen into a state where it is unable to perform a task autonomously.

<Third embodiment>
In the third embodiment, the position and orientation information of a moving obstacle is calculated.

Figure 8 is a block diagram showing the configuration of an image display device according to the third embodiment. The configuration of the image display device in Figure 8 is the same as that of the image display device in Figure 1, with a moving obstacle information calculation unit 8 added.

The object according to the third embodiment includes at least one of a fixed obstacle and a moving obstacle. Fixed obstacles are obstacles whose position and posture change little, such as walls, doors, shelves, and desks. Moving obstacles are moving bodies such as people and mobile robots, and obstacles whose position and posture change often, such as chairs.

In this third embodiment, the obstacle information in the work environment information includes position and orientation information of fixed obstacles in the work space.

The moving obstacle information calculation unit 8 calculates the position and orientation information of the moving obstacle to be displayed on the display unit 6 separately from the fixed obstacle based on the position and orientation information of the object in the working space and the position and orientation information of the fixed obstacle included in the working environment information. The position and orientation information of the object in the working space is, for example, the position and orientation information of the object including at least one of the fixed obstacle and the moving obstacle recognized by a sensor mounted on the autonomous mobile robot 21 or the like. The moving obstacle information calculation unit 8 calculates the position and orientation information of the moving obstacle by subtracting the position and orientation information of the fixed obstacle from the position and orientation information of the object in the working space. For example, the distance information recognized by the LiDER or the stereo camera may be used as the position and orientation information of the object. In this case, the moving obstacle information calculation unit 8 may calculate the information of the moving obstacle by removing the position and orientation information of the fixed obstacle from the distance information.

The moving obstacle is highlighted, for example, in the objective viewpoint environment image of the superimposed image generating unit 5, using color, text, and other image effects. The highlighting in the objective viewpoint environment image of the superimposed image generating unit 5 is reflected in the work situation reproduction image displayed on the display unit 6. Note that the moving obstacle indicated by the calculated position and orientation information may be highlighted in the work situation reproduction image of the display unit 6 without passing through the objective viewpoint environment image of the superimposed image generating unit 5.

In the above configuration, if a model of the moving obstacle is obtained in advance, the moving obstacle information calculation unit 8 may recognize the moving obstacle by model matching from among the objects in the captured image that does not include distance information, and calculate the position and orientation information of the moving obstacle. For example, models of moving obstacles such as chairs, people, and mobile robots may be included in the work environment information, and the moving obstacle information calculation unit 8 may perform model matching by referring to the models. For people, instead of precise model matching, a lump-sum model matching of people may be performed. In addition, the moving obstacle information calculation unit 8 may generate tag information that distinguishes between objects such as people, mobile robots, and chairs by tagging the objects. This tag information may be used to change the emphasis expression according to the type of moving obstacle, or to add the name of the moving obstacle, in the objective viewpoint environment image of the superimposed image generation unit 5 or the work situation reproduction image of the display unit 6.

In the position and orientation information of fixed obstacles included in the work environment information, fixed obstacles and moving obstacles may be switched during operation. For example, if the recognition result based on distance information differs from the previous recognition result, the fixed obstacle at the time of the previous recognition may be switched to a moving obstacle. Conversely, if the recognition result based on distance information does not change for a certain period of time, the moving obstacle at the time of the previous recognition may be switched to a fixed obstacle.

<Summary of the Third Embodiment>
The influence of fixed obstacles is anticipated in advance when planning a task, and is therefore unlikely to hinder the task execution of the autonomous mobile robot 21. On the other hand, the influence of moving obstacles cannot be fully anticipated in advance when planning a task, and is therefore relatively likely to hinder the task execution of the autonomous mobile robot 21.

Therefore, according to the third embodiment, the position and orientation information of the moving obstacle that is displayed on the display unit 6 separately from the fixed obstacle is calculated. With this configuration, the moving obstacle can be displayed separately from the fixed obstacle, so that the operator can more easily grasp the situation of the autonomous mobile robot 21 that has fallen into a state where it is unable to perform a task autonomously.

<Fourth embodiment>
In the fourth embodiment, an image from a camera other than the camera mounted on the autonomous mobile robot 21 that has requested remote control can be used.

Figure 9 is a block diagram showing the configuration of an image display device according to the fourth embodiment. The configuration of the image display device in Figure 9 is the same as that of the image display device in Figure 1, except that a second viewpoint position extraction unit 9, which is a second viewpoint position extraction unit, is added.

The other-machine viewpoint position extraction unit 9 calculates a different viewpoint position for a different camera other than the camera mounted on the autonomous mobile robot 21 that has requested remote operation. In the following, we will mainly explain the case where the different camera is a camera mounted on another autonomous mobile robot (i.e., another mobile body) other than the autonomous mobile robot 21 that has requested remote operation.

Except for the difference in the movement path, the processing of the other-vehicle viewpoint position extraction unit 9 is the same as the processing of the own-vehicle viewpoint position extraction unit 4. In other words, the other-vehicle viewpoint position extraction unit 9 extracts, as another viewpoint position, a position included in the photographable area 24 from among positions on another movement path along which another autonomous mobile robot has moved in the past.

Specifically, after the photographable area 24 is calculated, the other-machine viewpoint position extraction unit 9 superimposes another moving path along which the other autonomous mobile robot has moved so far onto the photographable area 24 based on the moving path information of the other autonomous mobile robot, and extracts the portion of the other moving path that is included in the photographable area 24 as another extracted path. The other-machine viewpoint position extraction unit 9 calculates an evaluation value for each position in the other extracted path, and determines another viewpoint position based on the evaluation value.

The superimposed image generating unit 5 superimposes a corresponding position image onto another photographed image photographed at another viewpoint position based on another photographed image photographed from another autonomous mobile robot, the work position 22, and another viewpoint position determined by the other-machine viewpoint position extracting unit 9. By performing such superimposition, the superimposed image generating unit 5 generates a work situation reproduction image displayed on the display unit 6. Note that the superimposed image generating unit 5 may compare the evaluation value of the viewpoint position calculated by the own-machine viewpoint position extracting unit 4 with the evaluation value of the other viewpoint position calculated by the other-machine viewpoint position extracting unit 9, and generate a work situation reproduction image only for the one with the higher evaluation value.

The superimposed image generating unit 5 may use images from, for example, a surveillance camera fixed to the ceiling, instead of images from a camera mounted on an autonomous mobile robot. In images from a surveillance camera, the position of a different viewpoint is not changed.

In addition, if a suitable alternative viewpoint cannot be obtained, the operator may move another autonomous mobile robot by remote control.

Figure 10 is a diagram showing an example of a display to an operator. In Figure 10, the fixed surveillance camera's viewpoint position 28 and shooting range 29, and the destination 30 of another autonomous mobile robot are added to the display of Figure 4. A map such as that of Figure 10 is displayed to an operator, and when the operator selects the destination 30 on the map, the other autonomous mobile robot may be configured to move to the destination 30. Note that the other autonomous mobile robot can take pictures even after the autonomous mobile robot 21 requests remote control. For this reason, when the display unit 6 displays an image taken by another autonomous mobile robot, it may be possible to distinguish whether the image is an image taken after a remote control request is made or an image taken before a remote control request is made by using the color of a display icon or the like. The display unit 6 may also display an image taken at a time specified by a seek bar or the like.

<Summary of the Fourth Embodiment>
According to the fourth embodiment, it is possible to use images captured by another moving body, such as another autonomous mobile robot. This makes it possible to use images captured by another moving body when there is no suitable image captured by the autonomous mobile robot 21 within the captureable area 24, or when the operator wants an image captured in a different direction from the captured image. As a result, the operator can more easily grasp the situation of the autonomous mobile robot 21 that has fallen into a state where it is unable to perform a task autonomously.

<Fifth embodiment>
In the fifth embodiment, a simulation image can be generated in which the autonomous mobile robot 21 virtually performs a task based on remote control by an operator.

Figure 11 is a block diagram showing the configuration of an image display device according to the fifth embodiment. The configuration of the image display device in Figure 11 is the same as that of the image display device in Figure 1, with a virtual image generating unit 10 added.

The virtual image generating unit 10 generates a simulation image of the case where the autonomous mobile robot 21 is caused to virtually perform a task based on remote control from an operator. Before operating the autonomous mobile robot 21 by remote control, the virtual image generating unit 10 generates a simulation image of the case where a 3D model of the autonomous mobile robot 21 or the like is operated in accordance with the remote control. The simulation image is displayed superimposed on the work situation reproduction image on the display unit 6. Note that the simulation image may be superimposed on the objective viewpoint environment image of the superimposed image generating unit 5 so as to be superimposed on the work situation reproduction image on the display unit 6.

Summary of the Fifth Embodiment
It is assumed that an operator will remotely control multiple autonomous mobile robots, and in such an operation, a remote control error after a remote control request may lead to an accident. In particular, since there are many different types of autonomous mobile robots and each robot has a different operation method and behavior, the possibility of such an accident is considered to be relatively high.

In contrast, according to the fifth embodiment, a simulation image is generated in which the autonomous mobile robot 21 virtually performs a task based on remote control from the operator. With this configuration, the operator can check whether there are any problems with the remote control by looking at the simulation image superimposed on the work situation reproduction image before actually remotely controlling the autonomous mobile robot 21. This allows the operator to change the remote control until he or she is sure that there are no problems with the remote control, thereby reducing remote control errors.

<Sixth embodiment>
In this embodiment 6, when a remote control request is received and the work being performed by the autonomous mobile robot 21 is work performed by the autonomous mobile robot 21 on a work object, an image of the work object can be acquired.

Fig. 12 is a block diagram showing the configuration of an image display device according to the sixth embodiment. The configuration of the image display device in Fig. 12 is the same as that of the image display device in Fig. 1, with a work object image acquisition unit 11 added.

The work object image acquisition unit 11 acquires an image of the work object when the work being performed by the autonomous mobile robot 21 when a remote control request is acquired is work from the autonomous mobile robot 21 to the work object. Work from the autonomous mobile robot 21 to the work object includes work using an end effector of the autonomous mobile robot 21, etc. To capture the image of the work object, for example, a mobile camera that can capture three-dimensional spatial images in real time and is movable from the autonomous mobile robot 21 body is used among the mounted cameras. The work object image acquisition unit 11 may change the viewpoint position of the mobile camera by moving the mobile camera so that an image in which the work object is easily recognized is captured.

First, in the sixth embodiment, the work object image acquisition unit 11 determines whether or not a three-dimensional spatial image of the work object or the end effector is required. For example, if the work content being performed by the autonomous mobile robot 21 when the remote operation request is acquired is an operation on the work object using the end effector, it is determined that a three-dimensional spatial image is required. It may also be determined that a three-dimensional spatial image is required when the end effector is being operated or when there is an object near the end effector. If it is determined that a three-dimensional spatial image is required, the work object image acquisition unit 11 determines the viewpoint position of the mobile camera that is photographing the work object as the work object photographing position.

The display unit 6 displays to the operator an image of the work object from the work object photographing position. When only the end effector operates before and after the work when remote operation is requested, the display unit 6 may display only the image of the work object without displaying the objective viewpoint environment image.

The image suitable for an operator varies depending on the type of work. When moving the autonomous mobile robot 21 body, an image from an objective viewpoint that allows the operator to see the entire autonomous mobile robot 21 is suitable for the operator's judgment of the situation. On the other hand, when the autonomous mobile robot 21 uses an end effector to perform a task such as transferring an object to be worked on, a three-dimensional spatial image that is centered on the end effector and the object to be worked on and in which the viewpoint position can be freely changed is suitable for the operator's judgment of the situation.

In particular, when operating a work object, it is difficult to grasp the sense of distance from an image with only a fixed viewpoint, so it is desirable for the worker to be able to check the work while changing the viewpoint position. On the other hand, when performing work that moves the work object by a large distance, it is desirable for the worker to be able to simultaneously check both the state of the autonomous mobile robot 21 main body and the state of the work object by the end effector. Therefore, based on the work content at the time of remote operation request, the display unit 6 may selectively display either the objective viewpoint environment image and the work object image side by side, or the objective viewpoint environment image and the work object image. Note that displaying the objective viewpoint environment image and the work object image side by side is not only useful when the worker checks the work when remote operation is requested, but is also useful when the worker performs remote operation.

The work object image acquisition unit 11 may reduce blind spots by combining a three-dimensional spatial image taken before the autonomous mobile robot 21 starts work (for example, at the time of system startup or at another work time) with a real-time image of the work object. In addition, if the three-dimensional spatial image can be expanded, the expanded three-dimensional spatial image may be displayed on the display unit 6. When such a display is performed, if the operator changes the viewpoint position within the three-dimensional spatial image, the viewpoint position of the mobile camera may be changed. With this configuration, the operator can check the work object and the end effector by changing the viewpoint position within the three-dimensional spatial image.

Summary of the Sixth Embodiment
According to the sixth embodiment, an image of the work target object is acquired if the work being performed by the autonomous mobile robot 21 when a remote operation request is acquired is work performed by the autonomous mobile robot 21 on the work target object. With this configuration, the operator can easily understand the work to be performed on the work target object by the autonomous mobile robot 21 that has fallen into a state in which it is unable to perform the work autonomously.

<Seventh embodiment>
In the seventh embodiment, the autonomous mobile robot 21 is configured to be movable so that a photographed image can be acquired at a point in the work space a predetermined time after the photographed image is taken.

Figure 13 is a block diagram showing the configuration of an image display device according to the seventh embodiment. The configuration of the image display device in Figure 13 is the same as that of the image display device in Figure 1, with a shooting position acquisition unit 12 and a plan creation unit 13 added.

The image capture position acquisition unit 12 acquires, as an image capture position, a point in the work space where a predetermined time has passed since an image was captured. The image capture position acquisition unit 12 manages the time when an image was captured at each point in the work space based on the work information and the work environment information, and acquires, if there is a point where no image has been captured for a certain period of time, such as 15 minutes, the point as an image capture position.

The plan creation unit 13 creates a plan for the route along which the autonomous mobile robot 21 should move, i.e., a plan for the movement route of the autonomous mobile robot 21, based on the shooting position. The plan creation unit 13 plans the movement route and work instructions based on the work plan of the autonomous mobile robot 21, but if the shooting position acquisition unit 12 has acquired the shooting position, the plan creates the movement route and work instructions taking into account the shooting position. The shooting position may be used as the destination of the movement route of the autonomous mobile robot 21, or as a stopover on the movement route. Methods for planning the movement route of the autonomous mobile robot 21 are widely known, so detailed explanations thereof will be omitted.

Note that, similar to the display of candidate viewpoint positions and the selection of candidates by the operator described in the modified example of the first embodiment, candidate shooting positions may be displayed and a candidate may be selected by the selector. For example, the shooting position acquisition unit 12 may calculate a demand level for each candidate shooting position that is proportional to the time that has elapsed since the image was captured, and display each candidate shooting position and its demand level. The shooting position acquisition unit 12 may then increase the demand level of the candidate shooting position selected by the operator by a certain value, and acquire the candidate shooting position whose demand level is equal to or greater than the threshold value as the shooting position.

Summary of the Seventh Embodiment
According to the seventh embodiment, a point in the working space where a predetermined time has elapsed since the captured image was captured is moved to the autonomous mobile robot 21. With this configuration, the captured image at that point can be updated, and the possibility of obtaining an appropriate captured image for the objective viewpoint environment image can be increased.

<Embodiment 8>
In the eighth embodiment, the autonomous mobile robot 21 is configured to be movable so that a photographic image can be acquired at a point in the work space where a person has completed a task.

Figure 14 is a block diagram showing the configuration of an image display device according to the eighth embodiment. The configuration of the image display device in Figure 14 is the same as that of the image display device in Figure 1, with a plan creation unit 13 and a task completion position acquisition unit 14 added.

The work completion position acquisition unit 14 acquires the point in the work space where a person has completed work as the photographing position. For example, the work completion position acquisition unit 14 acquires the point to which a person moved after remaining there for a certain period of time as the point where the person completed work, that is, the photographing position. The movement of the person may be determined by tracking using camera images, etc., or, if the person is an employee, may be determined from the work information of the worker.

The plan creation unit 13 is the same as the plan creation unit 13 in embodiment 7.

<Summary of the Eighth Embodiment>
After a person has completed a task, it is highly likely that the installation status of an installed object has been changed. For example, in an office environment, a change in the installation status of an installed object may include moving a chair. In a sales facility, a change in the installation status of an installed object may include changing the installation position of the item. In a restaurant, it is highly likely that the installation status of an installed object has been changed after a store clerk and a customer have completed their task.

Therefore, according to the eighth embodiment, the autonomous mobile robot 21 is moved to a point in the work space where a person has completed a task. With this configuration, an image can be taken from the environment after a person has completed a task, where the work environment is likely to have changed, and therefore an image can be secured of a point where remote operation by an operator is likely to be required.

<Other Modifications>
The acquisition unit 1, the work position calculation unit 2, the photographable area calculation unit 3, the aircraft viewpoint position extraction unit 4, and the superimposed image generation unit 5 in Fig. 1 described above will be hereinafter referred to as "the acquisition unit 1, etc." The acquisition unit 1, etc. is realized by a processing circuit 81 shown in Fig. 15. That is, the processing circuit 81 includes an acquisition unit 1 that acquires a request for remote control of the mobile body, work information indicating a work including the movement of the mobile body, and a photographed image taken from the mobile body, a work position calculation unit 2 that calculates the work position of the mobile body when the request is acquired based on the request and the work information, a photographable area calculation unit 3 that calculates an area in which a photographed image including an image of the work position can be photographed as a photographable area based on work environment information including map information indicating the work space of the mobile body and obstacle information regarding obstacles in the work space and the work position, a machine viewpoint position extraction unit 4 that extracts a position included in the photographable area as a viewpoint position from among positions in a moving path in which the mobile body has moved in the past based on the work information and the photographable area, and a superimposed image generation unit 5 that generates a superimposed image in which a corresponding position image indicating the work position is superimposed on a photographed image photographed at the viewpoint position based on the photographed image, the work position, and the viewpoint position. Dedicated hardware may be applied to the processing circuit 81, or a processor that executes a program stored in a memory may be applied. The processor may be, for example, a central processing unit, a processing unit, an arithmetic unit, a microprocessor, a microcomputer, or a DSP (Digital Signal Processor).

When the processing circuit 81 is dedicated hardware, the processing circuit 81 corresponds to, for example, a single circuit, a composite circuit, a programmed processor, a parallel programmed processor, an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array), or a combination of these. Each function of the acquisition unit 1, etc. may be realized by a circuit with distributed processing circuits, or the functions of each unit may be realized by a single processing circuit.

When the processing circuit 81 is a processor, the functions of the acquisition unit 1, etc. are realized in combination with software, etc. Note that software, etc. includes, for example, software, firmware, or software and firmware. Software, etc. is written as a program and stored in memory. As shown in FIG. 16, the processor 82 applied to the processing circuit 81 realizes the functions of each unit by reading and executing a program stored in memory 83. That is, the image display device includes a memory 83 for storing a program that, when executed by the processing circuit 81, results in the execution of the following steps: a request for remote control of the mobile body, work information indicating a work including the movement of the mobile body, and a captured image captured from the mobile body; a step of calculating the work position of the mobile body when the request is acquired based on the request and the work information; a step of calculating an area in which a captured image including an image of the work position can be captured as a captureable area based on the work environment information including map information indicating the work space of the mobile body and obstacle information regarding obstacles in the work space and the work position; a step of extracting a position included in the captureable area as a viewpoint position from among positions in a moving path in which the mobile body has moved in the past based on the work information and the captureable area; a step of generating a superimposed image in which a corresponding position image indicating the work position is superimposed on a captured image captured at the viewpoint position based on the captured image, the work position, and the viewpoint position; and a step of displaying the superimposed image. In other words, this program can be said to cause a computer to execute the procedure or method of the acquisition unit 1, etc. Here, the memory 83 may be, for example, a non-volatile or volatile semiconductor memory such as a random access memory (RAM), a read only memory (ROM), a flash memory, an erasable programmable read only memory (EPROM), or an electrically erasable programmable read only memory (EEPROM), a hard disk drive (HDD), a magnetic disk, a flexible disk, an optical disk, a compact disk, a mini disk, a digital versatile disk (DVD), a drive device for any of these, or any storage medium to be used in the future.

A configuration has been described above in which the functions of the acquisition unit 1, etc. are realized either by hardware or software, etc. However, this is not limited to this, and a configuration in which part of the acquisition unit 1, etc. is realized by dedicated hardware and another part is realized by software, etc. For example, the function of the acquisition unit 1 can be realized by a processing circuit 81 as dedicated hardware, and the other functions can be realized by the processing circuit 81 as a processor 82 reading and executing a program stored in a memory 83.

As described above, the processing circuit 81 can realize each of the above-mentioned functions through hardware, software, etc., or a combination of these.

The image display device described above can also be applied to an image display system constructed as a system by appropriately combining an information processing device, a communication terminal including a mobile terminal such as a mobile phone, a smartphone, and a tablet, the functions of an application installed on the information processing device, and a server. In this case, each function or each component of the image display device described above may be distributed and disposed in each device that constructs the system, or may be concentrated and disposed in one of the devices.

The embodiments and variations can be freely combined, and each embodiment and variation can be modified or omitted as appropriate.

Various aspects of this disclosure are summarized below as appendices.

(Appendix 1)
an acquisition unit that acquires a request for remote control of a moving object, work information indicating a work including movement of the moving object, and a captured image captured from the moving object;
a work position calculation unit that calculates a work position of the moving object when the request is acquired based on the request and the work information;
a captureable area calculation unit that calculates an area in which the captured image including an image of the work position can be captured as a captureable area based on work environment information including map information indicating a work space of the moving body and obstacle information regarding obstacles in the work space, and the work position;
a first viewpoint position extraction unit that extracts, based on the work information and the imageable area, a position included in the imageable area from among positions within a movement path along which the moving object has moved in the past, as a viewpoint position;
a superimposed image generating unit that generates a superimposed image by superimposing a corresponding position image indicating the work position on the photographed image photographed at the viewpoint position based on the photographed image, the work position, and the viewpoint position;
and a display unit that displays the superimposed image.

(Appendix 2)
2. The image display device according to claim 1, wherein the corresponding position image includes a task image showing a task that the moving object was performing when the request was acquired.

(Appendix 3)
the work environment information includes three-dimensional spatial position and orientation information of a fixed object in the work space,
The image display device described in Appendix 1 or Appendix 2, further comprising an environment image generation unit that generates a three-dimensional virtual environment image that can be changed in time series based on the captured images in time series and the position and orientation information as the captured images to be used in the superimposed image generation unit.

(Appendix 4)
The obstacle information of the work environment information includes position and orientation information of a fixed obstacle in the work space,
The image display device according to any one of Supplementary Note 1 to Supplementary Note 3, further comprising a moving obstacle information calculation unit that calculates position and orientation information of a moving obstacle to be displayed on the display unit in distinction from the fixed obstacle, based on position and orientation information of an object in the working space and the position and orientation information of the fixed obstacle in the working space.

(Appendix 5)
a second viewpoint position extraction unit that extracts a position included in the imageable area as another viewpoint position from among positions on another moving path along which another moving body has moved in the past;
The superimposed image generating unit
An image display device as described in any one of Appendix 1 to Appendix 4, which generates the superimposed image displayed on the display unit by superimposing the corresponding position image on the other captured image taken at the other viewpoint position based on another captured image taken from the other moving body, the work position, and the other viewpoint position.

(Appendix 6)
The image display device according to any one of Supplementary Note 1 to Supplementary Note 5, further comprising a virtual image generation unit that generates a simulation image that is superimposed on the superimposed image on the display unit and that generates a simulation image of a case in which the moving body is virtually performing work based on the remote operation from an operator.

(Appendix 7)
The image display device described in any one of Appendix 1 to Appendix 6, further comprising a work object image acquisition unit that acquires an image of the work object to be displayed on the display unit when the work being performed by the moving body at the time the request is acquired is work from the moving body to a work object.

(Appendix 8)
an image capturing position acquisition unit that acquires, as an image capturing position, a point in the work space at which a predetermined time has elapsed since the image was captured;
The image display device according to any one of claims 1 to 7, further comprising a plan creation unit that creates a plan for a route along which the moving object should move based on the shooting positions.

(Appendix 9)
A task completion position acquisition unit that acquires a point in the work space where a person completes a task as a photographing position;
The image display device according to any one of claims 1 to 7, further comprising a plan creation unit that creates a plan for a route along which the moving object should move based on the shooting positions.

(Appendix 10)
acquiring a request for remote control of a moving body, work information indicating a work including movement of the moving body, and a photographed image taken from the moving body;
Calculating a work position of the moving object at the time the request was acquired based on the request and the work information;
calculating an area in which the captured image including an image of the work position can be captured as a captureable area based on work environment information including map information indicating a work space of the mobile body and obstacle information regarding obstacles in the work space, and the work position;
extracting, as a viewpoint position, a position included in the image capture area from among positions within a movement path along which the moving object has moved in the past, based on the work information and the image capture area;
generating a superimposed image by superimposing a corresponding position image indicating the work position on the photographed image photographed at the viewpoint position based on the photographed image, the work position, and the viewpoint position;
An image display method comprising: displaying the superimposed image.

1 Acquisition unit, 2 Work position calculation unit, 3 Shootable area calculation unit, 4 Self-vehicle viewpoint position extraction unit, 5 Superimposed image generation unit, 6 Display unit, 7 Environmental image generation unit, 8 Moving obstacle information calculation unit, 9 Other vehicle viewpoint position extraction unit, 10 Virtual image generation unit, 11 Worked object image acquisition unit, 12 Shooting position acquisition unit, 13 Plan creation unit, 14 Work completion position acquisition unit, 21 Autonomous mobile robot, 22 Work position, 23 Obstacle, 24 Shootable area, 25 Movement route.

Claims (10)

an acquisition unit that acquires a request for remote control of a moving object, work information indicating a work including movement of the moving object, and a captured image captured from the moving object;
a work position calculation unit that calculates a work position of the moving object when the request is acquired based on the request and the work information;
a captureable area calculation unit that calculates an area in which the captured image including an image of the work position can be captured as a captureable area based on work environment information including map information indicating a work space of the moving body and obstacle information regarding obstacles in the work space, and the work position;
a first viewpoint position extraction unit that extracts, based on the work information and the imageable area, a position included in the imageable area from among positions within a movement path along which the moving object has moved in the past, as a viewpoint position;
a superimposed image generating unit that generates a superimposed image by superimposing a corresponding position image indicating the work position on the photographed image photographed at the viewpoint position based on the photographed image, the work position, and the viewpoint position;
and a display unit that displays the superimposed image. The image display device according to claim 1 ,
The corresponding position image includes a task image showing the task that the mobile object was performing when the request was acquired. 3. The image display device according to claim 1,
the work environment information includes three-dimensional spatial position and orientation information of a fixed object in the work space,
The image display device further includes an environment image generation unit that generates a three-dimensional virtual environment image that can be changed in time series based on the captured images in time series and the position and orientation information, as the captured images to be used in the superimposed image generation unit. 3. The image display device according to claim 1,
The obstacle information of the work environment information includes position and orientation information of a fixed obstacle in the work space,
The image display device further includes a moving obstacle information calculation unit that calculates position and orientation information of a moving obstacle to be displayed on the display unit in distinction from the fixed obstacle, based on position and orientation information of an object in the working space and the position and orientation information of the fixed obstacle in the working space. 3. The image display device according to claim 1,
a second viewpoint position extraction unit that extracts a position included in the imageable area as another viewpoint position from among positions on another moving path along which another moving body has moved in the past;
The superimposed image generating unit
An image display device that generates the superimposed image displayed on the display unit by superimposing the corresponding position image on the other captured image taken at the other viewpoint position based on another captured image taken from the other moving body, the work position, and the other viewpoint position. 3. The image display device according to claim 1,
The image display device further includes a virtual image generation unit that generates a simulation image that is superimposed on the superimposed image on the display unit and that simulates a task being virtually performed by the moving body based on the remote control from an operator. 3. The image display device according to claim 1,
The image display device further includes a work object image acquisition unit that acquires an image of the work object to be displayed on the display unit when the work being performed by the moving body at the time the request is acquired is work from the moving body to a work object. 3. The image display device according to claim 1,
an image capturing position acquisition unit that acquires, as an image capturing position, a point in the work space at which a predetermined time has elapsed since the image was captured;
a plan creation unit that creates a plan for a route along which the moving object should move based on the photographing positions. 3. The image display device according to claim 1,
A task completion position acquisition unit that acquires a point in the work space where a person completes a task as a photographing position;
a plan creation unit that creates a plan for a route along which the moving object should move based on the photographing positions. acquiring a request for remote control of a moving body, work information indicating a work including movement of the moving body, and a photographed image taken from the moving body;
Calculating a work position of the moving object at the time the request was acquired based on the request and the work information;
calculating an area in which the captured image including an image of the work position can be captured as a captureable area based on work environment information including map information indicating a work space of the mobile body and obstacle information regarding obstacles in the work space, and the work position;
extracting, as a viewpoint position, a position included in the image capture area from among positions within a movement path along which the moving object has moved in the past, based on the work information and the image capture area;
generating a superimposed image by superimposing a corresponding position image indicating the work position on the photographed image photographed at the viewpoint position based on the photographed image, the work position, and the viewpoint position;
An image display method comprising: displaying the superimposed image.

Images