KR102140077B1 - Master device, slave device and control method thereof - Google Patents

Abstract

The server is started. A server that provides content to a user terminal device that provides a virtual reality service includes at least one source device, a communication unit performing communication with the user terminal device, and content transmission for a predetermined location from the user terminal device When a request is received, it includes a processor that receives the content photographed in real time from the source device at a predetermined location based on the location information received from at least one source device and provides it to the user terminal device.

Description

Server, user terminal device and its control method {MASTER DEVICE, SLAVE DEVICE AND CONTROL METHOD THEREOF}

The present invention relates to a server, a user terminal device and a control method thereof, and more particularly, to a server providing a VR service, a user terminal device and a control method thereof.

In general, VR (Virtual Reality) technology is all the senses of the human body (visual, auditory, olfactory, taste, tactile) in a 3D virtual environment similar to the real environment created through computer graphic (CG) technology. It refers to a technology that ultimately aims to maximize the utilization of information by immersing human beings in this virtual space in three dimensions and immersing humans in this virtual space by immersing them in the virtually created world.

For example, with a VR headset, a ski slope covered with white snow appears in front of you. It is right in the middle of the ski area even if you turn your head up and down, as well as before and after. The snowboard on the feet starts to slide and suddenly a cliff appears. At the moment, the feet are empty. The head recognizes that this is a virtual reality, but the realism makes the virtual ski area in the headset feel like reality.

As such, the VR service provides a new experience to the user, but there is a problem that it is not commercially serviced in relation to content purchase and consumption.

The present invention is in accordance with the above-described needs, and an object of the present invention is to provide a server, a user terminal device, and a control method for providing a virtual experience service by linking a plurality of devices.

A server that provides content to a user terminal device that provides a virtual reality service according to an embodiment of the present invention for achieving the above object, performs communication with at least one source device and a user terminal device When a content transmission request for a predetermined location is received from the communication unit and the user terminal device, content captured in real time is received from the source device at the preset location based on the location information received from the at least one source device. It includes a processor for transmitting to the user terminal device.

In addition, the processor sequentially transmits a plurality of photographed contents received from a plurality of source devices to the user terminal device based on location information received from the plurality of source devices, or photographs received from the plurality of source devices Contents may be synthesized based on location information received from the plurality of source devices, and the synthesized content may be transmitted to the user terminal device.

In addition, the processor, by selecting the content corresponding to the permission information from the content received from the source device, based on the permission information for setting the content sharing level that the at least one source device can share with the user terminal device It can be transmitted to the user terminal device.

In addition, the processor may process the content received from the source device according to a preset condition set in the user terminal device and transmit it to the user terminal device.

In addition, when the content received from the source device is 3D content, the processor generates a left-eye stereoscopic space image and a right-eye stereoscopic space image by projecting a left-eye image and a right-eye image included in the 3D content into a virtual stereoscopic space, respectively. And, when information corresponding to the motion state of the user terminal device is received from the user terminal device, the viewpoint area corresponding to the motion state of the user terminal device is extracted from the generated left-eye and right-eye stereoscopic images. Device.

In addition, a user terminal device providing a virtual reality service, when a display, a communication unit performing communication with the server, and a user command specifying a predetermined location are input, transmits information related to the location to the server And a processor that receives real-time shooting content related to a corresponding location from the server and displays it through the display.

In addition, the processor may receive only the content selected by the server based on permission information that sets a content sharing level that can be shared by the user terminal device by at least one source device that provides the real-time shooting content.

In addition, when a user command for setting a first point and a second point is input on the UI screen, the processor transmits information corresponding to a predetermined path connecting the first point and the second point to the server, , It may be controlled to receive and display a plurality of contents related to the predetermined path from the server.

In addition, when a user command for setting a first point and a second point is input on the UI screen, the processor transmits information corresponding to a predetermined path connecting the first point and the second point to the server, In addition, it is possible to control to receive and display the synthesized content obtained by synthesizing a plurality of contents related to the predetermined path from the server.

Further, when a user command for setting a preset region is input, the processor transmits information related to the region to the server, and receives location information for at least one source device located in the preset region from the server. It may be controlled to receive and display a GUI indicating a location for the at least one source device.

In addition, the processor may control the at least one source device from the server to receive permission information on a content sharing level that can be shared with the user terminal device, and display it in an area adjacent to a GUI indicating the location of the source device. Can.

In addition, when a user command for specifying the preset location is input on a UI screen providing a virtual experience service, the processor transmits information related to the location to the server, and the virtual experience photographed at the location from the server Content can be controlled to be received and displayed.

Further, the virtual experience service may include at least one of a virtual road experience service, a virtual tour service, and a virtual event experience service.

On the other hand, a control method of a server providing content to a user terminal device providing a virtual reality service according to an embodiment of the present invention comprises: receiving a content transmission request for a preset location from the user terminal device And, based on the location information received from at least one source device may include the step of receiving the content photographed in real time from the source device of the predetermined location and providing it in the form of streaming to the user terminal device.

In addition, the step of providing in the form of streaming to the user terminal device may sequentially transmit a plurality of photographed contents received from a plurality of source devices to the user terminal device based on location information received from the plurality of source devices, The photographed content received from the plurality of source devices may be synthesized based on the location information received from the plurality of source devices, and the synthesized content may be transmitted to the user terminal device.

In addition, the step of providing in the form of streaming to the user terminal device may include, among the contents received from the source device, based on permission information that sets a content sharing level that the at least one source device can share with the user terminal device. Content corresponding to the authority information may be selected and transmitted to the user terminal device.

In addition, in the step of providing in the form of streaming to the user terminal device, the content received from the source device may be processed according to a preset condition set in the user terminal device and transmitted to the user terminal device.

In addition, in the method of controlling a user terminal device providing a virtual reality service according to an embodiment of the present invention, when a user command for specifying a preset location is input, information related to the location is transmitted to the server And receiving the real-time shooting content related to the corresponding location from the server and displaying it through the display.

Further, in the displaying, at least one source device that provides the real-time photographed content receives and displays only the content selected by the server based on permission information that sets a content sharing level that can be shared with the user terminal device. can do.

In addition, in the step of transmitting to the server, when a user command for setting the first point and the second point is input on the UI screen, the information corresponding to the predetermined path connecting the first point and the second point is The step of transmitting to the server and displaying the received content is displayed by receiving a plurality of contents related to the predetermined path from the server, or receiving and displaying a composite content obtained by synthesizing a plurality of contents related to the predetermined path from the server. can do.

As described above, according to various embodiments of the present invention, it is possible to provide a variety of experiences to a user because it is possible to virtually experience the realism of a remote location using information acquired through another person's wearable device.

1 is a view showing a display device providing a virtual reality service according to an embodiment of the present invention.
2 is a view for explaining a content providing system according to an embodiment of the present invention.
3A and 3B are block diagrams showing the configuration of a server according to an embodiment of the present invention.
4A and 4B are block diagrams illustrating a configuration of a user terminal device according to an embodiment of the present invention.
5 is a diagram for explaining a method of generating and outputting panoramic VR content according to an embodiment of the present invention.
6A and 6B are diagrams illustrating a method of streaming content from a server according to an embodiment of the present invention.
7 is a flowchart illustrating an operation of a source device according to an embodiment of the present invention.
8 is a sequence diagram for explaining the operation of the server according to an embodiment of the present invention.
9 is a sequence diagram illustrating an operation of a user terminal device according to an embodiment of the present invention.
10 to 12 are diagrams for explaining a method of providing a virtual experience service according to various embodiments of the present invention.
11 is a view for explaining a method of providing a virtual tour service according to an embodiment of the present invention.
12 is a view for explaining a method of providing a virtual event experience service according to an embodiment of the present invention.
13 is a flowchart illustrating a method of controlling a server according to an embodiment of the present invention.
14 is a flowchart illustrating a control method of a user terminal device according to an embodiment of the present invention.

Hereinafter, the present invention will be described in more detail with reference to the accompanying drawings.

1 is a view showing a display device providing a virtual reality service according to an embodiment of the present invention.

According to FIG. 1, the user terminal device 100 according to an embodiment of the present invention may be worn on a user's head to function as a head mounted display (HMD). For example, the user terminal device 100 is implemented as a smart phone or a tablet, and is attached to the front surface of a body (eg, a housing) having the shape of glasses, a headset, or a helmet to provide images to both eyes of a user. have.

For example, when the user terminal device 100 implemented as a smart phone is attached to the front of the headset body, the display provided on the smart phone is positioned close to both eyes of the user. Accordingly, the user terminal device 100 may display an image through a display, thereby providing an image in front of the user. In this case, a pair of glasses legs or bends to be worn on the user's head may be formed on the back of the headset body. In addition, the headset body may be equipped with a track pad for operation, a return button, a volume adjustment key, and the like.

Alternatively, the user terminal device 100 has a display itself, and may be implemented as a device having a shape that can be worn on a user's head. In this case, the display can be fixedly installed rather than detachable.

Meanwhile, when the user wears the user terminal device 100 on the head, an area viewed by the user's left eye and an area viewed by the right eye on the display may be spatially separated. Accordingly, the user terminal device 100 may display different images in a region viewed by the user's left eye and a region viewed by the right eye on the display so that different images are incident to the left and right eyes.

Meanwhile, the user terminal device 100 is implemented to immediately update the field of view image by tracking the user's head movement, and may be implemented to provide a 3D image as well as a 2D image. For example, when the user wears the user terminal device 100 on the head, the user's field of view is completely controlled to provide a 360-degree stereoscopic image and sound, and the gyro sensor and acceleration sensor provided in the user terminal device 100 are used. It can be implemented to sense the user moving his head up and down or left and right to create a visual effect suitable for the direction.

Accordingly, the user can be provided with a 3D image existing in a direction of his gaze among panoramic 3D images, so that a user can experience a virtual reality (VR).

2 is a view for explaining a content providing system according to an embodiment of the present invention.

2, the content providing system according to an embodiment of the present invention is a user terminal device 100, a server 200 and at least one source device (300-1, 300-2, 300-3) It includes.

The user terminal device 100 may communicate with the server 200 to receive content transmitted from various types of source devices 200-1, 200-2, 200-3 to the server 200 in real-time streaming. have. In this case, the source device (300-1, 300-2, 300-3) may be implemented as a wearable device such as a smart glass (glass), a smart watch (watch) that can be taken, but a smart phone, camcorder, camera, etc. Any device having a photographing function is not limited to this and can be applied.

Accordingly, the user terminal device 100 is provided with the image captured in real time at a remote location through the server 200 to provide a virtual experience service to the user in the following, with reference to the drawings, the present invention Various embodiments of the present invention will be described in detail.

3A and 3B are block diagrams showing the configuration of a server according to an embodiment of the present invention.

According to FIG. 3A, the server 200 according to an embodiment of the present invention includes a communication unit 210 and a processor 220.

Here, the server 200 may be implemented as a central server (or an integrated server) in charge of interaction between various operating systems and applications in all network systems, or a cloud server using cloud computing technology. Here, cloud computing refers to Internet-based computing technology, and is a web-based software service that uses a program on a utility data server on the Internet and then calls it to a computer or mobile phone.

The communication unit 210 communicates with the user terminal device 100 and the source devices 300-1, 300-2, and 300-3.

The communication unit 210 may be implemented as a wireless communication module connected to an external network according to a wireless communication protocol such as Wi-Fi or IEEE to perform communication. In addition, the wireless communication module is a user terminal device 100 and a source device (300-1, 300-2) through various communication methods such as 3G (3rd Generation), 3GPP (3rd Generation Partnership Project), and LTE (Long Term Evoloution). , 300-3). Here, the communication unit 210 may communicate with the user terminal device 100 and the source devices 300-1, 300-2, 300-3 through one communication module, but communicates through a separate communication module. It may be implemented to perform. Here, the separate communication modules may be implemented in a form supporting the same communication method, or may be implemented in a form supporting different communication methods.

For example, when a predetermined event occurs, the communication unit 110 performs communication according to a predetermined communication method between the display device 100 and the source devices 300-1, 300-2, 300-3. It can become a state. Here, the linkage is an operation in which communication is initiated between the server 200 and the display device 100, and between the server 200 and the source devices 300-1, 300-2, 300-3, an operation in which a network is formed, and pairing It may mean any state in which communication is possible, such as an operation performed. For example, communication between the server 200 and the display device 100 may be performed according to an event driving an application that provides a virtual experience service according to the present invention in the display device 100.

The communication unit 210 receives images captured in real time from the source devices 300-1, 300-2, 300-3 under the control of the processor 220, which will be described later, and the received images to the display device 100 You can stream.

The processor 220 controls the overall operation of the server 200.

Particularly, when a request for content transmission for a preset location is received from the user terminal device 100, the processor 220 is configured to access the location information received from at least one source device (300-1, 300-2, 300-3). On the basis of this, the content photographed from the source device at a predetermined location may be received in real time and provided in the form of streaming to the user terminal device 100. Here, the at least one source device (300-1, 300-2, 300-3) is implemented as a wearable device, such as a smart glass (glass), smart watch (watch) as described above, a place for the source device You can shoot. Accordingly, the user of the user terminal device 100 does not directly visit the corresponding place, and experiences the current environment of the corresponding place in real time through the image captured by the source devices 300-1, 300-2, 300-3. I can do it.

On the other hand, the content received from the source device (300-1, 300-2, 300-3) may be a general 2D video, but is not limited thereto, and 3D including a left-eye and right-eye image for VR service provision. This may include images, panoramic VR content, and general 2D images.

In addition, when the content transmission request for the changed location is received from the user terminal device 100, the processor 220 photographs in real time from the source devices 300-1, 300-2, and 300-3 corresponding to the changed location information. The received content can be received in real time and streamed to the user terminal device 100. For example, when a user command for selecting a different location is input from the user terminal device 100, a content transmission request for the corresponding location may be transmitted to the server 200.

In this case, when the processor 220 receives a request for a virtual experience service for a specific place from the user terminal device 100, the source device 300-1, 300-2, 300-3 based on the map information of the corresponding place Through the command to transmit the information indicating the location of the user terminal device 100, the user of the user terminal device 100 based on the information displayed on the screen to select the source device that wants to share the captured image The content transmission request for the location may be transmitted to the server 200. However, in some cases, it is also possible to display the location of the user of the source device that can share the image within the current gaze range by applying AR technology to the currently displayed image. In this case, the corresponding source device Distance information to the user may be additionally displayed.

In addition, the processor 220 may receive a plurality of photographed contents received from a plurality of source devices 300-1, 300-2, 300-3 from a plurality of source devices 300-1, 300-2, 300-3. On the basis of the received location information, it is sequentially transmitted to the user terminal device 100, or a plurality of photographed contents received from a plurality of source devices 300-1, 300-2, 300-3 are transmitted to the plurality of source devices 300 -1, 300-2, 300-3) can be synthesized based on the location information received from the, and transmit the synthesized content to the user terminal device 100. For example, when the shooting content is transmitted to the user terminal device 100 on a path via a location → b location → c location, a plurality of source devices 300- corresponding to each of a location, b location, and c location 1, 300-2, 300-3) sequentially provide the captured images to the user terminal device 100, or a plurality of source devices 300-1, 300 corresponding to each of a, b, and c positions -2, 300-3) can be transmitted to the user terminal device 100 by constructing a composite image sequentially connected to the captured image.

In this case, the processor 220 is based on the content information that varies according to the resolution information, the display arrangement state, the screen size, etc. of the plurality of source devices (300-1, 300-2, 300-3) and the user terminal device 100 Content received from a plurality of source devices 300-1, 300-2, 300-3 may be processed to be suitable for the user terminal device 100 and transmitted to the user terminal device 100. However, in some cases, it may be possible for the user terminal device 100 to process and display the received content to match the display format.

In addition, the processor 220 processes a plurality of photographed contents received from the plurality of source devices 300-1, 300-2, and 300-3 to match the display format of the user terminal device 100, and the processed contents It can be synthesized and transmitted to the user terminal device 100.

In addition, the processor 220 may transmit, to the user terminal device 100, only at least some content corresponding to the authority information, based on the authority information for the at least one source device (300-1, 300-2, 300-3 ). Can. Here, the authority information refers to information in which a user of at least one source device (300-1, 300-2, 300-3) sets a sharing level of content that he or she can share with other terminal devices. Also, when a plurality of contents are synthesized and provided, the processor 220 may synthesize and transmit at least some of the contents corresponding to the authority information to the user terminal device 100.

That is, the user of the at least one source device (300-1, 300-2, 300-3) can set the content sharing level such as audio data, video data, location information, whether or not a message is received, and the processor 220 Among content received from at least one source device (300-1, 300-2, 300-3), only content that can be shared by permission information may be selected and transmitted to the user terminal device 100.

For example, when the user of the at least one source device (300-1, 300-2, 300-3) only sets the video data and sets the permission information not to share the audio data, only the video data is used by the user terminal device. (100). As another example, when the user of the at least one source device (300-1, 300-2, 300-3) sets the authority information to receive a message from another terminal device, the user of the user terminal device 100 is at least It is also possible to transmit a message to one source device (300-1, 300-2, 300-3).

On the other hand, in the above-described embodiment, the server 200 determines the authority information, and selects the content that can be shared by the authority information among the contents received from at least one source device (300-1, 300-2, 300-3). It has been described as being transmitted to the user terminal device 100, but according to another embodiment, only the content corresponding to the authority information from at least one source device (300-1, 300-2, 300-3) to the server 200 It is also possible to transmit. For example, if the user of at least one source device (300-1, 300-2, 300-3) sets the permission information to share only video data and not audio data, the server 200 may Only video data may be received from at least one source device (300-1, 300-2, 300-3).

Also, the processor 220 may process the content received from at least one source device 300-1, 300-2, 300-3 and transmit it to the user terminal device 100.

In this case, the processor 220 processes the content received from the at least one source device (300-1, 300-2, 300-3) according to a preset condition set in the user terminal device 100 to process the user terminal device ( 100).

For example, when the user of the user terminal device 100 is set to receive the virtual experience service in the form, the processor 220 receives from at least one source device (300-1, 300-2, 300-3) The processed content may be processed into a panorama content form and transmitted to the user terminal device 100.

Specifically, the processor 220 projects the left-eye image and the right-eye image included in the 3D content received from the source devices 300-1, 300-2, and 300-3 into a virtual stereoscopic space, respectively, and the left-eye stereoscopic spatial image and A right-eye stereoscopic spatial image can be generated. Thereafter, when the information corresponding to the motion state of the user terminal device 100 is received from the user terminal device 100, the processor 120 receives the motion state of the user terminal device 100 from the generated left-eye and right-eye stereoscopic images. It is possible to extract the viewpoint region corresponding to and transmit it to the user terminal device 100.

In this case, the processor 220 may determine a point of view (or direction) area corresponding to the motion state of the user terminal device 100 among 3D images input based on the motion information received from the user terminal device 100. Thereafter, the processor 220 may transmit the determined image of the view area to the user terminal device 100. Here, the motion information may be real-time direction information and angle information for all directions of the user terminal device 100, but is not limited thereto, and movement speed information, rotation information, and the like may also be included. Further, after the initial direction information and the angle information are received from the user terminal device 100, it is also possible to receive the moved direction and angle information based on the previously received direction and angle information, but is not limited thereto.

In this case, the processor 220 may transmit an image corresponding to the user terminal device 100 to the corresponding user terminal device 100 based on device information received from the user terminal device 100, for example, a device ID. . In addition, the processor 220 secures a region of the image buffer as a transmission buffer based on the device display information received from the user terminal device 100, and the image corresponding to the motion information of the user terminal device 100 in the transmission buffer Can be buffered and transmitted to the corresponding user terminal device 100. In this case, the processor 220 may separately secure a transmission buffer corresponding to each user terminal device 100.

In addition, the processor 220 decompresses (or decodes) the input image, stores it in an image buffer, extracts an area corresponding to motion information of the user terminal device 100 from the image stored in the image buffer, and then extracts the extracted image. It may be compressed (or encoded) and transmitted to the user terminal device 100.

Alternatively, the processor 220 may extract an area corresponding to the motion information of the user terminal device 100, decode it, and transmit it to the user terminal device 100, and a margin other than the previously transmitted image among the areas corresponding to the motion information It is also possible to decode and transmit only the region.

Alternatively, the processor 220 may be determined based on the motion information of the user terminal device 100 together with the main video decoded based on the content received from the source devices 300-1, 300-2, 300-3. The margin image for the region may be transmitted to the user terminal device 100 as a low-quality image or an encoded image using a codec. Here, the margin image is an image excluding the decoded main image among regions determined based on motion information of the user terminal device 100

Can be. In this case, the user terminal device 100 may decode the received margin image and synthesize and output the received main image, thereby minimizing delay due to image transmission.

In addition, the processor 220 predicts future motion information of the slave devices 200-1, 200-2, and 200-3 based on the motion information of the currently received user terminal device 100, and the predicted motion information On the basis of the input image, an image of a viewpoint region corresponding to a future motion state of the user terminal device 100 may be transmitted to the user terminal device 100. For example, the processor 220 may track and predict a future motion state of the user terminal device 100 using a filtering method such as an existing Kalman filter.

In addition, when the motion information received from the plurality of user terminal devices 100 falls within a predetermined threshold range, the processor 220 groups the plurality of user terminal devices into one group, and user terminal devices belonging to the same group ( 100) may transmit an image of the same viewpoint area. For example, the processor 220 groups the first and second user terminal devices having motion information between 10° and 15° in the right direction into a group to remove an image of the same viewpoint area corresponding to the motion information. It can be transmitted to the first and second user terminal devices.

3B is a block diagram showing a specific configuration of the server shown in FIG. 3A. According to FIG. 3B, the server 200 ′ includes a communication unit 210, a processor 220, and a storage unit 230. Details of parts overlapping with those shown in FIG. 3A among the configurations shown in FIG. 3B will be omitted.

The storage unit 230 stores various data such as O/S (Operating System) software module for driving the server 200 and various information required to perform the functions of the server 200. For example, the storage 230 may store authority information for each source device (300-1, 300-2, 300-3).

Meanwhile, the operation of the above-described processor 220 may be performed by a program stored in the storage unit 230.

Specifically, the processor 220 includes a RAM 221, a ROM 222, a main CPU 223, a graphics processor 224, first to n interfaces 225-1 to 225-n, and a bus 226. Includes.

The RAM 221, the ROM 222, the main CPU 223, the graphic processing unit 224, and the first to n interfaces 225-1 to 225-n may be connected to each other through the bus 226.

The first to n interfaces 225-1 to 225-n are connected to various components described above. One of the interfaces may be a network interface connected to an external device through a network.

The main CPU 223 accesses the storage unit 230 to boot using the O/S stored in the storage unit 230. In addition, various operations are performed using various programs, contents, and data stored in the storage unit 230. For example, the main CPU 223 uses a variety of modules stored in the storage unit 230 to determine the source device at a location desired by the user terminal device 100, and the user terminal device to view the captured image received from the source device. It can be provided as (100).

The ROM 222 stores a set of instructions for booting the system and the like. When the turn-on command is input and power is supplied, the main CPU 223 copies the O/S stored in the storage unit 230 to the RAM 221 according to the command stored in the ROM 222, and executes the O/S. Boot the system. When the booting is completed, the main CPU 223 copies various application programs stored in the storage unit 230 to the RAM 221 and executes the application programs copied to the RAM 221 to perform various operations.

The graphic processing unit 224 generates a screen including various objects such as icons, images, and text using an operation unit (not shown) and a rendering unit (not shown). The operation unit (not shown) calculates attribute values such as coordinate values, shapes, sizes, colors, etc. to be displayed according to the layout of the screen based on the received control command. The rendering unit (not shown) generates screens of various layouts including objects based on attribute values calculated by the calculation unit (not shown).

Meanwhile, the processor 220 may provide content to the user terminal device 100 by using pre-stored data in some cases.

Specifically, the processor 220 may select and provide an image according to the request of the user terminal device 100 from a database in which a captured image is previously stored, or may synthesize and provide the selected image. For example, when a user of the user terminal device 100 sets a start point and an arrival point in the virtual space, an optimal route is generated, and image data from the database to the start and end are selected according to the generated route, , It is possible to acquire the landscape images one by one from the starting point and transmit them to the user terminal device 100. In addition, the virtual space is set by mapping the image obtained from the database to the virtual space, and by generating a virtual camera in the virtual space through the VR engine, the image of the virtual space projected by the virtual camera is transmitted to the user terminal device 100. Can provide. In this case, the position of the camera in the virtual space is determined according to the generated path, and the direction of the camera can be determined from the direction of the HMD.

Also, the processor 220 may provide various modes for each virtual experience service. For example, in the case of a virtual path finding experience service, an automatic car mode, a walking mode, a running mode, and the like may be provided, and audio feedback and haptic feedback may be provided according to a user's selection. The car mode is a mode that provides a virtual road experience while moving in a path progressing direction within landscape data obtained from a database. In this movement mode, the close distance moves in the current stereoscopic space image, and if a long distance movement is needed, if there is a connected input image or a connected stereoscopic space image, it can be provided by combining the current stereoscopic space image and the stereoscopic space image when moving. have. In this case, based on the metadata generated or stored through image analysis, information such as a corresponding audio/haptic may be obtained from a database and provided to the user terminal device 100. Here, examples of meta data include natural sounds such as bird sounds, cicada sounds, rain sounds, car sounds, train sounds, airplane sounds, artificial sound data such as airplane sounds, haptic data having profiles of various cycles and strengths, and adjusting the screen brightness of images. Data.

On the other hand, the walking mode and the running mode are basically similar modes, but there are differences in the movement speed in the same landscape, and there may be a difference in the position of the virtual camera in the virtual space. In this case, when a path is selected by a user input and a walking mode and a running mode are selected, a corresponding path is generated and ripples are generated in a vertical direction of the generated path. Here, the period and amplitude of the pulse wave vary depending on the walking mode and the running mode, and may have a predetermined size. If the image obtained from the database is a Full 3D image, the image may be provided through a change in camera position/direction according to pulse waves and a virtual VR engine. Alternatively, if the image obtained from the database is a binocular parallax image, depth information may be extracted to generate a virtual viewpoint image, and information of an exposed region may be virtually generated based on the extracted depth information. Alternatively, if the image obtained from the database is a monocular image, the image may be provided based only on the conversion of the position/direction of the virtual camera. Meanwhile, a zoom function may be provided according to a user's selection during movement according to each mode. Through the zoom function, the user can view the image by enlarging the landscape. When changing the head motion in the zoom state, for example, the amount of change in the head motion becomes larger than a certain value compared to the start of the zoom function, or the user's zoom mode release input If you enter, you can exit the zoom function.

4A and 4B are block diagrams illustrating a configuration of a user terminal device according to an embodiment of the present invention.

According to FIG. 4A, the user terminal device 100 according to an embodiment of the present invention includes a display 110, a communication unit 120, and a processor 130.

The display 110 displays various information.

Here, the display 110 may be implemented as a liquid crystal display panel (LCD), organic light emitting diodes (OLED), or the like, but is not limited thereto. In addition, the display 110 may be implemented as a flexible display, a transparent display, and the like in some cases. Meanwhile, as mentioned in FIG. 1, when a smartphone is connected and used as a monitor, the display 110 may be implemented as a display of a smartphone.

The communication unit 120 communicates with the server 200. The communication unit 210 may be implemented as a wireless communication module connected to an external network according to a wireless communication protocol such as Wi-Fi or IEEE to perform communication. In addition, the wireless communication module may perform communication with the server 200 through various communication methods such as 3G (3rd Generation), 3GPP (3rd Generation Partnership Project), and LTE (Long Term Evoloution).

The processor 130 controls the overall operation of the user terminal device 100.

In particular, when a user command for specifying a preset location is input, the processor 130 transmits information related to the location to the server 200 and receives and displays real-time shooting content related to the location from the server 200 ( 110).

Here, the user command for specifying a preset location may be a user command for selecting a specific place, a user command for selecting an event at a specific place, or the like.

In addition, the processor 130 receives only the content selected by the server 200 based on permission information that sets a content sharing level that can be shared by the user terminal device 100 by at least one source device that provides real-time shooting content. can do.

In addition, when a user command for setting a first point and a second point is input on the UI screen, the processor 130 transmits information corresponding to a predetermined path connecting the first point and the second point to the server 200. Then, a plurality of contents related to a predetermined path may be received from the server 200 and displayed on the screen.

In addition, when a user command for setting a first point and a second point is input on the UI screen, the processor 130 transmits information corresponding to a predetermined path connecting the first point and the second point to the server 200. Then, the composite content obtained by synthesizing a plurality of contents related to a predetermined path from the server 200 may be received and displayed on the screen.

In addition, when a user command for setting a preset region is input, the processor 130 transmits information related to the region to the server 200, and transmits information related to the region to the server 200 to at least one source device located in the preset region. A GUI indicating location of at least one source device may be displayed on the screen by receiving location information for the device.

In addition, the processor 130 receives the authority information for the content sharing level that at least one source device can share with the user terminal device 100 from the server 200, and displays it in an adjacent area of the GUI indicating the location of the source device can do.

In addition, when a user command for specifying a preset location is input on a UI screen that provides a virtual experience service, the processor 130 transmits information related to the location to the server 200 and from the server 200 at the location. The captured virtual experience content may be received and displayed. Here, the virtual experience service may include at least one of a virtual road experience service, a virtual tour service, and a virtual event experience service.

Meanwhile, the processor 130 may zoom in or zoom out the displayed content according to a user's zoom in/out command to enlarge or reduce an image.

Specifically, when the user terminal device 100 is implemented as an HMD, the display 110 displays a left-eye image in a display area viewed by the user's left eye and a right-eye image in a display region viewed by the user's right eye to the user It is possible to provide a stereoscopic image. Here, the left-eye image and the right-eye image may be implemented as a panoramic 3D image.

In this case, the processor 130 may generate a left-eye stereoscopic space image and a right-eye stereoscopic space image by projecting each of the left-eye image and the right-eye image constituting the 3D image into a virtual stereoscopic space. Here, the generated stereoscopic spatial image may be a panoramic 3D image. That is, the processor 130 projects each of the left-eye image and the right-eye image into the virtual sphere space, so that the left-eye image and the right-eye image exist in the 360° direction based on the center of the virtual sphere space, respectively. It is possible to generate a stereoscopic spatial image.

In addition, the processor 130 may extract an image positioned in a direction viewed by a user wearing the HMD from the stereoscopic space image, and display the extracted image through the display 110. Specifically, the processor 130 determines a user's gaze direction using a sensor (not shown) provided in the display apparatus 100, and uses a virtual camera located at the center of each of the left-eye stereoscopic image and the right-eye stereoscopic spatial image. Thus, an image existing in a region facing the user's gaze may be extracted from each of the left-eye stereoscopic space image and the right-eye stereoscopic space image and displayed. To this end, the processor 130 detects a direction in which the head of the user wearing the display device 100 moves by using at least one of a gyro sensor, an acceleration sensor, and a geomagnetic sensor provided in the display device 100 so that the user You can judge the direction you are looking at.

Then, the processor 130 positions the virtual camera having a constant field of view (FOV) at the center of each of the left-eye stereoscopic spatial image and the right-eye stereoscopic spatial image, and then rotates the virtual camera in the user's gaze direction to rotate the left-eye stereoscopic image. An image existing in a user's gaze direction may be extracted from each of the spatial image and the right-eye stereoscopic spatial image and displayed. For example, when it is determined that the user is looking at a 60° direction to the left with respect to the front, the processor 130 moves the virtual camera located at the center of each of the left-eye stereoscopic image and the right-eye stereoscopic image to the left 60° direction. By rotating to, the image existing in the left 60° direction can be extracted from each of the left-eye stereoscopic space image and the right-eye stereoscopic space image.

In this case, the processor 130 displays the image extracted from the left-eye stereoscopic space image in an area viewed by the left eye of the user wearing the display device 100 on the display 110, and displays the image extracted from the right-eye stereoscopic space image. The right eye of the user wearing the display device 100 may display the area viewed from the display 110. Accordingly, the user can be provided with a 3D image of a direction that his gaze is directed from among the panoramic 3D images.

Meanwhile, when a user command for enlarging or reducing a stereoscopic image is input, the processor 130 adjusts the stereoscopic effect for the stereoscopic image to correspond to the user command and displays the stereoscopic image of the stereoscopic image to display the enlarged or reduced image with the stereoscopic effect adjusted thereon 110 Can be controlled. Here, the enlargement command includes a full enlargement command for enlarging the entire stereoscopic image and a partial enlargement command for enlarging a part of the stereoscopic image, and the reduce command is a full reduction command for reducing the entire stereoscopic image and a part of the stereoscopic image It includes a partial reduction command to reduce.

The processor 130 moves each of the left-eye image and the right-eye image included in the stereoscopic image, and projects each of the moved left-eye image and right-eye image to a virtual stereoscopic space image to generate a left-eye stereoscopic space image and a right-eye stereoscopic space image, An image corresponding to a user command may be extracted from each of the left-eye stereoscopic space image and the right-eye stereoscopic space image and displayed. That is, the processor 130 moves each of the left-eye image and the right-eye image so as to correspond to a user command before projecting the left-eye image and the right-eye image into the virtual three-dimensional space, and then moves each of the moved left-eye image and right-eye image into the virtual three-dimensional space. The left-eye stereoscopic space image and the right-eye stereoscopic space image may be generated by projecting the image on the image.

Specifically, when a full enlargement command for enlarging the entire stereoscopic image is input, the processor 130 moves the left-eye image to the left and moves the right-eye image to the right, the left-eye image moved to the left, and the right-eye image moved to the right. A left-eye stereoscopic space image and a right-eye stereoscopic space image may be generated using each of the images, and an image corresponding to an entire enlargement command may be extracted from each of the left-eye stereoscopic space image and the right-eye stereoscopic space image and displayed. That is, when the full zoom command is input, the processor 130 moves the left-eye image and the right-eye image to the right to generate a left-eye stereoscopic space image and a right-eye stereoscopic space image, and zooms in each of the left-eye stereoscopic space image and the right-eye stereoscopic space image. Images can be extracted and displayed.

Specifically, the processor 130 moves the left-eye image to the left, and generates a left-eye stereoscopic space image using the left-eye image moved to the left. In addition, the processor 130 reduces the amount of the angle of view of the virtual camera existing in the left-eye stereoscopic space image by an enlargement ratio for the entire enlargement command, and uses the virtual camera with the reduced amount of angle of view to enlarge the entire angle in the left-eye stereoscopic space image. The enlarged image corresponding to the command can be extracted. In this case, the processor 130 only adjusts the angle of view of the virtual camera, but does not change the focus position of the virtual camera. That is, the processor 130 maintains the focal position of the virtual camera the same as before the full magnification command is input, and displays the magnified image corresponding to the full magnification command in the left-eye stereoscopic space image generated based on the left-eye image moved to the left. Can be extracted. As described above, when a left-eye stereoscopic image is generated by a left-eye image moved to the left, the left-eye stereoscopic image may have a state rotated counterclockwise compared to before moving to the left.

Therefore, when the enlarged image is extracted from the left-eye stereoscopic space image rotated counterclockwise while maintaining the focus position of the virtual camera, the left-eye stereoscopic image is not moved to the right side than when the enlarged image is extracted. It is possible to acquire an existing enlarged image. That is, when an enlarged image is extracted by moving the left-eye image to the left, an enlarged image existing on the right side may be extracted than when the enlarged image is extracted from the left-eye image before moving.

Meanwhile, the processor 130 may display the obtained enlarged image through the display 110. Specifically, the processor 130 may display an enlarged image obtained from the left-eye stereoscopic image in the area viewed by the user's left eye on the display 110. In this case, the processor 130 may adjust and display the size of the enlarged image to have the same size as the image displayed on the display 110 before the entire enlargement command is input.

However, the above-described image processing, generating a panoramic stereoscopic image, extracting an image according to a motion direction, and extracting an image according to a zoom in/out command may be performed by the server 200, and the user terminal device 100 may perform a server 200 Of course, it may be implemented to display only the image received from ). For example, when a zoom-in command is input from the user terminal device 100, the user terminal device 100 transmits the command to the server 200, and the server 200 performs image processing according to the zoom-in command. The obtained image may be provided to the user terminal device 100.

4B is a block diagram illustrating a specific configuration of the user terminal device shown in FIG. 4A. According to FIG. 4B, the user terminal device 100 ′ includes a display 110, a communication unit 120, a processor 130, a storage unit 140, an image processing unit 150, and a sensor unit 160. Details of parts overlapping with those shown in FIG. 4A among the configurations shown in FIG. 4B will be omitted.

The storage unit 140 stores various data such as an operating system (O/S) software module and various multimedia contents for driving the user terminal device 100'.

The user interface unit 150 receives various user commands. Specifically, the user interface unit 150 may receive user commands for various UI screens for function control of the user terminal device 100 ′ provided through the display 110. In this case, the user interface unit 150 may be implemented with a keypad, a touch pad, or the like provided on one side of the user terminal device 100, but in some cases, a camera for motion recognition, a microphone for voice recognition, etc. Can be.

The sensor unit 160 senses the motion state of the user terminal device 100'.

Specifically, the sensor unit 160 includes a geomagnetic sensor, a gyro sensor, and an acceleration sensor for sensing a motion state. The geomagnetic sensor is a sensor for detecting a rotation state and a moving direction of the user terminal device 100'. The gyro sensor is a sensor for detecting the rotation angle of the user terminal device 100'. Both the geomagnetic sensor and the gyro sensor may be provided, but even if only one of them is provided, the user terminal device 100 ′ may detect a rotation state. The acceleration sensor is a sensor for sensing the degree of inclination of the user terminal device 100'.

In addition, the sensor unit 160 may further include a touch sensor, a proximity sensor, and the like for sensing a touch or hovering operation through a touch pad attached to the device.

In this case, the processor 130 transmits the information corresponding to the motion state sensed by the sensor unit 160 to the server 200 and receives the content corresponding to the motion state from the server 200 to display the display 110. Can be displayed.

The image processing unit 170 functions to render the received image and provide it to the display 110.

Meanwhile, since the detailed configuration of the processor 130 illustrated in FIG. 4B is similar to the detailed configuration of the processor 220 of the server 200 illustrated in FIG. 3B, a detailed description thereof will be omitted.

5 is a diagram for explaining a method of generating and outputting panoramic VR content according to an embodiment of the present invention.

5, when the 3D image is input from at least one source device (300-1, 300-2, 300-3), the server 200 separates the left and right images (510), and virtualizes each of them. A left-eye stereoscopic image and a right-eye stereoscopic image are generated by projecting on the stereoscopic circular space of (520). In this case, the generated stereoscopic image may be a panoramic VR image, and the existing 3D-based avatar, object, space, and photorealistic panoramic VR and photo object (Photo) Object) A panoramic VR image can be generated using various VR techniques such as VR.

However, when receiving a stereo image in which the left-eye image and the right-eye image are separated, the corresponding left-right image may be used as it is, or a 3D image may be generated based on the 2D image according to the 2D-3D conversion technique, but detailed description thereof Is omitted.

Thereafter, a desired viewpoint image is selected through a virtual camera engine and transmitted to a user terminal device (eg, HMD) 100, and the user terminal device 100 provides 3D VR by providing the received images as left and right images respectively. It is possible to provide an image.

Meanwhile, when the changed motion information is received from the user terminal device 100, the server 200 analyzes the received motion information (540) and adjusts the direction of the virtual camera based on the analyzed motion information (550). The left and right images of the viewpoint region corresponding to the corresponding motion information may be extracted through the virtual camera engine 530 and transmitted to the user terminal device 100. That is, if it is assumed that there is a virtual camera in the circular stereoscopic space, the direction of the virtual camera is adjusted in a direction corresponding to the motion of the user terminal device 100 and the virtual camera whose orientation is adjusted among the images mapped in the circular stereoscopic space is adjusted. The captured image can be extracted and transmitted to the user terminal device 100.

6A and 6B are diagrams illustrating a method of streaming content from a server according to an embodiment of the present invention.

According to FIG. 6A, when a plurality of images 610 to 640 photographed from a plurality of source devices are received, the server 200 may respectively provide them to the output buffer 650 and individually transmit them to the user terminal device 100. have. In this case, the output buffer 650 may transmit a plurality of images 610 to 640 sequentially or simultaneously to the user terminal device 100. Here, the plurality of images (610 to 640) may be a general 2D video, but is not limited thereto, and 3D video including a left-eye and right-eye video for providing VR service, panoramic VR content, general 2D video, etc. may also be included. Can.

According to FIG. 6B, when a plurality of images 610 to 640 photographed from a plurality of source devices are received, the server 200 synthesizes them (660) and provides them to the output buffer 650 to provide a user terminal device 100 Can be transferred to. In this case, the received plurality of images 610 to 640 may be synthesized based on the location information of the plurality of source devices providing them.

On the other hand, the server 200 receives the image based on the content information that varies according to the resolution information, display arrangement state, screen size, etc. of the plurality of source devices and the user terminal device 100 providing the plurality of images 610 to 640 It may be processed to be suitable for the user terminal device 100 and transmitted to the user terminal device 100, or the processed content may be synthesized and transmitted to the user terminal device 100.

7 is a flowchart illustrating an operation of a source device according to an embodiment of the present invention.

According to the flowchart illustrated in FIG. 7, the source devices 300-1, 300-2, and 300-3 start providing information according to a preset event (S711). Here, the preset event may be various events that can be viewed as starting to provide information to a third party, such as an event of installing/driving an application for providing the corresponding service, an event of subscribing to the corresponding service.

On the other hand, when the user of the source devices 300-1, 300-2, and 300-3 starts providing information, it requests a connection to the server 200 and sets the information providing level (S720). For example, the level of information sharing with third parties, such as audio data, video data, location information, and whether or not a message is received, can be set.

Subsequently, when an information provision request is received from the server 200 (S730:Y), the corresponding information is transmitted to the server 200 according to the information provision level set in step S720 (S740).

8 is a sequence diagram for explaining the operation of the server according to an embodiment of the present invention.

According to the flowchart shown in FIG. 8, the server 200 registers/manages an information provider according to a preset event (S810). For example, when a user of the source device (300-1, 300-2, 300-3) subscribes to the corresponding service, the user of the registered source device (300-1, 300-2, 300-3) is registered And manage it. In this case, the server 200 may also manage information providing levels set by the users of the source devices 300-1, 300-2, and 300-3 for each user.

Subsequently, when a connection request is made from the user terminal device 100 (S820 :Y), a user of the user terminal device 100 that requested the connection is checked (S830 ). For example, the server 200 may check whether the corresponding user is a user subscribed to a service provided by the server 200. In this case, whether the user is a user subscribed to the service provided by the server 200 through user authentication information (for example, ID, password, etc.) as well as device information (for example, device identification information of the user terminal device). Can be confirmed. However, since a single device may be used by multiple users, it may be desirable to confirm the user through user authentication information.

Meanwhile, when the user of the user terminal device 100 is identified in step S830, the user terminal device 100 provides the location of the information provider, that is, the source devices 300-1, 300-2, 300-3 (S840). ). In this case, the location of the information provider for the place set by the user terminal device 100 may be provided. For example, when the user of the user terminal device 100 requests information about the vicinity of Gangnam Station, the user of the location of the source device 300-1, 300-2, 300-3 owned by the information provider for the vicinity of Gangnam Station It can be provided to the terminal device 100. In this case, the server 200 may provide an image (for example, a map image) in which the location of the information provider is displayed, but it is also possible to provide only the location information of the information provider.

Subsequently, when an information request for at least one specific location is received from the user terminal device 100 (S850), the image received from the source device of the corresponding location is transmitted to the user terminal device 100 (S860). In this case, the server 200 may buffer and synthesize the received image or provide it by streaming in real time.

9 is a sequence diagram illustrating an operation of a user terminal device according to an embodiment of the present invention.

According to the flowchart illustrated in FIG. 9, the user terminal device 100 accesses a server 200 that provides a virtual experience service (S910), and sets an area to access (S920).

Subsequently, the user terminal device 100 sets an information request authority level (S930). For example, an information request permission level may be set, such as whether audio data and video data are to be provided, whether only video data is to be provided, or whether to send a message to an information provider.

Thereafter, the user terminal device 100 receives and renders information of the level requested from the server 200, for example, an image (S940). In this case, when the level of information requested by the user terminal device 100 is higher than the level of information provided by the information provider, that is, the source devices 300-1, 300-2, and 300-3, the corresponding source device 300-1, Information may be provided within the information providing level of 300-2 and 300-3). For example, the user terminal device 100 requests video and audio from the first source device 300-1, but when the information providing level of the first source device 300-1 is set to video only, the server 200 ) May transmit only video to the user terminal device 100.

Thereafter, the user terminal device 100 displays the rendered image (S950). That is, when the user terminal device 100 is implemented as an HMD, a user of the user terminal device 100 is provided with a real-time image of a specific place through a displayed image to enable a virtual experience.

10 to 12 are diagrams for explaining a method of providing a virtual experience service according to various embodiments of the present invention.

As illustrated in FIGS. 10 to 12, a user terminal device 100, that is, an HMD user, is provided to another user's field of view by receiving an image of a specific place from an external user using a source device, that is, a wearable device It can provide a service in the form of receiving. In this case, the user of the wearable device can set the content sharing level through the permission setting of his/her device. For example, the level of sharing video only through the camera, the level of sharing sound information only through the microphone, the level of sharing both video and sound, the level of not sharing both the video and sound, and text messages from the user separately from each sharing level The content sharing level can be set to a level that provides permission to allow input of voice information.

10 is a view for explaining a method of providing a virtual road experience service according to an embodiment of the present invention.

As illustrated in FIG. 10, when an HMD user selects an area for receiving a virtual road experience service, the locations of a plurality of users 1011, 1012, and 1013.. with wearable devices in the selected area are displayed (1021, 1022, 1023..) may be displayed UI screen 1020 may be provided. In this case, the content sharing level set by the wearable device users may also be displayed. In this case, the HMD user can share the video (A, B, C..) of the wearable device user through the server 200 by selecting the user corresponding to the desired content sharing level and location.

Alternatively, by locating the user's location close to the destination route in connection with the route finding function, the wearable device users (1011, 1012, 1013..) located on the route are provided with images and sounds in sequence in order to simulate Real-time route conditions can be predicted. In this case, when the distance between the wearable device users located on the path is greater than or equal to a predetermined range, data such as a pre-stored street view may be provided to the HMD user instead of sharing the view of the wearable device user.

11 is a view for explaining a method of providing a virtual tour service according to an embodiment of the present invention.

As illustrated in FIG. 11, when an HMD user selects a region for receiving a virtual tour service, the locations of a plurality of users 1111, 1112, 1113.. with wearable devices in the selected region are displayed (1121, 1122) , 1123..) UI screen 1120 may be provided. In this case, the content sharing level set by the wearable device users may also be displayed. In this case, the HMD user can share the video of the wearable device user through the server 200 by selecting the user corresponding to the desired content sharing level and location.

12 is a view for explaining a method of providing a virtual event experience service according to an embodiment of the present invention.

As illustrated in FIG. 12, when a HMD user selects a performance for receiving a virtual performance experience service through a corresponding UI screen 1220, a performance image received from a plurality of users having a wearable device in the selected venue is shared. It becomes possible. In some cases, a UI screen displaying locations of a plurality of users who can share content in a venue may be provided. In this case, the content sharing level set by the wearable device users may also be displayed. Accordingly, the HMD user can selectively share the video captured in the desired seat even in the venue.

Alternatively, the audio information is obtained from a user who is closest to the performance location among a plurality of wearable device users in the venue, and the image information is collected/combined by each user, and various viewpoints between users are changed according to user input. You can also provide a video.

13A and 13B are diagrams illustrating an example of an implementation of a UI screen provided in a user terminal device according to an embodiment of the present invention.

As illustrated in FIG. 13A, when a user command for setting a preset region is input, the user terminal device 100 transmits information related to the region to the server 200 and locates it in the preset region from the server 200. The UI screens indicating the locations 1311 to 1314 for at least one source device may be displayed by receiving location information for at least one source device. In this case, as illustrated, at least one source device from the server 200 may receive and display authority information on a content sharing level that can be shared with the user terminal device 100.

As shown in FIG. 13B, it is also possible to display the locations 1321 to 1324 of the user of the source device that can share the image within the current gaze range by applying AR technology to the image displayed at the current virtual experience location. In this case, as illustrated, distance information to the corresponding source device user may be additionally displayed.

On the other hand, the user of the user terminal device 100 is based on the information displayed on the UI screen as shown in FIGS. 13A and 13B, content for a specific location through a command to select a source device that wants to share the captured image The transmission request may be transmitted to the server 200.

14 is a flowchart illustrating a method of controlling a server according to an embodiment of the present invention.

According to the control method of the server illustrated in FIG. 14, when a content transmission request for a predetermined location is received from a user terminal device (S1410:Y), the server determines the location based on the location information received from at least one source device. The real-time photographed content is received from the source device at the set location (S1420).

Subsequently, the received content is provided in a streaming form to the user terminal device (S1430).

Meanwhile, in step S1330 in which the received content is provided in a streaming form to the user terminal device, the plurality of photographed content received from the plurality of source devices are sequentially transmitted to the user terminal device based on the location information received from the plurality of source devices. Alternatively, the captured content received from the plurality of source devices may be synthesized based on the location information received from the plurality of source devices, and the synthesized content may be transmitted to the user terminal device.

In addition, in step S1430 in which the received content is provided in a streaming form to the user terminal device, the content received from the source device is based on the authority information that sets the content sharing level that at least one source device can share with the user terminal device. Content corresponding to the corresponding authority information may be selected and transmitted to the user terminal device.

In addition, in step S1430 of providing the received content in a streaming form to the user terminal device, the content received from the source device may be processed according to a preset condition set in the user terminal device and transmitted to the user terminal device.

In addition, the control method, when the content received from the source device is 3D content, generating a left-eye stereoscopic space image and a right-eye stereoscopic space image by projecting the left-eye image and the right-eye image included in the 3D content into a virtual stereoscopic space, respectively. , When receiving information corresponding to the motion state of the user terminal device from the user terminal device, extracting a viewpoint region corresponding to the motion state of the user terminal device from the generated left-eye and right-eye stereoscopic space images and transmitting the extracted area to the user terminal device It may further include.

15 is a flowchart illustrating a control method of a user terminal device according to an embodiment of the present invention.

According to the control method of the user terminal device shown in FIG. 15, when a user command for specifying a predetermined location is input (S1510: Y), information related to the location is transmitted to the server (S1520).

Thereafter, real-time shooting content related to the corresponding location is received from the server and displayed through a display (S1530).

In step S1530 of receiving and displaying the content, at least one source device providing real-time shooting content may receive only the content selected by the server based on the permission information that sets the content sharing level that can be shared with the user terminal device. . However, in some cases, it is also possible for the source device to transmit only the selected content based on the authority information to the server.

In addition, in steps S1510, S1520, and S1530, when a user command for setting the first point and the second point is input on the UI screen, information corresponding to a preset route connecting the first point and the second point is transmitted to the server. In addition, the server may receive and display a plurality of contents related to the predetermined path from the server, or receive and display a composite content obtained by synthesizing a plurality of contents related to the predetermined path from the server.

In addition, the control method, when a user command to set a preset region is input, transmits information related to the region to the server, receives at least one location information for at least one source device located in the preset region from the server at least The method may further include displaying a GUI indicating a location for one source device.

In this case, at least one source device may receive permission information on a content sharing level that can be shared with the user terminal device from the server and display it in an area adjacent to the GUI indicating the location of the source device.

In addition, in steps S1510, S1520, and S1530, when a user command for specifying a preset location is input on a UI screen providing a virtual experience service, information related to the location is transmitted to the server, and the virtual image captured at the location from the server is transmitted. The experience content can be received and displayed. Here, the virtual experience service may include at least one of a virtual road experience service, a virtual tour service, and a virtual event experience service.

As described above, according to various embodiments of the present invention, it is possible to experience a sense of realism at a remote location by using information obtained through another person's wearable device. In addition, it is possible to provide a service that directly or indirectly pays a user of a wearable device located at a remote location.

Meanwhile, methods according to various embodiments of the present disclosure may be programmed and stored in various storage media. Accordingly, methods according to various embodiments described above may be implemented in various types of electronic devices that execute a storage medium.

Specifically, according to an embodiment of the present invention, receiving a content transmission request for a predetermined location from a user terminal device, and a source device at a predetermined location based on location information received from at least one source device A non-transitory computer readable medium may be provided in which a program that sequentially performs steps of receiving the photographed content in real time and providing it in a streaming form to a user terminal device is provided.

The non-transitory readable medium means a medium that stores data semi-permanently and that can be read by a device, rather than a medium that stores data for a short time, such as registers, caches, and memory. Specifically, the various applications or programs described above may be stored and provided in a non-transitory readable medium such as a CD, DVD, hard disk, Blu-ray disk, USB, memory card, ROM, and the like.

In addition, although the preferred embodiments of the present invention have been illustrated and described above, the present invention is not limited to the specific embodiments described above, and the technical field to which the present invention pertains without departing from the gist of the present invention claimed in the claims. In addition, various modifications can be made by those skilled in the art, and these modifications should not be individually understood from the technical idea or prospect of the present invention.

100: user terminal device 110: display
120: communication unit 130: processor
200: user terminal device 210: communication unit
220: processor

Claims (21)

In the server that provides the content,
Receiving location information from a plurality of devices,
A content transmission request including a left-eye image and a right-eye image for a plurality of locations is received from the first device,
A communication unit configured to receive content photographed from second and third devices included in the plurality of devices corresponding to the plurality of positions based on the received position information; And
And a processor configured to synthesize the received content and control the communication unit to transmit the synthesized content to the first device. According to claim 1,
The processor,
The content corresponding to the permission information among the content received from the plurality of devices is selected and transmitted to the first device based on the permission information that sets the content sharing level that the plurality of devices can share with the first device. Server characterized in that. According to claim 1,
The processor,
A server characterized in that the content received from the plurality of devices is processed according to a preset condition set in the first device and transmitted to the first device. According to claim 3,
The processor,
When the content received from the plurality of devices is 3D content, the left-eye image and the right-eye image included in the 3D content are respectively projected into a virtual three-dimensional space to generate a left-eye stereoscopic space image and a right-eye stereoscopic space image, and the When information corresponding to the motion state of the first device is received from the first device, a viewpoint region corresponding to the motion state of the first device is extracted from the generated left-eye and right-eye stereoscopic images and transmitted to the first device Server characterized in that. According to claim 1,
The synthesized content,
A server in which the content photographed from the second device and the third device in the plurality of positions are sequentially connected content. In the first device,
display;
Communication department; And
When a command for specifying a plurality of points is input, a content transmission request for a plurality of positions corresponding to the plurality of points is transmitted to the server through the communication unit, and the content captured at the plurality of positions is synthesized from the server. Includes a processor for controlling the display to receive and display the composite content; includes,
The photographed content is a first device that is content including a left-eye image and a right-eye image. The method of claim 6,
The processor,
A first device, characterized in that a plurality of devices that provide the photographed content receive only the content selected by the server based on permission information that sets a content sharing level that can be shared with the first device. The method of claim 6,
The processor,
When a command for specifying the plurality of points is input, information corresponding to a preset route connecting the plurality of points is transmitted to the server, and a plurality of contents related to the preset route is received and displayed from the server. The first device, characterized in that for controlling the display. The method of claim 6,
The processor,
When a command for setting a region is input, information related to the region is transmitted to the server, and a GUI indicating a location for the plurality of devices is received by receiving location information for a plurality of devices located in the area from the server A first device comprising controlling the display to display. The method of claim 9,
The processor,
And controlling the display to receive the authority information on a content sharing level that can be shared by the plurality of devices from the server and display the area adjacent to a GUI indicating the location of the plurality of devices. First device. The method of claim 10,
The processor,
When a command for specifying the location is input on a UI screen providing a virtual experience service, information related to the location is transmitted to the server, and the display is displayed to receive and display virtual experience content photographed at the location from the server. A first device characterized in that it is controlled. The method of claim 11,
The virtual experience service,
A first device comprising at least one of a virtual road experience service, a virtual tour service, or a virtual event experience service. The method of claim 6,
The composite content,
A first device in which content photographed from a plurality of devices corresponding to the plurality of positions is sequentially connected content. In the control method of the server that provides the content,
Receiving location information from a plurality of devices;
Receiving a content transmission request including a left-eye image and a right-eye image for a plurality of locations from a first device; And
Based on the received location information, content captured from second and third devices included in the plurality of devices corresponding to the plurality of positions is received, the received content is synthesized, and the synthesized content is And transmitting to the first device. The method of claim 14,
The step of transmitting to the first device,
The content corresponding to the permission information among the content received from the plurality of devices is selected and transmitted to the first device based on the permission information that sets the content sharing level that the plurality of devices can share with the first device. Control method characterized in that. The method of claim 14,
The step of transmitting to the first device,
Control method characterized in that the content received from the plurality of devices is processed according to a preset condition set in the first device and transmitted to the first device. The method of claim 14,
The synthesized content,
The control method, wherein the content photographed from the second device and the third device in the plurality of positions are sequentially connected content. In the control method of the first device,
When a command for specifying a plurality of points is input, transmitting a content transmission request for a plurality of locations corresponding to the plurality of points to a server; And
It includes the step of receiving and displaying the synthesized content of the content captured at the plurality of locations from the server;
The photographed content is a control method that is content including a left-eye image and a right-eye image. The method of claim 18,
The displaying step,
A control method characterized in that a plurality of devices that provide the photographed content receive and display only the content selected by the server based on permission information that sets a content sharing level that can be shared with the first device. The method of claim 18,
The step of transmitting to the server,
When a command for specifying the plurality of points is input, information corresponding to a preset route connecting the plurality of points is transmitted to the server,
The displaying step,
And receiving and displaying a plurality of contents related to the preset route from the server. The method of claim 18,
The composite content,
A control method in which contents photographed from a plurality of devices corresponding to the plurality of locations are sequentially connected contents.

Images