KR102358997B1 - The service platform for multi-user supporting extended reality experience - Google Patents

Abstract

The present invention provides a service platform for providing a multi-party extended reality experience in which a plurality of users can interact with virtual objects and users appearing in the real world as a background,
An extended reality world that blends the real world and the VR world by combining a 3D recognition stereo camera for recognizing an indoor background and objects decorated according to a real background or scenario in the real world, and an HMD device for displaying 360-degree 3D XR content Content that implements a haptic-based user controller and a high-performance backpack PC for 3D content synthesis and synchronization processing as experience equipment, and provides audiovisual information about the service operation server and the on-site background screen XR story object Configuration server, a content link server for synchronizing information on interaction events between players who are currently playing, as a content server, multi-party simultaneous experience, spectator experience, real-time online relay, and social interaction Realize the reality (XR) experience service.

Description

A service platform for multilateral extended reality experience {THE SERVICE PLATFORM FOR MULTI-USER SUPPORTING EXTENDED REALITY EXPERIENCE}

The present invention relates to an extended reality (XR, eXtended Reality) technology that intends to expand and amplify the sense of reality beyond time and space by mixing the real world space with the virtual reality (VR) world and giving a high sense of reality, It relates to a service platform for a multi-party extended reality experience in which multiple users can interact with virtual objects appearing in the real world as a background.

Virtual reality (VR) uses special devices and technologies to block the real world to users and control the five senses of sight, hearing, smell, taste, and touch to show virtual images in cyberspace separated from reality. , Augmented Reality (AR) is a technology that displays additional virtual 3D images or media information over the real world by estimating positions and postures based on images input through sensors or cameras built into smart devices.

VR technically puts a video or photo taken with a 360-degree camera in a virtual space as a background, and wears a headset-type (HMD, Head Mounted Display) terminal covering the entire eye to make it feel as if the user is in a virtual space in front of them. provide an experience On the other hand, AR does not experience a virtual space like VR, but virtual content or objects are overlaid on top of the real world seen through a transparent display such as a mobile phone or smart glass. A service that directly displays navigation information on a display is a representative example of augmented reality.

XR (eXtended Reality) is an ultra-realistic service technology that encompasses both Virtual Reality (VR) and Augmented Reality (AR) technologies. can XR technology does not have a standardized implementation method. It projects a real background by attaching a camera to the existing VR technology. It is also called Mixed Reality (MR) in terms of mixing reality and virtuality by overlaying 3D objects of Also called XR technology can provide a variety of new types of experience services that break free from the limitations of reality through convergence and fusion with various fields as well as games.

Korean Patent Publication No. 10-2020-0009648 of Prior Patent Document 1 (HMD communication system and method using optical camera communication) and Korean Patent Publication No. 10-2020-0033064 of Prior Patent Document 2 (MR graphic) Display and graphic content processing technology for MR technology can be referenced through the platform service system for providing support content and its driving method), and it is difficult to find an XR service implementation case in Korea.

Typically, VR uses a headset-type (HMD) terminal that covers the entire eye as shown in FIG. 1, and AR uses a terminal such as a smartphone, tablet, or smart glasses. On the other hand, XR expresses freely expanded reality by pursuing individual or mixed use of VR or AR technology. High-performance computing power and graphic processing power are required to display objects in real time, and 5G (5G) mobile communication is required to support motion blur (afterimage generated when the screen is moved) and real-time interaction without delay in a wireless environment. As such, ultra-low latency and large-capacity data transmission technology is also a prerequisite.

The HoloLens developed by Microsoft (MS) is a glasses-type device, but it can be viewed as a form of XR in that it displays an optimized 3D hologram by identifying real space and object information, but it is still too expensive to apply to popular services. When XR technology evolves, it is usually transparent glasses, but when AR service is required, information is displayed on the glasses, and when VR service is needed, the glasses become opaque and display format that displays information through the entire field of view is also proposed. It is still in the concept establishment stage, and there are limits to its immediate use.

Accordingly, the inventors of the present invention came up with a service platform for a multi-party extended reality experience in which multiple users can interact with virtual objects and users appearing in the real world as a background.

1. Republic of Korea Patent Publication No. 10-2020-0009648 (HMD communication system and method using optical camera communication) 2. Republic of Korea Patent Publication No. 10-2020-0033064 (Platform service system for providing MR graphic support content and its driving method)

In the present invention, in order to solve the problems of the prior art as described above, a realistic experience equipment and service platform configuration method for a multi-party extended reality experience in which a plurality of users can interact between users and virtual objects appearing in the real world as a background It aims to provide an XR service platform through the data processing process between the content server and the user.

In configuring a service platform for providing a multi-party extended reality (XR, eXtended Reality) experience in which a plurality of users can interact between users and virtual objects appearing in the real world as a background,

Experiencer equipment to support two or more users to experience the extended reality together, a content server to configure the background of the real world and virtual object information, configure a story, and recognize the location information of the experiencer in real time It is characterized in that it includes equipment for the experience space for the purpose, and equipment for spectators to passively watch the play of the field experiencer.

The experience equipment combines a 3D recognition stereo camera for recognizing an indoor background and objects decorated according to an actual background or scenario of the real world and an HMD display (HMD - Head Mounted Display) device for displaying 360-degree 3D XR content. Realizing a world in an extended reality that mixes the world and the VR world,

In addition, it is characterized by including a haptic-based user controller to provide hyper-realistic five senses to the user and a backpack PC for real-time large-capacity 3D content synthesis and synchronization processing.

The content server includes a service operation server that provides operation and control functions for the overall XR experience service, a content configuration server that provides audiovisual screen information on the floor and walls of the experience space, location information of current players and real and virtual It is characterized in that it is composed of a content interworking server that provides content configuration information for an interaction event between objects,

The experience space equipment has four base station sensors arranged in a rectangle in the real space in order to recognize and track the location information of players who are currently playing in real time, and two projectors are displayed on at least one side of the wall and floor of the experience space. It is characterized in that it consists of a number of projectors that realistically compose the experience background by installing it at a certain angle so as to display the background in the story.

In the present invention, non-experienced users can watch in real time how experienced people wearing HMD devices with 3D-recognized stereo cameras are playing in the extended reality world, and XR play is relayed online in real time or shared through an external display. Social interaction with multiple viewers is possible.

Through the present invention, the real space is projected (PATH-THROUGH), not the space of the virtual world, through the 3D-recognized stereo camera attached to the HMD, and the virtual object is realistically expressed in the real space with the high-resolution stereo video capture processing method. and maximizing immersion.

The present invention recognizes all user actions such as jumping, crouching, and avoiding virtual objects only with sensors and user controllers, and matches virtual objects in three dimensions to fit the spatial structure and reflects them on the screen (INSIDE-OUT TRACKING). there is.

In addition, according to the present invention, since real lights and objects in the real world affect the virtual 3D object, the user can more realistically interact with the virtual object. (DMYNAMIC SPACE MAPPING)

On the other hand, the present invention allows the currently playing experience service to be performed while interacting with other experiencers, to allow on-site spectators to share in real time at the same time, and to provide online social interaction (MULTI-USER XR EXPERIENCE). be able to

The service platform for multilateral XR experience provided by the present invention can provide a variety of new types of experience services that go beyond the limitations of reality in the production of convergence and complex application contents with various fields as well as games, thereby creating an innovative trend in the video media industry. can be formed

1 is a configuration diagram of a user equipment for an extended reality service according to the present invention;
2 is a configuration diagram of an experience space for an extended reality service according to an embodiment of the present invention;
3 is a sensor arrangement diagram for recognizing the location of an experienced person according to an embodiment of the present invention;
4 is a configuration diagram of an indoor background screen using a projector according to an embodiment of the present invention;
5 is a configuration diagram of a device platform for an extended reality service according to the present invention;
6 is a data processing process diagram for an extended reality service according to the present invention

Since the present invention can have various changes and can have various embodiments, specific embodiments are illustrated in the drawings and will be described with reference to the detailed description. However, this is not intended to limit the present invention to specific embodiments, and it should be understood to include all modifications, equivalents, and substitutes included in the spirit and scope of the present invention. In describing each figure, like reference numerals have been used for like components. In describing the present invention, if it is determined that a detailed description of a related known technology may obscure the gist of the present invention, the detailed description thereof will be omitted.

In the present invention, a 3D recognition stereo camera for recognizing real real world space and objects and an HMD display (Head Mounted Display) device for displaying 360-degree 3D XR content are combined as shown in FIG. 1 to combine the real world and the VR world In order to implement the world in the extended reality mixed with It is characterized in that it includes a backpack PC directly connected to a 3D recognition stereo camera and an HMD display device by wire as the experience equipment.

The present invention provides a service platform for providing a multi-party extended reality (XR) experience in which a plurality of users can interact between virtual objects and users appearing in the real world,

5, the experiencer equipment 100 for supporting two or more users to simultaneously experience the extended reality; a content server 400 for configuring backgrounds and object information of the real and virtual worlds; an experience space equipment 300 for composing a story and recognizing the location information of the experiencer in real time; It is characterized in that it is composed of; the spectator equipment 200 to passively watch the play of the field experiencer.

The experience equipment includes a 3D recognition stereo camera 110 for recognizing an indoor background and objects decorated according to an actual background or scenario of the real world; HMD device 120 for displaying a 360-degree 3D XR content so that the user can be immersed in the experience story by displaying virtual objects and NPC (NON-PLAYER CHARACTER) objects; a haptic-based user controller 130 that provides interaction with virtual objects, NPCs, and other experiencers, and provides hyper-realistic five senses to the experiencers when an action event occurs; It consists of a high-performance backpack PC 140 for providing realistic play with expanded space and time to the experiencer through image processing of real-world images and virtual objects input through a 3D-recognized stereo camera, and all interaction results.

The content server 400 includes: a service operation server 410 that provides operation and control functions for the overall service, such as start and end of the experience service, stop and resume, user registration and selection of the experience service; a content configuration server 420 that provides visual information about the scene background screen and NPC objects; a content interworking server 430 for receiving information about the location of the players currently playing, and an interaction event between real and virtual objects, from the experiencer's equipment, synthesizing it with content configuration information, and providing it to the viewer;

In addition, the experience space equipment 300 includes four squares arranged in a real space so as to recognize in real time the location information of those who are currently playing in the experience space decorated according to the real world or scenario and gesture information using the user controller. base station position sensor 310; When composing the indoor background of the experience space, install it at a certain angle so that two projectors can be displayed on at least one side of the wall and the floor, and display the background in the story. consists of ;

The spectator equipment 200 is characterized in that it is composed of a flat-panel display device 210 that allows the experiencers to watch in real time how they play in the extended reality world.

As shown in FIG. 6, the present invention provides a data processing process configuration of a service platform for multi-party XR experience.

The backpack PC 140 of the equipment for the experiencer implements the world in the extended reality by synthesizing the virtual object according to the scenario with the real world image input through the 3D recognition stereo camera worn by the experiencer (141);

When the base station sensor scans the user's location information and the gesture information using the user controller, synchronizing the corresponding content and transmitting the information to the content interworking server so that it can be shared with other users and spectators (142);

displaying the synchronized 360-degree 3D XR content on the HMD of the experiencer in real time so that the user can interact with the virtual object through the user controller (143);

The user's interaction information is transmitted back to the content interworking server to be shared with other experienced users and spectators (step 144); characterized in that it includes a data processing process.

The content configuration server 420 includes a virtual object information module 421 for VR story development; a background screen and environment object information module 422 for configuring a background through an indoor wall when the real world is an indoor environment structured according to a scenario; NPC physical object information module 423 that does not play on the content but appears for story progression; and a synchronization information module 424 for mutual interworking between the configuration modules.

The content interworking server 430 stores the location, sight, hearing and timing information of the actual experiencers received from each experiencer, and various event information that occurs in the process of interacting with virtual objects in the extended reality with the experience information of other experiencers. a multi-party content synchronization step (431) of synchronizing and allowing multiple users to participate in and play the story at the same time in one space in the extended reality world;

By linking the synchronized play information in the extended reality world with the background and story objects of the content configuration server, the delay of 360-degree XR content is minimized, and viewers can share the way they play in the extended reality world at the same time. Multi-party content interworking step (432);

The data processing process of the social interaction support step (433) of allowing the synchronized play information in the extended reality to be shared online through the Internet or an external projector environment, and to react through SNS comments or multi-chats in real time; It is characterized in that it includes.

The service operation server 410 is an experience service control step 411 for performing operation and control of the overall service, such as starting and ending, stopping and resuming the experience service, registering the experiencer and selecting the experience service; and

The data processing process of the user UI/UX control step 412 for performing control functions such as start and end, stop and resume, and change settings on the experience story of the experiencer through the user controller individually used by the users in the experience; It is characterized in that it includes.

As shown in FIG. 2, in the configuration of the experiencer device and the experience space for the multi-party extended reality (XR) experience, the present invention,

A built-in gyroscope is built-in to scan the location and movement information such as crouching and jumping in an indoor experience space that is crumpled according to the physical real world or scenario, enabling 360-degree vertical, left and right 3D space recognition, and A user controller 130 that delivers a real-time haptic effect according to an action event in order to provide a hyper-realistic five senses;

As shown in Fig. 3, including at least four base stations 310 arranged in a rectangle in the real background space so that the user's location in the real world and gestures using the user controller can be recognized in real time,

All actions of the user, such as jumping, crouching, and avoiding virtual objects, are recognized only by sensors and user controllers, and virtual objects are matched in three dimensions to fit the spatial structure and reflected on the screen (INSIDE-OUT TRACKING). characterized.

In composing the real world, the experience space can physically implement the real world by installing a structure in an outdoor or indoor space, or project a background screen on an indoor wall according to a scenario to compose the real world in the story by screen composition. When configuring the background screen, as shown in FIG. 4, the projected background screen is installed at a certain angle so that two projectors 320 can be displayed on at least one side of the wall and floor of the indoor experience space. It is characterized in that it maximizes the sense of realism by configuring (321 to 324) to form a single space in succession in the left and right and up and down directions.

In the present invention, in the configuration of a service platform for multi-party extended reality experience,

It is characterized by enabling multi-party play that allows not only the individual experience service but also at least two or more experiencers to simultaneously enter the world in one extended reality, interact with virtual objects, and collaborate.

To this end, there is a content interworking server 430 that performs a content interworking function,

In the extended reality world, by synchronizing the location, sight, hearing, and timing information of the experiencers in the real world received from each multilateral experiencer, and various event information that occurs in the process of interaction with virtual objects in the extended reality with the experience information of other experiencers a step of synchronizing multi-party content so that multiple users can participate in and play the story at the same time in one space (431);

By linking the synchronized play information in the extended reality world with the background and story objects of the content configuration server, the delay of 360-degree XR content is minimized, and viewers can share the story at the same time as they play in the extended reality world. a multi-party content interworking step (432);

Including a step (433) of supporting social interaction so that the synchronized play information in the extended reality can be shared online through the Internet or an external projector environment, and reacted through SNS comments or multi-chat in real time; is a feature on the service platform.

Preferably, through the service platform for the multi-party extended reality experience presented in the present invention,

Maximizes realism and immersion by projecting the real real space instead of the virtual world space according to the eye of the experiencer (PATH-THROUGH) and expressing the virtual object in the real space using the high-resolution stereo video capture processing method,

All user actions such as jumping, crouching, and avoiding virtual objects are recognized only by sensors and user controllers, and virtual objects are matched in three dimensions to fit the spatial structure and reflected on the screen (INSIDE-OUT TRACKING) (5c),

As real lights and objects in the real world affect the virtual 3D object, the experiencer can interact with the virtual object more realistically.(DYNAMIC SPACE MAPPING)(5d)

An XR experience that allows the currently playing experience service to be performed while interacting with other experiencers at the same time, allowing on-site viewers to share in real time at the same time, and providing online social interaction (MULTI-USER XR EXPERIENCE) (5e) service can be implemented.

100: Experiencer Equipment
110: 3D recognition stereo camera
120: HMD device
130: user controller
140: backpack PC
200: spectator equipment
210: flat display device
220: HMD device
300: Experience space equipment
310: base station sensor
320: background screen composition projector
400: content server
410: service operation server
420: content configuration server
430: content interworking server

Claims (6)

In configuring a service platform for providing a multi-party extended reality (XR, eXtended Reality) experience in which multiple users can interact between virtual objects and users appearing in the real world as a background,
Experiencer equipment 100 for supporting two or more users to experience extended reality together;
Content server 400 for configuring the background and object information of the real and virtual world,
an experience space equipment 300 for composing a story and recognizing the location information of the experiencer in real time; and
Spectator equipment 200 for passively viewing the play of field experiencers;

The experience equipment 100,
3D recognition stereo camera 110 for recognizing indoor backgrounds and objects decorated according to real backgrounds or scenarios in the real world;
HMD display device 120 for displaying a 360-degree 3D XR content so that the user can be immersed in the experience story by displaying the virtual object and the NPC (NON-PLAYER CHARACTER) object;
a haptic-based user controller 130 that provides interaction with virtual objects, NPCs, and other experiencers, and provides hyper-realistic five senses to the experiencers when an action event occurs; and
A high-performance backpack PC 140 for providing realistic play with expanded space and time to the experiencer through image processing of real world images and virtual objects input through a 3D-recognized stereo camera, and all interaction results; Consists of;

The spectator equipment 200,
It consists of a flat-panel display device 210 or HMD device 220;

The experience space equipment 300,
Four base station position sensors 310 arranged in a rectangle in the real space so as to recognize in real time the location information of those who are currently playing and gesture information using the user controller on the indoor background screen decorated according to the real world or scenario; and
When configuring the indoor background of the experience space, two projectors are installed at a certain angle so that at least one side of the wall and the floor can be displayed, and a plurality of projects 320 that realistically configure the experience background by displaying the background in the story; is composed of,

The content server 400,
a service operation server 410 that provides operation and control functions for the overall service, such as start and end of the experience service, stop and resume, registration of the experiencer and selection of the experience service;
a content configuration server 420 that provides visual information about the scene background screen and NPC objects; and
A content interworking server 430 for receiving information about the location of the players currently playing, the interaction event between real and virtual objects from the experiencer's equipment, combining it with the content configuration information, and providing it to the viewer;

The content configuration server 420 is
Virtual object information module 421 for VR story development;
a background screen and environment object information module 422 for configuring a background through an indoor wall when the real world is an indoor environment structured according to a scenario;
NPC physical object information module 423 that does not play on the content but appears for story progression; and
Including; a synchronization information module 424 for mutual interworking between the respective modules;

The service operation server 410 is
Controlling the experience service for operation and control of the overall service, such as starting and ending, stopping and resuming the experience service, registering the experiencer and selecting the experience service (411); and
Data processing of the user UI/UX control step (412) for performing control functions such as start and end, stop and resume, and setting change on the experience story of the individual experiencer through the user controller individually used by the users in the experience; process, including

The content interworking server 430 is
A space in the extended reality world by synchronizing the location, visual, auditory, and timing information of the actual experiencers received from each experiencer and various event information that occurs during the interaction with virtual objects in the extended reality with the experience information of other experiencers a multi-party content synchronization step (431) in which a plurality of experiencers can participate in and play the story at the same time;
By linking the synchronized play information in the extended reality world with the background and story objects of the content configuration server, the delay of 360-degree XR content is minimized, and viewers can share the way they play in the extended reality world at the same time. Multi-party content interworking step (432); and
The data processing process of the social interaction support step (433) of allowing the synchronized play information in the expanded reality to be shared online through the Internet or an external projector environment, and to react through SNS comments or multi-chat in real time; A multi-party extended reality experience service platform that includes
delete According to claim 1,
The backpack PC 140 among the experience equipment
Realizing the world in the extended reality by synthesizing the virtual object according to the scenario with the real world image input through the 3D recognition stereo camera worn by the experiencer (141);
When the base station sensor scans the user's location information and gesture information using the user controller,
synchronizing the corresponding content and transmitting the information to the content interworking server so that it can be shared with other users and spectators (142);
displaying the synchronized 360-degree 3D XR content on the HMD of the user in real time so that the user can interact with the virtual object through the user controller (143);
A multi-party extended reality experience service platform, comprising the data processing process of step (144), in which the user's interaction information is transmitted back to the content linkage server to be shared with other experienced users and spectators.
delete According to claim 1,
The experience space for the multilateral extended reality experience includes a 360-degree up, down, left and right 3D space recognition function to scan the location and movement information such as crouching, jumping, etc. a user controller 130;
including at least four base station position sensors 310 arranged in a rectangle in the real background space to recognize the user's location in the real world and gestures using the user controller in real time;
Multi-party expansion, characterized in that all actions of the user, such as jumping, crouching, and avoiding virtual objects, are recognized only by the position sensor and user controller, and the virtual objects are matched in three dimensions to match the spatial structure and reflected on the screen Reality experience service platform.
delete

Images