WO2018131789A1

WO2018131789A1 - Home social robot system for recognizing and sharing everyday activity information by analyzing various sensor data including life noise by using synthetic sensor and situation recognizer

Info

Publication number: WO2018131789A1
Application number: PCT/KR2017/013401
Authority: WO
Inventors: 김진우; 정광민; 김병관; 심혜린; 이해성; 김태윤; 성지현; 김아람; 김혜미
Original assignee: 주식회사 하이
Priority date: 2017-01-12
Filing date: 2017-11-23
Publication date: 2018-07-19

Abstract

Disclosed in the present disclosure is a synthetic sensor-based home social robot management system for sharing everyday activity information, wherein the system recognizes people's everyday life activities and takes appropriate measures, without infringing on privacy. In the present disclosure, an acoustic preprocessing technique to divide and process sounds generated in everyday life in an optimal state for acoustic recognition, a technique to effectively remove additive white noise that is ubiquitous in life activities, a technique to extract feature vectors for effectively recognizing life activity sounds in preprocessed signals, a technique to extract kinds of life noise by applying machine learning to the extracted feature vectors, and the like are used. According to the present disclosure, particularly when applied to an environment where the elderly live alone, it is possible to prevent mistakes such as going out without turning off a gas range while cooking food, thereby enhancing safety of the elderly and minimizing an economic loss.

Description

Home social robot system that recognizes and shares daily activity information by analyzing various sensor data including living noise using synthetic sensor and situation recognizer

The present disclosure relates to a home social robot management system for sharing living activity information. Specifically, the home social robot that acquires and reads the living noise of a household of a single household through a home social robot installed by a user and shares it with other users A robot management system.

Home social robots are attracting attention as one of the tools to solve the various complex social problems in the changing modern society, such as population aging, increasing single-person households, and deepening individualism. Home social robots are emotion-oriented robots that interact with people, unlike traditional robots that simply replace physical tasks that are difficult for people to do, and help people not feel isolated.

One-person households are found to be tired of their relationships but at the same time feel lonely. This tendency of social relations is the cause of social isolation. Looking at the problems faced by single-person households, these households are found to be tired of their relationships. Recently, due to the stress of human relations, more people prefer to enjoy their own time rather than trying to make good relationships with others, and the new word “Gwantaegi”, which combines “relationship” and “taetaegi,” has emerged. The word refers to the appearance of feeling irritated and skeptical of unnecessary and exhausting relationships without feeling the need for new relationships. Experts diagnosed in this age of taekwondo as a product of modern society, which is used to living alone rather than training to connect with people. However, just because they feel tired of their relationships and voluntarily become single-person households does not mean they are not lonely. In a previous study of single-person households, loneliness was a major part of the challenges of living alone.

Home social robots have emerged as a way to solve such problems in single households. In the case of the elderly in particular, the need for a home social robot capable of practically guiding life in the elderly's life by providing feedback to the elderly appropriately while monitoring the living pattern of the elderly from the outside is emerging.

[Preceding technical literature]

1. Korean Patent Publication No. 10-2008-0011422

2. Korean Patent Publication No. 10-2017-7003990

3. Korean Patent Publication No. 10-2009-0084267

4. United States Patent No. 8909370

5. US Patent No. 9288594

6. Chinese Patent Publication No. 105818165

The present disclosure aims to promote companionship to solve the social isolation of single households, and devise a method of sharing activity information. Among the activities to be shared, the most appropriate activity for the living pattern of a single household was considered as home activity, but since home is the most private space, there may be concern about privacy infringement in sharing information of home activity. In the present disclosure, we have focused on sharing the activity information in the home with 'Social Robot' and 'Hearing' as a way to reduce the concern about privacy infringement, and the physical clue of the social clues is' the physical implementation of the agent 'and the voice clue' We tried to derive from the research study how to share activity information that can reduce social involvement with friends, companionship with agents, and frequent mistakes. As a result of the experiment, when the agent is physically implemented and the friend's activity information is transmitted through the robot's voice, the social connection to the friend, companionship to the agent, and companionship to the friend are the highest. Based on the experimental research results, a home social robot was produced to audibly inform the home activity information.

In the present disclosure, in order to manufacture a home social robot, in particular, a core recognition module of the IoT system is a "synthetic sensor: a distance measuring sensor, a temperature and humidity measuring sensor, an illuminance measuring sensor, an acoustic measuring sensor, a grid-eye sensor, a gyro acceleration sensor. It has adopted an artificial intelligence module that recognizes and analyzes user's activity through the combination of existing sensors, etc.), and recognizes the user's activity pattern and the sound of opening and closing of home appliances. The situation can be inferred.

Synthetic sensors, one of the core research areas of Human Computer Interaction (HCI) and Human Robot Interaction (HRI), go beyond the limits of existing individual sensors and combine various types of sensors to pursue effective and economical sensing.

Instead of providing on-off one-dimensional information by a single sensor, various measurement values inputted through the synthesized sensor are stored by time zone, and the current user situation is analyzed in a high-dimensional manner based on the entire set of data. In providing comprehensive activity analysis information for the present disclosure, the present disclosure is unique and distinguishes it from existing sensor network systems.

The problem to be solved by the present disclosure is not limited to the above-mentioned problem, and other problems not mentioned will be clearly understood by those skilled in the art from the following description.

In one aspect of the present disclosure, a home social robot management system for living activity sharing, comprising a plurality of home social robots and a management server communicatively connected to the home social robot, the home social robot comprising living noise and living environment information A synthesis sensor unit for receiving a signal, a speaker for outputting the state of the home social robot as a sound, a display for outputting the state of the home social robot as an image, a communication unit for communicating with a management server of the home social robot through a network; And a control unit connected to the synthesis sensor unit, the speaker, the display, and the communication unit, wherein the control unit transmits the living noise and living environment information obtained from the synthesis sensor unit to the management server. Based on the living noise and the living environment information, the situation information is determined and transmitted to the home social robot. Constructed, there is provided a home social robot control system.

In one embodiment, the synthesis sensor unit may include two or more of the distance measuring unit, gyro accelerating unit, temperature and humidity measuring unit, illuminance measuring unit, grid-eye unit, and sound measuring unit.

In one embodiment, the home social robot may include a hub robot and one or more edge robots communicatively connected to the hub robot.

In one embodiment, the distance measuring unit may include at least one of an infrared measuring device and an ultrasonic measuring device.

In one embodiment, the sound measurement unit includes a sound sensor and a sound recognizer, and the sound sensor may be configured to notify the sound recognizer to operate the sound recognizer only when the energy level of the sound is equal to or greater than a predetermined threshold. .

In one embodiment, the acoustic measurement unit may be configured to divide into different time magnitudes for the noise canceled input signal.

In one embodiment, the acoustic measurement unit to remove the noise primarily by the wavelet transform (Wavelet transform) for the input signal related to the living noise and living environment information, the noise by applying a median filter after the inverse wavelet transform Can be configured to remove secondary.

In one embodiment, the management server may be configured to first remove noise by wavelet transforming the input signal, and secondly remove noise by applying a median filter after the inverse wavelet transform.

In an embodiment, the management server may be configured to extract a feature vector by applying a wavelet transform to the input signal from which the noise is removed, and to classify a sound type based on the extracted feature vector.

In one embodiment, the situation information may include opening and closing the front door, opening and closing the window, turning on the tap, turning on the stove, turning on the microwave, opening the refrigerator, operating the vacuum cleaner, turning the lights on and off in the house, turning the lights on and off, and moving the people. It may include at least one.

In one embodiment, the management server may be configured to transmit the situation information to a terminal of a second user or a second home social robot to share life activity information with the user of the home social robot.

In one embodiment, the management server may instruct at least one of the facial expression and the voice of the robot to at least one of the home social robot and the second home social robot based on the situation information.

In an embodiment, when the management server determines that a dangerous situation occurs based on the situation information, the terminal of the second user or the second home social that share the living activity information with the user of the home social robot And send an emergency notification message to the robot.

The living noise may include at least one of a washing machine sound, a cleaner sound, a microwave oven sound, a gas stove sound, a keyboard sound, a window opening sound, a water sound, a front door sound, a refrigerator door sound, a visit sound, and a footstep sound. It may include.

In another aspect of the present disclosure, in the home social robot management system for living activity sharing, a method of determining a sound type, the method comprising: receiving living noise and living environment information as an input signal, and removing noise from the input signal And obtaining a feature vector by performing wavelet transform on the signal from which the noise has been removed, and determining the type of sound by applying the feature vector to a machine learning tool. Is provided.

In one embodiment, the removing of the noise may be performed by performing a wavelet transform on the input signal to remove the noise first, and performing the inverse wavelet transform on the signal from which the noise is first removed; And applying a median filter to the inverse wavelet transformed signal to remove noise secondarily.

In an embodiment, the step of obtaining the feature vector may include making the noise canceled signal dyed, and performing a discrete wavelet transform (DTW) on the dyed signal using a multi resolution analysis (MRA) method. And obtaining a second half of the feature vector representing the magnitude of energy for each location section in the time domain, and obtaining the first half of the feature vector representing the magnitude of energy for each bandwidth in the frequency domain. Connecting the first half and the second half of to obtain a final feature vector.

In another aspect of the present disclosure, in a home social robot management system for living activity sharing, comprising a plurality of home social robots and a management server communicatively connected to the home social robot, a method of sharing living noises, comprising: a home social robot Receiving the living noise and living environment information, the step of transmitting the living noise and living environment information to the management server via a network, and determining the status information based on the living noise and living environment information And transmitting the contextual information to at least one of a terminal of a second user or a second home social robot to share living activity information with the home social robot and the user of the home social robot. A noise sharing method is provided.

In one embodiment, the step of receiving the living noise and living environment information comprises the steps of determining whether an input value for the living noise and living environment information is greater than a predetermined threshold value, and if determined to be large, the left and right sound cards Recording at the same time, selecting data of a large energy of the left and right sounds recorded on the sound card as an analysis target, and dividing a sound of the analysis target data into a plurality of different sized data. Can be.

In an embodiment, the method may further include controlling, by the management server, at least one of a facial expression and a voice of the robot to at least one of the home social robot and the second home social robot based on the situation information. Can be.

In an embodiment, when the management server determines that a dangerous situation occurs based on the situation information, a terminal or a second home of a second user who has decided to share living activity information with the user of the home social robot The method may further include transmitting an emergency notification message to the social robot.

According to an embodiment of the present disclosure, a home social robot management system for sharing life activities that recognizes people's daily life activities and takes appropriate measures without violating privacy can be provided. In addition, according to one embodiment of the present disclosure, when the elderly living alone can be applied to an environment where they live alone, it is possible to prevent mistakes such as going out without turning off the gas stove, thereby increasing safety of the elderly and minimizing economic losses. In addition, according to the disclosed embodiment, the social isolation problem may be solved by connecting the user with another user through interaction with a social robot capable of social interaction with a person to expand the social connection of the single household.

In addition, according to an embodiment of the present disclosure, since the present disclosure can easily adjust the degree of information sharing selectively according to a user's situation, privacy balancing is possible.

In addition, according to an embodiment of the present disclosure, by employing a composite sensor, problems such as cost increase and battery / power supply caused by installing a chip for each device are solved.

1 is a diagram illustrating a system environment for controlling a home social robot performed by a management server of a home social robot according to one embodiment of the present disclosure.

2 is a block diagram illustrating a home social robot according to one embodiment of the present disclosure.

3 is a diagram illustrating the internal components of the synthesis sensor unit according to an embodiment of the present disclosure.

4 is a conceptual diagram of removing noise and block phenomenon using spatial correlation according to an embodiment of the present disclosure.

5 is a diagram illustrating a two-stage downsampling of an input signal according to an embodiment of the present disclosure.

6 is a diagram illustrating a decomposition of the input signal in the frequency domain according to an embodiment of the present disclosure.

7 is a flowchart illustrating an operation process of an acoustic measuring unit according to an exemplary embodiment of the present disclosure.

FIG. 8 is a flowchart for describing a procedure of controlling, by a controller, a situation analysis of a robot according to an embodiment of the present disclosure.

Advantages and features of the present disclosure and a method of accomplishing the same will be apparent with reference to the embodiments described below in detail with the accompanying drawings. However, the present disclosure is not limited to the embodiments disclosed below, but may be implemented in various forms, and the present embodiments merely allow the disclosure of the present disclosure to be complete and have ordinary skill in the art to which the present disclosure belongs. It is provided to fully inform the scope of the invention, and the present disclosure is defined only by the scope of the claims.

The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the disclosure. For example, a component expressed in the singular should be understood as a concept including a plurality of components unless the context clearly indicates only the singular. In addition, in the specification of the present disclosure, the terms 'comprise' or 'having' are merely intended to designate that there exists a feature, a number, a step, an operation, a component, a part, or a combination thereof described on the specification. The use of the term does not exclude the possibility of the presence or addition of one or more other features or numbers, steps, operations, components, parts or combinations thereof. In addition, in the embodiments described herein, 'module' or 'unit' may refer to a functional part performing at least one function or operation.

In addition, all terms used herein, including technical or scientific terms, unless defined otherwise, have the same meaning as commonly understood by one of ordinary skill in the art to which this disclosure belongs. Terms such as those defined in the commonly used dictionaries should be interpreted as having meanings consistent with the meanings in the context of the related art, and should be interpreted in ideal or excessively formal meanings unless expressly defined in the specification of the present disclosure. It doesn't work.

Hereinafter, embodiments of the present disclosure will be described in detail with reference to the accompanying drawings. However, in the following description, when there is a risk of unnecessarily obscuring the subject matter of the present disclosure, a detailed description of well-known functions and configurations will be omitted.

Referring to FIG. 1, the system environment 100 for controlling a home social robot according to an embodiment of the present disclosure may include a management server 110. The management server 110 may be communicatively connected to the home social robot 120 and the plurality of second user terminals 130-1, 130-2, 130-n of the user through the network 140. Here, the network 140 may include an internet network, a mobile communication network (for example, a WCDMA network, a GSM network, a CDMA network, etc.), a wireless internet network (for example, a Wibro network, a Wimax network, etc.). Including but not limited to wired / wireless networks.

In one embodiment, the home social robot 120 is configured to act as a hub, and one or more edge robots 150-1, 150-2 ... 150-n that are communicatively connected to the home social robot 120. It may further include. For example, the home social robot 120 serving as a hub is installed in the kitchen or living room, and the edge robots 150-1, 150-2 ... 150-n are installed in the bathroom, the bedroom, the porch, and the like. It is possible to monitor almost anywhere in the living room.

In one embodiment, the management server 110 may be a collocated server, a hosting server, or a cloud server, such as a dedicated server. The management server 110 receives sound from the user's home social robot 120 via the network 140 and stores the sound in a database (not shown) of the management server 110 (eg, washer sound). , The sound of a cleaner, a microwave, a gas stove, a keyboard, a window opening, a water, a front door sound, a refrigerator door sound, a visit sound, a footstep sound) It may be transmitted to the home social robot (not shown) or the terminals 130-1, 130-2, 130-n of the second user. It is possible to adjust the category of the living noise to be shared according to the intimacy of the user and the second user. The living environment information may include information such as temperature, humidity, illuminance of the indoor space.

The management server 110 may be stored in a database in the management server 110 for each category by machine learning about the living noise that may occur in the home. The management server 110 may continuously update the frequency of the living noise through the machine learning. In addition, the management server 110 may use the machine learning to match the sound from the user's home social robot 120 with a certain category of living noise.

Although not shown, the user's terminal (not shown) of the home social robot 120 may communicate with the user through WiFi, Bluetooth, infrared communication, WiMax, and the like. After the user installs an app on the terminal and undergoes user registration (for example, interworking with the user's home social robot 120 and the terminal), the user may control the home social robot 120 through the app.

The user's terminal and the plurality of second user terminals 130-1, 130-2,..., 130-n are wireless devices based on various types of handhelds such as laptops, portable terminals such as note pads, and smart phones. Although it may include a communication device, a computer, a server, and the like, which can communicate with other devices through the network 140, the terminal of the user and the plurality of second user terminals 130-1, 130-2,. The type of is not limited to this.

Although not shown, if the second user has also installed the home social robot, the management server 110 detects the living noise from the user's home social robot 120 by the second user terminals 130-1 and 130-2. 130-n), it is possible to transmit the living noise from the home social robot 120 of the user to the home social robot of the second user. Also in another embodiment, to transmit living noise from the user's home social robot 120 to both the second user terminal 130-1, 130-2 ... 130-n and the second user's home social robot. It is also possible.

The acoustic measurement unit 244 or the management server 110 performs a wavelet transform on the input signal and removes noise first, and removes the noise by applying a median filter after the inverse wavelet transform. It can be configured to. In addition, the management server 110 may be configured to extract a feature vector by applying a wavelet transform to the input signal from which the noise is removed, and to classify a sound type based on the extracted feature vector.

2 is a block diagram illustrating a home social robot according to one embodiment of the present disclosure. The home social robot 120 includes a synthetic sensor unit 240 for receiving living noise and living environment information such as sound, a storage unit 230 for storing the received living noise and living environment information, and living noise and living environment information. A speaker 250 for outputting auditory information about the display, a display 260 for outputting visual information about living noise and living environment information, and a subscriber terminal (not shown) and a home social robot through the network 140. The communication unit 220 communicates with the management server 110 of the control unit 210 is connected to the synthesis sensor unit 240, storage unit 230, speaker 250, display 260, and communication unit 220 ).

The controller 210 may transmit the living noise and the living environment information to the management server 110 through the network 140. In one embodiment, when the control unit 210 transmits the living noise and living environment information received through the synthesis sensor unit 240 to the management server 110, the management server 110 stores the living noise and living environment information database After matching the living noise of the category stored in the home noise (e.g. door opening noise) to the home social robot or terminal of one or more second users to share the living noise with the user send. The operation of matching the living noise and the living environment information with a specific category of the living noise and the living environment information can be performed by using machine learning that is advanced through a large amount of data learning.

In one embodiment, the control unit 210 of the home social robot 120 may be configured to receive the sound information of the second user to share the living noise from the management server 110. The control unit 210 may be configured to receive sound information of a second user who is to share living noise and convert the sound information of the second user as it is, or convert the sound information into a robot sound or a human voice.

Synthetic sensor unit 240 is a distance measuring unit 241, temperature and humidity measuring unit 242, illuminance measuring unit 243, sound measuring unit 244, grid-eye unit 245, gyro acceleration unit 246, etc. It includes (see Figure 3).

The communication unit 220 may be configured to implement a communication protocol that supports transmission and reception of various information under the control of the control unit 210. In this case, the communication protocol may be implemented with appropriate hardware and / or firmware. In some implementations, the communication protocol can include a Transmission Control Protocol / Internet Protocol (TCP / IP) protocol and / or a User Datagram Protocol (UDP) protocol. The communication unit 220 may be implemented with hardware and / or firmware that implements various Radio Access Technologies (RATs) including LTE / LTE-A. In some implementations, the communication unit 220 may be implemented to comply with a wireless communication interface standard such as LTE-Ue. The communication unit 220 may control the management server 110 and the plurality of second user terminals 130-1, 130-2,..., 130-n, and a home social robot (not shown) under the control of the controller 210. Communicate

The storage unit 230 may store frequencies of living noise for each category. The storage unit 230 may also store software / firmware and / or data for the operation of the controller 210, and store data input / output.

The storage unit 230 may include a flash memory type, a hard disk type, a multimedia card micro type, a card type memory (for example, SD or XD memory), Random Access Memory (RAM), Static Random Access Memory (SRAM), Read-Only Memory (ROM), Electrically Erasable Programmable Read-Only Memory (EEPROM), Programmable Read-Only Memory (PROM) Magnetic Memory, Magnetic It may include a storage medium of at least one type of a disk, an optical disk.

Synthesis sensor unit 240, for example, distance measuring unit 241, temperature and humidity measuring unit 242, illuminance measuring unit 243, acoustic measuring unit 244, grid-eye unit 245, gyro acceleration unit And a plurality of sensors, such as 246.

The distance measuring unit 241 may include, for example, an infrared sensor or an ultrasonic sensor. The distance measuring unit 241 periodically measures the distance of a specific object in front of the robot and stores the measured value. This is necessary for the robot to perceive and react to objects appearing in front of it. Infrared and ultrasonic sensors may use either one, or both.

When a distance measurement value is acquired from an infrared / ultrasound sensor in the distance measuring thread of the distance measuring unit 241, and the measured value is transmitted to the management server 110, the management server 110 may execute time and IP (Internet). Protocol), robot universally unique identifier (UUID), owner's ID (identification), and distance value.

The robot 120 stores values in arrays of size N global variables, the stored value being distance. N is a natural number representing the size of the buffer and can be set to 100, 200, or the like. Therefore, old stored values are removed in order as new stored values increase.

The distance measuring unit 241 waits for a predetermined time (for example, 5 seconds), and obtains the distance measurement value from the sensor again.

Temperature and humidity measurement unit 242 periodically measures the temperature and humidity of the environment in which the robot is located, and stores it. This is necessary for the robot to perceive and react to the comfort of its environment.

When measuring the temperature and humidity in the temperature and humidity thread of the temperature and humidity measurement unit 242, and transmits the measured value to the management server 110, the management server 110, time, robot UUID, owner's ID, temperature value and time It stores the robot UUID, owner ID, and humidity value.

The robot 120 stores the temperature value in the N-size global variable arrays, and also stores the humidity value in the N-size global variable arrays. N is a natural number representing the size of the buffer and can be set to 100, 200, or the like. Therefore, old stored values are removed in order as new stored values increase.

The illuminance measuring unit 243 periodically measures the ambient brightness of the environment where the robot is located and stores it. This is necessary for the robot 120 to perceive and react to whether the lighting of the environment in which the robot 120 is located is on / off.

When the temperature and humidity are measured by the illuminance thread of the illuminance measuring unit 243, and the measured value is transmitted to the management server 110, the management server 110 displays time, robot UUID, owner's ID, temperature value, and time. Stores the robot's UUID, owner's ID, and illuminance.

The robot 120 stores the illuminance value in the N variable global variable arrays. N is a natural number representing the size of the buffer and can be set to 100, 200, or the like. Therefore, old stored values are removed in order as new stored values increase.

The acoustic measurement unit 244 periodically measures the ambient noise of the environment in which the robot 120 is located and stores it. The acoustic measurement unit 244 may include a sound sensor 2441 and a sound recognizer 2442. The sound recognizer 2442 can operate only when the sound sensor 2441 notifies the sound recognizer 2442 about sound recognition. That is, the sound recognizer 2442 does not always operate, but the sound sensor first recognizes that the sound is generated, and the sound recognizer 2442 attempts to recognize the result only when the sound energy is greater than or equal to a predetermined threshold. It is preferable. This is to reduce the system load by not always operating the sound recognizer, which requires very large computational complexity.

The gyro acceleration unit 245 may detect vibrations when a person moves and detect movement of a person or falls.

The grid-eye unit 246 may detect the presence / movement of a person through a grid-eye using infrared array sensors. The grid-eye unit 246 only detects the presence / movement of a person, and unlike a camera, the grid-eye unit 246 does not generate an image that can identify an individual, and thus does not cause a concern about a person's privacy.

For example, it is preferable that the home social robot 120 serving as a hub has all the functions of the synthesis sensor unit 240, but the edge robots 150-1, 150-2. Depending on where it is installed, it may include only some of the functions of the synthesis sensor unit 240. For example, in the case of the edge robot installed in the toilet, only the acoustic measurement unit 244 and the gyro acceleration unit 245 may be included. In this case, only a part of the function of the synthesis sensor unit 240 of the edge robot may be configured to be activated, and includes only the distance measuring unit 241, the acoustic measurement unit 244, and the gyro acceleration unit 245. The part 240 may be manufactured. In the case of the synthesis sensor unit 240 can detect the movement of the person in the bathroom (for example, the fall of the person).

For example, in the case of the edge robot installed in the front door, only the distance measuring unit 241, the acoustic measuring unit 244, the grid-eye unit 245, and the gyro acceleration unit 245 may be included. In this case, the synthesis sensor unit 240 may detect the movement of the person (for example, the movement / exit of the person).

The function of the acoustic measurement unit 244 will be described in more detail. In particular, the denoising technique and feature vector extraction methodology employed in one embodiment of the present disclosure will be described.

잡음제거(denoising) 기술Denoising technology

I. Denoising technology that operates on both image (2D signal) and sound (1D signal)

The key to noise cancellation is to distinguish between the boundary components of noise and the boundary components of a signal in the wavelet transform domain. In view of the problem of noise reduction, the algorithm proposed in the present disclosure should have better performance than all existing noise reduction algorithms, and at the same time, have no large computational complexity. In particular, the noise to be removed in the present disclosure aims at an additive white Gaussian as a primary target, but is not limited thereto.

Therefore, the noise to be addressed for the development of the theory in the present disclosure intends to limit its kind only to Additive, which is considered to be a reasonable and appropriate choice when considering the objectives of the present disclosure.

1.1. Interpretation of Spatial Correlation

The basic definitions for the discussion from now on are as follows. For convenience, it is revealed in advance that the theory will be developed for the one-dimensional signal. In this chapter, f (x) refers to the pure original signal without any noise, and n (x) refers to the additive noise to be inserted into this signal. Therefore, I (x) = f (x) + n (x) is the input signal to be used for noise cancellation of the present disclosure. O (x) means the wavelet transformed result of the input signal. On the other hand, a (x), b (x), a '(x), and b' (x) are defined by equations (1-1) and (1-2), respectively.

(1-1)

(1-2)

Wavelet transforms the input signal, and then converts the result values using the following equations (1-3) and (1-4) so that the magnitude distribution of the result values is 0 to 255.

(1-3)

(1-4)

Now define the boundary line as [Definition 1]. In fact, the mathematical model of the boundary line includes a step edge, a roof edge, and a ridge edge. However, in the present disclosure, it is assumed that the boundary line existing in the sound is simply a step edge, and [Definition 1] is introduced.

[Definition 2]

Local Modulus Maximum at x = x ₀

When you have this zero or positive Lipschitz Regularity and you have zero or positive Lipschitz Regularity for larger scales,

Is defined as the boundary component in the wavelet transformed region.

Using [Definition 1], it is easy to derive the following useful [Theorem 1].

[Theorem 1]

[proof]

According to the equations (1-1) and (1-2), a (x) and a '(x) are wavelet transform results of the pure original signal without noise. It is known that the wavelet transform of a signal is equal to the result of smoothing and then differentiating the signal. That is, a (x) and a '(x) mean the result of smoothing and differentiating a given signal. Thus, all boundary components present in a given signal are represented as Local Modulus Maxima at these a (x) and a '(x). On the other hand, in this case, by applying [Definition 1] to derive a set of x that does not belong to the boundary component

Same as And for all x

Is true, then x satisfying this set is

Will be satisfied.

In general, the area occupied by the borderline components within a given sound is very small compared to the areas of the parts that do not correspond to the borderline components. Therefore, the probability that noise is inserted in the region of the boundary components is very low compared to the probability that noise is inserted in the region other than the boundary components. Therefore, the following [Assumption 1] can be introduced naturally. [Assumption 1] means that there is no noise in the boundary components of the wavelet transformed region.

[Home 1]

Next, in the present disclosure, Spatial Correlation D (x) is defined as follows.

[Definition 2]

And this spatial correlation D (x) can be easily analyzed using [Theorem 1] and [Assumption 1]. The results are shown in [Theorem 2].

[Theorem 2]

[proof]

first

Suppose that is In this case, according to [Assumption 1]

This holds true. therefore

Is established. to the next

Suppose that is In this case, according to [Theorem 1]

This holds true. therefore

Is established.

1.2. Noise Reduction and Block Phenomenon based on D (x)

Among the results obtained above, using [Theorem 1] and [Assumption 1], O ₁ (x) and O ₂ (x) can be expressed as [Theorem 3] below.

[Theorem 3]

[proof]

The proof method of [Theorem 3] is the same as the proof method of [Theorem 2].

In general, consider a case in which a boundary line can be detected in a sound in which noise exists through human vision in the wavelet transform region. In this case, the magnitudes of the conversion result values due to the noise will be considerably smaller than the magnitudes of the conversion result values due to the boundary line. Summarizing these ideas, we obtain the following valid [Assumption 2].

[Home 2]

Meanwhile

this

It is known that s decreases in proportion to scale. On the contrary, according to [Definition 1] of the present disclosure, it can be seen that values of Local Modulus Maxima of a boundary line are constant regardless of scale s. therefore,

Is

It can be seen that it implies. Another useful [Theorem 4] derives from these results.

[Theorem 4]

[proof]

According to [Home 2]

Is established, and also this

I mentioned earlier that it implies that Thus, [Theorem 4] above is naturally established.

[Theorem 4] and [Theorem 2] described above provide useful information for distinguishing boundary components from noise and block boundary components in the wavelet transform region. Meanwhile, the method for removing noise components and block boundary components in the wavelet transform region by modifying [Theorem 3] can be represented by the following [Coordination 1].

[Compensation 1]

Therefore, O ₁ (x) and O ₂ (x) are maintained as they are.

So set O ₁ (x) and O ₂ (x) to zero.

However, the process of removing noise and block components described in [Adjustment 1] shows that Spatial Correlation D (x)

This can be implemented by setting the values of O ₁ (x) and O ₂ (x) at the x position smaller than or equal to 0. The reason for this is as follows. First, in [Theorem 2], the value of D (x) depends on whether the position of x belongs to the boundary component or the noise and block component.

Wow

It can be seen that it is divided into. And referring to [Theorem 4]

Is

You can see that it is always smaller. Therefore, the magnitude of the D (x) value

The region of smaller x will contain all noise and block components and some border components. Therefore, as described earlier, Spatial Correlation D (x)

Using the method of setting the O ₁ (x) and O ₂ (x) values at the x position smaller than or equal to 0, it is concluded that noise and block components can be removed. 4 illustrates this method. Of course, this method removes some of the borderline components as shown in the circled portion of FIG. 4. A discussion of this problem will be presented in the future.

Now remove the noise and block components in this way to obtain new O ₁ (x) and O ₂ (x) respectively.

Named as Now this

By inverse wavelet transform, we obtain a new signal from which noise or blocks are removed.

However, in the present disclosure, it was confirmed through various experiments that the above-described method leaves a new kind of noise near the boundary of the signal from which the noise is removed. The main reason is that the above method is used to remove some of the values corresponding to the boundary components together with noise and block phenomena. In addition, [Assumption 1] also contributes to one cause. Because even if the probability is low, noise can be inserted in the boundary of a given sound. In the present disclosure, however, the characteristics of the newly generated noise are analyzed to show that the type is quite similar to the impulsive noise. As a result of estimating the main cause, the following explanation could be introduced. The circled portions in FIG. 4 are boundary components that should not be removed in practice, but indicate boundary components that are inevitably removed by introducing the noise and block phenomenon removing method described above. That is, the circled portion of FIG. 4 represents borderline components but the value falls to zero. And as mentioned many times, from a mathematical point of view, O ₁ (x) and O ₂ (x) are the derivatives of a given signal. Therefore, if we look at the part where the value changed from O ₁ (x), O ₂ (x) to 0 in the original signal, that is, integrating O ₁ (x) and O ₂ (x), there is no change in the slope in that part. It makes sense. Therefore, the circled part of FIG. 4 serves as a kind of discontinuity, that is, a step edge when restored to the original signal through the inverse wavelet. And when modeling these discontinuities as noise, the impact noise will be the closest model.

To solve this problem, although at the edge of the border components,

Borderline component values smaller than D (x) must also be restored by some amount. In FIG. 4, the portion indicated by circles is referred. In one possible way, the algorithm implemented in the present disclosure starts at the point where the boundary components are truncated and starts with O ₁ (x), O ₂ (x) until the slope of the D (x) Profile does not change, i.e. To restore the values. However, the truncated portion of the borderline component corresponding to the right circle of FIG. 4 is almost completely restored when the algorithm is applied, while the truncated portion of the borderline component corresponding to the left circle is not completely restored. In other words, there is a limit to this restoration method. Therefore, in the present disclosure, the median filter is applied to the primary result signal from which the noise and the block phenomenon are removed through the foregoing method. This is because the median filter is the easiest to implement in the elimination of impulsive noise and shows a good performance without the large computational complexity. In particular, the region to which the median filter is applied may be limited to portions in which the values of O ₁ (x) and O ₂ (x) are modified to 0, or may be extended to the entire region of the sound. The experimental results confirmed that when the amount of noise or block phenomena inserted into the sound was small, the median filter was applied to the areas where O ₁ (x) and O ₂ (x) were modified to 0. On the contrary, when the amount of noise inserted in the sound or the degree of block phenomenon is large, a better result can be obtained when the median filter application area is considered as the whole sound.

II. New Method for Noise Reduction

The algorithm proposed in the present disclosure, which is a synthesis of the results of the first chapter, is as follows.

2.1. New Algorithm for Noise Reduction and Block Phenomenon

① Wavelet transforms the given noise insertion sound in the x-axis direction and y-axis direction in step 2 ¹ (step 1 conversion) and step 2 ² (step 2 conversion), respectively.

,

Modulus values using

,

Calculate sure

to be. Rescale these modulus values from 0 to 255

Obtain

②

Calculate Among the values of D (x, y), the maximum value due to noise or block

Obtain

③ From the whole (x, y) definition

For regions of phosphorus (x, y)

,

Set to 0. The area of (x, y) is stored in Set (x, y).

(4) Apply the method described in Section 1.2 above to restore the edge values of the cutout boundary components. This way

,

Get

⑤ obtained in the previous step

,

Inverse wavelet transform to obtain the first target sound without noise.

⑥ Perform median filtering only in the area of Set (x, y) obtained in ③ above to get the final target sound without noise and block phenomenon. If the noise or block phenomena inserted in the sound is very large, it is more effective to apply this median filter to the entire area of the sound.

III. Additional Issues and Resolutions

3.1.

How to find

Finally, to use the algorithm presented in Chapter 2,

Must be found. However, to find this value, we must distinguish which part of the entire domain of D (x, y) corresponds to the boundary component and which part corresponds to the noise component without the boundary. In some ways, it seems that none of the problems are solved. This is because the boundary components and the noise components have to be distinguished again. In fact, however, it is important to realize that the discussion of the present disclosure has taken a step forward in solving the problem through the above long theoretical considerations. This is because now it is necessary to find only one region that can be determined to be composed of pure noise and block components without having to find the exact position of the boundary components for noise reduction. And the maximum value in this random region

This can be assumed.

To discuss this, let's first look at the aspect of noise reduction. Noise to be removed in this disclosure is additive white Gaussian noise. And this assumption is possible due to the fundamental nature of additive white Gaussian noise. This is because the property of white means that the statistical distribution of wavelet transformed noise component values is tentatively coincident through all domains of x. In other words, if only a part of the pure noise region without boundary lines is reliably included in the wavelet transformed sound, the distribution pattern of the wavelet transform values in the remaining pure noise region is also compared with the wavelet transform values in this specific region. It will have a similar distribution pattern. But another important problem arises. When the orthogonal wavelet filter is used, the noise distribution remains white even after the wavelet transform. However, when the biorthogonal wavelet filter is used, the noise distribution is white. The result is noise. Fortunately, the Redundant Biorthogonal wavelet filter used in the present disclosure was experimentally confirmed that it may be assumed to preserve some of the whiteness after filtering the noise.

In this disclosure, the variance difference between each region is used in the wavelet transformed sound of step 2 ² , that is, O ₂ (x, y), to find this random pure noise region. This was due to the assumption that the region containing both boundary and noise components was much larger than the region containing purely noise components. Table 1 below shows that this assumption is very valid (for convenience, we used data about images instead of sound).

Table 1

(Difference of variance values by area)

Based on the experimental results in Table 1, this study classifies 16x16 Pixels sized areas into variance values and assumes that the area with less than 100 variance values is a pure noise area. , y) is the maximum value of the entire area

Set to a value.

피처 벡터(feature vector) 추출 방법론Feature vector extraction methodology

The acoustic measurement unit 244 performs a machine learning function of automatically recognizing and determining what kind of signal this signal is based on the signal from which the noise is removed. There are many methods used for machine learning, but it is actually called a neural network, and according to the recent classification, the neural network is divided into a shallow neural network (SNN) and a deep neural network (DNN).

The difference between the SNN and the DNN depends on how many layers of the hidden layer are present.However, in actual theoretical terms, the difference between the SNN and the DNN is that the SNN forms the feature vector to be inserted into the Neural Network's input node and the input The neural network part that executes the optimization classification is separated, and in the case of the DNN, even the part constituting the feature vector is included in the neural network.

Recent studies have shown that the performance of DNNs is superior to that of SNNs, which is obvious. SNN has to optimize the feature vector and classifier sections, while the DNN is the feature vector and classifier sections. This is because it optimizes at the same time. However, once you find a good feature vector and make a successful machine learning machine using SNN, you can convert it to DNN.

Thus, a key part of this disclosure is how to construct the feature vector.

In order to configure the feature vector used in the present disclosure, the input signal is subjected to a wavelet transform. The wavelet transform can be thought of as a generalized version of the Fourier transform that transforms given time (or spatial) information into frequency information. Whereas the Fourier transform converts time information into all frequency information, wavelet transform maps the given time information into time-frequency space simultaneously. Therefore, the wavelet transform provides spatial information, which is difficult to extract from the conventional Fourier transform, and thus obtains much richer data to interpret the input signal.

A method of extracting a feature vector from a signal (eg, a sound) received in the present disclosure is as follows.

1. Make the input signal dyyadic.

Diadic means that a number is a power of two. For example, 3, 5, and 9 are not multiples of 2, so they are not diadic, but 2, 4, 8, 16, and 32 are 2 squared, so they are diadic. For 6 and 10, although they are even numbers divided by 2, they are not squared because they are not squared.

To perform wavelet transformation, as in the case of fast fourier transform (FFT), it is convenient for the input signal to be dialy.

Therefore, if there is an input signal, first check if the signal is diadic, and if not, continue to add zero-value data to this signal to make the number of data in this signal dia.

2. DWT the DIAD input signal by MRA method.

Although there are various forms of wavelet transform, the present disclosure performs a discrete wavelet transform (DWT) using a multi resolution analysis (MRA) method.

Its basic method is to separate a given signal into Low Frequency and High Frequency, and downsample them, respectively. And then downsample each one. This step proceeds until there is only one data in the low frequency range. 5 illustrates this.

5 is a diagram illustrating a two-stage downsampling process of an input signal according to an embodiment of the present disclosure, and FIG. 6 is a diagram illustrating a decomposition aspect of the input signal in the frequency domain according to an embodiment of the present disclosure.

Referring to FIG. 5, c _{j + 1} represents an input signal, g (−n) represents a high frequency, h (−n) represents a low frequency, and a downward arrow represents downsampling. c _{j +} ₁ is down-sampled to the d _j and c _{_j,} c _j is down-sampled back to d _j-1 and _j-1 c.

And from the perspective of the frequency domain it is expressed as shown in FIG.

C _j-1 in FIG. 5 corresponds to υ ₀ band which is the lowest frequency band in FIG. 6, d _j-1 in FIG. 5 corresponds to ω ₀ band in FIG. 6, and c _j in FIG. 5 is 6 corresponds to the ω ₁ band, and d _j in FIG. 5 corresponds to the ω ₂ band in FIG. 6.

3. Construct the second half of the feature vector during the MRA.

The feature vector of the present disclosure may be divided into a first half and a second half, and the first half expresses the magnitude of energy for each bandwidth in the frequency domain, and the second half expresses the magnitude of energy for each location section in the time domain.

The dimension (number of elements) constituting the first half of the dimension of the feature vector is log ₂ (N) +1. For example, if the size (number) of the input signal is 8, the number of first half elements is 4, 16 is 5, and 1024 is 11. In fact, this coincides with the number of bandwidths generated when MRA of an input signal by one difference.

On the other hand, the number of elements constituting the second half of the feature vector is the closest number to the number of elements in the first half. For example, if the size (number) of the input signal is 8, the number of second half elements is 4, 16 is 4, and 1024 is 8.

The latter half of the vector represents the magnitude of each energy in the time domain, which can be derived during the MRA. In principle, the MRA lasts until there is only one element corresponding to the low frequency. During this period, there is a point where the size of the low frequency region coincides with the number of the latter elements. This is because both the magnitude of the input signal and the number of latter elements are diadic.

If the size of the low frequency region is equal to the number of elements in the latter part of the feature vector, then the smallest value among the low frequency data is obtained. If this value is less than 0, all data are adjusted upward by the absolute value of this value. Make it positive.

The next step is to sum up the latter elements and divide each element by this value, which is a kind of normalization. If normalization is not performed, the probability of statistical bias is increased.

Then add the second element, the second element, the third element, and so on, in the second half, and find the position of the element just before the sum exceeds 0.5. The element is then moved in parallel so that the position of the element is just before the center of the latter vector (eg, fourth if the latter vector is 8). This is a way to overcome this, even if the input signal is shifted.

4. After finishing the MRA, construct the first half of the feature vector.

After completing the MRA, the total bandwidth plus one equals the number of first half elements in the feature vector.

The absolute value of each bandwidth is taken and then summed to obtain energy for each bandwidth.

Calculate the total energy by adding up the energy of each bandwidth, then divide each bandwidth energy by this number. That is, to perform some kind of normalization process.

5. Complete the final feature vector by joining the first and second half of the feature vectors.

Applying the machine learning tool to the obtained feature vector makes it possible to determine the type of sound.

The acoustic measurement unit 244 includes a sound sensor 2441 and a sound recognizer 2442, and the sound recognizer may operate only when the sound sensor 2441 is "on".

The sound sensor 2441 measures an input value (S710), and determines whether the input value is larger than a predetermined threshold value (S720). If the input value is less than or equal to the predetermined threshold, the controller waits for a predetermined time (for example, 3 seconds, 5 seconds, 10 seconds, etc.) (S730), and measures the input value again (S710).

If the input value is larger than the predetermined threshold value, the left / right sound card is recorded at the same time (S740), and the large energy data among the left / right sounds is selected as the analysis target (S750). Thus, by using large energy data among the left and right sounds, more accurate sound processing is possible.

Thereafter, voice filtering may be performed as necessary (S760). Although the present disclosure mainly deals with processing on living noise, the processing of voice signals may be performed in parallel in order to facilitate communication between a user and friends.

For example, you can help your friends communicate by dealing with words like "Wonder", "Like", "Hate", and "Sorrow". If the robot recognizes the voice "Wonder," and notifies the friend robot, the friend robot will display, for example, "Wonder Friend, Wondering" while displaying the "Wonder Expression" on the display screen at a predetermined time (for example, 5 seconds). You can do the output.

Thereafter, the input value is divided into data having sizes t1, t2, t3, and the like (S770). The present inventor needs about 5 seconds of time to determine, for example, whether it is a "washing machine" sound, but "cleaner", "microwave oven", "gas range", "keyboard", "window close", "water sound" It was found that a time of about 1 second was required to determine whether the sound was a sound of a lamp, and a time of about 0.5 seconds was required to determine whether it was a sound of "front door", "fridge door", "visit", and the like.

Therefore, the type of sound is determined based on sound data divided into data having sizes of t1, t2, t3, and the like (S780).

The controller 210 periodically rotates and analyzes the current situation of the robot 120 itself. The information used at this time is the history of the information of the synthesis sensor unit 240 and the type of the recognized sound. Based on this, the robot 120 selects and drives an appropriate motion, facial expression, and language to be performed by itself.

The controller thread obtains the latest value and the past value of the history array stored in the global variables for each sensor of the synthesis sensor unit 240 to construct a plurality of input vectors which are global variables (S801).

For example, the following 38-dimensional input vector can be configured.

Component 1: Illuminance Variation: Number of permutations that take two out of three: dark, appropriate, and bright = 9 levels

Element 2: Current illuminance: 3 levels of dark, appropriate, bright

Component 3: (Infrared) Distance Variation: Number of permutations that take three of three: far, appropriate, and near = 9 levels

Element 4: Current Distance (Infrared): 3 levels of Far, Apt, Near

Component 5: (Ultrasonic) Distance Variation: Number of permutations that take three of three: far, appropriate, and near = 9 levels

Element 6: (Ultrasonic) Current Distance: 3 Levels of Far, Apt, Close

Component 7: Temperature variation: number of permutations that select three out of three: cold, moderate, and hot = 9 levels

Element 8: Current temperature: 3 levels of cold, appropriate, and hot

Component 9: Humidity Variation: Number of permutations of three of three: wet, moderate, dry

Component 10: Current Humidity: Humid, Proper, 3 levels of drying

Component 11: Washing Machine Sound Variation: Number of Permutations to Choose Two of Two, Washing or Not Washing = Level 4

Element 12: Current Washing Machine Sound: 2 Levels of Washing, Not Washing

Component 13: Variation of Cleaner Sounds: Number of Permutations to Take Two of Two: Clean or Not Clean = Level 4

Element 14: Current cleaner sound: 2 levels of cleaning, not cleaning

Component 15: Keyboard Sound Change: Number of permutations that take two of two, operation or non-operation = 4 levels

Element 16: Current keyboard sound: two levels of operation, not operation

Component 17: Microwave Sound Change: Number of Permutations to Take Two of Two: Run or Not Run = Level 4

Element 18: Current Microwave Sound: Two Levels of Operation, Non-Operation

Component 19: Gas Stove Sound Change: Number of Permutations to Choose Two of Two: Run or Not Run = Level 4

Element 20: Current stove sound: Operation, non-operation 2 level

Component 21: Window Sound Change: Number of permutations to choose two of two: opening and closing, no sound = level 4

Element 22: Current window sound: 2 levels of opening and closing, no sound

Component 23: Front Door Sound Change: Number of Permutations to Take Two of Two: Close or No Sound = Level 4

Element 24: Current front door sound: 2 levels of close, no sound

Component 25: Refrigerator Door Sound Change: Number of permutations to choose two of two: Close or No Sound = Level 4

Element 26: Current Refrigerator Door Sound: Close, Two Levels of No Sound

Component 27: "Wonderful" Sound Change: Number of permutations that take two out of "Uncertain" or "Uncertain" = 4 levels

Element 28: Current "Uncertain" Sounds: Level 2 of "Uncertain", "Uncertain"

Component 29: "Like" Sound Changes: Number of permutations that take two out of "like" or "like" = 4 levels

Component 30: Current "Like" Sound: Level 2 of "Like", "Like"

Component 31: "No" Sound Changes: Number of permutations that take two of two "no", "no" = 4 levels

Component 32: Current "No" Sounds: Level 2 of "No", "No"

Component 33: "Sad" Sound Changes: Number of permutations that take two of two: "Sad" or "Sad" = Level 4

Component 34: Current "Sad" Sounds: Level 2 of "Sad", "Sad"

Component 35: Final time voice prompted washing machine sound

Component 36: End Time Voice Prompted Cleaner Sound

Component 37: End time voiced keyboard sound

Component 38: Final time of voice prompt for microwave sound

Thereafter, an output matrix corresponding to the plurality of input vectors is constructed (S802). For example, a 17 * 4 output matrix corresponding to a 38-dimensional input vector is constructed. For example, row values include specific contextual information about a change in conditions, such as {sound volume change, illuminance change, user distance change, temperature change, humidity change, washing machine sound change, cleaner sound change, keyboard sound change, microwave sound. Change, gas range sound change, window sound change, front door sound change, refrigerator door sound change, wonder change, liking change, dislike change, sad change}. Voice, friend's robot expression, friend's robot voice}. This is shown in the table below.

TABLE 2

(Example of 17 * 4 output matrix)

Operation starts at i = 0, the row number of the matrix (S803).

The i-th row is selected (S804), and the time, IP, the robot UUID, the owner ID, the energy size, the input vector, the output vector, and the UUIDs of the friend robots are stored in the management server 110 (S805).

It is determined whether the audio output time is possible (S806). If the user has set the voice output possible time from 8:00 am to 10 pm, if it is 11 pm, since the voice output is not possible, the robot 120 outputs only an expression without voice (S807). However, if it is 8 pm, the robot 120 outputs a voice and facial expression (S808).

It is determined whether the output vector is a set sharing activity (S809). For example, when the sound of the "front door" has been recognized and it is determined that the sound is to be shared with the friend robot, the output vector is transmitted to the friend robot's UUID channel (S810).

If the sound of the "front door" is not to be shared with a friend robot, i = i + 1 (S812), and determines whether i is greater than N (N is a natural number, for example, "17"). If i is smaller than N (S812), the i-th row is selected (S804).

If i is larger than N, the operation is waited for a predetermined time (for example, 1 second, 3 seconds, 5 seconds, etc.) (S813), and the operation is resumed from step S801.

In an embodiment of the present disclosure, when the management server 100 determines that a dangerous situation occurs based on the situation information, the management server 100 of the second user who has decided to share living activity information with the user of the home social robot 120. And send an emergency notification message to the terminal or the second home social robot. The situation information may include at least one of opening and closing the door, opening and closing the window, tapping the water, turning on the stove, turning on the microwave, opening the refrigerator, operating the cleaner, turning on and off the lights in the house, turning the TV on and off, and human movement. have. That is, when the management server 100 judges based on the status information such as "turning on the gas range" and "opening the front door", opening and closing the front door in the home of the single household elderly-> turning on the gas range-> opening and closing the front door If there is a change in the second home (for example, family, social worker, etc.) and / or your own terminal to decide to share the living activity information with the user by judging that you turned out the gas range in the home and going out An emergency notification message (eg, a mobile phone text message) can be sent. In this case, turning on the gas range may be detected by, for example, the acoustic measurement unit 244 and the temperature and humidity measurement unit 242 of the synthesis sensor unit 240, and opening and closing the front door may be performed by the acoustic measurement unit of the synthesis sensor unit 240. 244, the grid-eye unit 245, and the gyro accelerator unit 246 can be detected.

In the embodiments disclosed herein, the arrangement of the components shown may vary depending on the environment or requirements on which the invention is implemented. For example, some components may be omitted or several components may be integrated and implemented as one. In addition, the arrangement order and connection of some components may be changed.

Components of the embodiments of the present disclosure described above may be implemented in hardware, software, firmware, middleware, or a combination thereof, and may be utilized as systems, subsystems, components, or subcomponents thereof. It must be understood. If implemented in software, the elements of the present disclosure may be instructions / code segments for performing the necessary tasks. The program or code segments may be stored in a machine readable medium, a computer program product, such as a processor readable medium. Machine-readable media or processor-readable media can include any medium that can store or transmit information in a form readable and executable by a machine (eg, processor, computer, etc.).

Although various embodiments of the present disclosure have been shown and described, the present disclosure is not limited to the above-described specific embodiments, and the above-described embodiments deviate from the gist of the present disclosure as claimed in the appended claims. Without this, various modifications can be made by those skilled in the art without departing from the scope of the present invention, and these modified embodiments should not be understood separately from the spirit or scope of the present disclosure. Accordingly, the technical scope of the present disclosure should be defined only by the appended claims.

Claims

As a home social robot management system for sharing daily activity information based on synthetic sensor,

A plurality of home social robots and a management server connected to communicate with the home social robots;

The home social robot

Synthetic sensor unit for receiving the living noise and living environment information,

Speaker to output the status of the home social robot as a sound,

A display for outputting the status of the home social robot as an image;

A communication unit for communicating with a management server of a home social robot through a network;

And a control unit connected to the synthesis sensor unit, the speaker, the display, and the communication unit,

The control unit transmits the living noise and living environment information obtained from the synthesis sensor unit to the management server,

And the management server is configured to determine context information based on the living noise and living environment information and transmit the situation information to the home social robot.
The method of claim 1,

The synthesis sensor unit includes at least two of the distance measuring unit, gyro acceleration unit, temperature and humidity measuring unit, illuminance measuring unit, grid-eye unit, and sound measuring unit, home social robot management system.
The method of claim 2,

And the home social robot comprises a hub robot and one or more edge robots communicatively coupled to the hub robot.
The method of claim 3,

The distance measuring unit includes at least one of an infrared meter and an ultrasonic meter, home social robot management system.
The method of claim 1,

The acoustic measuring unit includes a sound sensor and a sound recognizer,

And the sound sensor is configured to notify the sound recognizer to operate the sound recognizer only when the energy magnitude of the sound is above a predetermined threshold.
The method of claim 5,

And the acoustic measurement unit is configured to divide the noise canceled input signal into different time magnitudes.
The method of claim 6,

The acoustic measurement unit first removes noise by performing wavelet transform on the input signal related to the living noise and living environment information, and removes the noise second by applying a median filter after inverse wavelet transformation. Configured, home social robot management system.
The method of claim 7, wherein

And the management server is configured to firstly remove noise by wavelet transforming the input signal and secondly remove noise by applying a median filter after the inverse wavelet transform.
The method according to claim 7 or 8,

And the management server is configured to apply a wavelet transform to the noise-free input signal to extract a feature vector and to classify a sound type based on the extracted feature vector.
The method of claim 9,

The situation information may include at least one of opening and closing of the front door, opening and closing of the window, turning on the tap, turning on the stove, turning on the microwave, opening the refrigerator, operating the vacuum cleaner, turning on and off the lights in the house, turning the TV on and off, and human movement. , Home social robot management system.
The method of claim 10,

And the management server is configured to transmit the contextual information to a terminal of a second user or a second home social robot that has decided to share living activity information with the user of the home social robot.
The method of claim 11,

And the management server instructs at least one of a facial expression and a voice of the robot to at least one of the home social robot and the second home social robot based on the situation information.
The method of claim 12,

When the management server determines that a dangerous situation has occurred based on the situation information, an emergency notification message to a terminal of a second user or a second home social robot that is to share life activity information with the user of the home social robot. A home social robot management system, configured to send it.
The method of claim 13,

The living noise may include at least one of a washing machine sound, a cleaner sound, a microwave oven sound, a gas stove sound, a keyboard sound, a window opening sound, a water sound, a front door sound, a refrigerator door sound, a visit sound, and a footstep sound. Robot management system.
In the home social robot management system for living activities sharing, as a method of determining the sound type,

Receiving living noise and living environment information as an input signal;

Removing noise from the input signal;

Performing a wavelet transform on the noise-free signal to obtain a feature vector;

Determining the type of sound by applying the feature vector to a machine learning tool.

A sound type determination method comprising a.
The method of claim 15,

Removing the noise

Performing noiselet transform on the input signal to remove noise primarily;

Performing an inverse wavelet transform on the first noise canceled signal;

Secondly removing a noise by applying a median filter to the inverse wavelet transformed signal

A sound type determination method comprising a.
The method of claim 16,

Obtaining the feature vector

Making the noise canceled signal dyed;

Discrete Wavelet Transform (DTW) of the dilated signal by MRA

Obtaining a second half of the feature vector representing the magnitude of energy for each location section in the time domain;

Obtaining a first half of the feature vector representing the magnitude of energy for each bandwidth in the frequency domain;

Connecting the first half and the second half of the feature vector to obtain a final feature vector

A sound type determination method comprising a.