KR20200023013A - Video Service device for supporting search of video clip and Method thereof - Google Patents

Abstract

According to an embodiment of the present invention, a video service device can comprise: a memory storing a video clip; and a processor functionally connected to the memory. The processor can be configured to acquire a video clip stored in the memory, divide the video clip in scene units, extract candidate keywords by performing at least one recognition operation on at least one divided scene of the video clip, and select a representative keyword for the video clip based on the extracted candidate keywords.

Description

Video service device for supporting video content search and support for video content search {Video Service device for supporting search of video clip and Method

The present invention relates to video content search. In particular, a video service apparatus and video content search support that support video content search for extracting a representative keyword of video content and mapping the video content to the video content for easy retrieval It is about a method.

Recently, the form of consuming content is rapidly shifting from text to image or video, and in particular, the consumption of video (or video or video) content information is rapidly increasing. In addition, if the conventional video content was merely a tool in terms of media consumption, the current video content is also used as a tool for communication and information retrieval.

On the other hand, the video content is flooded, but due to the busy living environment it is difficult to watch or replay the video content of 1 to 2 hours. Therefore, recently, contents that provide a short time (eg, 1 to 2 hours) to extract a core part of content for 2 to 3 minutes are provided rather than replaying the entire content. In order to meet this consumption trend, the broadcasting station produces and provides a short clip image which extracts the video in about 3 to 5 minutes in addition to the entire video of 1 to 2 hours when serving the broadcast contents.

On the other hand, in the case of the original media image, the basic metadata that the content creator has worked with is relatively rich and accurate (represented characters, shooting locations, synopsis, genre, etc.), and the original image is reproduced in about 2 to 3 minutes. The edited short clip video is difficult to secure metadata about characters in the clip, the plot of the clip, and the background. In addition, the short clip image has a problem that it is impossible to secure the original information for extracting the metadata. In addition, in the case of UGC, when a general user, not a professional producer, produces a meta input for a title or a detailed description of a video, it is often incorrect.

Therefore, in the case of providing a video content streaming service including full video and clip video, the main keyword information about the content (the hash tag # used to compress the video information and use it for search or recommendation) is provided manually by a person. However, there is a problem that information inaccuracy is high and information providing efficiency is very low.

Korean Laid-open Patent No. 10-1640317, registered on July 11, 2016 (Name: Image storage and retrieval device and storage and retrieval method including video and audio data)

The present invention is to meet the above-described needs, an image service apparatus that supports video content retrieval that can automatically generate a representative keyword (hash tag) that describes video content such as a movie or broadcast or UGC content based on recognition technology. And an image content search support method.

In addition, the present invention provides a video service apparatus and a video content search support method for supporting video content search, which provides a viewer with summary information of a corresponding broadcast clip content, and further effectively searches and recommends a broadcast clip content unit. .

An image service apparatus according to an embodiment of the present invention may include a memory for storing an image clip and a processor operatively connected to the memory. The processor acquires an image clip stored in the memory, divides the image clip into scene units, and performs at least one recognition operation on the divided scenes or at least one scene of the image clip to at least one candidate. A keyword may be extracted and a representative keyword for the video clip may be selected based on the at least one candidate keyword.

Alternatively, the at least one candidate keyword may be extracted through graphic subtitles applied to the at least one scene.

The at least one candidate keyword may be extracted according to a complex application of the Feature Vector extracted for each frame included in the at least one scene and the original Video Feature Vector (Global Feature) of the video clip.

The at least one candidate keyword may be assigned a priority through a score model.

In particular, the processor may be set to select the highest priority as a representative keyword for the video clip.

The processor may be configured to perform at least one recognition operation among face recognition, time or season recognition, place recognition, dialogue recognition, music recognition, situation or event recognition.

According to an embodiment of the present invention, a method of supporting image content search includes: acquiring, by a video service apparatus, an image clip requiring a keyword selection, dividing the image clip into scene units, and at least one divided scene of the image clip The method may include extracting at least one candidate keyword by performing at least one recognition operation on, and selecting a representative keyword for the video clip based on the at least one candidate keyword.

In the video content search support method, the video clip may include video content having a reproduction time length of several minutes to several ten minutes provided by editing at least one scene of the original video.

The present invention utilizes recognition technology such as image recognition, subtitle recognition, voice recognition, etc. for scene content having a certain length of time, as well as video clip content or UGC of 2-3 minutes, and includes scene meta information (eg: Candidate keywords) can be automatically generated and analyzed to automatically select keywords representing image content, thereby providing more accurate keyword selection and work efficiency.

1 is a diagram illustrating an example of a network environment according to an exemplary embodiment of the present invention.
2 is a diagram illustrating a schematic form of a video service device according to an exemplary embodiment of the present invention.
3 is a diagram illustrating an example of a processor of an image service apparatus according to an exemplary embodiment of the present invention.
4 is a diagram illustrating an example of a scene meta extractor of the present invention.
5 is a diagram illustrating an example of a scene recognition method according to an exemplary embodiment of the present invention.
FIG. 6 is a diagram illustrating an example of a method of generating a keyword map associated with image content search support according to an exemplary embodiment of the present invention.

Hereinafter, various embodiments of the present invention will be described with reference to the accompanying drawings. It is to be understood, however, that the intention is not to limit the invention to particular embodiments, but to cover various modifications, equivalents, and / or alternatives to the embodiments of the invention. In connection with the description of the drawings, similar reference numerals may be used for similar components.

1 is a diagram illustrating an example of a network environment according to an exemplary embodiment of the present invention.

Referring to FIG. 1, a network environment 10 according to an embodiment of the present invention may include a network 50, an external server device 300, an image service device 200, and a terminal 100.

The net of the present invention described above? The environment provides representative keywords by analyzing information contained in media, especially (video) content, and automatically generates keyword information for effectively describing and expressing video content, and based on the keyword information, a summary description and search of the video. It can be used for various purposes such as and recommendation. In particular, the present invention divides the video content by scenes, and a minority characterizing the video based on detailed metadata or context information acquired through image recognition and voice recognition (or subtitle recognition, OCR recognition, etc.) for each divided scene. It provides a way to select key keywords from. The present invention overcomes the limitation of providing simple information related to an actor (eg, providing information on a scene (viewing point) in which a specific actor appears) occurring in a scene recognition process for each scene using a conventional simple face recognition technology. It supports to calculate the representative keyword information characterizing.

The network 50 is not only an IP-based wired communication network such as the Internet network, but also a long term evolution (LTE) network, a mobile communication network such as a WCDMA network, various types of wireless networks such as a Wi-Fi network, and a combination thereof. Can be made. That is, the network environment 10 related to image content search support according to the present invention may be applied to both wired and wireless communication networks without distinction. In more detail, the network 50 may establish a communication channel between the video service device 200 and the terminal 100. Alternatively, the network 50 may establish a communication channel between the video service device 200 and the external server device 300. For example, the network 50 may support at least one of 3G, 4G, and 5G wireless mobile communication methods that the video service device 200, the external server device 300, or the terminal 100 may operate. Can be. Alternatively, the network 50 may form a communication channel between the video service device 200 and the external server device 300 or a communication channel between the terminal 100 and the video service device 200 on a wired basis. Such a network 50 should be interpreted as a concept including various wired networks, wireless networks, and combination networks thereof that are currently developed and commercialized, or may be developed and commercialized in the future.

The terminal 100 may access the video service device 200 through the network 50. The terminal 100 may serve as a consumer that can search for, view, or download various image contents provided by the image service apparatus 200. In this regard, the terminal 100 may display a terminal communication circuit for accessing the image service apparatus 200, a display capable of outputting a web page for searching for image contents received from the image service apparatus 200, and image content search. It may include an input unit for generating an input signal related to the memory, a memory for temporarily or semi-permanently storing the received image, a control unit for controlling the above-described configuration, for example, the terminal communication circuit, the display, the input unit, the memory, etc. . On the other hand, the terminal 100 of the present invention may serve as a provider to upload the video content (for example, a short clip video) produced by the terminal to the video service device 200. In this case, the terminal 100 may provide an editing function for producing a short clip image generated from image contents stored for a long time (for example, an image of several tens of minutes to several hours).

The external server device 300 may store at least one video content and provide the at least one video content according to a request of the video service device 200. Alternatively, the external server device 300 may generate various algorithms necessary while the video service device 200 generates keyword information corresponding to video content having a predetermined length of time (eg, a short clip video, hereinafter referred to as a “video clip”) or Various meta candidate information may be provided. Alternatively, the external server device 300 may perform various algorithms and data (eg, a face recognition algorithm and a face recognition DB, an audio recognition algorithm and an audio recognition DB, a background analysis algorithm, and a background DB) for scene analysis of an image clip. And a place DB, a music recognition algorithm, and a music DB) may be stored and provided to the image service device 200. The external server device 300 may be included in the video service device 200.

The video service device 200 may receive and store at least one video clip from the external server device 300 or the terminal 100. Alternatively, the image service device 200 may receive and store at least one image clip through a separate data receiving path (eg, an input interface or a memory device connection). The video service device 200 divides the received video clip by scene, applies various recognition modules to the divided scene, obtains various meta information (eg, candidate keywords), and applies the obtained meta information to the corresponding scene. By analyzing corresponding to the video clip, it is possible to select at least one representative keyword that can represent the video clip. The video service device 200 may map the selected keyword to a corresponding video clip and store the same. In this case, the image service apparatus 200 may store and manage the selected keyword as a hash tag of the corresponding video clip so that the selected keyword may be searched with the selected keyword.

2 is a diagram illustrating a schematic form of a video service device according to an exemplary embodiment of the present invention.

Referring to FIG. 2, the video service device 200 of the present invention may include a communication circuit 210, a memory 230, and a processor 250.

The communication circuit 210 may form a communication channel of the video service device 200. For example, the communication circuit 210 may establish a communication channel with the external server device 300 and receive at least one image clip from the external server device 300 at regular intervals or according to an administrator's request. The communication circuit 210 may communicate with the external server device 300 to collect external data of at least one scene included in the image clip 231 in the process of analyzing the image clip 231 stored in the memory 230. It may also form a communication channel. Meanwhile, the communication circuit 210 may form a communication channel supporting the connection of the terminal 100 through, for example, the network 50. In this case, the communication circuit 210 may provide a screen for inputting a search word to the connected terminal 100 and receive a search word (or keyword) for searching for a specific image clip 231 from the terminal 100. The communication circuit 210 may provide the terminal 100 with an image clip 231 corresponding to a keyword provided by the terminal 100 in response to the control of the processor 250.

The memory 230 may store at least one image clip 231, a meta DB 233, and a keyword map 235. The image clip 231 may be received from the external server device 300 or the terminal 100. Alternatively, the image clip 231 may be manufactured and stored by the image service apparatus 200 itself. The image clip 231 may include, for example, short image content having a relatively short playing time (eg, several minutes to several minutes) generated from long time image content. For example, the video clip 231 may include the aforementioned Short Clip image or UGC image. Alternatively, the image clip 231 may include image content having a size equal to or less than a specified amount of meta information. The memory 230 may store and manage a plurality of image clips 231. The image clip 231 may be recognized by various recognition modules and may be assigned at least one keyword selected based on a result of recognition, that is, meta information. The keyword may be used to search for the video clip 231.

The image clip 231 may include at least one scene. The at least one scene may be, for example, a plurality of frames constituting an image or I (Intra) -frames among the plurality of frames. Alternatively, the scene may include a plurality of frames in which the background screen does not change over a certain area. Accordingly, at least one person, animal, or object may be disposed in the image clip 231, and even if the object displays a certain movement, the background screen may be processed as one scene when the background screen is maintained over a certain area.

The meta DB 233 may store meta information about the image clip 231 recognized by various recognition modules. Meta information about the image clip 231 may be used to select a keyword of the image clip 231. In addition, the meta DB 233 may store at least one keyword information to be allocated to the image clip 231. Each keyword may be mapped to at least one meta information. Accordingly, the processor 250 calculates at least one keyword based on the meta information obtained in the recognition process of the image clip 231, and selects the representative keyword based on the appearance time, the number of appearances or the importance of the calculated keywords. Can be selected. Various meta information and keywords stored in the meta DB 233 may be updated by information provided by the external server device 300.

The keyword map 235 may include information that maps identification information of the image clip 231 and at least one keyword mapped to the identification information. In addition, the keyword map 235 may include information that maps identification information of a specific video clip 231 and representative keyword information representing the video clip. The keyword map 235 may be updated as a process of recognizing and calculating a keyword is performed after a new image clip 231 is received.

The processor 250 may provide a function of selecting a keyword and searching an image clip 231 of the image service apparatus 200. In this regard, the processor 250 receives at least one image clip from the external server device 300 or the terminal 100 in response to a predetermined period or an administrator request or an external server device 300 request, and stores the image clip in the memory 230. Can be stored. When a new image clip is stored in the memory 230 or a predetermined time arrives, the processor 250 may perform a keyword selection operation on the image clip 231 stored in the memory 230. In the keyword selection operation, the memory 230 classifies and analyzes the scene information on the image clip 231, various recognition processing on the divided scenes, and classifies and analyzes meta information obtained according to the recognition process according to a result of analysis. Selection of one keyword or representative keyword can be performed. The processor 250 may provide a search function for the image clip 231 by storing and managing the selected representative keyword as a hash tag of the corresponding image clip 231.

3 is a diagram illustrating an example of a processor of an image service apparatus according to an exemplary embodiment of the present invention.

Referring to FIG. 3, the processor 250 may include a scene meta extractor 251, an image integrated meta extractor 253, an external data extractor 255, an analyzer 257, and a keyword selector 259. have. As such, the processor 250 of the present invention may automatically generate a representative keyword of an image clip based on an image recognition technology. Alternatively, the processor 250 may automatically generate summary information describing the content of the video clip. For example, the processor 250 may provide a multi-label classification task (a function of providing a plurality of key keyword information describing a corresponding video) for one image clip file. In this operation, the processor 250 may select up and list a keyword having significant or high importance among tens of thousands of meta extracted in scene units by various image recognition technologies, and provide a predetermined number of keywords.

The scene meta extractor 251 is a recognizer (or a recognition module or a recognition processor) for automatically recognizing scene meta. The meta information extractor 251 may use meta information which may well describe information of image content according to the principle below. Can be generated. For example, the scene meta extractor 251 may automatically generate who is appearing on the screen through face recognition technology, and generate background place information where a story is unfolded through place recognition technology, and image content through context recognition technology. You can create the status / event information that is in progress. In addition, the scene meta extractor 251 may generate main object information appearing in the image content through object recognition technology, and analyzes the subtitles and subtitles written on the screen through voice recognition (metabolism) and subtitle recognition technology. Additional Meta can be extracted. In addition, the scene meta extractor 251 may generate special subtitle information by recognizing graphic processed special subtitles inserted by a media producer on the screen (using an Optical Character Recognition (OCR) recognition technology). For example, the scene meta extractor 251 may extract detailed place information such as special place information #Paris Sangeri Street #New York Times Square applied to a special subtitle, such as entertainment content. As described above, the scene meta extractor 251 divides an image clip into scene units and localizes each scene unit based on metadata extracted in a section divided by scene units (or image units or local units). Feature) can be extracted.

The image integrated meta extractor 253 may recognize the entire meta of the image clip. The image integrated meta extractor 253 may classify the entire video as a single feature vector instead of viewing information in units of frame images. That is, the image integrated meta extractor 253 may comprehensively grasp the contents of the entire image to specify the genre or to extract one keyword having representativeness. For example, the image integrated meta extractor 253 may learn Music Video in large quantities, and classify keywords as Music Video when the video clip is associated with the music video. Alternatively, the image integrated meta extractor 253 may learn a large amount of soccer images, and classify keywords of soccer-related video clips as soccer. Here, the image integrated meta extractor 253 classifies the image contents in a batch to access the map learning method (Video Feature Vector), connects the vector extracted for each frame of the image content, and classifies and approaches the aggregate (Aggregate Vector), And a method of utilizing data having continuity throughout the image, such as background music / metabolism, to increase the accuracy and variety of classification results. As described above, the image integrated meta extractor 253 may grasp and extract the global feature by grasping the overall feature of the video file. For example, the video integrated meta extractor 253 classifies a comprehensive keyword for a video clip by combining keywords corresponding to Genre, a relationship between front and rear frames of video content, and a complex relationship between an image and an audio file. I can do it.

The external data extractor 255 is a component that collects data from an external database (eg, a database of the external server device 300) through a web crawler. Analyze data from information sources. The data collected by the external data extractor 255 may be text data, and may be transformed into keyword candidate data of noun or adjective form through a topic model and a morpheme analyzer.

The analyzer 257 generates a model corresponding to Meta (data extracted by the scene meta extractor 251) and Video unified Meta (data extracted by the image integrated meta extractor 253) for each frame of the extracted video clip, and the corresponding model. A plurality of Keyword List candidate groups can be generated by mapping (categorizing) keywords corresponding to the. On the other hand, since the values included in the keyword list candidate group are vastly larger than the representative keywords characterizing the video clip, the analyzer 257 may perform scoring modeling to extract important and meaningful keywords from the values included in the keyword list candidate group. have. In the Scoring Model, priorities are determined from the extracted keywords, and score factors can be used for this. The score factor may include a frequency score, a confidence score, a co-occurrence score, and a special score. The frequency score may include the frequency at which each Meta appears in the image. For example, when a person who appears in a scene is recognized as an actor "Jung Woo Sung", the number of frames that Jung Woo Sung appears and recognized may be a frequency score. The confidence score may be a confidence score when each meta is extracted from the recognizer. For example, when it is recognized that a specific scene of the video clip is a bungee jumping scene, the confidence score may be a confidence value to which the bungee jumping scene is correct. The co-occurrence score may include a score that raises the priority due to the simultaneous appearance when a specific keyword simultaneously appears in the recognized Meta value and data collected through external crawling. The special score is a score for assigning a high score to keywords such as kiss scenes and wedding scenes that want to adjust their priorities according to service planners' intentions. These special scores can change the logic for assigning scores according to the planner's intent and genre.

The keyword selecting unit 259 may check final scores of the keywords output through the analyzer 257 through the above-described process, and select a representative keyword for the video clip according to the final score. Alternatively, the keyword selector 259 may select at least one neighboring keyword for the video clip according to the scores of the candidate keywords included in the keyword candidate group list. For example, the keyword selecting unit 259 may select a keyword having the highest score as a representative keyword, and select at least one keyword having a next or second ranking score as a neighboring keyword. The keyword selector 259 may update the keyword map 235 when keyword selection is completed. For example, the keyword selector 259 may newly write a mapping value obtained by mapping a representative keyword or at least one neighboring keyword calculated in a specific image clip to the keyword map 235.

The above-described embodiments may be modified in various forms. For example, the processor 250 may variously configure a recognition unit, a score model, an external data extractor, etc. according to the characteristics or genres of image clips. In addition, a recognition engine (or a recognition algorithm or a recognition module, etc.) operated by the processor 250 may be variously combined, and elements and logic reflected in the score model may be adjusted.

In addition, the processor 250 recognizes all frames in the frame-by-frame recognition process of the video clip or skips every predetermined frame in consideration of trade-off of data processing speed, accuracy, and utility of the video clip. Conditions such as whether to process may vary.

4 is a diagram illustrating an example of a scene meta extractor of the present invention.

Referring to FIG. 4, the scene meta extractor 251 may include a face analyzer 251a, a situation analyzer 251b, and an OCR recognizer 251C.

The scene meta extractor 251 may extract a keyword through image recognition for each scene divided based on scene (or scene) segmentation. In this case, since the content related to a person, a place, an event, and an object in a scene based on a story line is repeatedly recorded for a predetermined period of time and appeared over a plurality of frames, the scene meta extractor 251 is frame-by-frame. Each recognition engine can be crossed and processed.

The face analyzer 251a may analyze the face of the person included in the scene and extract meta information about the face of the actor. If a scene includes a plurality of people, the face analyzer 251a may extract meta information (eg, name information) about the plurality of people.

The situation analyzer 251 b may recognize a background included in a scene, for example, a place, a time or a seasonal background, and extract meta information corresponding to a recognition result. Alternatively, the situation analyzer 251 b may recognize the situation of the scene and may extract meta information about the recognized situation (eg, driving situation, exercise situation, marriage situation, shootout, etc.). In addition, the context analyzer 251b may recognize at least one object included in the scene, and may extract meta information (eg, desk, chair, mountain, river, sea, etc.) corresponding to the recognized object.

The OCR recognizer 251C may extract special subtitle information written in the scene.

The scene meta extractor 251 may classify a plurality of frames included in a specific scene of an image clip in units of time stamps and sequentially apply the above-described analyzers. For example, when the scene is composed of hundreds of frames, the scene meta extractor 251 recognizes a person who appears in the scene through face recognition using the face analyzer 251a for a frame corresponding to timestamp 1, and timestamp 2 For a frame corresponding to, a time / seasonal background of the scene may be recognized through time recognition using the situation analyzer 251b. The scene meta extractor 251 recognizes a place appearing in the scene by using a situation analyzer 251b for a frame corresponding to timestamp 3, and uses a situation analyzer 251b for a frame corresponding to timestamp 4. In this case, the situation can be recognized through the situation recognition, and the frame corresponding to the timestamp 5 can be recognized through the object recognition using the situation analyzer 251b. The scene meta extractor 251 may cross process each recognition engine used in the above-described operation. For example, suppose a scene is 3 minutes long and processes timestamps at 1 second intervals, the scene meta extractor 251 processes face recognition, time recognition, place recognition, situation recognition, and object recognition 36 times each. It can provide the processing result. Here, the video length and timestamp unit time interval of the video clip may be adjusted. For example, a relatively large number of keywords may be extracted by processing a narrower time stamp interval by an administrator request or according to a keyword policy for a video clip, or only some keywords may be extracted by further increasing the time stamp interval. In this case, the corresponding information may be missed at a specific timestamp, but it may be processed repeatedly and each keyword information may be extracted, and the post-processing time that needs to be sorted / refined may be more efficient when the same information appears several times.

The meta information processed by the scene meta extractor 251 described above may include the contents of the following six principles.

For example, the part about Who is related to which actor / celebrity is appearing in the corresponding broadcast content. The face recognition function can be solved when applied by learning / creating a face recognition function based on a celebrity DB. When passing through the face recognition engine, the scene meta extractor 251 may generate a keyword (hash tag) about a person through automatic recognition in the case of an actor / entertainer registered in a celebrity DB.

The part about When relates to time or season information when the broadcast content is captured, and may recognize time-season information by recognizing a specific entity capable of inferring time / season information. For example, when the scene meta extractor 251 recognizes the cherry blossoms on the screen, it is determined that it is spring, and when the winter padding or snow scene is recognized, it is determined to be #winter #snow, and when it recognizes the foliage or cosmos, it is determined to be autumn. Can be.

Where (Where) corresponds to the place information where the broadcast content is photographed, it is possible to determine the place information by recognizing the buildings, landmarks, etc. representing the main place. For example, the scene meta extractor 251 may recognize key facilities such as a church / cathedral, an amphitheater, an airport, or may recognize key landmarks such as the Eiffel Tower, the Blue House, Cheomseongdae, etc., and provide name information.

What is related to what is happening in the broadcast content, including situations involving sports / hobbies activities, events such as birthday parties, weddings, and sixtieth birthday feasts. can do. The scene meta extractor 251 may recognize a tool object characterizing the sport / hobbies, or may recognize the object characterizing the event and provide meta information about the decision “what”. For example, if the scene meta extractor 251 recognizes a baseball bat / baseball field, #recognizes a baseball, a fishing rod, and bait #recognizes a fishing, a balloon, and a birthday cake #recognizes a birthday party, a wedding dress, and a wedding hall # Meta information such as a wedding can be provided.

The scene meta extractor 251 may generate more rich and accurate keywords using the OCR recognizer 251C. For example, the screen may recognize and utilize a graphic processed subtitle inserted by a media producer. In this case, the scene meta extractor 251 may recognize a special caption recognized as graphic data rather than text data using the OCR recognizer 251C.

As another example, the scene meta extractor 251 may cross process the above-described recognition engines (eg, face recognition, time recognition, place recognition, situation recognition, and object recognition) while arbitrarily skipping a frame without discriminating a scene. have. Since each recognition engine processes multiple images more than once within a scene, the division between scenes may not significantly affect the extraction of the entire keyword. Accordingly, the arbitrary frame skipping process may cancel a portion that affects the overall keyword extraction due to the performance of the scene classification algorithm. The scene meta extractor 251 extracts the meta information passing through each recognition engine by frame skipping the entire video clip at arbitrary time intervals without distinguishing the scene, and then selects a keyword that is representative of the video clip based on this. Can be extracted. The scene meta extractor 251 may compare the extracted keywords according to the scene-specific analysis with the extracted keywords based on the selected frame analysis according to a certain skip policy, and select a representative keyword based on the comparison result. .

5 is a diagram illustrating an example of a scene recognition method according to an exemplary embodiment of the present invention.

Referring to FIG. 5, in relation to a scene recognition method according to an embodiment of the present disclosure, the processor 250 of the image service apparatus 200 may acquire an image clip 231 in step 501. For example, the processor 250 may read an image clip 231 requiring keyword selection from the memory 230.

In operation 503, the processor 250 may convert an image clip 231 into an image. For example, the processor 250 may convert an image clip 231 having a video structure to be viewed in units of frames.

In operation 505, the processor 250 may divide the converted video clip in scene switching units. That is, the processor 250 may recognize a time point at which the scene is changed, and may divide the frames into scene units based on the time point. For example, the processor 250 may compare the intra frames used for the scene change, and if the predetermined data is changed more than the predetermined time, the processor 250 may recognize the corresponding view as the scene dividing view.

In operation 507, the processor 250 may perform face recognition for each scene. For example, the processor 250 may recognize a face of a person and may extract a recognition result (eg, an actor's name corresponding to the person).

In operation 509, the processor 250 may perform time and season recognition of the scene. For example, the processor 250 may classify day or night through illumination applied to a scene, recognize time and season based on snow, rain, and the like, and extract a recognition result.

In operation 511, the processor 250 may perform scene / place recognition for each scene. For example, the processor 250 may recognize a place or a landmark by using a structure included in a scene or a natural landscape shown in the background. Alternatively, the processor 250 may perform recognition of a scene or a place of a scene through captions or dialogue.

In operation 513, the processor 250 may recognize a scene-specific activity or event. For example, the processor 250 may perform situational awareness including various activities or events such as weddings, sports, games, and the like, based on the actions or appearances of the characters.

In operation 515, the processor 250 may determine whether the last frame is in the same scene. If it is not the last frame, the processor 250 branches to step 507 and may perform the following operation on another frame.

In operation 517, when the processor 250 is the last frame in the same scene, the processor 250 may collect the recognition results and map the keyword to the corresponding scene.

In operation 519, the processor 250 may determine whether it is the last frame in the corresponding image, and if it is not the last frame, the processor 250 may branch to a previous step, for example, step 507, and perform the following operation again. In the case of the last frame, scene recognition for the image clip 231 may be terminated.

Meanwhile, in the above description, the order of steps 507 to 513 may be changed. In addition, the scene recognition process performed in steps 507 to 513 may be performed for each frame included in the scene, or one frame may be allocated to one recognition process. Alternatively, one recognition process may be performed on the plurality of frames. Alternatively, after sampling a plurality of frames included in a scene, the above-described recognition processes of steps 507 to 517 may be applied to the sampled frames singly or in combination.

FIG. 6 is a diagram illustrating an example of a method of generating a keyword map associated with support of image content search according to an embodiment of the present invention.

Referring to FIG. 6, in relation to a keyword map generation method related to image content search support, in step 601, the processor 250 may acquire an image clip 231 in step 601. For example, an image clip 231 requiring keyword selection may be received from the external server device 300, or an image clip 231 stored in the memory 230 may be obtained.

In operation 603, the processor 250 may process a vector conversion of the original video file. In operation 605, the processor 250 may perform a video feature vector analysis. The video feature vector analysis method may be a method of collectively classifying image contents and approaching supervised learning.

In operation 607, the processor 250 may perform aggregate meta analysis. The aggregated meta analysis method may be a method of accessing by classifying and classifying a vector extracted for each frame of image content. In operation 609, the processor 250 may perform music recognition. Music recognition may recognize music applied to the entire image clip or at least one music applied to a specific scene. In operation 611, the processor 250 may process metabolic recognition. In the metabolic recognition process, the processor 250 may extract a noun or an adjective included in the metabolism. Recognition of the image clip 231 of the processor 250 may be performed in parallel without a separate order. For example, the processor 250 may perform video feature vector analysis, aggregated meta analysis, music analysis, metabolic recognition, etc. regardless of the order or in a different order than the above-described order.

In operation 602, the processor 250 processes the image transformation as described above with reference to FIG. 5, and in operation 604, the processor 250 may process face recognition. In operation 606, the processor 250 may perform situation / event recognition, and in operation 608, the processor 250 may perform place / attraction recognition. In operation 610, the processor 250 may perform metabolic / subtitle recognition. In operation 612, the processor 250 may determine whether it is the last frame in the corresponding image, and if it is not the last frame, the processor 250 branches to step 604 to perform the following operation again. The processor 250 may perform the above-described various recognition processes regardless of the order or in a different order than the above-described order. In step 612, the frame performing the current recognition is

In the case of the last frame, in step 613, the processor 250 assigns a score to the meta information extracted based on the scoring model or to candidate keywords included in the keyword candidate list, and performs a score comparison. Representative keywords can be extracted.

According to the present invention, by generating representative keyword information for a video clip, labor cost for calculating representative keyword information can be saved and efficient work can be performed. In addition, the present invention supports the keyword generation of the (video) video clip content through the image recognition technology in almost real time processing. In addition, the present invention, after extracting a multi-label of a specific image through an image recognizer, by combining the appearance frequency, recognition accuracy, etc. of each label (keyword) information, extract the representative keywords, or derive additional value such as genre analysis Can also be used. In addition, the present invention may facilitate collection of learning databases for specific keywords. The present invention can extract and utilize a learning database from image content (video clip or video clip) based on keyword information for each character, time / season, place, and situation. For example, when the present invention collects a video about a wedding scene and wants to learn a wedding situation, the present invention can obtain various tag information about an image clip and use it again to secure a learning DB of an image recognition engine.

The method according to the invention can be implemented in software form readable by various computer means and recorded on a computer readable recording medium. Here, the recording medium may include a program command, a data file, a data structure, etc. alone or in combination. The program instructions recorded on the recording medium may be those specially designed and constructed for the present invention, or may be known and available to those skilled in computer software. For example, the recording medium may be a magnetic medium such as a hard disk, a floppy disk, or a magnetic tape, an optical media such as a compact disk read only memory (CD-ROM), a digital video disk (DVD), or a floppy. Magnetic-Optical Media, such as floppy disks, and hardware devices specifically configured to store and execute program instructions, such as ROM, random access memory (RAM), flash memory, and the like. do. Examples of program instructions may include high-level language code that can be executed by a computer using an interpreter, as well as machine code such as produced by a compiler. Such hardware devices may be configured to operate as one or more software modules to perform the operations of the present invention, and vice versa.

In addition, the functional operations and subject matter implementations described herein may be implemented in other types of digital electronic circuitry, computer software, firmware, or hardware including the structures and structural equivalents disclosed herein, or It can be implemented in combination with one or more of. Implementations of the subject matter described herein relate to one or more computer program products, ie computer program instructions encoded on a program storage medium of tangible type for controlling or by the operation of an apparatus according to the invention. It can be implemented as the above module. The computer readable medium may be a machine readable storage device, a machine readable storage substrate, a memory device, a composition of materials affecting a machine readable propagated signal, or a combination of one or more thereof.

In addition, although the specification includes details of numerous specific implementations, these should not be construed as limited to any invention or scope of the claims, but rather to features that may be specific to a particular embodiment of a particular invention. It should be understood as an explanation. Certain features that are described in this specification in the context of separate embodiments may be implemented in combination in a single embodiment. Conversely, various features that are described in the context of a single embodiment can also be implemented in multiple embodiments individually or in any suitable subcombination. Furthermore, while the features operate in a specific combination and may be depicted as such initially claimed, one or more features from the claimed combination may in some cases be excluded from the combination, the claimed combination being a subcombination. Or a combination of subcombinations.

Likewise, although the operations are depicted in the drawings in a specific order, it should not be understood that such operations must be performed in the specific order or sequential order shown in order to obtain desirable results or that all illustrated operations must be performed. In certain cases, multitasking and parallel processing may be advantageous. In addition, the separation of the various system components of the above-described embodiments should not be understood as requiring such separation in all embodiments, and the described program components and systems will generally be integrated together into a single software product or packaged into multiple software products. It should be understood that it can.

Specific embodiments of the subject matter described in this specification have been described. Other embodiments are within the scope of the following claims. For example, the actions recited in the claims can be performed in a different order but still achieve desirable results. As an example, the process depicted in the accompanying drawings does not necessarily require that particular illustrated or sequential order to obtain the desired results. In certain implementations, multitasking and parallel processing may be advantageous.

The foregoing description presents the best mode of the invention, and provides examples to illustrate the invention and to enable those skilled in the art to make and use the invention. The specification thus produced is not intended to limit the invention to the specific terms presented. Thus, while the present invention has been described in detail with reference to the examples described above, those skilled in the art can make modifications, changes and variations to the examples without departing from the scope of the invention.

Therefore, the scope of the present invention should be determined by the claims rather than by the described embodiments.

FIELD OF THE INVENTION The present invention applies to the field of telecommunications and, in particular, relates to video content retrieval technology.

In particular, the present invention can provide an efficient keyword selection operation by selecting a representative keyword of a keyword that is highly related to the image clip by using a variety of recognition modules while selecting a representative keyword for a variety of video clips, the accuracy of image clip search Can increase.

10: Network environment
100: terminal
200: video service device
210: communication circuit
230: memory
250: processor
251: Scene Meta Extractor
253: Image Integration Meta Extractor
255: external data extractor
257: analyzer
259: keyword selection unit
251a: face analyzer
251b: Situation Analyzer
251c: OCR analyzer
300: external server device

Claims (8)

A memory for storing an image clip;
A processor operatively coupled to the memory,
The processor is
Obtaining an image clip stored in the memory,
Splitting the video clip into scene units;
Extracting a candidate keyword by performing at least one recognition operation on at least one divided scene of the video clip;
And a representative keyword for the video clip is selected based on the extracted at least one candidate keyword. The method of claim 1,
The at least one candidate keyword is
And image extracted from the graphic subtitles applied to the at least one scene. The method of claim 1,
The at least one candidate keyword is
And a feature vector extracted for each frame included in the at least one scene according to a complex application of an original video feature vector (global feature) of the video clip. The method of claim 1,
The at least one candidate keyword is
Video service device, characterized in that priority is assigned through the score model. The method of claim 4, wherein
The processor is
And a highest priority is selected as a representative keyword for the video clip. The method of claim 1,
The processor is
And at least one of a face recognition, a time or season recognition, a place recognition, a dialogue recognition, a music recognition, a situation or an event recognition. Video service device,
Obtaining an image clip requiring keyword selection;
Dividing the video clip into scene units;
Extracting at least one candidate keyword by performing at least one recognition operation on at least one divided scene of the video clip;
And selecting a representative keyword for the video clip based on the at least one candidate keyword. The method of claim 7, wherein
The video clip
The video content retrieval method of claim 1, wherein the at least one scene of the original video is edited to include video content having a reproduction time length of several minutes to several tens of minutes.

Images