CN116129286A - Method for classifying graphic neural network remote sensing images based on knowledge graph - Google Patents
Method for classifying graphic neural network remote sensing images based on knowledge graph Download PDFInfo
- Publication number
- CN116129286A CN116129286A CN202310143228.5A CN202310143228A CN116129286A CN 116129286 A CN116129286 A CN 116129286A CN 202310143228 A CN202310143228 A CN 202310143228A CN 116129286 A CN116129286 A CN 116129286A
- Authority
- CN
- China
- Prior art keywords
- knowledge
- graph
- remote sensing
- neural network
- knowledge graph
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V20/00—Scenes; Scene-specific elements
- G06V20/10—Terrestrial scenes
- G06V20/13—Satellite images
-
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/30—Information retrieval; Database structures therefor; File system structures therefor of unstructured textual data
- G06F16/36—Creation of semantic tools, e.g. ontology or thesauri
- G06F16/367—Ontology
-
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
-
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/08—Learning methods
-
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N5/00—Computing arrangements using knowledge-based models
- G06N5/02—Knowledge representation; Symbolic representation
- G06N5/027—Frames
-
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N5/00—Computing arrangements using knowledge-based models
- G06N5/04—Inference or reasoning models
- G06N5/041—Abduction
-
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/40—Extraction of image or video features
- G06V10/46—Descriptors for shape, contour or point-related descriptors, e.g. scale invariant feature transform [SIFT] or bags of words [BoW]; Salient regional features
- G06V10/462—Salient features, e.g. scale invariant feature transforms [SIFT]
-
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/40—Extraction of image or video features
- G06V10/46—Descriptors for shape, contour or point-related descriptors, e.g. scale invariant feature transform [SIFT] or bags of words [BoW]; Salient regional features
- G06V10/462—Salient features, e.g. scale invariant feature transforms [SIFT]
- G06V10/464—Salient features, e.g. scale invariant feature transforms [SIFT] using a plurality of salient features, e.g. bag-of-words [BoW] representations
-
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/40—Extraction of image or video features
- G06V10/46—Descriptors for shape, contour or point-related descriptors, e.g. scale invariant feature transform [SIFT] or bags of words [BoW]; Salient regional features
- G06V10/467—Encoded features or binary features, e.g. local binary patterns [LBP]
-
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
- G06V10/762—Arrangements for image or video recognition or understanding using pattern recognition or machine learning using clustering, e.g. of similar faces in social networks
- G06V10/763—Non-hierarchical techniques, e.g. based on statistics of modelling distributions
-
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
- G06V10/764—Arrangements for image or video recognition or understanding using pattern recognition or machine learning using classification, e.g. of video objects
- G06V10/765—Arrangements for image or video recognition or understanding using pattern recognition or machine learning using classification, e.g. of video objects using rules for classification or partitioning the feature space
-
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
- G06V10/82—Arrangements for image or video recognition or understanding using pattern recognition or machine learning using neural networks
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Evolutionary Computation (AREA)
- Software Systems (AREA)
- Computing Systems (AREA)
- Artificial Intelligence (AREA)
- Multimedia (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Computational Linguistics (AREA)
- General Health & Medical Sciences (AREA)
- General Engineering & Computer Science (AREA)
- Data Mining & Analysis (AREA)
- Health & Medical Sciences (AREA)
- Databases & Information Systems (AREA)
- Mathematical Physics (AREA)
- Life Sciences & Earth Sciences (AREA)
- Medical Informatics (AREA)
- Biomedical Technology (AREA)
- Biophysics (AREA)
- Molecular Biology (AREA)
- Probability & Statistics with Applications (AREA)
- Animal Behavior & Ethology (AREA)
- Remote Sensing (AREA)
- Astronomy & Astrophysics (AREA)
- Image Analysis (AREA)
Abstract
The invention provides a method for classifying remote sensing images of a graph neural network based on a knowledge graph, which comprises the following steps: acquiring remote sensing samples under different scenes, and collecting remote sensing knowledge related to a remote sensing sample set; constructing a knowledge graph of the remote sensing sample based on remote sensing knowledge to obtain a first knowledge graph; constructing a graph neural network model, inputting a first knowledge graph into the graph neural network model for learning, perfecting and fusing the first knowledge graph, and obtaining a second knowledge graph; performing iterative training on the graph neural network model by using the remote sensing sample and the second knowledge graph, and optimizing the graph neural network model; and acquiring the remote sensing images to be classified, and classifying the remote sensing images to be classified according to the optimized graph neural network model to obtain a classification result. According to the invention, the knowledge graph and the graph neural network are combined and mutually optimized, so that the classification accuracy of the graph neural network model can be greatly improved.
Description
Technical Field
The invention relates to a remote sensing image classification method, in particular to a map neural network remote sensing image classification method based on a knowledge graph, and belongs to the field of remote sensing image classification.
Background
Remote sensing services are an important ring of geospatial information services, and providing information services is an ultimate goal. How to realize only search, flow modeling and task allocation; how to interoperate the remote sensing information service with other spatial disciplines under unified standards; how to construct a service-driven semantic model so that a computer can understand remote sensing data, information and requirements semantically and provide more intelligent remote sensing services is a problem faced by space information services. With the development of artificial intelligence, knowledge maps are attracting attention in both academia and industry. At present, the typical universal knowledge patterns compared abroad are as follows: DBpedia, YAGO, wikiData, and the like, a typical general knowledge map for domestic comparison is: CN-DBpedia, zhishi. Me, techKG, etc. Compared with the general knowledge graph, the domain knowledge graph is oriented to a specific domain, and can conduct more finely divided knowledge reasoning, auxiliary analysis and decision. But the domain knowledge graph has higher requirements on the specialty and the accuracy. Although research in the remote sensing field introduces standard knowledge or forms for constructing a knowledge graph to assist analysis and decision, the scale of the graph and the coverage of the field are smaller, and a unified theoretical framework similar to the knowledge graph in the field of geography and a large remote sensing knowledge graph are not formed yet.
A graph is an abstract data structure representing the association between objects, comprising a set of nodes and a set of edges, and their attribute features. Large scale graph data can express rich relationships. The graph shows learning, and data of different sources and different types can be fused into one graph for analysis, so that a result which is difficult to find when the isolated data are analyzed is obtained. Artificial intelligence has made an important breakthrough in many tasks such as image classification, video processing, speech recognition, etc. During processing of these tasks, usually European spatial data is processed, and non-European spatial data is emerging in more and more applications. To better analyze non-European data, graph representation learning is a research hotspot in the current artificial intelligence field and is widely used in industry.
Graph representation learning is a method for efficiently mining the potential value of graph data, and graph neural networks are one of the most effective tools for achieving graph representation learning. A graphic neural network is made up of a plurality of structural components. Each layer of the graphic neural network comprises structural components such as an attention mechanism function, a node aggregation function, an activation function and the like. The objective of the graph neural network modeling is to select proper component values in each structural component to form a high-efficiency graph neural network structure, and simultaneously train on graph data and acquire model parameters. However, the graphic neural network has not been well applied in the remote sensing field at present.
Disclosure of Invention
Based on the problems of the lack of the remote sensing knowledge graph and insufficient application of the graph neural network, the invention provides a remote sensing image classification method based on the combination of the knowledge graph and the graph neural network.
The invention provides a method for classifying remote sensing images of a graph neural network based on a knowledge graph, which comprises the following steps:
s1, acquiring remote sensing samples under different scenes, and collecting remote sensing knowledge related to the remote sensing samples;
s2, constructing a knowledge graph of the remote sensing sample based on remote sensing knowledge to obtain a first knowledge graph;
s3, constructing a graph neural network model, inputting the first knowledge graph into the graph neural network model for learning, perfecting and fusing the first knowledge graph, and obtaining a second knowledge graph;
s4, performing iterative training on the graph neural network model by using the remote sensing sample and the second knowledge graph, and optimizing the graph neural network model;
s5, acquiring the remote sensing images to be classified, and classifying the remote sensing images to be classified according to the optimized graph neural network model to obtain a classification result.
In an embodiment of the present invention, step S1 includes:
acquiring a plurality of remote sensing samples, and classifying the remote sensing samples according to a visual word bag method to obtain remote sensing samples in different scenes;
remote sensing knowledge related to a plurality of remote sensing samples is collected, wherein the remote sensing knowledge includes text knowledge and image knowledge.
In an embodiment of the present invention, step S2 includes:
performing mode design according to the characteristics of remote sensing sample data of different scenes to obtain a remote sensing body;
carrying out knowledge extraction on the remote sensing knowledge, including text knowledge extraction and image knowledge extraction, wherein the text knowledge extraction is to establish a mapping relation between entities and remote sensing ontology, identify the entities and establish a relation between the entities, and the image knowledge extraction is to structure the entities and the relation to obtain a plurality of triples;
and connecting the triples to form a first knowledge graph.
In one embodiment of the present invention, the remote sensing ontology includes classes, subclasses, attributes, and attribute constraints.
In an embodiment of the present invention, step S3 includes:
constructing a graph neural network model, wherein the graph neural network model comprises graph structure data, and the graph structure data comprises characterization vectors of nodes and characterization vectors of edges, wherein the edges represent association relations between the connected nodes;
inputting the first knowledge graph into a graph neural network model, aligning the entities of the first knowledge graph based on a conditional random field to fuse the first knowledge graph, and utilizing graph structure data to infer and predict the fused first knowledge graph to perfect graph information so as to obtain a second knowledge graph.
In one embodiment of the invention, the graph neural network model comprises a graph convolutional network, a graph annotation network, a graph self-encoder and a graph generation network model.
In an embodiment of the present invention, the process of fusing the first knowledge-graph is:
combining isomorphic entity pairs in the plurality of first knowledge maps to obtain a combined entity set;
and (3) carrying out multi-directional linking on the combined entity set by using the conditional random field, and fusing a plurality of first knowledge maps.
In an embodiment of the present invention, the process of reasoning and predicting the fused first knowledge-graph includes:
the process for carrying out knowledge reasoning on the fused first knowledge graph comprises the following steps:
mapping the entities and the relations thereof in the fused first knowledge graph to a low-dimensional continuous vector space, and updating the entities and the relations thereof by using a graph neural network model to obtain an updated low-dimensional vector representation;
when updating the low-dimensional vector representation, automatically capturing and reasoning the required characteristics, so that the fused first knowledge graph automatically realizes reasoning in a low-dimensional continuous vector space;
the process of predicting the link relation between the entities comprises the following steps:
an automatic encoder is introduced, comprising an entity encoder that maps each entity to a true value vector, and a decoder that reconstructs the relationship between the entities from the vector representations of the entities, thereby predicting the linkage relationship between the entities.
In an embodiment of the present invention, the process of reasoning and predicting the fused first knowledge graph includes:
carrying out knowledge reasoning on the fused first knowledge graph, and deducing to obtain a new relation between the entities;
and utilizing the graph neural network model to introduce information of adjacent entities and corresponding relations, and predicting the link relation between the entities.
The beneficial effects of the invention are as follows:
(1) According to the invention, the knowledge patterns are combined with the graph neural network, the knowledge patterns obtained in a plurality of different scenes are fused by utilizing the recognition and analysis capability of the graph neural network on the graph data, so that the learning capability of learning the knowledge patterns is greatly enhanced, and more accurate classification results can be obtained when the knowledge patterns are used for recommending and classifying.
(2) The invention also carries out reasoning and perfecting on the fused knowledge graph according to the characteristics of the graph neural network, so that knowledge of the scene remote sensing image with a large area can be covered by the knowledge graph, the generalization performance of the knowledge graph is improved, the knowledge graph has a continuous self-learning function, the content of the graph can be continuously optimized in the subsequent application, and the application prospect in the remote sensing field is very wide.
(3) According to the invention, the graphic neural network can be fed back by using the knowledge graph after the completion, the remote sensing samples are guided to train and optimize the graphic neural network model according to the triplet characteristics of the knowledge graph, and in the optimizing process, the knowledge graph can improve the generalization performance of the graphic neural network model, so that the graphic neural network model has good classifying effect on different remote sensing samples, the remote sensing samples can be better utilized, and the classifying precision of the graphic neural network model is improved.
Drawings
In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings that are needed in the embodiments will be briefly described below, and it is obvious that the drawings in the following description are only some embodiments of the present invention, and other drawings may be obtained according to these drawings without inventive effort for a person skilled in the art.
FIG. 1 is a flowchart of a remote sensing image classification method according to an embodiment of the present invention;
fig. 2 is a schematic diagram of a portion of a remote sensing entity according to an embodiment of the present invention.
Detailed Description
The following description of the embodiments of the present invention will be made clearly and completely with reference to the accompanying drawings, in which it is apparent that the embodiments described are only some embodiments of the present invention, but not all embodiments. It should be noted that, as long as no conflict is formed, each embodiment of the present invention and each feature of each embodiment may be combined with each other, and the formed technical solutions are all within the protection scope of the present invention.
It is noted that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of exemplary embodiments according to the present invention. As used herein, the singular is also intended to include the plural unless the context clearly indicates otherwise, and furthermore, it is to be understood that the terms "comprises" and/or "comprising" when used in this specification are taken to specify the presence of stated features, steps, operations, devices, components, and/or combinations thereof.
Referring to fig. 1, the invention provides a method for classifying remote sensing images of a graph neural network based on a knowledge graph, which comprises the following steps:
s1, acquiring remote sensing samples under different scenes, and collecting remote sensing knowledge related to the remote sensing samples;
s2, constructing a knowledge graph of the remote sensing sample based on remote sensing knowledge to obtain a first knowledge graph;
s3, constructing a graph neural network model, inputting the first knowledge graph into the graph neural network model for learning, perfecting and fusing the first knowledge graph, and obtaining a second knowledge graph;
s4, performing iterative training on the graph neural network model by using the remote sensing sample and the second knowledge graph, and optimizing the graph neural network model;
s5, acquiring the remote sensing images to be classified, and classifying the remote sensing images to be classified according to the optimized graph neural network model to obtain a classification result.
The technical process of the invention is as follows: firstly, acquiring remote sensing samples under different scenes, and collecting remote sensing knowledge related to a remote sensing sample set; then constructing a knowledge graph of the remote sensing sample based on remote sensing knowledge to obtain a first knowledge graph; constructing a graph neural network model, inputting a first knowledge graph into the graph neural network model for learning, perfecting and fusing the first knowledge graph, and obtaining a second knowledge graph; then, carrying out iterative training on the graph neural network model by using the remote sensing sample and the second knowledge graph, and optimizing the graph neural network model; and acquiring the remote sensing images to be classified, and classifying the remote sensing images to be classified according to the optimized graph neural network model to obtain a classification result.
Firstly, a plurality of remote sensing samples are obtained from a sample library, the remote sensing samples are provided with tag information, and the remote sensing samples under different scenes which are distinguished according to the scenes can be directly obtained; or firstly obtaining remote sensing samples which are not subjected to scene distinction, and then carrying out scene classification on the remote sensing samples according to a visual word bag method to obtain remote sensing samples in different scenes: (1) Performing feature extraction on the remote sensing sample by using a SIFT algorithm to obtain SIFT features; (2) Clustering SIFT features by using a K-Means algorithm, taking each clustering center as a visual word, and taking the values of the K clustering centers and the corresponding visual word numbers as a visual vocabulary; (3) Performing visual word mapping on the remote sensing sample to generate a visual word distribution map; (4) Performing LBP conversion on the visual word distribution map to obtain LBP histogram representation of the visual word distribution map; (5) And inputting the LBP histogram representation of the visual word distribution diagram and the visual word number into an SVM classifier for training and classifying to obtain remote sensing samples under different scenes.
Remote sensing knowledge related to a plurality of remote sensing samples is collected, wherein the remote sensing knowledge includes text knowledge and image knowledge. The text knowledge is the information for carrying out text description on the remote sensing knowledge, and comprises the material weather information, the ground surface coverage ground material, the image recording information, the environment recording information and the like; the image knowledge is the characteristics of the remote sensing image, including the spatial relationship such as topology, direction, distance and the like among various objects in the remote sensing image, the texture characteristics of the remote sensing image, the spatial spectrum characteristics of the remote sensing image and the like.
After the remote sensing knowledge is collected, the first knowledge graph is constructed based on the remote sensing knowledge, and the specific process is as follows:
(1) And performing mode design according to the characteristics of remote sensing sample data of different scenes to obtain a map frame. The pattern design is also an ontology design, and the ontology abstracts real world objects into concepts and explicitly and normalized descriptions the concepts through attributes and attribute constraints. The ontology is composed of classes, subclasses, attributes and attribute constraints. A class is a conceptual abstraction for a specific field, referring to fig. 2, in an example of a part of the remote sensing ontology, the remote sensing image is all available images, and the high resolution remote sensing image is a sub-class of the remote sensing image. The attribute is description of the class, and can expand the class and restrict the constructed knowledge graph. The ontology carries out structural organization on knowledge and data, and gives semantic relevance to the data. The relationship of the remote sensing body is divided into semantic relationship, spatial relationship and time relationship. The construction of the remote sensing ontology follows a course from coarse to fine. According to the characteristics of the image data of the remote sensing samples in different scenes, determining the field and the range of the knowledge graph, obtaining key concepts in the graph, and then expanding the concepts to complete the body concepts.
(2) And carrying out knowledge extraction on the remote sensing knowledge, including text knowledge extraction and image knowledge extraction, wherein the text knowledge extraction is to establish a mapping relation between the entity and the remote sensing ontology, identify the entity and establish a relation between the entities, and the image knowledge extraction is to structure the entity and the relation to obtain a plurality of triples. The text knowledge extraction can extract structural information from the text description so as to establish the mapping relation between the entity and the remote sensing body, and can identify important phrases and words in the field by using a vocabulary mining technology, and identify the entity and establish the specific relation between the entities by means of entity identification, entity classification, entity link and the like. The image knowledge is extracted into a combination of a top-down mode and a bottom-up mode, the relationship between entities in the remote sensing sample is established according to the label information of the remote sensing sample, an entity candidate frame is generated, object features including category features and position features are extracted according to the candidate frame, then a deep learning network which can be RNN, GNN or other deep learning networks is constructed, the object features are input into the deep learning network for relationship reasoning and prediction, the entities and the relationships can be structurally represented based on the relationship between the entities in the remote sensing sample and the related concepts in the remote sensing knowledge, triples of (entity 1, relationship and entity 2) are formed, and a series of triples under the same scene are used as a remote sensing scene graph.
(3) And taking the remote sensing scene graph as a sub-graph, fusing a plurality of sub-graphs in a graph fusion mode, specifically, cross-connecting the triples to form a first knowledge graph, wherein the number of the obtained first knowledge graphs is the same as the number of the scenes of the remote sensing sample.
The graph neural network model can be constructed as a graph convolution network, a graph annotation network, a graph self-encoder, a graph generation network and the like. The present embodiment is a relationship graph convolutional network model (R-GCN model). The R-GCN model comprises graph structure data, wherein the graph structure data comprises characterization vectors of nodes and characterization vectors of edges, and the edges represent association relations between the connected nodes;
inputting the first knowledge-graph into an R-GCN model, wherein the isomorphic sub-graph can be identified by graph structure data, and similar neighbors exist around entity pairs, namely, entity 3 and entity pairs (entity 1 and entity 2) have certain isomorphic characteristics, then the first knowledge-graph is identified by the R-GCN model to obtain isomorphic characteristics, the entities which describe the same target and are learned from a plurality of first knowledge-graphs are combined to obtain a combined entity set, and then the local and global information of the combined entity set are linked in multiple directions by adopting a conditional random field to complete fusion of the plurality of first knowledge-graphs.
It should be noted that, each first knowledge graph is a knowledge graph which primarily summarizes knowledge and content of different scenes in the remote sensing field, the capability of the graph neural network is utilized to fuse each first knowledge graph, so that the heterogeneous problem of the first knowledge graph can be solved, knowledge sharing can be performed on each first knowledge graph, the fused first knowledge graph is a knowledge graph, the knowledge graph contains knowledge content in the complicated and various remote sensing fields, the learning capability is high, and the classification accuracy can be improved when the remote sensing image classification is performed subsequently.
Carrying out knowledge reasoning on the fused first knowledge graph, deducing to obtain a new relation between the entities, introducing information of adjacent entities and corresponding relations by using a graph neural network model, and predicting the link relation between the entities, wherein the specific process is as follows:
(1) Learning low-dimensional vector representations
Mapping the entities and the relations in the fused first knowledge graph to a low-dimensional continuous vector space, and learning a low-dimensional vector representation for the entities and the relations, wherein the low-dimensional vector representation comprises semantic information. And meanwhile, the R-GCN model is applied to integrate topological structure information and attribute characteristic information in the atlas, so that when each entity performs low-dimensional vector representation learning, information in other entities related to the entity can be utilized, and thus the low-dimensional vector representation of more complete and richer entities and relations is obtained through learning.
(2) Reasoning based on learned low-dimensional vector representation
When learning the low-dimensional vector representation, automatically capturing and reasoning the required characteristics, and automatically realizing the reasoning of the fused first knowledge graph in a low-dimensional continuous vector space through training and learning.
(3) Link prediction based on learned low-dimensional vector representations
When the low-dimensional vector representation learning is carried out on the entities and the relations, the R-GCN model is utilized to introduce information of adjacent entities and corresponding relations for the entities, specifically, an automatic encoder is introduced, the automatic encoder comprises an entity encoder and a decoder, the entity encoder maps each entity to a true value vector, the decoder is a scoring function, and the decoder represents the sides of the reconstructed graph according to the nodes, namely, the relations among the reconstructed entities are represented according to the vectors of the entities. The steps can learn more comprehensive entity representation, so that the link relation between the entities is predicted, and the fused first knowledge graph is perfected to obtain a second knowledge graph.
It should be noted that, the graph neural network has excellent recognition and analysis capability on the data of the graph structure, when the knowledge graph is inferred and perfected, the graph neural network can exchange information with the knowledge graph to guide the knowledge graph to improve the learning capability of the user, and the knowledge graph can be continuously perfected when being applied later, the content and learning capability are continuous, and the application prospect of the graph neural network containing the knowledge graph in the remote sensing field is very wide.
Training the graph neural network model according to the remote sensing sample and the second knowledge graph until the graph neural network model converges, wherein the process is as follows:
the graph neural network model is an R-GCN model, and the R-GCN model is composed of a plurality of R-GCNs. Wherein the first R-GCN layer is an input layer, and the last R-GCN layer comprises a classifier. Firstly inputting a second knowledge graph into an R-GCN model and storing the second knowledge graph, inputting a remote sensing sample into the R-GCN model, carrying out knowledge extraction on the remote sensing sample by utilizing the second knowledge graph to obtain an entity of the remote sensing sample, introducing an automatic encoder comprising the encoder and the decoder, wherein the encoder is an R-GCN for generating implicit characteristic representation of the entity, the decoder is a scoring function, carrying out vector representation and relation type related weight matrix calculation output information on each entity by utilizing the automatic encoder, aggregating the output information and generating new entity representation, reconstructing the relation between the entities according to the vector representation of the entity, namely predicting the label of the entity, comparing the label of the predicted entity with the label of the corresponding entity in the second knowledge graph to adjust the parameters of the classifier, and iterating the steps until the R-GCN model converges to obtain the trained R-GCN model with the classifier.
And acquiring the remote sensing images to be classified, and inputting the remote sensing images to be classified into a trained graphic neural network model to obtain the classification result of the remote sensing images to be classified.
It should be noted that after the knowledge graph is completed, the graph neural network can be used for feeding back, in the process of training the graph neural network model, the knowledge graph can play a guiding role on the performance of the network, the content of the knowledge graph is rich, the utilization rate of different remote sensing samples can be improved, for example, remote sensing samples which are difficult to classify can be obtained, knowledge graph can learn and infer the remote sensing samples, the labels of the predictions can be given by combining the knowledge related to the samples and the labels of similar samples, and the graph neural network model can be adjusted according to the label results of the knowledge graph, so that the generalization performance of the graph neural network model can be improved, and the classification accuracy of the graph neural network model is improved.
The foregoing is merely illustrative of the present invention, and the present invention is not limited thereto, and any changes or substitutions easily contemplated by those skilled in the art within the scope of the present invention should be included in the present invention. Therefore, the protection scope of the invention is subject to the protection scope of the claims.
Claims (10)
1. The method for classifying the remote sensing images of the graph neural network based on the knowledge graph is characterized by comprising the following steps of:
s1, acquiring remote sensing samples under different scenes, and collecting remote sensing knowledge related to the remote sensing samples;
s2, constructing a knowledge graph of the remote sensing sample based on remote sensing knowledge to obtain a first knowledge graph;
s3, constructing a graph neural network model, inputting the first knowledge graph into the graph neural network model for learning, perfecting and fusing the first knowledge graph, and obtaining a second knowledge graph;
s4, performing iterative training on the graph neural network model by using the remote sensing sample and the second knowledge graph, and optimizing the graph neural network model;
s5, acquiring the remote sensing images to be classified, and classifying the remote sensing images to be classified according to the optimized graph neural network model to obtain a classification result.
2. The method for classifying remote sensing images of a graph neural network based on a knowledge graph according to claim 1, wherein the step S1 comprises:
acquiring a plurality of remote sensing samples, and classifying the remote sensing samples according to a visual word bag method to obtain remote sensing samples in different scenes;
remote sensing knowledge related to a plurality of remote sensing samples is collected, wherein the remote sensing knowledge includes text knowledge and image knowledge.
3. The method for classifying remote sensing images of a graph neural network based on a knowledge graph according to claim 2, wherein the step S2 comprises:
performing mode design according to the characteristics of remote sensing sample data of different scenes to obtain a remote sensing body;
carrying out knowledge extraction on the remote sensing knowledge, including text knowledge extraction and image knowledge extraction, wherein the text knowledge extraction is to establish a mapping relation between entities and remote sensing ontology, identify the entities and establish a relation between the entities, and the image knowledge extraction is to structure the entities and the relation to obtain a plurality of triples;
and connecting the triples to form a first knowledge graph.
4. The method for classifying the remote sensing images of the graphic neural network based on the knowledge graph according to claim 3, wherein the remote sensing ontology comprises classes, subclasses, attributes and attribute constraints.
5. The method for classifying a remote sensing image of a graph neural network based on a knowledge graph according to claim 3, wherein the step S3 comprises:
constructing a graph neural network model, wherein the graph neural network model comprises graph structure data, and the graph structure data comprises characterization vectors of nodes and characterization vectors of edges, wherein the edges represent association relations between the connected nodes;
inputting the first knowledge graph into a graph neural network model, aligning the entities of the first knowledge graph based on a conditional random field to fuse the first knowledge graph, and utilizing graph structure data to infer and predict the fused first knowledge graph to perfect graph information so as to obtain a second knowledge graph.
6. The knowledge-based graphic neural network remote sensing image classification method according to claim 5, wherein the graphic neural network model comprises a graphic convolution network, a graphic annotation network, a graphic self-encoder and a graphic generation network model.
7. The method for classifying the remote sensing images of the graph neural network based on the knowledge graph according to claim 5, wherein the process of fusing the first knowledge graph is as follows:
combining isomorphic entity pairs in the plurality of first knowledge maps to obtain a combined entity set;
and (3) carrying out multi-directional linking on the combined entity set by using the conditional random field, and fusing a plurality of first knowledge maps.
8. The method for classifying the remote sensing images of the graph neural network based on the knowledge graph according to claim 5, wherein the process of reasoning and predicting the fused first knowledge graph is as follows:
carrying out knowledge reasoning on the fused first knowledge graph, and deducing to obtain a new relation between the entities;
and utilizing the graph neural network model to introduce information of adjacent entities and corresponding relations, and predicting the link relation between the entities.
9. The method for classifying remote sensing images of a graph neural network of knowledge maps according to claim 8, wherein the process of performing knowledge reasoning on the fused first knowledge maps comprises:
mapping the entities and the relations thereof in the fused first knowledge graph to a low-dimensional continuous vector space, and updating the entities and the relations thereof by using a graph neural network model to obtain an updated low-dimensional vector representation;
when updating the low-dimensional vector representation, automatically capturing and reasoning the required characteristics, so that the fused first knowledge graph automatically realizes reasoning in a low-dimensional continuous vector space;
the process of predicting the link relation between the entities comprises the following steps:
an automatic encoder is introduced, comprising an entity encoder that maps each entity to a true value vector, and a decoder that reconstructs the relationship between the entities from the vector representations of the entities, thereby predicting the linkage relationship between the entities.
10. The method for classifying remote sensing images based on a graph neural network of claim 1, wherein the number of first knowledge-graphs is the same as the number of scenes.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310143228.5A CN116129286A (en) | 2023-02-10 | 2023-02-10 | Method for classifying graphic neural network remote sensing images based on knowledge graph |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310143228.5A CN116129286A (en) | 2023-02-10 | 2023-02-10 | Method for classifying graphic neural network remote sensing images based on knowledge graph |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN116129286A true CN116129286A (en) | 2023-05-16 |
Family
ID=86306217
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202310143228.5A Pending CN116129286A (en) | 2023-02-10 | 2023-02-10 | Method for classifying graphic neural network remote sensing images based on knowledge graph |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN116129286A (en) |
Cited By (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN116468960A (en) * | 2023-06-19 | 2023-07-21 | 南京朵盛信息技术有限公司 | Video image analysis and retrieval method and system |
| CN117392470A (en) * | 2023-12-11 | 2024-01-12 | 安徽中医药大学 | Fundus image multi-label classification model generation method and system based on knowledge graph |
| CN118036902A (en) * | 2024-04-11 | 2024-05-14 | 中国科学院自动化研究所 | Knowledge graph-based ocean typical scene evaluation index system construction method and device, electronic equipment and storage medium |
| CN120111079A (en) * | 2025-05-07 | 2025-06-06 | 宁波弘泰水利信息科技有限公司 | A reservoir intelligent inspection system based on knowledge graph and digital twin |
-
2023
- 2023-02-10 CN CN202310143228.5A patent/CN116129286A/en active Pending
Cited By (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN116468960A (en) * | 2023-06-19 | 2023-07-21 | 南京朵盛信息技术有限公司 | Video image analysis and retrieval method and system |
| CN116468960B (en) * | 2023-06-19 | 2023-08-25 | 南京朵盛信息技术有限公司 | Video image analysis and retrieval method and system |
| CN117392470A (en) * | 2023-12-11 | 2024-01-12 | 安徽中医药大学 | Fundus image multi-label classification model generation method and system based on knowledge graph |
| CN117392470B (en) * | 2023-12-11 | 2024-03-01 | 安徽中医药大学 | Fundus image multi-label classification model generation method and system based on knowledge graph |
| CN118036902A (en) * | 2024-04-11 | 2024-05-14 | 中国科学院自动化研究所 | Knowledge graph-based ocean typical scene evaluation index system construction method and device, electronic equipment and storage medium |
| CN120111079A (en) * | 2025-05-07 | 2025-06-06 | 宁波弘泰水利信息科技有限公司 | A reservoir intelligent inspection system based on knowledge graph and digital twin |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| Alzubaidi et al. | A survey on deep learning tools dealing with data scarcity: definitions, challenges, solutions, tips, and applications | |
| Xu et al. | Survey on multi-output learning | |
| Zheng | Methodologies for cross-domain data fusion: An overview | |
| Guo et al. | Supplier selection based on hierarchical potential support vector machine | |
| Huang et al. | Deep learning driven visual path prediction from a single image | |
| CN116129286A (en) | Method for classifying graphic neural network remote sensing images based on knowledge graph | |
| Yang et al. | Dynamic graph representation learning with neural networks: A survey | |
| CN107506793B (en) | Method and system for clothing recognition based on weakly labeled images | |
| CN110111340A (en) | The Weakly supervised example dividing method cut based on multichannel | |
| CN103544496B (en) | The robot scene recognition methods merged with temporal information based on space | |
| CN112819065B (en) | Unsupervised pedestrian sample mining method and unsupervised pedestrian sample mining system based on multi-clustering information | |
| Zhou et al. | Multi-label image classification via category prototype compositional learning | |
| CN110993037A (en) | Protein activity prediction device based on multi-view classification model | |
| Chen et al. | A review of few-shot and zero-shot learning for node classification in social networks | |
| Wu et al. | Transformer driven matching selection mechanism for multi-label image classification | |
| Zhang et al. | A survey of generative techniques for spatial-temporal data mining | |
| Patel et al. | A study on video semantics; overview, challenges, and applications | |
| Wang et al. | R2-trans: Fine-grained visual categorization with redundancy reduction | |
| Ma et al. | Semantic-aware dual contrastive learning for multi-label image classification | |
| Feng et al. | Spatial-temporal exclusive capsule network for open set action recognition | |
| CN111046965A (en) | A latent class discovery and classification method in multi-label classification | |
| CN114170625B (en) | A context-aware, noise-robust pedestrian search method | |
| Lu | Design of visual communication based on deep learning approaches: L. Lu | |
| Lu et al. | Editorial introduction to responsible artificial intelligence for autonomous driving | |
| Qi et al. | IMC-Det: Intra–Inter Modality Contrastive Learning for Video Object Detection |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |