CN105653569A - Image data expansion processing method and apparatus - Google Patents

Abstract

The invention discloses an image data expansion processing method and apparatus. The method comprises the following steps of receiving a data processing request for to-be-processed image data, determining whether a data type for the to-be-processed image data is built, executing a data type building step if the data type is not built, matching the data type and executing a stored data processing process if the data type is built and the stored data processing process can be reused, with data processing resource stored, customizing a data processing data corresponding to the to-be-processed image data based on the stored data processing resources when the data type is built but the stored data processing process cannot be reused, and matching the data type and executing the customized data processing process. With abstractive remote-sensing image types and effective rule collection, a DB dynamic management model is built and a work flow method capable of being expanded can be built; and expansion processing for massive image data can be achieved.

Description

Image data extension processing method and device

technical field

The invention relates to the technical field of remote sensing satellite data processing, in particular to an SSI (Struts Spring Ibatis)-based image data expansion processing method and device.

Background technique

In recent years, remote sensing satellite data has become a hot topic in the fields of national security, foreign policy, and scientific research on global environmental change. With the continuous acceleration of the research and development of my country's earth observation series satellites and the launch of the national high-resolution earth observation system project, the remote sensing satellite application industry has become a national strategic high-tech industry, and has transformed from a test application type to a business service type , Satellite applications have become an important support for my country's economic construction, social development and government decision-making. Over the past ten years, my country has built a series of earth observation satellites such as the China-Pakistan Earth Resources Satellite, the Environmental Disaster Reduction Satellite, the Surveying and Mapping Satellite, and the High-Resolution Satellite. Satellite data resources form a satellite remote sensing service and application system.

At present, due to the limited image resources of a single satellite, and the satellite data will be affected by the environment such as weather, remote sensing applications usually involve the comprehensive application of multi-satellite and multi-load data. Taking remote sensing flood emergency monitoring as an example, the satellite data involved not only includes high-score data, but also may include data from resource satellites, environmental disaster reduction satellites, foreign satellites, military satellites, and aerial images. Sensor data that may be used include visible light , radar and infrared. For the research of multi-source remote sensing data information service system, the recognition and extraction of the remote sensing data obtained by each satellite in the multi-source satellite remote sensing data and the corresponding metadata information is an extremely important task. The key to database construction in the data information service system is also the core part to further distinguish the quality of the service system and whether it can provide users with fast and convenient services.

The processing of remote sensing image data has a high degree of complexity, mainly including: a wide variety of data, inconsistent formats, and the characteristics of the same data type often change, which leads to rapid changes in the law of demand. With the rapid expansion of my country's earth observation information, the problem of how to effectively manage, use, and share these heterogeneous, multi-source, and massive data has become increasingly prominent. The above problems are concentrated in the image data management layer. The prominent problem is that each system needs to develop a data management model and data storage and storage process separately, so as to adapt to the input data format and meet the data processing requirements of the ground processing system. A large amount of research and development work is concentrated in the In terms of the adjustment of the underlying data model, and the scalability of the data management category of the system is poor, adding a new data format requires code-level modification by the R&D personnel, and the workload is heavy.

Contents of the invention

In view of this, in order to overcome at least one of the above disadvantages, and provide at least one of the following advantages. Aiming at the integrated and automated management requirements of heterogeneous, multi-source, and massive remote sensing image data in the ground application system, an extended processing method and device for image data based on SSI is proposed. Collection, establish a DB dynamic management model and a workflow method that can be expanded and configured to realize the extended processing of massive image data. Specifically, the present invention adopts the following technical solutions:

An extended processing method for image data, comprising the following steps:

A data processing request carrying image data to be processed is received;

Judging whether the data category of the image data to be processed has been created, if it is judged otherwise, execute the step of creating the data category;

If it is judged to be yes and it is judged to reuse the stored data processing flow, then match the data category and execute the stored data processing flow;

If it is determined to be yes and it is determined not to reuse the stored data processing flow, then in the case of stored data processing resources, customize the data processing flow corresponding to the image data to be processed based on the stored data processing resources, matching The data category and execute the customized data processing flow.

The extended processing method for image data as described above, in a possible implementation manner, further includes:

If it is determined that the data category of the image data to be processed has been created, and it is determined that the stored data processing flow is not to be reused, then if no data processing resource is stored, add the data corresponding to the image data to be processed The data processing resource customizes a data processing flow corresponding to the image data to be processed based on the added data processing resource, matches the data category and executes the customized data processing flow.

In the extended processing method for image data as described above, in a possible implementation, the step of creating the data category includes: creating a data category code, creating data structure information, creating a data index analysis template, and creating a data index Validation rules.

In the extended processing method for image data described above, in a possible implementation, the step of creating the data category includes: creating a data category code, creating data structure information, importing a data index analysis template, and data index verification rule.

In the extended processing method of image data as described above, in a possible implementation, the data index includes a basic index and an extended index, and the basic index includes satellite code, sensor code, product level code, longitude and latitude coordinate information Any one or more of , the extended index is attribute information of the various image data to be processed.

In a possible implementation of the above-mentioned extended processing method for image data, the data processing flow further includes: a step of analyzing data index information, a step of extracting data browsing map, a step of recording data index information, and creating a data storage directory Steps, data import corresponding directory steps, data processing integrity check steps.

In the extended processing method for image data as described above, in a possible implementation manner, the data index information parsing step includes processing the to-be The image data is processed and analyzed to obtain the format of the data, and based on the data index verification rule, it is verified whether the format of the parsed data conforms to the defined rules.

In a possible implementation of the above-mentioned extended processing method for image data, the data categories include small data types, standard image types, 1-to-N associated image types, and N-associated product image types classified according to logical characteristics ; and standard formats and independent file formats classified according to data format characteristics.

An extended processing device for image data, comprising:

A receiving module, configured to receive a data processing request carrying image data to be processed;

A judging module, configured to judge whether the data category of the image data to be processed has been created;

A data category creation module, configured to create the data category when the judging module judges as no;

a matching module, configured to match the data category;

The flow customization module is used to customize the data corresponding to the stored data processing resources based on the stored data processing resources when the judgment module judges to be yes and the stored data processing flow is not to be reused. the data processing flow of the image data to be processed; and

An executing module, configured to execute the stored or customized data processing flow after the matching module matches the data category.

The extended processing device for image data as described above, in a possible implementation manner, further includes:

A resource adding module, configured to determine that the data category of the image data to be processed has been created by the judging module, and judge that the stored data processing flow is not to be reused, if no data processing resources are stored, adding data processing resources corresponding to the image data to be processed;

The customization module is further configured to customize the data processing flow corresponding to the image data to be processed based on the data processing resources added by the resource adding module _.

By adopting the above-mentioned technical solution, the beneficial effects achieved by the present invention are: the core framework based on the method provided by the present invention is SSI, which is used to realize the extended definition of data categories and the fully automatic management of image data in storage. The SSI framework adopts a semi-automatic ORM mechanism, allowing direct embedding of complex database SQL operations and control logic, compatible with complex database processing logic, and more suitable for massive remote sensing image management needs with complex historical data, large data processing volume, and strict performance requirements . Based on SSI technology, the data category definition at the human-computer interaction business level and the loosely coupled connection method between the data management business model and the underlying database service model are realized, which avoids the problems of hard-coded data model management and poor scalability in traditional image management systems, and can provide The data category defines an interactive service mode to realize the dynamic expansion capability of the data management system category. Moreover, the method realizes data archiving, extracts specific process definitions, and realizes the ability to analyze and expand data in special data formats such as HDF and NC through the expansion of the workflow.

Description of drawings

In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following will briefly introduce the accompanying drawings that need to be used in the description of the embodiments of the present invention. Obviously, the accompanying drawings in the following description are only some embodiments of the present invention , for those skilled in the art, other drawings can also be obtained according to the content of the embodiment of the present invention and these drawings without any creative effort.

FIG. 1 is a flow chart of a part of a method for processing image data provided by an embodiment of the present invention;

Fig. 2 is another part of the flow chart of the image data processing method provided by one embodiment of the present invention;

FIG. 3 is a schematic structural diagram of 1-to-N associated image type image data provided by an embodiment of the present invention;

FIG. 4 is a schematic structural diagram of an image data processing device provided by another embodiment of the present invention.

detailed description

In order to make the technical problems solved by the present invention, the technical solutions adopted and the technical effects achieved clearer, the technical solutions of the embodiments of the present invention will be further described in detail below in conjunction with the accompanying drawings. Obviously, the described embodiments are only the technical solutions of the present invention. Some, but not all, embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative efforts fall within the protection scope of the present invention.

The technical solutions of the present invention will be further described below in conjunction with the accompanying drawings and through specific implementation methods.

Based on the analysis of the above-mentioned background technology, it can be seen that the main difficulty in managing remote sensing image data lies in the variety of image data formats. The establishment of query indexes for image data in different formats basically requires customized development, and the system cannot dynamically expand the manageable image product categories. , if it must be expanded, professional engineers will need to pay a lot of work, modify the business logic, and the maintenance cost will be high. Based on the analysis of the characteristics and storage requirements of remote sensing image big data, the inventors have carried out research work on the storage model and logical structure design, standardized processing, and multivariate heterogeneous data integration of massive remote sensing images to achieve the above-mentioned problems. Scalable automatic storage and effective integration and sharing of massive image data. The extended processing method for image data proposed by the present invention will be described below with specific embodiments.

Example 1

As shown in Figure 1 and Figure 2, it is a flowchart of an image data processing method provided by an embodiment of the present invention, the method includes the following steps:

Step S11. A data processing request is received, and the request carries image data to be processed;

Step S12. According to the system prompt and the data category definition information stored in the system, it is judged whether the data category corresponding to the image data to be processed has been created in the system. If it has not been created, perform step S13; if it has been created, perform step S14;

Among them, data categories can be classified according to logical characteristics, divided into small data types, standard image types, 1-to-N associated image types, and N-associated product image types; they can also be classified according to data format characteristics, divided into standard formats and independent files Format.

Step S13, execute the step of creating data categories; specifically,

Determine whether to manually create a data category, if so, execute step S19, otherwise execute step S110;

Step S14, judging whether the stored data processing flow can be reused, if yes, execute step S21, otherwise execute S15;

Step S15, judging whether there is a data processing resource corresponding to the image data to be processed stored in the resource database, if there is a corresponding data processing resource stored, execute step S16, if there is no corresponding data processing resource stored, execute step S17;

In addition, the composition method, data format, file naming rules, meta-information extraction method, format, etc. of the image data are defined in the database to realize an extensible database service model.

Step S16, customize the data processing flow corresponding to the image data to be processed based on the stored data processing resources, and execute step S21;

Step S17, create (or add) a data processing resource corresponding to the image data to be processed, and customize a data processing flow corresponding to the image data to be processed based on the created resource, and execute step S21;

This method provides an extended configuration of data processing, and the existing processing flow of the multiplexing system can be selected in step S14, or new processing resources can be added in step S17 and the processing flow can be customized based on the newly added processing resources;

Step S21, judging whether the stored data category can be automatically matched, if yes, execute step S22, otherwise execute step S23;

Step S22, matching the data processing flow corresponding to the data category of the image data to be processed, and proceeding to step S24;

Step S23, prompting that it cannot be recognized, and ending the processing flow;

Step S24, execute the data processing flow;

The system will automatically form the background database definition language according to the set conversion rules, and automatically complete the creation and startup of the underlying database service. The process is simple and no special maintenance is required. For example, the above-mentioned model rules, ie, data index analysis and verification rules, are converted into template language in XML format, and XML template interpretation services are implemented, so as to realize automatic import and export services of image product model data.

Among them, the data processing processes stored in the system include preset processes and user-defined processes, and the system preset processes include archiving, extraction, and sharing. The user-defined process is set to select the existing process resources in the process resource management library, or add process resources that conform to the system interface rules. The system resource management library provides interfaces for adding, deleting and modifying all process resources in the system. Specifically, the data processing flow includes the steps of data index information analysis, data browsing map extraction, data index information recording, data storage directory creation, data import corresponding directory steps, and data processing integrity inspection steps. After completing these steps, the data processing process ends, and users can retrieve and browse data index information and download data in the system.

Step S18, create data category codes, create data structure information, create data index analysis templates, create data index verification rules, complete creation of remote sensing data categories, and execute step S14;

Step S19: Create data category codes, create data structure information, import data category definition templates, that is, import data index analysis templates and data index verification rules, complete creation of remote sensing data categories, and execute step S14.

Among them, the data category code is a predefined data category identifier, which can be represented by any combination of numbers, letters, and symbols; the data structure information includes information such as data file composition, image file naming rules, and the number of files contained in the data; The index parsing template is used to parse the image data to be processed, and the format of the data is parsed according to the defined metadata format and parsing rules; the data index verification rule is used to verify whether the format of the data conforms to the defined rules. Data indexes include basic indexes and extended indexes. The basic data index is the main index that can uniquely determine a piece of image data during the image data retrieval process, and generally contains information such as satellite code, sensor code, product level code, longitude and latitude coordinate information, etc. The extended index is the unique attribute information of various image data, such as the features and target names contained in the image data. The system needs to pre-organize and abstractly define the definition domain and value domain of commonly used indexes of image data, including name, type, uniqueness, value domain, etc., so as to form a management model for data index analysis and verification rules.

Among the above data types, the small data type represents image data with a simple data format and fewer index elements. For example, common GPS files, forecast orbit root files, etc., generally only need to establish an index including time, satellite code, and data code to ensure the unique calibration of the data.

The standard image type refers to standard product data such as environmental series satellites. This image data has no specific and necessary relationship with other image data except for its own remote sensing application characteristics, and the product data format is regular. Although the data length varies, the management logic Simpler.

An image of the 1-to-N associated image type generally includes a set of strip overview data, a set of overview data containing N groups of logical scenes, and a set of standardized strip images. The 1-to-N association relationship between strips and logical scenes is shown in Figure 3 As shown, a region in the logical scene strip image generally distinguishes the division range of each logical scene by defining its corner coordinates. Due to the wide land observation range of large strip images, industry users are often only interested in a small part of the entire strip. The use of logical scenes can reduce the amount of data processing and improve the speed of application processing and data transmission. Therefore, associations need to be established between the stripes and the corresponding N groups of logical scenes. It should be noted that the logical scene itself is the index of the strip image, and does not need to be processed as an independent image product. However, in industrial applications, the logical scene index is often used as the strip image extraction condition to obtain a part of the image data in the large strip.

N-associated product image type means that there is an inevitable relationship between N groups of products, but each group of products is also a standard image type, and a combined archiving and extraction method needs to be provided externally, as well as an independent archiving and extraction for each product. method.

In addition, according to the file structure of the image data, the image data can also be divided into a file collection structure and an independent file structure, which are used for data structure definition.

For image products with a file collection structure, each product is composed of multiple files. These files are all necessary and will serve in remote sensing processing and application sharing. For example: metadata index files are used for image retrieval, auxiliary data files are used for For application processing, the browser map file is used for image browsing, and the data entity is the object of application processing. For an independent file structure, that is, a product contains only one file, typical file standards such as HDF5, NC, TIFF, etc.

In the method provided by the present invention, the image product management method definition can be implemented by importing configuration files, or a wizard can be adopted, and this module guides users to define image product management methods. The wizard definition process allows users to create new meta information Elements, and the system automatically completes type matching and value range selection according to user definitions.

Example 2

As shown in FIG. 4 , it is a schematic structural diagram of an image data processing device provided by an embodiment of the present invention. The image data processing device 100 includes: a receiving module 10, a judging module 20, a data category creating module 30, a matching module 40, Process customization module 50 and execution module 60 .

Wherein, the receiving module 10 is used to receive the data processing request that carries the image data to be processed; the judging module 20 is used to judge whether the data category of the image data to be processed has been created; In the case of creating the data category; the matching module 40 is used for judging by the judging module 20 and judging that the stored data processing flow is reused, or the data processing flow corresponding to the image data to be processed has been customized Next, match the data category; the process customization module 50 is used to determine that the judgment module 20 is yes and judges not to reuse the stored data processing flow, then in the case of storing data processing resources, based on the stored data processing resources Customize the data processing flow corresponding to the image data to be processed; the execution module 60 is used to execute the stored or customized data processing flow after the matching module 40 matches the data category.

The image data processing device 100 above also includes: a resource adding module 70, used for judging in the judging module 20 that the data category of the image data to be processed has been created, and judging that the stored data processing flow is not to be reused; In the case of data processing resources, add data processing resources corresponding to the image data to be processed; then the flow customization module 50 is also used to customize the data processing flow corresponding to the image data to be processed based on the data processing resources added by the resource adding module 70 .

The beneficial effects of the present invention mainly include the following two aspects:

(1) Solve the problem of automatic extended storage of remote sensing image big data in common file formats. Due to the characteristics of remote sensing image big data, such as various types, large quantities, large storage capacity, diversified storage formats, and scattered metadata, it is difficult to organize and manage these data in a unified manner, access them efficiently, and share and exchange them. Bottlenecks for shared utilization. Based on SSI, a loosely coupled dynamic database expansion management model is built to provide interactive configuration management services, thus solving the problem of automatic expansion storage of remote sensing image big data.

(2) Solved the problem of automatic extended storage of images in special scientific data formats. Scientific data and space weather scientific data have many types and complex structures, such as HDF and CDF formats. There are problems such as common data access interfaces, large number of interfaces, and low data access granularity and abstraction level, which lead to strong customization of this type of data format. There are many restrictions on integrated applications, and data integration systems are often required for targeted and compatible research and development. The invention abstracts the logical structure and data access interface method of such data into a special data analysis model library, builds a prototype system based on scalable workflow services, and solves the problem of automatic extended storage of images in special scientific data formats.

All or part of the technical solutions provided by the above embodiments can be realized by software programming, and the software program is stored in a readable storage medium, such as a hard disk, an optical disk or a floppy disk in a computer.

Note that the above are only preferred embodiments of the present invention and applied technical principles. Those skilled in the art will understand that the present invention is not limited to the specific embodiments described herein, and that various obvious changes, readjustments and substitutions can be made by those skilled in the art without departing from the protection scope of the present invention. Therefore, although the present invention has been described in detail through the above embodiments, the present invention is not limited to the above embodiments, and can also include more other equivalent embodiments without departing from the concept of the present invention, and the present invention The scope is determined by the scope of the appended claims.

Claims (10)

1. An extended processing method for image data, comprising the following steps: A data processing request carrying image data to be processed is received; Judging whether the data category of the image data to be processed has been created, if it is judged otherwise, execute the step of creating the data category; If it is judged to be yes and it is judged to reuse the stored data processing flow, then match the data category and execute the stored data processing flow; If it is determined to be yes and it is determined not to reuse the stored data processing flow, then in the case of stored data processing resources, customize the data processing flow corresponding to the image data to be processed based on the stored data processing resources, matching The data category and execute the customized data processing flow. 2. The extended processing method of image data as claimed in claim 1, further comprising: If it is determined that the data category of the image data to be processed has been created, and it is determined that the stored data processing flow is not to be reused, then if no data processing resource is stored, add the data corresponding to the image data to be processed The data processing resource customizes a data processing flow corresponding to the image data to be processed based on the added data processing resource, matches the data category and executes the customized data processing flow. 3. The extended processing method of image data according to claim 1, wherein the step of creating the data category comprises: creating a data category code, creating data structure information, creating a data index parsing template and creating a data index Validation rules. 4. The extended processing method of image data according to claim 1, wherein the step of creating the data category comprises: creating a data category code, creating data structure information, importing a data index analysis template and data index verification rule. 5. The extended processing method of image data as claimed in claim 3 or 4, wherein the data index includes a basic index and an extended index, and the basic index includes a satellite code, a sensor code, a product level code, latitude and longitude coordinates Any one or more of the information, the extended index is the attribute information of the various image data to be processed. 6. The extended processing method of image data as claimed in claim 3 or 4, characterized in that, the data processing flow also includes: a data index information analysis step, a data browsing map extraction step, a data index information recording step, and a data storage Directory creation steps, data import corresponding directory steps, and data processing integrity inspection steps. 7. The extended processing method of image data as claimed in claim 6, wherein said data index information parsing step includes analyzing the format and parsing rules of the meta information defined in the data index parsing template for The image data is processed and analyzed to obtain the format of the data, and based on the data index verification rule, it is verified whether the format of the analyzed data conforms to the defined rules. 8. The extended processing method of image data according to claim 1, wherein the data categories include small data types, standard image types, 1-to-N associated image types, and N-associated product image types classified according to logical characteristics ; and standard formats and independent file formats classified according to data format characteristics. 9. An extended processing device for image data, comprising: A receiving module, configured to receive a data processing request carrying image data to be processed; A judging module, configured to judge whether the data category of the image data to be processed has been created; A data category creation module, configured to create the data category when the judging module judges as no; a matching module, configured to match the data category; A flow customization module, configured to customize the data processing flow corresponding to the stored data processing resource based on the stored data processing resource when the judging module judges to be yes and the stored data processing flow is not to be reused. the data processing flow of the image data to be processed; and An executing module, configured to execute the stored or customized data processing flow after the matching module matches the data category. 10. The image data extension processing device according to claim 9, further comprising: The resource adding module is used for determining that the data category of the image data to be processed has been created by the judging module, and judging that the stored data processing flow is not to be reused, if there is no data processing resource stored, adding data processing resources corresponding to the image data to be processed; The process customization module is further configured to customize the data processing process corresponding to the image data to be processed based on the data processing resources added by the resource adding module.