CN116644202A - Method and device for storing large-data-volume remote sensing image data - Google Patents

Abstract

The present application relates to the field of big data, and in particular to a method and device for storing remote sensing image data with a large amount of data, which includes a method for storing remote sensing image data with a large amount of data, including: dividing the data into N segments; dividing the storage space into M nodes; set one of the M nodes as a configuration node, and the remaining M-1 nodes are set as working nodes for storing data; use the configuration node to store the data of N fragments to the corresponding storage data M‑1 worker nodes. It also includes: adding K working nodes; configuring nodes to allocate data storage of M+K‑1 nodes after adding working nodes. This application has the effect of improving efficiency in the storage process and ensuring the stability and reliability of the system.

Description

Storage method and device for large amount of remote sensing image data

technical field

The present application relates to the field of big data, in particular to a method and device for storing remote sensing image data with a large amount of data.

Background technique

At present, in the field of remote sensing spatial data storage, when the amount of data in the database reaches tens of millions or even hundreds of millions, data retrieval (attribute retrieval, spatial retrieval, spatial aggregation, etc.) will take a long time, resulting in slow system response and interface timeout. The traditional sub-database sub-table is a technology for horizontally partitioning data in a relational database. In this case, the main goal is to divide the large amount of spatial data into smaller data blocks, which are then allocated to different databases to improve the performance and scalability of the database. Under normal circumstances, when the amount of data reaches an acceptable range, it is necessary to divide the database and table. This technology is based on the concept of data dispersion, splitting a large database into multiple smaller databases, and then dispersing spatial data into these databases. Sub-database and sub-table will increase the number of components in the system, resulting in more management and configuration work, which will increase the difficulty of development, deployment and maintenance of the entire system. The target data may be scattered in various libraries. Spatial operations such as aggregation require multiple operations to obtain results, and the storage efficiency is not significantly improved.

Contents of the invention

In order to solve the technical problem that the efficiency improvement in the storage process is not obvious, the present application provides a method and device for storing remote sensing image data with a large amount of data.

A method for storing remote sensing image data with a large amount of data provided by this application adopts the following technical scheme:

In the first aspect, a method for storing remote sensing image data with a large amount of data is provided, including:

Dividing the large amount of remote sensing image data into N segments;

Divide the storage space into M nodes;

Setting one of the M nodes as a configuration node, and setting the remaining M-1 nodes as working nodes for storing large amounts of remote sensing image data;

The configuration nodes are used to store the data of the N fragments to the corresponding M-1 working nodes that store large amount of remote sensing image data.

Preferably, it also includes:

Add K working nodes;

Configure nodes to allocate data storage for M+K-1 nodes after adding working nodes.

Preferably, it also includes:

Copy P copies of the N fragments and distribute them to M-1 working nodes.

Preferably, said dividing the remote sensing image data with a large amount of data into N segments includes:

Create a distributed index;

According to the structure of the distributed index, the table is divided into multiple fragments;

Divide the data into N fragments and add them to the corresponding table divided into multiple fragments.

Preferably, the distributed index includes: distributed B-Tree and/or hash index.

In the second aspect, a remote sensing image data storage device with a large amount of data is also provided, including:

The first division module: for dividing the remote sensing image data with a large amount of data into N segments;

The second division module: for dividing the storage space into M nodes;

Setting module: used to set one of the M nodes as a configuration node, and set the remaining M-1 nodes as working nodes for storing large amounts of remote sensing image data;

The first storage module: used to store the remote sensing image data of N fragments with a large amount of data into the corresponding M-1 working nodes storing the remote sensing image data with a large amount of data by using the configuration node.

Preferably, it also includes:

Add module: used to add K working nodes;

The second storage module: it is used to configure the node to allocate data storage of M+K-1 nodes after adding working nodes.

Preferably, it also includes:

Copying module: for copying P copies of the N fragments and distributing them to M-1 working nodes.

Preferably, the first division module includes: a creation module: used to create a distributed index;

Allocation module: used to divide the table into multiple fragments according to the structure of the distributed index;

The third storage module: used to divide the remote sensing image data with a large amount of data into N pieces, and add them to the corresponding table divided into multiple pieces.

Preferably, the distributed index includes: distributed B-Tree and/or hash index.

In summary, the present application includes at least one of the following beneficial technical effects:

1. It can support efficient storage of large-scale data. In terms of storage of spatial remote sensing metadata, data can be horizontally sharded and different data can be distributed and stored on different nodes, so as to avoid single point of failure and data skew, and improve the reliability and scalability of the entire system.

2. The data can be distributed to different nodes by means of spatial hashing. In the storage of spatial remote sensing metadata, the spatial hash algorithm can be used to hash the data according to the spatial coordinates, so that adjacent data can be divided into the same node. This can improve query efficiency while reducing network overhead and improving overall system performance.

3. The number of nodes and data fragmentation can be adjusted in real time according to the growth of data volume, so as to ensure the stability and reliability of the system.

Description of drawings

Fig. 1 is a first embodiment diagram of a method for storing remote sensing image data with a large amount of data;

Fig. 2 is a second embodiment diagram of a method for storing remote sensing image data with a large amount of data;

Fig. 3 is a diagram of a third embodiment of a method for storing remote sensing image data with a large amount of data;

Fig. 4 is the embodiment figure that divides data into N segments;

Fig. 5 is a diagram of the first embodiment of a remote sensing image data storage device with a large amount of data;

Fig. 6 is a diagram of a second embodiment of a remote sensing image data storage device with a large amount of data;

Fig. 7 is a diagram of a third embodiment of a remote sensing image data storage device with a large amount of data;

Fig. 8 is an embodiment diagram of the first dividing module.

Explanation of reference numerals: 1. A remote sensing image data storage device with a large amount of data; 11. The first division module; 12. The second division module; 13. The setting module; 14. The first storage module; 15. The increase module; 16 . The second storage module; 17. The copy module; 111. The creation module; 112. The allocation module; 113. The third storage module.

Detailed ways

In order to make the purpose, technical solution and advantages of the present application clearer, the present application will be further described in detail below in conjunction with the accompanying drawings 1-8 and embodiments. It should be understood that the specific embodiments described here are only used to explain the present application, not to limit the present application.

A data storage method provided by this application adopts the following technical solution:

In the first aspect, as shown in FIG. 1, a method for storing remote sensing image data with a large amount of data is provided, including:

S1: Divide the large amount of remote sensing image data into N segments; the data volume of the large amount of remote sensing image data will be large, so it needs to be processed in segments. If it is not segmented, it may cause cross-node or The cross-server situation occurs. At this time, although the storage is convenient, there will be various problems such as access timeout when accessing.

S2: Divide the storage space into M nodes; the storage space includes various storage servers or storage media. Multiple storage devices can be divided into a single node; a single server can also be divided into multiple nodes. In this embodiment, multiple servers in the storage space are divided into M nodes; the single node may include a single server or multiple servers. The purpose of dividing the storage space into M nodes is to adapt to the data being divided into N pieces, that is, to make the data divided into N pieces can be properly allocated to the M nodes of the storage space.

S3: Set one of the M nodes as a configuration node, and set the remaining M−1 nodes as working nodes for storing data; the configuration node only needs one node. However, a node can include multiple servers, or just one server. The main role of the configuration node is to coordinate the work of data distribution, storage, retrieval and access of the remaining M-1 nodes. The role of the working node is mainly to store data. As for how to store data, it is the main job of the configuration node. For example, the configuration node divides the data into segments according to the length of the data, and then matches the data segment with the storage space according to the existing storage space of the working node that stores the data. Appropriate matching rules, assign different data segments to servers of different working nodes.

S4: Use the configuration node to store the remote sensing image data of N fragments with a large amount of data to the corresponding M-1 working nodes that store the remote sensing image data with a large amount of data. The purpose of this step is to better store large amounts of remote sensing image data.

Preferably, as shown in Figure 2, it also includes:

S5: Add K working nodes; in the actual production process, due to the lack of servers at any time, the ability to store data may be insufficient. Then, it is necessary to add K working nodes, that is, adding K working nodes that are not managed by the original configured nodes. Then, how to reasonably store data on the additional K working nodes is a technical problem to be solved urgently in this embodiment.

S6: Configure nodes to allocate data storage for M+K-1 nodes after adding working nodes. If K nodes are added as working nodes, then the positions of these K nodes need to be configured in the configuration node. That is, the information of these K nodes should be allocated in the configuration node. The details are as follows: Scan table: All tables that need to be rebalanced will be scanned, and the number of partitions of each table will be recorded. Determine the number of partitions: For each table, the number of all its partitions is calculated and the number of these partitions is recorded. Determine the allocation strategy: The allocation strategy will be determined according to the number of partitions in each table. For example, if a table has the same number of partitions, you can use a load-balancing-based allocation strategy; if a certain table has many more partitions than others, you can use an importance-based allocation strategy. Allocation tasks will apply the allocation strategy to each table and assign query tasks to the most suitable nodes. This process involves multiple algorithms and technologies, such as load balancing algorithms, distance algorithms, and importance algorithms. Adjust task distribution: After tasks are assigned, the task distribution will be checked and adjusted as needed. For example, if the number of tasks on a certain node is too high, some tasks may be assigned to other nodes to balance the task distribution. Repeat process: The above steps will be repeated until the number of partitions of all tables is rebalanced.

Preferably, as shown in Figure 3, it also includes:

S7: Copy P copies of the N fragments and distribute them to M-1 working nodes. To increase the number of copies by P is to increase the number of copies. Replica sets can improve the availability and performance of the database because when one node fails, other nodes can take over its work. At the same time, replica sets can also improve data redundancy and security, because any node can provide a copy of the data. However, an increase in the number of replicas also increases storage capacity and computing resource requirements. In addition, the replicas will be allocated to M-1 working nodes, in order to ensure the respective security and redundancy of the replicas of the N fragments.

Preferably, as shown in FIG. 4, the data is divided into N segments, including:

S11: Create a distributed index;

S12: Divide the table into multiple fragments according to the structure of the distributed index; perform an index creation operation on the table to create a distributed index. Divide the table into fragments based on the structure of the indexes. The size of each fragment is fixed and can be set according to actual needs.

S13: Divide the large amount of remote sensing image data into N segments, and add them to the corresponding table divided into multiple segments. Load the data page into the corresponding fragment. The data page is the core data structure of the shard, and each data page contains a certain number of row data. When the data page is inserted, updated, or deleted, the state of the fragment needs to be updated.

Preferably, the distributed index includes: distributed B-Tree and/or hash index. These indexes can be partitioned across multiple nodes. This enables this embodiment to handle tables with large numbers of rows and complex geometric objects. Horizontally partition the spatial table so that different spatial data can be stored on different nodes. This enables this embodiment to speed up spatial queries and support larger datasets. It can automatically process queries in parallel on multiple nodes, improve the parallel execution capability of spatial data operations, and make query and analysis more efficient. Provides the capability of horizontal expansion, which can be easily extended to multiple nodes, so that the entire system can be easily expanded to cope with more data requirements. The data can be distributed to different nodes by means of spatial hashing. In the storage of spatial remote sensing metadata, the spatial hash algorithm can be used to hash the data according to the spatial coordinates, so that adjacent data can be divided into the same node. This can improve query efficiency while reducing network overhead and improving overall system performance.

In the second aspect, as shown in FIG. 5 , there is also provided a remote sensing image data storage device 1 with a large amount of data, including:

The first division module 11: for dividing the remote sensing image data with a large amount of data into N segments;

The second division module 12: for dividing the storage space into M nodes;

Setting module 13: used to set one of the M nodes as a configuration node, and set the remaining M-1 nodes as working nodes for storing large amounts of remote sensing image data;

The first storage module 14 : used to store the data of N fragments to M−1 working nodes corresponding to the storage data by using the configuration node.

Preferably, as shown in Figure 6, it also includes:

Add module 15: used to add K working nodes;

The second storage module 16: used for configuring the data storage of the node allocation and adding M+K-1 nodes after the working nodes are added.

Preferably, as shown in Figure 7, it also includes:

Copying module 17: for copying P copies of the N fragments and distributing them to M-1 working nodes.

Preferably, as shown in FIG. 8, the first division module 11 includes:

Creation module 111: for creating a distributed index;

Allocation module 112: for dividing the table into multiple fragments according to the structure of the distributed index;

The third storage module 113: used to divide the large amount of remote sensing image data into N segments, and add them to the corresponding table divided into multiple segments.

Preferably, the distributed index includes: distributed B-Tree and/or hash index.

In summary, the present application includes at least one of the following beneficial technical effects:

1. It can support efficient storage of large-scale data. In terms of storage of spatial remote sensing metadata, data can be horizontally sharded and different data can be distributed and stored on different nodes, so as to avoid single point of failure and data skew, and improve the reliability and scalability of the entire system.

2. The data can be distributed to different nodes by means of spatial hashing. In the storage of spatial remote sensing metadata, the spatial hash algorithm can be used to hash the data according to the spatial coordinates, so that adjacent data can be divided into the same node. This can improve query efficiency while reducing network overhead and improving overall system performance.

3. The number of nodes and data fragmentation can be adjusted in real time according to the growth of data volume, so as to ensure the stability and reliability of the system.

All of the above are preferred embodiments of the application, and are not intended to limit the scope of protection of the application. Any feature disclosed in this specification (including abstracts and drawings), unless specifically stated, can be used by other equivalent or similar Alternative features for the purpose are replaced. That is, unless expressly stated otherwise, each feature is one example only of a series of equivalent or similar features.

Claims (10)

1. A storage method for large amount of remote sensing image data, characterized in that it comprises: dividing the data into N segments; Divide the storage space into M nodes; Setting one of the M nodes as a configuration node, and setting the remaining M-1 nodes as working nodes for storing large amounts of remote sensing image data; The configuration nodes are used to store the data of the N fragments to the corresponding M-1 working nodes that store large amount of remote sensing image data. 2. The method according to claim 1, further comprising: Add K working nodes; Configure nodes to allocate data storage for M+K-1 nodes after adding working nodes. 3. The method according to claim 1, further comprising: Copy P copies of the N fragments and distribute them to M-1 working nodes. 4. The method according to claim 1, wherein said dividing data into N segments comprises: Create a distributed index; According to the structure of the distributed index, the table is divided into multiple fragments; The remote sensing image data with a large amount of data is divided into N segments, and added to the corresponding table divided into multiple segments. 5. The method according to claim 4, wherein the distributed index comprises: a distributed B-Tree and/or a hash index. 6. A storage device for large amount of remote sensing image data, characterized in that it comprises: The first division module: used to divide the data of the large amount of remote sensing image data into N segments; The second division module: for dividing the storage space into M nodes; Setting module: used to set one of the M nodes as a configuration node, and set the remaining M-1 nodes as working nodes for storing large amounts of remote sensing image data; The first storage module: used for using the configuration node to store the data of N fragments to the corresponding M-1 working nodes that store large amount of remote sensing image data. 7. The device according to claim 6, further comprising: Add module: used to add K working nodes; The second storage module: it is used to configure the node to allocate data storage of M+K-1 nodes after adding working nodes. 8. The device according to claim 6, further comprising: Copying module: for copying P copies of the N fragments and distributing them to M-1 working nodes. 9. The device according to claim 6, wherein the first division module comprises: a creation module: used to create a distributed index; Allocation module: used to divide the table into multiple fragments according to the structure of the distributed index; The third storage module: used to divide the large amount of remote sensing image data into N segments, and add them to the corresponding table divided into multiple segments. 10. The device according to claim 9, wherein the distributed index comprises: a distributed B-Tree and/or a hash index.