CN119652901A

CN119652901A - A method, device, electronic device and storage medium for out-of-order synchronization of ship-shore data

Info

Publication number: CN119652901A
Application number: CN202411700348.1A
Authority: CN
Inventors: 李锐; 张辉; 李刚; 李�昊; 肖奕楠; 余杭涛; 王刚龙
Original assignee: Guangzhou Marine Geological Survey
Current assignee: Guangzhou Marine Geological Survey
Priority date: 2024-11-26
Filing date: 2024-11-26
Publication date: 2025-03-18

Abstract

The present invention discloses a method, device, electronic device and storage medium for synchronizing ship-shore data out of order. By initiating a data synchronization request to the shore end, the compressed information of the compressed package to be synchronized is sent to the shore end, so that the shore end performs data synchronization detection based on the compressed information, obtains a full bitmap of the data synchronization situation of the shore end; performs breakpoint detection on the data blocks based on the full bitmap, obtains breakpoint data; when the number of breakpoint data is greater than the first number, resends the breakpoint data to the shore end; otherwise, synchronizes the data blocks after the target block to the shore end in sequence; obtains the block reception situation of the shore end regularly, and updates the full bitmap based on the block reception situation; returns to execute the step of performing breakpoint detection on the data blocks based on the full bitmap, obtains the breakpoint data, and performs a new round of data synchronization iteration until all data blocks of the compressed package are synchronized. The present invention can efficiently realize ship-shore data synchronization and can be widely used in the field of data processing technology.

Description

Method and device for out-of-order synchronization of ship-shore data, electronic equipment and storage medium

Technical Field

The present invention relates to the field of data processing technologies, and in particular, to a method and apparatus for out-of-order synchronization of shore data, an electronic device, and a storage medium.

Background

In the current ship-to-shore data synchronization technology, the main stream scheme is mainly implemented by relying on TCP (transmission control protocol). This scheme is consistent with the idea of synchronizing data between two points on land, i.e. a TCP communication channel is first established between the two points, and then the transmission of synchronized data through this channel is started. However, the prior art has a significant problem in that since data is sequentially transmitted and retransmitted, much data that has been successfully received by the receiving end is still repeatedly transmitted. Such repeated transmission results in a large amount of retransmission data, which in turn affects the overall transmission speed. Accordingly, there remains a need in the art for further improvements and optimizations in terms of improving transmission efficiency.

Disclosure of Invention

The present invention aims to solve the problems of the limitations of the related art at least to some extent. Therefore, the invention provides a method, a device, electronic equipment and a storage medium for out-of-order synchronization of ship-shore data, which can efficiently realize the synchronization of ship-shore data.

In one aspect, the embodiment of the invention provides a method for out-of-order synchronization of ship-shore data, which is applied to a ship end, and comprises the following steps:

transmitting compression information of a compression packet to be synchronized to a shore end by a data synchronization request initiated by the shore end so that the shore end performs data synchronization detection based on the compression information to obtain a full bitmap of the data synchronization condition of the shore end;

The full bitmap comprises a sequence number and a synchronization state of each data block of the compressed packet, wherein the synchronization state comprises the completion of synchronization, synchronization neutralization and waiting for synchronization;

Performing breakpoint detection on the data blocks based on the full bitmap to obtain breakpoint data, retransmitting the breakpoint data to the shore end when the number of the breakpoint data is larger than the first number, and synchronizing the data blocks after the target blocks to the shore end sequentially;

the method comprises the steps of determining a synchronization state of breakpoint data, wherein the synchronization state of the breakpoint data is in synchronization, and the breakpoint data represents that a data block with the synchronization state being the completion of synchronization exists behind the breakpoint data;

periodically acquiring the block receiving condition of the bank end, and updating the full bitmap based on the block receiving condition;

and returning to execute the step of detecting the break points of the data blocks based on the full bitmap to obtain the break point data, and carrying out a new round of data synchronization iteration until all the data blocks of the compressed package are synchronized.

Optionally, periodically acquiring the block receiving condition of the shore end includes at least one of the following steps:

when the number of the data blocks received by the bank end in one round of data synchronization iteration is larger than the second number, responding to the reply of the bank end to acquire the block receiving condition of the bank end;

And periodically sending a block check request and a heartbeat request to the shore end, so that the shore end replies to the block receiving condition in response to the block check request and/or the heartbeat request, and further periodically acquiring the block receiving condition of the shore end.

Optionally, the block receiving situation includes a bitmap start ID and incremental bitmap information, where the bitmap start ID corresponds to a largest sequence number in all data blocks for which data synchronization is completed, the bitmap start ID characterizes the sequence number corresponding to the start bitmap, the synchronization state of the start bitmap is completed, and the start bitmap characterizes that no data block with the largest sequence number in synchronization exists before, and updates the full bitmap based on the block receiving situation, including the following steps:

updating the synchronous state of the initial bitmap and all the data blocks in front of the initial bitmap to complete synchronization;

The incremental bitmap information comprises synchronization states of a preset number of data blocks behind the data blocks corresponding to the initial bitmap;

based on the incremental bitmap information, carrying out state update on the synchronous state of the data blocks with preset quantity behind the initial bitmap;

the state updating comprises updating the synchronous state of the data blocks to be transmitted in the next round of data synchronous iteration from waiting for synchronization to in synchronization.

On the other hand, the embodiment of the invention provides a ship-shore data out-of-order synchronization device, which is applied to a ship terminal and comprises:

The first module is used for sending a data synchronization request initiated to the shore end, and sending the compression information of the compression packet to be synchronized to the shore end, so that the shore end performs data synchronization detection based on the compression information, and a full bitmap of the data synchronization condition of the shore end is obtained;

the second module is used for carrying out breakpoint detection on the data blocks based on the full bitmap to obtain breakpoint data, retransmitting the breakpoint data to the shore end when the quantity of the breakpoint data is larger than the first quantity, and synchronizing the sequence of the data blocks after the target blocks to the shore end;

the third module is used for periodically acquiring the block receiving condition of the bank end and updating the full bitmap based on the block receiving condition;

and the fourth module is used for returning to execute the second module and carrying out a new round of data synchronization iteration until all data blocks of the compressed package are synchronized.

On the other hand, the embodiment of the invention provides a ship-shore data disorder synchronization method, which is applied to a shore end and is interactively applied to the ship-shore data disorder synchronization method, and the method comprises the following steps:

The method comprises the steps of responding to a data synchronization request of a ship end, obtaining compression information of a compression packet to be synchronized, carrying out data synchronization detection based on the compression information to obtain a full-quantity bitmap of the data synchronization condition, and sending the full-quantity bitmap to the ship end;

The method comprises the steps that data blocks sent by a ship end are received in disorder, and the received data blocks are stored sequentially based on sequence numbers;

based on the storage condition of the data blocks, periodically replying the block receiving condition to the ship end so that the ship end updates the full bitmap of the ship end based on the block receiving condition;

And returning to the step of executing the data block sent by the out-of-order receiving ship end, and carrying out a new round of data synchronization iteration until all the data blocks of the compressed package are synchronized.

Optionally, the ship end periodically sends a block check request and/or a heartbeat request to the shore end, periodically replies the ship end with a block receiving condition, and the method comprises at least one of the following steps:

When the number of the data blocks received in one round of data synchronization iteration is larger than the second number, returning to the ship end to the block receiving condition;

Responding to the blocking verification request and/or the heartbeat request sent by the ship end periodically, and replying to the blocking receiving condition to the ship end.

Optionally, the ship end periodically sends a block verification request to the shore end, and the method further comprises the following steps:

Responding to a blocking verification request of a ship end, and acquiring a first verification value of a data block sent by the ship end from the ship end;

processing through a preset information abstract algorithm to obtain a second check value of the received data block;

performing data consistency verification based on the first verification value and the second verification value corresponding to each data partition;

And sending a retransmission request of the corresponding data block to the ship end when the data consistency check is not passed, so that the ship end retransmits the corresponding data block in response to the retransmission request.

On the other hand, the embodiment of the invention provides a ship-shore data disorder synchronization device, which is interactively applied to the ship-shore data disorder synchronization device, and comprises the following components:

the fifth module is used for responding to the data synchronization request of the ship end, acquiring the compression information of the compression packet to be synchronized, carrying out data synchronization detection based on the compression information, and obtaining a full bitmap of the data synchronization condition;

A sixth module, configured to receive, in an out-of-order manner, the data blocks sent by the ship end, and store the received data blocks sequentially based on the sequence number;

A seventh module, configured to periodically reply to the ship end with a block reception condition based on the storage condition of the data block, so that the ship end updates a full-scale bitmap of the ship end based on the block reception condition;

and the eighth module is used for returning to the sixth module and carrying out a new round of data synchronization iteration until all data blocks of the compressed package are synchronized.

Optionally, the ship end periodically sends a block verification request to the shore end, and the device further comprises:

A ninth module, configured to obtain, from the ship end, a first check value of a data block sent by the ship end in response to a block check request of the ship end;

A tenth module, configured to obtain a second check value of the received data partition through processing by using a preset information summary algorithm;

an eleventh module, configured to perform data consistency verification based on the first verification value and the second verification value corresponding to each synchronized data partition;

and the twelfth module is used for sending a retransmission request of the corresponding data block to the ship end when the data consistency check is not passed, so that the ship end retransmits the corresponding data block in response to the retransmission request.

On the other hand, the embodiment of the invention provides electronic equipment which comprises a processor and a memory, wherein the memory is used for storing a program, and the processor executes the program to realize the ship-shore data out-of-order synchronization method.

In another aspect, an embodiment of the present invention provides a computer storage medium in which a program executable by a processor is stored, the program executable by the processor being configured to implement the above-described method for out-of-order synchronization of ship-shore data when executed by the processor.

The embodiment of the invention comprises the steps of sending compressed information of a compressed packet to be synchronized to a shore through a data synchronization request initiated by the shore, so that the shore performs data synchronization detection based on the compressed information to obtain a full bitmap of the data synchronization condition of the shore, wherein the full bitmap comprises a serial number and a synchronization state of each data block of the compressed packet, the synchronization condition comprises the completion of synchronization, synchronization and waiting for synchronization, breakpoint detection is performed on the data blocks based on the full bitmap to obtain breakpoint data, when the number of the breakpoint data is larger than a first number, the breakpoint data is retransmitted to the shore, otherwise, the data blocks after the target blocks are sequentially synchronized to the shore, the synchronization state of the breakpoint data is that the data represents the data blocks with the synchronization completion state, the target blocks represent the data blocks with the largest serial numbers, the block receiving condition of the data blocks with the synchronization completion state, updating the full bitmap based on the block receiving condition is periodically obtained, and the step of performing breakpoint detection on the data blocks based on the full bitmap is performed, and the step of obtaining the breakpoint data is performed until all the data blocks of the data synchronization of a new round of the data synchronization packet are completed. The embodiment of the invention at least comprises the following beneficial effects:

the synchronization efficiency is improved by sending the compressed information of the compressed packet to the shore end, so that the shore end can perform data synchronization detection based on the information, thereby reducing unnecessary data transmission and improving the synchronization efficiency.

Breakpoint continuous transmission capability, namely breakpoint detection is carried out on the data blocks in the scheme, when the quantity of breakpoint data is larger than the first quantity, the breakpoint data is only retransmitted to the bank end, so that repeated transmission of a large quantity of synchronized data is avoided, and the synchronization efficiency is further improved.

And (3) an intelligent synchronization strategy, namely dynamically adjusting the synchronization strategy according to the data block synchronization state in the full bitmap. For the synchronized data blocks, repeated transmission is not performed, and the data blocks after the target blocks are sequentially synchronized to the shore end, so that an intelligent synchronization process is realized.

And updating and iterating in real time, namely periodically acquiring the block receiving condition of the bank end, updating the full bitmap based on the block receiving condition, and then returning to execute the breakpoint detection step to perform a new round of data synchronization iteration. This real-time update and iteration mechanism ensures the persistence and accuracy of the synchronization process.

And the full-volume bitmap comprises the sequence numbers and the synchronization states of each data block of the compressed packet, so that the monitoring of the synchronization process is more comprehensive and careful, and the problem can be found and solved in time.

In summary, the technical scheme has the beneficial effects of improving the synchronization efficiency, realizing breakpoint continuous transmission, adopting an intelligent synchronization strategy, updating and iterating in real time, comprehensively monitoring the synchronization state and the like, and remarkably improving the performance and reliability of the ship end-to-shore end data synchronization.

Drawings

The accompanying drawings are included to provide a further understanding of the invention and are incorporated in and constitute a part of this specification, illustrate and do not limit the invention.

FIG. 1 is a schematic diagram of an implementation environment for performing out-of-order synchronization of shore data according to an embodiment of the present invention;

fig. 2 is a schematic flow chart of a method for out-of-order synchronization of ship-shore data applied to a ship end according to an embodiment of the present invention;

FIG. 3 is a schematic diagram of a full-volume bitmap of a breakpoint data synchronization interaction according to an embodiment of the present invention;

fig. 4 is a schematic flow chart of a method for out-of-order synchronization of ship-shore data applied to a shore end according to an embodiment of the present invention;

FIG. 5 is a schematic flow diagram of a conventional shore data synchronous interaction principle;

fig. 6 is a schematic diagram of an expansion flow of a ship-shore data disorder synchronization method applied to a shore end according to an embodiment of the present invention;

fig. 7 is a schematic diagram of an overall business flow of a method for out-of-order synchronization of ship-shore data according to an embodiment of the present invention;

fig. 8 is a schematic structural diagram of a ship-shore data disorder synchronization device applied to a ship end according to an embodiment of the present invention;

fig. 9 is a schematic structural diagram of a ship-shore data disorder synchronization device applied to a shore end according to an embodiment of the present invention;

fig. 10 is a schematic structural diagram of an electronic device according to an embodiment of the present invention.

Detailed Description

The present invention will be described in further detail with reference to the drawings and examples, in order to make the objects, technical solutions and advantages of the present invention more apparent. It should be understood that the specific embodiments described herein are for purposes of illustration only and are not intended to limit the scope of the invention.

It should be noted that although functional block diagrams are depicted as block diagrams, and logical sequences are shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than the block diagrams in the system. The terms first/S100, second/S200, and the like in the description and in the claims and in the above-described figures, are used for distinguishing between similar objects and not necessarily for describing a particular sequential or chronological order.

Reference in the specification to "an embodiment" means that a particular feature, structure, or characteristic described in connection with the embodiment may be included in at least one embodiment of the invention. The appearances of such phrases in various places in the specification are not necessarily all referring to the same embodiment, nor are separate or alternative embodiments mutually exclusive of other embodiments. Those of skill in the art will explicitly and implicitly appreciate that the described embodiments of the invention may be combined with other embodiments.

It can be understood that the method for out-of-order synchronization of ship-shore data provided by the embodiment of the invention can be applied to any computer equipment with data processing and computing capabilities, and the computer equipment can be various terminals or servers. When the computer device in the embodiment is a server, the server is an independent physical server, or a server cluster or a distributed system formed by a plurality of physical servers, or a cloud server providing cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDNs (Content Delivery Network, content distribution networks), basic cloud computing services such as big data and artificial intelligence platforms, and the like. Alternatively, the terminal is a smart phone, a tablet computer, a notebook computer, a desktop computer, or the like, but is not limited thereto.

In order to facilitate understanding of the technical solution of the present invention, first, proper nouns of technical features that may occur in the embodiments of the present invention are explained:

the VSAT satellite communication (VERY SMALL Aperture Terminal) is that the VSAT is also called a satellite small data station or a personal earth station because the VSAT is derived from a traditional satellite communication system, wherein the small refers to that the antenna caliber of small station equipment in the VSAT system is small, usually 0.3 m-1.4 m, the equipment structure is compact, solid and intelligent, the price is low, the installation is convenient, the requirement on the use environment is not high, the limitation of a ground network is avoided, and the networking is flexible.

A drill ship (DRILLING VESSEL) is a ship dedicated to drilling operations on subsea geological formations, primarily for marine geological exploration. With the deep implementation of national ocean strategy, drilling and production vessels in China are more and more frequently arranged in the international oceans such as the Pacific ocean, and communication means between the vessels and the onshore base are only satellites.

The TCP transmission control protocol (Transmission Control Protocol) is a connection-oriented, reliable, byte stream based transport layer communication protocol. TCP is intended to accommodate a layered protocol hierarchy that supports multiple network applications. Reliable communication services are provided by means of TCP between pairs of processes in host computers connected to different but interconnected computer communication networks. TCP assumes that it can obtain simple, possibly unreliable datagram services from lower level protocols. In principle, TCP should be able to operate over a variety of communication systems from hardwired to packet-switched or circuit-switched networks.

The UDP transport control protocol (User Datagram Protocol) is based on a connectionless, unreliable datagram transport mechanism, providing only source port, destination port, data length and checksum encapsulation, enabling fast and compact end-to-end communication. The method does not ensure orderly arrival, reliable transmission or flow control of data, and is suitable for application scenes which are sensitive to delay, can tolerate packet loss and have self-error correction capability, such as real-time audio and video streaming, online games, network broadcasting, DNS query and the like.

LZMA compression Algorithm in practical compression algorithms, LZMA (Lempel-Ziv-Markov chain Algorithm) is generally considered to be one that provides extremely high compression rates. It was proposed in 1998 as an improved version of the LZ77 algorithm for implementing the 7-Zip file format (.7z). LZMA uses a chained compression method and applies a modified LZ77 algorithm at the bit level to achieve excellent compression efficiency, especially when processing data with high repeatability of text, source code, etc.

LZMA2 compression algorithm LZMA2 further improves compression performance and multi-core processor support as a subsequent version of LZMA while maintaining high compression rate. In practical applications, such as using 7-Zip software, a very high compression rate is often obtained by selecting the LZMA2 compression method.

MD5 (Message-Digest Algorithm 5), which is a widely used cryptographic hash function designed to convert information of arbitrary length into a hash value of fixed length, is unidirectional, meaning that it is difficult to reverse-extrapolate the original information from the hash value. The core meaning of MD5 is to provide a data verification mechanism, which can efficiently compare whether data are consistent by calculating MD5 value for data, without directly comparing data itself. It is achieved by complex mathematical operations, even small data variations can result in large differences in the calculated MD5 values.

FIG. 1 is a schematic view of an embodiment of the invention. Referring to fig. 1, the implementation environment includes at least one terminal 102 and a server 101. The terminal 102 and the server 101 can be connected through a network in a wireless or wired mode to complete data transmission and exchange.

The server 101 may be an independent physical server, a server cluster or a distributed system formed by a plurality of physical servers, or a cloud server providing cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDNs (Content Delivery Network, content delivery networks), basic cloud computing services such as big data and artificial intelligent platforms, and the like.

In addition, server 101 may also be a node server in a blockchain network. The blockchain is a novel application mode of computer technologies such as distributed data storage, point-to-point transmission, a consensus mechanism, an encryption algorithm and the like.

The terminal 102 may be, but is not limited to, a smart phone, tablet, notebook, desktop, smart box, smart watch, etc. The terminal 102 and the server 101 may be directly or indirectly connected through wired or wireless communication, which is not limited in this embodiment of the present invention.

Based on the implementation environment shown in fig. 1, the embodiment of the present invention provides a method for out-of-order synchronization of ship-shore data, and the method for out-of-order synchronization of ship-shore data is described below by taking an example that the method for out-of-order synchronization of ship-shore data is applied to the server 101 as an example, and it can be understood that the method for out-of-order synchronization of ship-shore data can also be applied to the terminal 102.

Referring to fig. 2, fig. 2 is a flowchart of a method for out-of-order synchronization of ship-shore data applied to a server according to an embodiment of the present invention, where an execution body of the method for out-of-order synchronization of ship-shore data may be any one of the foregoing computer devices (including a server or a terminal). Referring to fig. 2, the method is applied to a ship end, and comprises the following steps:

S100, a data synchronization request initiated to a shore end sends compression information of a compressed packet to be synchronized to the shore end, so that the shore end performs data synchronization detection based on the compression information, and a full bitmap of the data synchronization condition of the shore end is obtained;

In some embodiments, the ship end sends a data synchronization request, and forwards the data synchronization request to the shore end through a satellite, wherein the data synchronization request mainly comprises detailed information of a data compression packet transmitted by the satellite, and the detailed information comprises file name, file size, file MD5, file creation time, final modification time and the like of the compression packet, and the information is mainly used for judging whether breakpoint continuous transmission is needed.

After the shore end receives the data synchronization request, the transmitted file at the shore end is checked, and the judging process is mainly as follows:

1) Checking the file name, and checking whether the file exists at the bank end. If not, this file is indicated as being the first transmission. If not, continuing to compare MD5 of the file;

2) Checking the file MD5, checking whether the bank end file MD5 is consistent with the target file MD 5. If not, it is stated that the target file is changed and retransmission is required. If so, continuing to compare the file sizes.

3) Checking the bitmap information and the file size, if the bitmap information and the file size are consistent, indicating that the file is transmitted before, and selecting breakpoint continuous transmission. If not, it is stated that the target file is changed and retransmission is required. This step is necessary because the file is stored in the buffer before the transfer is successful, and it may happen that the buffer is tampered with or cleared.

4) Checking bitmap information, if all data packets are received, the data synchronization is successful, otherwise, retransmission or breakpoint continuous transmission is needed. If the data is transmitted successfully, checking the final check mark of the shore end, if the check is completed, replying that the ship end is successful in synchronization, otherwise replying that the final check is needed.

After the inspection is completed, the monitored condition is directly fed back to the ship end, and the ship end mainly comprises file information (file name, MD5, file bitmap, file size, time modification, transmission progress and the like) existing at the ship end. Note that the bitmap here is a full volume bitmap, i.e. the entire bitmap needs to be synchronized to the ship side.

It should be noted that, in some embodiments, before the data synchronization, the method may further include a step of locally compressing the data of the file to be transmitted, and in some specific embodiments, the method may be implemented as follows:

The ship end has a plurality of files to be transmitted, mainly the test results of technical experiments, the formats include files such as tables, texts and pictures, and the high-definition videos are few. These data files need to be compressed before transmission, and an open source LZMA2 compression algorithm is adopted, and has a very high compression rate. Compression speeds are somewhat slower than other compression algorithms, but higher compression rates are still necessary in view of satellite bandwidth. In the on-board data transmission management regulations, transmission of high definition video is not recommended. This is because the existing video is usually in a compressed format, even if the file is compressed again by using LZMA2, the file compression rate will not be too high, and the video will occupy a lot of bandwidth to affect the transmission of other files, so that the high-definition video files are usually synchronized by direct copying after the ship is on shore.

Normally, there will be a plurality of files to be synchronized, and these files are ordered according to the last modification time, and the files will be compressed in a plurality of groups. Grouping is done here on a 100M basis, i.e. the sum of the file sizes of each grouping is around 100M, if a single file exceeds 100M, then the files are grouped individually. Compression is then performed for each group, with the compressed package name being MD5, where all file names are added, thus resulting in a series of compressed packages. This grouping criteria can be custom adjusted, but it is not recommended to exceed 200M, because the larger the compressed file, the higher the file transfer failure rate will be. The compressed packets are transmitted synchronously one by one, and the invention is aimed at synchronizing data by single compressed packet.

S200, performing breakpoint detection on the data blocks based on the full bitmap to obtain breakpoint data, retransmitting the breakpoint data to the shore end when the number of the breakpoint data is larger than the first number, and otherwise, synchronizing the data blocks after the target blocks to the shore end in sequence;

Illustratively, in some embodiments, based on the file transfer bitmap, if it is determined that the file has not yet been successfully synchronized, out-of-order sending of the data chunks is prepared. Referring to fig. 3, states of data are divided into three states of in-sync, waiting for synchronization, and completing synchronization. The sending strategy is to send breakpoint data preferentially, and then send the breakpoint data sequentially one by one, and adopts the strategy of fixing the number of to-be-synchronized maximally.

Because of the out-of-order transmission and the out-of-order reception and storage, and the data reception condition is confirmed based on the bitmap, after the packet is lost in the data transmission, the bitmap has a condition that part of the bitmap is not received, and referring to fig. 3, the block 2, the block 4 and the block 6 belong to the type of packet loss retransmission. When a block bank is found not to be received, the network is possibly congested, so that the block data is selected to be retransmitted, and the advantage of the method is that all data are guaranteed to be sent as soon as possible, and the transmission speed is improved.

When sending the data packet, firstly, counting the breakpoint situation, and sending preferentially, wherein the maximum sending packet number is WindowSize1 (the window size can be set in a self-defining way). If the breakpoint data exceeds WindowSize1, the breakpoint data is mainly sent, and the sending of the breakpoint data is retransmission and retransmission of the repeated sending data. This is done to facilitate periodic data checks.

When the quantity of breakpoint data is less than or equal toIn the method, sequential data is sent, and is sent by fixed window movement, and at most 2 pieces of data are sent each time (window size can be set in a self-defined mode, at least 1 and the recommended range is 1-6), and the number of data blocks in synchronization is at most 2 pieces of window size. Every time a transmission progress reply is received, the state of the data block is marked (partially modified from in-sync to complete in-sync, partially modified from waiting in-sync) based on the condition of the shore side data reception, and then the transmission window is moved and a new data block is prepared to be transmitted.

S300, periodically acquiring the block receiving condition of the bank end, and updating the full bitmap based on the block receiving condition;

It should be noted that, in some embodiments, periodically obtaining the block receiving condition of the shore end may include at least one of the following steps:

For example, in some embodiments, the shore end receives a large number of data packets successively and stores them sequentially, but the shore end does not send a reply every time a data block is received, which reduces the data transmission efficiency. The bank end can immediately reply to the data block receiving condition when the following three conditions are met:

case1 when received When (if not divisible, the whole number is taken upwards) the data block reception is resumed immediately, wherein the valid data block does not include a duplicate packet. For example, if WindowsSize is 3, the data receiving sequence is packet 1 (block 5), packet 2 (block 5), packet 3 (block 6), packet 4 (block 6), and when packet 3 is received, 2 valid data blocks (block 5 and block 6) are received, and the data block receiving situation is recovered. Packet 2 and packet 4 are duplicate packets and are discarded. It can be seen here that in general, the number of data retransmissions at the ship end is mostlyAnd each.

And Case2, when a block check request is received, immediately replying to the data block receiving condition.

Case3, when receiving the heartbeat request, immediately replying to the data block receiving condition.

In general, the situation that reply data is received by blocks is actively sent according to any one of cases 1,2 and 3, including file ID, bitmap start ID (also called start block ID), incremental bitmap information, and the like. Because the bitmap itself may be large, not the entire bitmap, but incremental bitmap information is transmitted here. The bitmap is basically transmitted in BitMapSize sizes (which can be custom-adjusted, typically 1K, i.e., 1024), and the bitmap start range is [ bitmap start ID, bitmap start id+ BitMapSize ] can be confirmed based on the bitmap start ID. This can be understood as a bitmap of BitMapSize blocks received up to date, the bitmap start ID is calculated dynamically, meaning that the data packet of [0, start block ID) has been synchronized, and the currently synchronized data packet is [ bitmap start ID, bitmap start id+ BitMapSize). The incremental bitmap can reduce transmission bandwidth pressure and increase transmission speed.

It should be noted that, the block receiving situation includes a bitmap start ID and incremental bitmap information, where the bitmap start ID corresponds to the largest sequence number in all data blocks for completing data synchronization; the bitmap start ID characterizes the sequence number corresponding to the start bitmap, the synchronization state of the start bitmap is complete synchronization, the start bitmap characterizes that no data blocks with the largest sequence number exist before the synchronization state is in synchronization, in some embodiments, updating the full bitmap based on the block receiving condition can comprise the steps of updating the synchronization states of the start bitmap and all data blocks before the start bitmap to complete synchronization, wherein the increment bitmap information comprises the synchronization states of a preset number of data blocks behind the data blocks corresponding to the start bitmap, and the state updating is performed on the synchronization states of the preset number of data blocks behind the start bitmap based on the increment bitmap information, wherein the state updating comprises updating the synchronization states of the data blocks to be transmitted in the next round of data synchronization iteration from waiting synchronization to synchronization.

In some embodiments, the method may further include performing transmission start-stop control of data synchronization according to the sent heartbeat request and timeliness of heartbeat reply of the shore end in response to the heartbeat request. The method can be realized as follows:

the ship end can periodically send heartbeat requests, and the request content comprises information such as heartbeat SeqID, file ID and the like, and every time the heartbeat is sent, seqID= SeaID +1. The transmission frequency SendFrequency can be custom controlled, typically once every 1s (500 ms to 1s is recommended). The main reasons for setting the heartbeat request are as follows:

1) And judging the network state. The ship end can judge the network state through the heartbeat request reply, if the heartbeat reply is not received for a plurality of times, the network quality is poor, and the sending progress can be adjusted at the moment, and even the sending is stopped.

2) The transmission speed is improved. The shore side can update the bitmap information immediately after receiving the heartbeat, and the ship side sends data based on the latest bitmap. Because the heartbeat is periodic, the mechanism well avoids the extreme situation that the ship end can not acquire the latest bitmap forever due to the loss of other data packets, accelerates the movement of the ship end sending window, and improves the transmission speed.

The shore end processes the heartbeat request and replies the information such as the heartbeat SeqID, the file ID, the bitmap initial ID, the increment bitmap information and the like to the ship end. The invention judges whether the data is lost or not based on the bitmap information, so that the data loss is not required to be judged.

And judging the network quality, namely, because the heartbeat packet is very concise, the shore end sending and the ship end receiving processing hardly consume time, and the network quality can be directly fed back once the heartbeat comes. If M times SendFrequency are consecutive, no heartbeat reply packet (M may be configured in a custom manner, and is generally 3) is received, that is, M heartbeat packets are not received, at this time, it may be determined that the network quality is particularly poor, all data transmission may be stopped immediately, but the transmission frequency of the heartbeat packets is still unchanged.

If a heartbeat reply is received, assuming that the received heartbeat reply is SeqID1, the latest heartbeat sent is SeqID2, if SeqID2-SeqID1> =m, this indicates that the network quality is particularly poor, resulting in a long heartbeat cycle time, and at this time, the data sending is still stopped. If SeqID2-SeqID1< M, the network quality is adequate, and the transmission of the data packet can be resumed or continued. In general, heartbeat is a bottom protection means for judging the network at the ship end, so that the transmission speed can be improved, and breakpoint retransmission can be performed after the network is recovered to be normal in time.

S400, returning to execute the step of performing breakpoint detection on the data blocks based on the full bitmap to obtain breakpoint data, and performing a new round of data synchronization iteration until all the data blocks of the compressed package are synchronized.

Referring to fig. 4, fig. 4 is a flowchart of a method for out-of-order synchronization of ship-shore data applied to a server according to an embodiment of the present invention, where an execution body of the method for out-of-order synchronization of ship-shore data may be any one of the foregoing computer devices (including a server or a terminal). Referring to fig. 4, the method is applied to the shore end, and includes the steps of:

The method comprises the steps of T100, responding to a data synchronization request of a ship end, obtaining compression information of a compression packet to be synchronized, carrying out data synchronization detection based on the compression information, obtaining a full bitmap of the data synchronization condition, and sending the full bitmap to the ship end;

It should be noted that, the specific embodiments of this step and the application principles thereof correspond to the principles of the method of the first aspect, and are not described herein.

T200, receiving the transmitted data blocks of the ship end in disorder, and sequentially storing the received data blocks based on the serial numbers;

In some embodiments, the shore end receives a plurality of UDP packets, including three pieces of information, such as a file ID, a block ID, and a data block. The data of which file is determined according to the file ID, and the block position is determined according to the block ID, wherein the block ID is the position in the bitmap. In consideration of network factors and breakpoint retransmission, packets received by the shore end do not necessarily arrive in sequence, but the shore end stores the packets in sequence according to the block ID, wherein the packets belong to "breakpoint" data when the packets are not received temporarily. When the bitmap is synchronized, the ship end retransmits according to the breakpoint data. Meanwhile, the shore end may receive the data packets with the same file ID and partition ID for many times, and if the data packets are already stored, duplicate data packets are directly discarded, and the data packets may be retransmitted data packets.

Referring to the data transmission flow of fig. 5, the normal situation is that the ship end transmits data and the shore end receives the data repeatedly, and at the same time, the shore end periodically checks the data and replies to the ship end. When the data is definitely lost, the breakpoint data are retransmitted preferentially, and after the transmission of the breakpoint data is almost completed, the new data are restarted to be sequentially transmitted until all the data are successfully received.

T300, periodically replying a block receiving condition to the ship end based on the storage condition of the data blocks, so that the ship end updates a full-quantity bitmap of the ship end based on the block receiving condition;

it should be noted that, the ship end periodically sends a block check request and/or a heartbeat request to the shore end, and in some embodiments, periodically replies a block receiving condition to the ship end, which includes at least one of replying to the ship end a block receiving condition when the number of data blocks received in one round of data synchronization iteration is greater than the second number, and replying to the ship end in response to the block check request and/or the heartbeat request sent by the ship end periodically.

In some embodiments, the ship end periodically transmits a block check request to the shore end, as shown in fig. 6, the method may further include the steps of obtaining a first check value of a data block transmitted by the ship end from the ship end in response to the block check request of the ship end, processing by a preset information summarization algorithm to obtain a second check value of the received data block, performing data consistency check by the ship end based on the first check value and the second check value corresponding to each synchronous data block, and transmitting a retransmission request of the corresponding data block to the ship end so that the ship end retransmits the corresponding data block in response to the retransmission request by the ship end.

In some embodiments, when a verification request is received, MD5 verification is performed by taking data according to the start block verification ID and the end block verification ID, and an MD5 verification value is obtained. Assuming that the MD5 check value calculated by the shore end is A1, and the MD5 check value calculated by the ship end is A2.

If A1 and A2 agree, then the reply check agrees and the latest delta bitmap is replied.

If A1 and A2 are inconsistent, indicating that the data may be tampered, clearing the data of [ the start block check ID, the end block check ID ], resetting the corresponding bitmap state to wait for synchronization, updating the latest bitmap at the ship end (namely, the latest start block ID is set as the start block check ID), then replying to the inconsistent check, and replying to the latest increment bitmap. This is also one of the core points of the present invention, namely that data failing the block check can be actively retransmitted.

If the verification is consistent, updating the latest initial verification partition ID to be the ending partition verification ID, and updating the latest bitmap at the same time.

If the verification is inconsistent, the data of the [ initial block verification ID, end block verification ID) are cleared, the bitmap of the ship end is directly covered based on the latest bitmap, and the phase of 'breakpoint' data retransmission is ready to enter. Based on experience, this is rarely the case.

It is important to note here that only in the case of inconsistent received data verification, the bitmap information of the ship side is directly covered by the delta bitmap of the shore side, and in other cases, the bitmap is synchronized based on the largest latest start block ID.

In some embodiments, the method may further include the step of performing an overall check of the synchronized data after all the data chunks have completed synchronization. The method can be realized as follows:

Based on the ship end bitmap, after the ship end judges that all data are received, the data synchronization is successful. At this point a verification request is sent, the requested content being mainly the information of the currently transmitted file (file name, file ID, MD5, file size, last modification time, etc.).

After the bank end receives the verification request, the MD5 of the file is recalculated, and then the comparison is carried out with the MD5 of the request, if the comparison is consistent, the file transmission is successful. If not, it is indicated that some of the packets may be tampered with. MD5 is consistent in most cases.

And T400, returning to the step of executing the data block sent by the out-of-order receiving ship end, and carrying out a new round of data synchronization iteration until all the data blocks of the compressed package are synchronized.

For the purpose of illustrating the principles of the present invention in detail, the following general flow chart of the present invention is described in connection with certain specific embodiments, and it is to be understood that the following is illustrative of the principles of the present invention and is not to be construed as limiting the present invention.

Firstly, it should be noted that, the main scheme of the current ship-shore data synchronization is that the current ship-end-to-shore data synchronization is mainly realized based on TCP, and is consistent with the data synchronization thought of two points on land, namely, a TCP communication channel is established between the two points, and then synchronous data is sent.

The existing working thought is that the data synchronization from ship end to shore end at present is divided into three stages of metadata preparation, data synchronization and final verification, and the transmission is characterized by sequential fragmented transmission, continuous transmission is carried out according to sequential break points, idle bandwidth is fully utilized, and the transmission speed is improved.

But the shortcomings of the current mainstream solutions:

1) The speed is low, namely the data from the ship end to the shore end is mainly transmitted through satellites at present, and the satellite bandwidth is low, so that the data synchronization speed is low.

2) The experience is poor, in consideration of the fact that satellite bandwidth is low, and data synchronization and on-board technicians are required to be networked, bandwidth contention occurs, and the user surfing experience is poor.

3) The success rate of synchronization is low, namely, the data synchronization is mainly transmitted based on TCP at present, the integrity of data transmission is theoretically guaranteed, but the situation that the data transmission is half and finally fails often occurs in consideration of the low bandwidth of satellites.

Drawbacks of the existing working solutions:

1) The retransmission bandwidth is large, because the existing work is sequentially transmitted and retransmitted, a lot of received data can still be retransmitted, so that the retransmission data volume is large, and the transmission speed still needs to be improved.

2) The success rate of synchronization is low, the data synchronization is easy to occur to more than 80%, and continuous retransmission is caused by network reasons, so that the data synchronization failure is finally caused.

The data is easy to be tampered, namely, only a final checking mechanism is adopted, the intermediate data is easy to be tampered, and the final data synchronization failure is caused.

In view of this, the invention proposes a method for data disorder synchronization from ship end to shore, referring to fig. 7, specifically comprising the following steps:

And 1, locally compressing the data. The ship end has a plurality of files to be transmitted, mainly the test results of technical experiments, the formats include files such as tables, texts and pictures, and the high-definition videos are few. These data files need to be compressed before transmission, and an open source LZMA2 compression algorithm is adopted, and has a very high compression rate. Compression speeds are somewhat slower than other compression algorithms, but higher compression rates are still necessary in view of satellite bandwidth. In the on-board data transmission management regulations, transmission of high definition video is not recommended. This is because the existing video is usually in a compressed format, even if the file is compressed again by using LZMA2, the file compression rate will not be too high, and the video will occupy a lot of bandwidth to affect the transmission of other files, so that the high-definition video files are usually synchronized by direct copying after the ship is on shore.

In most cases, the text file compression rate can reach about 60%, i.e. the compressed text file size is about 40% of the original size, and the single compressed package is about 40M in terms of 100M packets. In a practical scenario, the compression rate may even reach 70%.

Regarding grouping, for example, if 7 files of files a (size 80M), B (size 17M), C (size 60M), D (size 60M), E (size 70M), F (size 80M), G (170M), etc. have been ordered by the last modification time, then the grouping is as follows:

1) A (size 80M) and B (size 17M) are a group, because a+b=97m, around 100M, the compressed packet name is MD5 (AB);

2) C (60M in size) and D (60M in size) are a set, C is 60M, a distance of 100M is 40M, but c+d=120m, a distance of 100M is only 20M, so C and D are a set, and the compressed packet name is MD5 (CD);

3) E (size 70M) is a group alone, E is 70M, distance 100M is 30M, but e+f=150m, distance 100M is 50M more, so E is a group alone, compressed packet name MD5 (E);

4) F (80M) is a group alone for the same reasons as E;

5) G (size 170M) is a group alone;

in summary, 5 groups were separated, each group was packed using LZMA 2.

And 2, initiating a ship end data synchronization request. The ship end sends a data synchronization request, and the data synchronization request is forwarded to the shore end through a satellite, and mainly comprises detailed information of a compression packet with transmission data, wherein the detailed information comprises information such as file name, file size, file MD5, file creation time, final modification time and the like of the compression packet, and the information is mainly used for judging whether breakpoint continuous transmission is needed or not.

And 3, checking the data synchronization condition. After the shore end receives the data synchronization request, the transmitted file at the shore end is checked, and the judging process is mainly as follows:

The bitmap information is used to mark the situation that the file is received at the shore, assuming that the compressed file is 50M, each data packet is transmitted using UDP protocol, and each packet is about 1K, then there is a total of 50M/1 k=51200 bits, i.e. 51200 bits are needed to represent the status of each packet, 0 indicates that no data is received, 1 indicates that the size of the bitmap is 51200 bits, and 51200/8=6.25 kbytes are needed to represent the bitmap. The bitmap is mainly used for data retransmission, reduces the repeated bandwidth and improves the transmission speed.

And 4, recovering the synchronous condition of the shore data. After the inspection is completed, the monitored condition is directly fed back to the ship end, and the ship end mainly comprises file information (file name, MD5, file bitmap, file size, time modification, transmission progress and the like) existing at the ship end. Note that the bitmap here is a full volume bitmap, i.e. the entire bitmap needs to be synchronized to the ship side.

And 5, sending the data blocks out of order. Based on the file transfer bitmap, if it is determined that all data packets have been received, a final check of the data is required. The ship end checks whether the final check of the file is finished, and if the final check of the file is finished, the whole synchronous transmission flow is finished. If the file is not subjected to final verification, turning to step 13, and performing data verification.

Based on the file transfer bitmap, if it is determined that the file has not yet been successfully synchronized, data blocks are prepared to be sent out of order. Referring to fig. 3, states of data are divided into three states of in-sync, waiting for synchronization, and completing synchronization. The sending strategy is to send breakpoint data preferentially, and then send the breakpoint data sequentially one by one, and adopts the strategy of fixing the number of to-be-synchronized maximally.

Because of the out-of-order transmission and the out-of-order reception and storage, and the data reception condition is confirmed based on the bitmap, after the packet is lost in the data transmission, the bitmap has a condition that part of the bitmap is not received, and referring to fig. 3, the block 2, the block 4 and the block 6 belong to the type of packet loss retransmission. When a block bank is found not to be received, the network is possibly congested, so that the block data is selected to be retransmitted, and the advantage of the method is that all data are guaranteed to be sent as soon as possible, and the transmission speed is improved. Regarding how to determine bluff whether a data block is received or not, this later step will give a determination rule in step 12, which is not developed first.

For example, referring to fig. 2, assume that WindowSize1 is 2 and WindowSize2 is 3, and then, the blocks 2, 4, and 6 are of the type of packet loss retransmission, and it is necessary to retransmit the blocks 2 and 4 preferentially. When the synchronization of the block 2 and the block 4 is completed, only the block 6, i.e. the breakpoint data is equal to 1, is left in the packet loss retransmission, and at this time, the blocks 9, 10 and 11 are transmitted sequentially from the block 9. The packet loss data is retransmitted preferentially, so that the received data is ensured not to leave a breakpoint as far as possible, and a solid foundation is laid for periodical segment check of the following data.

And 6, receiving the data blocks out of order and storing the data blocks in sequence. The shore end receives a plurality of UDP data packets, wherein the UDP data packets comprise three parts of information such as file IDs, block IDs, data blocks and the like. The data of which file is determined according to the file ID, and the block position is determined according to the block ID, wherein the block ID is the position in the bitmap. In consideration of network factors and breakpoint retransmission, packets received by the shore end do not necessarily arrive in sequence, but the shore end stores the packets in sequence according to the block ID, wherein the packets belong to "breakpoint" data when the packets are not received temporarily. When the bitmap is synchronized, the ship end retransmits according to the breakpoint data. Meanwhile, the shore end may receive the data packets with the same file ID and partition ID for many times, and if the data packets are already stored, duplicate data packets are directly discarded, and the data packets may be retransmitted data packets.

The bitmap is updated each time a packet is received and is periodically synchronized according to rules, which are described in step 8.

And 7, periodically replying the data block receiving condition. The shore end continuously receives a large number of data packets and stores the data packets in sequence, but the shore end does not send a reply every time a data block is received, so that the data transmission efficiency is reduced. The bank end can immediately reply to the data block receiving condition when the following three conditions are met:

Here, for example, assuming that the compressed file is 50M, each packet is transmitted using UDP protocol, and each packet has about 1K, there is a total of 50M/1 k=50000 bits, i.e., 50000 bits are required to indicate the status of each packet, 0 indicates no receipt, and 1 indicates receipt, i.e., the bit map size is 50000/8=6250=6.1K bytes. The 6.1 kbyte bitmap is also relatively large, so here a transmission delta bitmap is chosen, assuming a start ID of 0 and bitmapsize of 1024, then the bitmap received by the data chunk is [0,1024 ].

With respect to the "breakpoint" retransmission strategy of step 5, there is a bit to be discussed herein. The shore end can send data block receiving conditions according to the Case1, case2 and Case3 fixed rules, and when the shore end receives the latest bitmap information, the shore end can actively check the breakpoint data conditions, stop the current data sending and preferentially send the breakpoint data. When a plurality of increment bitmap information is received, the maximum initial bitmap ID is taken as the latest progress (special case exception of data verification failure is caused, the ship end bitmap is directly covered in the case, and a large breakpoint retransmission state is entered).

And 8, periodically sending a blocking verification request. When the ship end receives the increment bitmap, the bitmap is synchronized, and the following operation is mainly completed.

1. Checking the data block of the synchronization [ 0] and the starting block ID), setting the state as the synchronization completion, synchronizing and updating the bitmap of the ship end.

2. Based on the increment bitmap [ initial block ID, bitmap initial ID+1K), directly covering, synchronizing and updating the bitmap of the ship end.

Meanwhile, the ship end may receive the incremental bitmap information for multiple times, and at this time, the maximum initial block ID is used as the latest bitmap, and other incremental bitmap information may be discarded. After the ship end updates to the latest bitmap, the ship end can send out a data verification request periodically. The ship end sends a verification request according to the latest initial verification block ID and data verification size VerifSize (which can be self-defined and adjusted to default to 100). Assuming that the latest start block ID is LASTRECVID, the latest start check block ID is LASTVERIFID.

Calculation of BatchSize according to the following equation

If the BatchSize is 0, it indicates that the latest received data is less than VerifSize, at this time, no check request is issued. If BatchSize >0, then the bank calculates the MD5 check value of [ LASTVERIFID, LASTVERIFID +BatchSize x VerifSize) and then issues a data check request for the [ LASTVERIFID, LASTVERIFID +BatchSize x VerifSize) range segment.

If LASTRECVID is the bitmap last ID, i.e., all data syncs complete, then a data check request is issued for the [ LASTVERIFID, LASTRECVID ] range segment.

The data verification request packet contains contents report, file ID, start block verification ID, end block verification ID and MD5 verification value.

Here, for example, if LASTRECVID is 1027, lastverifid is 200, verifsize is 100, at which time batchSize is 8, i.e. at which time a block ID check of the range 200, 1000) is issued.

It is explained here how to set the size of VerifSize, the bandwidth of the current shore classmates is around 2.5MB, i.e. 2500KB, assuming here that the data synchronization bandwidth is 1KB, taking into account the other bandwidth occupancy on the ship. VerifSize takes 100 (1 k for each packet, 100 or 100 s) to indicate that data verification is performed every 100s or so. VerifSize is set according to the current satellite bandwidth situation, and is not too large, and is recommended to not exceed 500 based on the current satellite bandwidth.

And 9, checking the received data. When a verification request is received, according to the start block verification ID and the end block verification ID, data is taken to carry out MD5 verification, and an MD5 verification value is obtained. Assuming that the MD5 check value calculated by the shore end is A1, and the MD5 check value calculated by the ship end is A2.

And 10, replying the data verification condition and selectively retransmitting.

And step 11, sending a heartbeat request. The ship end can periodically send heartbeat requests, and the request content comprises information such as heartbeat SeqID, file ID and the like, and every time the heartbeat is sent, seqID= SeaID +1. The transmission frequency SendFrequency can be custom controlled, typically once every 1s (500 ms to 1s is recommended). The main reasons for setting the heartbeat request are as follows:

And step 12, recovering the heartbeat. The shore end processes the heartbeat request and replies the information such as the heartbeat SeqID, the file ID, the bitmap initial ID, the increment bitmap information and the like to the ship end. The invention judges whether the data is lost or not based on the bitmap information, so that the data loss is not required to be judged.

For example, the ship-side heartbeat sending and receiving conditions are assumed to be sending heartbeat 1, receiving heartbeat reply 1, sending heartbeat 2, receiving heartbeat reply 2, sending heartbeat 3 and receiving heartbeat reply 3. The normal situation of the data network is that the network delay of the land is generally 30ms to 50ms, and the satellite transmission is considered, the time delay of the ship-shore network is estimated to be 200ms, namely, the reply of the heartbeat 1 can be received after 200ms, so that the data network belongs to the normal situation of the network.

The sending and receiving conditions of the ship-end heartbeat are assumed to be that the sending heartbeat 1, the sending heartbeat 2, the receiving heartbeat reply 1, the receiving heartbeat reply 2, the sending heartbeat 3 and the receiving heartbeat reply 3. This situation is a slightly worse network but may also continue to transmit.

Assuming that the sending and receiving conditions of the ship end heartbeat are that the sending heartbeat 1, the sending heartbeat 2, the sending heartbeat 3 and the sending heartbeat 4 are the sending heartbeat, and the continuous M heartbeat replies are not received at the moment, which means that the network quality is poor, and all data sending can be stopped immediately.

Assuming that the sending and receiving condition of the ship side heartbeat is that the heartbeat 4 is sent and the heartbeat reply 1 is received, at the moment, although the heartbeat reply is received, the reply is very early, and all data sending can still be stopped.

The condition of sending and receiving the heartbeat at the ship end is assumed to be that the heartbeat 4 is sent, the heartbeat reply 1 is received and the heartbeat reply 4 is received, and the condition of the network is indicated to be good at the moment, so that the data sending can be recovered.

And 13, successfully synchronizing the data and sending a verification request. Based on the ship end bitmap, after the ship end judges that all data are received, the data synchronization is successful. At this point a verification request is sent, the requested content being mainly the information of the currently transmitted file (file name, file ID, MD5, file size, last modification time, etc.).

And 14, checking the data locally. After the bank end receives the verification request, the MD5 of the file is recalculated, and then the comparison is carried out with the MD5 of the request, if the comparison is consistent, the file transmission is successful. If not, it is indicated that some of the packets may be tampered with. MD5 is consistent in most cases.

And 15, replying to the data verification condition. The shore replies to the MD5 verification, mainly the information of the current transmission file (file name, file ID, MD5, file size, last modification time, etc.). After receiving the reply, the bank end records the transmission condition, marks the file to finish synchronization, and informs the user. If MD5 is consistent, the user is notified that the file transfer was successful, otherwise the user is notified that the intermediate data was tampered with, requiring resynchronization (where the data is not reset and emptied, requiring active retransmission by the user). In short, the data with failed block verification can be actively retransmitted, but if the step of final verification is reached, the data with failed verification cannot be actively retransmitted, so that a user can be reminded and retransmission is selected.

When steps 13 through 15 are completed, the file is marked as synchronized and another file is transferred. It should be explained that in an extreme case, if the file is just successfully transmitted, the network is suddenly disconnected, the ship end cannot receive the data check reply at a later time, the file is still in a synchronous state, and after the network is recovered to be normal, the ship end can resend the check request until the check reply is received, and the transmission flow of the file is really finished.

In summary, the whole transmission is divided into three phases:

1) The preparation stage comprises steps 1 to 4, wherein the stage mainly comprises the steps of determining information of a shore-side transmission file, synchronizing the latest bitmap and belongs to a data preparation stage before transmission;

2) And the synchronization stage comprises steps 5 to 12, wherein the stage mainly synchronizes data, is responsible for data synchronization and bitmap synchronization, dynamically adjusts the sending rhythm through a heartbeat and block verification mechanism, and improves the file transmission efficiency.

3) The verification phase, which is mainly the last data verification, includes steps 13 to 15.

Compared with the prior art, the invention at least has the following beneficial effects:

1) And a primary bitmap mechanism is introduced to control the data transmission flow, reduce the data retransmission bandwidth and improve the transmission speed.

2) And out-of-order transmission is realized based on a fixed window transmission strategy, idle bandwidth is fully utilized, and transmission speed is improved.

3) And the intermediate data is checked regularly, the breakpoint continuous transmission is supported, and the data integrity is ensured from the beginning of data downloading to the completion of the whole flow.

On the other hand, as shown in fig. 8, an embodiment of the present invention provides a ship-shore data out-of-order synchronization device 800, which is applied to a ship end and may include:

A first module 801, configured to send, to a shore, compression information of a compressed packet to be synchronized to a data synchronization request initiated by the shore, so that the shore performs data synchronization detection based on the compression information, and obtain a full bitmap of data synchronization conditions of the shore;

A second module 802, configured to perform breakpoint detection on the data blocks based on the full bitmap to obtain breakpoint data, and retransmit the breakpoint data to the shore when the number of the breakpoint data is greater than the first number;

a third module 803, configured to periodically obtain a block reception condition of the shore end, and update the full bitmap based on the block reception condition;

and a fourth module 804, configured to return to the second module to perform a new round of data synchronization iteration until all data blocks of the compressed packet are synchronized.

On the other hand, as shown in fig. 9, an embodiment of the present invention provides a ship shore data disorder synchronization device 900, which is applied to a shore end and may include:

a fifth module 901, configured to respond to a data synchronization request of the ship end, obtain compression information of a compression packet to be synchronized;

A sixth module 902, configured to sequentially store the received data blocks based on the sequence number, where the data blocks are sent by the out-of-order receiving ship end;

a seventh module 903, configured to periodically reply to the ship end with a block reception condition based on the storage condition of the data block, so that the ship end updates a full bitmap of the ship end based on the block reception condition;

and an eighth module 904, configured to return to the sixth module to perform a new round of data synchronization iteration until all data blocks of the compressed packet are synchronized.

In some embodiments, the ship end periodically sends a block verification request to the shore end, and the ship-shore data disorder synchronization device applied to the shore end may further include:

The content of the method embodiment of the invention is suitable for the device embodiment, the specific function of the device embodiment is the same as that of the method embodiment, and the achieved beneficial effects are the same as those of the method.

On the other hand, the embodiment of the invention also provides electronic equipment, which comprises a memory and a processor, wherein the memory stores a computer program, and the processor realizes the hydrate stability domain bottom boundary prediction method when executing the computer program. The electronic equipment can be any intelligent terminal including a tablet personal computer, a vehicle-mounted computer and the like.

It can be understood that the content in the above method embodiment is applicable to the embodiment of the present apparatus, and the specific functions implemented by the embodiment of the present apparatus are the same as those of the embodiment of the above method, and the achieved beneficial effects are the same as those of the embodiment of the above method.

As shown in fig. 10, fig. 10 illustrates a hardware structure of an electronic device 1000 of another embodiment, and the electronic device 1000 includes:

The processor 1001 may be implemented by using a general purpose CPU (Central Processing Unit ), a microprocessor, an Application SPECIFIC INTEGRATED Circuit (ASIC), or one or more integrated circuits, etc. to execute related programs to implement the technical solution provided by the embodiments of the present invention;

The Memory 1002 may be implemented in the form of a Read Only Memory (ROM), a static storage device, a dynamic storage device, or a random access Memory (Random Access Memory, RAM). The memory 1002 may store an operating system and other application programs, and when the technical solutions provided in the embodiments of the present disclosure are implemented by software or firmware, relevant program codes are stored in the memory 1002, and the processor 1001 invokes a network node population optimization method for executing the embodiments of the present disclosure;

an input/output interface 1003 for implementing information input and output;

The communication interface 1004 is configured to implement communication interaction between the present device and other devices, and may implement communication in a wired manner (e.g. USB, network cable, etc.), or may implement communication in a wireless manner (e.g. mobile network, WIFI, bluetooth, etc.);

A bus 1005 for transferring information between the various components of the device (e.g., the processor 1001, memory 1002, input/output interface 1003, and communication interface 1004);

wherein the processor 1001, the memory 1002, the input/output interface 1003, and the communication interface 1004 realize communication connection between each other inside the device through the bus 1005.

The above described embodiments of the electronic device are merely illustrative, wherein the units described as separate components may or may not be physically separate, i.e. may be located in one place, or may be distributed over a plurality of network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

The content of the method embodiment of the invention is suitable for the electronic equipment embodiment, the functions of the electronic equipment embodiment are the same as those of the method embodiment, and the achieved beneficial effects are the same as those of the method.

Another aspect of the embodiments of the present invention also provides a computer-readable storage medium storing a program that is executed by a processor to implement the foregoing method.

It should be noted that, the computer readable medium shown in the embodiments of the present invention may be a computer readable signal medium or a computer readable storage medium, or any combination of the two. The computer readable storage medium can be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or a combination of any of the foregoing. More specific examples of a computer-readable storage medium may include, but are not limited to, an electrical connection having one or more wires, a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-Only Memory (ROM), an erasable programmable read-Only Memory (Erasable Programmable Read Only Memory, EPROM), a flash Memory, an optical fiber, a portable compact disc read-Only Memory (Compact Disc Read to Only Memory, CD to ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In the context of this document, a computer readable storage medium may be any tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device. In the present invention, however, the computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, with the computer-readable program code embodied therein. Such a propagated data signal may take any of a variety of forms, including, but not limited to, electro-magnetic, optical, or any suitable combination of the foregoing. A computer readable signal medium may also be any computer readable medium that is not a computer readable storage medium and that can communicate, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device. Program code embodied on a computer readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wireline, etc., or any suitable combination of the foregoing.

The content of the method embodiment of the invention is applicable to the computer readable storage medium embodiment, the functions of the computer readable storage medium embodiment are the same as those of the method embodiment, and the achieved beneficial effects are the same as those of the method.

Embodiments of the present invention also disclose a computer program product or computer program comprising computer instructions stored in a computer readable storage medium. The computer instructions may be read from a computer-readable storage medium by a processor of a computer device, and executed by the processor, to cause the computer device to perform the foregoing method.

The flowcharts and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to various embodiments of the present invention. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of code, which comprises one or more executable instructions for implementing the specified logical function(s). It should also be noted that, in some alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams or flowchart illustration, and combinations of blocks in the block diagrams or flowchart illustration, can be implemented by special purpose hardware-based systems which perform the specified functions or acts, or combinations of special purpose hardware and computer instructions.

It should be noted that although in the above detailed description several modules of a device for action execution are mentioned, such a division is not mandatory. Indeed, the features and functions of two or more modules or units described above may be embodied in one module or unit in accordance with embodiments of the invention. Conversely, the features and functions of one module or unit described above may be further divided into a plurality of modules or units to be embodied.

From the above description of embodiments, those skilled in the art will readily appreciate that the example embodiments described herein may be implemented in software, or may be implemented in software in combination with the necessary hardware. Thus, the technical solution according to the embodiments of the present invention may be embodied in the form of a software product, which may be stored in a non-volatile storage medium (may be a CD to ROM, a U-disc, a mobile hard disk, etc.) or on a network, and includes several instructions to cause a computing device (may be a personal computer, a server, a touch terminal, or a network device, etc.) to perform the method according to the embodiments of the present invention.

In some alternative embodiments, the functions/acts noted in the block diagrams may occur out of the order noted in the operational illustrations. For example, two blocks shown in succession may in fact be executed substantially concurrently or the blocks may sometimes be executed in the reverse order, depending upon the functionality/acts involved. Furthermore, the embodiments presented and described in the flowcharts of the present invention are provided by way of example in order to provide a more thorough understanding of the technology. The disclosed methods are not limited to the operations and logic flows presented by the present invention. Alternative embodiments are contemplated in which the order of various operations is changed, and in which sub-operations described as part of a larger operation are performed independently.

Furthermore, while the invention is described in the context of functional modules, it should be appreciated that, unless otherwise indicated, one or more of the functions and/or features may be integrated in a single physical device and/or software module or may be implemented in separate physical devices or software modules. It will also be appreciated that a detailed discussion of the actual implementation of each module is not necessary to an understanding of the present invention. Rather, the actual implementation of the various functional modules in the apparatus disclosed in the present invention will be understood within the ordinary skill of the engineer in view of their attributes, functions and internal relationships. Accordingly, one of ordinary skill in the art can implement the invention as set forth in the claims without undue experimentation. It is also to be understood that the specific concepts disclosed are merely illustrative and are not intended to be limiting upon the scope of the invention, which is to be defined in the appended claims and their full scope of equivalents.

The functions, if implemented in the form of software functional units and sold or used as a stand-alone product, may be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention may be embodied essentially or in a part contributing to the prior art or in a part of the technical solution in the form of a software product stored in a storage medium, comprising several instructions for causing a computer device (which may be a personal computer, a server, a network device, etc.) to perform all or part of the steps of the method of the embodiments of the present invention. The storage medium includes a usb disk, a removable hard disk, a Read-Only Memory (ROM), a random-access Memory (RAM, random Access Memory), a magnetic disk, an optical disk, or other various media capable of storing program codes.

Logic and/or steps represented in the flowcharts or otherwise described herein, e.g., a ordered listing of executable instructions for implementing logical functions, can be embodied in any computer-readable medium for use by or in connection with an instruction execution apparatus, device, or apparatus, such as a computer-based apparatus, processor-containing apparatus, or other apparatus that can fetch the instructions from the instruction execution apparatus, device, or apparatus and execute the instructions. For the purposes of this description, a "computer-readable medium" can be any means that can contain, store, communicate, propagate, or transport the program for use by or in connection with the instruction execution apparatus, device, or apparatus.

More specific examples (a non-exhaustive list) of the computer-readable medium would include an electrical connection (an electronic device) having one or more wires, a portable computer diskette (a magnetic device), a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber device, and a portable compact disc read-only memory (CDROM). Additionally, the computer-readable medium may even be paper or other suitable medium upon which the program is printed, as the program may be electronically captured, via, for instance, optical scanning of the paper or other medium, then compiled, interpreted or otherwise processed in a suitable manner, if necessary, and then stored in a computer memory.

It is to be understood that portions of the present invention may be implemented in hardware, software, firmware, or a combination thereof. In the above embodiments, the various steps or methods may be implemented in software or firmware stored in a memory and executed by a suitable instruction execution device. For example, if implemented in hardware, as in another embodiment, may be implemented using any one or combination of techniques known in the art, discrete logic circuits with logic gates for implementing logic functions on data signals, application specific integrated circuits with appropriate combinational logic gates, programmable Gate Arrays (PGAs), field Programmable Gate Arrays (FPGAs), and the like.

In the description of the present specification, a description referring to terms "one embodiment," "some embodiments," "examples," "specific examples," or "some examples," etc., means that a particular feature, structure, material, or characteristic described in connection with the embodiment or example is included in at least one embodiment or example of the present invention. In this specification, schematic representations of the above terms do not necessarily refer to the same embodiments or examples. Furthermore, the particular features, structures, materials, or characteristics described may be combined in any suitable manner in any one or more embodiments or examples.

Although embodiments of the present invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made therein without departing from the spirit and scope of the invention as defined by the appended claims and their equivalents.

While the preferred embodiment of the present invention has been described in detail, the present invention is not limited to the embodiments, and those skilled in the art can make various equivalent modifications or substitutions without departing from the spirit of the present invention, and the equivalent modifications or substitutions are intended to be included in the scope of the present invention as defined in the appended claims.

Claims

1. A method for synchronizing ship-shore data out of order, characterized in that it is applied to the ship side and comprises the following steps:

A data synchronization request is initiated to the shore end, and the compression information of the compressed package to be synchronized is sent to the shore end, so that the shore end performs data synchronization detection based on the compression information and obtains a full bitmap of the data synchronization status of the shore end;

The full bitmap includes the sequence number and synchronization status of each data block of the compressed package; the synchronization status includes completed synchronization, synchronizing, and waiting for synchronization;

Performing breakpoint detection on the data blocks based on the full bitmap to obtain breakpoint data; when the amount of the breakpoint data is greater than a first amount, resending the breakpoint data to the shore end; otherwise, synchronizing the data blocks after the target block to the shore end in sequence;

The synchronization state of the breakpoint data is in synchronization, and the breakpoint data indicates that there is a data block whose synchronization state is completed synchronization; the target block indicates the data block with the largest sequence number whose synchronization state is completed synchronization;

Periodically obtaining the block reception status of the shore end, and updating the full quantity bitmap based on the block reception status;

Return to the step of performing breakpoint detection on the data blocks based on the full bitmap to obtain breakpoint data, and perform a new round of data synchronization iteration until all the data blocks of the compressed package are synchronized.

2. The method for out-of-order synchronization of ship-shore data according to claim 1, characterized in that the periodic acquisition of the block receiving status of the shore end comprises at least one of the following steps:

When the number of the data blocks received by the shore end in a round of data synchronization iteration is greater than a second number, obtaining the block reception status of the shore end in response to a reply from the shore end;

Periodically send block check requests and heartbeat requests to the shore end, so that the shore end replies with the block reception status in response to the block check request and/or the heartbeat request, and then periodically obtain the block reception status of the shore end.

3. The method for out-of-order synchronization of ship-shore data according to claim 1 is characterized in that the block reception status includes a bitmap start ID and incremental bitmap information, the bitmap start ID corresponds to the largest sequence number of all the data blocks that have completed data synchronization; the bitmap start ID represents the sequence number corresponding to the start bitmap, the synchronization state of the start bitmap is the completed synchronization, and the start bitmap represents that there is no data block with the largest sequence number in the synchronization state; the updating of the full bitmap based on the block reception status includes the following steps:

Updating the synchronization status of the starting bitmap and all the data blocks preceding it to the synchronization completion;

Wherein, the incremental bitmap information includes the synchronization status of a preset number of data blocks following the data block corresponding to the start bitmap;

Based on the incremental bitmap information, updating the synchronization status of the preset number of data blocks after the starting bitmap;

The state update includes updating the synchronization state of the data blocks to be transmitted in the next round of data synchronization iteration from waiting for synchronization to synchronizing.

4. A ship-shore data out-of-order synchronization device, characterized in that it is applied to the ship side and comprises:

The first module is used to initiate a data synchronization request to the shore end, and send the compression information of the compressed package to be synchronized to the shore end, so that the shore end performs data synchronization detection based on the compression information, and obtains a full bitmap of the data synchronization status of the shore end;

The second module is used to perform breakpoint detection on the data block based on the full bitmap to obtain breakpoint data; when the amount of the breakpoint data is greater than the first amount, resend the breakpoint data to the shore end; otherwise, synchronize the data block sequence after the target block to the shore end;

The third module is used to periodically obtain the block reception status of the shore end, and update the full amount bitmap based on the block reception status;

The fourth module is used to return to execute the second module and perform a new round of data synchronization iteration until all the data blocks of the compressed package are synchronized.

5. A method for synchronizing ship-shore data out of order, characterized in that the method is applied to the shore side, and the method is interactively applied with the method for synchronizing ship-shore data out of order as claimed in any one of claims 1 to 3, and the method comprises the following steps:

In response to a data synchronization request from the ship side, obtaining compression information of a compression package to be synchronized; performing data synchronization detection based on the compression information to obtain a full bitmap of data synchronization status; and sending the full bitmap to the ship side;

receiving the data blocks sent by the ship end out of order, and sequentially storing the received data blocks based on the sequence numbers;

Based on the storage status of the data blocks, periodically replying the block reception status to the ship end, so that the ship end updates the full quantity bitmap of the ship end based on the block reception status;

Return to the step of receiving the data blocks sent by the ship in disorder, and perform a new round of data synchronization iteration until all the data blocks of the compressed package are synchronized.

6. The method for synchronizing ship-shore data out of order according to claim 5, characterized in that the ship end periodically sends a block verification request and/or a heartbeat request to the shore end, and the periodic reply of the block reception status to the ship end comprises at least one of the following steps:

When the number of the data blocks received in a round of data synchronization iteration is greater than a second number, replying the block reception status to the ship end;

In response to the block check request and/or the heartbeat request periodically sent by the ship end, the block reception status is replied to the ship end.

7. The method for synchronizing ship-shore data out of order according to claim 5, characterized in that the ship end periodically sends a block verification request to the shore end, and the method further comprises the following steps:

In response to the block check request of the ship end, obtaining, from the ship end, a first check value of the data block sent by the ship end;

Obtaining a second check value of the received data block by processing with a preset information digest algorithm;

Performing data consistency verification based on the first verification value and the second verification value corresponding to each of the synchronized data blocks;

When the data consistency check fails, a retransmission request of the corresponding data block is sent to the ship end, so that the ship end retransmits the corresponding data block in response to the retransmission request.

8. A ship-shore data out-of-order synchronization device, characterized in that it is applied to the shore side, and the device is interactively applied with the ship-shore data out-of-order synchronization device as claimed in claim 4, comprising:

The fifth module is used to respond to the data synchronization request of the ship end, obtain the compression information of the compressed package to be synchronized; perform data synchronization detection based on the compression information to obtain a full bitmap of the data synchronization situation; and send the full bitmap to the ship end;

A sixth module is used to receive the data blocks sent by the ship end out of order, and sequentially store the received data blocks based on the sequence numbers;

A seventh module is used to periodically reply the block reception status to the ship end based on the storage status of the data blocks, so that the ship end updates the full amount bitmap of the ship end based on the block reception status;

The eighth module is used to return to execute the sixth module and perform a new round of data synchronization iteration until all the data blocks of the compressed package are synchronized.

9. An electronic device, comprising a processor and a memory;

The memory is used to store programs;

The processor executes the program to implement the method according to any one of claims 1 to 3 or 5 to 7.

10. A computer storage medium storing a processor-executable program, wherein the processor-executable program is used to implement the method according to any one of claims 1 to 3 or 5 to 7 when executed by the processor.