CN119854313A

CN119854313A - Ship-shore weak network data synchronization method and device, electronic equipment and storage medium

Info

Publication number: CN119854313A
Application number: CN202411819861.2A
Authority: CN
Inventors: 张辉; 李刚; 李�昊; 巫鹏程; 黄钰峰; 余杭涛
Original assignee: Guangzhou Marine Geological Survey
Current assignee: Guangzhou Marine Geological Survey
Priority date: 2024-12-11
Filing date: 2024-12-11
Publication date: 2025-04-18

Abstract

The invention discloses a method, a device, electronic equipment and a storage medium for synchronizing ship-shore weak network data, wherein the method comprises the steps of carrying out packet compression on first data to be synchronized in a target ship end to obtain second data; the method comprises the steps of transmitting compressed information to a target bank, comparing the compressed information through the target bank based on transmitted files to obtain a transmission mode and a transmission progress of each compressed packet in second data, initializing metadata at the target bank and the target bank based on the transmission mode and the transmission progress to complete transmission preparation, sequentially transmitting the compressed packets of the target bank to the target bank in blocks based on a sliding window, synchronizing any number of data blocks in front of the sliding window when the target bank completes synchronization of the data blocks in the sliding window, and moving the sliding window backwards by a corresponding number of data blocks until the target bank completes synchronization of all the data blocks. The invention can realize data synchronization with high efficiency and can be widely applied to the technical field of data processing.

Description

Ship-shore weak network data synchronization method and device, electronic equipment and storage medium

Technical Field

The present invention relates to the field of data processing technologies, and in particular, to a method and apparatus for synchronizing data of a weak network on a ship, an electronic device, and a storage medium.

Background

At present, the data synchronization from ship end to shore end is mainly realized based on TCP, and is consistent with the data synchronization thought of two points on land, namely, a TCP communication channel is established between the two points, and then synchronous data is sent. However, the prior art has the following disadvantages:

1) The speed is low, namely the data from the ship end to the shore end is mainly transmitted through satellites at present, and the satellite bandwidth is low, so that the data synchronization speed is low.

2) The experience is poor, in consideration of the fact that satellite bandwidth is low, and data synchronization and on-board technicians are required to be networked, bandwidth contention occurs, and the user surfing experience is poor.

3) The success rate of synchronization is low, namely, the data synchronization is mainly transmitted based on TCP at present, the integrity of data transmission is theoretically guaranteed, but the situation that the data transmission is half and finally fails often occurs in consideration of the low bandwidth of satellites.

Disclosure of Invention

The present invention aims to solve the problems of the limitations of the related art at least to some extent. Therefore, the invention provides a method and a device for synchronizing ship shore weak network data, electronic equipment and a storage medium, which can realize data synchronization with high efficiency.

In one aspect, an embodiment of the present invention provides a method for synchronizing weak network data on a ship shore, including:

The method comprises the steps of carrying out packet compression on first data to be synchronized in a target ship end to obtain second data, wherein the second data comprises at least one compression packet and compression information corresponding to the compression packet;

Transmitting the compressed information to a target bank terminal, and comparing the compressed information based on the transmitted file by the target bank terminal to obtain a transmission mode and a transmission progress of each compressed packet in the second data, wherein the transmission mode comprises de-header transmission and breakpoint transmission, and the transmission progress of the de-header transmission is 0;

Based on the transmission mode and the transmission progress, initializing metadata at a target ship end and a target shore end respectively to finish transmission preparation;

Sequentially transmitting the compressed packets of the target ship end to the target shore end in blocks based on the sliding window;

The sliding window comprises a preset number of data blocks which are sequentially arranged, when the target bank end completes synchronization of any number of data blocks in front of the sliding window, the sliding window is moved backwards by a corresponding number of data blocks until the target bank end completes synchronization of all data blocks of the compressed package.

Optionally, the compressed information comprises file name, file size, file MD5 and last modification time, and the first data to be synchronized in the target ship end is compressed in a grouping way to obtain second data, and the method comprises the following steps:

sorting the data files in the first data according to the modification time to obtain a file sequence;

Sequentially traversing the file sequence based on a preset size reference, and grouping all data files in the file sequence into a plurality of data groups;

carrying out file compression on each data packet through a preset compression algorithm to obtain a corresponding compression packet;

the file names are obtained based on name combinations of all data files in the data packet, the file size and the file MD5 are obtained based on a preset algorithm, and finally the modification time is determined based on the modification time of the last data file in the data packet.

Optionally, the compressed information comprises file name, file size, file MD5 and last modification time, and the transmission mode and transmission progress of each compressed packet in the second data are obtained by comparing the compressed information based on the transmitted file through the target bank end, comprising the following steps:

comparing file names through the target bank end based on names in file information of each file in the transmitted files, wherein the file information is determined according to compression information of compression packets historically transmitted by the target bank end;

Determining that the transmission mode of the compressed package is the de-header transmission when the file name of the compressed package does not exist in the transmitted file of the target bank end, otherwise, comparing the file MD5 of the MD5 in the file information of the target file with the same file name by the target bank end;

determining that the transmission mode of the compression packet is the de-header transmission when the MD5 in the file information of the target file is inconsistent with the MD5, otherwise, comparing the file sizes according to the size in the file information of the target file through the target bank end;

Determining that the transmission mode of the compressed package is the transmission from the beginning when the size of the file information of the target file is inconsistent with the size of the file, otherwise, determining that the transmission mode of the compressed package is breakpoint transmission, and determining the transmission progress based on the actual file content of the target file;

And comparing the last modification time by the target shore end based on the target file, and sending an early warning prompt to the target ship end when the modification time of the target file is inconsistent with the last modification time, so as to determine the transmission mode and the transmission progress of the compressed packet in response to the selection instruction of the target ship end.

Optionally, the compressed packets of the target ship end are sequentially transmitted to the target shore end in blocks based on the sliding window, and the method comprises the following steps:

Partitioning the compressed packet at the target ship end based on the preset data size to obtain a plurality of data partitions which are sequentially arranged;

selecting a preset number of data blocks as synchronous data after the transmission progress through a sliding window;

sequentially sending each data block in the synchronous data to a target bank end so that the target bank end synchronizes the data blocks in the synchronous data one by one;

If the transmission progress is not changed, the data retransmission is carried out on the synchronous data, otherwise, the transmission progress is updated based on the synchronous progress, and the step of selecting the preset number of data blocks as synchronous data after the transmission progress is selected through the sliding window is carried out until the target bank end completes the synchronization of all the data blocks of the compressed package.

Optionally, the method further comprises the steps of:

And when the target shore ends complete the synchronization of the data blocks of the target quantity or the target shore ends receive the heartbeat request of the target ship ends, sending the synchronization progress of the data blocks of the compressed package to the target ship ends through the target shore ends.

Optionally, the method further comprises the steps of:

Periodically sending a heartbeat request to a target shore end through a target ship end;

determining network quality of the target ship end and the target ship end according to timeliness of heartbeat reply of the target ship end to the target ship end based on the heartbeat request;

And stopping data transmission from the target ship end to the target shore end when the delay of the network quality is larger than a preset threshold value.

Optionally, the method further comprises the steps of:

When the target bank end completes synchronization of all data blocks of the compressed packet, MD5 calculation is carried out on the synchronized data to obtain verification MD5;

comparing the verification MD5 with the file MD5 in the compressed information corresponding to the compressed packet;

And when the comparison results are inconsistent, retransmitting and feeding back to the target ship end.

On the other hand, the embodiment of the invention provides a ship shore weak network data synchronization device, which comprises:

The first module is used for carrying out packet compression on first data to be synchronized in the target ship end to obtain second data, wherein the second data comprises at least one compression packet and compression information corresponding to the compression packet;

The second module is used for sending the compressed information to the target bank terminal, and comparing the compressed information based on the transmitted file through the target bank terminal to obtain a transmission mode and a transmission progress of each compressed packet in the second data, wherein the transmission mode comprises head transmission and breakpoint transmission, and the transmission progress of the head transmission is 0;

The third module is used for initializing metadata at the target ship end and the target shore end respectively based on the transmission mode and the transmission progress to finish transmission preparation;

A fourth module, configured to sequentially block-transmit the compressed packets of the target ship end to the target shore end based on the sliding window;

Optionally, the apparatus further comprises:

And the fifth module is used for sending the synchronous progress of the data blocks of the compressed packet to the target ship end through the target ship end when the target ship end completes the synchronization of the data blocks of the target number or when the target ship end receives the heartbeat request of the target ship end.

Optionally, the apparatus further comprises:

a sixth module, configured to periodically send a heartbeat request to the target shore end through the target ship end;

a seventh module, configured to determine network quality of the target ship end and the target shore end according to an age of heartbeat reply from the target shore end to the target ship end based on the heartbeat request;

And an eighth module, configured to stop data transmission from the target ship end to the target shore end when the delay of the network quality is greater than a preset threshold.

Optionally, the apparatus further comprises:

a ninth module, configured to perform MD5 calculation on the synchronized data to obtain a verification MD5 when the target bank end completes synchronization of all data blocks of the compressed packet;

a tenth module, configured to compare the verification MD5 with the file MD5 in the compression information corresponding to the compression packet;

and the eleventh module is used for carrying out retransmission feedback to the target ship end when the comparison results are inconsistent.

On the other hand, the embodiment of the invention provides electronic equipment which comprises a processor and a memory, wherein the memory is used for storing a program, and the processor executes the program to realize the ship shore weak network data synchronization method.

In another aspect, an embodiment of the present invention provides a computer storage medium in which a program executable by a processor is stored, where the program executable by the processor is used to implement the method for synchronizing data of a ship-shore weak network.

The method comprises the steps of carrying out packet compression on first data to be synchronized in a target ship end to obtain second data, wherein the second data comprise at least one compression packet and compression information corresponding to the compression packet, sending the compression information to the target land end, comparing the compression information through the target land end based on transmitted files to obtain a transmission mode and transmission progress of each compression packet in the second data, wherein the transmission mode comprises head transmission and breakpoint transmission, the transmission progress of the head transmission is 0, initializing metadata at the target ship end and the target land end based on the transmission mode and the transmission progress respectively to complete transmission preparation, sequentially transmitting the compression packets of the target ship end to the target land end in blocks based on a sliding window, wherein the sliding window comprises a preset number of data blocks which are sequentially arranged, and when the target land end completes synchronization of any number of data blocks in front of the sliding window, the sliding window is moved backwards by a corresponding number of data blocks until the target land end completes synchronization of all data blocks of the compression packets. The embodiment of the invention realizes breakpoint continuous transmission based on breakpoint transmission, improves the success rate of ship-shore data synchronization, provides a strategy for sequential transmission, ensures the data integrity and improves the data synchronization speed. The invention can realize data synchronization with high efficiency.

Drawings

The accompanying drawings are included to provide a further understanding of the invention and are incorporated in and constitute a part of this specification, illustrate and do not limit the invention.

FIG. 1 is a schematic diagram of an implementation environment for data synchronization according to an embodiment of the present invention;

fig. 2 is a schematic flow chart of a method for synchronizing weak network data on a ship shore according to an embodiment of the present invention;

FIG. 3 is a schematic diagram of a communication flow between a ship end and a shore end according to an embodiment of the present invention;

fig. 4 is a schematic diagram of an expansion flow of packet compression according to an embodiment of the present invention;

Fig. 5 is a schematic diagram of an expansion flow for determining a transmission mode and a transmission progress of a compressed packet according to an embodiment of the present invention;

FIG. 6 is a schematic diagram of data synchronization interaction according to an embodiment of the present invention;

fig. 7 is a schematic diagram of data retransmission according to an embodiment of the present invention;

fig. 8 is a schematic diagram of an overall business flow of a method for synchronizing data of a weak network on a ship shore according to an embodiment of the present invention;

Fig. 9 is a schematic structural diagram of a weak network data synchronization device on a ship shore according to an embodiment of the present invention;

fig. 10 is a schematic structural diagram of an electronic device according to an embodiment of the present invention.

Detailed Description

The present invention will be described in further detail with reference to the drawings and examples, in order to make the objects, technical solutions and advantages of the present invention more apparent. It should be understood that the specific embodiments described herein are for purposes of illustration only and are not intended to limit the scope of the invention.

It should be noted that although functional block diagrams are depicted as block diagrams, and logical sequences are shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than the block diagrams in the system. The terms first/S100, second/S200, and the like in the description and in the claims and in the above-described figures, are used for distinguishing between similar objects and not necessarily for describing a particular sequential or chronological order.

Reference in the specification to "an embodiment" means that a particular feature, structure, or characteristic described in connection with the embodiment may be included in at least one embodiment of the invention. The appearances of such phrases in various places in the specification are not necessarily all referring to the same embodiment, nor are separate or alternative embodiments mutually exclusive of other embodiments. Those of skill in the art will explicitly and implicitly appreciate that the described embodiments of the invention may be combined with other embodiments.

It can be understood that the method for synchronizing the ship-shore weak network data provided by the embodiment of the invention can be applied to any computer equipment with data processing and computing capabilities, and the computer equipment can be various terminals or servers. When the computer device in the embodiment is a server, the server is an independent physical server, or a server cluster or a distributed system formed by a plurality of physical servers, or a cloud server providing cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDNs (Content Delivery Network, content distribution networks), basic cloud computing services such as big data and artificial intelligence platforms, and the like. Alternatively, the terminal is a smart phone, a tablet computer, a notebook computer, a desktop computer, or the like, but is not limited thereto.

In order to facilitate understanding of the technical solution of the present invention, first, proper nouns of technical features that may occur in the embodiments of the present invention are explained:

the VSAT satellite communication (VERY SMALL Aperture Terminal) is that the VSAT is also called a satellite small data station or a personal earth station because the VSAT is derived from a traditional satellite communication system, wherein the small refers to that the antenna caliber of small station equipment in the VSAT system is small, usually 0.3 m-1.4 m, the equipment structure is compact, solid and intelligent, the price is low, the installation is convenient, the requirement on the use environment is not high, the limitation of a ground network is avoided, and the networking is flexible.

A drill ship (DRILLING VESSEL) is a ship dedicated to drilling operations on subsea geological formations, primarily for marine geological exploration. With the deep implementation of ocean strategies, drilling and production vessels are in the international oceans such as the pacific ocean, and at this time, the communication means between the vessels and the onshore base are only satellites, and the communication flow is shown in fig. 3.

The TCP transmission control protocol (Transmission Control Protocol) is a connection-oriented, reliable, byte stream based transport layer communication protocol. TCP is intended to accommodate a layered protocol hierarchy that supports multiple network applications. Reliable communication services are provided by means of TCP between pairs of processes in host computers connected to different but interconnected computer communication networks. TCP assumes that it can obtain simple, possibly unreliable datagram services from lower level protocols. In principle, TCP should be able to operate over a variety of communication systems from hardwired to packet-switched or circuit-switched networks.

The UDP transport control protocol (User Datagram Protocol) is based on a connectionless, unreliable datagram transport mechanism, providing only source port, destination port, data length and checksum encapsulation, enabling fast and compact end-to-end communication. The method does not ensure orderly arrival, reliable transmission or flow control of data, and is suitable for application scenes which are sensitive to delay, can tolerate packet loss and have self-error correction capability, such as real-time audio and video streaming, online games, network broadcasting, DNS query and the like.

LZMA compression Algorithm in practical compression algorithms, LZMA (Lempel-Ziv-Markov chain Algorithm) is generally considered to be one that provides extremely high compression rates. It was proposed in 1998 as an improved version of the LZ77 algorithm for implementing the 7-Zip file format (.7z). LZMA uses a chained compression method and applies a modified LZ77 algorithm at the bit level to achieve excellent compression efficiency, especially when processing data with high repeatability of text, source code, etc.

LZMA2 compression algorithm LZMA2 further improves compression performance and multi-core processor support as a subsequent version of LZMA while maintaining high compression rate. In practical applications, such as using 7-Zip software, a very high compression rate is often obtained by selecting the LZMA2 compression method.

MD5 (Message-Digest Algorithm 5), which is a widely used cryptographic hash function designed to convert information of arbitrary length into a hash value of fixed length, is unidirectional, meaning that it is difficult to reverse-extrapolate the original information from the hash value. The core meaning of MD5 is to provide a data verification mechanism, which can efficiently compare whether data are consistent by calculating MD5 value for data, without directly comparing data itself. It is achieved by complex mathematical operations, even small data variations can result in large differences in the calculated MD5 values.

FIG. 1 is a schematic view of an embodiment of the invention. Referring to fig. 1, the implementation environment includes at least one terminal 102 and a server 101. The terminal 102 and the server 101 can be connected through a network in a wireless or wired mode to complete data transmission and exchange.

The server 101 may be an independent physical server, a server cluster or a distributed system formed by a plurality of physical servers, or a cloud server providing cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDNs (Content Delivery Network, content delivery networks), basic cloud computing services such as big data and artificial intelligent platforms, and the like.

In addition, server 101 may also be a node server in a blockchain network. The blockchain is a novel application mode of computer technologies such as distributed data storage, point-to-point transmission, a consensus mechanism, an encryption algorithm and the like.

The terminal 102 may be, but is not limited to, a smart phone, tablet, notebook, desktop, smart box, smart watch, etc. The terminal 102 and the server 101 may be directly or indirectly connected through wired or wireless communication, which is not limited in this embodiment of the present invention.

Based on the implementation environment shown in fig. 1, the embodiment of the present invention provides a method for synchronizing data of a weak network on a ship, and the method for synchronizing data of a weak network on a ship is described below by taking an application of the method for synchronizing data of a weak network on a ship to a server 101 as an example, and it is understood that the method for synchronizing data of a weak network on a ship can also be applied to a terminal 102.

Referring to fig. 2, fig. 2 is a flowchart of a method for synchronizing data of a weak network on a ship shore, which is applied to a server according to an embodiment of the present invention, and an execution body of the method for synchronizing data of a weak network on a ship shore may be any one of the foregoing computer devices (including a server or a terminal). Referring to fig. 2, the method includes the steps of:

s100, carrying out packet compression on first data to be synchronized in a target ship end to obtain second data;

the second data comprises at least one compression packet and compression information corresponding to the compression packet;

It should be noted that the compression information includes a file name, a file size, a file MD5 and a last modification time, in some embodiments, as shown in fig. 4, step S100 may include the steps of S101, sorting data files in the first data according to the modification time to obtain a file sequence, S102, traversing the file sequence in sequence based on a preset size reference, grouping all data files in the file sequence into a plurality of data packets, S103, compressing each data packet by a preset compression algorithm to obtain a corresponding compression packet, where the file name is obtained based on a combination of names of all data files in the data packet, the file size and the file MD5 are obtained based on a measurement and calculation of the preset algorithm, and the last modification time is determined based on the modification time of the last data file in the data packet.

Illustratively, in some embodiments, the compression of data locally for the ship side may be implemented specifically as follows:

the ship end has a plurality of files to be transmitted, mainly the test results of technical experiments, the formats include files such as tables, texts and pictures, and the high-definition videos are few. The data files need to be compressed before transmission, and the invention adopts an open-source LZMA2 compression algorithm which has very high compression rate. Compression speeds are somewhat slower than other compression algorithms, but higher compression rates are still necessary in view of satellite bandwidth. In the on-board data transmission management regulations, transmission of high definition video is not recommended. This is because the existing video is usually in a compressed format, even if the file is compressed again by using LZMA2, the file compression rate will not be too high, and the video will occupy a lot of bandwidth to affect the transmission of other files, so that the high-definition video files are usually synchronized by direct copying after the ship is on shore.

Normally, there will be a plurality of files to be synchronized, and these files are ordered according to the last modification time, and the files will be compressed in a plurality of groups. The invention groups according to 100M, that is, the sum of file sizes of each group is about 100M, if a single file exceeds 100M, then the files are grouped individually. Compression is then performed for each group, with the compressed package name being MD5, where all file names are added, thus resulting in a series of compressed packages. This grouping criteria can be custom adjusted, but it is not recommended to exceed 200M, because the larger the compressed file, the higher the file transfer failure rate will be. The compressed packets are transmitted synchronously one by one, and the invention is aimed at synchronizing data by single compressed packet.

In most cases, the compression rate of the text invention can reach about 60%, that is, the size of the compressed text invention is about 40% of the original size, and according to 100M grouping, a single compressed package is about 40M. In a practical scenario, the compression rate may even reach 70%.

S200, transmitting the compressed information to a target bank terminal, and comparing the compressed information through the target bank terminal based on the transmitted file to obtain a transmission mode and a transmission progress of each compressed packet in the second data;

The transmission mode comprises a head transmission mode and a breakpoint transmission mode, wherein the transmission progress of the head transmission mode is 0;

It should be noted that the compressed information includes file name, file size, file MD5 and last modification time; in some embodiments, as shown in fig. 5, the method for transmitting each compressed packet in the second data and the transmission progress are obtained by comparing the compressed information with the transmitted files through the target bank, which may include the steps of S201, comparing the file names with the names in the file information of each file in the transmitted files through the target bank, determining the compressed packet transmission method according to the compressed information of the compressed packets transmitted through the target bank, S202, determining the compressed packet transmission method as the header transmission when the file names of the compressed packets do not exist in the transmitted files of the target bank, otherwise, determining the compressed packet transmission method as the header transmission method according to the MD5 in the file information of the target file with the same file names through the target bank, S203, determining the compressed packet transmission method as the header transmission when the MD5 in the file information of the target file is inconsistent with the MD5, otherwise, determining the compressed packet transmission method as the header transmission method according to the size in the file information of the target file, S204, determining the compressed packet transmission method as the header transmission method when the size in the file information of the target file is inconsistent with the file size, determining the compressed packet transmission method as the header transmission method when the data is not consistent with the file size, determining the header transmission method 205, and finally, determining the modified file transmission progress is determined based on the actual file transmission method and the time is not consistent with the target bank, and the modified file transmission progress is determined according to the actual transmission progress.

Illustratively, in some embodiments, the ship end data synchronization request is initiated first, which may be implemented specifically as follows:

the ship end sends a data synchronization request, and the data synchronization request is forwarded to the shore end through a satellite, and mainly comprises detailed information of a compression packet with transmission data, wherein the detailed information comprises information such as file name, file size, file MD5, file creation time, final modification time and the like of the compression packet, and the information is mainly used for judging whether breakpoint continuous transmission is needed or not.

Then, checking the data synchronization condition, specifically, the following can be realized:

1) Checking the file name, and checking whether the file exists at the bank end. If not, this file is indicated as being the first transmission. If so, continuing to compare MD5 of the file;

2) Checking the file MD5, checking whether the bank end file MD5 is consistent with the target file MD 5. If not, it is stated that the target file is changed and retransmission is required. If so, continuing to compare the file sizes.

3) Checking the file size, if it is consistent, indicating that the file was previously transferred, a breakpoint resume may be selected. If not, it is stated that the target file is changed and retransmission is required. This step is necessary because the file is stored in the buffer before the transfer is successful, and it may happen that the buffer is tampered with or cleared.

4) Checking the final modification time, if the final modification time is consistent, indicating that the file is subjected to breakpoint continuous transmission after last compression, otherwise, indicating that the file is a new compression packet, and actively reminding a ship end.

Further, the data synchronization condition at the shore end is recovered. After the inspection is completed, the monitored condition is directly fed back to the ship end, and the file information (file name, MD5, file size, last modification time, transmission progress and the like) existing at the ship end, transmission advice (from scratch or breakpoint) and other information are mainly fed back.

S300, initializing metadata at a target ship end and a target shore end respectively based on a transmission mode and a transmission progress, and completing transmission preparation;

Illustratively, in some embodiments, first, the metadata is explicitly synchronized and initialized, and may be specifically implemented as follows:

and after receiving the feedback information of the shore end, reminding the user of the transmission mode. According to the judgment condition of the preamble step, if the files are inconsistent, only the de-header transmission can be certainly selected, and if the consistent files exist at the bank end, the de-header transmission or the breakpoint continuous transmission can be selected. In most cases, to improve transmission efficiency, breakpoint resume can be selected, or else, from scratch.

Furthermore, the metadata may be initialized as follows:

after an explicit choice of transmission from scratch or breakpoint, metadata is initialized at the ship end, including information of the currently transmitted file (file name, file ID, MD5, file size, last modification time, etc.), and current transmission progress.

The UDP single data packet is generally 1KB (1024 bytes), the invention selects 1KB to be transmitted in a blocking sequence, and the transmission progress saves the size of the transmitted file. If the transmission is started from the beginning, the transmission progress is 0, and if the transmission is continued from the breakpoint, the transmission progress is the last transmission progress.

Finally, metadata initialization reply is carried out, after metadata of the ship end is received, a part of metadata is initialized at the shore end, and after the initialization is successful, the ship end is recovered. The ship can start transmitting data after receiving the reply.

S400, sequentially transmitting the compressed packets of the target ship end to the target shore end in blocks based on the sliding window;

It should be noted that, in some embodiments, step S400 may include the steps of performing a blocking process on the compressed packet of the target ship end based on a preset data size to obtain a plurality of data blocks arranged in sequence, selecting a preset number of data blocks as synchronous data after the transmission progress through the sliding window, sequentially transmitting each data block in the synchronous data to the target shore end so as to synchronize the data blocks in the synchronous data one by the target shore end, obtaining the synchronization progress of the target shore end on the data blocks in the synchronous data, retransmitting the data of the synchronous data when the synchronization progress is unchanged from the transmission progress, otherwise, updating the transmission progress based on the synchronization progress, and returning to perform the step of selecting the preset number of data blocks as synchronous data after the transmission progress through the sliding window until the target shore end completes synchronization of all the data blocks of the compressed packet.

Illustratively, in some embodiments, the sequential block transfer may be implemented specifically as follows:

The invention selects UDP data with 1KB size for block transmission, so that a file can be divided into a plurality of blocks according to the 1KB size, and each UDP data packet comprises three parts of information of a file ID, a block ID and a data block. Referring to fig. 6, each block has three states, namely complete synchronization, in-process synchronization and waiting for synchronization, the proposed data synchronization strategy of the present invention is sequential synchronization, i.e. all data blocks are sent one by one for synchronization, and are sent strictly in block order.

The sending strategy is fixed window sending, and the maximum number of data blocks with windowSize is sent each time (the window size can be self-defined and set at least 1, the recommended range is 1-6), and the maximum number of data blocks with windowSize is in synchronization. Every time a transmission progress reply is received, the state of the data block is marked (partially modified from in-sync to complete in-sync, partially modified from waiting in-sync) based on the condition of the shore side data reception, and then the transmission window is moved and a new data block is prepared to be transmitted.

The retransmission strategy is to retransmit all missing data sequentially, namely, when the bank end is found to not receive the block, all data after the block is retransmitted automatically, when the bank end is found to not receive one block, the network is possibly congested, and the subsequent block of the block is not received at the same time, so that the block and the subsequent data are retransmitted selectively, and the advantage of doing so is to ensure that all data are stored sequentially as much as possible, and the transmission speed is improved.

The method comprises the step that a bank terminal receives data blocks, wherein the bank terminal can receive a plurality of UDP data packets, and the plurality of UDP data packets comprise three parts of information such as file IDs, block IDs and data blocks. And determining which file is the data of the file according to the file ID, determining the blocking position according to the blocking ID, and storing the final bank end according to the blocking ID sequence. Meanwhile, the shore end may receive the data packets with the same file ID and partition ID for many times, and if the data packets are already stored, duplicate data packets are directly discarded, and the data packets may be retransmitted data packets. Every time a packet is received, an attempt is made to update the transmission schedule and to synchronize the latest transmission schedule according to rules, which are described later.

Referring to the data transmission flow of fig. 7, in the case of normal data synchronization, repeated interaction between data transmission and data reception is performed, if data loss is found, the retransmission state is immediately entered, the current transmission progress is stopped, the lost data block is retransmitted, and then whether the data is received is waited for confirmation, and if the data is not received yet, the retransmission is continued until the data is received.

In some embodiments, the method may further include the step of sending, by the target shore, a synchronization progress of the data blocks of the compressed packet to the target ship when the target shore completes synchronization of the target number of data blocks, or when the target shore receives a heartbeat request from the target ship.

Illustratively, in some embodiments, the shore end may periodically reply to the data chunk reception. The shore end continuously receives a large number of data packets and stores the data packets in sequence, but the shore end does not send a reply every time a data block is received, so that the data transmission efficiency is reduced. The bank end can immediately reply to the data block receiving condition when the following two conditions are satisfied:

Case1 when received When (if not divisible, the whole number is taken upwards) the data block reception is resumed immediately, wherein the valid data block does not include a duplicate packet. For example, if WindowsSize is 3, the data receiving sequence is packet 1 (block 5), packet 2 (block 5), packet 3 (block 6), packet 4 (block 6), and when packet 3 is received, 2 valid data blocks (block 5 and block 6) are received, and the data block receiving situation is recovered. Packet 2 and packet 4 are duplicate packets and are discarded. It can be seen here that in general, the number of data retransmissions at the ship end is mostlyAnd each.

Case2, when receiving the heartbeat request, immediately replying to the data block receiving condition.

The progress of the reply includes information such as a heartbeat SeqID, a file ID, a continuously stored latest ID, and the like. For example. If blocks 1-6 have been received, blocks 8-9, the latest ID stored consecutively at this time is block 6.

In some embodiments, the method further comprises the steps of periodically sending a heartbeat request to the target shore through the target ship end, determining network quality of the target ship end and the target shore according to timeliness of heartbeat reply of the target ship end to the target ship end based on the heartbeat request, and stopping data transmission from the target ship end to the target shore when delay of the network quality is greater than a preset threshold.

For example, in some embodiments, the ship end may send a heartbeat request periodically, where the request content includes information of a heartbeat SeqID, a file ID, a latest sending progress, and each time a heartbeat is sent, seqid= SeaID +1. The transmission frequency SendFrequency can be custom controlled, typically once every 1s (500 ms to 1s is recommended). The main reasons for setting the heartbeat request are as follows:

1) And judging the network state. The ship end can judge the network state through the heartbeat request reply, if the heartbeat reply is not received for a plurality of times, the network quality is poor, and the sending progress can be adjusted at the moment, and even the sending is stopped.

2) The transmission speed is improved. The shore end can immediately synchronize the data transmission progress after receiving the heartbeat, and the ship end can immediately adjust the sending strategy after receiving the latest progress. Because the heartbeat is periodic, the mechanism well avoids the extreme situation that the ship end can not acquire the data progress forever due to the loss of other data packets, accelerates the movement of the ship end sending window, and improves the transmission speed.

The shore end processes the heartbeat request and replies information such as the heartbeat SeqID, the file ID, the continuously stored latest ID and the like to the ship end. The ship end needs to make two operations of network quality judgment and data loss judgment according to the heartbeat reply.

And judging the network quality, namely, because the heartbeat packet is very concise, the shore end sending and the ship end receiving processing hardly consume time, and the network quality can be directly fed back once the heartbeat comes. If M times SendFrequency are consecutive, no heartbeat reply packet (M may be configured in a custom manner, and is generally 3) is received, that is, M heartbeat packets are not received, at this time, it may be determined that the network quality is particularly poor, all data transmission may be stopped immediately, but the transmission frequency of the heartbeat packets is still unchanged.

If a heartbeat reply is received, assuming that the received heartbeat reply is SeqID1, the latest heartbeat sent is SeqID2, if SeqID2-SeqID1> =m, this indicates that the network quality is particularly poor, resulting in a long heartbeat cycle time, and at this time, the data sending is still stopped. If SeqID2-SeqID1< M, the network quality is adequate, and the transmission of the data packet can be resumed or continued. In general, heartbeat is a bottom protection means for judging the network at the ship end, so that the transmission speed can be improved, and breakpoint retransmission can be performed after the network is recovered to be normal in time.

In some embodiments, the method may further include the steps of performing MD5 calculation on the synchronized data to obtain a verification MD5 when the target shore end completes synchronization of all data blocks of the compressed packet, comparing the verification MD5 with a file MD5 in compression information corresponding to the compressed packet, and performing retransmission feedback to the target shore end when the comparison result is inconsistent.

Illustratively, in some embodiments, based on the latest data transmission progress, the ship end determines that all data is received, which indicates that the data synchronization is successful. At this point a verification request is sent, the requested content being mainly the information of the currently transmitted file (file name, file ID, MD5, file size, last modification time, etc.).

After receiving the verification request, the ship end recalculates the MD5 of the file, compares the calculated MD5 with the requested MD5, and indicates that the file transmission is successful if the calculated MD5 is consistent with the requested MD 5. If not, it is indicated that some of the packets may be tampered with. MD5 is consistent in most cases.

The ship end replies to MD5 verification, mainly the information of the current transmission file (file name, file ID, MD5, file size, last modification time, etc.). After receiving the reply, the bank end records the transmission condition, marks the file to finish synchronization, and informs the user. If MD5 is consistent, the user is notified that the file transfer was successful, otherwise the user is notified that the intermediate data was tampered with, requiring resynchronization (where the data is not reset and emptied, requiring active retransmission by the user).

For the purpose of illustrating the principles of the present invention in detail, the following general flow chart of the present invention is described in connection with certain specific embodiments, and it is to be understood that the following is illustrative of the principles of the present invention and is not to be construed as limiting the present invention.

Referring to fig. 8, the present invention proposes a method for synchronizing data (sequence) from ship end to shore end, and the specific steps may be implemented as follows:

And 1, locally compressing the data. The ship end has a plurality of files to be transmitted, mainly the test results of technical experiments, the formats include files such as tables, texts and pictures, and the high-definition videos are few. The data files need to be compressed before transmission, and the invention adopts an open-source LZMA2 compression algorithm which has very high compression rate. Compression speeds are somewhat slower than other compression algorithms, but higher compression rates are still necessary in view of satellite bandwidth. In the on-board data transmission management regulations, transmission of high definition video is not recommended. This is because the existing video is usually in a compressed format, even if the file is compressed again by using LZMA2, the file compression rate will not be too high, and the video will occupy a lot of bandwidth to affect the transmission of other files, so that the high-definition video files are usually synchronized by direct copying after the ship is on shore.

Regarding grouping, for example, if 7 files of files a (size 80M), B (size 17M), C (size 60M), D (size 60M), E (size 70M), F (size 80M), G (170M), etc. have been ordered by the last modification time, then the grouping is as follows:

1) A (size 80M) and B (size 17M) are a group, because a+b=97m, around 100M, the compressed packet name is MD5 (AB);

2) C (60M in size) and D (60M in size) are a set, C is 60M, a distance of 100M is 40M, but c+d=120m, a distance of 100M is only 20M, so C and D are a set, and the compressed packet name is MD5 (CD);

3) E (size 70M) is a group alone, E is 70M, distance 100M is 30M, but e+f=150m, distance 100M is 50M more, so E is a group alone, compressed packet name is MD5 (E);

4) F (80M) is a group alone for the same reasons as E;

5) G (size 170M) is a group alone;

to sum up, 5 groups were separated, each group was packed using LZMA2 compression.

And 2, initiating a ship end data synchronization request. The ship end sends a data synchronization request, and the data synchronization request is forwarded to the shore end through a satellite, and mainly comprises detailed information of a compression packet with transmission data, wherein the detailed information comprises information such as file name, file size, file MD5, file creation time, final modification time and the like of the compression packet, and the information is mainly used for judging whether breakpoint continuous transmission is needed or not.

And 3, checking the data synchronization condition. After the shore end receives the data synchronization request, the transmitted file is checked, and the judging process is mainly as follows:

And 4, recovering the synchronous condition of the shore data. After the inspection is completed, the monitored condition is directly fed back to the ship end, and the file information (file name, MD5, file size, last modification time, transmission progress and the like) existing at the ship end, transmission advice (from scratch or breakpoint) and other information are mainly fed back.

And 5, defining a synchronous mode and initializing metadata. And after receiving the feedback information of the shore end, reminding the user of the transmission mode. According to the judgment condition in the step 3, if the files are inconsistent, only the de-header transmission can be selected certainly, and if the consistent files exist at the bank end, the de-header transmission or the breakpoint continuous transmission can be selected. In most cases, to improve transmission efficiency, breakpoint resume can be selected, or else, from scratch.

And 6, initializing metadata. After an explicit choice of transmission from scratch or breakpoint, metadata is initialized at the ship end, including information of the currently transmitted file (file name, file ID, MD5, file size, last modification time, etc.), and current transmission progress.

And 7, initializing and replying the metadata. After receiving the metadata of the ship end, initializing a part of metadata at the shore end, and recovering the ship end after the initialization is successful. The ship can start transmitting data after receiving the reply.

And 8, sequentially transmitting the data blocks. The invention selects UDP data with 1KB size for block transmission, so that a file can be divided into a plurality of blocks according to the 1KB size, and each UDP data packet comprises three parts of information of a file ID, a block ID and a data block. Referring to fig. 6, each block has three states, namely complete synchronization, in-process synchronization and waiting for synchronization, the proposed data synchronization strategy of the present invention is sequential synchronization, i.e. all data blocks are sent one by one for synchronization, and are sent strictly in block order.

The retransmission strategy is to retransmit all missing data sequentially, namely, when the bank end is found to not receive the block, all data after the block is retransmitted automatically, when the bank end is found to not receive one block, the network is possibly congested, and the subsequent block of the block is not received at the same time, so that the block and the subsequent data are retransmitted selectively, and the advantage of doing so is to ensure that all data are stored sequentially as much as possible, and the transmission speed is improved. Regarding how to determine bluff whether a data block is received or not, this later step will give a determination rule in step 12, which is not developed first.

For example, referring to fig. 6, assuming WindowSize is 3, after synchronization of block 5 is completed, blocks 6, 7, and 8 are automatically transmitted, at which time a maximum of 3 data blocks are in synchronization. At this time, it is found that the synchronization of the block 6 is completed, and the block 7 and the block 8 are still in the synchronization state, then the block 9 is sent immediately, and the sending policy is that the full force guarantees that at most 3 data blocks are in the synchronization state. If the bank end is found not to receive the block 6, the block 7 and the block 8 are retransmitted immediately, if the bank end is found not to receive the block 7, the block 7 and the block 8 are retransmitted immediately, and if the bank end is found not to receive the block 8, the block 8 is retransmitted immediately.

And 9, receiving the data blocks. The shore end receives a plurality of UDP data packets, wherein the UDP data packets comprise three parts of information such as file IDs, block IDs, data blocks and the like. And determining which file is the data of the file according to the file ID, determining the blocking position according to the blocking ID, and storing the final bank end according to the blocking ID sequence. Meanwhile, the shore end may receive the data packets with the same file ID and partition ID for many times, and if the data packets are already stored, duplicate data packets are directly discarded, and the data packets may be retransmitted data packets. Every time a packet is received, an attempt is made to update the transmission schedule and to synchronize the latest transmission schedule according to rules, which are described in step 10.

And 10, periodically replying to the data block receiving condition. The shore end continuously receives a large number of data packets and stores the data packets in sequence, but the shore end does not send a reply every time a data block is received, so that the data transmission efficiency is reduced. The bank end can immediately reply to the data block receiving condition when the following two conditions are satisfied:

With respect to the data retransmission strategy of step 8, there is a bit to be discussed here. The shore end can send data according to the fixed rules of Case1 and Case2 and receive the situation in blocks, when the ship end receives the transmission progress, the ship end can firstly pause sending data, and the data ending progress is processed preferentially, and the maximum value is taken as the latest progress. Based on the latest progress, data retransmission is selected or data is normally transmitted. For example, if the received data progress is the block 6, the block 7, and the block 8, respectively, the data reception progress at this time is the block 8, and then retransmission or normal transmission is started from the block 9.

And step 11, sending a heartbeat request. The ship end can periodically send heartbeat requests, and the request content comprises heartbeat SeqID, file ID, latest sending progress and other information, and every time a heartbeat is sent, seqID= SeaID +1. The transmission frequency SendFrequency can be custom controlled, typically once every 1s (500 ms to 1s is recommended). The main reasons for setting the heartbeat request are as follows:

And step 12, recovering the heartbeat. The shore end receives the heartbeat request and processes the heartbeat request, and replies information such as the heartbeat SeqID, the file ID, the continuously stored latest ID and the like to the ship end. The ship end needs to make two operations of network quality judgment and data loss judgment according to the heartbeat reply.

For example, the ship-side heartbeat sending and receiving conditions are assumed to be sending heartbeat 1, receiving heartbeat reply 1, sending heartbeat 2, receiving heartbeat reply 2, sending heartbeat 3 and receiving heartbeat reply 3. The normal situation of the data network is that the network delay of the land is generally 30ms to 50ms, and the satellite transmission is considered, the time delay of the ship-shore network is estimated to be 200ms, namely, the reply of the heartbeat 1 can be received after 200ms, so that the data network belongs to the normal situation of the network.

The sending and receiving conditions of the ship-end heartbeat are assumed to be that the sending heartbeat 1, the sending heartbeat 2, the receiving heartbeat reply 1, the receiving heartbeat reply 2, the sending heartbeat 3 and the receiving heartbeat reply 3. This situation is a slightly worse network but may also continue to transmit.

Assuming that the sending and receiving conditions of the ship end heartbeat are that the sending heartbeat 1, the sending heartbeat 2, the sending heartbeat 3 and the sending heartbeat 4 are the sending heartbeat, and the continuous M heartbeat replies are not received at the moment, which means that the network quality is poor, and all data sending can be stopped immediately.

Assuming that the sending and receiving condition of the ship side heartbeat is that the heartbeat 4 is sent and the heartbeat reply 1 is received, at the moment, although the heartbeat reply is received, the reply is very early, and all data sending can still be stopped.

The condition of sending and receiving the heartbeat at the ship end is assumed to be that the heartbeat 4 is sent, the heartbeat reply 1 is received and the heartbeat reply 4 is received, and the condition of the network is indicated to be good at the moment, so that the data sending can be recovered.

And under normal conditions, the data transmission progress is continuously increased, so that the ship end can continuously move the transmission window and continuously transmit data. However, when the network is particularly poor, the situation of data packet loss may occur, and the data loss is determined to be divided into two cases

1) If no heartbeat reply packet (M may be configured in a custom manner, and is generally 3) is received in a continuous m× SendFrequency time, it is determined that all data in the transmission window is lost, and the data packet transmission is stopped. And the network resumes and resends. This situation is a poor network and requires that the data transmission be stopped, at which point it is determined that the data is lost.

2) If no transmission progress reply (which can be custom set, typically 200ms-1000ms, typically 400 ms) is received at the post-transmission interval LostInterval, then a data loss is determined. This situation is a poor network where data is lost with a high probability, and at this time, it is determined that data is lost and the lost data packets are retransmitted immediately.

It is noted that fig. 1 shows a data synchronization interaction diagram for describing a basic flow, wherein steps 8 to 12 belong to a synchronization data phase, and are repeatedly performed until all data blocks are successfully synchronized.

And 13, successfully synchronizing the data and sending a verification request. Based on the latest data transmission progress, the ship side judges that all data are received, and then indicates that the data synchronization is successful. At this point a verification request is sent, the requested content being mainly the information of the currently transmitted file (file name, file ID, MD5, file size, last modification time, etc.).

And 14, checking the data locally. After receiving the verification request, the ship end recalculates the MD5 of the file, compares the calculated MD5 with the requested MD5, and indicates that the file transmission is successful if the calculated MD5 is consistent with the requested MD 5. If not, it is indicated that some of the packets may be tampered with. MD5 is consistent in most cases.

And 15, replying to the data verification condition. The ship end replies to MD5 verification, mainly the information of the current transmission file (file name, file ID, MD5, file size, last modification time, etc.). After receiving the reply, the bank end records the transmission condition, marks the file to finish synchronization, and informs the user. If MD5 is consistent, the user is notified that the file transfer was successful, otherwise the user is notified that the intermediate data was tampered with, requiring resynchronization (where the data is not reset and emptied, requiring active retransmission by the user).

When steps 13 through 15 are completed, the file is marked as synchronized and another file is transferred. It should be explained that in an extreme case, if the file is just successfully transmitted, the network is suddenly disconnected, the ship end cannot receive the data check reply at a later time, the file is still in a synchronous state, and after the network is recovered to be normal, the ship end can resend the check request until the check reply is received, and the transmission flow of the file is really finished.

In summary, the whole transmission is divided into three phases:

1) The preparation stage comprises steps 1 to 7, wherein the stage mainly comprises the steps of determining information of a shore-side transmission file, determining whether the file is transmitted from the beginning or the breakpoint continuous transmission, and belongs to a data preparation stage before transmission;

2) And the synchronization stage comprises steps 8 to 12, wherein the stage mainly synchronizes data and is responsible for data synchronization and retransmission, and the transmission rhythm is dynamically adjusted through the heartbeat, so that the file transmission efficiency is improved.

3) The verification phase, which is mainly the last data verification, includes steps 13 to 15.

Compared with the prior art, the invention at least has the following beneficial effects:

1) And idle bandwidth is utilized as much as possible, so that the data transmission speed is improved, bandwidth competing with technicians is avoided, and user experience is improved.

2) And the breakpoint continuous transmission is supported, and the shore data synchronization success rate is improved.

3) Based on UDP protocol, the strategy of sequential transmission is provided, which not only ensures the data integrity, but also improves the data synchronization speed.

On the other hand, as shown in fig. 9, an embodiment of the present invention provides a device 900 for synchronizing weak network data on a ship shore, which may include:

The first module 901 is configured to perform packet compression on first data to be synchronized in a target ship end to obtain second data, where the second data includes at least one compression packet and compression information corresponding to the compression packet;

A second module 902, configured to send compressed information to a target bank, and compare the compressed information based on the transmitted file by the target bank to obtain a transmission mode and a transmission progress of each compressed packet in the second data, where the transmission mode includes a de-header transmission and a breakpoint transmission, and the transmission progress of the de-header transmission is 0;

A third module 903, configured to initialize metadata at a target ship end and a target shore end based on a transmission manner and a transmission progress, so as to complete transmission preparation;

a fourth module 904, configured to sequentially block-transmit the compressed packets of the target ship end to the target shore end based on the sliding window;

In some embodiments, the apparatus may further include:

The content of the method embodiment of the invention is suitable for the device embodiment, the specific function of the device embodiment is the same as that of the method embodiment, and the achieved beneficial effects are the same as those of the method.

On the other hand, the embodiment of the invention also provides electronic equipment, which comprises a memory and a processor, wherein the memory stores a computer program, and the processor realizes the hydrate stability domain bottom boundary prediction method when executing the computer program. The electronic equipment can be any intelligent terminal including a tablet personal computer, a vehicle-mounted computer and the like.

It can be understood that the content in the above method embodiment is applicable to the embodiment of the present apparatus, and the specific functions implemented by the embodiment of the present apparatus are the same as those of the embodiment of the above method, and the achieved beneficial effects are the same as those of the embodiment of the above method.

As shown in fig. 10, fig. 10 illustrates a hardware structure of an electronic device 1000 of another embodiment, and the electronic device 1000 includes:

The processor 1001 may be implemented by using a general purpose CPU (Central Processing Unit ), a microprocessor, an Application SPECIFIC INTEGRATED Circuit (ASIC), or one or more integrated circuits, etc. to execute related programs to implement the technical solution provided by the embodiments of the present invention;

the Memory 1002 may be implemented in the form of a Read Only Memory (ROM), a static storage device, a dynamic storage device, or a random access Memory (Random Access Memory, RAM). The memory 1002 may store an operating system and other application programs, and when the technical solutions provided in the embodiments of the present disclosure are implemented by software or firmware, relevant program codes are stored in the memory 1002, and the processor 1001 invokes a network node population optimization method for executing the embodiments of the present disclosure;

An input/output interface 1003 for implementing information input and output;

the communication interface 1004 is configured to implement communication interaction between the present device and other devices, and may implement communication in a wired manner (e.g. USB, network cable, etc.), or may implement communication in a wireless manner (e.g. mobile network, WIFI, bluetooth, etc.);

A bus 1005 for transferring information between the various components of the device (e.g., the processor 1001, memory 1002, input/output interface 1003, and communication interface 1004);

Wherein the processor 1001, the memory 1002, the input/output interface 1003, and the communication interface 1004 realize communication connection between each other inside the device through the bus 1005.

The above described embodiments of the electronic device are merely illustrative, wherein the units described as separate components may or may not be physically separate, i.e. may be located in one place, or may be distributed over a plurality of network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

The content of the method embodiment of the invention is suitable for the electronic equipment embodiment, the functions of the electronic equipment embodiment are the same as those of the method embodiment, and the achieved beneficial effects are the same as those of the method.

Another aspect of the embodiments of the present invention also provides a computer-readable storage medium storing a program that is executed by a processor to implement the foregoing method.

It should be noted that, the computer readable medium shown in the embodiments of the present invention may be a computer readable signal medium or a computer readable storage medium, or any combination of the two. The computer readable storage medium can be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or a combination of any of the foregoing. More specific examples of a computer-readable storage medium may include, but are not limited to, an electrical connection having one or more wires, a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-Only Memory (ROM), an erasable programmable read-Only Memory (Erasable Programmable Read Only Memory, EPROM), a flash Memory, an optical fiber, a portable compact disc read-Only Memory (Compact Disc Read to Only Memory, CD to ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In the context of this document, a computer readable storage medium may be any tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device. In the present invention, however, the computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, with the computer-readable program code embodied therein. Such a propagated data signal may take any of a variety of forms, including, but not limited to, electro-magnetic, optical, or any suitable combination of the foregoing. A computer readable signal medium may also be any computer readable medium that is not a computer readable storage medium and that can communicate, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device. Program code embodied on a computer readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wireline, etc., or any suitable combination of the foregoing.

The content of the method embodiment of the invention is applicable to the computer readable storage medium embodiment, the functions of the computer readable storage medium embodiment are the same as those of the method embodiment, and the achieved beneficial effects are the same as those of the method.

Embodiments of the present invention also disclose a computer program product or computer program comprising computer instructions stored in a computer readable storage medium. The computer instructions may be read from a computer-readable storage medium by a processor of a computer device, and executed by the processor, to cause the computer device to perform the foregoing method.

The flowcharts and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to various embodiments of the present invention. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of code, which comprises one or more executable instructions for implementing the specified logical function(s). It should also be noted that, in some alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams or flowchart illustration, and combinations of blocks in the block diagrams or flowchart illustration, can be implemented by special purpose hardware-based systems which perform the specified functions or acts, or combinations of special purpose hardware and computer instructions.

It should be noted that although in the above detailed description several modules of a device for action execution are mentioned, such a division is not mandatory. Indeed, the features and functions of two or more modules or units described above may be embodied in one module or unit in accordance with embodiments of the invention. Conversely, the features and functions of one module or unit described above may be further divided into a plurality of modules or units to be embodied.

From the above description of embodiments, those skilled in the art will readily appreciate that the example embodiments described herein may be implemented in software, or may be implemented in software in combination with the necessary hardware. Thus, the technical solution according to the embodiments of the present invention may be embodied in the form of a software product, which may be stored in a non-volatile storage medium (may be a CD to ROM, a U-disc, a mobile hard disk, etc.) or on a network, and includes several instructions to cause a computing device (may be a personal computer, a server, a touch terminal, or a network device, etc.) to perform the method according to the embodiments of the present invention.

In some alternative embodiments, the functions/acts noted in the block diagrams may occur out of the order noted in the operational illustrations. For example, two blocks shown in succession may in fact be executed substantially concurrently or the blocks may sometimes be executed in the reverse order, depending upon the functionality/acts involved. Furthermore, the embodiments presented and described in the flowcharts of the present invention are provided by way of example in order to provide a more thorough understanding of the technology. The disclosed methods are not limited to the operations and logic flows presented by the present invention. Alternative embodiments are contemplated in which the order of various operations is changed, and in which sub-operations described as part of a larger operation are performed independently.

Furthermore, while the invention is described in the context of functional modules, it should be appreciated that, unless otherwise indicated, one or more of the functions and/or features may be integrated in a single physical device and/or software module or may be implemented in separate physical devices or software modules. It will also be appreciated that a detailed discussion of the actual implementation of each module is not necessary to an understanding of the present invention. Rather, the actual implementation of the various functional modules in the apparatus disclosed in the present invention will be understood within the ordinary skill of the engineer in view of their attributes, functions and internal relationships. Accordingly, one of ordinary skill in the art can implement the invention as set forth in the claims without undue experimentation. It is also to be understood that the specific concepts disclosed are merely illustrative and are not intended to be limiting upon the scope of the invention, which is to be defined in the appended claims and their full scope of equivalents.

The functions, if implemented in the form of software functional units and sold or used as a stand-alone product, may be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention may be embodied essentially or in a part contributing to the prior art or in a part of the technical solution in the form of a software product stored in a storage medium, comprising several instructions for causing a computer device (which may be a personal computer, a server, a network device, etc.) to perform all or part of the steps of the method of the embodiments of the present invention. The storage medium includes a usb disk, a removable hard disk, a Read-Only Memory (ROM), a random-access Memory (RAM, random Access Memory), a magnetic disk, an optical disk, or other various media capable of storing program codes.

Logic and/or steps represented in the flowcharts or otherwise described herein, e.g., a ordered listing of executable instructions for implementing logical functions, can be embodied in any computer-readable medium for use by or in connection with an instruction execution apparatus, device, or apparatus, such as a computer-based apparatus, processor-containing apparatus, or other apparatus that can fetch the instructions from the instruction execution apparatus, device, or apparatus and execute the instructions. For the purposes of this description, a "computer-readable medium" can be any means that can contain, store, communicate, propagate, or transport the program for use by or in connection with the instruction execution apparatus, device, or apparatus.

More specific examples (a non-exhaustive list) of the computer-readable medium would include an electrical connection (an electronic device) having one or more wires, a portable computer diskette (a magnetic device), a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber device, and a portable compact disc read-only memory (CDROM). Additionally, the computer-readable medium may even be paper or other suitable medium upon which the program is printed, as the program may be electronically captured, via, for instance, optical scanning of the paper or other medium, then compiled, interpreted or otherwise processed in a suitable manner, if necessary, and then stored in a computer memory.

It is to be understood that portions of the present invention may be implemented in hardware, software, firmware, or a combination thereof. In the above embodiments, the various steps or methods may be implemented in software or firmware stored in a memory and executed by a suitable instruction execution device. For example, if implemented in hardware, as in another embodiment, may be implemented using any one or combination of techniques known in the art, discrete logic circuits with logic gates for implementing logic functions on data signals, application specific integrated circuits with appropriate combinational logic gates, programmable Gate Arrays (PGAs), field Programmable Gate Arrays (FPGAs), and the like.

In the description of the present specification, a description referring to terms "one embodiment," "some embodiments," "examples," "specific examples," or "some examples," etc., means that a particular feature, structure, material, or characteristic described in connection with the embodiment or example is included in at least one embodiment or example of the present invention. In this specification, schematic representations of the above terms do not necessarily refer to the same embodiments or examples. Furthermore, the particular features, structures, materials, or characteristics described may be combined in any suitable manner in any one or more embodiments or examples.

Although embodiments of the present invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made therein without departing from the spirit and scope of the invention as defined by the appended claims and their equivalents.

While the preferred embodiment of the present invention has been described in detail, the present invention is not limited to the embodiments, and those skilled in the art can make various equivalent modifications or substitutions without departing from the spirit of the present invention, and the equivalent modifications or substitutions are intended to be included in the scope of the present invention as defined in the appended claims.

Claims

1. A ship-shore weak network data synchronization method, characterized in that it comprises the following steps:

Grouping and compressing the first data to be synchronized in the target ship end to obtain second data; the second data includes at least one compressed package and compression information corresponding to the compressed package;

The compressed information is sent to the target shore end, and the target shore end compares the compressed information based on the transmitted file to obtain the transmission mode and transmission progress of each compressed packet in the second data; the transmission mode includes transmission from the beginning and transmission at a breakpoint, and the transmission progress of the transmission from the beginning is 0;

Based on the transmission mode and the transmission progress, initializing metadata at the target ship end and the target shore end respectively to complete transmission preparation;

Transmitting the compressed packets of the target ship end to the target shore end in blocks in sequence based on a sliding window;

Wherein, the sliding window includes a preset number of data blocks arranged in sequence. When the target shore completes the synchronization of any number of the data blocks in front of the sliding window, the sliding window is moved back by a corresponding number of the data blocks until the target shore completes the synchronization of all the data blocks of the compressed package.

2. The ship-shore weak network data synchronization method according to claim 1, characterized in that the compressed information includes file name, file size, file MD5 and last modification time; the first data to be synchronized in the target ship end is grouped and compressed to obtain the second data, comprising the following steps:

Sort the data files in the first data according to modification time to obtain a file sequence;

Based on a preset size benchmark, sequentially traverse the file sequence and group all the data files in the file sequence into a plurality of data groups;

Perform file compression on each of the data groups using a preset compression algorithm to obtain the corresponding compressed package;

Among them, the file name is obtained based on the combination of the names of all the data files in the data group, the file size and the file MD5 are calculated based on a preset algorithm, and the last modification time is determined based on the modification time of the last data file in the data group.

3. The ship-shore weak network data synchronization method according to claim 1 is characterized in that the compressed information includes file name, file size, file MD5 and last modification time; the target shore end compares the compressed information based on the transmitted file to obtain the transmission mode and transmission progress of each compressed package in the second data, comprising the following steps:

The target shore end compares the file name based on the name in the file information of each file in the transmitted file; the file information is determined according to the compression information of the compressed package historically transmitted by the target ship end;

When the file name of the compressed package does not exist in the transmitted file of the target shore, determine that the transmission mode of the compressed package is the retransmission; otherwise, compare the file MD5 with the MD5 in the file information of the target file with the same file name by the target shore;

When the MD5 in the file information of the target file is inconsistent with the MD5 of the file, determining that the transmission mode of the compressed package is the retransmission; otherwise, comparing the file size based on the size in the file information of the target file by the target shore;

When the size in the file information of the target file is inconsistent with the file size, determining that the transmission mode of the compressed package is the re-scratch transmission; otherwise, determining that the transmission mode of the compressed package is the breakpoint transmission, and determining the transmission progress based on the actual file content of the target file;

The target shore end compares the last modification time based on the target file. When the modification time of the target file is inconsistent with the last modification time, an early warning reminder is sent to the target ship end, and then the transmission mode and the transmission progress of the compressed package are determined in response to the selection instruction of the target ship end.

4. The ship-shore weak network data synchronization method according to claim 1, characterized in that the sequential block transmission of the compressed package of the target ship end to the target shore end based on the sliding window comprises the following steps:

The compressed packet of the target ship is processed into blocks based on a preset data size to obtain a plurality of sequentially arranged data blocks;

Selecting the preset number of data blocks after the transmission progress as synchronization data through the sliding window;

Sending each of the data blocks in the synchronization data to the target shore end in sequence, so that the target shore end synchronizes the data blocks in the synchronization data one by one; obtaining the synchronization progress of the data blocks in the synchronization data by the target shore end;

When the synchronization progress does not change compared to the transmission progress, the synchronization data is retransmitted; otherwise, the transmission progress is updated based on the synchronization progress, and the step of selecting the preset number of data blocks after the transmission progress as the synchronization data through the sliding window is returned to execute until the target shore completes the synchronization of all the data blocks of the compressed package.

5. The ship-shore weak network data synchronization method according to claim 1, characterized in that the method further comprises the following steps:

When the target shore end completes the synchronization of the target number of data blocks, or when the target shore end receives the heartbeat request of the target ship end, the synchronization progress of the data blocks of the compressed package is sent to the target ship end through the target shore end.

6. The ship-shore weak network data synchronization method according to claim 1, characterized in that the method further comprises the following steps:

Periodically sending a heartbeat request to the target shore end through the target ship end;

Determine the network quality of the target ship end and the target shore end according to the timeliness of the heartbeat reply of the target shore end to the target ship end based on the heartbeat request;

When the delay of the network quality is greater than a preset threshold, the data transmission from the target ship end to the target shore end is stopped.

7. The ship-shore weak network data synchronization method according to claim 1, characterized in that the method further comprises the following steps:

When the target shore end completes synchronization of all the data blocks of the compressed package, MD5 calculation is performed on the synchronized data to obtain verification MD5;

Compare the verification MD5 with the file MD5 in the compressed information corresponding to the compressed package;

When the comparison results are inconsistent, retransmission feedback is performed to the target ship end.

8. A ship-shore weak network data synchronization device, characterized by comprising:

The first module is used to group and compress the first data to be synchronized in the target ship end to obtain second data; the second data includes at least one compressed package and compression information corresponding to the compressed package;

The second module is used to send the compressed information to the target shore end, and the target shore end compares the compressed information based on the transmitted file to obtain the transmission mode and transmission progress of each compressed packet in the second data; the transmission mode includes transmission from the beginning and breakpoint transmission, and the transmission progress of the transmission from the beginning is 0;

The third module is used to initialize metadata at the target ship end and the target shore end respectively based on the transmission mode and the transmission progress to complete the transmission preparation;

A fourth module is used to sequentially transmit the compressed packets of the target ship end to the target shore end in blocks based on a sliding window;

9. An electronic device, comprising a processor and a memory;

The memory is used to store programs;

The processor executes the program to implement the method according to any one of claims 1 to 7.

10. A computer storage medium storing a program executable by a processor, wherein the program executable by the processor is used to implement the method according to any one of claims 1 to 7 when executed by the processor.