+

WO2012005609A1 - Système de transfert de fichier - Google Patents

Système de transfert de fichier Download PDF

Info

Publication number
WO2012005609A1
WO2012005609A1 PCT/NZ2011/000126 NZ2011000126W WO2012005609A1 WO 2012005609 A1 WO2012005609 A1 WO 2012005609A1 NZ 2011000126 W NZ2011000126 W NZ 2011000126W WO 2012005609 A1 WO2012005609 A1 WO 2012005609A1
Authority
WO
WIPO (PCT)
Prior art keywords
computer
chunk
transfer
file
sender
Prior art date
Application number
PCT/NZ2011/000126
Other languages
English (en)
Inventor
Valentine Boiarkine
Original Assignee
Blade Limited
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Blade Limited filed Critical Blade Limited
Publication of WO2012005609A1 publication Critical patent/WO2012005609A1/fr

Links

Classifications

    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L67/00—Network arrangements or protocols for supporting network services or applications
    • H04L67/01—Protocols
    • H04L67/06—Protocols specially adapted for file transfer, e.g. file transfer protocol [FTP]
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L1/00—Arrangements for detecting or preventing errors in the information received
    • H04L1/12—Arrangements for detecting or preventing errors in the information received by using return channel
    • H04L1/16—Arrangements for detecting or preventing errors in the information received by using return channel in which the return channel carries supervisory signals, e.g. repetition request signals
    • H04L1/18—Automatic repetition systems, e.g. Van Duuren systems
    • H04L1/1829—Arrangements specially adapted for the receiver end

Definitions

  • the invention relates to a file transfer system. More particularly it relates to a file transfer system for transferring files over a network.
  • File transfer systems allow files of various sizes to be transferred from one location to another. These systems usually involve transferring files electronically over a network to a server utilizing file transfer protocols.
  • a common prior art method of transferring files is utilizing an internet, WAN or LAN server where files are uploaded to a server by a sender and the uploaded files are downloaded by recipients when required.
  • Dedicated software such as internet download manager software and most internet browsers allow files to be uploaded to and downloaded from servers in this manner.
  • the reliability and speed of the download/upload of files change depending on the network and server type/settings and also the transfer conditions. Hence various methods have been used in the prior art to transfer files with increasing speed and reliability.
  • oc define a maximum number of streams that can be uploaded/downloaded at a given time.
  • chunk size is usually predetermined by software or by a user and does not allow for any flexibility in chunk size if the transfer conditions become altered or failed during a transfer of a large file.
  • Large files present particular problems because the file transfer time is often so long that transfer conditions will alter during the transfer, and any initial setup conditions may not be optimum at the later stages of the transfer.
  • Transfer failure and corruption Another problem with some existing file transfer methods is that the transfer may fail or data becomes corrupt after the transfer.
  • Some common causes of transfer failure and corruption include packet loss, high latency network connectivity, low speed network connectivity, high network congestion, network interruption including cable pull, loss of intermediate device such as switch or router, loss of firewall, network swap e.g.: start upload over wireless, swap to Ethernet network/IP scheme 1, swap to Ethernet network/IP scheme 2, etc, user defined pause and resume at any time during the transfer or graceful restart or instant power loss of server/client computers. It has been difficult to overcome failure and file corruption caused by these factors using prior art file transfer methods
  • a further a problem with some prior art methods is that files cannot be transferred securely to a destination.
  • Some file transfer protocols that are used in the transfer of files such as FTP, Skype and Bit torrent protocols have no method of data encryption and the data can be obtained by third parties. Therefore these methods cause a serious threat to the privacy of the transferred files and can also cause data to become corrupt or be infected by viruses.
  • the invention resides in a method of transferring one or more files from a sender's computer to a recipient's computer over a network, the method comprising the steps of:
  • each chunk is attached to a web service call as a binary attachment and each web service call having a data envelope containing information on the size, position and/or global unique ID of the attached chunk,
  • the chunk(s) have not been successfully received after one or more transfer attempts and/or only a part of the chunk has been received by the recipient's computer
  • the network is the internet, WAN or LAN.
  • the web service calls are XML web service calls that are sent over HTTPS protocol.
  • the sender's computer is a client computer and the receiver's computer is a server computer in an upload transfer and the sender's computer is a server computer and the receiver's computer is a client computer in a download transfer.
  • the client computer runs a 'client transfer agent' and 'client launcher software' and the server computer runs a 'server transfer service' and optional 'server monitor software'.
  • the server computer runs a 'server transfer service' and optional 'server monitor software'.
  • two or more chunks are sent to recipient's computer simultaneously.
  • More preferably four sequential chunks are sent to the recipient's computer concurrently.
  • a transfer of a chunk(s) is interrupted or fails due to transfer failure causes defined before, the resending of the chunk(s) is initiated automatically.
  • the chunks have size and CRC checks embedded.
  • the chunks change size in accordance with transfer conditions.
  • the incrementing of chunk sizes sent from the sender's computer occurs until a predefined maximum chunk size is reached wherein the size of chunks are not incremented when this limit is reached.
  • the decrementing of chunk sizes sent from the sender's computer occurs until a successful transfer of a chunk is achieved or until a predefined minimum chunk size is reached wherein the size of chunks are not decremented when this limit is reached.
  • the initial file(s) on the sender's computer and the transferred file(s) on the recipient's computer are binary identical where the transfer method allows for zero file corruption tolerance.
  • the invention resides in a file transfer system for transferring at least one file between a client computer(s) and a server computer(s) over a network, the file transfer system comprising:
  • a server transfer service running on the server computer - wherein the client transfer agent sends messages to the server transfer service, the messages being either file upload requests or file download requests
  • the client transfer agent separates a file into one or more chunks and transfers the one or more chunks of the file to the server computer as web service calls
  • each chunk is attached to a web service call as a binary attachment and each web service call also having information on the size, position and/or global unique ID of the attached chunk and,
  • the server transfer service receives the web service calls from the client transfer agent, retrieves the chunks of each web service call and assembles the chunks to form an identical copy of the file(s) at the server computer using information on the size, position and/or global unique ID of each chunk,
  • the server transfer service separates the file into one or more chunks and transfers the one or more chunks of the file to the client computer as web service calls, wherein each chunk is attached to a web service call as a binary attachment and each web service call also having information on the size, position and/or global unique ID of the attached chunk and,
  • the client transfer agent receives the web service calls from the server transfer service, retrieves the chunks of each web service call and assembles the chunks to form an identical copy of the file(s) at the client computer using information on the size, position and/or global unique ID of each chunk,
  • the server transfer service is an XML web service of the server which is accessible using HTTPS over the internet, LAN or WAN.
  • the web service calls are XML web service calls.
  • the client computer includes client launcher software in the form of a graphical user interface, a windows service, a continuous replication service and/or a command line utility for initiating a file transfer from the client transfer agent and/or displaying, logging or relaying messages from the client transfer agent to a user.
  • client launcher software in the form of a graphical user interface, a windows service, a continuous replication service and/or a command line utility for initiating a file transfer from the client transfer agent and/or displaying, logging or relaying messages from the client transfer agent to a user.
  • the server computer includes server monitoring software for providing services such as email notification, logging progress and other transfer metadata to a database, monitoring scheduled transfers and/or reporting on activity and usage.
  • server monitoring software for providing services such as email notification, logging progress and other transfer metadata to a database, monitoring scheduled transfers and/or reporting on activity and usage.
  • This invention may also broadly be said to consist in the parts, elements and features referred to or indicated in the specification of the application, individually or collectively, and any or all collectively of any two or more of the parts, elements or features, and where specific integers are mentioned herein which have known equivalents such equivalents are deemed to be incorporated herein as if individually set forth.
  • Figure 1 is a block diagram showing an initialisation phase of a file upload in accordance with a first preferred embodiment of the invention.
  • Figure la-c are screenshots of a graphic user interface showing the initialisation of a file transfer.
  • Figure 2 is a block diagram showing a transfer phase of a file upload.
  • Figure 2a is a screenshot of a graphic user interface showing the progress of a file transfer.
  • Figure 3 is a block diagram showing a completion phase of a file upload.
  • Figure 3a is a screenshot of a graphic user interface showing the completion of a file transfer.
  • Figure 4 is a block diagram showing an initialisation phase of a file download.
  • Figure 5 is a block diagram showing a transfer phase of a file download.
  • Figure 6 is a block diagram showing a completion phase of a file download.
  • Figure 7 is a block diagram showing the transfer phase of a file upload illustrating the adaptive chunk size feature of the invention.
  • the transfer system includes four main software components which are the client launcher software 101, the client transfer agent 103, the server transfer service 105 and the server monitoring software 107 as shown in figures 1-6. While only the client transfer agent and the server transfer service are core elements of this invention, the client launcher and server monitor components are expected to be present in some form when the file transfer system is in use. All four components are described in more detail below.
  • Client Launcher Software The purpose of this software is to call the client transfer agent and instruct it to transfer files. This software will also display, log, or otherwise relay messages from the transfer agent to the end user.
  • suitable launcher software are (a) a graphical user interface (GUI) giving users the ability to upload and download files to a server (figure la-c) (b) a Windows service that replicates files between a local data store and the server (c) a command line utility that reconciles a local data store with a directory on the server (d) a continuous replication service that continuously transfers changes made to a log file to the server, to keep the contents of the log file in near real-time synchronization.
  • GUI graphical user interface
  • Client Transfer Agent The client transfer agent controls the transfer progress. It is responsible for sending and receiving messages to the server transfer service and controlling the logic around fault tolerance and resume.
  • the client transfer agent communicates with the client launcher software via event messages.
  • the client transfer agent communicates with the server transfer service via XML web service calls.
  • the server transfer service is an XML web service running on the server computer and accessible using HTTPS over the internet, LAN or WAN.
  • the server transfer service is responsible for receiving data from the client and writing it to the file on the server during upload. During download, the server sends chunks of data to the client transfer agent as specified by the client transfer agent.
  • the server saves files and retrieves files to and from the designated data store directory located locally on the server hosting server transfer service or on a UNC share.
  • the server transfer service communicates with the server monitoring software via event messages.
  • Server Monitoring Software This is the software providing additional functionality for the server. This functionality may include, but is not limited to, e-mail notification, logging progress and other transfer metadata to a database, monitoring scheduled transfers, reporting on activity and usage.
  • the file transfers are performed in three phases.
  • the three phases are an initialisation phase, transfer phase and a completion phase.
  • Each phase exchanges one or more messages via XML web service calls between the client computer and the server as explained in more detail later.
  • Each message carries envelope data and an optional binary attachment/payload for transfer of chunks of files.
  • FIG. 1 shows a flowchart of the initialisation phase of a file upload transfer.
  • the initialization phase starts when client launcher software 101 (e.g.: a GUI as shown in figure la-c) signals to the client transfer agent 103 that a new transfer should be performed 109.
  • client launcher software 101 e.g.: a GUI as shown in figure la-c
  • the client launcher software requires the following information to initiate an upload transfer:
  • Optional information such as folder on the server the file should be placed in, notification settings, etc (as shown at 133 in figure lc).
  • the transfer agent running on the client computer creates an initialization request message 111 as shown in figure 1.
  • This XML message is populated with (1) indication whether this transfer is an upload or a download (2) name of file to be transferred (3) expected size of file (4) whether this is an overwrite file request or a continuous replication request.
  • the server transfer service 105 authenticates the client computer and either accepts or rejects the connection. If the server accepts the connection, a new user impersonation context will be created, impersonating the user entity based on the authentication credentials that the client computer provided at transfer initiation. This is essential as when the files are created or accessed, this user context is used, thus the correct files are shown to the user based on permissions given to the user and when files are created they are owned by this user. Finally, any activity performed by a user is audited in accordance with Microsoft Windows auditing methodology.
  • the server 105 When the server 105 receives the initialization request message 111, it creates a unique transfer ID at 113 to allow logging and tracking, allocates space 114 in either a temporary file (if this is an overwrite file request), or an existing file, if this is a continuous replication request. Furthermore the server monitor 107 logs all transfer details such as file name, time, file size, etc 115.
  • the server 105 will send back to the transfer agent 103 at 116 an 'Initialization Response' XML response received at 117 and containing: (1) Error information (if any); (2) the unique ID for this transfer - this unique ID is the name of the temporary file or name of the server file being continuously replicated by continuous replication; (3) the maximum chunk size supported by this server.
  • Examples of a Transfer ID are "d40dfd3e-8b5c-4528-af9e- dl6596729693" - randomly allocated globally unique ID (GUID) for a new transfer - this is the name of the temporary file the server holds while receiving the file to prevent an incomplete file being visible on the server.
  • Transfer ID For a continuous replication transfer, an example of a Transfer ID would be "Replicating_File.log" - the name of the file being continuously replicated.
  • the maximum chunk size information allows the server 105 to dictate the maximum size of a chunk it can receive. As chunk data is buffered in server memory, servers that expect many concurrent users can limit the maximum chunk size for performance reasons. The default maximum chunk size on the server is unlimited i.e.: accepts any chunk size the client computer may send.
  • the transfer agent 103 running on the client computer signals to the client launcher software at 118 that the transfer has been initialized 119, or was unable to be initialized because of an error e.g.: "the server was out of disk space", "the server certificate was un-trusted", etc. Normally the initialization succeeds at 118 the transfer agent persists at 120 and moves on to the transfer phase at 121. If the transfer agent 103 does not receive a response from the server 105 preferably within a 90 second timeout period, or is unable to find the server e.g.: network error, the transfer agent queues the transfer, and will continue re-trying to initialize the transfer preferably every 90 seconds, until cancelled by client launcher software 101 or initialization is successful.
  • the transfer may be queued at 124 for later initialization and resumption.
  • the transfer phase begins (figure 2).
  • the actual file data is transferred (uploaded).
  • the transfer agent 103 (figure 2) on the client computer spawns preferably four chunk transfer processes 201 simultaneously, each chunk starting with consecutive positions on transferred the file. However any other number of chunk transfer processes can be spawned depending on the server limitations. These positions will be determined by the starting chunk size configured on the client computer, with the default chunk size being 1.0MB (1024KB) of file data. If a chunk transfer succeeds on the first try, the chunk size used will increase by a predetermined value (e.g.: 0.5MB). Upon further consecutive successes the chunk size used will continue to grow by that value (e.g.: 1.5MB -> 2MB -> 2.5MB and so on) until the maximum chunk size is reached (preferably 5MB). If a chunk transfer fails more times than a predetermined value (preferably 4 times), the next chunk size used is reduced by a predetermined value (e.g.: 0.5MB or less) until a minimum chunk size (preferably 0.1MB) is reached.
  • a predetermined value e.g.: 0.5MB or less
  • the default chunk sizes, chunk growth and chunk shrinkage values, minimum and maximum chunk sizes and all timeout values are configurable and can be dictated by the client launcher software 101. It is expected that these values will change continuously, as network connectivity is expected to improve in the future. They can also be tuned for a specific situation e.g.: satellite, slow WAN, congested WAN, etc.
  • Each chunk transfer process 201 first reads the current chunk size from the launcher at 202 then asynchronously sends an upload data request message at 203, as shown in figure 2.
  • an upload data request message is received at 205, the data written to a temporary file at 206 and an upload data response message sent at 207..
  • an incoming response is continually monitored for at 208 and if received is passed to a response type sensor 209. If no response is received a check is made at 210 for an upload timeout (typically at 8 minutes) and if this time is exceeded the chunk is resent. If a fatal error is received at 209 then the process is aborted at 212 and signalled to the client launcher at 215. Where the received response indicates at 211 that the end of a chunk has been reached the file transferred position for that file chunk is updated at 214, a signal sent to the launcher process and another chunk is started.
  • the XML message 203 envelope contains metadata i.e.: (1) Transfer ID (2) chunk start position; and has a binary attachment which is the actual chunk data.
  • the binary attachment infrastructure carries an expected length and a cyclic redundancy check (CRC) value of the chunk. If the server does not receive the entire chunk, or the chunk received by the server does not match the CRC value, the server will respond with an error, causing the client to resubmit the chunk again.
  • the server when it receives the message 203, checks the chunk for integrity 205. If no integrity errors are found, the server writes the binary data contained in the attachment to the position specified by the envelope message, in the file dictated by the Transfer ID contained in the envelope at 206. Furthermore the server monitor software 107 logs all activity 217 as in the initialisation phase.
  • the server If the server was successfully able to write the binary data contained in the attachment to the correct position, the server responds with an upload data response message 207 containing no error details i.e.: acknowledgement of success. If the server encounters an error, the error is classified as fatal (e.g.: access denied) or transient (e.g.: the chunk CRC did not compute meaning the chunk was corrupt on reception). The error details are returned to the client in the upload data response message 207.
  • fatal e.g.: access denied
  • transient e.g.: the chunk CRC did not compute meaning the chunk was corrupt on reception
  • the client transfer agent 103 monitors for a response at 208, looping through timeout monitor 210 until a response is received or the transfer times out, when a resend of the same chunk is instigated.. If it receives a response with a fatal error at 209 the transfer process is aborted at 212 and signalled to the client launcher at 215.
  • the launcher software will show these details on the screen (as shown in Figure 2a) or log them to a log file, or perform some other action.
  • a transient error, or receipt of a network error at 211 also causes the transfer agent to reinstigate transfer of the same chunk. The transfer agent re-submits the same chunk, with the same metadata again.
  • the transfer agent will continue re-trying for a number of times (preferably 20), and then wait a specified amount of time (preferably 20 seconds) until continuing to retry. The delay is added to ensure the re-try process is not tying up too many resources.
  • the client computer will continue re-trying until the client launcher software instructs the transfer agent to stop, or a successful transfer is obtained.
  • the transfer agent 103 If the client transfer agent 103 receives a response containing a fatal error 209, the transfer agent will signal the client launcher software 215 that there was a fatal error, and pass the error details to be displayed on screen, logged to event log, etc. The client transfer agent will then abort the transfer and exit.
  • the transfer agent receives a response message with an end of file notification at 214 the completion of that chunk is notified to the signal process at 215 and the process stops.
  • Other response messages signify the receipt of a message with no errors and the chunk position is updated at 216 and returns to start the transfer of another chunk.
  • Information stored relating to a chunk transfer must be sufficient for the transfer to resume from this point at which it was persisted without requiring any further information.
  • the information persisted include transfer ID, upload or download, full path to folder, file name, server URL, size of file and current position.
  • the transfer agent signals the progress to the client launcher software 215 to display on a progress bar 135 (figure 2a), log to a log file or performs some other action with this information.
  • the completion phase will begin. If not, the transfer agent will move onto the next file. If the end of the file is not reached, the transfer agent will spawn additional chunk transfer processes 201 to maintain multiple concurrently running processes (preferably 4).
  • the client software can cause the transfer agent to interrupt itself i.e.: pause, and then resume.
  • the transfer agent When paused, the transfer agent will abandon any chunks being currently transferred, and exit.
  • resumed the transfer agent will read progress data from the last persistence point (as persisted on disk) and resume transfer (starting from the current point persisted). This allows the transfer to be resumed even after the client computer loses power.
  • the upload transfer phase is complete (i.e. the last chunk of the file was successfully transferred and acknowledged by the server), the client will finalize the transfer in the upload completion phase as shown in figure 3.
  • the client computer sends a completion request message 301 (without any attachments) to the server.
  • the XML completion request message contains information such as transfer ID, file name and destination folder.
  • the server 105 Upon receipt of the completion request message, the server 105 moves the temporary file to its final destination 303 (i.e.: appropriate folder, and renames it to the correct name). If the transfer was a part of a continuous replication, the server does not need to move or rename the file.
  • the server 105 then signals completion to the server monitor software 107 where the data is logged in a file 305 or database or displayed on screen and may send a notification of the completion, by email for instance.
  • the server sends a completion response message at 306 which is received at 307 carrying error details (if any) to the client transfer agent.
  • the transfer agent receives a server error at 311 the client computer will signal completion to the client launcher software 309 to be shown on screen as shown in Figure 3 a. Receipt of a success message (no error) at 308 from the server allows the success of the transfer to be persisted to the client disk at 310 before the success is shown on screen,
  • the transfer agent clears the persistence point data from disk, so that any new transfers of this file starts from the beginning of the file. If it is indeed a continuous replication scenario, persistence point data stays intact, so that any subsequent transfers of the file will begin from the point where the transfer ended (i.e. only changes are replicated).
  • FIG 4-6 show the three phases of a file download transfer using the file transfer system of this invention. Similar to the upload phases, messages are exchanged in the form of XML web service calls between the client computer and the server and each message carries envelope data and an optional binary attachment payload for transferring chunks of files.
  • the download initialization phase begins when the client launcher software 101 signals to the transfer agent 103 that a new transfer should be performed 401 (figure 4).
  • the client launcher software requires the following information to initiate a download transfer:
  • Relative path of each file, located on the server - this path is relative to the server's data store directory (i.e.: the directory the server is exposing to the internet)
  • the server authenticates the client computer and either accepts or rejects the client connection, similar to the file upload scenario described before. If the server accepts the connection, a new user impersonation context is created, impersonating the user entity based on the authentication credentials that the client computer provided at transfer initiation. This is essential as when the files are created or accessed, this user context is used, thus the correct files are shown to the user, based on permissions given to the user and when files are created they are owned by this user. Finally, any activity performed by a user is audited in accordance with Microsoft Windows auditing methodology.
  • the server 105 When the server 105 receives the initialization message 403, it checks for existence of the file(s) specified for download, and whether the user context has permissions to read the file 405. As mentioned before, the server monitor 107 logs all transfer details such as file name, time, file size, etc 407.
  • the server returns to the client transfer agent a response 409 containing: (1) Error information (if any); (2) Unique ID for this transfer - this unique ID is the name of the temporary file or name of the server file being continuously replicated by continuous replication; (3) Maximum chunk size supported by this server.
  • An example of a transfer ID is "d40dfd3e-8b5c-4528-af9e-d 16596729693" - randomly allocated GUID for a new transfer.
  • the transfer ID is the same for continuous replication transfer and basic file transfer.
  • the maximum chunk size information allows the server to dictate the maximum size of the chunk it can send. As the chunk data is buffered in server memory, servers that expect many concurrent users may wish to limit the maximum chunk size for performance reasons.
  • the default maximum chunk size on the server is unlimited i.e.: sends any chunk size length requested by the client transfer agent.
  • the transfer agent 103 running on the client computer will allocate space in a temporary file located in the folder specified during download initialization and signal to the client launcher software 101 that the transfer has been initialized 411, or was unable to be initialized because of an error e.g.: "the computer was out of disk space", "the server certificate was un-trusted”, etc. If at 414 the transfer agent does not receive a response from the server preferably within a 90 second timeout period, or is unable to find the server (e.g. network error), the transfer agent queues the transfer at 415, and will continue re-trying to initialize the transfer preferably every 90 seconds, until cancelled by the client launcher software or initialization is successful.
  • an error e.g.: "the computer was out of disk space", "the server certificate was un-trusted”, etc.
  • the transfer agent of the client computer spawns preferably 4 chunk transfer processes 501 simultaneously, each chunk starting with consecutive positions on transferred the file.
  • the chunk size used will increase by a predetermined value (e.g.: 0.5MB).
  • a predetermined value e.g.: 0.5MB
  • the chunk size used will continue to grow by that value (e.g.: 1.5MB -> 2MB -> 2.5MB and so on) until the maximum chunk size is reached (preferably 5MB).
  • a chunk transfer fails more times than a predetermined value (preferably 4)
  • the next chunk size used reduced by a predetermined value e.g.: 0.5MB
  • a minimum chunk size preferably 0.1MB
  • the default chunk sizes, chunk growth and chunk shrinkage values, minimum and maximum chunk sizes and all timeout values are configurable and can be dictated by the client launcher software 101. It is expected that these values will change continuously, as network connectivit is expected to improve in the future. They can also be tuned for a specific situation e.g.: satellite, slow WAN, congested WAN, etc.
  • Each chunk transfer process 501 asynchronously sends a download data request message 503, as shown in figure 5 and waits for a server response 507, an error (e.g. server could not be reached) or a timeout with a defined value (default 8 minutes).
  • the message envelope contains the following metadata: (1) Transfer ID and (2) Chunk start position.
  • transfer progress and errors are logged by server monitor software 517.
  • the server receives a download data request message 505, it reads a chunk from the required position in a file at 506 and stores it in memory. It then sends a download data response message 507, which has a binary attachment containing the actual chunk data.
  • the binary attachment infrastructure carries an expected length and a cyclic redundancy check (CRC) value of the chunk.
  • CRC cyclic redundancy check
  • the transfer agent If the transfer agent does not receive the entire chunk, or the chunk received by the transfer agent does not match the CRC value, the transfer agent will re-submit the download data request 509. If no integrity errors are found, the transfer agent writes the binary data contained in the attachment to the position specified by the envelope message 505, in the file dictated by the temporary file name. If the server encounters an error, the error is classified as fatal (e.g. access denied) 509 or transient 511 (e.g. lost network connection). The error details are returned to the transfer agent in the download data response message 507.
  • fatal e.g. access denied
  • transient 511 e.g. lost network connection
  • the transfer agent receives a transient error 511, receives a network error, the transfer agent will signal the client launcher software that a transmission error was experienced 515.
  • the launcher software will show these details on the screen or log them to a log file, or perform some other appropriate action.
  • the transfer agent will also send another download data request message asking for the same chunk, at the same position, with the same metadata again 503.
  • the transfer agent continues re-trying for a number of times (preferably 20), and then wait a specified amount of time (preferably 20 seconds) until continuing to retry. The delay is added to ensure the re-try process is not tying up unnecessary resources.
  • the transfer agent will continue re-trying until the client launcher software instructs the transfer agent to stop, or a successful transfer is obtained.
  • the transfer agent If the transfer agent receives a response containing a fatal error 509, the transfer agent signals the client launcher software 515 that there was a fatal error, and passes the error details to be displayed on screen, logged to event log, etc. The transfer agent will then abort the transfer and exit at 512.
  • the transfer agent If the transfer agent writes the attachment data to file without error at 516 (i.e. success), the transfer agent persists its position on disk. Information persisted must be sufficient for the transfer to resume from this point without requiring any further information. The information persisted include transfer ID, upload or download, full path to folder, file name, server URL, size of file and current position.
  • the transfer agent signals the progress to the client launcher software to display on a progress bar, log to a log file or to perform some other action with this information. Once the progress was persisted successfully, the chunk transfer process exits and the transfer agent checks whether the end of file is reached at 514. If so, file completion is signalled to the client launcher software 515. If the last file in the sequence is reached, the completion phase will begin. If not, the transfer of the next file will begin. If the end of the file is not reached, the transfer agent spawns additional chunk transfer processes 501 to maintain multiple concurrently running processes (preferably 4).
  • the client software can cause the transfer agent to interrupt itself i.e.: pause, and then resume.
  • pause the transfer agent will abandon any chunks being currently transferred, and exit.
  • resumed the transfer agent will read progress data from the last persistence point (as persisted on disk) and resume transfer (starting from the current point persisted). This allows the transfer to be resumed even after the client computer loses power.
  • the transfer agent finalizes the transfer in the download completion phase.
  • the transfer agent 103 does not need to send any data to the server 105 to finalize the transfer as shown in figure 6.
  • the transfer agent renames the temporary file 601 (e.g.: download_file.zip.tmp) to the correct name (i.e.: download.zip). If the transfer was a part of the continuous replication, the transfer agent does not need to rename the file.
  • the transfer agent also signals completion to the client software 603 so that it is displayed on screen, logged to file, etc.
  • the transfer agent clears the persistence point data from disk at 602, so that any new transfers of this file starts from the beginning of the file. If the download was in fact a continuous replication scenario, persistence point data stays intact, so that any subsequent transfers of this file begin from the point where this transfer ended (i.e. only changes are replicated).
  • Figure 7 shows a file upload transfer process as described in Figure 2 in which the chunk size file adjustment process is identified. As the file transfer proceeds the chunk size is adjusted to take advantage of good transfer conditions or compensate for poor transfer conditions as explained previously.
  • an upload data request message 203 is sent as an Upload Data message at 704, received at 205 and is either correctly received as at 705, when the data is written to the indicated position 707 and a "success" response sent at708 or the receive fails at 706 and a "failure” response is sent at 713.
  • the response is determined as fatal or otherwise at 209 on the basis of the type of failure or the number of previous failure responses. If fatal the process is aborted at 211 and an eventual restart may be attempted. Where the failure is judged non-fatal at 209 the chunk size last used is decreased at 714 as explained previously and a read of that chunk carried out at 203 for transfer.
  • the size of the chunk step increase or decrease will depend on the number of failures and the data rate, their being a calculable maximum achievable transfer rate based on the number and size of failed uploads.
  • the chunk size adjustment process for a file download transfer (not illustrated) is similar to the download transfer phase shown in figure 5 where the chunk size is incremented upon a successful transfer 513 and the chunk size is decremented on subsequent attempts 509 upon a transfer failure.
  • the file transfer system disclosed in this specification has the ability to change the chunk size of a transferred file depending on the transfer conditions during a transfer. As explained previously, the chunk size increases with each successful chunk transfer until chunk size reaches a predefined maximum size and decreases each time the chunk transfer fails until a minimum chunk size is reached. This allows the file transfer system to adapt efficiently to the transfer conditions so that bigger chunks are transferred in a strong server connection that does not often fail and smaller chunks are transferred if the server connection often fails. Further, when conditions change over the time of an extended transfer, the chunk size will adapt to the changed conditions by either increasing or decreasing the chunk size transferred.
  • the file transfer system of this specification is immune to transfer failure and corruption caused by a number of problems that can occur during file transfer. More specifically, no progress is lost and transfer recovers and continues automatically in the case of packet loss, high latency network connectivity, low speed network connectivity, high network congestion, network interruption including cable pull, loss of intermediate device such as switch or router, loss of firewall, network swap e.g.: start upload over wireless, swap to Ethernet network/IP scheme 1, swap to Ethernet network/IP scheme 2, etc, user defined pause and resume at any time during the transfer or graceful restart or instant power loss of server/client computers. In the case of power loss or restart of the client computer, the transfer resumes automatically or manually depending on the settings of the client launcher software. Furthermore CRC checks are performed on transferred files to ensure that the files are identical copies of the original files and are free from corruption as explained previously.
  • HTTPS Hypertext Transfer Protocol Secure
  • SSL Secure Socket Layer
  • the file transfer system has the ability to transfer unlimited amount of data and the software does not have any limitations in terms of maximum transfer size. In testing the system, individual files of up to 100 GB in size have been transferred. The data transfer limit may only be restricted by the limitations of the client and server computers and the server connection.
  • file transfer system of this invention uses the HTTPS protocol to transfer files to a server using XML web service calls, other protocols and other forms of digital messaging can be used. It is expected that file transfer protocols and messaging will become more efficient and increasingly secure in future. Therefore the system can be adapted for use with those new or already existing technologies currently under development.
  • All file sizes, minimum and maximum chunk sizes and timeout values are configurable and can be dictated by the client launcher software as mentioned previously.
  • the values disclosed in this specification are for example only and can be varied depending on the individual network and conditions. It is expected that these values will change continuously, as network connectivity is expected to improve in the future. They can also be tuned for a specific situation e.g.: satellite, slow WAN, congested WAN, etc.
  • the word "comprise” and variations of that word such as “comprising” and “comprises” are not intended to exclude other additives, components, integers or steps.

Landscapes

  • Engineering & Computer Science (AREA)
  • Computer Networks & Wireless Communication (AREA)
  • Signal Processing (AREA)
  • Information Transfer Between Computers (AREA)

Abstract

Un processus de transfert approprié pour de grands fichiers (> 100 Mbits à de nombreux Gbits) d'un ordinateur d'expéditeur à un ordinateur de destinataire sur un réseau implique la lecture du fichier et la mémorisation de fragments du fichier dans une mémoire tampon, ensuite l'agent de transfert sur l'ordinateur de client engendre quatre processus de transfert de fragments simultanément, chaque fragment commençant par des positions consécutives sur le fichier transféré, la taille de fragment par défaut étant de 1,5 Moctets de données de fichier. Si un transfert de fragment réussit au premier essai, la taille de fragment utilisée augmentera d'une valeur prédéterminée. Lors d'autres réussites consécutives, la taille de fragment utilisée continuera d'augmenter de cette valeur jusqu'à ce que la taille de fragment maximum atteigne généralement 5 Moctets. Si un transfert de fragment échoue un nombre de fois supérieur à une valeur prédéterminée, la taille de fragment suivante utilisée est réduite d'une valeur prédéterminée jusqu'à ce qu'une taille de fragment minimum (de préférence, 0,1 Moctet) soit atteinte.
PCT/NZ2011/000126 2010-07-08 2011-07-05 Système de transfert de fichier WO2012005609A1 (fr)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
NZ58668110 2010-07-08
NZ586681 2010-07-08

Publications (1)

Publication Number Publication Date
WO2012005609A1 true WO2012005609A1 (fr) 2012-01-12

Family

ID=45441398

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/NZ2011/000126 WO2012005609A1 (fr) 2010-07-08 2011-07-05 Système de transfert de fichier

Country Status (1)

Country Link
WO (1) WO2012005609A1 (fr)

Cited By (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20140189038A1 (en) * 2012-12-28 2014-07-03 Brother Kogyo Kabushiki Kaisha Intermediate server, communication apparatus and computer program
EP3119062A1 (fr) * 2015-07-17 2017-01-18 Bio-Rad Laboratories, Inc. Transfert de réseau de fichiers de grande taille dans des environnements de réseau instables
CN106856503A (zh) * 2015-12-08 2017-06-16 三星电子株式会社 用于对设备的上传大小进行控制的方法和装置
CN108366112A (zh) * 2018-02-06 2018-08-03 杭州朗和科技有限公司 客户端的数据传输方法及系统、介质和计算设备
CN109495434A (zh) * 2017-09-13 2019-03-19 北京国双科技有限公司 一种文件传输方法及装置
US10664170B2 (en) 2016-12-14 2020-05-26 Microsoft Technology Licensing, Llc Partial storage of large files in distinct storage systems

Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO1991007038A2 (fr) * 1989-11-03 1991-05-16 Microcom Systems, Inc. Procede et appareil effectuant des transmissions efficaces de donnees
US6061733A (en) * 1997-10-16 2000-05-09 International Business Machines Corp. Method and apparatus for improving internet download integrity via client/server dynamic file sizes
US20060133278A1 (en) * 2004-12-03 2006-06-22 Microsoft Corporation Efficient transfer of messages using reliable messaging protocols for web services
US20070076626A1 (en) * 2005-09-30 2007-04-05 Research In Motion Limited Methods And Apparatus For Dynamically Adjusting A Data Packet Window Size For Data Packet Transmission In A Wireless Communication Network

Patent Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO1991007038A2 (fr) * 1989-11-03 1991-05-16 Microcom Systems, Inc. Procede et appareil effectuant des transmissions efficaces de donnees
US6061733A (en) * 1997-10-16 2000-05-09 International Business Machines Corp. Method and apparatus for improving internet download integrity via client/server dynamic file sizes
US20060133278A1 (en) * 2004-12-03 2006-06-22 Microsoft Corporation Efficient transfer of messages using reliable messaging protocols for web services
US20070076626A1 (en) * 2005-09-30 2007-04-05 Research In Motion Limited Methods And Apparatus For Dynamically Adjusting A Data Packet Window Size For Data Packet Transmission In A Wireless Communication Network

Non-Patent Citations (2)

* Cited by examiner, † Cited by third party
Title
BOOTH, D. ET AL.: "Web Services Architecture", W3C WORKING GROUP NOTE, 11 February 2004 (2004-02-11), Retrieved from the Internet <URL:http://web.archive.org_/web/20040402051851/http://www.w3.org/TR/ws-arch> [retrieved on 20111109] *
GAILEY, J. H.: "Sending Files, Attachments, and SOAP Messages Via Direct Internet Message Encapsulation", MSDN MAGAZINE, December 2002 (2002-12-01), Retrieved from the Internet <URL:http://web.archive.org/web/20100129044410/http://msdn.microsoft.com/en-us/magazine/cc188797.aspx> [retrieved on 20111109] *

Cited By (14)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US10298686B2 (en) * 2012-12-28 2019-05-21 Brother Kogyo Kabushiki Kaisha Intermediate server, communication apparatus and computer program
US20140189038A1 (en) * 2012-12-28 2014-07-03 Brother Kogyo Kabushiki Kaisha Intermediate server, communication apparatus and computer program
EP3119062A1 (fr) * 2015-07-17 2017-01-18 Bio-Rad Laboratories, Inc. Transfert de réseau de fichiers de grande taille dans des environnements de réseau instables
CN106453474A (zh) * 2015-07-17 2017-02-22 生物辐射实验室股份有限公司 在不稳定网络环境中的大文件的网络传输
CN112910967B (zh) * 2015-07-17 2024-04-09 生物辐射实验室股份有限公司 在不稳定网络环境中的大文件的网络传输
US10033794B2 (en) 2015-07-17 2018-07-24 Bio-Rad Laboratories, Inc. Network transfer of large files in unstable network environments
CN112910967A (zh) * 2015-07-17 2021-06-04 生物辐射实验室股份有限公司 在不稳定网络环境中的大文件的网络传输
CN106453474B (zh) * 2015-07-17 2021-02-05 生物辐射实验室股份有限公司 在不稳定网络环境中的大文件的网络传输
EP3384703A4 (fr) * 2015-12-08 2018-12-19 Samsung Electronics Co., Ltd. Procédé et appareil de régulation de la taille de téléchargement de dispositif
US10887372B2 (en) 2015-12-08 2021-01-05 Samsung Electronics Co., Ltd. Method and apparatus for controlling upload size of device
CN106856503A (zh) * 2015-12-08 2017-06-16 三星电子株式会社 用于对设备的上传大小进行控制的方法和装置
US10664170B2 (en) 2016-12-14 2020-05-26 Microsoft Technology Licensing, Llc Partial storage of large files in distinct storage systems
CN109495434A (zh) * 2017-09-13 2019-03-19 北京国双科技有限公司 一种文件传输方法及装置
CN108366112A (zh) * 2018-02-06 2018-08-03 杭州朗和科技有限公司 客户端的数据传输方法及系统、介质和计算设备

Similar Documents

Publication Publication Date Title
WO2012005609A1 (fr) Système de transfert de fichier
JP4734592B2 (ja) クライアントリダイレクトによるプライベートネットワークへの安全なアクセス提供方法およびシステム
US9521217B2 (en) System and method for remote access to cloud-enabled network devices
US7921215B2 (en) Method and apparatus for optimizing and prioritizing the creation of a large number of VPN tunnels
TW412685B (en) System and method for managing client requests in client-server networks
US8751682B2 (en) Data transfer using high speed connection, high integrity connection, and descriptor
KR100806492B1 (ko) Ｔｃｐ 상태천이를 이용한 서비스거부 공격의 차단방법
EP3179701B1 (fr) Procédés de téléchargement de fichier et serveur associé
US20100011435A1 (en) Method and System for Providing Guaranteed File Transfer in Corporate Environment Behind Firewall
JP6745821B2 (ja) ハイパーテキスト・トランスファ・プロトコル要求の再送方法及びデバイス並びにクライアント端末
CA3158089A1 (fr) Decouverte et reglage d&#39;une unite de transmission maximale de chemin
US20100098092A1 (en) Accelerating data communication using tunnels
US8788576B2 (en) High speed parallel data exchange with receiver side data handling
US20080104252A1 (en) Resuming a computing session when rebooting a computing device
US7089311B2 (en) Methods, systems and computer program products for resuming SNA application-client communications after loss of an IP network connection
US9985913B2 (en) Apparatus and method for client-side flow control in a remote access environment
CN108234089A (zh) 低时延通信
CN112055088B (zh) 一种基于光闸的文件可靠传输系统及其方法
CN107294877B (zh) 一种tcp流重组方法和装置
US20200007608A1 (en) System and method for performing fast file transfers
CN118573729A (zh) 一种内网穿透系统和内网穿透方法
JP2015201689A (ja) 通信システム、送信装置、受信装置及び通信方法
FR2988940A1 (fr) Procede et systeme de transmission d&#39;une donnee sous forme de n blocs entre un equipement source et un equipement cible via un routeur

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 11803862

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 11803862

Country of ref document: EP

Kind code of ref document: A1

点击 这是indexloc提供的php浏览器服务，不要输入任何密码和下载