CN116225314A - Data writing method, device, computer equipment and storage medium - Google Patents

Abstract

The present application relates to a data writing method, device, computer equipment, storage medium and computer program product. The method includes: obtaining a plurality of writing processes corresponding to the data writing request; determining the target data block of the target writing process for the target writing process; dividing the target data block according to the size of the preset stripes to obtain The size of the first data block and the remaining data blocks that do not meet the preset stripe size, and write the data in the first data block corresponding to the target write process; if there is a next write process of the target write process, Through the preset data transmission network, the remaining data blocks are transmitted to the next writing process of the target writing process, so that the next writing process determines the remaining data blocks and the original data blocks of the next writing process as the target data blocks, and as the new The target write process executes the step of dividing the target data block according to the preset stripe size.

Description

Data writing method, device, computer equipment and storage medium

technical field

The present application relates to the field of storage, in particular to a data writing method, device, computer equipment, storage medium and computer program product.

Background technique

In order to ensure the consistency of data in the distributed file storage system, it needs to be protected by distributed interval locks. Specifically, the data in the distributed file storage system can be divided into stripes, and each stripe needs to be protected by a distributed interval lock.

In related technologies, when cross-process data needs to be written to the same stripe, distributed interval locks are used to prevent different writing processes from processing the storage space in the same stripe, and each stripe reads This lock needs to be called when fetching. When reading multiple stripes, it will lead to competition of distributed interval locks, resulting in low data writing efficiency.

Contents of the invention

Based on this, it is necessary to address the above technical problems and provide a data writing method, device, computer equipment, computer readable storage medium and computer program product that can avoid distributed lock competition.

In a first aspect, the present application provides a data writing method. The methods include:

Responding to the data write request, acquiring multiple write processes corresponding to the data write request;

For a target writing process among multiple writing processes, determine a target data block of the target writing process;

Divide the target data block according to the size of the preset stripe, obtain the first data block that meets the size of the preset stripe and the remaining data blocks that do not meet the size of the preset stripe, and pass the A target write process, writing data in the first data block corresponding to the target write process;

In the case that the next writing process of the target writing process exists, the remaining data block is transmitted to the next writing process of the target writing process through a preset data transmission network, so that the next writing process Determining the remaining data blocks and the original data blocks of the next writing process as target data blocks, and performing the step of dividing the target data blocks according to the size of the preset stripes as a new target writing process .

Based on the above scheme, in the initial stage of data writing, the data to be written can be divided into full stripes, and the data that is not full of stripes and resulting in cross-process writing can be exchanged to other writing processes for writing, avoiding the use of distributed intervals Lock, and high-speed data transmission through the preset data transmission network, can communicate, exchange, splicing and writing data at high speed, ensure strict stripe alignment, and further ensure the efficiency of data writing.

In one of the embodiments, the target data block is divided according to the size of the preset stripe to obtain the first data block that satisfies the size of the preset stripe and the first data block that does not meet the size of the preset stripe After the step of remaining data blocks of size, the method further includes:

If there is no next writing process in the target writing process, the remaining data blocks corresponding to the target writing process are cached in a preset cache space.

Based on the above solution, by caching the remaining data blocks, data integrity and data writing timeliness can be guaranteed.

In one embodiment, the method also includes:

After detecting the close instruction of the file to be written corresponding to the data write request, acquiring the remaining data blocks in the preset cache space, and concatenating the remaining data blocks into the file to be written.

Based on the above solutions, the remaining data blocks can be written in time, avoiding data loss and improving the efficiency of data writing.

In one of the embodiments, for each target writing process in the multiple writing processes, determining the target data block of the target writing process includes:

For the target writing process, there is a previous writing process of the target writing process in the target writing process, obtaining the remaining data blocks of the previous writing process, and obtaining the original data blocks of the target writing process;

Splicing the remaining data blocks of the last writing process and the original data blocks of the target writing process to obtain the target data blocks of the target writing process.

Based on the above scheme, the data block of the previous writing process can be exchanged to the target writing process, so as to realize data writing without distributed lock competition and improve data writing efficiency.

In one embodiment, the method also includes:

For the target writing process, the previous writing process of the target writing process does not exist in the target writing process, and the original data block of the target writing process is determined as the target data block of the target writing process.

Based on the above solution, the original data block of the writing process can be used as the target data block to implement data writing with strict stripe alignment.

In one embodiment, the method also includes:

For the target writing process, when the target writing process has a previous writing process of the target writing process and the remaining data block does not exist in the previous writing process, determine the target writing process The original data block of is the target data block of the target write process.

Based on the above solution, the original data block of the writing process can be used as the target data block to implement data writing with strict stripe alignment.

In a second aspect, the present application also provides a data writing device. The devices include:

A first acquiring module, configured to acquire multiple writing processes corresponding to the data writing request in response to the data writing request;

A first determining module, configured to, for a target writing process among multiple writing processes, determine a target data block of the target writing process;

A write module, configured to divide the target data block according to the size of the preset stripe, to obtain the first data block that meets the size of the preset stripe and the remaining data that does not meet the size of the preset stripe block, and write the data in the first data block corresponding to the target writing process through the target writing process;

A transmission module, configured to transmit the remaining data blocks to the next writing process of the target writing process through a preset data transmission network in the case that there is a next writing process of the target writing process, so that all The next write process determines that the remaining data blocks and the original data blocks of the next write process are target data blocks, and as a new target write process, executes the process of performing the processing of the target data according to the size of the preset stripe Steps to divide blocks.

In one of the embodiments, the data writing device also includes:

The caching module is configured to cache the remaining data blocks corresponding to the target writing process in a preset cache space when there is no next writing process in the target writing process.

In one of the embodiments, the data writing device also includes:

The second acquisition module is configured to acquire the remaining data blocks in the preset cache space after detecting the closing instruction of the file to be written corresponding to the data writing request, and concatenate and write the remaining data blocks The file to be written.

In one of the embodiments, the first determining module is specifically configured to:

For the target writing process, there is a previous writing process of the target writing process in the target writing process, obtaining the remaining data blocks of the previous writing process, and obtaining the original data blocks of the target writing process;

Splicing the remaining data blocks of the last writing process and the original data blocks of the target writing process to obtain the target data blocks of the target writing process.

In one of the embodiments, the data writing device also includes:

The second determining module is configured to, for the target writing process, determine that the original data block of the target writing process is the target writing process when the target writing process does not have a previous writing process of the target writing process target data block.

In one of the embodiments, the data writing device also includes:

The third determining module is configured to, for the target writing process, when the target writing process has a previous writing process of the target writing process, and the remaining data block does not exist in the previous writing process , determining the original data block of the target writing process as the target data block of the target writing process.

In a third aspect, the present application also provides a computer device. The computer device includes a memory and a processor, the memory stores a computer program, and the processor implements the following steps when executing the computer program:

Responding to the data write request, acquiring multiple write processes corresponding to the data write request;

For a target writing process among multiple writing processes, determine a target data block of the target writing process;

Divide the target data block according to the size of the preset stripe, obtain the first data block that meets the size of the preset stripe and the remaining data blocks that do not meet the size of the preset stripe, and pass the A target write process, writing data in the first data block corresponding to the target write process;

In the case that the next writing process of the target writing process exists, the remaining data block is transmitted to the next writing process of the target writing process through a preset data transmission network, so that the next writing process Determining the remaining data blocks and the original data blocks of the next writing process as target data blocks, and performing the step of dividing the target data blocks according to the size of the preset stripes as a new target writing process .

In a fourth aspect, the present application also provides a computer-readable storage medium. The computer-readable storage medium has a computer program stored thereon, and when the computer program is executed by a processor, the following steps are implemented:

Responding to the data write request, acquiring multiple write processes corresponding to the data write request;

For a target writing process among multiple writing processes, determine a target data block of the target writing process;

Divide the target data block according to the size of the preset stripe, obtain the first data block that meets the size of the preset stripe and the remaining data blocks that do not meet the size of the preset stripe, and pass the A target write process, writing data in the first data block corresponding to the target write process;

In the case that the next writing process of the target writing process exists, the remaining data block is transmitted to the next writing process of the target writing process through a preset data transmission network, so that the next writing process Determining the remaining data blocks and the original data blocks of the next writing process as target data blocks, and performing the step of dividing the target data blocks according to the size of the preset stripes as a new target writing process .

In a fifth aspect, the present application also provides a computer program product. The computer program product includes a computer program, and when the computer program is executed by a processor, the following steps are implemented:

Responding to the data write request, acquiring multiple write processes corresponding to the data write request;

For a target writing process among multiple writing processes, determine a target data block of the target writing process;

Divide the target data block according to the size of the preset stripe, obtain the first data block that meets the size of the preset stripe and the remaining data blocks that do not meet the size of the preset stripe, and pass the A target write process, writing data in the first data block corresponding to the target write process;

In the case that the next writing process of the target writing process exists, the remaining data block is transmitted to the next writing process of the target writing process through a preset data transmission network, so that the next writing process Determining the remaining data blocks and the original data blocks of the next writing process as target data blocks, and performing the step of dividing the target data blocks according to the size of the preset stripes as a new target writing process .

The above data writing method, device, computer equipment, storage medium and computer program product, wherein the method includes: in response to a data writing request, obtaining multiple writing processes corresponding to the data writing request; The target writing process in the process determines the target data block of the target writing process; divides the target data block according to the size of the preset stripe, and obtains the first data block satisfying the size of the preset stripe and For the remaining data blocks that do not meet the preset stripe size, write the data in the first data block corresponding to the target writing process through the target writing process; In the case of a process, the remaining data blocks are transmitted to the next writing process of the target writing process through a preset data transmission network, so that the next writing process can determine the remaining data blocks and the next The original data block of the writing process is the target data block, and as a new target writing process, the step of dividing the target data block according to the preset stripe size is performed. By adopting this method, at the initial stage of data writing, the data to be written can be divided into full stripes, and the data that is not full of stripes and resulting in cross-process writing can be exchanged to other writing processes for writing, avoiding the use of distributed Interval locks, and high-speed data transmission through the preset data transmission network, write data through exchange and splicing, ensure strict stripe alignment, and further improve data writing efficiency.

Description of drawings

Fig. 1 is a schematic flow chart of a data writing method in an embodiment;

Figure 2a is a schematic diagram of division of data blocks corresponding to a writing process in the data writing method in an embodiment;

Figure 2b is a schematic diagram of division of data blocks corresponding to a writing process in the data writing method in an embodiment;

Fig. 3 is a schematic flow chart of the step of determining the target data block in one embodiment;

Fig. 4 is a schematic diagram of distributed lock competition in an embodiment;

Fig. 5 is a structural block diagram of a data writing device in an embodiment;

Figure 6 is an internal block diagram of a computer device in one embodiment.

Detailed ways

In order to make the purpose, technical solution and advantages of the present application clearer, the present application will be further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described here are only used to explain the present application, and are not intended to limit the present application.

In one embodiment, as shown in FIG. 1 , a method for writing data is provided. This embodiment uses the method applied to a terminal as an example for illustration. It can be understood that this method can also be applied to a server, and can also be applied It is based on a system including a terminal and a server, and is realized through the interaction between the terminal and the server. The above-mentioned terminal can be but not limited to various personal computers, notebook computers, smart phones, tablet computers, etc., and the server can be an independent server or multiple servers Formed server clusters to achieve. In this embodiment, the data writing method includes the following steps:

Step 102, in response to the data write request, acquire multiple write processes corresponding to the data write request.

Wherein, the data write request may be a write request received by the terminal, and the write request may be a request for instructing the terminal to perform a data write operation to the distributed file system, and the distributed file system may be a file system managed The physical storage resource may be connected to each node through a computer network, or may be a file system composed of multiple different logical disk partitions, and the data writing request may include at least one writing process.

Optionally, the terminal may also obtain multiple writing processes that perform data writing operations on the distributed file system within a preset time period.

In implementation, other terminal devices can generate multiple write processes based on the requirements of the actual application scenario, and generate a data write request based on the write process, so that the data write request can be sent to the terminal corresponding to the distributed file system. After receiving the data writing request, the terminal may analyze the data writing request to obtain a writing process or multiple writing processes included in the data writing request. The terminal may also receive a plurality of data writing requests within a preset time period, and respectively obtain the writing processes included in the data writing requests.

Step 104, for the target writing process among the multiple writing processes, determine the target data block of the target writing process.

Wherein, the target writing process may be any one writing process among multiple writing processes. The target data block of the target writing process may be the original data block carried by the target writing process, or may be a data block generated by merging the original data block with data blocks of other writing processes.

In implementation, after acquiring the multiple writing processes corresponding to the distributed file system, the terminal may sort the writing processes according to the time sequence corresponding to the writing processes, and process the writing processes sequentially. In this way, the specific processing procedure of the terminal for the target writing process among the multiple writing processes may be: after the terminal determines the target writing process, it needs to determine the target data block corresponding to the target writing process.

Step 106: Divide the target data block according to the size of the preset stripe, obtain the first data block that meets the size of the preset stripe and the remaining data blocks that do not meet the size of the preset stripe, and write the target data block through the target write process. Enter the data in the first data block corresponding to the target write process.

Wherein, the size of the preset stripe may be the length of data included in the preset stripe in the distributed file system, for example, it may be A. The data length of the first data block that meets the preset stripe size can be the same as the length of data that can be contained in the pre-configured preset stripe, and the remaining data blocks that do not meet the preset stripe size can have a data length smaller than the pre-configured The length of data blocks that a preset stripe can contain.

In implementation, for the target writing process, after the terminal obtains the target data block corresponding to the target writing process, it can segment the target data block according to the data length of the preset stripe to obtain one or more first data blocks (stripe-aligned data), and the remaining data blocks. After the terminal obtains the first data block and the remaining data blocks corresponding to the target writing process, the terminal can write the data in the first data block to the distributed file system respectively, so as to ensure the timeliness of data writing.

Step 108, if there is a next writing process of the target writing process, transmit the remaining data blocks to the next writing process of the target writing process through the preset data transmission network, so that the next writing process can determine the remaining data blocks and The original data block of the next writing process is the target data block, and as a new target writing process, the step of dividing the target data block according to the preset stripe size is performed.

Wherein, the preset data transmission network may be an OpenMPI network (a high-performance message passing library), which is used for data transmission between processes.

In implementation, after the terminal determines the first data block and the remaining data blocks corresponding to the target writing process, the terminal can write the first data block into the distributed file system corresponding to the data write request, and at the same time, the terminal can determine whether the Whether there is a next write process after the target write process.

After the terminal determines that there is a next write process in the target write process, the terminal can exchange the remaining data blocks from the target write process to the next write of the target write process through the preset data transmission network process, that is, the next write process that transfers the remaining data blocks from the target write process to the target write process. In this way, after the terminal determines that the next write process of the target write process has received the remaining data block of the target write process, the terminal can determine the remaining data block and the original data block of the next write process of the target write process as the target data block, and The next write process of the target write process is used as a new target write process, and the target data block is divided according to the size of the preset stripe in the above embodiment, and the first data block that meets the size of the preset stripe and the first data block that does not meet the requirements are obtained. A step of presetting the remaining data blocks of the stripe size, and writing the data in the first data block corresponding to the target writing process through the target writing process.

In an example, as shown in Figure 2a, the target writing process can be rank0, and the target data block corresponding to the rank can be data block ①; the terminal divides the data block ① according to the size of the preset stripe, and the obtained data satisfies the preset The schematic diagram of the first data block of the stripe size and the remaining data blocks that do not meet the preset stripe size can be shown in Figure 2b, the first data block can be data block ③ and data block ④, and the remaining data blocks can be data block ⑤, that is, the terminal can divide the data block ① into data block ③, data block ④ and data block ⑤. The terminal can write data block ③ and data block ④ into the distributed file system.

As shown in Figure 2b, the terminal can determine that there is a next writing process after the target writing process (rank0), that is, rank1. Based on this, the terminal can transfer the remaining data blocks of rank0 ( Data block ⑤), exchanged from rank0 to rank1. That is to say, the next writing process of the target writing process can be rank1, the original data block corresponding to rank1 can be data block ②, and the target data block of rank1 can be the remaining data block (data block ⑤) and the original data block (data block ②) For the merged data block, after the terminal determines the target data block of rank1, it can use rank1 as a new target writing process to perform the steps of the above embodiment, that is, divide the target data block of rank1 according to the size of the preset stripe, Obtain a new first data block and a new remaining data block. The new first data block can be a data block composed of data block ⑤+data block ⑥, data block ⑦, and data block ⑧. The new remaining data block can be Data block ⑨.

In the above data writing method, in response to the data writing request, multiple writing processes corresponding to the data writing request are acquired. A target data block of the target write process is determined for a target write process among the multiple write processes. The target data block is divided according to the size of the preset stripe, and the first data block that meets the size of the preset stripe and the remaining data blocks that do not meet the size of the preset stripe are obtained, and are written into the target write process through the target write process. The data in the first data block corresponding to the process. In the case that there is a next write process of the target write process, the remaining data blocks are transmitted to the next write process of the target write process through the preset data transmission network, so that the next write process can determine the remaining data blocks and the next write process The original data block of the process is the target data block, and as a new target writing process, the step of dividing the target data block according to the size of the preset stripe is performed. By adopting this method, at the initial stage of data writing, the data to be written can be divided into full stripes, and the data that is not full of stripes and resulting in cross-process writing can be exchanged to other writing processes for writing, avoiding the use of distributed Interval locks, and high-speed data transmission through the preset data transmission network, can communicate, exchange, splice and write data at high speed, ensure strict stripe alignment, and further ensure the efficiency of data writing by improving.

In one embodiment, after the step of dividing the target data block according to the preset stripe size to obtain the first data block meeting the preset stripe size and the remaining data blocks not meeting the preset stripe size, The data writing method also includes:

If there is no next writing process in the target writing process, the remaining data blocks corresponding to the target writing process are cached in the preset cache space.

In implementation, when the terminal determines that there is no next writing process after the target writing process, the terminal may store the remaining data blocks in the preset remaining buffer space, that is, cache the remaining data blocks.

In this embodiment, by caching the remaining data blocks, data integrity and timeliness of data writing can be guaranteed.

In one embodiment, the data writing method also includes:

After detecting the closing command of the file to be written corresponding to the data write request, the remaining data blocks in the preset cache space are obtained, and the remaining data blocks are concatenated and written into the file to be written.

Wherein, the file to be written corresponding to the data write request may be a file located in the distributed file system, the address of the file to be written may be the destination storage address of the data carried by the writing process, and the closing command of the file to be written may be It is an operation performed after the file to be written has been written, and is used to instruct the terminal to close the file to be written.

In implementation, if the terminal detects that there is a close instruction for the file to be written corresponding to the data write request, the terminal can perform splicing writing on the remaining blocks cached in the preset cache space, that is, splicing writing To the file to be written in the distributed file system.

In an example, after the terminal stores the remaining data blocks in the preset cache space, if it is determined that there is no other writing process within the preset time period, the terminal can concatenate the remaining data blocks in the preset cache space and write them to be written. Into the file, that is, after the pen is written and before the next pen is written, the remaining data blocks in the preset cache space are spliced and written into the file to be written.

In this embodiment, the remaining data blocks can be written in time, avoiding data loss and improving the efficiency of data writing.

In one embodiment, as shown in FIG. 3 , the specific processing of the step "determining the target data block of the target writing process for each of the multiple writing processes" includes:

Step 302, for the target writing process, there is a previous writing process of the target writing process in the target writing process, obtain the remaining data blocks of the previous writing process, and obtain the original data blocks of the target writing process.

Step 304, splicing the remaining data block of the previous writing process and the original data block of the target writing process to obtain the target data block of the target writing process.

Wherein, the previous writing process of the target writing process may be a process in the migration order of the target writing process according to the arrangement order of the processes.

In implementation, for the target writing process, the terminal can determine whether there are other processes before the target writing process, that is, determine whether the target writing process is the initial writing process, and the initial writing process can be determined according to the order of the processes at the first place the writing process. When the terminal determines that there is a previous writing process in the target writing process, the terminal can obtain the remaining data blocks of the previous writing process through the preset data transmission network, so that the terminal can also obtain the original data carried by the target writing process itself block, and splicing the remaining data block of the previous writing process of the target writing process and the original data block of the target writing process to obtain the target data block of the target writing process.

In this embodiment, the data blocks of the previous writing process can be exchanged to the target writing process, so as to implement data writing without distributed lock competition and improve data writing efficiency.

In one embodiment, the data writing method also includes:

For the target writing process, if there is no previous writing process of the target writing process in the target writing process, the original data block of the target writing process is determined as the target data block of the target writing process.

In implementation, for the target writing process, the terminal can determine whether there are other processes before the target writing process, that is, determine whether the target writing process is the initial writing process, and the initial writing process can be determined according to the order of the processes at the first place the writing process. When the terminal determines that the target writing process does not have a previous writing process, that is, when the terminal determines that the target writing process is the initial writing process, the terminal can obtain the original data block carried by the target writing process itself, and The original data block is used as the target data block of the target writing process.

In this embodiment, the original data block of the writing process can be used as the target data block to implement data writing with strict stripe alignment.

In one embodiment, the data writing method also includes:

For the target writing process, if there is a previous writing process of the target writing process in the target writing process, and there is no remaining data block in the previous writing process, determine that the original data block of the target writing process is the target data block of the target writing process .

In implementation, when the terminal determines that there is a previous write process of the target write process in the target write process, but there is no remaining data block in the previous write process of the target write process, the terminal may not need to pass The preset data transmission network performs data exchange, so that the terminal can determine that the original data block of the target writing process is the target data block of the target writing process.

In this embodiment, the original data block of the writing process can be used as the target data block to implement data writing with strict stripe alignment.

In this embodiment, in order to ensure the consistency of data information in the distributed system, it needs to be protected by distributed locks (distributed interval locks). Distributed applications operate the same file synchronously based on multiple access points, which involves the competition of distributed locks, and the resulting distributed lock requests and recalls. Therefore, the competition of distributed locks is a performance bottleneck for OpenMPI applications to write files. That is to say, OpenMPI is a multi-process application. (rank0 and rank1) When requesting a distributed lock, if the lock is used by other access points, the recall operation will be performed first to clear the local cache of the original access point, and then the distributed lock server will cache the distributed lock to the requesting node local.

A data writing method provided by this application can strictly ensure the alignment of written data stripes through data exchange and splicing, fundamentally avoiding the competition of distributed locks and the occurrence of supplementary reading, and greatly improving the writing efficiency . And for OpenMPI application scenarios where there is only writing but no reading, you can use the ioctl interface provided by the distributed file system to close the distributed interval lock and completely skip the logic of the distributed interval lock, which can further improve the writing efficiency. This application provides The data writing method of the OpenMPI process can use the high-speed communication of the OpenMPI process to exchange and splicing the written data to ensure strict stripe alignment and avoid the overhead of distributed locks in the distributed file system. For non-OpenMPI applications, during the writing process, it will not call ioctl interface, no change in writing logic, ensuring compatibility.

It should be understood that although the steps in the flow charts involved in the above embodiments are shown sequentially according to the arrows, these steps are not necessarily executed sequentially in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order restriction on the execution of these steps, and these steps can be executed in other orders. Moreover, at least some of the steps in the flow charts involved in the above embodiments may include multiple steps or stages, and these steps or stages are not necessarily executed at the same time, but may be executed at different times, The execution order of these steps or stages is not necessarily performed sequentially, but may be executed in turn or alternately with other steps or at least a part of steps or stages in other steps.

Based on the same inventive concept, an embodiment of the present application further provides a data writing device for implementing the above-mentioned data writing method. The solution to the problem provided by the device is similar to the implementation described in the above method, so the specific limitations in one or more embodiments of the data writing device provided below can be referred to above for the data writing method limited and will not be repeated here.

In one embodiment, as shown in FIG. 5, a data writing device 500 is provided, including:

The first acquiring module 502 is configured to acquire multiple writing processes corresponding to the data writing request in response to the data writing request;

The first determining module 504 is configured to determine the target data block of the target writing process for the target writing process among the multiple writing processes;

The write module 506 is configured to divide the target data block according to the size of the preset stripe, obtain the first data block that meets the size of the preset stripe and the remaining data blocks that do not meet the size of the preset stripe, and pass the target Write process, write the data in the first data block corresponding to the target write process;

The transmission module 508 is configured to transmit the remaining data blocks to the next writing process of the target writing process through a preset data transmission network if there is a next writing process of the target writing process, so that the next writing process can determine the remaining The data block and the original data block of the next writing process are the target data block, and as a new target writing process, the step of dividing the target data block according to the size of the preset stripe is performed.

In one of the embodiments, the data writing device 500 also includes:

The caching module is configured to cache the remaining data blocks corresponding to the target writing process in a preset cache space when there is no next writing process in the target writing process.

In one of the embodiments, the data writing device 500 also includes:

The second acquisition module is configured to acquire the remaining data blocks in the preset cache space after detecting the closing instruction of the file to be written corresponding to the data writing request, and concatenate and write the remaining data blocks The file to be written.

In one of the embodiments, the first determining module 504 is specifically configured to:

For the target writing process, there is a previous writing process of the target writing process in the target writing process, obtaining the remaining data blocks of the previous writing process, and obtaining the original data blocks of the target writing process;

Splicing the remaining data blocks of the last writing process and the original data blocks of the target writing process to obtain the target data blocks of the target writing process.

In one of the embodiments, the data writing device 500 also includes:

The second determining module is configured to, for the target writing process, determine that the original data block of the target writing process is the target writing process when the target writing process does not have a previous writing process of the target writing process target data block.

In one of the embodiments, the data writing device 500 also includes:

The third determining module is configured to, for the target writing process, when the target writing process has a previous writing process of the target writing process, and the remaining data block does not exist in the previous writing process , determining the original data block of the target writing process as the target data block of the target writing process.

Each module in the above-mentioned data writing device 500 may be realized in whole or in part by software, hardware or a combination thereof. The above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, and can also be stored in the memory of the computer device in the form of software, so that the processor can invoke and execute the corresponding operations of the above-mentioned modules.

In one embodiment, a computer device is provided. The computer device may be a server, and its internal structure may be as shown in FIG. 6 . The computer device includes a processor, memory and a network interface connected by a system bus. Wherein, the processor of the computer device is used to provide calculation and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, computer programs and databases. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage medium. The database of the computer device is used to store written data. The network interface of the computer device is used to communicate with an external terminal via a network connection. When the computer program is executed by the processor, a data writing method is realized.

Those skilled in the art can understand that the structure shown in FIG. 6 is only a block diagram of a part of the structure related to the solution of this application, and does not constitute a limitation on the computer equipment to which the solution of this application is applied. The specific computer equipment can be More or fewer components than shown in the figures may be included, or some components may be combined, or have a different arrangement of components.

In one embodiment, there is also provided a computer device, including a memory and a processor, where a computer program is stored in the memory, and the processor implements the steps in the above method embodiments when executing the computer program.

In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored, and when the computer program is executed by a processor, the steps in the foregoing method embodiments are implemented.

In one embodiment, a computer program product is provided, including a computer program, and when the computer program is executed by a processor, the steps in the foregoing method embodiments are implemented.

Those of ordinary skill in the art can understand that all or part of the processes in the methods of the above-mentioned embodiments can be completed by instructing related hardware through computer programs, and the computer programs can be stored in a non-volatile computer-readable memory In the medium, when the computer program is executed, it may include the processes of the embodiments of the above-mentioned methods. Wherein, any reference to storage, database or other media used in the various embodiments provided in the present application may include at least one of non-volatile and volatile storage. Non-volatile memory can include read-only memory (Read-Only Memory, ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive variable memory (ReRAM), magnetic variable memory (Magnetoresistive Random Access Memory, MRAM), Ferroelectric Random Access Memory (FRAM), Phase Change Memory (Phase Change Memory, PCM), graphene memory, etc. The volatile memory may include random access memory (Random Access Memory, RAM) or external cache memory. As an illustration and not a limitation, RAM can be in various forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (Dynamic Random Access Memory, DRAM). The databases involved in the various embodiments provided in this application may include at least one of a relational database and a non-relational database. The non-relational database may include a blockchain-based distributed database, etc., but is not limited thereto. The processors involved in the various embodiments provided by this application can be general-purpose processors, central processing units, graphics processors, digital signal processors, programmable logic devices, data processing logic devices based on quantum computing, etc., and are not limited to this.

The technical features of the above embodiments can be combined arbitrarily. To make the description concise, all possible combinations of the technical features in the above embodiments are not described. However, as long as there is no contradiction in the combination of these technical features, they should be It is considered to be within the range described in this specification.

The above-mentioned embodiments only express several implementation modes of the present application, and the description thereof is relatively specific and detailed, but should not be construed as limiting the patent scope of the present application. It should be noted that those skilled in the art can make several modifications and improvements without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application should be determined by the appended claims.

Claims (10)

1. A data writing method, characterized in that the method comprises: Responding to the data write request, acquiring multiple write processes corresponding to the data write request; For a target writing process among multiple writing processes, determine a target data block of the target writing process; Divide the target data block according to the size of the preset stripe, obtain the first data block that meets the size of the preset stripe and the remaining data blocks that do not meet the size of the preset stripe, and pass the A target write process, writing data in the first data block corresponding to the target write process; In the case that the next writing process of the target writing process exists, the remaining data block is transmitted to the next writing process of the target writing process through a preset data transmission network, so that the next writing process Determining the remaining data blocks and the original data blocks of the next writing process as target data blocks, and performing the step of dividing the target data blocks according to the size of the preset stripes as a new target writing process . 2. The method according to claim 1, wherein the target data block is divided according to the size of the preset stripe to obtain the first data block satisfying the size of the preset stripe and After the step of remaining data blocks that do not meet the preset stripe size, the method further includes: If there is no next writing process in the target writing process, the remaining data blocks corresponding to the target writing process are cached in a preset cache space. 3. The method according to claim 2, wherein the method further comprises: After detecting the close instruction of the file to be written corresponding to the data write request, acquiring the remaining data blocks in the preset cache space, and concatenating the remaining data blocks into the file to be written. 4. The method according to claim 1, wherein, for each target writing process in a plurality of writing processes, determining the target data block of the target writing process comprises: For the target writing process, there is a previous writing process of the target writing process in the target writing process, obtaining the remaining data blocks of the previous writing process, and obtaining the original data blocks of the target writing process; Splicing the remaining data blocks of the last writing process and the original data blocks of the target writing process to obtain the target data blocks of the target writing process. 5. method according to claim 4, is characterized in that, described method also comprises: For the target writing process, the previous writing process of the target writing process does not exist in the target writing process, and the original data block of the target writing process is determined as the target data block of the target writing process. 6. The method according to claim 4, characterized in that the method further comprises: For the target writing process, when the target writing process has a previous writing process of the target writing process and the remaining data block does not exist in the previous writing process, determine the target writing process The original data block of is the target data block of the target write process. 7. A data writing device, characterized in that the device comprises: A first acquiring module, configured to acquire multiple writing processes corresponding to the data writing request in response to the data writing request; A first determining module, configured to, for a target writing process among multiple writing processes, determine a target data block of the target writing process; A write module, configured to divide the target data block according to the size of the preset stripe, to obtain the first data block that meets the size of the preset stripe and the remaining data that does not meet the size of the preset stripe block, and write the data in the first data block corresponding to the target writing process through the target writing process; A transmission module, configured to transmit the remaining data blocks to the next writing process of the target writing process through a preset data transmission network in the case that there is a next writing process of the target writing process, so that all The next write process determines that the remaining data blocks and the original data blocks of the next write process are target data blocks, and as a new target write process, executes the process of performing the processing of the target data according to the size of the preset stripe Steps to divide blocks. 8. A computer device, comprising a memory and a processor, the memory stores a computer program, wherein the processor implements the method according to any one of claims 1 to 6 when executing the computer program step. 9. A computer-readable storage medium, on which a computer program is stored, wherein when the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are realized. 10. A computer program product, comprising a computer program, characterized in that the computer program When executed by a processor, the steps of the method according to any one of claims 1 to 6 are realized.

Images