CN114327800A - Method and device for writing in topic, processor and stream processing platform - Google Patents

Abstract

The application provides a topic writing method, a writing device, a processor and a stream processing platform, wherein the method comprises the following steps: acquiring incoming data of a stream processing platform to obtain target data; generating a plurality of records according to the target data, wherein the records comprise partition fields, and the partition fields are used for representing the corresponding relation between the records and topic of the stream processing platform; and calling the current transaction, and writing a plurality of records into corresponding topics according to the partition fields, wherein the records correspond to the topics one by one. In the method, because the partition field can represent the corresponding relation between the record and the topic of the stream processing platform, a plurality of topics can be written in only through one connector, and only one node needs to be established by one connector.

Description

Topic writing method, writing device, processor and stream processing platform

technical field

The present application relates to the field of topic writing, and in particular, to a topic writing method, a writing device, a computer-readable storage medium, a processor, and a stream processing platform.

Background technique

In Kafka, if you want to write multiple topic records at the same time, you can submit these records through the same transaction. Flink's native Kafka connector only uses Kafka transactions in Semantic.EXACTLY_ONCE mode to ensure that a single record is written exactly once, but it does not support writing multiple topics at the same time. Currently, to implement Flink multi-topic writing, you can add multiple sinks, one sink corresponds to one Kafka connector. The method steps are as follows:

(1) Create a Kafka connector FlinkKafkaProducer corresponding to a topic;

(2) Add the created FlinkKafkaProducers to the task topology one by one through the addSink() method of DataStream.

Although the multi-sink scheme realizes that a record can be sent to multiple topics, this scheme has many disadvantages:

(1) Strong consistency between topics cannot be guaranteed. The writing of multiple topics is not in the same transaction, and there is no guarantee that the writing will succeed or fail at the same time, so strong consistency between topics cannot be guaranteed;

(2) Reduce task processing performance. If there are thousands of topics, then thousands of sinks need to be created. For Flink, adding a node also means that Flink needs to allocate corresponding resources to run the tasks corresponding to this node. At the same time, data is routed to the sink, and network transmission is also required in the middle. Therefore, creating thousands of sinks requires a lot of resources. It will greatly reduce the task processing performance.

Although the multi-sink scheme realizes that a record can be sent to multiple topics, this scheme has many disadvantages:

(1) Strong consistency between topics cannot be guaranteed. The writing of multiple topics is not in the same transaction, and there is no guarantee that the writing will succeed or fail at the same time, so strong consistency between topics cannot be guaranteed;

(2) Reduce task processing performance. If there are thousands of topics, then thousands of sinks need to be created. For Flink, adding a node also means that Flink needs to allocate corresponding resources to run the tasks corresponding to this node. At the same time, data is routed to the sink, and network transmission is also required in the middle. Therefore, creating thousands of sinks requires a lot of resources. It will greatly reduce the task processing performance.

The above information disclosed in this Background section is only for enhancement of understanding of the background of the technology described in this article and therefore it may contain certain information that does not form part of the already known in this country to a person of ordinary skill in the art known prior art.

SUMMARY OF THE INVENTION

The main purpose of the present application is to provide a topic writing method, writing device, computer-readable storage medium, processor and stream processing platform, so as to solve the problem that multiple topic writing occupies too many resources in the prior art.

According to an aspect of the embodiments of the present invention, a method for writing a topic is provided, including: acquiring incoming data of a stream processing platform to obtain target data; and generating a plurality of records according to the target data, and the records include a partition field , the partition field is used to represent the correspondence between the record and the topic of the stream processing platform; call the current transaction, and write a plurality of the records into the corresponding topic according to the partition field, the record One-to-one correspondence with the topic.

Optionally, before calling the current transaction and writing a plurality of the records into the corresponding topic according to the partition field, the method further includes: querying the status of the current transaction; when the current transaction is in In the case of the to-be-committed state, the current transaction is added to the to-be-committed transaction set, and a new current transaction is generated.

Optionally, after invoking the current transaction and writing a plurality of the records into the corresponding topic according to the partition field, the method further includes: adding the current transaction to a set of transactions to be submitted; The transactions of the set of transactions to be committed are sequentially committed.

Optionally, generating multiple records according to the target data includes: obtaining a corresponding task configuration file according to the task code of the target data, where the configuration mode of the task configuration file includes an output field determination mode and a partition mode; The target data is processed by the output field determination method and the partition method to generate a plurality of the records.

Optionally, the determining manner of the output field includes a manner of outputting all fields and a manner of outputting a specified part of the fields.

Optionally, the partition mode includes a custom partition mode and a default partition mode.

According to another aspect of the embodiments of the present invention, a device for writing a topic is further provided, including: an acquiring unit for acquiring incoming data from a stream processing platform to obtain target data; a generating unit for acquiring data according to the target The data generates multiple records, the records include a partition field, and the partition field is used to represent the corresponding relationship between the record and the topic of the stream processing platform; the writing unit is used to call the current transaction, and according to the partition The field writes a plurality of the records into the corresponding topic, and the records are in one-to-one correspondence with the topic.

According to yet another aspect of the embodiments of the present invention, a computer-readable storage medium is further provided, the computer-readable storage medium includes a stored program, wherein the program executes any one of the methods.

According to yet another aspect of the embodiments of the present invention, a processor is also provided, and the processor is configured to run a program, wherein any one of the methods is executed when the program is run.

According to an aspect of the embodiments of the present invention, a stream processing platform is also provided, including a connector, one or more processors, a memory, a display device, and one or more programs, wherein the connector is used to write a record into the topic, the one or more programs are stored in the memory and configured to be executed by the one or more processors, the one or more programs including for performing any of the described method.

In the embodiment of the present invention, in the above topic writing method, first, the incoming data of the stream processing platform is obtained to obtain target data; then, a plurality of records are generated according to the above target data, and the above records include a partition field, and the above partition field It is used to represent the correspondence between the above records and the topics of the above stream processing platform; finally, the current transaction is called, and a plurality of the above records are written into the corresponding above topics according to the above partition fields, and the above records are in one-to-one correspondence with the above topics. This method calls the current transaction to write multiple records to the corresponding topic. Since the partition field can represent the correspondence between the above record and the topic of the above stream processing platform, only one connector is needed to complete the writing of multiple topics. , a connector only needs to create one node, which greatly reduces the resource occupancy rate compared to the prior art to create one node for one topic, and solves the problem that multiple topic writes occupy too many resources in the prior art.

Description of drawings

The accompanying drawings that form a part of the present application are used to provide further understanding of the present application, and the schematic embodiments and descriptions of the present application are used to explain the present application and do not constitute improper limitations on the present application. In the attached image:

FIG. 1 shows a schematic diagram of a method for writing a topic according to an embodiment of the present application;

FIG. 2 shows a schematic diagram of a writing device for a topic according to an embodiment of the present application;

FIG. 3 shows a schematic diagram of processing logic of a connector according to an embodiment of the present application.

Detailed ways

It should be noted that the embodiments in the present application and the features of the embodiments may be combined with each other in the case of no conflict. The present application will be described in detail below with reference to the accompanying drawings and in conjunction with the embodiments.

In order to make those skilled in the art better understand the solutions of the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are only The embodiments are part of the present application, but not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative work shall fall within the scope of protection of the present application.

It should be noted that the terms "first", "second", etc. in the description and claims of the present application and the above drawings are used to distinguish similar objects, and are not necessarily used to describe a specific sequence or sequence. It is to be understood that the data so used are interchangeable under appropriate circumstances for the embodiments of the application described herein. Furthermore, the terms "comprising" and "having" and any variations thereof, are intended to cover non-exclusive inclusion, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those expressly listed Rather, those steps or units may include other steps or units not expressly listed or inherent to these processes, methods, products or devices.

For the convenience of description, some nouns or terms involved in the embodiments of the present application are described below:

Flink: Apache Flink is a framework and distributed processing engine for stateful computation on unbounded and bounded data streams.

Kafka: Apache Kafka is a distributed stream processing platform.

topic: topic is a topic in Kafka, a category or feed name to publish records to.

FlinkKafkaProducer: Flink's native Kafka connector for writing data in Kafka topics.

Checkpoint: Flink is a state fault-tolerant mechanism. It persists the state of the operator and forms checkpoints for failure recovery.

As mentioned in the background art, multi-topic writing in the prior art occupies too many resources. In order to solve the above problem, a typical implementation of the present application provides a topic writing method, writing Apparatus, computer readable storage medium, processor and stream processing platform.

According to an embodiment of the present application, a method for writing a topic is provided.

FIG. 1 is a flowchart of a method for writing a topic according to an embodiment of the present application. As shown in Figure 1, the method includes the following steps:

Step S101, acquiring incoming data of the stream processing platform to obtain target data;

Step S102, generate a plurality of records according to the above-mentioned target data, the above-mentioned records include a partition field, and the above-mentioned partition fields are used to represent the corresponding relationship between the above-mentioned records and the topics of the above-mentioned stream processing platform;

Step S103, calling the current transaction, and writing a plurality of the above records into the corresponding above-mentioned topic according to the above-mentioned partition field, and the above-mentioned records are in one-to-one correspondence with the above-mentioned topic.

In the above topic writing method, first, the incoming data of the stream processing platform is obtained to obtain target data; then, a plurality of records are generated according to the above target data, and the above records include a partition field, and the above partition field is used to represent the above records and the above The correspondence between the topics of the stream processing platform; finally, the current transaction is called, and a plurality of the above records are written into the corresponding above topics according to the above partition fields, and the above records are in one-to-one correspondence with the above topics. This method calls the current transaction to write multiple records to the corresponding topic. Since the partition field can represent the correspondence between the above record and the topic of the above stream processing platform, only one connector is needed to complete the writing of multiple topics. , a connector only needs to create one node, which greatly reduces the resource occupancy rate compared to the prior art to create one node for one topic, and solves the problem that multiple topic writes occupy too many resources in the prior art.

In the actual application process, multiple topic names can be configured under the kafka node. For example, tb-dpdtl-curt-acc-topic and dpst-open-clos-acct-book-info-topic can both be topic names.

It should be noted that the steps shown in the flowcharts of the accompanying drawings may be executed in a computer system, such as a set of computer-executable instructions, and, although a logical sequence is shown in the flowcharts, in some cases, Steps shown or described may be performed in an order different from that herein.

In an embodiment of the present application, before calling the current transaction and writing a plurality of the records into the corresponding topic according to the partition field, the method further includes: querying the status of the current transaction; when the current transaction is pending In the case of committing, add the above current transaction to the set of transactions to be committed, and generate a new current transaction. Then, by calling the current transaction, write multiple records into the corresponding topic according to the partition field in the record, and further It realizes writing multiple topics through one connector, further solves the problem that multiple topic writing occupies too many resources in the prior art, and further ensures the strong consistency of data between topics.

Specifically, in the actual application process, when Flink initiates a checkpoint, it will call the snapshotState() method of the connector to take a snapshot; call the preCommit() method to add the current transaction to the set of transactions to be committed, and call the beginTransactionInternal() method at the same time Generate a new current transaction.

In a specific embodiment of the present application, when the current transaction is in a commit failure state, the transaction is rolled back, the operations on the transaction are not saved, and the behavior is restored to the last correct state. For example, in the scenario of transferring money, if there is an error in the transfer process, that is, the money in the bank account of the requesting party has been deducted, but the money in the bank account of the other party has not increased, so an error will occur, so in this kind of In the scenario, the transaction needs to be rolled back, that is, the operation of deducting money from the requested bank account is not saved, so that it can return to the state before the transaction is executed, that is, the money in the requester's bank account has not been deducted. If successful, the money in the other party's bank account has not increased.

In another embodiment of the present application, after calling the current transaction and writing a plurality of the records into the corresponding topic according to the partition field, the method further includes: adding the current transaction to the set of transactions to be submitted; Add transactions that commit the above set of pending transactions in order. In this embodiment, the transactions in the to-be-committed transaction set are submitted according to the sequence of adding the current transaction to the to-be-committed transaction set, so as to ensure that whether the transaction is successfully executed can be known in a timely manner.

In a specific embodiment of the present application, after the checkpoint is completed, Flink will call back the notifyCheckpointComplete() method of the connector, and call the commit() method to submit the transactions to be submitted one by one.

In order to generate multiple records relatively simply and efficiently, in another embodiment of the present application, generating multiple records according to the above target data includes: obtaining a corresponding task configuration file according to the task code of the above target data, and the above task configuration file The configuration mode includes an output field determination mode and a partition mode; the target data is processed according to the output field determination mode and the partition mode to generate a plurality of the above records.

In a specific embodiment of this application, when a record enters, the invoke() method of the connector will be called; in the invoke() method, multiple records are generated one-to-one corresponding to multiple topics, and the current transaction is used to send the record.

In the actual application process, the above task configuration file can be a yaml task configuration file.

In still another embodiment of the present application, the above-mentioned output field determination method includes a method of outputting all fields and a method of outputting a specified part of the fields, which ensures that the target data can be processed flexibly according to the output field determination method in the future, and further This ensures that multiple records are obtained more efficiently.

Specifically, the all_fields node can be used to indicate whether all fields are output. When the value of all_fields is true, the method of outputting all fields is determined; when the value of all_fields is false, the method of outputting the specified part of the fields is determined. Of course, When all_fields is false, you can also specify the fields to output by configuring the optional_fields node.

In order to ensure that the target data can be processed flexibly according to the partition mode in the future to obtain multiple records, in an embodiment of the present application, the partition mode includes a custom partition mode and a default partition mode.

Specifically, the partition_field node can be used to configure the partition mode, and the partition_class node can be used to configure a custom partition mode and perform custom partitions. Of course, you can also select a configuration from the partition_field node and the partition_class node to implement the custom partition; when the partition_field node When neither partition_class nor partition_class is configured, the default partition mode is round-robin.

This embodiment of the present application further provides a topic writing apparatus. It should be noted that the topic writing apparatus of the present application embodiment may be used to execute the topic writing method provided by the present application embodiment. The following describes the topic writing apparatus provided by the embodiments of the present application.

FIG. 2 is a schematic diagram of a writing apparatus for a topic according to an embodiment of the present application. As shown in Figure 2, the device includes:

The obtaining unit 10 is used to obtain the incoming data of the stream processing platform, and obtain the target data;

The generating unit 20 is configured to generate a plurality of records according to the above-mentioned target data, the above-mentioned records include a partition field, and the above-mentioned partition field is used to represent the corresponding relationship between the above-mentioned record and the topic of the above-mentioned stream processing platform;

The writing unit 30 is configured to call the current transaction, and write a plurality of the above records into the corresponding above-mentioned topic according to the above-mentioned partition field, and the above-mentioned records are in one-to-one correspondence with the above-mentioned topic.

In the above-mentioned topic writing device, the acquiring unit is used to acquire the incoming data of the stream processing platform to obtain the target data; the generating unit is used to generate a plurality of records according to the above-mentioned target data, and the above-mentioned records include a partition field, and the above-mentioned partition field is used for Indicates the correspondence between the above record and the topic of the above stream processing platform; the writing unit is used to call the current transaction, and write a plurality of the above records into the corresponding above topic according to the above partition field, and the above records are in one-to-one correspondence with the above topics. The device can call the current transaction to write multiple records to the corresponding topic. Since the partition field can represent the correspondence between the records and the topic of the stream processing platform, only one connector can be used to complete the writing of multiple topics. , a connector only needs to create one node, which greatly reduces the resource occupancy rate compared to the prior art to create one node for one topic, and solves the problem that multiple topic writes occupy too many resources in the prior art.

In the actual application process, multiple topic names can be configured under the kafka node. For example, tb-dpdtl-curt-acc-topic and dpst-open-clos-acct-book-info-topic can both be topic names.

In an embodiment of the present application, the above-mentioned apparatus further includes a query unit and a first addition unit, wherein the above-mentioned query unit is used for invoking the current transaction and before writing the plurality of above-mentioned records into the corresponding above-mentioned topics according to the above-mentioned partition fields , query the state of the current transaction; the first adding unit is used to add the current transaction to the transaction set to be submitted when the current transaction is in the state to be submitted, and generate a new current transaction, and subsequently call the current transaction by calling the current transaction. , according to the partition field in the record, multiple records are written into the corresponding topic, which further realizes the writing of multiple topics through one connector, and further solves the problem that multiple topic writing occupies too many resources in the prior art The problem further ensures the strong consistency of data between topics.

Specifically, in the actual application process, when Flink initiates a checkpoint, it will call the snapshotState() method of the connector to take a snapshot; call the preCommit() method to add the current transaction to the set of transactions to be committed, and call the beginTransactionInternal() method at the same time Generate a new current transaction.

In a specific embodiment of the present application, when the current transaction is in a commit failure state, the transaction is rolled back, the operations on the transaction are not saved, and the behavior is restored to the last correct state. For example, in the scenario of transferring money, if there is an error in the transfer process, that is, the money in the bank account of the requesting party has been deducted, but the money in the bank account of the other party has not increased, so an error will occur, so in this kind of In the scenario, the transaction needs to be rolled back, that is, the operation of deducting money from the requested bank account is not saved, so that it can return to the state before the transaction is executed, that is, the money in the requester's bank account has not been deducted. If successful, the money in the other party's bank account has not increased.

In another embodiment of the present application, the above-mentioned apparatus further includes a second adding unit and a submitting unit, wherein the above-mentioned second adding unit is used to call the current transaction and write a plurality of the above-mentioned records into the corresponding partition fields according to the above-mentioned partition fields. After the above topic, the above current transaction is added to the set of transactions to be submitted; the above submission unit is configured to submit the transactions of the above set of transactions to be submitted in the order of addition. In this embodiment, the transactions in the to-be-committed transaction set are submitted according to the sequence of adding the current transaction to the to-be-committed transaction set, so as to ensure that whether the transaction is successfully executed can be known in a timely manner.

In a specific embodiment of the present application, after the checkpoint is completed, Flink will call back the notifyCheckpointComplete() method of the connector, and call the commit() method to submit the transactions to be submitted one by one.

In order to generate multiple records relatively simply and efficiently, in another embodiment of the present application, the generating unit includes an acquisition module and a processing module, wherein the acquisition module is configured to acquire the corresponding task configuration according to the task code of the target data The configuration method of the task configuration file includes an output field determination method and a partition method; the processing module is configured to process the target data according to the output field determination method and the partition method to generate a plurality of the above records.

In a specific embodiment of this application, when a record enters, the invoke() method of the connector will be called; in the invoke() method, multiple records are generated one-to-one corresponding to multiple topics, and the current transaction is used to send the record.

In the actual application process, the above task configuration file can be a yaml task configuration file.

In still another embodiment of the present application, the above-mentioned output field determination method includes a method of outputting all fields and a method of outputting a specified part of the fields, which ensures that the target data can be processed flexibly according to the output field determination method in the future, and further This ensures that multiple records are obtained more efficiently.

Specifically, the all_fields node can be used to indicate whether all fields are output. When the value of all_fields is true, the method of outputting all fields is determined; when the value of all_fields is false, the method of outputting the specified part of the fields is determined. Of course, When all_fields is false, you can also specify the fields to output by configuring the optional_fields node.

In order to ensure that the target data can be processed flexibly according to the partition mode in the future to obtain multiple records, in an embodiment of the present application, the partition mode includes a custom partition mode and a default partition mode.

Specifically, the partition_field node can be used to configure the partition mode, and the partition_class node can be used to configure a custom partition mode and perform custom partitions. Of course, you can also select a configuration from the partition_field node and the partition_class node to implement the custom partition; when the partition_field node When neither partition_class nor partition_class is configured, the default partition mode is round-robin.

The above topic writing device includes a processor and a memory. The above acquisition unit, generation unit, and writing unit are all stored in the memory as program units, and the processor executes the program units stored in the memory to implement corresponding functions.

The processor includes a kernel, and the kernel calls the corresponding program unit from the memory. The kernel can be set to one or more, and the problem that multiple topic writes occupy too many resources in the prior art can be solved by adjusting the kernel parameters.

Memory may include non-persistent memory in computer readable media, random access memory (RAM) and/or non-volatile memory, such as read only memory (ROM) or flash memory (flash RAM), the memory including at least one memory chip.

An embodiment of the present invention provides a computer-readable storage medium on which a program is stored, and when the program is executed by a processor, the above-mentioned method for writing a topic is implemented.

An embodiment of the present invention provides a processor, where the processor is used to run a program, wherein the method for writing the topic is executed when the program is running.

In a typical embodiment of the present application, a stream processing platform is also provided, the stream processing platform includes a connector, one or more processors, a memory, a display device and one or more programs, wherein the above-mentioned connection The processor is used to write records to the topic, and the one or more programs described above are stored in the above-mentioned memory and are configured to be executed by the one or more processors described above, and the one or more programs described above include a program for executing any of the above method.

The above-mentioned stream processing platform can execute any of the above-mentioned methods. In the above-mentioned topic writing method, firstly, the incoming data of the stream processing platform is obtained to obtain the target data; then, a plurality of records are generated according to the above-mentioned target data, and the above-mentioned The record includes a partition field, and the above-mentioned partition field is used to represent the corresponding relationship between the above-mentioned record and the topic of the above-mentioned stream processing platform; finally, the current transaction is called, and a plurality of the above-mentioned records are written into the corresponding above-mentioned topic according to the above-mentioned partition field, and the above-mentioned records are related to the above-mentioned topic. The above topics correspond one-to-one. This method calls the current transaction to write multiple records to the corresponding topic. Since the partition field can represent the correspondence between the above record and the topic of the above stream processing platform, only one connector is needed to complete the writing of multiple topics. , a connector only needs to create one node, which greatly reduces the resource occupancy rate compared to the prior art to create one node for one topic, and solves the problem that multiple topic writes occupy too many resources in the prior art.

In order to make those skilled in the art understand the technical solutions of the present application more clearly, the following will be described in conjunction with specific embodiments:

Example

This application can support the writing of multiple topics through a custom Kafka connector, that is, the Kafka connector can be named as MultiTopicFlinkKafkaProducer. The functions of each class/interface are as follows: CheckpointedFunction interface, which is used to perform checkpoint operations, The initializeState() and snapshotState() methods are used to initialize the state and take snapshots of the state respectively; CheckpointListener interface: This interface is used to monitor checkpoint activities, notifyCheckpointComplete() and notifyCheckpointAborted() will be called when the checkpoint is successful and canceled respectively; RichSinkFunction interface : This interface is the rich function interface of Sink, which provides life cycle methods. The open() and close() methods will be called when the operator is created and destroyed respectively; TwoPhaseCommitSinkFunction interface: This abstract class encapsulates the logic template submitted in two phases. Subclasses can implement their own transaction commit logic by implementing beginTransaction(), preCommit(), and commit() methods.

As shown in Figure 3, the connector MultiTopicFlinkKafkaProducer inherits the TwoPhaseCommitSinkFunction interface, and its processing logic is: first, the checkpoint processing logic is performed. When Flink initiates a checkpoint, the snapshotState() method of the connector is called to take a snapshot; in the preCommit() method , add the current transaction to the transaction set to be submitted, and call the beginTransactionInternal() method to generate a new current transaction; secondly, process a single record, that is, the record enters the processing logic, when a record enters, the invoke() method of the Kafka connector will be called; in the invoke() method, multiple records are generated one-to-one corresponding to multiple topics, and the current transaction is used to send the records. Among them, when creating a record, the task code will be read from the incoming data, the corresponding yaml task configuration file will be read according to the task code, the output field determination method and partitioning method will be defined in the task configuration file, and corresponding records will be generated according to the configuration. ;Finally, call back after the checkpoint is completed. When the checkpoint is completed, Flink will call back the notifyCheckpointComplete() method of the connector; call the commit() method to submit the transactions to be committed one by one.

An embodiment of the present invention provides a device. The device includes a processor, a memory, and a program stored in the memory and running on the processor. The processor implements at least the following steps when executing the program:

Step S101, acquiring incoming data of the stream processing platform to obtain target data;

Step S102, generate a plurality of records according to the above-mentioned target data, the above-mentioned records include a partition field, and the above-mentioned partition fields are used to represent the corresponding relationship between the above-mentioned records and the topics of the above-mentioned stream processing platform;

Step S103, calling the current transaction, and writing a plurality of the above records into the corresponding above-mentioned topic according to the above-mentioned partition field, and the above-mentioned records are in one-to-one correspondence with the above-mentioned topic.

The devices in this article can be servers, PCs, PADs, mobile phones, and so on.

The present application also provides a computer program product that, when executed on a data processing device, is adapted to execute a program initialized with at least the following method steps:

Step S101, acquiring incoming data of the stream processing platform to obtain target data;

Step S102, generate a plurality of records according to the above-mentioned target data, the above-mentioned records include a partition field, and the above-mentioned partition fields are used to represent the corresponding relationship between the above-mentioned records and the topics of the above-mentioned stream processing platform;

Step S103, calling the current transaction, and writing a plurality of the above records into the corresponding above-mentioned topic according to the above-mentioned partition field, and the above-mentioned records are in one-to-one correspondence with the above-mentioned topic.

In the above-mentioned embodiments of the present invention, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, reference may be made to related descriptions of other embodiments.

In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. The device embodiments described above are only illustrative. For example, the division of the above-mentioned units may be a logical function division. In actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated. to another system, or some features can be ignored, or not implemented. On the other hand, the shown or discussed mutual coupling or direct coupling or communication connection may be through some interfaces, indirect coupling or communication connection of units or modules, and may be in electrical or other forms.

The units described above as separate components may or may not be physically separated, and components shown as units may or may not be physical units, that is, may be located in one place, or may be distributed to multiple units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution in this embodiment.

In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically alone, or two or more units may be integrated into one unit. The above-mentioned integrated units may be implemented in the form of hardware, or may be implemented in the form of software functional units.

If the above-mentioned integrated units are implemented in the form of software functional units and sold or used as independent products, they may be stored in a computer-readable computer-readable storage medium. Based on such understanding, the technical solution of the present invention essentially or the part that contributes to the prior art or the whole or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a computer-readable The storage medium includes several instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the above-mentioned methods of the various embodiments of the present invention. The aforementioned computer-readable storage medium includes: U disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), mobile hard disk, magnetic disk or optical disk and other various programs that can store programs medium of code.

From the above description, it can be seen that the above-mentioned embodiments of the present application achieve the following technical effects:

1) In the topic writing method of the present application, first, the incoming data of the stream processing platform is obtained to obtain the target data; then, a plurality of records are generated according to the above target data, and the above records include partition fields, and the above partition fields are used for The correspondence between the above records and the topics of the above stream processing platform is represented; finally, the current transaction is called, and a plurality of the above records are written into the corresponding above topics according to the above partition fields, and the above records are in one-to-one correspondence with the above topics. This method calls the current transaction to write multiple records to the corresponding topic. Since the partition field can represent the correspondence between the above record and the topic of the above stream processing platform, only one connector is needed to complete the writing of multiple topics. , a connector only needs to create one node, which greatly reduces the resource occupancy rate compared to the prior art to create one node for one topic, and solves the problem that multiple topic writes occupy too many resources in the prior art.

2) In the topic writing device of the present application, the acquisition unit is used to acquire the incoming data of the stream processing platform to obtain the target data; the generation unit is used to generate a plurality of records according to the above-mentioned target data, and the above-mentioned records include a partition field, and the above-mentioned The partition field is used to represent the corresponding relationship between the above record and the topic of the above stream processing platform; the writing unit is used to call the current transaction, and write a plurality of the above records into the corresponding above topic according to the above partition field, and the above record and the above topic are one. A correspondence. The device can call the current transaction to write multiple records into the corresponding topic. Since the partition field can represent the correspondence between the records and the topic of the stream processing platform, multiple topics can be written through only one connector. , a connector only needs to create one node, which greatly reduces the resource occupancy rate compared to the prior art to establish one node for one topic, and solves the problem that multiple topic writes occupy too many resources in the prior art.

3) The stream processing platform of the present application can execute any one of the above methods. In the above topic writing method, first, the incoming data of the stream processing platform is obtained to obtain target data; record, the record includes a partition field, and the partition field is used to represent the corresponding relationship between the record and the topic of the stream processing platform; finally, the current transaction is called, and a plurality of the records are written into the corresponding topic according to the partition field , the above records are in one-to-one correspondence with the above topics. This method calls the current transaction to write multiple records to the corresponding topic. Since the partition field can represent the correspondence between the above record and the topic of the above stream processing platform, only one connector is needed to complete the writing of multiple topics. , a connector only needs to create one node, which greatly reduces the resource occupancy rate compared to the prior art to create one node for one topic, and solves the problem that multiple topic writes occupy too many resources in the prior art.

The above descriptions are only preferred embodiments of the present application, and are not intended to limit the present application. For those skilled in the art, the present application may have various modifications and changes. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of this application shall be included within the protection scope of this application.

Claims (10)

1. A method of writing to topoic, comprising:

acquiring incoming data of a stream processing platform to obtain target data;

generating a plurality of records according to the target data, wherein the records comprise partition fields, and the partition fields are used for representing the corresponding relation between the records and topic of the stream processing platform;

and calling the current transaction, and writing a plurality of records into the corresponding topic according to the partition field, wherein the records correspond to the topic one by one.

2. The method of claim 1, wherein prior to invoking a current transaction and writing a plurality of the records to the corresponding topic according to the partition field, the method further comprises:

querying the state of the current transaction;

and under the condition that the current transaction is in a to-be-committed state, adding the current transaction to a to-be-committed transaction set, and generating a new current transaction.

3. The method of claim 1, wherein after invoking a current transaction and writing a plurality of the records to the corresponding topic according to the partition field, the method further comprises:

adding the current transaction to a set of transactions to be committed;

and submitting the transactions of the transaction set to be submitted according to the adding sequence.

4. The method of any of claims 1 to 3, wherein generating a plurality of records from the target data comprises:

acquiring a corresponding task configuration file according to the task code of the target data, wherein the configuration mode of the task configuration file comprises an output field determining mode and a partition mode;

and processing the target data according to the output field determining mode and the partition mode to generate a plurality of records.

5. The method of claim 4, wherein the output field determination manner comprises a manner of outputting all fields and a manner of outputting a specified partial field.

6. The method of claim 4, wherein the partition modes comprise a custom partition mode and a default partition mode.

7. A topic writer, comprising:

the acquisition unit is used for acquiring the incoming data of the stream processing platform to obtain target data;

a generating unit, configured to generate a plurality of records according to the target data, where the records include partition fields, and the partition fields are used to represent a correspondence between the records and topic of the stream processing platform;

and the writing unit is used for calling the current transaction and writing a plurality of records into the corresponding topic according to the partition field, wherein the records correspond to the topic one by one.

8. A computer-readable storage medium, characterized in that the computer-readable storage medium comprises a stored program, wherein the program performs the method of any one of claims 1 to 6.

9. A processor, characterized in that the processor is configured to run a program, wherein the program when running performs the method of any of claims 1 to 6.

10. A stream processing platform comprising a connector for writing a record to topic, one or more processors, memory, a display device, and one or more programs stored in the memory and configured for execution by the one or more processors, the one or more programs comprising instructions for performing the method of any of claims 1 to 6.

Images