CN112860750A - Data processing method and device, electronic equipment and storage medium - Google Patents
Data processing method and device, electronic equipment and storage medium Download PDFInfo
- Publication number
- CN112860750A CN112860750A CN202110264709.2A CN202110264709A CN112860750A CN 112860750 A CN112860750 A CN 112860750A CN 202110264709 A CN202110264709 A CN 202110264709A CN 112860750 A CN112860750 A CN 112860750A
- Authority
- CN
- China
- Prior art keywords
- target
- database
- topic
- data
- task
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/245—Query processing
- G06F16/2455—Query execution
- G06F16/24552—Database cache management
-
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/245—Query processing
- G06F16/2457—Query processing with adaptation to user needs
- G06F16/24578—Query processing with adaptation to user needs using ranking
-
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/27—Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
-
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/28—Databases characterised by their database models, e.g. relational or object models
- G06F16/284—Relational databases
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
- Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management
Landscapes
- Engineering & Computer Science (AREA)
- Databases & Information Systems (AREA)
- Theoretical Computer Science (AREA)
- Data Mining & Analysis (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Computational Linguistics (AREA)
- Computing Systems (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The application discloses a data processing method, a data processing device and electronic equipment, wherein the method comprises the following steps: acquiring aggregate messages to be processed from a cache queue, wherein the aggregate messages comprise target identifications and task categories of target objects, and the task categories represent categories of task processing items executed on the target objects; executing the task processing items on the target object according to the target identification and the task category to obtain a processing result; and storing the processing result according to a preset storage rule. The method can reduce the data processing amount in the system, thereby relieving the pressure of the database and improving the reading and writing speed of the data.
Description
Technical Field
The present disclosure relates to the field of internet data processing technologies, and in particular, to a data processing method, an apparatus, an electronic device, and a computer-readable storage medium.
Background
In the internet field, a user can issue rich and diverse objects through terminal equipment, wherein the objects can be videos, characters, images and the like; after the terminal equipment sends the object to be issued to the server, the server can aggregate the objects to the same category by aggregating the objects; therefore, when other users browse the objects by using the terminal equipment, the server can display the objects in the categories which are interested by the users for the users to view so as to improve the user viscosity.
In the process of implementing the present application, the inventors find that, in the existing method, when aggregation processing is performed, aggregation processing is often performed on an object at the same time after the object is successfully issued, and obtained aggregated data is synchronously written into a database, for example, a mysql database; however, during the peak time of user activity, for example, at night or on holidays, the issuing amount of the object is increased sharply, and therefore, the corresponding aggregation processing is also increased, and the existing mode of writing the aggregated data into the database synchronously after the object is successfully issued will cause the data request amount of the database to increase suddenly during the peak time, which causes a large pressure to the database, and the database is generally deployed according to the maximum data request amount, so that the problem of waste of database resources during the off-peak time also exists.
In the above, the aggregation processing for an object after the object is successfully issued is taken as an example to illustrate the problems existing in the existing method; in practice, after an object is successfully published, when related information of the object needs to be updated, the existing method also has the above problem, and therefore, it is necessary to provide a data processing method to aggregate data according to business characteristics, so as to avoid the problems of database over-pressure and resource waste which may be brought about.
Disclosure of Invention
The embodiment of the disclosure provides a data processing method, a data processing device, an electronic device, a storage medium and a data acquisition method, so that when data aggregation processing is performed on the electronic device, data can be aggregated and stored according to service characteristics, a database is efficiently utilized, and data writing and reading speeds are improved.
In a first aspect of the present disclosure, a data processing method is provided, where the method includes:
acquiring aggregate messages to be processed from a cache queue, wherein the aggregate messages comprise target identifications and task categories of target objects, and the task categories represent categories of task processing items executed on the target objects;
executing the task processing items on the target object according to the target identification and the task category to obtain a processing result; and the number of the first and second groups,
and storing the processing result according to a preset storage rule.
In a second aspect of the present disclosure, there is also provided a data obtaining method, including:
receiving an object acquisition request sent by terminal equipment, wherein the object acquisition request comprises a target identifier of a target object to be acquired, and the target object is associated with a target topic;
in the case that the target object is judged not to be stored in a third database according to the target identification and the topic identification of the target topic, acquiring the target object from a second database according to the target identification and the topic identification, wherein the second database is a non-relational database for storing data in a non-volatile memory, and the third database is a non-relational database for storing data in a volatile memory;
and providing the target object to the terminal equipment.
In a third aspect of the present disclosure, there is also provided a data processing apparatus, including:
the device comprises an aggregation message acquisition module, a cache queue processing module and a processing module, wherein the aggregation message acquisition module is used for acquiring to-be-processed aggregation messages from the cache queue, the aggregation messages comprise target identifications and task categories of target objects, and the task categories represent the categories of task processing items executed on the target objects;
a processing result obtaining module, configured to execute the task processing item on the target object according to the target identifier and the task category, and obtain a processing result; and the number of the first and second groups,
and the storage module is used for storing the processing result according to a preset storage rule.
According to a fourth aspect of the present disclosure, there is also provided an electronic device, including:
a memory for storing executable instructions;
a processor configured to execute the electronic device to perform the method according to the first or second aspect of the disclosure according to the control of the executable instructions.
According to a fifth aspect of the present disclosure, there is also provided a computer readable storage medium storing a computer program readable and executable by a computer, the computer program being configured to perform the method according to the first or second aspect of the present disclosure when read and executed by the computer.
The method has the beneficial effects that according to the method of the embodiment of the disclosure, after obtaining the aggregation message to be processed from the cache queue, the electronic device, for example, the server, can execute the corresponding task processing item on the target object according to the task category of the message and obtain the processing result, and then store the processing result by using the preset rule. According to the method, the cache queue is set, and the aggregation message needing to be processed is placed in the cache queue, so that the server can asynchronously process the aggregation task in the service, the data request amount at the same time, particularly in a peak period, is reduced, the pressure of a database is relieved, the reading and writing speed of data is improved, and the use efficiency of database resources is improved.
Other features of the present disclosure and advantages thereof will become apparent from the following detailed description of exemplary embodiments thereof, which proceeds with reference to the accompanying drawings.
Drawings
The accompanying drawings, which are incorporated in and constitute a part of the specification, illustrate embodiments of the disclosure and together with the description, serve to explain the principles of the disclosure.
Fig. 1 is a schematic flow chart of a data processing method according to an embodiment of the present disclosure.
Fig. 2 is a schematic diagram of a data processing framework provided by an embodiment of the present disclosure.
Fig. 3 is a schematic flow chart diagram of a data acquisition method provided in an embodiment of the present disclosure.
Fig. 4 is a functional block diagram of a data processing apparatus provided in an embodiment of the present disclosure.
Fig. 5 is a schematic diagram of a hardware structure of an electronic device provided in an embodiment of the present disclosure.
Detailed Description
Various exemplary embodiments of the present disclosure will now be described in detail with reference to the accompanying drawings. It should be noted that: the relative arrangement of the components and steps, the numerical expressions, and numerical values set forth in these embodiments do not limit the scope of the present disclosure unless specifically stated otherwise.
The following description of at least one exemplary embodiment is merely illustrative in nature and is in no way intended to limit the disclosure, its application, or uses.
Techniques, methods, and apparatus known to those of ordinary skill in the relevant art may not be discussed in detail but are intended to be part of the specification where appropriate.
In all examples shown and discussed herein, any particular value should be construed as merely illustrative, and not limiting. Thus, other examples of the exemplary embodiments may have different values.
The terms first, second and the like in the description and in the claims of the present disclosure are used for distinguishing between similar elements and not necessarily for describing a particular sequential or chronological order. It is to be understood that the data so used is interchangeable under appropriate circumstances such that embodiments of the disclosure can be practiced in sequences other than those illustrated or described herein.
It should be noted that: like reference numbers and letters refer to like items in the following figures, and thus, once an item is defined in one figure, further discussion thereof is not required in subsequent figures.
< method embodiment I >
Fig. 1 is a schematic flow chart of a data processing method according to an embodiment of the present disclosure. The method can be implemented by an electronic device, and the electronic device can be a server, for example, a blade server, a rack server, or the like, or can also be a server cluster deployed in a cloud, which is not particularly limited herein; of course, in the specific implementation, the method may also be applied to the terminal device according to actual needs, and is not particularly limited herein.
As shown in fig. 1, the method of the present embodiment may include the following steps S1100-S1300, which are described in detail below.
Step S1100, obtaining an aggregated message to be processed from a cache queue, where the aggregated message includes a target identifier of a target object and a task category, and the task category represents a category of a task processing item executed on the target object.
Specifically, the method aims at the problems that the existing method generally carries out real-time aggregation processing on objects after the objects are successfully issued, and synchronously writes the obtained aggregated data into a database, and carries out processing such as database writing processing and the like synchronously in real time when the related information of the objects needs to be updated, and the like, so that the data request amount is increased suddenly, the database pressure is large, and the data reading and writing speed is influenced. In the present application, when an electronic device, for example, a server, performs data processing, tasks that do not need to be processed in real time are placed in a cache queue according to characteristics of different services, and a message processing timing is controlled to reduce a data request amount in a system at the same time, so that database pressure can be reduced, and a data read/write speed in the system can be increased, which will be described in detail below.
In this embodiment, the aggregation message is a message including task transactions to be processed corresponding to the target object, where the task transactions in this embodiment may be transactions for aggregating the target object on a certain topic, or transactions for aggregating related information of the target object, for example, attributes such as likes and recommended values; of course, in the specific implementation, the task processing items may be other items, and are not particularly limited herein.
Topics used to describe common characteristics of objects or describe categories to which objects belong in content-class applications, such as short video applications, news applications, knowledge sharing applications, etc., for example, a topic tag "# Dance" may be used to indicate that an object is an object related to Dance.
The target object can be video, text, image and other objects, for example, short video produced by a user in a short video application; for another example, the news application may be news produced by a user, and is not particularly limited herein. In this embodiment, if no special description is given, the target object is taken as an example of a video in a short video application.
In this embodiment, the buffer queue is a queue for storing aggregate messages to be processed, and may be, for example, a message queue based on rocktmq, where rocktmq is a distributed message publish-subscribe system, and of course, the buffer queue may also be implemented in other manners, and may also be, for example, a queue based on Rabbitmq, which is not limited herein.
In one embodiment, the buffer queue may be maintained by: receiving an object publishing request, wherein the object publishing request comprises a first object to be published and an identification of a first topic associated with the first object; responding to the object publishing request, publishing the first object, and obtaining the identifier of the first object; and obtaining an identification of the first topic; and generating an aggregation message with a task category as a first category according to the identification of the first object and the identification of the first topic, and writing the aggregation message into the cache queue, wherein the first category represents that topic aggregation items are executed on the first object.
Specifically, in the short video application, when the user produces an object through the terminal device, for example, when the short video "video a" is produced and uploaded to the server using the mobile phone, step S1101 may be executed first, and the mobile phone may transmit a distribution request for distributing "video a" to the server, where the distribution request may include the short video and a topic associated with the short video selected or input by the user, for example, the topic "# lance"; after receiving the publishing request, the server needs to timely return a response message indicating whether the short video is successfully published to the terminal device, and therefore, the server needs to perform step S1102 to perform short video publishing processing in real time, where the publishing processing specifically includes generating a video identifier of the short video, allocating a storage space to the video identifier, acquiring a storage path of the video identifier, and storing relevant information of the short video, such as a name of the short video, a publishing user, a storage address, a praise value, and the like, in a short video list; after the short video is successfully distributed, the server may execute step S1103, and return a response message indicating that the distribution is successful to the terminal device; in this embodiment, different from the prior art that after a short video is successfully published, it is further required to synchronously perform a topic aggregation process, that is, a process of aggregating the short video into a topic associated with the short video, in this embodiment, the object publishing process and the topic aggregation process may be decoupled, that is, after the object is successfully published through the above step S1103, step S1104 is executed, a video identifier of "video a" is obtained, and a topic identifier of the topic "# lance" associated with the object is obtained, so that an aggregation message with a first task category is generated, and the aggregation message is placed in a cache queue, so that a topic aggregation task of "video a" can be asynchronously processed.
As can be seen from the above description, in this embodiment, when the electronic device performs data processing, for a core service, for example, an object publishing service may be processed in real time, so as to provide a response to a user in time; for secondary services, for example, topic aggregation services, the topic aggregation services and object publishing services can be decoupled and processed asynchronously, so that the data request volume in the system, especially the data request volume in a peak period, is reduced, and the data read-write speed is further improved.
In order to further reduce the request amount in the system in the implementation, the data request amount in the system may be further reduced by updating the traffic for other non-core traffic, for example, for the hot value of the target object, and putting the updated traffic into the cache queue.
Specifically, in one embodiment, the buffer queue may also be maintained by: receiving an object heat update request, wherein the object heat update request comprises an identifier of a second object to be updated with a heat value; and generating an aggregation message with the task class being a second class according to the identification of the second object, and writing the aggregation message into the cache queue, wherein the second class represents that the hot update item is executed on the second object. In this embodiment, the popularity value represents a degree to which the object is paid attention by the user or a degree to which the user has an interest in the object, and in a specific implementation, the popularity value may be an approval value of the object; alternatively, the update time of the object may be used; or, a preset heat calculation algorithm may be used to integrate the calculated values according to the weight of the relevant information of the object, where the relevant information of the object may be, for example, a like value, an update time, and the like. In this embodiment, for convenience of description, the praise value directly targeted for the heat value is taken as an example for description.
In specific implementation, when displaying the objects belonging to the same topic to the user, the terminal device may first sort the objects included in the object according to the hot value of the object, and display the objects whose sort meets a set condition, for example, the object whose hot value is 100 th, for the user to view.
It should be noted that, in order to increase the data reading speed, when a general electronic device, such as a server, stores an object and information of the object, such as a heat value, the general electronic device may generally store data in a manner of using a "cache database + a persistent storage database", that is, data with a high user access frequency, for example, a high heat value, is stored in the cache database, so that the data is read in time and displayed to a user for viewing; and storing the data with low user access frequency, for example, low heat value in a persistent database, wherein the cache database may be a volatile memory-based, for example, a redis database of a memory, and the persistent database may be a non-volatile memory-based, for example, a mysql database of a disk.
Specifically, in this embodiment, after the user approves the object, for example, "video a", through the terminal device, the terminal device sends an object popularity update request to the server; different from the prior art, the server sends the write request to the database, for example, the persistent storage database and/or the cache database in real time for each object hot update request, which causes a sudden increase in the request amount of the database and further may cause a problem of high database pressure. In this embodiment, after receiving the update request, the server responds to the update request, instead of sending a write request to the database in real time, by obtaining the video identifier of "video a" from the request, generating an aggregation message with the task class of the second class, and placing the aggregation message into the cache queue, the hot update task of "video a" can be asynchronously processed, and the purpose of reducing the request volume of the database can be achieved.
It should be noted that, the above is a specific embodiment of maintaining a buffer queue provided in this embodiment, and during specific implementation, the aggregation message may not include a task category, but a corresponding buffer queue is set for each task processing item; of course, the buffer queue may also include other types of aggregation processing items, which are not described herein again.
In an embodiment, after the cache queue is obtained through the maintenance in the above steps, when the messages in the cache queue are consumed, the aggregated message may be further obtained from the cache queue according to a preset reading algorithm, where the preset reading algorithm includes at least one of a message processing time corresponding to the aggregated message according to a preset time interval, so as to maintain the data request amount in the system within a set number range, where the preset time interval and the message processing time set for each aggregated message may be set according to needs, for example, the message processing time may be subdivided according to the priority of each task, and details are not described here.
After step S1100, step S1200 is executed to execute the task processing item on the target object according to the target identifier and the task category, and obtain a processing result.
After the step S1100, the aggregate message to be processed is obtained from the cache queue, and then the corresponding task processing item can be executed on the target object according to the task type in the aggregate message and the target identifier of the target object.
In one embodiment, the task category includes a first category representing performing a topic aggregation item on the target object, the aggregation message further including a target topic identification therein, the target topic identification being an identification of a target topic associated with the target object; the executing the task processing item to the target object according to the target identifier and the task category to obtain a processing result includes: and under the condition that the task category is the first category, establishing mapping data reflecting the association relation between the target object and the target topic as the processing result according to the target identification and the target topic identification.
Specifically, when the task category in the aggregated message is the first category, the target topic identifier of the target topic associated with the target object may also be obtained from the message, wherein the related processing on how to obtain the target topic identifier is not repeated here; then, the aggregation of the target objects under the associated target topics is represented by establishing list data similar to [ target topic identification, target identification ].
For example, for video a and its associated topic "# lance," video a may be aggregated under topic "# lance" by adding the identification of video a to the topic list corresponding to that topic.
It should be noted that, in this embodiment, when publishing an object, a user selects a topic as an object, and then the electronic device acquires a topic identifier of a corresponding topic to aggregate the object under the corresponding topic; in specific implementation, only the object may be published without selecting a topic by the user, but the electronic device automatically acquires the identification of the topic associated with the target object by extracting the feature information of the object after obtaining the aggregated message for the target object; further, the object —. topic can be at least one, not specifically limited herein.
The above describes how to execute the corresponding task processing item on the target object and obtain the processing result, taking the task category as the first category, i.e., the topic aggregation item as an example. In the specific implementation, the task type may be other types, for example, the second type described in step S1100, that is, the heat update item, which will be described in detail below.
In one embodiment, the task category includes a second category representing performing a hotness update transaction on the target object; the executing the task processing item to the target object according to the target identifier and the task category to obtain a processing result includes: and under the condition that the task type is the second type, acquiring a heat value of the target object in a first time range as the processing result according to the target identification, wherein the first time range comprises a time range taking the current first time as a starting time and taking a second time as an ending time, and the second time is obtained by adding the first time and a preset time length.
In this embodiment, the hot value may be an like value of the target object; in a specific implementation, the obtaining, according to the target identifier, a heat value of the target object in a first time range as the processing result includes: and counting a new praise value of the target object in the first time range as the processing result according to the target identification.
Specifically, for the hotness update items corresponding to the target object, for example, the like value update items, the server may not need to perform database write operation in real time for each like value; instead, after obtaining the aggregated message with the task category being the second category, the new value of praise of the target object in a preset time period, for example, a one-minute time range, may be counted from the current first time, and the new value of praise in the time range may be used as the processing result.
In the above, how to execute the corresponding task processing items on the target object to obtain the processing results corresponding to the target object and the task category is described in detail for different task categories in the aggregated message, and after the processing results are obtained, the processing results can be stored according to the preset storage rule.
After step S1200, step S1300 is executed, and the processing result is stored according to a preset storage rule.
In specific implementation, a design of "redis + mysql" may be generally adopted to store data, where a redis database is used as a cache database to store data with high user access frequency, for example, high heat; whereas mysql is used to persist the stored data. However, since the redis is based on volatile memory, such as memory, to store data, it is costly and has limited capacity, which makes it impractical to cache the full amount of data in the redis database.
For example, tens of thousands of videos may be aggregated under a topic, and the video objects are sorted with like values directly as heat values; based on capacity considerations, generally only videos ranked top 100 in the hotness value of the topic are taken to be cached in the redis database. In this case, when the user uses the terminal device to view the video, if only the video ranked at the top 100 on the topic is viewed, the server directly obtains the data from the redis database and provides the data for the terminal device to display; when the user needs to view the video ranked after 100, the user generally needs to read data from mysql, that is, cache penetration occurs.
However, since mysql is a relational database that stores data based on a non-volatile memory, this results in a high time consumption for reading data from mysql when cache penetration occurs, which may affect the user experience; especially in the case of a large amount of data requests, the database pressure may be further increased, which may cause the performance of the database to be degraded, thereby affecting the stability and reliability of the whole system. In a specific implementation, it is of course also possible to cache as much data as possible by increasing the memory capacity of the redis, however, this approach is costly and may also cause a problem of resource waste when caching more data.
To solve the above problem, in the present embodiment, unlike a simple way of increasing the memory capacity, the present embodiment adds a non-relational database for storing data based on a non-volatile memory to perform data caching, and stores the data by using a preset storage rule to solve the above problem, which will be described in detail below.
In one embodiment, the storing the processing result according to a preset storage rule includes: and writing the mapping data into a first database and a second database, wherein the first database is a non-relational database, the second database is a relational database, and the data of the first database and the data of the second database are both stored in a nonvolatile memory.
In this embodiment, the first database may be a key-value-based database that manages data and stores the data in a nonvolatile memory, and the database may be, for example, a pika database, where pika is a key-value database that can persistently store data, has high processing performance, and can implement persistent storage of data; the second database, for example, may be an existing mysql database for persisted data.
In one embodiment, in the case that the task category is a first category, for mapping data established to represent an association relationship between a target object and a target topic, the writing the mapping data into a first database includes: acquiring a real-time heat value of the target object according to the target identification; obtaining a topic record to be stored according to the mapping data and the real-time heat value; writing the topic records to be stored into a target topic data list corresponding to the target topic identifier, wherein the target topic data list is an ordered list in the first database, and the target topic data list takes the heat value of an object as a key value and sequences the stored topic records.
Specifically, after obtaining mapping data representing the association relationship between the target object and the target topic, when storing the mapping data, a topic record to be stored, that is, aggregated object list data corresponding to the topic, may be obtained by obtaining a real-time popularity value, for example, a like value, of the target object; and stores the data in the first database, i.e., the ordered list data structure (SortedSet) for pika.
In this embodiment, by storing data using the above storage rule, when cache penetration occurs, that is, when data that needs to be shown by the terminal device is not cached in a third database for caching data, for example, redis data, the server may not need to directly read data from the second database, for example, mysql database, but obtain data from the first database with relatively high processing performance; in addition, because the ordered list is used for storing the corresponding correspondence of the topics in the first database, and the list can automatically sort the data in the first database according to the set key value, the data does not need to be sorted every time the data is read, and the data reading speed can be further improved.
In this embodiment, the storing the processing result according to a preset storage rule further includes: acquiring a real-time heat value of the target object according to the target identification; judging whether the sequence of the target object in all the objects related to the target topic meets a preset condition or not according to the real-time heat value to obtain a judgment result; and writing the mapping data into a third database under the condition that the judgment result shows that the sequencing of the target objects meets a preset condition, wherein the third database is a non-relational database for storing data in a volatile memory.
That is, when the processing result is stored, for example, when the mapping data is stored, in order to ensure the validity of the data in the third database, for example, the redis cache, the hot value of the target object may also be obtained in real time while the mapping data is stored, so as to determine whether to write the mapping data into the redis database at the same time, so as to avoid the cache penetration phenomenon as much as possible.
Of course, the above is only one embodiment provided by the present application, and in specific implementation, it is not necessary to determine whether to write the mapping data into the third database when storing the mapping data, but an existing cache updating mechanism is used to update the cache data in the third database, which is not limited herein.
In one embodiment, in the case that the task category is the second category, for a heat value of the target object in the first time range obtained by aggregation, for example, a new praise value of the target object in the first time range, the storing the processing result according to a preset storage rule includes: obtaining a second moment according to the first moment and the preset duration; and at the second moment, updating the heat degree value of the target object stored in a first database and a second database according to the target identification and the heat degree value, wherein the first database is a non-relational database, the second database is a relational database, and the data of the first database and the data of the second database are both stored in a nonvolatile memory.
Please refer to fig. 2, which is a schematic diagram of a data processing framework according to an embodiment of the disclosure. In fig. 2, a terminal device is taken as an example to be described as a terminal running a short video application, as shown in fig. 2, in a specific implementation, the terminal device may send a request for object publishing, a request for object approval, and the like to a server; after the server obtains the request, the server may process, in real time, core services that need to be processed in real time, for example, a published object, according to service characteristics of the request, and may generate corresponding task messages for non-core services that have low real-time requirements, for example, services such as topic aggregation, like a praise update, and the like, and the task messages are placed in a rocktmq queue shown in fig. 2, and then the messages in the queue are asynchronously consumed, so that the data request volume in the system is reduced, and further, the database pressure is reduced. In addition, as shown in fig. 2, the storage system of this embodiment includes multiple levels of caches, for example, a third database that can be regarded as a first level cache, that is, a redis database; a first database, i.e., pika database, which may be considered a second level cache; and a second database, i.e., mysql database, which may be considered a third level cache; when storing data, the electronic device may cache the hotspot data in a redis database, and for non-hotspot data, may persistently store it in the pika database and the mysql database at the same time; when the read data is subjected to cache penetration, the data does not need to be read in the mysql database, but the data is obtained from the pika database with the processing performance higher than that of the mysql database, and since the data stored by the pika is sorted, the read data does not need to be read and then sorted, but can be directly provided for the terminal equipment, so that the speed of displaying the data by the terminal equipment is improved, and the user experience is further improved.
Referring to fig. 2, in this embodiment, after receiving an object acquisition request sent by a terminal device, an electronic device, for example, a server, may acquire a topic identifier of a target topic associated with a target object according to a target identifier included in the object acquisition request; then, it may be determined whether the target object is stored in a first-level cache, for example, a third database, redis shown in fig. 2, according to the topic identifier and the target identifier; if the target object does not exist, the data does not need to be acquired from a second database with lower processing performance, namely mysql, but the target object can be acquired from the full amount of data stored in the second-level cache and provided for the terminal device, so that the terminal device can rapidly display the target object for the user to view. In addition, since the database used as the second-level cache in this embodiment is based on a non-volatile memory, for example, a disk pika database, the expansion cost is much lower than that of a manner in which a memory is used as an expansion.
In summary, in the data processing method provided in this embodiment, after obtaining the aggregate message to be processed from the cache queue, the electronic device, for example, the server, may execute the corresponding task processing item on the target object according to the task category of the message and obtain the processing result, and then store the processing result by using the preset rule. The method sets a buffer queue and puts the aggregation message to be processed into the buffer queue, so that a server can asynchronously process the aggregation message in service to reduce the data processing amount at the same time, especially at a peak time, thereby reducing the pressure of a database and improving the reading and writing speed of data; in addition, when data storage is carried out, the first database is added, for example, the pika database is used as a second-level cache in the system, so that the cost can be saved, and the data acquisition speed can be further improved.
< method example two >
Corresponding to the first method embodiment, the present embodiment further provides a data obtaining method, which may be applied to an electronic device, for example, may be used in a server.
Please refer to fig. 3, which is a flowchart illustrating a data acquisition method according to an embodiment of the disclosure. As shown in FIG. 3, the method includes the following steps S3100-S3300, which are described in detail below.
Step S3100, receiving an object acquisition request sent by a terminal device, where the object acquisition request includes a target identifier of the target object to be acquired, and the target object is associated with a target topic.
Step S3200, in a case where it is determined that the target object is not stored in a third database according to the target identifier and the topic identifier of the target topic, acquiring the target object from a second database according to the target identifier and the topic identifier, wherein the second database is a non-relational database that stores data in a non-volatile memory, and the third database is a non-relational database that stores data in a volatile memory.
And step S3300, providing the target object to the terminal equipment.
< apparatus embodiment >
Corresponding to the above embodiments, in this embodiment, a data processing apparatus is further provided, and as shown in fig. 4, the apparatus 4000 may include an aggregation message obtaining module 4100, a processing result obtaining module 4200, and a storage module 4300.
The aggregate message acquiring module 4100 is configured to acquire an aggregate message to be processed from a cache queue, where the aggregate message includes a target identifier of a target object and a task category, and the task category represents a category of task processing items executed on the target object; the processing result obtaining module 4200 is configured to execute the task processing item on the target object according to the target identifier and the task category to obtain a processing result; the storage module 4300 is configured to store the processing result according to a preset storage rule.
< apparatus embodiment >
In the present embodiment, an electronic device is also provided, as shown in fig. 5, the electronic device 5000 may further include a processor 5200 and a memory 5100, the memory 5100 being configured to store executable instructions; the processor 5200 is configured to operate the electronic device to perform a method according to any embodiment of the present disclosure, as controlled by the instructions.
The electronic device 5000 may be a server, or may be other types of devices, such as a terminal device, and is not limited herein.
< media examples >
Corresponding to the above method embodiments, in this embodiment, a computer-readable storage medium is further provided, where a computer program that can be read and executed by a computer is stored, and when the computer program is read and executed by the computer, the computer program is configured to perform the method according to any of the above embodiments of the present disclosure.
The flowchart and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of instructions, which comprises one or more executable instructions for implementing the specified logical function(s). In some alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams and/or flowchart illustration, and combinations of blocks in the block diagrams and/or flowchart illustration, can be implemented by special purpose hardware-based systems which perform the specified functions or acts, or combinations of special purpose hardware and computer instructions. It is well known to those skilled in the art that implementation by hardware, by software, and by a combination of software and hardware are equivalent.
Having described embodiments of the present disclosure, the foregoing description is intended to be exemplary, not exhaustive, and not limited to the disclosed embodiments. Many modifications and variations will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is chosen in order to best explain the principles of the embodiments, the practical application, or improvements made to the technology in the marketplace, or to enable others of ordinary skill in the art to understand the embodiments disclosed herein. The scope of the present disclosure is defined by the appended claims.
Claims (15)
1. A data processing method, comprising:
acquiring aggregate messages to be processed from a cache queue, wherein the aggregate messages comprise target identifications and task categories of target objects, and the task categories represent categories of task processing items executed on the target objects;
executing the task processing items on the target object according to the target identification and the task category to obtain a processing result; and the number of the first and second groups,
and storing the processing result according to a preset storage rule.
2. The method of claim 1, wherein the method maintains the buffer queue by:
receiving an object publishing request, wherein the object publishing request comprises a first object to be published and a first topic associated with the first object;
responding to the object publishing request, publishing the first object, and obtaining the identifier of the first object; and the number of the first and second groups,
obtaining an identification of the first topic;
and generating an aggregation message with a task category as a first category according to the identification of the first object and the identification of the first topic, and writing the aggregation message into the cache queue, wherein the first category represents that topic aggregation items are executed on the first object.
3. The method of claim 1, wherein the method maintains the buffer queue by:
receiving an object heat update request, wherein the object heat update request comprises an identifier of a second object to be updated with a heat value;
and generating an aggregation message with the task class being a second class according to the identification of the second object, and writing the aggregation message into the cache queue, wherein the second class represents that the hot update item is executed on the second object.
4. The method of claim 1, wherein the task category comprises a first category representing performing a topic aggregation item for the target object, wherein the aggregated message further comprises a target topic identification, wherein the target topic identification is an identification of a target topic associated with the target object;
the executing the task processing item to the target object according to the target identifier and the task category to obtain a processing result includes:
and under the condition that the task category is the first category, establishing mapping data reflecting the association relation between the target object and the target topic as the processing result according to the target identification and the target topic identification.
5. The method according to claim 4, wherein the storing the processing result according to a preset storage rule comprises:
and writing the mapping data into a first database and a second database, wherein the first database is a non-relational database, the second database is a relational database, and the data of the first database and the data of the second database are both stored in a nonvolatile memory.
6. The method of claim 5, wherein writing the mapping data to the first database comprises:
acquiring a real-time heat value of the target object according to the target identification;
obtaining a topic record to be stored according to the mapping data and the real-time heat value;
writing the topic records to be stored into a target topic data list corresponding to the target topic identifier, wherein the target topic data list is an ordered list in the first database, and the target topic data list takes the heat value of an object as a key value and sequences the stored topic records.
7. The method according to claim 5, wherein the storing the processing result according to a preset storage rule further comprises:
acquiring a real-time heat value of the target object according to the target identification;
judging whether the sequence of the target object in all the objects related to the target topic meets a preset condition or not according to the real-time heat value to obtain a judgment result;
and writing the mapping data into a third database under the condition that the judgment result shows that the sequencing of the target objects meets a preset condition, wherein the third database is a non-relational database for storing data in a volatile memory.
8. The method of claim 1, wherein the task category comprises a second category representing performing a hotness update transaction on the target object;
the executing the task processing item to the target object according to the target identifier and the task category to obtain a processing result includes:
and under the condition that the task type is the second type, acquiring a heat value of the target object in a first time range as the processing result according to the target identification, wherein the first time range comprises a time range taking the current first time as a starting time and taking a second time as an ending time, and the second time is obtained by adding the first time and a preset time length.
9. The method of claim 8, wherein the heat value comprises an like value of the target object;
the obtaining, according to the target identifier, a heat value of the target object in a first time range as the processing result includes:
and counting a new praise value of the target object in the first time range as the processing result according to the target identification.
10. The method according to claim 8, wherein the storing the processing result according to a preset storage rule comprises:
obtaining a second moment according to the first moment and the preset duration;
and at the second moment, updating the heat degree value of the target object stored in a first database and a second database according to the target identification and the heat degree value, wherein the first database is a non-relational database, the second database is a relational database, and the data of the first database and the data of the second database are both stored in a nonvolatile memory.
11. The method of claim 1, wherein obtaining the pending aggregated message from the buffer queue comprises:
and acquiring the aggregated message from the buffer queue according to a preset reading algorithm, wherein the preset reading algorithm comprises at least one item of message processing time corresponding to the aggregated message according to a preset time interval.
12. A method of data acquisition, comprising:
receiving an object acquisition request sent by terminal equipment, wherein the object acquisition request comprises a target identifier of a target object to be acquired, and the target object is associated with a target topic;
in the case that the target object is judged not to be stored in a third database according to the target identification and the topic identification of the target topic, acquiring the target object from a second database according to the target identification and the topic identification, wherein the second database is a non-relational database for storing data in a non-volatile memory, and the third database is a non-relational database for storing data in a volatile memory;
and providing the target object to the terminal equipment.
13. A data processing apparatus, comprising:
the device comprises an aggregation message acquisition module, a cache queue processing module and a processing module, wherein the aggregation message acquisition module is used for acquiring to-be-processed aggregation messages from the cache queue, the aggregation messages comprise target identifications and task categories of target objects, and the task categories represent the categories of task processing items executed on the target objects;
a processing result obtaining module, configured to execute the task processing item on the target object according to the target identifier and the task category, and obtain a processing result; and the number of the first and second groups,
and the storage module is used for storing the processing result according to a preset storage rule.
14. An electronic device, comprising:
a memory for storing executable instructions;
a processor configured to execute the electronic device to perform the method according to the control of the instruction, wherein the method is as claimed in any one of claims 1 to 12.
15. A computer-readable storage medium, in which a computer program is stored which is readable and executable by a computer, and which, when read and executed by the computer, is adapted to perform the method according to any one of claims 1 to 12.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110264709.2A CN112860750B (en) | 2021-03-11 | 2021-03-11 | Data processing method, device, electronic equipment and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110264709.2A CN112860750B (en) | 2021-03-11 | 2021-03-11 | Data processing method, device, electronic equipment and storage medium |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN112860750A true CN112860750A (en) | 2021-05-28 |
| CN112860750B CN112860750B (en) | 2023-11-17 |
Family
ID=75993982
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202110264709.2A Active CN112860750B (en) | 2021-03-11 | 2021-03-11 | Data processing method, device, electronic equipment and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN112860750B (en) |
Cited By (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113656423A (en) * | 2021-08-18 | 2021-11-16 | 北京百度网讯科技有限公司 | Method and device for updating data, electronic equipment and storage medium |
| CN113918857A (en) * | 2021-10-14 | 2022-01-11 | 国家税务总局山东省税务局 | Three-level cache acceleration method for improving performance of distributed WEB application system |
| CN114064268A (en) * | 2021-10-26 | 2022-02-18 | 上海钧正网络科技有限公司 | A message processing method, device and device |
| CN114138838A (en) * | 2021-12-07 | 2022-03-04 | 胜斗士(上海)科技技术发展有限公司 | Data processing method and device, equipment and medium |
| CN114510509A (en) * | 2022-01-21 | 2022-05-17 | 广州市百果园信息技术有限公司 | Data sampling method, device, electronic equipment and storage medium |
| CN114519056A (en) * | 2022-02-03 | 2022-05-20 | 百果园技术(新加坡)有限公司 | Friend relationship management method and device, electronic equipment and storage medium |
| WO2024179361A1 (en) * | 2023-02-28 | 2024-09-06 | 北京字跳网络技术有限公司 | Message processing method and apparatus, device, and medium |
Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103034722A (en) * | 2012-12-13 | 2013-04-10 | 合一网络技术(北京)有限公司 | Network video comment gathering device and network video comment gathering method |
| CN104244112A (en) * | 2013-06-14 | 2014-12-24 | 腾讯科技（深圳）有限公司 | Multimedia processing method and device and server |
| CN106033438A (en) * | 2015-03-13 | 2016-10-19 | 北大方正集团有限公司 | Public opinion data storage method and server |
| CN108984675A (en) * | 2018-07-02 | 2018-12-11 | 北京百度网讯科技有限公司 | Data query method and apparatus based on evaluation |
| CN109597818A (en) * | 2018-11-28 | 2019-04-09 | 优刻得科技股份有限公司 | Data-updating method, device, storage medium and equipment |
| CN111259246A (en) * | 2020-01-17 | 2020-06-09 | 北京达佳互联信息技术有限公司 | Information pushing method and device, electronic equipment and storage medium |
| CN112307046A (en) * | 2020-11-26 | 2021-02-02 | 北京金堤征信服务有限公司 | Data acquisition method and device, computer readable storage medium and electronic equipment |
-
2021
- 2021-03-11 CN CN202110264709.2A patent/CN112860750B/en active Active
Patent Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103034722A (en) * | 2012-12-13 | 2013-04-10 | 合一网络技术(北京)有限公司 | Network video comment gathering device and network video comment gathering method |
| CN104244112A (en) * | 2013-06-14 | 2014-12-24 | 腾讯科技（深圳）有限公司 | Multimedia processing method and device and server |
| CN106033438A (en) * | 2015-03-13 | 2016-10-19 | 北大方正集团有限公司 | Public opinion data storage method and server |
| CN108984675A (en) * | 2018-07-02 | 2018-12-11 | 北京百度网讯科技有限公司 | Data query method and apparatus based on evaluation |
| CN109597818A (en) * | 2018-11-28 | 2019-04-09 | 优刻得科技股份有限公司 | Data-updating method, device, storage medium and equipment |
| CN111259246A (en) * | 2020-01-17 | 2020-06-09 | 北京达佳互联信息技术有限公司 | Information pushing method and device, electronic equipment and storage medium |
| CN112307046A (en) * | 2020-11-26 | 2021-02-02 | 北京金堤征信服务有限公司 | Data acquisition method and device, computer readable storage medium and electronic equipment |
Cited By (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113656423A (en) * | 2021-08-18 | 2021-11-16 | 北京百度网讯科技有限公司 | Method and device for updating data, electronic equipment and storage medium |
| CN113918857A (en) * | 2021-10-14 | 2022-01-11 | 国家税务总局山东省税务局 | Three-level cache acceleration method for improving performance of distributed WEB application system |
| CN114064268A (en) * | 2021-10-26 | 2022-02-18 | 上海钧正网络科技有限公司 | A message processing method, device and device |
| CN114064268B (en) * | 2021-10-26 | 2025-02-11 | 杭州哈行网络科技有限公司 | A message processing method, device and equipment |
| CN114138838A (en) * | 2021-12-07 | 2022-03-04 | 胜斗士(上海)科技技术发展有限公司 | Data processing method and device, equipment and medium |
| CN114138838B (en) * | 2021-12-07 | 2025-02-07 | 胜斗士(上海)科技技术发展有限公司 | Data processing method, device, equipment and medium |
| CN114510509A (en) * | 2022-01-21 | 2022-05-17 | 广州市百果园信息技术有限公司 | Data sampling method, device, electronic equipment and storage medium |
| CN114519056A (en) * | 2022-02-03 | 2022-05-20 | 百果园技术(新加坡)有限公司 | Friend relationship management method and device, electronic equipment and storage medium |
| WO2024179361A1 (en) * | 2023-02-28 | 2024-09-06 | 北京字跳网络技术有限公司 | Message processing method and apparatus, device, and medium |
Also Published As
| Publication number | Publication date |
|---|---|
| CN112860750B (en) | 2023-11-17 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN112860750B (en) | Data processing method, device, electronic equipment and storage medium | |
| CN105827706B (en) | Message pushing device and method | |
| US20210397630A1 (en) | Content resonance | |
| US8539161B2 (en) | Pre-fetching content items based on social distance | |
| CN110175306A (en) | A kind of processing method and processing device of advertising information | |
| CN111966887B (en) | Dynamic caching method and device, electronic equipment and storage medium | |
| CN113157777B (en) | Distributed real-time data query method, cluster, system and storage medium | |
| CN113805864A (en) | Project engineering generation method and device, electronic equipment and storage medium | |
| CN114489770B (en) | Grayscale release method, device, computer equipment and storage medium | |
| US20220300534A1 (en) | Data processing method, data querying method, and server device | |
| CN107526828B (en) | Page information recommendation method and device | |
| CN115328406A (en) | Data writing and acquiring method and device, electronic equipment and computer medium | |
| CN116842292A (en) | Dynamic page caching methods, electronic devices, vehicles and storage media | |
| CN113672834B (en) | Data processing method, device, electronic device and computer readable medium | |
| CN113761102A (en) | Data processing method, device, server, system and storage medium | |
| CN107016008B (en) | Information processing method and device and dynamic information display method and device | |
| CN107480269B (en) | Object display method and system, medium and computing equipment | |
| CN114417112A (en) | Data processing method, data processing apparatus, electronic device, and storage medium | |
| CN108363773A (en) | A kind of method and device of registering based on binary operation | |
| CN112473149A (en) | Ranking list processing method | |
| CN112883316A (en) | Data processing method and device, electronic equipment and storage medium | |
| CN109246035A (en) | A kind of method and device of data transfer management | |
| CN113177173B (en) | Data access method, device, equipment and storage medium | |
| CN115757976A (en) | Information pushing method, system, medium and equipment based on product subscription keywords | |
| CN113742010B (en) | Page multilingual display method and device |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |