CN106021335A - A database accessing method and device - Google Patents
A database accessing method and device Download PDFInfo
- Publication number
- CN106021335A CN106021335A CN201610299223.1A CN201610299223A CN106021335A CN 106021335 A CN106021335 A CN 106021335A CN 201610299223 A CN201610299223 A CN 201610299223A CN 106021335 A CN106021335 A CN 106021335A
- Authority
- CN
- China
- Prior art keywords
- request
- log
- database
- data
- write
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
-
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/27—Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
-
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/245—Query processing
- G06F16/2458—Special types of queries, e.g. statistical queries, fuzzy queries or distributed queries
- G06F16/2471—Distributed queries
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- Databases & Information Systems (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Data Mining & Analysis (AREA)
- Computing Systems (AREA)
- Fuzzy Systems (AREA)
- Mathematical Physics (AREA)
- Probability & Statistics with Applications (AREA)
- Software Systems (AREA)
- Computational Linguistics (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
本发明提供了一种访问数据库的方法及装置,该方法包括:接收来自客户端的访问数据库的请求;对所述请求进行类型分析,以确定所述请求是否为数据写入请求;以及当所述请求为数据写入请求时,将所述请求相应的写入操作记录至本地日志,并将所述请求相应的数据写入所述数据库的缓存层。本发明实施例直接将数据写入请求相应的数据写入数据库的缓存层,并向客户端返回响应消息,其能够提高后续对所述数据读取的效率。
The present invention provides a method and device for accessing a database. The method includes: receiving a database access request from a client; performing type analysis on the request to determine whether the request is a data writing request; and when the When the request is a data write request, record the write operation corresponding to the request to the local log, and write the data corresponding to the request into the cache layer of the database. The embodiment of the present invention directly writes the data corresponding to the data writing request into the cache layer of the database, and returns a response message to the client, which can improve the efficiency of subsequent reading of the data.
Description
技术领域technical field
本发明涉及计算机技术领域,特别是一种访问数据库的方法及装置。The invention relates to the field of computer technology, in particular to a method and device for accessing a database.
背景技术Background technique
随着社会的发展,大量的数据需要存储起来,对存储的数据,根据不同用户的需求应用一定的数据访问方法进行处理,数据库就是适应这种需求的一种技术。简单来说,数据库是数据以一种能够持久保存,并且可以被操作的方式来保存的数据集合。With the development of society, a large amount of data needs to be stored, and certain data access methods are used to process the stored data according to the needs of different users. Database is a technology that meets this demand. Simply put, a database is a collection of data stored in a way that can be persisted and manipulated.
在实际应用中,用户可以对数据库中的数据进行读写操作,如增加、删除、修改、查询等操作。相关技术中,以leveldb为代表的LSM Tree数据库为例,当用户对数据库中的热数据进行读取操作时,数据库接收到用户客户端发出的读取操作请求后,会从本地文件中去查找,之后向用户客户端返回响应消息。这样的读取方式,其读性能较低,满足不了需要大量请求热数据的情况。因而,亟待解决这一技术问题。In practical applications, users can read and write data in the database, such as adding, deleting, modifying, querying and other operations. In related technologies, the LSM Tree database represented by leveldb is taken as an example. When a user reads hot data in the database, the database will search for it from the local file after receiving the read operation request from the user client. , and then return a response message to the user client. Such a reading method has low reading performance and cannot satisfy the situation where a large number of hot data requests are required. Therefore, it is urgent to solve this technical problem.
发明内容Contents of the invention
鉴于上述问题,提出了本发明以便提供一种克服上述问题或者至少部分地解决上述问题的访问数据库的方法及相应的装置。In view of the above problems, the present invention is proposed to provide a method for accessing a database and a corresponding device that overcome the above problems or at least partially solve the above problems.
依据本发明的一方面,提供了一种访问数据库的方法,包括:According to one aspect of the present invention, a method for accessing a database is provided, including:
接收来自客户端的访问数据库的请求;Receive a request from the client to access the database;
对所述请求进行类型分析,以确定所述请求是否为数据写入请求;以及performing type analysis on the request to determine whether the request is a data writing request; and
当所述请求为数据写入请求时,将所述请求相应的写入操作记录至本地日志,并将所述请求相应的数据写入所述数据库的缓存层。When the request is a data write request, record the write operation corresponding to the request to a local log, and write the data corresponding to the request into the cache layer of the database.
可选地,所述方法还包括:Optionally, the method also includes:
当所述请求为数据的读取请求时,在所述缓存层中查找所述请求相应的数据,并将查找到的所述数据返回给所述客户端。When the request is a data read request, the data corresponding to the request is searched in the cache layer, and the found data is returned to the client.
可选地,所述将所述请求相应的写入操作记录至本地日志,并将所述请求相应的数据写入所述数据库的缓存层的步骤进一步包括:Optionally, the step of recording the corresponding write operation of the request to a local log, and writing the corresponding data of the request into the cache layer of the database further includes:
调用第一线程,并利用所述第一线程将所述请求相应的写入操作记录至所述本地日志并将所述请求相应的数据写入所述缓存层。Invoking the first thread, and using the first thread to record the write operation corresponding to the request to the local log and write the data corresponding to the request into the cache layer.
可选地,所述方法还包括:Optionally, the method also includes:
调用第二线程并利用所述第二线程扫描所述本地日志以确定所述本地日志中是否存在日志增量,其中所述第二线程与所述第一线程异步;以及invoking a second thread and scanning the local log with the second thread to determine whether a log delta exists in the local log, wherein the second thread is asynchronous to the first thread; and
若发现所述本地日志中存在日志增量,则将所述日志增量相应的数据写入所述数据库中。If it is found that there is a log increment in the local log, write the data corresponding to the log increment into the database.
可选地,所述利用所述第二线程扫描所述本地日志以确定所述本地日志中是否存在日志增量的步骤进一步包括:Optionally, the step of using the second thread to scan the local log to determine whether there is a log increment in the local log further includes:
利用所述第二线程扫描所述本地日志,确定所述本地日志中是否存在未被标记的日志记录;以及Using the second thread to scan the local log to determine whether there are unmarked log records in the local log; and
若是,则确定所述未被标记的日志记录为所述日志增量。If yes, determine that the unmarked log record is the log increment.
可选地,所述将所述日志增量相应的数据写入所述数据库中之后,所述方法还包括:Optionally, after writing the data corresponding to the log increment into the database, the method further includes:
在所述本地日志中对所述日志增量相应的日志记录进行标记。Mark the log record corresponding to the log increment in the local log.
可选地,所述方法还包括:Optionally, the method also includes:
删除所述本地日志中已被标记的日志记录。Delete marked log records in the local log.
可选地,所述方法还包括:Optionally, the method also includes:
在所述数据库出现故障重新启动后,扫描所述本地日志,确定所述本地日志中是否存在未被标记的日志记录;以及After the database fails and restarts, scan the local log to determine whether there are unmarked log records in the local log; and
若是,则将所述未被标记的日志记录相应的数据写入所述数据库,并在所述本地日志中对所述未被标记的日志记录进行标记。If yes, write the corresponding data of the unmarked log record into the database, and mark the unmarked log record in the local log.
可选地,所述方法还包括:Optionally, the method also includes:
当所述请求为数据的读取请求时,若在所述缓存层中未查找到所述请求相应的数据,则将所述请求转发至所述数据库;以及When the request is a read request for data, if the data corresponding to the request is not found in the cache layer, forward the request to the database; and
在所述数据库中查找所述请求相应的数据,将查找到的所述请求相应的数据返回给所述客户端,并在所述缓存层中缓存所述请求相应的数据。Search the database for data corresponding to the request, return the found data corresponding to the request to the client, and cache the data corresponding to the request in the cache layer.
可选地,所述缓存层为分布式的高速缓存系统memcached。Optionally, the cache layer is a distributed cache system memcached.
依据本发明的另一方面,还提供了一种访问数据库的装置,包括:According to another aspect of the present invention, a device for accessing a database is also provided, including:
请求接收模块,适于接收来自客户端的访问数据库的请求;A request receiving module, adapted to receive a request from a client to access a database;
请求分析模块,适于对所述请求进行类型分析,以确定所述请求是否为数据写入请求;以及A request analysis module, adapted to perform type analysis on the request to determine whether the request is a data writing request; and
第一写入模块,适于当所述请求分析模块确定所述请求为数据的写入请求时,将所述请求相应的写入操作记录至本地日志,并将所述请求相应的数据写入所述数据库的缓存层。The first write module is adapted to record the write operation corresponding to the request to a local log and write the data corresponding to the request when the request analyzing module determines that the request is a write request of data The caching layer of the database.
可选地,所述装置还包括:Optionally, the device also includes:
读取模块,适于当所述请求分析模块确定所述请求为数据的读取请求时,在所述缓存层中查找所述请求相应的数据,并将查找到的所述数据返回给所述客户端。The reading module is adapted to, when the request analysis module determines that the request is a read request for data, search the cache layer for data corresponding to the request, and return the found data to the client.
可选地,所述第一写入模块还适于:Optionally, the first writing module is also suitable for:
调用第一线程,并利用所述第一线程将所述请求相应的写入操作记录至所述本地日志并将所述请求相应的数据写入所述缓存层。Invoking the first thread, and using the first thread to record the write operation corresponding to the request to the local log and write the data corresponding to the request into the cache layer.
可选地,所述装置还包括:Optionally, the device also includes:
第二写入模块,适于调用第二线程并利用所述第二线程扫描所述本地日志以确定所述本地日志中是否存在日志增量,其中所述第二线程与所述第一线程异步;以及若发现所述本地日志中存在日志增量,则将所述日志增量相应的数据写入所述数据库中。A second writing module, adapted to call a second thread and utilize the second thread to scan the local log to determine whether there is a log increment in the local log, wherein the second thread is asynchronous to the first thread ; and if it is found that there is a log increment in the local log, writing the data corresponding to the log increment into the database.
可选地,所述第二写入模块还适于:Optionally, the second writing module is also suitable for:
利用所述第二线程扫描所述本地日志,确定所述本地日志中是否存在未被标记的日志记录;以及Using the second thread to scan the local log to determine whether there are unmarked log records in the local log; and
若是,则确定所述未被标记的日志记录为所述日志增量。If yes, determine that the unmarked log record is the log increment.
可选地,所述装置还包括:Optionally, the device also includes:
标记模块,适于在所述第二写入模块将所述日志增量相应的数据写入所述数据库中之后,在所述本地日志中对所述日志增量相应的日志记录进行标记。The marking module is adapted to mark the log record corresponding to the log increment in the local log after the second writing module writes the data corresponding to the log increment into the database.
可选地,所述装置还包括:Optionally, the device also includes:
删除模块,适于删除所述本地日志中已被标记的日志记录。A deletion module, adapted to delete marked log records in the local log.
可选地,所述装置还包括:Optionally, the device also includes:
故障处理模块,适于在所述数据库出现故障重新启动后,扫描所述本地日志,确定所述本地日志中是否存在未被标记的日志记录;以及The failure processing module is adapted to scan the local log to determine whether there are unmarked log records in the local log after the database fails and restarts; and
若是,则将所述未被标记的日志记录相应的数据写入所述数据库,并在所述本地日志中对所述未被标记的日志记录进行标记。If yes, write the corresponding data of the unmarked log record into the database, and mark the unmarked log record in the local log.
可选地,所述读取模块还适于:Optionally, the reading module is also suitable for:
当所述请求分析模块确定所述请求为数据的读取请求时,若在所述缓存层中未查找到所述请求相应的数据,则将所述请求转发至所述数据库;以及When the request analysis module determines that the request is a read request for data, if the data corresponding to the request is not found in the cache layer, the request is forwarded to the database; and
在所述数据库中查找所述请求相应的数据,将查找到的所述请求相应的数据返回给所述客户端,并在所述缓存层中缓存所述请求相应的数据。Search the database for data corresponding to the request, return the found data corresponding to the request to the client, and cache the data corresponding to the request in the cache layer.
可选地,所述缓存层为分布式的高速缓存系统memcached。Optionally, the cache layer is a distributed cache system memcached.
在本发明实施例中,当接收到来自客户端的访问数据库的请求时,对该请求进行类型分析以确定该请求是否为数据写入请求。当确定该请求为数据写入请求时,将该数据写入请求相应的写入操作记录至本地日志,并将该数据写入请求相应的数据写入数据库的缓存层。由此可知,本发明实施例直接将数据写入请求相应的数据写入数据库的缓存层,并向客户端返回响应消息,其能够提升后续对所述数据读取的效率。并且,本发明实施例将数据写入请求相应的写入操作记录至本地日志,从而可以利用本地日志异步地将写入请求相应的数据写入数据库,保证数据库中数据的持久化和完整性。另外,在数据库出现故障重新启动后,也可以利用本地日志对数据库进行数据恢复,实现故障修复的目的。In the embodiment of the present invention, when a request from the client for accessing the database is received, type analysis is performed on the request to determine whether the request is a data writing request. When it is determined that the request is a data write request, the write operation corresponding to the data write request is recorded in the local log, and the data corresponding to the data write request is written into the cache layer of the database. It can be seen that the embodiment of the present invention directly writes the data corresponding to the data write request into the cache layer of the database, and returns a response message to the client, which can improve the efficiency of subsequent reading of the data. Moreover, the embodiment of the present invention records the write operation corresponding to the data write request to the local log, so that the data corresponding to the write request can be written into the database asynchronously by using the local log, ensuring the persistence and integrity of the data in the database. In addition, after the database fails and restarts, the local log can also be used to restore the data of the database to achieve the purpose of fault recovery.
上述说明仅是本发明技术方案的概述,为了能够更清楚了解本发明的技术手段,而可依照说明书的内容予以实施,并且为了让本发明的上述和其它目的、特征和优点能够更明显易懂,以下特举本发明的具体实施方式。The above description is only an overview of the technical solution of the present invention. In order to better understand the technical means of the present invention, it can be implemented according to the contents of the description, and in order to make the above and other purposes, features and advantages of the present invention more obvious and understandable , the specific embodiments of the present invention are enumerated below.
根据下文结合附图对本发明具体实施例的详细描述,本领域技术人员将会更加明了本发明的上述以及其他目的、优点和特征。Those skilled in the art will be more aware of the above and other objects, advantages and features of the present invention according to the following detailed description of specific embodiments of the present invention in conjunction with the accompanying drawings.
附图说明Description of drawings
通过阅读下文优选实施方式的详细描述,各种其他的优点和益处对于本领域普通技术人员将变得清楚明了。附图仅用于示出优选实施方式的目的,而并不认为是对本发明的限制。而且在整个附图中,用相同的参考符号表示相同的部件。在附图中:Various other advantages and benefits will become apparent to those of ordinary skill in the art upon reading the following detailed description of the preferred embodiment. The drawings are only for the purpose of illustrating a preferred embodiment and are not to be considered as limiting the invention. Also throughout the drawings, the same reference numerals are used to designate the same parts. In the attached picture:
图1示出了根据本发明一个实施例的访问数据库的方法的流程示意图;FIG. 1 shows a schematic flow diagram of a method for accessing a database according to an embodiment of the present invention;
图2示出了根据本发明另一个实施例的访问数据库的方法的流程示意图;FIG. 2 shows a schematic flowchart of a method for accessing a database according to another embodiment of the present invention;
图3示出了根据本发明一个实施例的访问数据库的装置的结构示意图;以及FIG. 3 shows a schematic structural diagram of a device for accessing a database according to an embodiment of the present invention; and
图4示出了根据本发明另一个实施例的访问数据库的装置的结构示意图。Fig. 4 shows a schematic structural diagram of an apparatus for accessing a database according to another embodiment of the present invention.
具体实施方式detailed description
下面将参照附图更详细地描述本公开的示例性实施例。虽然附图中显示了本公开的示例性实施例,然而应当理解,可以以各种形式实现本公开而不应被这里阐述的实施例所限制。相反,提供这些实施例是为了能够更透彻地理解本公开,并且能够将本公开的范围完整的传达给本领域的技术人员。Exemplary embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although exemplary embodiments of the present disclosure are shown in the drawings, it should be understood that the present disclosure may be embodied in various forms and should not be limited by the embodiments set forth herein. Rather, these embodiments are provided for more thorough understanding of the present disclosure and to fully convey the scope of the present disclosure to those skilled in the art.
为解决上述技术问题,本发明实施例提供了一种访问数据库的方法。图1示出了根据本发明一个实施例的访问数据库的方法的流程示意图。如图1所示,该方法至少可以包括步骤S102、步骤S104以及步骤S106:In order to solve the above technical problem, an embodiment of the present invention provides a method for accessing a database. Fig. 1 shows a schematic flowchart of a method for accessing a database according to an embodiment of the present invention. As shown in Figure 1, the method may at least include step S102, step S104 and step S106:
步骤S102,接收来自客户端的访问数据库的请求;Step S102, receiving a request from a client to access a database;
步骤S104,对该请求进行类型分析,以确定该请求是否为数据写入请求;Step S104, performing type analysis on the request to determine whether the request is a data writing request;
步骤S106,当该请求为数据写入请求时,将该请求相应的写入操作记录至本地日志,并将该请求相应的数据写入数据库的缓存层。Step S106, when the request is a data write request, record the write operation corresponding to the request to the local log, and write the data corresponding to the request into the cache layer of the database.
在该步骤中,本地日志是指数据库在本地物理机上硬盘存储的日志文件,它用来记录数据库被访问的操作。In this step, the local log refers to the log file stored by the database on the hard disk of the local physical machine, which is used to record the operation of the database being accessed.
在本发明实施例中,当接收到来自客户端的访问数据库的请求时,对该请求进行类型分析以确定该请求是否为数据写入请求。当确定该请求为数据写入请求时,将该数据写入请求相应的写入操作记录至本地日志,并将该数据写入请求相应的数据写入数据库的缓存层。由此可知,本发明实施例直接将数据写入请求相应的数据写入数据库的缓存层,并向客户端返回响应消息,而无需将数据写入数据库,能够提高后续对所述数据读取的效率。并且,本发明实施例将数据写入请求相应的写入操作记录至本地日志,从而可以利用本地日志异步地将写入请求相应的数据写入数据库,保证数据库中数据的持久化和完整性。另外,在数据库出现故障重新启动后,也可以利用本地日志对数据库进行数据恢复,实现故障修复的目的。In the embodiment of the present invention, when a request from the client for accessing the database is received, type analysis is performed on the request to determine whether the request is a data writing request. When it is determined that the request is a data write request, the write operation corresponding to the data write request is recorded in the local log, and the data corresponding to the data write request is written into the cache layer of the database. It can be seen that the embodiment of the present invention directly writes the data corresponding to the data write request into the cache layer of the database, and returns a response message to the client without writing the data into the database, which can improve the subsequent reading of the data. efficiency. Moreover, the embodiment of the present invention records the write operation corresponding to the data write request to the local log, so that the data corresponding to the write request can be written into the database asynchronously by using the local log, ensuring the persistence and integrity of the data in the database. In addition, after the database fails and restarts, the local log can also be used to restore the data of the database to achieve the purpose of fault recovery.
在本发明的可选实施例中,为了提高数据库访问的安全性,在上文步骤S102接收到来自客户端的访问数据库的请求后,可以对该请求的合法性进行验证,在验证通过后进行后续的步骤S104以及步骤S106,若验证不通过,则进行拦截处理,并发出告警提示。这里的合法性是根据预先设置的安全认证策略来度量的,安全认证策略可以包括权限的合法性、请求的有效性或者攻击检查等,本发明不限于此。例如,安全认证策略可以设置为根据哪个IP地址的客户端对数据库的查询是不允许的;除了某个IP地址外的客户端是不能删除、修改数据库中的数据,等等策略。In an optional embodiment of the present invention, in order to improve the security of database access, after receiving the request for accessing the database from the client in step S102 above, the legitimacy of the request can be verified, and subsequent verification can be performed after the verification is passed. In step S104 and step S106, if the verification fails, interception processing is performed and an alarm prompt is issued. The legitimacy here is measured according to the preset security authentication strategy, which may include the legitimacy of the authority, validity of the request, or attack check, etc., and the present invention is not limited thereto. For example, the security authentication policy can be set according to which IP address the client is not allowed to query the database; clients other than a certain IP address cannot delete or modify data in the database, and other policies.
在本发明的可选实施例中,在步骤S104对该请求进行类型分析时,可以从该请求中解析出访问数据库的语句,分析该语句是哪种类型的操作,从而确定该请求是否为数据写入请求。这里的类型如增加、删除、修改、查询等,当语句为增加操作的语句时,则确定该请求为数据写入请求;当语句为删除操作的语句时,则确定该请求为数据写入请求;当语句为修改操作的语句时,则确定该请求为数据写入请求;当语句为查询操作的语句时,则确定该请求为数据读取请求,等等。In an optional embodiment of the present invention, when the type analysis is performed on the request in step S104, the statement for accessing the database can be parsed out from the request, and the type of operation of the statement is analyzed to determine whether the request is a data request. write request. The types here are such as add, delete, modify, query, etc. When the statement is a statement of an increase operation, the request is determined to be a data write request; when the statement is a statement of a delete operation, the request is determined to be a data write request ; When the statement is a statement of a modify operation, it is determined that the request is a data write request; when the statement is a statement of a query operation, it is determined that the request is a data read request, and so on.
在本发明的可选实施例中,上文步骤S106中提及的缓存层可以是在数据库之外新建的缓存层。由于数据库本身不具备缓存层,本发明实施例选择在数据库之外新建缓存层用于临时存储数据。例如,在将写入请求相应的写入操作记录至本地日志后,将写入请求相应的数据写入新建的缓存层,并向客户端返回响应消息,从而提高后续对所述数据读取的效率。这里,可以根据业务需求选取相应存储容量的缓存层,缓存层的类型可以如分布式的高速缓存系统memcached或者基于内容存的redis等。In an optional embodiment of the present invention, the cache layer mentioned in step S106 above may be a newly created cache layer outside the database. Since the database itself does not have a cache layer, the embodiment of the present invention chooses to create a new cache layer outside the database for temporary storage of data. For example, after the write operation corresponding to the write request is recorded in the local log, the data corresponding to the write request is written into the newly created cache layer, and a response message is returned to the client, thereby improving the subsequent reading of the data. efficiency. Here, a cache layer with corresponding storage capacity can be selected according to business requirements. The type of the cache layer can be, for example, the distributed cache system memcached or the content-based redis.
在本发明的可选实施例中,上文步骤S106将该请求相应的写入操作记录至本地日志,并将该请求相应的数据写入数据库的缓存层,本发明实施例提供了一种可选的方案,在该方案中,可以调用第一线程,并利用第一线程将请求相应的写入操作记录至本地日志并将请求相应的数据写入缓存层。也就是说,由第一线程将请求相应的写入操作记录至本地日志,随后将请求相应的数据写入缓存层,之后向客户端返回表示已完成数据库写入的响应消息。In an optional embodiment of the present invention, the above step S106 records the write operation corresponding to the request to the local log, and writes the data corresponding to the request into the cache layer of the database. The embodiment of the present invention provides a The selected solution, in this solution, the first thread can be called, and the first thread can be used to record the corresponding write operation of the request to the local log and write the corresponding data of the request into the cache layer. That is to say, the first thread records the write operation corresponding to the request to the local log, then writes the data corresponding to the request into the cache layer, and then returns a response message indicating that the database write has been completed to the client.
进一步地,为了保证数据库中数据的持久化和完整性,本发明实施例可以利用本地日志异步地将写入请求相应的数据写入数据库,即,调用第二线程并利用第二线程扫描本地日志以确定本地日志中是否存在日志增量,其中第二线程与第一线程异步,若发现本地日志中存在日志增量,则将日志增量相应的数据写入数据库中。这里,第二线程可以实时地对本地日志进行扫描,也可以设定指定的扫描周期(如0.5秒或1秒等)对本地日志进行扫描。Further, in order to ensure the persistence and integrity of the data in the database, the embodiment of the present invention can use the local log to asynchronously write the data corresponding to the write request into the database, that is, call the second thread and use the second thread to scan the local log To determine whether there is a log increment in the local log, wherein the second thread is asynchronous with the first thread, and if it is found that there is a log increment in the local log, then write the corresponding data of the log increment into the database. Here, the second thread may scan the local log in real time, or may scan the local log at a specified scan period (such as 0.5 second or 1 second, etc.).
在本发明的可选实施例中,在确定日志增量时,本发明实施例提供了一种可选的方案,即,利用第二线程扫描本地日志,确定本地日志中是否存在未被标记的日志记录,若存在未被标记的日志记录,则确定未被标记的日志记录为日志增量。进一步地,将日志增量相应的数据写入数据库中之后,在本地日志中对日志增量相应的日志记录进行标记,以表示日志增量相应的数据已写入数据库中,可以避免重复写入的问题。例如,将日志增量相应的数据写入数据库中之后,可以在本地日志中标记日志增量相应的日志记录为“已操作”或“已记录”等。In an optional embodiment of the present invention, when determining the log increment, the embodiment of the present invention provides an optional solution, that is, using the second thread to scan the local log to determine whether there is an unmarked log in the local log Log records, if there are unmarked log records, determine the unmarked log records as log increments. Further, after the data corresponding to the log increment is written into the database, the log record corresponding to the log increment is marked in the local log to indicate that the data corresponding to the log increment has been written into the database, which can avoid repeated writing The problem. For example, after the data corresponding to the log increment is written into the database, the log record corresponding to the log increment can be marked as "operated" or "recorded" in the local log.
在本发明的可选实施例中,为了节省本地存储空间,可以删除本地日志中已被标记的日志记录,例如,对本地日志进行设置,每隔预定时间可以删除已被标记的日志记录。另外,本地日志可以按照大小切分,当某个切分区域中的日志记录已全部被标记,则可以删除该切分区域中的日志记录。In an optional embodiment of the present invention, in order to save local storage space, marked log records in the local log may be deleted, for example, the local log may be set to delete marked log records at predetermined intervals. In addition, the local log can be split according to the size. When all the log records in a split area have been marked, the log records in the split area can be deleted.
在本发明的可选实施例中,如果因为宕机或其他因素导致数据库出现故障,使得新写在缓存层中的数据还未在数据库中持久化,则在下次重启数据库后,可以利用本地日志对数据库进行数据恢复,实现故障修复的目的。具体地,在数据库出现故障重新启动后,启用扫描线程,并利用扫描线程扫描本地日志,确定本地日志中是否存在未被标记的日志记录,若是,则将未被标记的日志记录相应的数据写入数据库,并在本地日志中对未被标记的日志记录进行标记。例如,新写在缓存层中的数据“b=2”未在数据库中持久化(“b=2”相应的写入操作已记录至本地日志),数据库突发故障,在数据库重新启动后,启用扫描线程,并利用扫描线程扫描本地日志,发现本地日志中存在未被标记的日志记录,即“b=2”相应的写入操作,则将“b=2”写入数据库,并在本地日志中对未被标记的日志记录(即“b=2”相应的写入操作)进行标记,如标记为“已操作”或“已记录”等。In an optional embodiment of the present invention, if the database fails due to downtime or other factors, so that the data newly written in the cache layer has not been persisted in the database, after the next restart of the database, the local log can be used to Perform data recovery on the database to achieve the purpose of fault repair. Specifically, after the database fails and restarts, enable the scanning thread, and use the scanning thread to scan the local log to determine whether there are unmarked log records in the local log, and if so, write the corresponding data of the unmarked log record to into the database, and mark unmarked log records in the local log. For example, the data "b=2" newly written in the cache layer is not persisted in the database (the corresponding write operation of "b=2" has been recorded in the local log), the database suddenly fails, and after the database is restarted, Enable the scanning thread, and use the scanning thread to scan the local log, and find that there are unmarked log records in the local log, that is, the corresponding write operation of "b=2", then write "b=2" into the database, and locally Unmarked log records (that is, the write operation corresponding to "b=2") are marked in the log, such as marked as "operated" or "recorded".
进一步地,在数据恢复后数据库正常工作,缓存层中也有新的数据写入(这里,可以采用前文提及的第一线程将请求相应的写入操作记录至本地日志并将请求相应的数据写入缓存层),此时可以继续利用扫描线程扫描本地日志,以确定本地日志中是否存在日志增量,若发现本地日志中存在日志增量,则将日志增量相应的数据写入数据库中。这里的扫描线程与前文提及的第二线程作用相同,扫描线程与第一线程异步。Furthermore, after the data recovery, the database works normally, and new data is written in the cache layer (here, the first thread mentioned above can be used to record the corresponding write operation of the request to the local log and write the corresponding data of the request In this case, you can continue to use the scanning thread to scan the local log to determine whether there is a log increment in the local log. If you find that there is a log increment in the local log, write the corresponding data of the log increment into the database. The scanning thread here has the same function as the second thread mentioned above, and the scanning thread is asynchronous to the first thread.
在本发明的可选实施例中,由于在缓存层对数据进行了缓存,因而当来自客户端的访问数据库的请求为数据的读取请求时,则可以直接在缓存层中查找该读取请求相应的数据,并将查找到的数据返回给客户端,从而可以减少对数据库的读取访问,提高数据读取的效率。In an optional embodiment of the present invention, since the data is cached at the cache layer, when the request from the client for accessing the database is a read request for data, the corresponding data of the read request can be directly searched in the cache layer. , and return the found data to the client, thereby reducing the reading access to the database and improving the efficiency of data reading.
在本发明的可选实施例中,如果数据库故障或其他因素导致缓存层数据部分或全部丢失,使得在缓存层中未查找到读取请求相应的数据,则本发明实施例可以将读取请求转发至数据库,在数据库中查找读取请求相应的数据,将查找到的读取请求相应的数据返回给客户端,并在缓存层中缓存读取请求相应的数据。In an optional embodiment of the present invention, if a database failure or other factors cause some or all of the data in the cache layer to be lost, so that the data corresponding to the read request cannot be found in the cache layer, the embodiment of the present invention can send the read request to Forward to the database, find the data corresponding to the read request in the database, return the found data corresponding to the read request to the client, and cache the data corresponding to the read request in the cache layer.
下面将通过一具体实施例详细介绍本发明的访问数据库的方法的实现过程。在该实施例中,根据业务需求选取相应存储容量的缓存层,为数据库新建缓存层,缓存层的类型可以如分布式的高速缓存系统memcached或者基于内存的redis等。The implementation process of the method for accessing the database of the present invention will be introduced in detail below through a specific embodiment. In this embodiment, a cache layer with a corresponding storage capacity is selected according to business requirements, and a new cache layer is created for the database. The type of the cache layer can be, for example, a distributed cache system memcached or a memory-based redis.
图2示出了根据本发明另一个实施例的访问数据库的方法的流程示意图。如图2所示,该方法至少可以包括步骤S202、步骤S204、步骤S206、步骤S208以及步骤S210。Fig. 2 shows a schematic flowchart of a method for accessing a database according to another embodiment of the present invention. As shown in FIG. 2, the method may at least include step S202, step S204, step S206, step S208 and step S210.
步骤S202,为数据库新建缓存层,接收来自客户端的访问数据库的请求。Step S202, creating a cache layer for the database, and receiving a request from a client to access the database.
步骤S204,对该请求进行类型分析,确定该请求是否为数据写入请求,若是,则继续执行步骤S206;若否,则继续执行步骤S208。Step S204, analyze the type of the request to determine whether the request is a data write request, if yes, proceed to step S206; if not, proceed to step S208.
步骤S206,调用第一线程,并利用第一线程将请求相应的写入操作记录至本地日志并将请求相应的数据写入缓存层。Step S206, call the first thread, and use the first thread to record the write operation corresponding to the request to the local log and write the data corresponding to the request into the cache layer.
在该步骤中,本地日志是指数据库在本地物理机上硬盘存储的日志文件,它用来记录数据库被访问的操作。In this step, the local log refers to the log file stored by the database on the hard disk of the local physical machine, which is used to record the operation of the database being accessed.
步骤S208,当来自客户端的访问数据库的请求为数据的读取请求时,在缓存层中查找该读取请求相应的数据,并将查找到的数据返回给客户端。Step S208, when the request from the client to access the database is a data read request, search the cache layer for data corresponding to the read request, and return the found data to the client.
在该步骤中,如果数据库故障或其他因素导致缓存层数据部分或全部丢失,使得在缓存层中未查找到读取请求相应的数据,则本发明实施例可以将读取请求转发至数据库,在数据库中查找读取请求相应的数据,将查找到的读取请求相应的数据返回给客户端,并在缓存层中缓存读取请求相应的数据。In this step, if the data in the cache layer is partially or completely lost due to database failure or other factors, so that the data corresponding to the read request cannot be found in the cache layer, the embodiment of the present invention may forward the read request to the database, and then Find the data corresponding to the read request in the database, return the found data corresponding to the read request to the client, and cache the data corresponding to the read request in the cache layer.
步骤S210,调用第二线程并利用第二线程扫描本地日志以确定本地日志中是否存在日志增量,其中第二线程与第一线程异步,若发现本地日志中存在日志增量,则将日志增量相应的数据写入数据库中。Step S210, calling the second thread and using the second thread to scan the local log to determine whether there is a log increment in the local log, wherein the second thread is asynchronous with the first thread, and if it is found that there is a log increment in the local log, then the log is incremented. The corresponding amount of data is written into the database.
在确定日志增量时,可以利用第二线程扫描本地日志,确定本地日志中是否存在未被标记的日志记录,若存在未被标记的日志记录,则确定未被标记的日志记录为日志增量。进一步地,将日志增量相应的数据写入数据库中之后,在本地日志中对日志增量相应的日志记录进行标记,以表示日志增量相应的数据已写入数据库中,可以避免重复写入的问题。例如,将日志增量相应的数据写入数据库中之后,可以在本地日志中标记日志增量相应的日志记录为“已操作”或“已记录”等。另外,为了节省本地存储空间,可以删除本地日志中已被标记的日志记录。When determining the log increment, the second thread can be used to scan the local log to determine whether there is an unmarked log record in the local log, and if there is an unmarked log record, then determine that the unmarked log record is the log increment . Further, after the data corresponding to the log increment is written into the database, the log record corresponding to the log increment is marked in the local log to indicate that the data corresponding to the log increment has been written into the database, which can avoid repeated writing The problem. For example, after the data corresponding to the log increment is written into the database, the log record corresponding to the log increment can be marked as "operated" or "recorded" in the local log. In addition, in order to save local storage space, marked log records in the local log can be deleted.
在本发明实施例中,如果因为宕机或其他因素导致数据库出现故障,使得新写在缓存层中的数据还未在数据库中持久化,则在下次重启数据库后,可以利用本地日志对数据库进行数据恢复,实现故障修复的目的。具体地,在数据库出现故障重新启动后,启用扫描线程,并利用扫描线程扫描本地日志,确定本地日志中是否存在未被标记的日志记录,若是,则将未被标记的日志记录相应的数据写入数据库,并在本地日志中对未被标记的日志记录进行标记。In the embodiment of the present invention, if the database fails due to downtime or other factors, so that the data newly written in the cache layer has not been persisted in the database, after the next restart of the database, the local log can be used to update the database. Data recovery, to achieve the purpose of fault repair. Specifically, after the database fails and restarts, enable the scanning thread, and use the scanning thread to scan the local log to determine whether there are unmarked log records in the local log, and if so, write the corresponding data of the unmarked log record to into the database, and mark unmarked log records in the local log.
进一步地,在数据恢复后数据库正常工作,缓存层中也有新的数据写入(这里,可以采用前文提及的第一线程将请求相应的写入操作记录至本地日志并将请求相应的数据写入缓存层),此时可以继续利用扫描线程扫描本地日志,以确定本地日志中是否存在日志增量,若发现本地日志中存在日志增量,则将日志增量相应的数据写入数据库中。这里的扫描线程与前文提及的第二线程作用相同,扫描线程与第一线程异步。Furthermore, after the data recovery, the database works normally, and new data is written in the cache layer (here, the first thread mentioned above can be used to record the corresponding write operation of the request to the local log and write the corresponding data of the request In this case, you can continue to use the scanning thread to scan the local log to determine whether there is a log increment in the local log. If you find that there is a log increment in the local log, write the corresponding data of the log increment into the database. The scanning thread here has the same function as the second thread mentioned above, and the scanning thread is asynchronous to the first thread.
需要说明的是,在实际应用中,上述所有可选实施方式可以采用结合的方式任意组合,形成本发明的可选实施例,在此不再一一赘述。It should be noted that, in practical applications, all the above optional implementation manners may be combined arbitrarily in combination to form optional embodiments of the present invention, which will not be repeated here.
基于上文各个实施例提供的访问数据库的方法,基于同一发明构思,本发明实施例还提供了一种访问数据库的装置。Based on the method for accessing a database provided in the above embodiments and based on the same inventive concept, an embodiment of the present invention also provides a device for accessing a database.
图3示出了根据本发明一个实施例的访问数据库的装置的结构示意图。如图3所示,该装置300至少可以包括请求接收模块310、请求分析模块320以及第一写入模块330。Fig. 3 shows a schematic structural diagram of an apparatus for accessing a database according to an embodiment of the present invention. As shown in FIG. 3 , the apparatus 300 may at least include a request receiving module 310 , a request analyzing module 320 and a first writing module 330 .
现介绍本发明实施例的访问数据库的装置300的各组成或器件的功能以及各部分间的连接关系:Now introduce the functions of each component or device of the apparatus 300 for accessing the database in the embodiment of the present invention and the connection relationship between each part:
请求接收模块310,适于接收来自客户端的访问数据库的请求;A request receiving module 310, adapted to receive a request from a client to access a database;
请求分析模块320,与请求接收模块310相耦合,适于对请求进行类型分析,以确定请求是否为数据写入请求;以及The request analysis module 320, coupled with the request receiving module 310, is adapted to perform type analysis on the request to determine whether the request is a data writing request; and
第一写入模块330,与请求分析模块320相耦合,适于当请求分析模块确定请求为数据的写入请求时,将请求相应的写入操作记录至本地日志,并将请求相应的数据写入数据库的缓存层。The first write module 330, coupled with the request analysis module 320, is adapted to record the corresponding write operation of the request to the local log when the request analysis module determines that the request is a write request of data, and write the corresponding data of the request to into the cache layer of the database.
在本发明一实施例中,如图4所示,图3展示的装置还可以包括:In an embodiment of the present invention, as shown in FIG. 4, the device shown in FIG. 3 may further include:
读取模块340,与请求分析模块320、第一写入模块330相耦合,适于当请求分析模块320确定请求为数据的读取请求时,在缓存层中查找请求相应的数据,并将查找到的数据返回给客户端。The reading module 340, coupled with the request analyzing module 320 and the first writing module 330, is suitable for searching the corresponding data in the cache layer when the request analyzing module 320 determines that the request is a read request of data, and will search The received data is returned to the client.
在本发明一实施例中,第一写入模块330还适于:In an embodiment of the present invention, the first writing module 330 is also suitable for:
调用第一线程,并利用第一线程将请求相应的写入操作记录至本地日志并将请求相应的数据写入缓存层。The first thread is called, and the first thread is used to record the corresponding write operation of the request to the local log and write the corresponding data of the request to the cache layer.
在本发明一实施例中,如图4所示,图3展示的装置还可以包括:In an embodiment of the present invention, as shown in FIG. 4, the device shown in FIG. 3 may further include:
第二写入模块350,与第一写入模块330相耦合,适于调用第二线程并利用第二线程扫描本地日志以确定本地日志中是否存在日志增量,其中第二线程与第一线程异步;以及若发现本地日志中存在日志增量,则将日志增量相应的数据写入数据库中。The second write module 350, coupled with the first write module 330, is adapted to call the second thread and utilize the second thread to scan the local log to determine whether there is a log increment in the local log, wherein the second thread and the first thread Asynchronous; and if it is found that there is a log increment in the local log, write the corresponding data of the log increment into the database.
在本发明一实施例中,第二写入模块350还适于:In an embodiment of the present invention, the second writing module 350 is also suitable for:
利用第二线程扫描本地日志,确定本地日志中是否存在未被标记的日志记录;以及Utilize the second thread to scan the local log to determine whether there are unmarked log records in the local log; and
若是,则确定未被标记的日志记录为日志增量。If so, it is determined that unmarked log records are log increments.
在本发明一实施例中,如图4所示,图3展示的装置还可以包括:In an embodiment of the present invention, as shown in FIG. 4, the device shown in FIG. 3 may further include:
标记模块360,与第二写入模块350相耦合,适于在第二写入模块350将日志增量相应的数据写入数据库中之后,在本地日志中对日志增量相应的日志记录进行标记。The marking module 360, coupled with the second writing module 350, is adapted to mark the log record corresponding to the log increment in the local log after the second writing module 350 writes the data corresponding to the log increment into the database .
在本发明一实施例中,如图4所示,图3展示的装置还可以包括:In an embodiment of the present invention, as shown in FIG. 4, the device shown in FIG. 3 may further include:
删除模块370,与标记模块360相耦合,适于删除本地日志中已被标记的日志记录。The deletion module 370, coupled with the marking module 360, is adapted to delete the marked log records in the local log.
在本发明一实施例中,如图4所示,图3展示的装置还可以包括:In an embodiment of the present invention, as shown in FIG. 4, the device shown in FIG. 3 may further include:
故障处理模块380,与第一写入模块330、标记模块360相耦合,适于在数据库出现故障重新启动后,扫描本地日志,确定本地日志中是否存在未被标记的日志记录;以及The failure processing module 380, coupled with the first writing module 330 and the marking module 360, is adapted to scan the local log after the database fails and restarts, and determines whether there are unmarked log records in the local log; and
若是,则将未被标记的日志记录相应的数据写入数据库,并在本地日志中对未被标记的日志记录进行标记。If so, write the corresponding data of the unmarked log records into the database, and mark the unmarked log records in the local log.
在本发明一实施例中,读取模块340还适于:In an embodiment of the present invention, the reading module 340 is also suitable for:
当请求分析模块确定请求为数据的读取请求时,若在缓存层中未查找到请求相应的数据,则将请求转发至数据库;When the request analysis module determines that the request is a read request for data, if the data corresponding to the request is not found in the cache layer, the request is forwarded to the database;
在数据库中查找请求相应的数据,将查找到的请求相应的数据返回给客户端,并在缓存层中缓存请求相应的数据。Find the data corresponding to the request in the database, return the found data corresponding to the request to the client, and cache the corresponding data in the cache layer.
在本发明一实施例中,缓存层为分布式的高速缓存系统memcached。In an embodiment of the present invention, the cache layer is a distributed cache system memcached.
根据上述任意一个优选实施例或多个优选实施例的组合,本发明实施例能够达到如下有益效果:According to any one of the above preferred embodiments or a combination of multiple preferred embodiments, the embodiments of the present invention can achieve the following beneficial effects:
在本发明实施例中,当接收到来自客户端的访问数据库的请求时,对该请求进行类型分析以确定该请求是否为数据写入请求。当确定该请求为数据写入请求时,将该数据写入请求相应的写入操作记录至本地日志,并将该数据写入请求相应的数据写入数据库的缓存层。由此可知,本发明实施例直接将数据写入请求相应的数据写入数据库的缓存层,并向客户端返回响应消息,其能够提高后续对所述数据读取的效率。并且,本发明实施例将数据写入请求相应的写入操作记录至本地日志,从而可以利用本地日志异步地将写入请求相应的数据写入数据库,保证数据库中数据的持久化和完整性。另外,在数据库出现故障重新启动后,也可以利用本地日志对数据库进行数据恢复,实现故障修复的目的。In the embodiment of the present invention, when a request from the client for accessing the database is received, type analysis is performed on the request to determine whether the request is a data writing request. When it is determined that the request is a data write request, the write operation corresponding to the data write request is recorded in the local log, and the data corresponding to the data write request is written into the cache layer of the database. It can be seen that the embodiment of the present invention directly writes the data corresponding to the data write request into the cache layer of the database, and returns a response message to the client, which can improve the efficiency of subsequent reading of the data. Moreover, the embodiment of the present invention records the write operation corresponding to the data write request to the local log, so that the data corresponding to the write request can be written into the database asynchronously by using the local log, ensuring the persistence and integrity of the data in the database. In addition, after the database fails and restarts, the local log can also be used to restore the data of the database to achieve the purpose of fault recovery.
在此处所提供的说明书中,说明了大量具体细节。然而,能够理解,本发明的实施例可以在没有这些具体细节的情况下实践。在一些实例中,并未详细示出公知的方法、结构和技术,以便不模糊对本说明书的理解。In the description provided herein, numerous specific details are set forth. However, it is understood that embodiments of the invention may be practiced without these specific details. In some instances, well-known methods, structures and techniques have not been shown in detail in order not to obscure the understanding of this description.
类似地,应当理解,为了精简本公开并帮助理解各个发明方面中的一个或多个,在上面对本发明的示例性实施例的描述中,本发明的各个特征有时被一起分组到单个实施例、图、或者对其的描述中。然而,并不应将该公开的方法解释成反映如下意图:即所要求保护的本发明要求比在每个权利要求中所明确记载的特征更多的特征。更确切地说,如下面的权利要求书所反映的那样,发明方面在于少于前面公开的单个实施例的所有特征。因此,遵循具体实施方式的权利要求书由此明确地并入该具体实施方式,其中每个权利要求本身都作为本发明的单独实施例。Similarly, it should be appreciated that in the foregoing description of exemplary embodiments of the invention, in order to streamline this disclosure and to facilitate an understanding of one or more of the various inventive aspects, various features of the invention are sometimes grouped together in a single embodiment, figure, or its description. This method of disclosure, however, is not to be interpreted as reflecting an intention that the claimed invention requires more features than are expressly recited in each claim. Rather, as the following claims reflect, inventive aspects lie in less than all features of a single foregoing disclosed embodiment. Thus, the claims following the Detailed Description are hereby expressly incorporated into this Detailed Description, with each claim standing on its own as a separate embodiment of this invention.
本领域那些技术人员可以理解,可以对实施例中的设备中的模块进行自适应性地改变并且把它们设置在与该实施例不同的一个或多个设备中。可以把实施例中的模块或单元或组件组合成一个模块或单元或组件,以及此外可以把它们分成多个子模块或子单元或子组件。除了这样的特征和/或过程或者单元中的至少一些是相互排斥之外,可以采用任何组合对本说明书(包括伴随的权利要求、摘要和附图)中公开的所有特征以及如此公开的任何方法或者设备的所有过程或单元进行组合。除非另外明确陈述,本说明书(包括伴随的权利要求、摘要和附图)中公开的每个特征可以由提供相同、等同或相似目的的替代特征来代替。Those skilled in the art can understand that the modules in the device in the embodiment can be adaptively changed and arranged in one or more devices different from the embodiment. Modules or units or components in the embodiments may be combined into one module or unit or component, and furthermore may be divided into a plurality of sub-modules or sub-units or sub-assemblies. All features disclosed in this specification (including accompanying claims, abstract and drawings) and any method or method so disclosed may be used in any combination, except that at least some of such features and/or processes or units are mutually exclusive. All processes or units of equipment are combined. Each feature disclosed in this specification (including accompanying claims, abstract and drawings) may be replaced by alternative features serving the same, equivalent or similar purpose, unless expressly stated otherwise.
此外,本领域的技术人员能够理解,尽管在此所述的一些实施例包括其它实施例中所包括的某些特征而不是其它特征,但是不同实施例的特征的组合意味着处于本发明的范围之内并且形成不同的实施例。例如,在权利要求书中,所要求保护的实施例的任意之一都可以以任意的组合方式来使用。Furthermore, those skilled in the art will understand that although some embodiments described herein include some features included in other embodiments but not others, combinations of features from different embodiments are meant to be within the scope of the invention. and form different embodiments. For example, in the claims, any one of the claimed embodiments can be used in any combination.
本发明的各个部件实施例可以以硬件实现,或者以在一个或者多个处理器上运行的软件模块实现,或者以它们的组合实现。本领域的技术人员应当理解,可以在实践中使用微处理器或者数字信号处理器(DSP)来实现根据本发明实施例的访问数据库的装置中的一些或者全部部件的一些或者全部功能。本发明还可以实现为用于执行这里所描述的方法的一部分或者全部的设备或者装置程序(例如,计算机程序和计算机程序产品)。这样的实现本发明的程序可以存储在计算机可读介质上,或者可以具有一个或者多个信号的形式。这样的信号可以从因特网网站上下载得到,或者在载体信号上提供,或者以任何其他形式提供。The various component embodiments of the present invention may be implemented in hardware, or in software modules running on one or more processors, or in a combination thereof. Those skilled in the art should understand that a microprocessor or a digital signal processor (DSP) may be used in practice to implement some or all functions of some or all components in the apparatus for accessing a database according to the embodiment of the present invention. The present invention can also be implemented as an apparatus or an apparatus program (for example, a computer program and a computer program product) for performing a part or all of the methods described herein. Such a program for realizing the present invention may be stored on a computer-readable medium, or may be in the form of one or more signals. Such a signal may be downloaded from an Internet site, or provided on a carrier signal, or provided in any other form.
应该注意的是上述实施例对本发明进行说明而不是对本发明进行限制,并且本领域技术人员在不脱离所附权利要求的范围的情况下可设计出替换实施例。在权利要求中,不应将位于括号之间的任何参考符号构造成对权利要求的限制。单词“包含”不排除存在未列在权利要求中的元件或步骤。位于元件之前的单词“一”或“一个”不排除存在多个这样的元件。本发明可以借助于包括有若干不同元件的硬件以及借助于适当编程的计算机来实现。在列举了若干装置的单元权利要求中,这些装置中的若干个可以是通过同一个硬件项来具体体现。单词第一、第二、以及第三等的使用不表示任何顺序。可将这些单词解释为名称。It should be noted that the above-mentioned embodiments illustrate rather than limit the invention, and that those skilled in the art will be able to design alternative embodiments without departing from the scope of the appended claims. In the claims, any reference signs placed between parentheses shall not be construed as limiting the claim. The word "comprising" does not exclude the presence of elements or steps not listed in a claim. The word "a" or "an" preceding an element does not exclude the presence of a plurality of such elements. The invention can be implemented by means of hardware comprising several distinct elements, and by means of a suitably programmed computer. In a unit claim enumerating several means, several of these means can be embodied by one and the same item of hardware. The use of the words first, second, and third, etc. does not indicate any order. These words can be interpreted as names.
至此,本领域技术人员应认识到,虽然本文已详尽示出和描述了本发明的多个示例性实施例,但是,在不脱离本发明精神和范围的情况下,仍可根据本发明公开的内容直接确定或推导出符合本发明原理的许多其他变型或修改。因此,本发明的范围应被理解和认定为覆盖了所有这些其他变型或修改。So far, those skilled in the art should appreciate that, although a number of exemplary embodiments of the present invention have been shown and described in detail herein, without departing from the spirit and scope of the present invention, the disclosed embodiments of the present invention can still be used. Many other variations or modifications consistent with the principles of the invention are directly identified or derived from the content. Accordingly, the scope of the present invention should be understood and deemed to cover all such other variations or modifications.
本发明实施例的一方面公开了A1、一种访问数据库的方法,包括:One aspect of the embodiments of the present invention discloses A1, a method for accessing a database, including:
接收来自客户端的访问数据库的请求;Receive a request from the client to access the database;
对所述请求进行类型分析,以确定所述请求是否为数据写入请求;以及performing type analysis on the request to determine whether the request is a data writing request; and
当所述请求为数据写入请求时,将所述请求相应的写入操作记录至本地日志,并将所述请求相应的数据写入所述数据库的缓存层。When the request is a data write request, record the write operation corresponding to the request to a local log, and write the data corresponding to the request into the cache layer of the database.
A2、根据A1所述的方法,其中,还包括:A2. The method according to A1, which also includes:
当所述请求为数据的读取请求时,在所述缓存层中查找所述请求相应的数据,并将查找到的所述数据返回给所述客户端。When the request is a data read request, the data corresponding to the request is searched in the cache layer, and the found data is returned to the client.
A3、根据A1或A2所述的方法,其中,所述将所述请求相应的写入操作记录至本地日志,并将所述请求相应的数据写入所述数据库的缓存层的步骤进一步包括:A3. The method according to A1 or A2, wherein the step of recording the corresponding write operation of the request to a local log, and writing the corresponding data of the request into the cache layer of the database further includes:
调用第一线程,并利用所述第一线程将所述请求相应的写入操作记录至所述本地日志并将所述请求相应的数据写入所述缓存层。Invoking the first thread, and using the first thread to record the write operation corresponding to the request to the local log and write the data corresponding to the request into the cache layer.
A4、根据A1-A3中任一项所述的方法,其中,还包括:A4. The method according to any one of A1-A3, further comprising:
调用第二线程并利用所述第二线程扫描所述本地日志以确定所述本地日志中是否存在日志增量,其中所述第二线程与所述第一线程异步;以及invoking a second thread and scanning the local log with the second thread to determine whether a log delta exists in the local log, wherein the second thread is asynchronous to the first thread; and
若发现所述本地日志中存在日志增量,则将所述日志增量相应的数据写入所述数据库中。If it is found that there is a log increment in the local log, write the data corresponding to the log increment into the database.
A5、根据A1-A4中任一项所述的方法,其中,所述利用所述第二线程扫描所述本地日志以确定所述本地日志中是否存在日志增量的步骤进一步包括:A5. The method according to any one of A1-A4, wherein the step of using the second thread to scan the local log to determine whether there is a log increment in the local log further includes:
利用所述第二线程扫描所述本地日志,确定所述本地日志中是否存在未被标记的日志记录;以及Using the second thread to scan the local log to determine whether there are unmarked log records in the local log; and
若是,则确定所述未被标记的日志记录为所述日志增量。If yes, determine that the unmarked log record is the log increment.
A6、根据A1-A5中任一项所述的方法,其中,所述将所述日志增量相应的数据写入所述数据库中之后,所述方法还包括:A6. The method according to any one of A1-A5, wherein, after writing the data corresponding to the log increment into the database, the method further includes:
在所述本地日志中对所述日志增量相应的日志记录进行标记。Mark the log record corresponding to the log increment in the local log.
A7、根据A1-A6中任一项所述的方法,其中,还包括:A7. The method according to any one of A1-A6, further comprising:
删除所述本地日志中已被标记的日志记录。Delete marked log records in the local log.
A8、根据A1-A7中任一项所述的方法,其中,还包括:A8. The method according to any one of A1-A7, further comprising:
在所述数据库出现故障重新启动后,扫描所述本地日志,确定所述本地日志中是否存在未被标记的日志记录;以及After the database fails and restarts, scan the local log to determine whether there are unmarked log records in the local log; and
若是,则将所述未被标记的日志记录相应的数据写入所述数据库,并在所述本地日志中对所述未被标记的日志记录进行标记。If yes, write the corresponding data of the unmarked log record into the database, and mark the unmarked log record in the local log.
A9、根据A1-A8中任一项所述的方法,其中,还包括:A9. The method according to any one of A1-A8, further comprising:
当所述请求为数据的读取请求时,若在所述缓存层中未查找到所述请求相应的数据,则将所述请求转发至所述数据库;以及When the request is a read request for data, if the data corresponding to the request is not found in the cache layer, forward the request to the database; and
在所述数据库中查找所述请求相应的数据,将查找到的所述请求相应的数据返回给所述客户端,并在所述缓存层中缓存所述请求相应的数据。Search the database for data corresponding to the request, return the found data corresponding to the request to the client, and cache the data corresponding to the request in the cache layer.
A10、根据A1-A9中任一项所述的方法,其中,所述缓存层为分布式的高速缓存系统memcached。A10. The method according to any one of A1-A9, wherein the cache layer is a distributed cache system memcached.
本发明实施例的另一方面还公开了B11、一种访问数据库的装置,包括:Another aspect of the embodiments of the present invention also discloses B11, a device for accessing a database, including:
请求接收模块,适于接收来自客户端的访问数据库的请求;A request receiving module, adapted to receive a request from a client to access a database;
请求分析模块,适于对所述请求进行类型分析,以确定所述请求是否为数据写入请求;以及A request analysis module, adapted to perform type analysis on the request to determine whether the request is a data writing request; and
第一写入模块,适于当所述请求分析模块确定所述请求为数据的写入请求时,将所述请求相应的写入操作记录至本地日志,并将所述请求相应的数据写入所述数据库的缓存层。The first write module is adapted to record the write operation corresponding to the request to a local log and write the data corresponding to the request when the request analyzing module determines that the request is a write request of data The caching layer of the database.
B12、根据B11所述的装置,其中,还包括:B12. The device according to B11, further comprising:
读取模块,适于当所述请求分析模块确定所述请求为数据的读取请求时,在所述缓存层中查找所述请求相应的数据,并将查找到的所述数据返回给所述客户端。The reading module is adapted to, when the request analysis module determines that the request is a read request for data, search the cache layer for data corresponding to the request, and return the found data to the client.
B13、根据B11或B12所述的装置,其中,所述第一写入模块还适于:B13. The device according to B11 or B12, wherein the first writing module is further adapted to:
调用第一线程,并利用所述第一线程将所述请求相应的写入操作记录至所述本地日志并将所述请求相应的数据写入所述缓存层。Invoking the first thread, and using the first thread to record the write operation corresponding to the request to the local log and write the data corresponding to the request into the cache layer.
B14、根据B11-B13中任一项所述的装置,其中,还包括:B14. The device according to any one of B11-B13, further comprising:
第二写入模块,适于调用第二线程并利用所述第二线程扫描所述本地日志以确定所述本地日志中是否存在日志增量,其中所述第二线程与所述第一线程异步;以及若发现所述本地日志中存在日志增量,则将所述日志增量相应的数据写入所述数据库中。A second writing module, adapted to call a second thread and utilize the second thread to scan the local log to determine whether there is a log increment in the local log, wherein the second thread is asynchronous to the first thread ; and if it is found that there is a log increment in the local log, writing the data corresponding to the log increment into the database.
B15、根据B11-B14中任一项所述的装置,其中,所述第二写入模块还适于:B15. The device according to any one of B11-B14, wherein the second writing module is further adapted to:
利用所述第二线程扫描所述本地日志,确定所述本地日志中是否存在未被标记的日志记录;以及Using the second thread to scan the local log to determine whether there are unmarked log records in the local log; and
若是,则确定所述未被标记的日志记录为所述日志增量。If yes, determine that the unmarked log record is the log increment.
B16、根据B11-B15中任一项所述的装置,其中,还包括:B16. The device according to any one of B11-B15, further comprising:
标记模块,适于在所述第二写入模块将所述日志增量相应的数据写入所述数据库中之后,在所述本地日志中对所述日志增量相应的日志记录进行标记。The marking module is adapted to mark the log record corresponding to the log increment in the local log after the second writing module writes the data corresponding to the log increment into the database.
B17、根据B11-B16中任一项所述的装置,其中,还包括:B17. The device according to any one of B11-B16, further comprising:
删除模块,适于删除所述本地日志中已被标记的日志记录。A deletion module, adapted to delete marked log records in the local log.
B18、根据B11-B17中任一项所述的装置,其中,还包括:B18. The device according to any one of B11-B17, further comprising:
故障处理模块,适于在所述数据库出现故障重新启动后,扫描所述本地日志,确定所述本地日志中是否存在未被标记的日志记录;以及The failure processing module is adapted to scan the local log to determine whether there are unmarked log records in the local log after the database fails and restarts; and
若是,则将所述未被标记的日志记录相应的数据写入所述数据库,并在所述本地日志中对所述未被标记的日志记录进行标记。If yes, write the corresponding data of the unmarked log record into the database, and mark the unmarked log record in the local log.
B19、根据B11-B18中任一项所述的装置,其中,所述读取模块还适于:B19. The device according to any one of B11-B18, wherein the reading module is further adapted to:
当所述请求分析模块确定所述请求为数据的读取请求时,若在所述缓存层中未查找到所述请求相应的数据,则将所述请求转发至所述数据库;以及When the request analysis module determines that the request is a read request for data, if the data corresponding to the request is not found in the cache layer, the request is forwarded to the database; and
在所述数据库中查找所述请求相应的数据,将查找到的所述请求相应的数据返回给所述客户端,并在所述缓存层中缓存所述请求相应的数据。Search the database for data corresponding to the request, return the found data corresponding to the request to the client, and cache the data corresponding to the request in the cache layer.
B20、根据B11-B19中任一项所述的装置,其中,所述缓存层为分布式的高速缓存系统memcached。B20. The device according to any one of B11-B19, wherein the cache layer is a distributed cache system memcached.
Claims (10)
Priority Applications (2)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201610299223.1A CN106021335A (en) | 2016-05-06 | 2016-05-06 | A database accessing method and device |
| PCT/CN2017/080290 WO2017190580A1 (en) | 2016-05-06 | 2017-04-12 | Method and device for accessing database |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201610299223.1A CN106021335A (en) | 2016-05-06 | 2016-05-06 | A database accessing method and device |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN106021335A true CN106021335A (en) | 2016-10-12 |
Family
ID=57082231
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201610299223.1A Pending CN106021335A (en) | 2016-05-06 | 2016-05-06 | A database accessing method and device |
Country Status (2)
| Country | Link |
|---|---|
| CN (1) | CN106021335A (en) |
| WO (1) | WO2017190580A1 (en) |
Cited By (10)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN106713450A (en) * | 2016-12-21 | 2017-05-24 | 广州优视网络科技有限公司 | Downloading acceleration method and apparatus based on read-write separation mode |
| WO2017190580A1 (en) * | 2016-05-06 | 2017-11-09 | 北京奇虎科技有限公司 | Method and device for accessing database |
| CN107341267A (en) * | 2017-07-24 | 2017-11-10 | 郑州云海信息技术有限公司 | A kind of distributed file system access method and platform |
| CN108733704A (en) * | 2017-04-20 | 2018-11-02 | 北京京东尚科信息技术有限公司 | Multiple database data processing method, device and storage medium and electronic equipment |
| CN109213432A (en) * | 2017-07-04 | 2019-01-15 | 华为技术有限公司 | Utilize the log-structured storage device and method thereof for merging tree and writing data into |
| CN110019345A (en) * | 2017-12-28 | 2019-07-16 | 北京京东尚科信息技术有限公司 | Data processing method, device, system and medium |
| CN110019132A (en) * | 2017-12-20 | 2019-07-16 | 北京潘达互娱科技有限公司 | Thumb up date storage method and equipment |
| CN110908838A (en) * | 2019-11-19 | 2020-03-24 | 杭州安恒信息技术股份有限公司 | Data processing method and device, electronic equipment and storage medium |
| CN113886350A (en) * | 2021-09-10 | 2022-01-04 | 阿里云计算有限公司 | Data processing method and system |
| WO2023116827A1 (en) * | 2021-12-23 | 2023-06-29 | 深圳市世强元件网络有限公司 | High-concurrency data storage method and system |
Families Citing this family (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107895039B (en) * | 2017-11-29 | 2020-11-24 | 华中科技大学 | A Construction Method of Campus Network Authentication System Log Database |
| CN108829730A (en) * | 2018-05-11 | 2018-11-16 | 中国平安人寿保险股份有限公司 | The method, apparatus and its readable storage medium storing program for executing that a kind of pair of daily record data is stored |
| CN109800260A (en) * | 2018-12-14 | 2019-05-24 | 深圳壹账通智能科技有限公司 | High concurrent date storage method, device, computer equipment and storage medium |
| CN111008233B (en) * | 2019-11-24 | 2023-02-28 | 浪潮电子信息产业股份有限公司 | A KV database access method, device and equipment |
| CN112231274B (en) * | 2020-10-16 | 2024-04-05 | 京东科技控股股份有限公司 | Log summarizing method and device, electronic equipment and storage medium |
Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN1858738A (en) * | 2006-02-15 | 2006-11-08 | 华为技术有限公司 | Method and device for access data bank |
| CN104317926A (en) * | 2014-10-31 | 2015-01-28 | 北京思特奇信息技术股份有限公司 | Persistent data storage method, persistent data query method, persistent data storage device and persistent data query system |
Family Cites Families (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103810056A (en) * | 2012-11-06 | 2014-05-21 | 南京壹进制信息技术有限公司 | Method for obtaining consistent backup on Linux platform |
| CN104462194A (en) * | 2014-10-28 | 2015-03-25 | 北京国双科技有限公司 | Service data processing method, device and server |
| CN104834724B (en) * | 2015-05-12 | 2018-12-18 | 广东睿江云计算股份有限公司 | A kind of database synchronization method and device |
| CN106021335A (en) * | 2016-05-06 | 2016-10-12 | 北京奇虎科技有限公司 | A database accessing method and device |
-
2016
- 2016-05-06 CN CN201610299223.1A patent/CN106021335A/en active Pending
-
2017
- 2017-04-12 WO PCT/CN2017/080290 patent/WO2017190580A1/en not_active Ceased
Patent Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN1858738A (en) * | 2006-02-15 | 2006-11-08 | 华为技术有限公司 | Method and device for access data bank |
| CN104317926A (en) * | 2014-10-31 | 2015-01-28 | 北京思特奇信息技术股份有限公司 | Persistent data storage method, persistent data query method, persistent data storage device and persistent data query system |
Cited By (13)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2017190580A1 (en) * | 2016-05-06 | 2017-11-09 | 北京奇虎科技有限公司 | Method and device for accessing database |
| CN106713450A (en) * | 2016-12-21 | 2017-05-24 | 广州优视网络科技有限公司 | Downloading acceleration method and apparatus based on read-write separation mode |
| CN108733704B (en) * | 2017-04-20 | 2020-11-20 | 北京京东尚科信息技术有限公司 | Multi-database data processing method and device, storage medium and electronic equipment |
| CN108733704A (en) * | 2017-04-20 | 2018-11-02 | 北京京东尚科信息技术有限公司 | Multiple database data processing method, device and storage medium and electronic equipment |
| CN109213432B (en) * | 2017-07-04 | 2021-10-26 | 华为技术有限公司 | Storage device for writing data using log structured merge tree and method thereof |
| CN109213432A (en) * | 2017-07-04 | 2019-01-15 | 华为技术有限公司 | Utilize the log-structured storage device and method thereof for merging tree and writing data into |
| CN107341267A (en) * | 2017-07-24 | 2017-11-10 | 郑州云海信息技术有限公司 | A kind of distributed file system access method and platform |
| CN110019132A (en) * | 2017-12-20 | 2019-07-16 | 北京潘达互娱科技有限公司 | Thumb up date storage method and equipment |
| CN110019345A (en) * | 2017-12-28 | 2019-07-16 | 北京京东尚科信息技术有限公司 | Data processing method, device, system and medium |
| CN110908838A (en) * | 2019-11-19 | 2020-03-24 | 杭州安恒信息技术股份有限公司 | Data processing method and device, electronic equipment and storage medium |
| CN110908838B (en) * | 2019-11-19 | 2022-09-02 | 杭州安恒信息技术股份有限公司 | Data processing method and device, electronic equipment and storage medium |
| CN113886350A (en) * | 2021-09-10 | 2022-01-04 | 阿里云计算有限公司 | Data processing method and system |
| WO2023116827A1 (en) * | 2021-12-23 | 2023-06-29 | 深圳市世强元件网络有限公司 | High-concurrency data storage method and system |
Also Published As
| Publication number | Publication date |
|---|---|
| WO2017190580A1 (en) | 2017-11-09 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN106021335A (en) | A database accessing method and device | |
| CN110046133B (en) | Metadata management method, device and system for storage file system | |
| JP6062987B2 (en) | Network storage management system | |
| US9875183B2 (en) | Method and apparatus for content derived data placement in memory | |
| CN110888837B (en) | Object storage small file merging method and device | |
| CN107807797B (en) | Data writing method and device and server | |
| WO2020088104A1 (en) | Method and apparatus for performing block chain record-keeping on webpage by means of file acquisition | |
| CN113377289B (en) | Cache management method, system, computing device and readable storage medium | |
| WO2021226822A1 (en) | Log write method and apparatus, electronic device, and storage medium | |
| CN112416710A (en) | User-operated recording method, device, electronic device, and storage medium | |
| CN106227769B (en) | Date storage method and device | |
| US11853284B2 (en) | In-place updates with concurrent reads in a decomposed state | |
| CN110399101B (en) | Disk write operation processing method, device, system and storage medium | |
| CN111382179A (en) | Data processing method and device and electronic equipment | |
| CN106844676A (en) | Data storage method and device | |
| CN107256233A (en) | A kind of date storage method and device | |
| CN109002503A (en) | A kind of metadata read method, device, equipment and readable storage medium storing program for executing | |
| CN116303590A (en) | A cache data access method, device, equipment and storage medium | |
| WO2021169163A1 (en) | File data access method and apparatus, and computer-readable storage medium | |
| CN109992469A (en) | A method and device for merging logs | |
| CN111625506A (en) | Distributed data deleting method, device and equipment based on deleting queue | |
| CN106254092A (en) | A kind of method for early warning, Apparatus and system | |
| CN111026333A (en) | Access request processing method, processing device, electronic equipment and storage medium | |
| CN115174158A (en) | Cloud product configuration checking method based on multi-cloud management platform | |
| CN106612283B (en) | A method and device for identifying the source of a downloaded file |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| C10 | Entry into substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20161012 |