CN115757384A - A big data-based government data processing method - Google Patents

Abstract

The invention relates to data processing, in particular to a big data-based government affair data processing method, which comprises the steps of constructing a government affair data sharing model for multi-place joint office, detecting whether abnormal data exist in government affair data by using the government affair data sharing model, and removing the abnormal data; determining a data optimization model for optimizing the government affair data from which the abnormal data are removed; carrying out optimization training on the data optimization model until an optimal data optimization model is obtained; the optimal data optimal model is used for optimizing the government affair data to obtain high-quality government affair data; the technical scheme provided by the invention can effectively overcome the defects that abnormal data in government affair data cannot be effectively removed and high-quality government affair data cannot be preferably obtained from the government affair data in the prior art.

Description

A big data-based government data processing method

technical field

The invention relates to data processing, in particular to a method for processing government affairs data based on big data.

Background technique

Government affairs refer to the transactional work related to the government. When a certain government affairs consultation is completed, it needs to be released so that the people and government staff can be informed in time. With the development of Internet technology, the release of government affairs has expanded from the initial paper documents to the Internet. After long-term development, government departments have recorded a large amount of government affairs data, which are important basis for daily management of government departments. Government data has a large amount of data, many data types, a wide range of sources, and complex data formats. With the development of big data and the Internet, the government has an increasing demand for mining the value of government data from various departments.

In recent years, with the advancement of technology and the guidance of policies, the reform of digital government proposes to break the "data islands". Improving the efficiency of work requires the data docking of various government departments. The increasingly serious security problems require that the data of various government departments can be effectively shared. , the optimization and upgrading of convenience services requires the cooperation of various government departments, which has prompted governments at all levels to actively promote cross-departmental government data sharing.

However, inter-departmental government data sharing is a very complex project. How to effectively eliminate abnormal data in government data and fully mine data value by optimizing high-quality government data is an urgent problem to be solved in current government data sharing.

Contents of the invention

(1) Solved technical problems

Aiming at the above-mentioned shortcomings existing in the prior art, the present invention provides a method for processing government affairs data based on big data, which can effectively overcome the problems existing in the prior art that the abnormal data in the government affairs data cannot be effectively eliminated and cannot be optimized from the government affairs data. Drawbacks of obtaining high-quality government data.

(2) Technical solution

To achieve the above object, the present invention is achieved through the following technical solutions:

A method for processing government affairs data based on big data, comprising the following steps:

S1. Build a government data sharing model for multi-site joint office, use the government data sharing model to detect whether there is abnormal data in the government data, and eliminate the abnormal data;

S2. Determine a data optimization model for optimizing government affairs data after removing abnormal data;

S3. Perform optimization training on the data optimization model until an optimal data optimization model is obtained;

S4. Using the optimal data optimization model to optimize the government affairs data to obtain high-quality government affairs data.

Preferably, in S1, a government affairs data sharing model for joint offices in multiple places is constructed, including:

Obtain a government node, and use the consensus algorithm in the blockchain to randomly select a node from the government nodes as the aggregation node of the current round;

After data dimensionality reduction, an isolated forest is constructed based on aggregated nodes, and node aggregation is completed after eliminating isolated parameter vectors;

Upload the hash value of the vector after removing the isolated parameter vector to the blockchain, and send the vector source data to the next round of participating nodes until the construction of the government data sharing model is completed.

Preferably, the data dimensionality reduction includes:

For each dimension of the data vector, the aggregation government node obtains the corresponding value of each dimension, and uses the dimensionality reduction graph method to perform data dimensionality reduction.

Preferably, said constructing an isolated forest based on aggregation nodes includes:

Aggregate the government nodes based on the aggregation nodes, construct an isolated forest with k isolated trees according to the data set after dimensionality reduction, and obtain the isolated parameter vector.

Preferably, the elimination of isolated parameter vectors includes:

For each dimension of the data vector of the government node, calculate the box plot function;

If all the function values in a dimension are within the set range, or more than half of the function values are outside the set range, then the data vector in this dimension is eliminated.

Preferably, in S2, a data optimization model for optimizing government affairs data after removing abnormal data is determined, including:

Select the model file containing the default network structure and hyperparameters according to the government affairs data, and determine the algorithm file of the iterative algorithm according to the preset loss expectation value.

Preferably, in S3, the data optimization model is optimized and trained until the optimal data optimization model is obtained, including:

Use the tuner to train the data optimization model to obtain the target data optimization model;

Based on the model parameters of the target data optimization model, the evaluator is used to evaluate the target data optimization model, and the model evaluation result is obtained;

Based on the model evaluation results, the optimizer is used to initialize the target data optimization model, and the target data optimization model is cyclically optimized by using the tuner and evaluator until the preset convergence condition is reached, and the optimal data optimization model is obtained.

Preferably, using the tuner to train the data optimization model to obtain the target data optimization model includes:

According to the preset optimization method, use the tuner to train the data optimization model to obtain the target data optimization model;

Among them, the preset optimization methods include reinforcement learning methods, non-derivative optimization methods and heuristic search methods.

Preferably, the model parameters of the optimal model based on target data use an evaluator to evaluate the optimal model of target data to obtain model evaluation results, including:

According to the preset evaluation method, based on the model parameters of the target data optimization model, the evaluator is used to evaluate the target data optimization model to obtain the model evaluation result.

Preferably, the optimizer is used to initialize the target data optimization model based on the model evaluation result, including:

The empirical learning algorithm is used to determine the optimal model parameters corresponding to the model evaluation results, and based on the optimal model parameters, the optimizer is used to initialize the target data optimization model.

(3) Beneficial effects

Compared with the prior art, a method for processing government affairs data based on big data provided by the present invention has the following beneficial effects:

1) Build a government data sharing model for multi-site joint office, use the government data sharing model to detect whether there is abnormal data in the government data, and remove the abnormal data, so that the government data sharing model can be used to effectively remove the abnormal data in the government data, Ensure the subsequent optimization of government affairs data to obtain the accuracy of high-quality government affairs data;

2) Determine the data optimization model used to optimize the government affairs data after removing abnormal data, optimize and train the data optimization model until the optimal data optimization model is obtained, use the optimal data optimization model to optimize the government affairs data, and obtain high-quality Quality government data, so that the optimal data optimization model can be used to optimize high-quality government data from government data, and provide data protection for the full mining of the value of government data.

Description of drawings

In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings that are required in the description of the embodiments or the prior art. Apparently, the drawings in the following description are only some embodiments of the present invention, and those skilled in the art can obtain other drawings according to these drawings without creative efforts.

Fig. 1 is a schematic flow sheet of the present invention;

Fig. 2 is a schematic flow chart of removing abnormal data in government affairs data in the present invention;

Fig. 3 is a schematic flow diagram for preferably obtaining high-quality government affairs data in the present invention.

Detailed ways

In order to make the purpose, technical solutions and advantages of the embodiments of the present invention more clear, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Apparently, the described embodiments are some, but not all, embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by persons of ordinary skill in the art without creative efforts fall within the protection scope of the present invention.

A government data processing method based on big data, as shown in Figure 1 and Figure 2, ① Construct a multi-site joint office data sharing model, use the government data sharing model to detect whether there is abnormal data in the government data, and eliminate abnormal data .

Among them, a government data sharing model for joint offices in multiple places is constructed, including:

Obtain a government node, and use the consensus algorithm in the blockchain to randomly select a node from the government nodes as the aggregation node of the current round;

After data dimensionality reduction, an isolated forest is constructed based on aggregated nodes, and node aggregation is completed after eliminating isolated parameter vectors;

Upload the hash value of the vector after removing the isolated parameter vector to the blockchain, and send the vector source data to the next round of participating nodes until the construction of the government data sharing model is completed.

1) Perform data dimensionality reduction, including:

For each dimension of the data vector, the aggregation government node obtains the corresponding value of each dimension, and uses the dimensionality reduction graph method to perform data dimensionality reduction.

2) Construct an isolated forest based on aggregation nodes, including:

Aggregate the government nodes based on the aggregation nodes, construct an isolated forest with k isolated trees according to the data set after dimensionality reduction, and obtain the isolated parameter vector.

3) Eliminate isolated parameter vectors, including:

For each dimension of the data vector of the government node, calculate the box plot function;

If all the function values in a dimension are within the set range, or more than half of the function values are outside the set range, then the data vector in this dimension is eliminated.

The above technical solution builds a government data sharing model for multi-site joint office, uses the government data sharing model to detect whether there is abnormal data in the government data, and eliminates the abnormal data, so that the government data sharing model can be used to realize the effective analysis of the abnormal data in the government data. Elimination to ensure the subsequent optimization of government affairs data and the accuracy of high-quality government affairs data.

As shown in Figure 1 and Figure 3, ② Determine the data optimization model used to optimize the government data after removing abnormal data, including:

Select the model file containing the default network structure and hyperparameters according to the government affairs data, and determine the algorithm file of the iterative algorithm according to the preset loss expectation value.

③ Optimize and train the data optimization model until the optimal data optimization model is obtained, specifically including:

Use the tuner to train the data optimization model to obtain the target data optimization model;

Based on the model parameters of the target data optimization model, the evaluator is used to evaluate the target data optimization model, and the model evaluation result is obtained;

Based on the model evaluation results, the optimizer is used to initialize the target data optimization model, and the target data optimization model is cyclically optimized by using the tuner and evaluator until the preset convergence condition is reached, and the optimal data optimization model is obtained.

1) Use the tuner to train the data optimization model to obtain the target data optimization model, including:

According to the preset optimization method, use the tuner to train the data optimization model to obtain the target data optimization model;

Among them, the preset optimization methods include reinforcement learning methods, non-derivative optimization methods and heuristic search methods.

2) Based on the model parameters of the target data optimization model, use the evaluator to evaluate the target data optimization model, and obtain the model evaluation results, including:

According to the preset evaluation method, based on the model parameters of the target data optimization model, the evaluator is used to evaluate the target data optimization model to obtain the model evaluation result.

3) Based on the model evaluation results, use the tuner to initialize the target data optimization model, including:

The empirical learning algorithm is used to determine the optimal model parameters corresponding to the model evaluation results, and based on the optimal model parameters, the optimizer is used to initialize the target data optimization model.

④Use the optimal data optimization model to optimize government affairs data to obtain high-quality government affairs data.

The above technical solution determines the data optimization model for optimizing the government affairs data after removing the abnormal data, optimizes and trains the data optimization model until the optimal data optimization model is obtained, and uses the optimal data optimization model to optimize the government affairs data, Obtain high-quality government affairs data, so that the optimal data optimization model can be used to optimize high-quality government affairs data from government affairs data, and provide data guarantee for the full mining of the value of government affairs data.

The above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that: it can still be described in the foregoing embodiments The recorded technical solutions are modified, or some of the technical features are replaced equivalently; and these modifications or replacements will not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims (10)

1. A method for processing government affairs data based on big data, characterized in that: comprising the following steps: S1. Build a government data sharing model for multi-site joint office, use the government data sharing model to detect whether there is abnormal data in the government data, and eliminate the abnormal data; S2. Determine a data optimization model for optimizing government affairs data after removing abnormal data; S3. Perform optimization training on the data optimization model until an optimal data optimization model is obtained; S4. Using the optimal data optimization model to optimize the government affairs data to obtain high-quality government affairs data. 2. The method for processing government affairs data based on big data according to claim 1, characterized in that: in S1, a government affairs data sharing model for joint offices in multiple places is constructed, including: Obtain a government node, and use the consensus algorithm in the blockchain to randomly select a node from the government nodes as the aggregation node of the current round; After data dimensionality reduction, an isolated forest is constructed based on aggregated nodes, and node aggregation is completed after eliminating isolated parameter vectors; Upload the hash value of the vector after removing the isolated parameter vector to the blockchain, and send the vector source data to the next round of participating nodes until the construction of the government data sharing model is completed. 3. The method for processing government affairs data based on big data according to claim 2, wherein said performing data dimension reduction includes: For each dimension of the data vector, the aggregation government node obtains the corresponding value of each dimension, and uses the dimensionality reduction graph method to perform data dimensionality reduction. 4. the government affairs data processing method based on big data according to claim 3, is characterized in that: described construction isolated forest based on aggregation node, comprises: Aggregate the government nodes based on the aggregation nodes, construct an isolated forest with k isolated trees according to the data set after dimensionality reduction, and obtain the isolated parameter vector. 5. the government affairs data processing method based on big data according to claim 4, is characterized in that: described eliminating isolated parameter vector, comprises: For each dimension of the data vector of the government node, calculate the box plot function; If all the function values in a dimension are within the set range, or more than half of the function values are outside the set range, then the data vector in this dimension is eliminated. 6. The method for processing government affairs data based on big data according to claim 1, characterized in that: in S2, it is determined that the data optimization model used to optimize the government affairs data after removing abnormal data includes: Select the model file containing the default network structure and hyperparameters according to the government affairs data, and determine the algorithm file of the iterative algorithm according to the preset loss expectation value. 7. The method for processing government affairs data based on big data according to claim 1, characterized in that: in S3, the data optimization model is optimized and trained until the optimal data optimization model is obtained, including: Use the tuner to train the data optimization model to obtain the target data optimization model; Based on the model parameters of the target data optimization model, the evaluator is used to evaluate the target data optimization model, and the model evaluation result is obtained; Based on the model evaluation results, the optimizer is used to initialize the target data optimization model, and the target data optimization model is cyclically optimized by using the tuner and evaluator until the preset convergence condition is reached, and the optimal data optimization model is obtained. 8. the government affairs data processing method based on big data according to claim 7, is characterized in that: described utilizes tuner to train data optimal model, obtains target data optimal model, comprises: According to the preset optimization method, use the tuner to train the data optimization model to obtain the target data optimization model; Among them, the preset optimization methods include reinforcement learning methods, non-derivative optimization methods and heuristic search methods. 9. The government affairs data processing method based on big data according to claim 7, characterized in that: the model parameters of the preferred model based on target data utilize an evaluator to evaluate the preferred model of target data, and obtain a model evaluation result, including: According to the preset evaluation method, based on the model parameters of the target data optimization model, the evaluator is used to evaluate the target data optimization model to obtain the model evaluation result. 10. The method for processing government affairs data based on big data according to claim 7, characterized in that: said model-based evaluation result uses an optimizer to initialize the target data optimization model, including: The empirical learning algorithm is used to determine the optimal model parameters corresponding to the model evaluation results, and based on the optimal model parameters, the optimizer is used to initialize the target data optimization model.

Images