WO2020258175A1

WO2020258175A1 - Method for automatically installing distributed big data platform cdh

Info

Publication number: WO2020258175A1
Application number: PCT/CN2019/093339
Authority: WO
Inventors: 刘洋; 沈磊; 李彦生; 郝建维; 张强; 安飞虎; 刘秋辉; 王嬛
Original assignee: 四川中电启明星信息技术有限公司; 国网信息通信产业集团有限公司
Priority date: 2019-06-27
Filing date: 2019-06-27
Publication date: 2020-12-30

Abstract

A method for automatically installing a distributed big data platform CDH, specifically comprising the following steps: S10, environment preparation: specifically, checking the basic environment of each node server, and preparing installation resources; S20, script configuration: completing the configurations of basic information of all node servers and custom information of a user according to configuration file requirements; S30, installation and deployment: configuring a cluster basic environment and a big data operation environment according to the operation process of an execution script, specifically comprising basic service configuration, NTP service building, SSH key-free login, java environment, CDH management console service installation, and service state verification; and S40, result checking: logging in a CDH management console to check the result. The method can effectively implement the automatic installation of the CDH management console, so that the big data cluster building process is more efficient, and the error rate is reduced.

Description

A CDH Method for Automatic Installation of Distributed Big Data Platform

Technical field

The invention relates to the technical field of computer big data, in particular, is a method for automatically installing a distributed big data platform CDH.

Background technique

With the rapid development of the information age, the use of big data has become more and more widespread. However, due to the difference in the installation system environment, many complicated operations have been generated to install a distributed big data platform. The more clusters, the higher the labor cost.

Because the distributed big data platform needs to use horizontally scalable computers, the number of clusters ranges from 3 to hundreds or thousands. The more data, the higher the probability of manual error and the higher the labor cost. It is urgent to solve the time cost and misoperation problems caused by manual operation.

Summary of the invention

The purpose of the present invention is to provide a method for automatically installing a distributed big data platform CDH, which effectively reduces the probability of errors, and reduces labor costs and time costs.

The present invention is realized through the following technical solutions:

A CDH method for automated installation of a distributed big data platform specifically includes the following steps:

S10: Environment preparation; specifically refers to: check the basic environment of each node server and prepare installation resources;

S20: Script configuration; according to the requirements of the configuration file, the basic information of all node servers and user-defined information are configured;

S30: Installation and deployment; configure the cluster basic environment and big data operating environment according to the running process of the execution script, including basic service configuration, building NTP service, SSH key-free login, java environment, installing CDH management console service, and verifying service status ；

S40: Check the result; log in to the CDH management console to check the result.

Further, in order to better implement the present invention, the basic environment of each node server is checked in the step S10, which specifically includes:

Step S11: Determine whether the operating system version of each node server meets the requirements;

If not, you need to reinstall the operating system;

Step S12: Check the system architecture of the operating system of each node server to determine whether the operating system is an x86_64 system architecture;

Step S13: Check the IP network of each node's server operating system to test whether each node is connected;

Step S14: Check the configuration file, installation package and execution script of each node server.

Further, in order to better implement the present invention, the step S20 specifically includes:

Step S21: Check that the IP addresses, SSH ports, passwords of root users, and custom information installed by the user big data platform of all node servers are configured in the deployment file;

If not, you need to modify the place where the prompt is not set reasonably;

Step S22: Check whether the configuration information of all node servers is correct, and whether the custom information configuration is reasonable;

If not, you need to make adjustments to abnormal and unreasonable information.

Further, in order to better implement the present invention, the step S30 specifically includes stand-alone configuration installation and online configuration installation: the stand-alone configuration installation specifically includes the following steps:

Step S31: All nodes set the IP of each server as static IP according to the corresponding relationship of the script configuration information, and restart the NETWORK service after the setting is completed;

Step S32: According to the corresponding relationship of the script configuration information, all the nodes set the machine name of each server in a corresponding unified format, and name them in order;

Step S33: All nodes modify the hosts file according to the corresponding relationship of the configuration file, adding each node IP and corresponding machine name;

Step S34: Confirm that all nodes can communicate with each other, and there is no packet loss phenomenon, to ensure the normal network;

Step S35: Variable setting of all nodes: specifically refers to: modifying the value of swappiness and the value of transparent_hugepage of all nodes; adjusting configuration of all nodes to self-startup modification;

Step S36: Turn off the firewall and selinux of all nodes; cancel the boot-up.

Further, in order to better implement the present invention, the online configuration installation specifically refers to: configuring the cluster operating environment according to the existing process, including NTP service, SSH key-free login, JAVA environment, installation of CDH management console service, execution When finished, verify each service status operation; specifically include the following steps:

Step S311: Configure the operating environment of all servers;

Step S312: all node servers trust each other; determine whether all nodes verify mutual trust successfully, and go to each node through SSH to verify whether the configuration is successful;

If not, check the reasons for the abnormal SSH status of individual servers and troubleshoot permissions;

Step S313: Check whether all nodes are in the Chinese time zone; if not, change to the Chinese time zone;

Step S314: All nodes configure the yum source of the system, with the management node as the HTTP server, and all other nodes are configured; verify that the yum source of all nodes is configured correctly;

Step S315: Install and build NTP service on all nodes, configure the time server of all nodes as the same time server, and verify whether it is successful;

If not, check the cause of the abnormality of the individual server, and check whether the NTP component installation fails or is not started;

Step S316: Install mysql service;

Step S317: Install the basic operation service component;

Step S318: start the service component; specifically: start the CDH management console service according to the configuration management node, and add the server to the self-startup; verify the service status according to the check; verify the startup through log check and port check success.

Further, in order to better implement the present invention, the step S316 specifically includes:

Install a mysql database according to the configuration management node;

Modify the mysql configuration file according to the configuration management node;

Set the mysql service to be self-starting after booting according to the configuration management node;

Initialize mysql data, create access users, and authorize access according to the configuration management node;

Verify the access status of the mysql database.

Further, in order to better implement the present invention, the step S317 specifically includes:

Install java operating environment on all nodes according to the configuration;

Install the mysql driver environment on all nodes according to the described configuration;

Install the CDH management console service according to the configuration management node;

Configure the metadata database used by the CDH management console service according to the described configuration.

Further, in order to better implement the present invention, the step S14 specifically refers to:

Determine whether the configuration file of the management node has modified the IP user, password, and modified machine name according to the current cluster machine situation;

Determine whether the configuration file and the installation source exist in the management node directory.

Compared with the prior art, the present invention has the following advantages and beneficial effects:

(1) The present invention can effectively realize the automatic installation of CDH management console;

(2) The present invention can make the process of building a big data cluster more efficient and reduce the error rate.

Detailed ways

The present invention will be further described in detail below in conjunction with examples, but the embodiments of the present invention are not limited thereto.

Example 1:

The present invention is realized through the following technical solutions. A method for automatically installing a distributed big data platform CDH specifically includes the following steps:

S10: Environment preparation; specifically refers to: inspecting the basic environment of each node server and preparing for installation resources; the node service and basic environment include the system version, system bitness, and network environment of each node server;

The checking of the basic environment of each node server in the step S10 specifically includes:

Step S11: Determine whether the operating system version of each node server meets the requirements; the preferred operating system version is Red Hat Enterprise Linux 6/CentOS 6.x series version;

If not, you need to reinstall the operating system;

Step S12: Check the system architecture of each node's server operating system to determine whether the operating system is an x86_64 system architecture; since the big data platform only supports the x86_64 system architecture, use the uname-r command to view this, and check whether the output contains x86_64.

Step S13: Check the IP network of each node's server operating system to test whether each node is connected; if not, check the server network environment;

Step S14: Check the configuration files, installation packages and execution scripts of each node server; specifically:

The other parts of this embodiment are the same as the above-mentioned embodiments, so they will not be repeated.

Example 2:

This embodiment is further optimized on the basis of the foregoing embodiment, and the step S20 specifically includes:

If not, you need to modify the place where the prompt is not set reasonably;

If not, you need to make adjustments to abnormal and unreasonable information.

Example 3:

This embodiment is further optimized on the basis of the foregoing embodiment. The step S30 specifically includes stand-alone configuration installation and online configuration installation: the stand-alone configuration installation specifically refers to: configuring the basic environment of all servers, which specifically includes the following steps:

Step S31: All nodes set the IP of each server to static IP according to the corresponding relationship of the script configuration information, and restart the NETWORK service after the setting is completed; if not, check the server network according to the abnormal situation of individual nodes and restore the network; if yes, then Go to step S32;

Step S34: Confirm that all nodes can be connected to each other, and there is no packet loss, to ensure that the network is normal; if it cannot be connected, check individual server network conditions and troubleshoot hardware problems; if it can be connected, go to step S35;

Step S35: Variable setting of all nodes: specifically refers to: modify the values of swappiness and transparent_hugepage of all nodes; adjust configuration of all nodes to be modified after booting; specifically refer to:

Modify the value of swappiness for all nodes, set it to change the setting at runtime, and set /proc/sys/vm/swappiness to 0;

All nodes are adjusted and configured to start automatically after booting. Add /etc/sysctl.conf to save this setting after restarting. This setting will not be lost after restarting cat>>/etc/sysctl.conf<<EOF vm.swappiness=0EOF ；

All nodes adjust the configuration, modify the value of transparent_hugepage, the execution takes effect immediately

echo never>/sys/kernel/mm/transparent_hugepage/defrag;

All nodes are adjusted and configured to start automatically after booting, and a startup script is added to make it permanent. echo "echo never>/sys/kernel/mm/transparent_hugepage/defrag">>/etc/rc.local;

Step S36: Turn off the firewall and selinux of all nodes; cancel the boot-up automatically; specifically:

Turn off the firewall of all nodes, service iptables save

service iptables stop;

chkconfig iptables off;

service ip6tables save;

service ip6tables stop;

chkconfig ip6tables off;

Preferably, it also includes:

Close selinux of all nodes,

setenforce 0sed-i.bak's/SELINUX=enforcing/SELINUX=disabled/'/etc/sysconfig/selinux;

sed-i.bak's/SELINUX=enforcing/SELINUX=disabled/'/etc/selinux/config;

All nodes perform reboot to restart;

Example 4:

This embodiment is further optimized on the basis of the above embodiment. The online configuration installation specifically refers to: configuring the cluster operating environment according to the existing process, including NTP service, SSH key-free login, JAVA environment, and installation of CDH management console After the service is executed, verify each service status operation; specifically include the following steps:

Step S311: Configure the operating environment of all servers;

Step S312: All node servers trust each other; specifically, this means: copy the private key of each node to each server, refer to the following command SSH-copy-id-i root@bigdata-a-001;

Each server described here includes this machine, and the corresponding relationship between host name and IP needs to be configured in /etc/hosts when copying the private key.

Judge whether all nodes verify mutual trust successfully, go to each node through SSH, and verify whether the configuration is successful;

If not, check the reason for the abnormal SSH status of individual servers, and troubleshoot permissions; if the configuration is successful, go to step S313;

Step S313: Check whether all nodes are in the Chinese time zone; if not, change to the Chinese time zone; specifically:

Check whether all nodes are in the Chinese time zone, and check the returned result by date, whether it contains CST;

If not, change to China time zone ln-sf/usr/share/zoneinfo/Asia/Shanghai/etc/localtime;

All nodes verify the yum status and check whether the output is normal by using yum search to verify whether the yum source is configured properly;

If not, check whether the configuration repo file in the yum.repos.d folder of the yum source is wrong;

Step S316: Install mysql service; specifically including

Install a mysql database according to the configuration management node;

Modify the mysql configuration file according to the configuration management node; the mysql configuration file includes the mysql port number, setting ignore case, and character set encoding;

According to the configuration management node initializes mysql data, creates access users, and authorizes access; verifies the access status of the mysql database; specifically refers to: the management node initialization database includes the creation of the CDH management console metadata database, and the remote login users defined by the user Create remote access users by name and authorize remote access;

All nodes verify the database connection status;

If not, check the causes of individual server exceptions, check whether the database is started, or the user's remote access authorization fails;

If yes, install java environment and database connection on all nodes, yum-y install oracle-j2sdk1.7mysql-connector-java;

If not, check the reason for the abnormality of the individual server and check whether there is an execution authority problem;

If it is, the management node installs the CDH management console service;

If not, check the reason for the abnormality of the management node server, and check whether there is a problem of missing dependencies;

If yes, the management node configures the metadata database used by the CDH management console service;

If not, check the mysql database of the management node server to check whether the database connection is abnormal;

Step S317: Install the basic operation service component;

The management node server starts the CDH management console service;

If not, check the abnormal log of the management node server to check whether there is an abnormal database connection;

If it is, the log check and port check are passed to verify that the startup is successful.

Example 5:

This embodiment is further optimized on the basis of the above embodiments. At present, most servers are LINUX servers, which support shell scripts. This embodiment provides an automated installation of distributed big data platform CDH based on shell scripts under LINUX servers. Method, "shell script" technology is a procedural script, a computer program and text file, the content is composed of a series of shell commands, and the content is literally translated by UNIX Shell to operate. It is designed as a scripting language, and its operation mode is similar to that of an interpreted language. UNIX SHELL plays the role of a command line interpreter. After reading the shell script, it runs the shell commands in sequence, and then outputs the results. Use shell scripts to perform system management, file operations, etc.

S10, environmental preparation;

It is required that the server has installed the Red Hat Enterprise Linux 6/CentOS6.x operating system, and the number of operating system bits must be x86_64 bits, the system IP network is normal, the firewall has been turned off, and the RPM installation package of the yum source has been uploaded to the management node.

In the specific implementation process, step S10 "Environment Preparation" needs to complete the following workflow, install the x86_64-bit Red Hat Enterprise Linux 6/CentOS6.x operating system, manually execute the command to turn off the firewall, check whether the network is normal, and ensure that all nodes interact with each other It can be accessed through SSH, and the RPM installation package of yum source is uploaded to the designated directory of the management node server.

S20, script configuration;

It is required to complete the modification of the configuration file according to the current situation of all nodes and the remarks of the configuration file.

In the specific execution process, step S20 "configuration script" needs to complete the following workflow, according to the actual situation of all nodes, according to the remarks in the configuration file, fill in the current node IP, root password, port number and big data platform customization The new static IP address and machine name that need to be changed are placed in the configuration script file, and the configuration file and execution script are uploaded to the designated directory of the management node.

S30, installation and deployment;

In the specific execution process, step S30 needs to execute a script on the management node, which will obtain the basic information of the current server from the configuration file, verify the basic environment of the system, and prompt for modification if it encounters an abnormality. After the operation and maintenance personnel are adjusted, The script will continue to execute. It will configure static IP and modify the machine name according to the process. After completion, it will install and configure the cluster operating environment according to the existing process, including NTP service, SSH key-free login, JAVA environment, and installation of CDH management console Services, etc., are executed and the status is verified.

S40, check the result, log in to the CDH management console interface to check the status;

In the embodiment of the present application, in step S40 "result check", it is necessary to log in to the CDH management console to verify its status.

In the specific execution process, step S40 only needs to log in to the CDH management console, check whether the service is started, verify whether the status is normal, and the process of this automated installation of the distributed big data platform CDH is completed.

The above are only preferred embodiments of the present invention and do not limit the present invention in any form. Any simple modification or equivalent change made to the above embodiments based on the technical essence of the present invention shall fall into the scope of the present invention. Within the scope of protection.

Claims

A method for automatically installing a distributed big data platform CDH is characterized in that it specifically includes the following steps:

S10: Environment preparation; specifically refers to: check the basic environment of each node server and prepare installation resources;

S20: Script configuration; according to the requirements of the configuration file, the basic information of all node servers and user-defined information are configured;

S30: Installation and deployment; configure the cluster basic environment and big data operating environment according to the running process of the execution script, including basic service configuration, building NTP service, SSH key-free login, java environment, installing CDH management console service, and verifying service status ；

S40: Check the result; log in to the CDH management console to check the result.
The method for automatically installing a distributed big data platform CDH according to claim 1, characterized in that: the basic environment of each node server is checked in the step S10, which specifically includes:

Step S11: Determine whether the operating system version of each node server meets the requirements;

If not, you need to reinstall the operating system;

Step S12: Check the system architecture of the operating system of each node server to determine whether the operating system is an x86_64 system architecture;

Step S13: Check the IP network of each node's server operating system to test whether each node is connected;

Step S14: Check the configuration file, installation package and execution script of each node server.
The method for automatically installing a distributed big data platform CDH according to claim 2, wherein said step S20 specifically includes:

Step S21: Check that the IP addresses, SSH ports, passwords of root users, and custom information installed by the user big data platform of all node servers are configured in the deployment file;

If not, you need to modify the place where the prompt is not set reasonably;

Step S22: Check whether the configuration information of all node servers is correct, and whether the custom information configuration is reasonable;

If not, you need to make adjustments to abnormal and unreasonable information.
The method for automatically installing a distributed big data platform CDH according to claim 1, characterized in that: the step S30 specifically includes a stand-alone configuration installation and an online configuration installation: the stand-alone configuration installation specifically includes the following steps:

Step S31: All nodes set the IP of each server as static IP according to the corresponding relationship of the script configuration information, and restart the NETWORK service after the setting is completed;

Step S32: According to the corresponding relationship of the script configuration information, all the nodes set the machine name of each server in a corresponding unified format, and name them in order;

Step S33: All nodes modify the hosts file according to the corresponding relationship of the configuration file, adding each node IP and corresponding machine name;

Step S34: Confirm that all nodes can communicate with each other, and there is no packet loss phenomenon, to ensure the normal network;

Step S35: Variable setting of all nodes: specifically refers to: modifying the value of swappiness and the value of transparent_hugepage of all nodes; adjusting configuration of all nodes to self-startup modification;

Step S36: Turn off the firewall and selinux of all nodes; cancel the boot-up.
The CDH method for automated installation of a distributed big data platform according to claim 4, characterized in that: the online configuration installation specifically refers to: configuring a cluster operating environment according to an existing process, including NTP service and SSH key-free login , JAVA environment, install CDH management console service, after execution, verify each service status operation; specifically include the following steps:

Step S311: Configure the operating environment of all servers;

Step S312: all node servers trust each other; determine whether all nodes verify mutual trust successfully, and go to each node through SSH to verify whether the configuration is successful;

If not, check the reasons for the abnormal SSH status of individual servers and troubleshoot permissions;

Step S313: Check whether all nodes are in the Chinese time zone; if not, change to the Chinese time zone;

Step S314: All nodes configure the yum source of the system, with the management node as the HTTP server, and all other nodes are configured; verify that the yum source of all nodes is configured correctly;

Step S315: Install and build NTP service on all nodes, configure the time server of all nodes as the same time server, and verify whether it is successful;

If not, check the cause of the abnormality of the individual server, and check whether the NTP component installation fails or is not started;

Step S316: Install mysql service;

Step S317: Install the basic operation service component;

Step S318: start the service component; specifically: start the CDH management console service according to the configuration management node, and add the server to the self-startup; verify the service status according to the check; verify the startup through log check and port check success.
The method for automatically installing a distributed big data platform CDH according to claim 5, wherein said step S316 specifically includes:

Install a mysql database according to the configuration management node;

Modify the mysql configuration file according to the configuration management node;

Set the mysql service to be self-starting after booting according to the configuration management node;

Initialize mysql data, create access users, and authorize access according to the configuration management node;

Verify the access status of the mysql database.
The method for automatically installing a distributed big data platform CDH according to claim 6, wherein said step S317 specifically includes:

Install java operating environment on all nodes according to the configuration;

Install the mysql driver environment on all nodes according to the described configuration;

Install the CDH management console service according to the configuration management node;

Configure the metadata database used by the CDH management console service according to the described configuration.
The method for automatically installing a distributed big data platform CDH according to claim 7, wherein the step S14 specifically refers to:

Determine whether the configuration file of the management node has modified the IP user, password, and modified machine name according to the current cluster machine situation;

Determine whether the configuration file and the installation source exist in the management node directory.