KR101378348B1 - Basic prototype of hadoop cluster based on private cloud infrastructure - Google Patents

Abstract

The present invention is a private cloud designed hardware in a blade method that can add hardware and improve performance by scale-out method, can be applied to a variety of tasks based on Hadoop, high availability and open in terms of hardware, operations, and applications It is about a basic prototype of a Hadoop cluster based on a private cloud infrastructure that can implement source management.
A basic prototype of a private cloud infrastructure based Hadoop cluster according to the present invention for achieving the above object is a Hadoop cluster including a Hadoop distributed file system and MapReduce, wherein the Hadoop distributed file system is a central processing unit And at least one PC type name node including a mainboard, and at least one PC type data node including a central processing unit and a mainboard, and connected to store data of the name node and the data node. 50, the network 60 for connecting to the Internet 70, the power supply 40 for operating the name node and data node, and the open source of Hadoop are upgraded by the version of the private cloud. Open source manags for open software management to perform functions to support automatic upgrades ement: 91).

Description

BASIC PROTOTYPE OF HADOOP CLUSTER BASED ON PRIVATE CLOUD INFRASTRUCTURE}

The present invention is a private cloud designed hardware in a blade method that can add hardware and improve performance by scale-out method, can be applied to a variety of tasks based on Hadoop, high availability and open in terms of hardware, operations, and applications It is about a basic prototype of a Hadoop cluster based on a private cloud infrastructure that can implement source management.

"Cloud computing can be defined as a model that provides a fast, supplyable, minimal management (network, server, storage, application, service) pool of computing resources anywhere, anytime, into a convenient consumer-centric network." By using cloud computing, users can access services from the Internet without the knowledge or control of the technical infrastructure they support, especially in the form of services that pay for software or other computer resources on demand. The initial cost is low because it is provided.

In addition, the availability of computers is high because virtualization technology and distributed computing technology provide server services to users by tying or dividing server resources. The user can implement a consistent user environment through the abstracted service and keep the user's data securely by storing it on a reliable server.

Hadoop was developed to support Nutch's distributed processing, providing a foundation for building and operating applications that can handle hundreds of gigabytes to terabytes or petabytes of data. The state is a data processing platform. Because Hadoop's data is typically at least several hundred gigabytes in size, data is not stored on a single computer, but divided into blocks and distributed across multiple computers. Therefore, Hadoop includes the Hadoop Distributed File System (HDFS), which allows the processing of incoming data separately.The distributed data stored in MapReduce was developed for parallel processing of large data in a cluster environment. It is processed by the process.

HDFS and MapReduce run on the same physical server. Both HDFS and MapReduce have a master / slave architecture consisting of one master and multiple slaves. In HDFS, the master is called the NameNode, and the slave is called the DataNode. In the case of MapReduce, the master is called the Job Tracker and the slave is called the TaskTracker.

In HDFS, the name node, the master, manages the file's meta-information, and the actual data is distributed and replicated across multiple data nodes. A map reduce program is called a job, and a job usually consists of one or more map tasks and a reduce map task.

The job tracker is a master service of the Hadoop MapReduce framework, which receives a Hadoop job execution request from a user and manages the job until the job is terminated.

5 is a conceptual diagram illustrating the flow of data during job processing in Hadoop MapReduce. An input file stores data for map reduction, which is typically stored in HDFS. Hadoop supports a variety of data formats, as well as textual data.

When a Job is started at the client's request, the InputFormat (101) determines how to divide and read the input file. In other words, inputSplit is returned by dividing the input file with respect to the data of the block, while generating and returning a RecordReader 102 that converts the InputSplit into a mapper-readable (key, value) form. InputSplit is the unit of data processed by a single map job in MapReduce. Hadoop has several types of input formats: TextInputFormat, KeyValueInputFormat, and SequenceInputFormat. The typical input format is TextInputFormat, which divides input files stored in block units based on each line to form InputSplit, which is a logical input unit, and extracts a record of the form (LongWritable, Text) from this InputSplit. Returns.

The returned RecordReader is responsible for reading the key-value pair record from InputSplit and passing it to the mapper during normal Map process. Mapper creates this record as a record of new keys and values through the process defined in Map. OutputFormat (103) is a format for outputting data generated during the MapReduce process to a file to HDFS. The output format is a key and value pair received as a result of MapReduce processing through RecordWriter 104, a subclass. Writing the configured record to HDFS terminates the data processing process.

On the other hand, big data refers to a large amount of data that is difficult to manage with a general database and software, and can be defined as a data set that exceeds data collection, storage, management, and analysis capabilities of existing database processing methods. Big data is characterized by a huge amount (volume) beyond the existing data unit, the speed at which data is generated and flowed very rapidly (Velocity), photographs, movies, etc. It has three attributes including information. Recently, big data is also expressed as 3V + 1C (Volume, Velocity, Variety, Complexity). This information is mainly composed of unstructured data that is not formatted in the form of pictures, videos, and voices, from text-oriented data in the past. This makes it difficult to collect, store, retrieve, analyze, and visualize billions of unstructured data using traditional methods or tools that have processed tens of millions of text-driven structured data. Therefore, the necessity of effectively responding to the coming big data era is emerging through research and development of analysis technology for big data.

Cloud computing technology is one of the many ways to address big data. Cloud computing virtualizes infrastructure to service in the form of infrastructure as a service (IaaS), or builds a platform using IaaS to provide software developers, etc., or to develop software using PaaS. The service is provided in the form of SaaS (Software as a Service) provided to the user. Cloud computing providers that provide services have the advantage of reducing surplus resources, and users who use them have the advantage of using only as many resources as they need or using multiple software as independent hardware.

Data mining can also be defined as the process of exploring and analyzing large amounts of data using automated or semi-automated tools to discover meaningful patterns and rules. In the age of big data, data mining has been utilized in various fields and has been researched and developed in combination with various fields. The major research areas of data mining that are currently being actively researched are as follows.

Business Data mining-Analyzes a large business database and provides an easy-to-use interface for end-users to make informed decisions, even if they do not have knowledge of statistics.

Bio Data mining-The development of new drugs by obtaining knowledge of life phenomena such as life evolution, heredity, adaptation to environment, and learning from vast amounts of biological molecular sequence data generated and stored from numerous molecular biology studies. It can contribute to the development of pharmacology, chemistry and ecology as well as the development of new therapies, the development of prevention and the development of new antibiotics.

Spatial Data Mining-The process of finding interesting information, spatial correlations, and various spatial patterns in a spatial database. Spatial data includes not only general attribute information composed of text but also spatial information composed of various objects such as points, lines, and faces existing in two- and three-dimensional spaces.

3D Visualization-The goal is to improve the overall data mining process through the convergence of visualization and data mining technologies.

Meanwhile, cloud computing may be classified into public cloud computing and private cloud computing according to a deployment method. Public cloud computing is an Internet-based cloud computing service for the public, which is provided in the form of utility computing using an external data center like a portal site. There is no particular restriction on the subject, and the fee can be paid according to the amount used. By using cloud computing services according to the purpose of use, the elasticity and utilization of the service usage can be maximized, and the maximum performance can be achieved with minimum investment. You can get the service in a timely manner and pay only for what you use. However, depending on the payment method, it may be cumbersome to pay the monthly fee, and it may be difficult to provide professional services for each service, thereby increasing the support cost. In addition, there is a problem that there is a lack of control over the use of the service because customers do not know where and how the service is provided. It is available as a service such as Amazon Web Service (AWS), Google Apps, Salesforce, Twitter, etc., and according to Carolyn Purcell & David Floyer, it is appropriate for companies with less than $ 1 billion in revenue.

Private cloud computing is a cloud computing environment centered on an internal cloud computing data center that provides services to internal customers with less management burden on each member's system. Specific mission-oriented application configurations are common, allowing companies to consolidate and manage data and gain control over the entire infrastructure. Having control over the infrastructure increases security and reliability, reduces network bandwidth constraints and enables service level agreements (SLAs). On the other hand, there is a disadvantage in that it is not possible to settle the costs according to usage, and a separate construction cost may occur. There are costs for equipment, hardware, and virtualization technologies, with separate data center deployment costs and high workforce costs expected, with relatively low resiliency. Services are available from large vendors such as IBM, HP, VMware, and EMC, and Carolyn Pur-cell & David Floyer says it's advantageous to build in companies with revenues of more than $ 1 billion.

In this situation, a prototype for building a private cloud infrastructure for big data processing suitable for SMBs is urgently needed. In particular, there are three main reasons for SMEs to adopt cloud computing. The first is to avoid capital expenditures of information security, IT support, software and hardware by outsourcing infrastructure / platforms / services. The second is to employ cloud computing because of the flexibility and scalability of IT resources, and the third because of business continuity and disaster recovery capabilities. Therefore, there is an urgent need for a prototype for building a private cloud infrastructure for big data processing suitable for SMBs.

The present invention has been proposed to solve the above problems, and an object of the present invention is to provide a basic prototype for building a Hadoop cluster based on a private cloud infrastructure as part of a special effort to support SMBs. .

In other words, SMBs' private clouds can be applied to a variety of tasks, providing high availability and scalability in terms of computing and storage storage, and private cloud features include blade server technology, security, high availability, open source management, and scalability. Its purpose is to provide a basic prototype of a Hadoop cluster based on a private cloud infrastructure that can provide such features.

A basic prototype of a private cloud infrastructure based Hadoop cluster according to the present invention for achieving the above object is a Hadoop cluster including a Hadoop distributed file system and MapReduce, wherein the Hadoop distributed file system is a central processing unit And at least one PC type name node including a mainboard, and at least one PC type data node including a central processing unit and a mainboard, and connected to store data of the name node and the data node. 50, the network 60 for connecting to the Internet 70, the power supply 40 for operating the name node and data node, and the open source of Hadoop are upgraded by the version of the private cloud. Open source manags for open software management to perform functions to support automatic upgrades ement: 91).

At this time, the private cloud is configured in a scale-out 100 method of the blade server method, the outer side of the cloud for security in Hadoop is a firewall 80, the inside is composed of abnormal detection and honeypot, high availability The hybrid cloud 200 may be configured by connecting resources through the network 60 in an external public cloud 210 for XA (Extended Availability: 92) and computing side scalability. .

delete

As described above, the prototype for building a private cloud infrastructure for big data processing suitable for small and medium-sized businesses (SMB) includes information security by outsourcing infrastructure / platforms / services, IT support, S / W, and H / W capital expenditure. To provide cloud computing with the flexibility and scalability of IT resources, and the business continuity and disaster recovery capabilities.

In addition, it can be applied to various tasks and provides high availability and scalability in terms of computing and storage storage, and private cloud functions include blade server technology, security, high availability, open source management, and scalability. It is effective to provide SMEs with high efficiency.

1 is a basic prototype of a private cloud infrastructure based Hadoop cluster according to an embodiment of the present invention,
2 is a basic prototype of a private cloud infrastructure based Hadoop cluster according to another embodiment of the present invention.
3 is a basic prototype of a private cloud infrastructure based Hadoop cluster according to another embodiment of the present invention,
4 is a hybrid cloud combined with scale-out of a private cloud according to an embodiment of the present invention;
5 is a conceptual diagram showing the flow of data during job processing in a conventional general Hadoop.

Hereinafter, a basic prototype of a private cloud infrastructure based Hadoop cluster according to the present invention will be described in detail with reference to the accompanying drawings.

1 is a basic prototype of a private cloud infrastructure based Hadoop cluster according to an embodiment of the present invention, Figure 2 is a basic prototype of a private cloud infrastructure based Hadoop cluster according to another embodiment of the present invention, Figure 3 A basic prototype of a private cloud infrastructure based Hadoop cluster according to another embodiment of the present invention, Figure 4 is a hybrid cloud combined with the scale-out of the private cloud according to an embodiment of the present invention.

In the drawings, the same reference numerals are given to the same elements even when they are shown in different drawings. In the drawings, the same reference numerals as used in the accompanying drawings are used to designate the same or similar elements. And detailed description of the configuration will be omitted. Also, directional terms such as "top", "bottom", "front", "back", "front", "forward", "rear", etc. are used in connection with the orientation of the disclosed drawing (s). Since the elements of the embodiments of the present invention can be positioned in various orientations, the directional terminology is used for illustrative purposes, not limitation.

The basic prototype of the preferred private cloud infrastructure based Hadoop cluster according to an embodiment of the present invention is a basic prototype of PC type, as shown in FIG. 1, with the case removed and one name node and at least one Hadoop configured as a data node, storage 50 connected to store data of the name node and data node, power supply for operating the network 60 and the name node and data node to be connected to the Internet 70. It is configured to include a supply device (40).

Here, the name node and the data node are constituted of a PC including the CPU 11, 21, 31 and the main boards 12, 22, 32, as known.

In addition, to develop in a rack form may be configured as shown in any one of Figs.

In other words, as shown in FIG. 2, a firewall 80 for security, high availability, and open source software management (OSM) 91 are added to the rack-type Hadoop.

Private cloud 110 of the SMB (SMB) is applicable to a variety of tasks and must provide high availability and scalability in terms of computing and storage storage as shown in FIG. In other words, private cloud features require blade server technology, security, high availability, open source management, and scalability.

Here, the server virtualization technology can be largely divided into a partitioning method and a blade server method using a virtual machine, and in particular, the private cloud 110 for the SMB is built by the blade server method. By implementing the blade method, it is possible to add hardware and improve performance at a low cost input by scale-out method 100, and when a large amount of computing and storage 50 is required, the public cloud by scalability and elasticity, which is an advantage of cloud computing 210 may be used as shown in FIG. 5, and a hybrid cloud 200 may be configured.

The security strategy of the private cloud 110 is largely divided into inside and outside based on the cloud. The external side of the cloud operates a security strategy by the firewall 80, and the internal security strategy by anomaly detection and honeypot.

XA (Extended Availability: 92) provides high availability, and SMB's private cloud 110 can be applied to various tasks based on Hadoop, and to support this, high availability and open source management functions ( Implement Open Source Managemant. High Availability provides hardware, operational, and application aspects of availability. Hardware availability provides fault detection and reporting of hardware such as SMB private cloud devices, storage devices and network equipment. Operational availability supports the availability of operating systems, storage systems, network drivers, and so on, in the system area of hardware. Application availability also supports continuous operation of applications such as Hadoop.

Open Software Management (OSM) will perform a function to support automatic upgrade by the version of the private cloud for SMB when the open source such as Hadoop constituting the SMB private cloud 110 is upgraded. .

Scalability on the computing side is elastically using available resources due to scalability which is an advantage of cloud computing when the usage of virtualized resources in the private cloud is reached. In order to support elastic usage, the public cloud 210 uses resources connected to the Internet.

In the present invention, in order to design a private cloud for SMB, first, the Hadoop installation was configured as a basic prototype by configuring a PC as a name node 10 and a data node 20 and 30.

Here, the basic prototype of the PC type can be configured as shown in any one of Figures 1 to 4 in order to remove the case and develop in a rack form.

In other words, as shown in FIG. 2, a firewall 80 for security, high availability, and open source software management (OSM) 91 are added to the rack-type Hadoop.

On the other hand, it is more desirable to provide scalability to the basic prototype. As a way to provide such scalability, by providing scalability to public cloud 210 such as Amazon Web Services (AWS) by open source such as boto library 93, It is desirable to construct a basic prototype.

Meanwhile, the hardware specification is shown in Table 1 as an example of implementing a private cloud for small and medium businesses as one name node and three data nodes based on a PC.

division details Quantity CPU Intel i5, 2.8 GHz,
4 Core cpu 4 Memory 4 GB 4 disk 500GB 4 Network 3Com, 8-Port Switching Hub One

On the other hand, the hardware specification is shown in Table 2 when the name prototype and the data node are configured by connecting four small motherboards consisting of an Intel i3 CPU, 4GB of RAM, and 320GB of hard disk as a router.

division details Quantity CPU Intel i3 3.3GHz,
Dual Core cpu 4 Memory 4 GB 4 disk 320 GB 4 Network NetGear, 4-Port Switching Hub One

In addition, WLANs can be added to the base prototype for high availability fault tolerance. It will have the effect of fault tolerance and network speed improvement by wired and wireless network redundancy.

In the configuration of the basic private cloud prototype as described above, performance tests on big data were performed as shown in Table 3 in FIGS. 1 to 4.

Table 3 tests were conducted using 11 GB of US flight flight statistics published by the American Standard Association (ASA), with performance in most cases of less than five to six minutes.

Test data Prototype version Processing time ASA
US Flight Data Example (Fig. 1) 5 minutes 10 seconds Another embodiment (FIG. 2) 5 minutes 42 seconds Another embodiment (Fig. 3) 5 minutes 42 seconds

Meanwhile, the present invention may implement a federation function with the public cloud to provide scalability and elasticity in the basic prototype.

The embodiments of the present invention described above and shown in the drawings should not be construed as limiting the technical idea of the present invention. The scope of protection of the present invention is limited only by the matters described in the claims, and those skilled in the art will be able to modify the technical idea of the present invention in various forms. Accordingly, such improvements and modifications will fall within the scope of the present invention if they are apparent to those skilled in the art.

10: name nodes 11, 21, 31: CPU
20, 30: data node 40: power supply
50: storage 60: network
70: Internet 80: Firewall
91: Open Source Management 92: Extended Availability (XA)
93: boto Library 95: Amazon Web Services (AWS)
100: scale out 110: private clyde
200: hybrid cloud 210: public cloud

Claims (3)

For Hadoop clusters containing Hadoop Distributed File System and MapReduce,
The Hadoop distributed file system includes a PC type name node including a central processing unit and a main board, and at least one PC type data node including a central processing unit and a main board,
A storage 50 connected to store data of the name node and the data node;
A network 60 for connecting with the Internet 70,
A power supply device 40 for operating said name node and data node;
When the open source of Hadoop is upgraded, the private cloud infrastructure is configured to include an open source management (OSM) 91 for open software management to perform a function for supporting an automatic upgrade by a version of the private cloud. Basic prototype of the underlying Hadoop cluster.
The method according to claim 1,
The private cloud is configured in a scale-out 100 method of the blade server method,
The outer side of the cloud for security in Hadoop is the firewall 80, the inside is composed of abnormal detection and honeypot,
Extended Availability (XA) to provide high availability,
Basic prototype of a private cloud infrastructure based Hadoop cluster, characterized in that the hybrid cloud 200 is configured by connecting resources from the external public cloud 210 through the network 60 for scalability on the computing side. . delete

Images