hadoop HA 之 QJM

前言

本文主要通过对hadoop2.2.0集群配置的过程加以梳理，所有的步骤都是通过自己实际测试。文档的结构也是根据自己的实际情况而定，同时也会加入自己在实际过程遇到的问题。搭建环境过程不重要，重要点在于搭建过程中遇到的问题，解决问题的过程。

可能自己遇到的问题在一些由经验的老者手上都不是问题，但是这些问题着实让自己耽误了很长时间，最后问题解决也是费了太大心血。也通过这篇文档，表现出来，算是总结，为后者提供意见。

Hadoop2.2.0体系结构

要想理解本节内容，首先需要了解hadoop1的体系结构。这里不过多的介绍基于hadoop1的体系架构，早在之前，曾搭建hadoop1.2.1伪分布式集群，详细请看hadoop学习（一）hadoop-1.2.1伪分布式配置及遇到的问题。这里主要介绍hadoop2的体系架构。

hadoop1的核心组成是两部分，即HDFS和MapReduce。在hadoop2中变为HDFS和Yarn。

新的HDFS中的NameNode不再是只有一个了，可以有多个（目前只支持2个）。每一个都有相同的职能。

这两个NameNode的地位如何：一个是active状态的，一个是standby状态的。当集群运行时，只有active状态的NameNode是正常工作的，standby状态的NameNode是处于待命状态的，时刻同步active状态 NameNode的数据。一旦active状态的NameNode不能工作，通过手工或者自动切换，standby状态的NameNode就可以转变为 active状态的，就可以继续工作了。这就是高可靠。

当NameNode发生故障时，他们的数据如何保持一致：在这里，2个NameNode的数据其实是实时共享的。新HDFS采用了一种共享机制，JournalNode集群或者NFS进行共享。NFS是操作系统层面的，JournalNode是hadoop层面的，我们这里使用JournalNode集群进行数据共享。

如何实现NameNode的自动切换：这就需要使用ZooKeeper集群进行选择了。HDFS集群中的两个NameNode都在ZooKeeper中注册，当active状态的NameNode出故障时，ZooKeeper能检测到这种情况，它就会自动把standby状态的NameNode切换为active状态。

HDFS Federation（HDFS联盟）：联盟的出现是有原因的。我们知道 NameNode是核心节点，维护着整个HDFS中的元数据信息，那么其容量是有限的，受制于服务器的内存空间。当NameNode服务器的内存装不下数据后，那么HDFS集群就装不下数据了，寿命也就到头了。因此其扩展性是受限的。HDFS联盟指的是有多个HDFS集群同时工作，那么其容量理论上就不受限了，夸张点说就是无限扩展。你可以理解成，一个总集群中，可以虚拟出两个或两个以上的单独的小集群，各个小集群之间数据是实时共享的。因为hadoop集群中已经不在单独存在namenode和datanode的概念。当一个其中一个小集群出故障，可以启动另一个小集群中的namenode节点，继续工作。因为数据是实时共享，即使namenode或datanode一起死掉，也不会影响整个集群的正常工作。

配置文件如下：

core-site.xml

 1 <?xml version="1.0" encoding="UTF-8"?>
 2 <?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
 3 <!--
 4   Licensed under the Apache License, Version 2.0 (the "License");
 5   you may not use this file except in compliance with the License.
 6   You may obtain a copy of the License at
 7 
 8     http://www.apache.org/licenses/LICENSE-2.0
 9 
10   Unless required by applicable law or agreed to in writing, software
11   distributed under the License is distributed on an "AS IS" BASIS,
12   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
13   See the License for the specific language governing permissions and
14   limitations under the License. See accompanying LICENSE file.
15 -->
16 
17 <!-- Put site-specific property overrides in this file. -->
18 
19 <configuration>
20      <property>  
21            <name>fs.defaultFS</name>  
22           <value>hdfs://cluster1</value>  
23      </property>
24      <property>
25               <name>io.file.buffer.size</name>
26               <value>131072</value>
27          </property>
28          <property>
29              <name>ha.zookeeper.quorum</name>
30              <value>moses.zookeeper0:2181,moses.zookeeper1:2181,moses.zookeeper2:2181,moses.zookeeper3:2181,moses.zookeeper4:2181</value>
31          </property>
32 </configuration>

hdfs-site.xml

  1 <?xml version="1.0" encoding="UTF-8"?>
  2 <?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
  3 <!--
  4   Licensed under the Apache License, Version 2.0 (the "License");
  5   you may not use this file except in compliance with the License.
  6   You may obtain a copy of the License at
  7 
  8     http://www.apache.org/licenses/LICENSE-2.0
  9 
 10   Unless required by applicable law or agreed to in writing, software
 11   distributed under the License is distributed on an "AS IS" BASIS,
 12   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 13   See the License for the specific language governing permissions and
 14   limitations under the License. See accompanying LICENSE file.
 15 -->
 16 
 17 <!-- Put site-specific property overrides in this file. -->
 18 
 19 <configuration>
 20      <property >  
 21          <name>dfs.replication</name>  
 22          <value>3</value>  
 23      </property>
 24      <property>
 25          <name>dfs.permissions.enabled</name>
 26          <value>false</value>
 27      </property>
 28      <property>
 29          <name>dfs.nameservices</name>
 30          <value>cluster1</value>
 31      </property>
 32      <property>
 33          <name>dfs.ha.namenodes.cluster1</name>
 34          <value>n1,n2</value>
 35      </property>
 36      <property>
 37          <name>dfs.namenode.rpc-address.cluster1.n1</name>
 38          <value>moses.namenode:9090</value>
 39      </property>
 40      <property>
 41          <name>dfs.namenode.http-address.cluster1.n1</name>
 42          <value>moses.namenode:50070</value>
 43      </property>
 44      <property>
 45          <name>dfs.namenode.rpc-address.cluster1.n2</name>
 46          <value>moses.datanode3:9090</value>
 47      </property>
 48      <property>
 49          <name>dfs.namenode.http-address.cluster1.n2</name>
 50          <value>moses.datanode3:50070</value>
 51      </property>
 52      <property>
 53          <name>dfs.namenode.servicerpc-address.cluster1.n1</name>
 54          <value>moses.namenode:53310</value>
 55      </property>
 56      <property>
 57          <name>dfs.namenode.servicerpc-address.cluster1.n2</name>
 58          <value>moses.datanode3:53310</value>
 59      </property>
 60      <property>
 61          <name>dfs.ha.automatic-failover.enabled.cluster1</name>
 62          <value>true</value>
 63      </property>
 64      <property>
 65          <name>dfs.namenode.shared.edits.dir</name>
 66          <value>qjournal://moses.namenode:8485;moses.datanode1:8485;moses.datanode2:8485;moses.datanode3:8485;moses.datanode4:8485/cluster1</value>
 67      </property>
 68      <property>
 69          <name>dfs.client.failover.proxy.provider.cluster1</name>
 70          <value>org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider</value>
 71      </property>
 72      <property>
 73          <name>dfs.journalnode.edits.dir</name>
 74          <value>/data/wapage/journal</value>
 75      </property>
 76      <property>
 77          <name>dfs.ha.fencing.methods</name>
 78          <value>sshfence</value>
 79      </property>
 80      <property>
 81          <name>dfs.ha.fencing.ssh.private-key-files</name>
 82          <value>/home/wapage/.ssh/id_rsa</value>
 83      </property>
 84      <property>
 85          <name>dfs.ha.fencing.ssh.connect-timeout</name>
 86          <value>10000</value>
 87      </property>
 88      <property>
 89          <name>dfs.namenode.handler.count</name>
 90          <value>60</value>
 91      </property>
 92 
 93      <property>
 94            <name>dfs.datanode.max.xcievers</name>
 95            <value>4096</value>
 96      </property>
 97      <property>
 98                 <name>dfs.namenode.secondary.http-address</name>
 99                <value>moses.data.namenode:9091</value>
100         </property>
101     <property>
102       <name>hadoop.tmp.dir</name>
103       <value>/data2/wapage/hadooptmp</value>
104       <description>A base for other temporary directories.</description>
105     </property>
106     <property>
107             <name>dfs.namenode.checkpoint.period</name>
108               <value>600</value>
109     </property>
110     <property>
111          <name>dfs.namenode.name.dir</name>
112          <value>/data/wapage/hadoopname,/data1/wapage/hadoopname,/data2/wapage/hadoopname,/data3/wapage/hadoopname,/data4/wapage/hadoopname,/data5/wapage/hadoopname,/data6/wapage/hadoopname,/data7/wapage/hadoopname</value>
113          <description>Determines where on the local filesystem the DFS name node
114                           should store the name table(fsimage).  If this is a comma-delimited list
115                                 of directories then the name table is replicated in all of the
116                                       directories, for redundancy. </description>
117     </property>
118     <property>
119          <name>dfs.datanode.data.dir</name>
120          <value>/data/wapage/hadoopdata,/data1/wapage/hadoopdata,/data2/wapage/hadoopdata,/data3/wapage/hadoopdata,/data4/wapage/hadoopdata,/data5/wapage/hadoopdata,/data6/wapage/hadoopdata,/data7/wapage/hadoopdata</value>
121          <description>Determines where on the local filesystem an DFS data node
122          should store its blocks.  If this is a comma-delimited
123                                                 list of directories, then data will be stored in all named
124                                                   directories, typically on different devices.
125                                                     Directories that do not exist are ignored.
126          </description>
127     </property>
128     <property>
129     　　<name>dfs.balance.bandwidthPerSec</name>  
130     　　<value>10485760</value>  
131     　　<description>  
132     　　　　Specifies the maximum amount of bandwidth that each datanode   
133     　　　　can utilize for the balancing purpose in term of   
134     　　　　the number of bytes per second.   
135     　　</description>
136     </property>
137 </configuration>