hadoop环境搭建笔记

一、配置Linux

（1）cat /etc/networks

（2）cat /etc/sysconfig/network

（3）vi /etc/udev/rules.d/70-persistent-net.rules

eth1 改为eth0 

（4）vi /etc/sysconfig/network-scripts/ifcfg-eth0

HWADDR改为eth1的地址

HWADDR=00:0c:29:79:e6:ec

DNS1=网关地址

（5）vi /etc/sysconfig/selinux

#SELINUX=disabled

（6）加sudo权限

vi /etc/sudoers

beifeng ALL=(root)NOPASSWD:ALL

（7）关闭防火墙

centos 7：
systemctl stop firewalld.service #停止
systemctl disable firewalld.service #禁用
之前的版本：
service iptables stop #

停止
chkconfig iptables off #禁用

chkconfig iptables off

service iptables status

 

二、安装jdk、maven、hadoop

chown beifeng:beifeng /opt/*

无密钥登陆ssh
ssh-keygen -t rsa
ssh-copy-id hostname02
ssh-copy-id hostname03
ssh-copy-id hostname04 

 

三、配置伪分布式

1. 修改hadoop-evn.sh

export JAVA_HOME=/opt/modules/jdk1.7.0_67

2. 修改yarn-env.sh

export JAVA_HOME=/opt/modules/jdk1.7.0_67

3. 修改mapred-env.sh

export JAVA_HOME=/opt/modules/jdk1.7.0_67

4. 修改core-site.xml

<configuration><property><name>fs.defaultFS</name><value>hdfs://localhost:9000</value></property><property><name>hadoop.tmp.dir</name><value>/opt/modules/hadoop-2.5.0/data/tmp</value></property><property><name>hadoop.http.staticuser.user</name><value>beifeng</value></property>
</configuration>

5. 修改hdfs-site.xml

<configuration><property><name>dfs.replication</name><value>1</value></property><property><name>dfs.namenode.secondary.http-address</name><value>beifeng-hadoop-02:50090</value></property><property><name>dfs.permissions.enabled</name><value>false</value></property>
</configuration>

6. 修改slaves

beifeng-hadoop-02

7. 修改yarn-site.xml

<configuration><!-- Site specific YARN configuration properties --><property><name>yarn.nodemanager.aux-services</name><value>mapreduce_shuffle</value></property><property><name>yarn.resourcemanager.hostname</name><value>beifeng-hadoop-02</value></property><!-- 是否启用日志聚集功能 --><property><name>yarn.log-aggregation-enable</name><value>true</value></property><!-- 日志保留时间(单位为秒) --><property><name>yarn.log-aggregation.retain-seconds</name><value>106800</value></property>
</configuration>

8. 修改mapred-site.xml

<configuration><property><name>mapreduce.framework.name</name><value>yarn</value></property>
</configuration>

9. 启动服务

bin/hdfs namenode -format

sbin/hadoop-daemon.sh start namenode
sbin/hadoop-daemon.sh start datanode
sbin/yarn-daemon.sh start resourcemanager
sbin/yarn-daemon.sh start nodemanager
sbin/mr-jobhistory-daemon.sh start historyserver
sbin/hadoop-daemon.sh start secondarynamenode


sbin/hadoop-daemon.sh stop namenode
sbin/hadoop-daemon.sh stop datanode
sbin/yarn-daemon.sh stop resourcemanager
sbin/yarn-daemon.sh stop nodemanager
sbin/mr-jobhistory-daemon.sh stop historyserver
sbin/hadoop-daemon.sh stop secondarynamenode

10. 服务可视化

hdfs可视化界面： http://beifeng-hadoop-01:50070/dfshealth.html#tab-overview
yarn可视化界面： http://beifeng-hadoop-01:8088/cluster
历史服务器可视化界面：http://beifeng-hadoop-01:19888/
secondarynamenode可视化界面 http://beifeng-hadoop-01:50090/status.html