前置环境:
操作系统: CentOS Linux release 7.7
java JDK : 1.8.0_231
1、准备工作
准备以下安装包:
Cloudera Manager:
cloudera-manager-agent-6.3.1-1466458.el7.x86_64.rpm
cloudera-manager-daemons-6.3.1-1466458.el7.x86_64.rpm
cloudera-manager-server-6.3.1-1466458.el7.x86_64.rpm
CDH Parcel:
CDH-6.3.2-1.cdh6.3.2.p0.1605554-el7.parcel
CDH-6.3.2-1.cdh6.3.2.p0.1605554-el7.parcel.sha1
manifest.json
MySQL:
mysql-community-common-5.7.19-1.el7.x86_64.rpm
mysql-community-libs-5.7.19-1.el7.x86_64.rpm
mysql-community-libs-compat-5.7.19-1.el7.x86_64.rpm
mysql-community-devel-5.7.19-1.el7.x86_64.rpm
mysql-community-client-5.7.19-1.el7.x86_64.rpm
mysql-community-server-5.7.19-1.el7.x86_64.rpm
全部放在/opt/CDH6.3.2 中
2、集群规划
hostname | IP | 角色 | CM组件 | 软件包 |
---|---|---|---|---|
bigdata1 | 172.16.1.62 | master | server,agent | JDK1.8,mysql5.7 |
bigdata2 | 172.16.1.63 | slave | agent | JDK1.8 |
bigdata3 | 172.16.1.64 | slave | agent | JDK1.8 |
bigdata4 | 172.16.1.65 | slave | agent | JDK1.8 |
bigdata5 | 172.16.1.67 | slave | agent | JDK1.8 |
bigdata6 | 172.16.1.68 | slave | agent | JDK1.8 |
3、系统准备
3.1 修改主机名及映射(全部节点)
vi /etc/hostname vi /etc/hosts
3.2 配置免密登录(SSH)(全部节点)
在用户主目录下,进入.ssh文件夹:
cd ~/.ssh
生成密钥:
ssh-keygen -t rsa
发放其他主机:
ssh-copy-id 主机名
3.3 时间同步NTP
所有节点安装ntp
yum -y install ntp
主节点:
vi /etc/ntp.conf server 127.127.1.0 fudge 127.127.1.0 stratum 10 注释掉其他server xxx irurst
其他节点
vi /etc/ntp.conf server 127.127.1.0 fudge 127.127.1.0 stratum 10 server bigdata1 iburst 注释掉其他server xxx irurst
添加server主机ip
vi /etc/sysconfig/ntpd
SYNC_CLOCK=yes
启动服务
systemctl start ntpd.service systemctl enable ntpd.service service ntpd status
其他节点:
crontab -e
写入:
*/10 * * * * /usr/sbin/ntpdate 主节点
3.4 修改系统参数(全部节点)
sysctl vm.swappiness=10 echo 'vm.swappiness=10'>> /etc/sysctl.conf echo never > /sys/kernel/mm/transparent_hugepage/defrag echo never > /sys/kernel/mm/transparent_hugepage/enabled echo 'echo never > /sys/kernel/mm/transparent_hugepage/defrag' >> /etc/rc.local echo 'echo never > /sys/kernel/mm/transparent_hugepage/enabled' >> /etc/rc.local
3.5 安装http(master节点)
yum -y install httpd
启动服务
service httpd start
4、安装软件
4.1 安装依赖(全部节点)
yum -y install chkconfig yum -y install bind-utils yum -y install chkconfig yum -y install cyrus-sasl-gssapi yum -y install cyrus-sasl-plain yum -y install fuse yum -y install fuse-libs yum -y install gcc yum -y install libxslt yum -y install mod_ssl yum -y install MySQL-python yum -y install openssl yum -y install openssl-devel yum -y install perl yum -y install portmap yum -y install postgresql-server yum -y install psmisc yum -y install python-devel yum -y install python-psycopg2 yum -y install python-setuptools yum -y install sed yum -y install sqlite yum -y install swig yum -y install zlib yum -y install lsb yum -y install libaio
4.2 安装JDK(全部节点)
1)上传JDK,在root用户下解压和配置,目录为:/usr/local/java/jdk1.8.0_231
mkdir -p /usr/share/java
将mysql-connector-java-5.1.46.jar拷贝至/usr/share/java/ 并更名为mysql-connector-java.jar
修改配置文件vi /etc/profile
export JAVA_HOME=/usr/local/java/jdk1.8.0_231 export JRE_HOME=${JAVA_HOME}/jre export CLASSPATH=.:${JAVA_HOME}/lib:${JRE_HOME}/lib export PATH=${JAVA_HOME}/bin:$PATH
添加软连接
ln -s /usr/local/java/jdk1.8.0_231/bin/java /usr/bin/java
mkdir /usr/java
ln -s /usr/local/java/jdk1.8.0_231 /usr/java/default
4.3 安装mysql(master节点)
按以下顺序安装
cd /opt/CDH6.3.2/mysql
rpm -ivh mysql-community-common-5.7.19-1.el7.x86_64.rpm --nodeps --force rpm -ivh mysql-community-libs-5.7.19-1.el7.x86_64.rpm --nodeps --force rpm -ivh mysql-community-libs-compat-5.7.19-1.el7.x86_64.rpm --nodeps --force rpm -ivh mysql-community-devel-5.7.19-1.el7.x86_64.rpm --nodeps --force rpm -ivh mysql-community-client-5.7.19-1.el7.x86_64.rpm --nodeps --force rpm -ivh mysql-community-server-5.7.19-1.el7.x86_64.rpm --nodeps --force
启动mysql
systemctl start mysqld
设置开机自启动mysql
systemctl enable mysqld
获取临时密码
cat /var/log/mysqld.log | grep password
查询到初始密码 2eB>8r;ZEO%2
使用初始密码登录mysql账号
mysql -uroot -p’2eB>8r;ZEO%2’
设置mysql root密码
set password=password('Pw#123456'); quit;
登陆设置允许外网访问当前的MySQL
mysql -uroot -pPw#123456
use mysql; update user set host='%' where host='localhost'; delete from user where host != '%'; flush privileges; quit;
建立相关数据库及用户
mysql -uroot -pPw#123456
grant all on *.* to root@'%' identified by "Pw#123456" with grant option; create database scm default character set utf8 default collate utf8_general_ci; grant all on scm.* to 'scm'@'%' identified by 'Pw#123456'; create database hue default character set utf8 default collate utf8_general_ci; grant all on hue.* to 'hue'@'%' identified by 'Pw#123456'; create database hive default character set utf8 default collate utf8_general_ci; grant all on hive.* to 'hive'@'%' identified by 'Pw#123456'; flush privileges;
5、安装CM
5.1 安装deamons(全部节点)
cd /opt/CDH6.3.2/cm rpm -ivh cloudera-manager-daemons-6.3.1-1466458.el7.x86_64.rpm
5.2 安装agent(全部节点)
cd /opt/CDH6.3.2/cm rpm -ivh cloudera-manager-agent-6.3.1-1466458.el7.x86_64.rpm
5.3 配置agent的server节点(全部节点)
cp /etc/cloudera-scm-agent/config.ini /etc/cloudera-scm-agent/config.ini.template vim /etc/cloudera-scm-agent/config.ini
server_host=bigdata1(主节点主机名)
5.4 安装server(master节点)
cd /opt/CDH6.3.2/cm rpm -ivh cloudera-manager-server-6.3.1-1466458.el7.x86_64.rpm
6、启动CM
6.1 上传CDH包到parcel-repo(master节点)
cp /opt/CDH6.3.2/parcel/* /opt/cloudera/parcel-repo/
修改sha1为sha
mv /opt/cloudera/parcel-repo/CDH-6.3.2-1.cdh6.3.2.p0.1605554-el7.parcel.sha1 CDH-6.3.2-1.cdh6.3.2.p0.1605554-el7.parcel.sha
mkdir -p /var/www/html/cdh6_parcel
将/opt/cloudera/parcel-repo 下的文件复制到 /var/www/html/cdh6_parcel/
cp /opt/cloudera/parcel-repo/* /var/www/html/cdh6_parcel/
6.2 修改配置(master节点)
vim /etc/cloudera-scm-server/db.properties
# 填入如下内容
com.cloudera.cmf.db.type=mysql com.cloudera.cmf.db.setupType=EXTERNAL com.cloudera.cmf.db.name=scm com.cloudera.cmf.db.host=bigdata1:3306 com.cloudera.cmf.db.user=root com.cloudera.cmf.db.password=Pw#123456
6.3 启动server(master节点)
service cloudera-scm-server start
查看server状态
systemctl status cloudera-scm-server
6.4 启动agent(全部节点)
service cloudera-scm-agent start
查看agent状态
systemctl status cloudera-scm-agent
7、安装CDH集群(master节点IP)
http://172.16.1.62:7180/cmf/login
账号:admin
密码:admin
进行访问,根据提示进行安装7.1 点击继续
7.2 同意协议
7.3 选择第一个版本Cloudera Express
7.4 点击继续
7.5 集群命名
7.6 选择服务器
7.7 使用Parcel安装
7.8 开始下载安装
7.9 检查机器
7.10 选择安装服务,自定义安装
7.11 自定义角色分配
7.12 数据库配置
7.13 审核更改
路径配置较复杂,不同环境可能不一样,需要一一确认