目录
一、前期准备
1.1 部署 docker
1.2 部署 docker-compose
1.3 版本说明
二 、镜像构建启动
2.1 系统镜像构建
2.2 安装包源镜像构建
2.3 kdc镜像构建
2.4 集群安装
2.5 容器导出为镜像
三、Ubuntu环境安装测试
3.1 环境准备
3.2 集群容器启动
一、前期准备
1.1 部署 docker
# 安装yum-config-manager配置工具
yum -y install yum-utils# 建议使用阿里云yum源:(推荐)
#yum-config-manager --add-repo https://download.docker.com/linux/centos/docker-ce.repo
yum-config-manager --add-repo http://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo# 安装docker-ce版本
yum install -y docker-ce# 启动并开机启动
systemctl enable --now docker
docker --version
docker info
1.2 部署 docker-compose
curl -SL https://github.com/docker/compose/releases/download/v2.16.0/docker-compose-linux-x86_64 -o /usr/local/bin/docker-composechmod +x /usr/local/bin/docker-compose
docker-compose --version
创建网络
docker network create hadoop-network
1.3 版本说明
组件 | 版本 |
操作系统 | Centos7.6/Ubuntu20.04.5 |
ambari | 2.7.4 |
HDP | 3.1.4 |
HDP-GPL | 3.1.4 |
HDP-UTILS | 1.1.0.22 |
JDK | jdk-8u162-linux-x64 |
如下安装只做测试。
二 、镜像构建启动
2.1 系统镜像构建
# 拉取 Centos 镜像
FROM centos:7.6.1810# 作者信息
LABEL kangll # 安装基本工具
RUN yum update -y && yum install -y \net-tools \vim \wget \curl \lrzsz \ nc \openssl \sed \telnet \openssh-server \httpd \chronyd \zip unzip \openssh-clients \expect \krb5-server \krb5-libs \ krb5-workstation \&& yum clean all# JDK
RUN mkdir -p /usr/java
ADD ./config/jdk-8u162-linux-x64.tar.gz /usr/java/
ENV JAVA_HOME=/usr/java/jdk1.8.0_162
ENV CLASSPATH=.:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jar
ENV PATH=$JAVA_HOME/bin:$PATH# 添加一个名为myuser的用户
RUN useradd -m winner_spark
# 设置myuser用户的密码,这里设置为mypassword
RUN echo 'winner_spark:123456' | chpasswd
# 设置myuser用户为sudoer
RUN echo 'winner_spark ALL=(ALL) NOPASSWD:ALL' >> /etc/sudoers# SSH
RUN sed -i '/^HostKey/'d /etc/ssh/sshd_config
RUN echo 'HostKey /etc/ssh/ssh_host_rsa_key' >> /etc/ssh/sshd_config
RUN mkdir /var/run/sshd/
RUN echo 'UseDNS no' >> /etc/ssh/sshd_config
RUN sed -i -e '/pam_loginuid.so/d' /etc/pam.d/sshdCOPY ./config/hosts /etc/#set default password
RUN echo 'root:123456' | chpasswd
RUN /usr/bin/ssh-keygen -ACOPY ./config/krb5.conf /etc#声明22端口
EXPOSE 22RUN mkdir -p /opt && mkdir -p /var/log/supervisord
RUN echo '/usr/sbin/sshd -D' >> /opt/run.sh
RUN chmod +x /opt/run.sh
CMD ["/opt/run.sh"]#
WORKDIR /opt/
系统镜像构建
docker build -f dockerfile-centos -t centos-jdk:1.0 .
2.2 安装包源镜像构建
dockerfile-httpd
FROM centos-jdk:1.0# centos7.6 JDK1.8
LABEL kangllRUN yum -y install httpd
RUN echo "crushlinux" >/var/www/html/index.html
RUN ln -sf /usr/share/zoneinfo/Asia/Shanghai /etc/localtime# 添加yum 源
COPY ./repo/ambari.repo /etc/yum.repos.d/
COPY ./repo/hdp.repo /etc/yum.repos.d/
COPY ./repo/hdp.gpl.repo /etc/yum.repos.d/RUN mkdir -p /hadoop/hdp
ADD ./HDP/ambari.tar.gz /var/www/html
ADD ./HDP/HDP-GPL.tar.gz /var/www/html
ADD ./HDP/HDP.tar.gz /var/www/html
ADD ./HDP/HDP-UTILS.tar.gz /var/www/htmlEXPOSE 80CMD ["httpd","-DFOREGROUND"]
源镜像构建
docker build -t ambari:2.7.4 -f dockerfile-ambari .
2.3 kdc镜像构建
dockerfile-kerberos
FROM centos:7.6.1810RUN yum install -y krb5-server krb5-libs krb5-auth-dialog krb5-workstationRUN yum install -y net-tools telnetCOPY ./config/krb5.conf /etc
COPY ./config/kdc.conf /var/kerberos/krb5kdc/
COPY ./config/kadm5.acl /var/kerberos/krb5kdc/CMD ["/usr/sbin/init"]
服务启动
# 进入容器 启动服务
Systemctl restart krb5kdc
Systemctl restart kadmin
docker-compose-base.yml 启动基础环境
version: '3'
services:mysql:image: registry.cn-hangzhou.aliyuncs.com/bigdata_cloudnative/mysql-test:5.7restart: alwaysnetworks:- hadoop-networkcontainer_name: mysqlhostname: mysql-testenvironment:MYSQL_ROOT_PASSWORD: Winner001TZ: Asia/Shanghaiports:- 3306:3306volumes:- /data:/var/lib/mysql- ./conf/my.cnf:/etc/mysql/my.cnfcommand:--max_connections=1000--character-set-server=utf8mb4--collation-server=utf8mb4_general_ci--default-authentication-plugin=mysql_native_passwordhealthcheck:test: ["CMD-SHELL", "curl --silent localhost:3306 >/dev/null || exit 1"]interval: 10stimeout: 10shttpd:container_name: httpdhostname: httpdimage: httpd:1.0networks:- hadoop-networkports:- "80:80"privileged: truerestart: alwaysvolumes:- "/etc/hosts:/etc/hosts"kdc:image: kerberos:1.0restart: alwaysnetworks:- hadoop-networkcontainer_name: kerberoshostname: kerberos.winnerports:- 88:88- 749:749- 750:750volumes:- /etc/hosts:/etc/hosts- /hadoop/kerberos/krb5kdc:/var/kerberos/krb5kdc- /hadoop/kerberos/keytabs:/etc/security/keytabscommand:- /bin/sh- -c- |/opt/apache/kerberosStart.shnetworks:hadoop-network:external:name: hadoop-network
Ambari启动
# 启动基础环境
docker-compose -f docker-compose-base.yaml up -d# 启动ambari , 在此容器中安装集群
docker-compose -f docker-compose-ambari.yaml up -d
2.4 集群安装
repo地址
节点和秘钥
agent安装
服务安装
由于是单机直接下一步
安装组件
输入密码
数据目录确认
用户确认下一步
服务参数确定
发布
安装完成
2.5 容器导出为像
docker ps -a
docker commit 容器ID ambari-1.0.tar.gz
三、Ubuntu环境安装测试
3.1 环境准备
我们准备Ubuntu 环境,在Ubuntu环境安装Ambari单机,相当于在Centos服务器安装的docker化的Ambari 导出为镜像 然后在Ubutntu 环境尝试启动。
准备Ubuntu 环境,版本:Ubuntu 20.04.5 LTS
将镜像 ambari-1.0.tar.gz 加载到Ubuntu 环境
docker load -i ambari-1.0.tar.gz
MySQL元数据库备份还原(Ambari、hive、ranger)
HDFS、kerberos 备份的数据拷贝到Ubuntu服务器并保持数据文件路径不变。
将在centos 安装完成的集群备份元数据库写入到新的库中 , 也就是Ubuntu环境要使用的元数据库。
# 备份的元数据写入新的数据库中
mysql -hmysql-test -uroot -pWinner001 ambari < ambari.sql
mysql -hmysql-test -uroot -pWinner001 ranger < ranger.sql
mysql -hmysql-test -uroot -pWinner001 hive < hive.sql
备份的kerberos和安装包解压
# 解压到 /usr 路径下,保存的是集群安装的数据
hdp.tar.gz# 在根路径解压保留的kerberos 和HDFS 初始化的元数据
hadoop.tar.gz
基础环境启动docker-compose-base.yaml
version: '3'
services:mysql:image: registry.cn-hangzhou.aliyuncs.com/bigdata_cloudnative/mysql-test:5.7restart: alwaysnetworks:- hadoop-networkcontainer_name: mysqlhostname: mysql-testenvironment:MYSQL_ROOT_PASSWORD: Winner001TZ: Asia/Shanghaiports:- 3306:3306volumes:- /data:/var/lib/mysql- ./conf/my.cnf:/etc/mysql/my.cnfcommand:--max_connections=1000--character-set-server=utf8mb4--collation-server=utf8mb4_general_ci--default-authentication-plugin=mysql_native_passwordhealthcheck:test: ["CMD-SHELL", "curl --silent localhost:3306 >/dev/null || exit 1"]interval: 10stimeout: 10shttpd:container_name: httpdhostname: httpdprivileged: trueimage: httpd:1.0networks:- hadoop-networkports:- "80:80"privileged: truerestart: alwaysvolumes:- "/etc/hosts:/etc/hosts"kdc:image: kerberos:1.0restart: alwaysprivileged: truenetworks:- hadoop-networkcontainer_name: kerberoshostname: kerberos.winnerports:- 888:88- 8749:749- 8750:750volumes:- /etc/hosts:/etc/hosts- /hadoop/kerberos/krb5kdc:/var/kerberos/krb5kdc- /hadoop/kerberos/keytabs:/etc/security/keytabsnetworks:hadoop-network:external:name: hadoop-network
启动
# 启动
docker-compose -f docker-compose-base.yaml up -d# 停止
docker-compose -f docker-compose-base.yaml down
3.2 集群容器启动
镜像
启动容器
version: '3'
services:ambari:image: ambari:2.7.4 # 修改为新的容器container_name: ambarihostname: ambari-serverrestart: alwaysprivileged: truenetworks:- hadoop-networkports:- 8080:8080- 8088:8088- 50070:50070- 50075:50075- 8188:8188- 8042:8042- 10000:10000- 9083:9083- 6080:6080- 16010:16010volumes:- /etc/hosts:/etc/hosts- /usr/hdp:/usr/hdp- /hadoop/hdfs:/hadoop/hdfs- /hadoop/etc/hadoop/:/etc/hadoop/- /hadoop/etc/hbase/:/etc/hbase/- /hadoop/etc/hive/:/etc/hive/- /hadoop/etc/livy2/:/etc/livy2/- /hadoop/etc/spark2/:/etc/spark2/- /hadoop/etc/tez/:/etc/tez/- /hadoop/etc/zookeeper/:/etc/zookeeper/- /hadoop/yarn:/hadoop/yarn- /hadoop/kerberos/krb5kdc:/var/kerberos/krb5kdc- /hadoop/kerberos/keytabs:/etc/security/keytabscommand:- /bin/sh- -c- |/opt/apache/bootstrap.sh
networks:hadoop-network:external:name: hadoop-network
启动
docker-compose -f docker-compose-ambari.yaml up -d
下次安装直接加载镜像,然后执行yaml文件即可
docker load -i ambari-1.0.tar.gz
docker load -i kerberos.tar.gz
docker load -i mysql.tar.gz
docker load -i HDP-REPO.tar.gz
脚本
bootstrap.sh
#!/bin/bashset -xambari_db=ambari
myurl=mysql-test
myuser=root
mypwd=Winner001
myport=3306
mydb=ipvacloud
java_home=/usr/java/jdk1.8.0_162
mysql_driver=/usr/share/java/mysql-connector-java.jarBASEDIR=$(cd "$(dirname "$0")"; pwd)wait_for() {echo Waiting for $1 to listen on $2...while ! nc -z $1 $2; do echo waiting...; sleep 1s; done
}setup_ambari() {# 强制替换文件cp -f /tmp/ambari.properties /etc/ambari-server/conf/ cp -f /tmp/password.dat /etc/ambari-server/conf/cp -f /tmp/ambari-agent.ini /etc/ambari-agent/conf/
}# wait for mysql and httpd
#wait_for windp-aio 3306
#wait_for httpd 80# 开始ambari安装
yum -y install ambari-server
yum -y install ambari-agentre=`mysql -h${myurl} -u${myuser} -p${mypwd} -e "show databases"|grep ambari|wc -l`
if [ $re -eq 0 ];then# sql 初始化mysql -h${myurl} -u${myuser} -p${mypwd} < /tmp/init_db.sql mysql -h${myurl} -u${myuser} -p${mypwd} ambari < /tmp/Ambari-DDL-MySQL-CREATE.sql
fi# 配置
setup_ambari# 启动ambari服务
/usr/sbin/ambari-server start
/usr/sbin/ambari-agent start/usr/sbin/sshd -Dtail -f /var/log/ambari-server/ambari-server.log
参考文章:https://www.cnblogs.com/liugp/p/17489616.html
Ubuntu 环境安装docker: 【云计算】Ubuntu多种安装docker方式_ubuntu安装docker-CSDN博客