hive安装部署
为了简单起见，都安装到node1服务器上。（集群：node1，node2，node3）
环境（已安装）：Hadoop3.4.0，jdk-8u171
需要安装：MySQL8.4.3，hive4.0.0
下载链接：
hive4.0.0：
http://archive.apache.org/dist/hive/hive-4.0.0/apache-hive-4.0.0-bin.tar.gz
MySQL驱动包：https://repo1.maven.org/maven2/mysql/mysql-connector-java/8.0.30/mysql-connector-java-8.0.30.jar
上传下面3个文件到虚拟机上node1上。
步骤1：安装MySQL数据库
我安装的是8.4.3版本的MySQL
# 我安装的是8.4.3版本的
[root@node1 server]# ls
hadoop jdk mysql-8.1.0-linux-glibc2.28-aarch64.tar.xz
hadoop-3.4.0 jdk1.8.0_171 mysql84-community-release-el7-1.noarch.rpm
[root@node1 server]# rpm -Uvh mysql84-community-release-el7-1.noarch.rpm
警告：mysql84-community-release-el7-1.noarch.rpm: 头V4 RSA/SHA256 Signature, 密钥 ID a8d3785c: NOKEY
准备中... ################################# [100%]
正在升级/安装...1:mysql84-community-release-el7-1 ################################# [100%]# 安装MySQL服务端[root@node1 ~]# yum -y install mysql-community-server
已加载插件：fastestmirror, langpacks
Loading mirror speeds from cached hostfile* base: mirrors.aliyun.com* extras: mirrors.aliyun.com* updates: mirrors.aliyun.com
...
总下载量：84 M
Downloading packages:
警告：/var/cache/yum/x86_64/7/mysql-8.4-lts-community/packages/mysql-community-client-plugins-8.4.3-1.el7.x86_64.rpm: 头V4 RSA/SHA256 Signature, 密钥 ID a8d3785c: NOKEY
mysql-community-client-plugins-8.4.3-1.el7.x86_64.rpm 的公钥尚未安装
(1/7): mysql-community-client-plugins-8.4.3-1.el7.x86_64.rpm | 3.5 MB 00:00:01
(2/7): mysql-community-common-8.4.3-1.el7.x86_64.rpm | 689 kB 00:00:00
(3/7): mysql-community-client-8.4.3-1.el7.x86_64.rpm | 15 MB 00:00:02
(4/7): mysql-community-icu-data-files-8.4.3-1.el7.x86_64.rpm | 2.2 MB 00:00:00
(5/7): mysql-community-libs-8.4.3-1.el7.x86_64.rpm | 1.5 MB 00:00:00
(6/7): mysql-community-libs-compat-8.4.3-1.el7.x86_64.rpm | 670 kB 00:00:00
(7/7): mysql-community-server-8.4.3-1.el7.x86_64.rpm | 61 MB 00:00:05
---------------------------------------------------------------------------------------
总计 9.6 MB/s | 84 MB 00:08
从 file:///etc/pki/rpm-gpg/RPM-GPG-KEY-mysql-2023 检索密钥
导入 GPG key 0xA8D3785C:用户ID : "MySQL Release Engineering <mysql-build@oss.oracle.com>"指纹 : bca4 3417 c3b4 85dd 128e c6d4 b7b3 b788 a8d3 785c软件包 : mysql84-community-release-el7-1.noarch (installed)来自 : /etc/pki/rpm-gpg/RPM-GPG-KEY-mysql-2023
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction
警告：RPM 数据库已被非 yum 程序修改。正在安装 : mysql-community-common-8.4.3-1.el7.x86_64 1/8 正在安装 : mysql-community-client-plugins-8.4.3-1.el7.x86_64 2/8 正在安装 : mysql-community-libs-8.4.3-1.el7.x86_64 3/8 正在安装 : mysql-community-client-8.4.3-1.el7.x86_64 4/8 正在安装 : mysql-community-icu-data-files-8.4.3-1.el7.x86_64 5/8 正在安装 : mysql-community-server-8.4.3-1.el7.x86_64 6/8 正在安装 : mysql-community-libs-compat-8.4.3-1.el7.x86_64 7/8 正在删除 : 1:mariadb-libs-5.5.68-1.el7.x86_64 8/8 验证中 : mysql-community-libs-compat-8.4.3-1.el7.x86_64 1/8 验证中 : mysql-community-client-plugins-8.4.3-1.el7.x86_64 2/8 验证中 : mysql-community-common-8.4.3-1.el7.x86_64 3/8 验证中 : mysql-community-icu-data-files-8.4.3-1.el7.x86_64 4/8 验证中 : mysql-community-libs-8.4.3-1.el7.x86_64 5/8 验证中 : mysql-community-client-8.4.3-1.el7.x86_64 6/8 验证中 : mysql-community-server-8.4.3-1.el7.x86_64 7/8 验证中 : 1:mariadb-libs-5.5.68-1.el7.x86_64 8/8
已安装:mysql-community-libs.x86_64 0:8.4.3-1.el7 mysql-community-libs-compat.x86_64 0:8.4.3-1.el7 mysql-community-server.x86_64 0:8.4.3-1.el7
作为依赖被安装:mysql-community-client.x86_64 0:8.4.3-1.el7 mysql-community-client-plugins.x86_64 0:8.4.3-1.el7 mysql-community-common.x86_64 0:8.4.3-1.el7 mysql-community-icu-data-files.x86_64 0:8.4.3-1.el7
替代:mariadb-libs.x86_64 1:5.5.68-1.el7
完毕！
# 启动Mysql设置开机启动
[root@node1 ~]# systemctl start mysqld # 有时可能是systemctl start/enable/status/.. mysql
[root@node1 ~]# systemctl enable mysqld
# 有时需要重启一下mysql：systemctl restart mysqld
# 检查Mysql服务状态，且服务状态必须得是：active (running)
[root@node1 ~]# systemctl status mysqld
● mysqld.service - MySQL ServerLoaded: loaded (/usr/lib/systemd/system/mysqld.service; enabled; vendor preset: disabled)Active: active (running) since 三 2024-10-23 10:33:09 CST; 38s agoDocs: man:mysqld(8)http://dev.mysql.com/doc/refman/en/using-systemd.htmlMain PID: 6716 (mysqld)Status: "Server is operational"CGroup: /system.slice/mysqld.service└─6716 /usr/sbin/mysqld
10月 23 10:32:57 node1 systemd[1]: Starting MySQL Server...
10月 23 10:33:09 node1 systemd[1]: Started MySQL Server.
# MySQL8.0安装成功后会在/var/log/mysqld.log文件中给root生成了一个默认密码，通过下面命令先找到临时密码，然后进行修改
[root@node1 ~]# cat /var/log/mysqld.log | grep 'password'
2024-10-23T02:33:03.166159Z 6 [Note] [MY-010454] [Server] A temporary password is generated for root@localhost: uu+Z4*w>!h:a
[root@node1 ~]# mysql -uroot -p
Enter password:
Welcome to the MySQL monitor. Commands end with ; or \g.
Your MySQL connection id is 9
Server version: 8.4.3
Copyright (c) 2000, 2024, Oracle and/or its affiliates.
Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.
Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.
mysql> ALTER USER 'root'@'localhost' IDENTIFIED BY 'Jieru141592@';
# 如果你想设置简单密码，需要降低Mysql的密码安全级别
mysql> set global validate_password.policy=0; # 密码安全级别低
# 或者：mysql> set global validate_password.policy=LOW;
mysql> set global validate_password.length=4; # 密码长度最低4位即可
mysql> ALTER USER 'root'@'localhost' IDENTIFIED BY '123456';
mysql> GRANT ALL ON *.* TO 'root'@'%';
ERROR 1410 (42000): You are not allowed to create a user with GRANT
mysql> update user set host='%' where user ='root';
Query OK, 1 row affected (0.00 sec)
Rows matched: 1 Changed: 1 Warnings: 0
mysql> flush privileges;
Query OK, 0 rows affected (0.00 sec)
mysql> GRANT ALL ON *.* TO 'root'@'%';
Query OK, 0 rows affected (0.00 sec)
mysql> flush privileges;
Query OK, 0 rows affected (0.00 sec)
# 授权root用户远程登录并且密码为你登录mysql所使用的密码
#（很重要！！！，方便后续hive远程连接MySQL）
grant all privileges on *.* to 'root'@'%' with grant option;
flush privileges;问题：如果一开始就将密码改为123456，则报错，因为： 查看官网：https://dev.mysql.com/doc/refman/8.4/en/linux-installation-yum-repo.html，发现“validate_password 默认安装。validate_password 实施的默认密码策略要求密码至少包含一个大写字母、一个小写字母、一个数字和一个特殊字符，并且密码总长度至少为 8 个字符。”。
所以，先把密码改为较为复杂且满足validate_password ，之后再降低Mysql的密码安全级别，最后再修改一次密码
# 问题
mysql> ALTER USER 'root'@'localhost' IDENTIFIED BY '123456';
ERROR 1819 (HY000): Your password does not satisfy the current policy requirements
# 如果你想设置简单密码，需要降低Mysql的密码安全级别
mysql> set global validate_password.policy=0; # 密码安全级别低
ERROR 1820 (HY000): You must reset your password using ALTER USER statement before executing this statement.
mysql> SHOW VARIABLES LIKE 'validate_password.%';
ERROR 1820 (HY000): You must reset your password using ALTER USER statement before executing this statement.
# ！！！！注意：在新版MySQL里，MySQL8.0开始（好像是），密码验证插件“validate_password.policy”，而MySQL5.7是“validate_password_policy”
# 查看官网：https://dev.mysql.com/doc/refman/8.4/en/linux-installation-yum-repo.html，发现“validate_password 默认安装。validate_password 实施的默认密码策略要求密码至少包含一个大写字母、一个小写字母、一个数字和一个特殊字符，并且密码总长度至少为 8 个字符。”。
# 添加远程登录用户
mysql> grant all privileges on *.* to root@"%" identified by '123456' with grant option;
ERROR 1064 (42000): You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near 'identified by '123456' with grant option' at line 1步骤2：配置Hadoop
Hive的运行依赖于Hadoop（HDFS、MapReduce、YARN都依赖）
同时涉及到HDFS文件系统的访问，所以需要配置Hadoop的代理用户
即设置hadoop用户允许代理（模拟）其它用户
配置如下内容在Hadoop（cd /export/server/hadoop/etc/hadoop)的core-site.xml中，并分发到其它节点，且重启HDFS集群
<!-- 整合hive代理用户配置-->
<property><name>hadoop.proxyuser.hadoop.hosts</name> <value>*</value>
</property>
<property><name>hadoop.proxyuser.hadoop.groups</name> <value>*</value>
</property>步骤3：下载解压Hive
[root@node1 ~]# cd /home/master
[root@node1 master]# ls
apache-hive-4.0.0-bin.tar.gz Downloads Pictures Videos Desktop Music Public 桌面 Documents mysql-connector-java-8.0.30.jar Templates
[root@node1 master]# mv mysql-connector-java-8.0.30.jar /export/server
[root@node1 master]# mv apache-hive-4.0.0-bin.tar.gz /export/server-
切换到hadoop用户
su - hadoop
-
下载Hive安装包：我下载的是hive-4.0.0，下载链接如下：
http://archive.apache.org/dist/hive/hive-4.0.0/apache-hive-4.0.0-bin.tar.gz
-
解压到node1服务器的：/export/server/内
tar -zxvf apache-hive-4.0.0-bin.tar.gz -C /export/server/
-
设置软连接
ln -s /export/server/apache-hive-4.0.0-bin /export/server/hive
步骤4：提供MySQL Driver包
-
下载MySQL驱动包：链接如下：
https://repo1.maven.org/maven2/mysql/mysql-connector-java/8.0.30/mysql-connector-java-8.0.30.jar
-
将下载好的驱动jar包，放入：Hive安装文件夹的lib目录内
mv mysql-connector-java-8.0.30.jar /export/server/hive/lib/
步骤5：配置Hive
在hive的conf目录内，有个hive-env.sh.template，将其改名为hive-env.sh，输入：
mv hive-env.sh.template hive-env.sh
并在其内添加如下配置:
export HADOOP_HOME=/export/server/hadoop export HIVE_CONF_DIR=/export/server/hive export HIVE_AUX_JARS_PATH=/export/server/hive/lib
在hive的conf目录内，新建hive-site.xml文件，填入以下内容：
注意：
1.配置 <name>javax.jdo.option.ConnectionURL</name> <value>jdbc:mysql://node1:3306/hive_metastore?createDatabaseIfNotExist=true&useSSL=false&useUnicode=true&characterEncoding=UTF-8&allowPublicKeyRetrieval=true</value>时，MySQL8.0版本必须要有allowPublicKeyRetrieval=true，其中hive_metastore是初始化Hive所需的元数据库（可自定义）。
<configuration>
<!-- 存储元数据mysql相关配置 -->
<property><name>javax.jdo.option.ConnectionURL</name><value>jdbc:mysql://node1:3306/hive_metastore?createDatabaseIfNotExist=true&useSSL=false&useUnicode=true&characterEncoding=UTF-8&allowPublicKeyRetrieval=true</value>
</property><property><name>javax.jdo.option.ConnectionDriverName</name><value>com.mysql.cj.jdbc.Driver</value>
</property><property><name>javax.jdo.option.ConnectionUserName</name><value>root</value>
</property><property><name>javax.jdo.option.ConnectionPassword</name><value>123456</value><description>登录MySQL的秘密</description>
</property><!-- H2S运行绑定host -->
<property><name>hive.server2.thrift.bind.host</name><value>node1</value>
</property><property><name>hive.server2.thrift.port</name><value>10000</value>
</property> <!-- 远程模式部署metastore metastore地址 -->
<property><name>hive.metastore.uris</name><value>thrift://node1:9083</value>
</property><!-- 关闭元数据存储授权 -->
<property><name>hive.metastore.event.db.notification.api.auth</name><value>false</value>
</property>
</configuration>步骤6：初始化元数据库
Hive的配置已经完成，现在在启动Hive前，需要先初始化Hive所需的元数据库。
[root@node1 ~]# mysql -uroot -p(密码：123456)
-
执行元数据库初始化命令：
[root@node1 ~]# cd /export/server/hive
[root@node1 ~]# bin/schematool -initSchema -dbType mysql -verbos # 初始化成功后，会在MySQL的hive库中新建x张元数据管理的表。（具体不知道多少张表，但有83行）
是否初始化成功的检验方法：
登录mysql，查看数据库，如果有hive3数据库，使用hive_metastore数据库后查看表，如果内部有表则初始化成功。
步骤7：启动Hive（使用hadoop用户）
-
在启动前，需要将hdfs和yarn集群启动：切换到hadoop用户，输入
start-all.sh -
确保Hive文件夹所属为hadoop用户
[root@node1 bin]# cd
[root@node1 ~]# cd /export/server
[root@node1 server]# ll
总用量 448044
drwxrwxr-x 11 hadoop hadoop 204 10月 23 12:26 apache-hive-4.0.0-bin
-rw-rw-r-- 1 master master 458782861 10月 23 12:21 apache-hive-4.0.0-bin.tar.gz
lrwxrwxrwx 1 hadoop hadoop 27 10月 18 21:57 hadoop -> /export/server/hadoop-3.4.0
drwxr-xr-x 11 hadoop hadoop 227 10月 18 23:18 hadoop-3.4.0
lrwxrwxrwx 1 hadoop hadoop 36 10月 23 12:27 hive -> /export/server/apache-hive-4.0.0-bin
lrwxrwxrwx. 1 hadoop hadoop 27 10月 18 15:06 jdk -> /export/server/jdk1.8.0_171
drwxr-xr-x. 8 hadoop hadoop 255 3月 29 2018 jdk1.8.0_171
-rw-rw-r-- 1 master master 11700 10月 23 09:41 mysql84-community-release-el7-1.noarch.rpm如上，显然不是，则输入：
[root@node1 server]# chown -R hadoop:hadoop apache-hive-4.0.0-bin hive
-
创建一个logs的日志文件夹：
mkdir /export/server/hive/logs
-
启动元数据管理服务（必须启动，否则无法工作）
前台启动：bin/hive --service metastore（特别慢）（启动完成后，再另起shell）
后台启动并写到日志文件里：nohup bin/hive --service metastore >> logs/metastore.log 2>&1 &
-
启动客户端，二选一（当前先选择Hive Shell方式）
Hive Shell方式（可以直接写SQL）： bin/hive
Hive ThriftServer方式（不可直接写SQL，需要外部客户端链接使用）：
bin/hive --service hiveserver2（启动完成后，再另起shell）
或者
nohup bin/hive --service hiveserver2 >> logs/hiveserver2.log 2>&1 &
可输入[hadoop@node1 hive]$ tail -f logs/hiveserver2.log来追踪
最后在hive目录下，启动bin/beeline，! connect jdbc:hive2://node1:10000
显示数据库：
show databases;
如果有default数据库则启动成功。
新建窗口，查看jps：若不启动hiveserver2，原本应该出现2个RunJar的，
遇到的问题：
问题1：解决日志jar包冲突
#在系统重查找所有的 SLF4J 绑定的 JAR 文件
[root@node1 lib]# find / -name "*slf4j*.jar"
find: ‘/run/user/1000/gvfs’: 权限不够
/export/server/hadoop-3.4.0/share/hadoop/common/lib/jul-to-slf4j-1.7.36.jar
/export/server/hadoop-3.4.0/share/hadoop/common/lib/slf4j-api-1.7.36.jar
/export/server/hadoop-3.4.0/share/hadoop/common/lib/slf4j-reload4j-1.7.36.jar
/export/server/apache-hive-4.0.0-bin/hcatalog/share/webhcat/svr/lib/jul-to-slf4j-1.7.30.jar
/export/server/apache-hive-4.0.0-bin/lib/log4j-slf4j-impl-2.18.0.jar#删除
mv $HIVE_HOME/lib/log4j-slf4j-impl-2.18.0.jar $HIVE_HOME/lib/log4j-slf4j-impl-2.18.0.bak#恢复 方法
#mv $HIVE_HOME/lib/log4j-slf4j-impl-2.18.0.bak $HIVE_HOME/lib/log4j-slf4j-impl-2.18.0.jar 问题2：连接hive2时，Could not open connection to the HS2 server. Please check the server URI and if the URI is correct, then ask the administrator to check the server status. Enable verbose error messages (--verbose=true) for more information.，说明，可能是在启动时没启动成功，即RunJar的端口号与对应应该要启动的端口号没有一一对应，
即：输入hive --service metastore，确认metastore是否启动成功，然后可通过输入jps查看RunJar前面的端口号，再输入ps -ef | grep HiveServer2/端口号等查看是否启动成功
之后依次确认其他启动是否成功。
[hadoop@node1 hive]$ beeline
...
Beeline version 4.0.0 by Apache Hive
beeline> !connect jdbc:hive2:://node1:10000/default
Driver scan complete in 9ms
No known driver to handle "jdbc:hive2:://node1:10000/default"
beeline> !connect jdbc:hive2://node1:10000/default
Connecting to jdbc:hive2://node1:10000/default
Enter username for jdbc:hive2://node1:10000/default: root
Enter password for jdbc:hive2://node1:10000/default:
24/10/28 17:45:32 [main]: WARN jdbc.HiveConnection: Failed to connect to node1:10000
Could not open connection to the HS2 server. Please check the server URI and if the URI is correct, then ask the administrator to check the server status. Enable verbose error messages (--verbose=true) for more information.
Error: Could not open client transport with JDBC Uri: jdbc:hive2://node1:10000/default: java.net.ConnectException: Connection refused (Connection refused) (state=08S01,code=0)
beeline> 
