前提条件:
运行环境:Hadoop 3.* + Hive 3.* + MySQL 8 ,如果还未安装相关环境,请参考:Hive 一文读懂
Centos7 安装Hadoop3 单机版本(伪分布式版本)
SpringBoot 2 集成Hive 3
pom.xml
<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0"xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd"><parent><artifactId>SpringBootCase</artifactId><groupId>org.example</groupId><version>1.0-SNAPSHOT</version></parent><modelVersion>4.0.0</modelVersion><artifactId>SpringBoot-Hive3</artifactId><properties><maven.compiler.source>8</maven.compiler.source><maven.compiler.target>8</maven.compiler.target></properties><dependencies><dependency><groupId>org.apache.hive</groupId><artifactId>hive-jdbc</artifactId><version>3.1.2</version><exclusions><exclusion><artifactId>slf4j-log4j12</artifactId><groupId>org.slf4j</groupId></exclusion><exclusion><artifactId>log4j-api</artifactId><groupId>org.apache.logging.log4j</groupId></exclusion><exclusion><artifactId>log4j-core</artifactId><groupId>org.apache.logging.log4j</groupId></exclusion><exclusion><artifactId>log4j</artifactId><groupId>log4j</groupId></exclusion><exclusion><artifactId>log4j-slf4j-impl</artifactId><groupId>org.apache.logging.log4j</groupId></exclusion><exclusion><groupId>org.eclipse.jetty</groupId><artifactId>jetty-runner</artifactId></exclusion></exclusions></dependency><dependency><groupId>org.springframework.boot</groupId><artifactId>spring-boot-starter-jdbc</artifactId></dependency></dependencies></project>
配置application.properties
server.port=8083
# hive 驱动名称
spring.datasource.driver-class-name=org.apache.hive.jdbc.HiveDriver
# hive 数据库地址 = jdbc:hive2://hive 服务器地址:10000/default(默认数据库名称)
spring.datasource.url=jdbc:hive2://192.168.43.11:10000/default
# hive 服务器用户名
spring.datasource.username=root
# hive 服务器密码
spring.datasource.password=123456
编写Controller和应用入口
我这边编写一个简单的Controller,打印Hive 默认数据库包含数据库名称。
package cn.zzg.hive.controller;import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.jdbc.core.JdbcTemplate;
import org.springframework.web.bind.annotation.RequestMapping;
import org.springframework.web.bind.annotation.RestController;import java.util.List;
import java.util.Map;@RestController
@RequestMapping("/hive")
public class HiveController {@Autowiredprivate JdbcTemplate jdbcTemplate;@RequestMapping("/list")public List<Map<String, Object>> list() {String sql = "show databases";List<Map<String, Object>> list = jdbcTemplate.queryForList(sql);return list;}}
package cn.zzg.hive;import org.springframework.boot.SpringApplication;
import org.springframework.boot.autoconfigure.SpringBootApplication;@SpringBootApplication
public class Application {public static void main(String[] args) {SpringApplication.run(Application.class, args);}
}
效果截图:
SpringBoot 2 集成Hive 3 遇到的问题
问题一:Class path contains multiple SLF4J bindings,日志依赖重复冲突。
造成此问题的原因是:spring boot 默认日志为logback, 而引用的hive-jdbc 及其关联jar 使用的日志为 log4j ,造成SLF4J 绑定冲突。
解决办法:移除冲突的日志:log4j
<exclusion><artifactId>slf4j-log4j12</artifactId><groupId>org.slf4j</groupId></exclusion><exclusion><artifactId>log4j-api</artifactId><groupId>org.apache.logging.log4j</groupId></exclusion><exclusion><artifactId>log4j-core</artifactId><groupId>org.apache.logging.log4j</groupId></exclusion><exclusion><artifactId>log4j</artifactId><groupId>log4j</groupId></exclusion><exclusion><artifactId>log4j-slf4j-impl</artifactId><groupId>org.apache.logging.log4j</groupId></exclusion>
问题二:SpringBoot 自带容器Tomcat 与Hive JDBC 关联Jetty 容器冲突
An attempt was made to call the method org.apache.tomcat.util.ExceptionUtils.preload()V but it does not exist. Its class, org.apache.tomcat.util.ExceptionUtils, is available from the following locations:jar:file:/E:/maven_repository/org/eclipse/jetty/jetty-runner/9.3.20.v20170531/jetty-runner-9.3.20.v20170531.jar!/org/apache/tomcat/util/ExceptionUtils.classjar:file:/E:/maven_repository/org/apache/tomcat/embed/tomcat-embed-core/9.0.12/tomcat-embed-core-9.0.12.jar!/org/apache/tomcat/util/ExceptionUtils.class
解决办法:移除Hive JDBC 依赖的Jetty 容器。
<exclusion><groupId>org.eclipse.jetty</groupId><artifactId>jetty-runner</artifactId></exclusion>
问题三:通过JDBC 连接Hive 数据库提示:
java.net.ConnectException: Connection refused
造成此类 问题的原因:hiveserver2 服务没有正常启动。
解决办法: 切换至hive 安装目录的bin/ 文件夹下(/usr/local/hive/bin),执行如下命令:
# 方式一
hiveserver2 &
# 方式二
hive --service hiveserver2
问题四:访问Hive 数据库,提示无权限问题:
造成此类问题的原因:hadoop 没有配置权限导致。
解决办法:切换至hadoop 安装目录的/etc文件夹下(/usr/local/hadoop/etc/hadoop/core-site.xml),添加如下配置:
<property><name>hadoop.proxyuser.root.hosts</name><value>*</value></property><property><name>hadoop.proxyuser.root.groups</name><value>*</value></property>