上一篇讲了普通构建spark项目 这次分享用Maven构建Spark项目,中间遇到了很多坑!其根本原因是Scala 与 Spark的版本不一致!
本次环境:
Java1.8
Scala 2.11.8
Spark spark-2.1.0-bin-hadoop2.6.tgz
新建Maven项目
接下来就是选取Maven环境的选取,简单的说一下。
如果你电脑上装了Maven 并且IDEA运行过Maven项目,则会默认给你带出来你以前配置的环境参数。 这里我对Spark项目单独建立了一个依赖文件夹下图E://M2//Repository3(个人喜欢而已)
org.apache.hadoop hadoop-common 2.6.0org.apache.hadoop hadoop-hdfs 2.6.0org.apache.spark spark-core_2.11 2.1.0org.apache.spark spark-sql_2.11 2.1.0org.apache.spark spark-hive_2.11 2.1.0org.apache.spark spark-streaming_2.11 2.1.0org.apache.spark spark-mllib_2.11 2.1.0com.databricks spark-avro_2.10 2.0.1
然后将上处常用的依赖加入pom文件中
上述的过程可能会有点慢,不过没关系我这边有阿里的Maven库镜像配置只需要将以下代码Copy到Maven安装目录下的setting文件中替换mirrors对应的位置就OK,这样下载任何依赖都是非常的快!!!
public
public-group-RELEASE
https://maven.aliyun.com/repository/public/
*
jcenter
jcenter-proxy-RELEASE
https://maven.aliyun.com/repository/jcenter/
*
spring
spring-proxy-RELEASE
https://maven.aliyun.com/repository/spring/
*
spring-plugin
spring-plugin-proxy-RELEASE
https://maven.aliyun.com/repository/spring-plugin/
*
google-proxy-RELEASE
https://maven.aliyun.com/repository/google/
*
central
central-proxy-RELEASE
https://maven.aliyun.com/repository/central/
*
grails-core
grails-core-proxy-RELEASE
https://maven.aliyun.com/repository/grails-core/
*
mapr-public
mapr-public-proxy-RELEASE
https://maven.aliyun.com/repository/mapr-public/
*
apache snapshots
apache snapshots-proxy-SNAPSHOT
https://maven.aliyun.com/repository/apache-snapshots/
*
gradle-plugin
gradle-plugin-proxy-RELEASE
https://maven.aliyun.com/repository/gradle-plugin/
*
releases
releases-hosted-RELEASE
https://maven.aliyun.com/repository/releases/
*
snapshots
snapshots-hosted-SNAPSHOT
https://maven.aliyun.com/repository/snapshots/
*
jboss-public-repository-group
JBoss Public Repository Group
http://repository.jboss.org/nexus/content/groups/public
*
有时IDEA会卡在这里,关闭重新打开就好
上面有个插件始终下载不下来,就删了它,我们也没有用它。然后再刷新Maven就没有错误了。
尝试运行
报错,我们把Test整个文件夹删掉。
再次运行
有需要文件or技术支持的请私信。
谢谢~~~
下班~~~