【spark源码修改】hive3.1.3 spark3.5.4编译，需要修改源码，最终编译成功

1. 准备安装包与maven编译环境
- 1.1 安装环境准备
- 1.2 修改pom
- 1.3 打包命令
2. 编译与问题解决
- 2.1 开始编译失败，缺包pentaho-aggdesigner-algorithm:pom:5.1.5-jhyde
- 2.2 Hive Spark Remote Client 模块编译失败，类找不到，原因是升级后，类不兼容
3. 修改源码
- 3.1 源码不兼容，源码SparkCounter分析（附带修改过程与结果，spark2,spark3源码对比）
- 3.2 修改代码 SparkCounter
- 3.3 修改代码 ShuffleWriteMetrics
4. 继续编译
- 4.1 有一个test的类不兼容，直接删掉就行，重新编译
- 4.2 Hive Llap Server 编译失败
- 4.3 编译完成
- 4.4 编译后的安装包

1. 准备安装包与maven编译环境

1.1 安装环境准备

安装包到官网找， hive.apache.org, spark.apache.org
maven环境准备：初始化只改了settings的ali源

1.2 修改pom

pom版本修改前：

    <spark.version>2.3.0</spark.version><scala.binary.version>2.11</scala.binary.version><scala.version>2.11.8</scala.version>

修改后：

    <spark.version>3.5.4</spark.version><scala.binary.version>2.13</scala.binary.version><scala.version>2.13.8</scala.version>

1.3 打包命令

mvn clean package -Pdist -DskipTests -Dmaven.javadoc.skip=true

2. 编译与问题解决

2.1 开始编译失败，缺包pentaho-aggdesigner-algorithm:pom:5.1.5-jhyde

pentaho-aggdesigner-algorithm:pom:5.1.5-jhyde 找不到

Downloaded from central: https://repo.maven.apache.org/maven2/org/apache/hadoop/hadoop-mapreduce-client-core/2.7.2/hadoop-mapreduce-client-core-2.7.2.pom (3.5 kB at 12 kB/s)
Downloading from central: https://repo.maven.apache.org/maven2/org/apache/hadoop/hadoop-common/2.6.4/hadoop-common-2.6.4.pom
Downloaded from central: https://repo.maven.apache.org/maven2/org/apache/hadoop/hadoop-common/2.6.4/hadoop-common-2.6.4.pom (31 kB at 77 kB/s)
Downloading from central: https://repo.maven.apache.org/maven2/org/apache/hadoop/hadoop-project-dist/2.6.4/hadoop-project-dist-2.6.4.pom
Downloaded from central: https://repo.maven.apache.org/maven2/org/apache/hadoop/hadoop-project-dist/2.6.4/hadoop-project-dist-2.6.4.pom (18 kB at 48 kB/s)
Downloading from central: https://repo.maven.apache.org/maven2/org/apache/hadoop/hadoop-project/2.6.4/hadoop-project-2.6.4.pom
Downloaded from central: https://repo.maven.apache.org/maven2/org/apache/hadoop/hadoop-project/2.6.4/hadoop-project-2.6.4.pom (43 kB at 106 kB/s)
Downloading from central: https://repo.maven.apache.org/maven2/org/apache/hadoop/hadoop-main/2.6.4/hadoop-main-2.6.4.pom
Downloaded from central: https://repo.maven.apache.org/maven2/org/apache/hadoop/hadoop-main/2.6.4/hadoop-main-2.6.4.pom (18 kB at 47 kB/s)
Downloading from central: https://repo.maven.apache.org/maven2/org/apache/hadoop/hadoop-annotations/2.6.4/hadoop-annotations-2.6.4.pom
Downloaded from central: https://repo.maven.apache.org/maven2/org/apache/hadoop/hadoop-annotations/2.6.4/hadoop-annotations-2.6.4.pom (2.4 kB at 8.0 kB/s)
Downloading from central: https://repo.maven.apache.org/maven2/org/apache/hadoop/hadoop-auth/2.6.4/hadoop-auth-2.6.4.pom
Downloaded from central: https://repo.maven.apache.org/maven2/org/apache/hadoop/hadoop-auth/2.6.4/hadoop-auth-2.6.4.pom (7.7 kB at 20 kB/s)
Downloading from central: https://repo.maven.apache.org/maven2/org/apache/hive/hive-storage-api/2.2.1/hive-storage-api-2.2.1.pom
Downloaded from central: https://repo.maven.apache.org/maven2/org/apache/hive/hive-storage-api/2.2.1/hive-storage-api-2.2.1.pom (5.4 kB at 12 kB/s)
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary for Hive 3.1.3:
[INFO] 
[INFO] Hive Upgrade Acid .................................. FAILURE [02:30 min]
[INFO] Hive ............................................... SKIPPED
[INFO] Hive Classifications ............................... SKIPPED
[INFO] Hive Shims Common .................................. SKIPPED
[INFO] Hive Shims 0.23 .................................... SKIPPED
[INFO] Hive Shims Scheduler ............................... SKIPPED
[INFO] Hive Shims ......................................... SKIPPED
[INFO] Hive Common ........................................ SKIPPED
[INFO] Hive Service RPC ................................... SKIPPED
[INFO] Hive Serde ......................................... SKIPPED
[INFO] Hive Standalone Metastore .......................... SKIPPED
[INFO] Hive Metastore ..................................... SKIPPED
[INFO] Hive Vector-Code-Gen Utilities ..................... SKIPPED
[INFO] Hive Llap Common ................................... SKIPPED
[INFO] Hive Llap Client ................................... SKIPPED
[INFO] Hive Llap Tez ...................................... SKIPPED
[INFO] Hive Spark Remote Client ........................... SKIPPED
[INFO] Hive Query Language ................................ SKIPPED
[INFO] Hive Llap Server ................................... SKIPPED
[INFO] Hive Service ....................................... SKIPPED
[INFO] Hive Accumulo Handler .............................. SKIPPED
[INFO] Hive JDBC .......................................... SKIPPED
[INFO] Hive Beeline ....................................... SKIPPED
[INFO] Hive CLI ........................................... SKIPPED
[INFO] Hive Contrib ....................................... SKIPPED
[INFO] Hive Druid Handler ................................. SKIPPED
[INFO] Hive HBase Handler ................................. SKIPPED
[INFO] Hive JDBC Handler .................................. SKIPPED
[INFO] Hive HCatalog ...................................... SKIPPED
[INFO] Hive HCatalog Core ................................. SKIPPED
[INFO] Hive HCatalog Pig Adapter .......................... SKIPPED
[INFO] Hive HCatalog Server Extensions .................... SKIPPED
[INFO] Hive HCatalog Webhcat Java Client .................. SKIPPED
[INFO] Hive HCatalog Webhcat .............................. SKIPPED
[INFO] Hive HCatalog Streaming ............................ SKIPPED
[INFO] Hive HPL/SQL ....................................... SKIPPED
[INFO] Hive Streaming ..................................... SKIPPED
[INFO] Hive Llap External Client .......................... SKIPPED
[INFO] Hive Shims Aggregator .....................