任务描述
1、了解搭建民航后端框架
2、使用IDEA创建基于SpringBoot、MyBatis、MySQL、Redis的Java项目
3、以原项目为参照搭建项目所涉及到的各个业务和底层服务
4、以原项目为例,具体介绍各个目录情况并参照创建相关文件夹
任务指导
1、讲框架的选择和原理
2、使用IDEA创建基于SpringBoot、MyBatis-Plus、MySQL、Redis的Java项目
- 创建项目后端:BigData-KongGuan
- 创建项目数据清洗端:BigData-Etl-KongGuan
3、以原项目为例,具体介绍各个目录情况并参照创建相关文件夹
- BigData-KongGuan项目(SpringBoot项目)
- BigData-Etl-KongGuan项目(SpringBoot项目)
任务实现
1、创建项目后端 BigData-KongGuan
BigData-KongGuan项目是当前民航项目的后端程序,负责访问数据为前端提供查询接口,同时提供一些定时任务,例如,将HBase数据查询出来,并推送到Kafka当中,来模拟航空数据的收集过程等。
- 创建一个基于SpringBoot+MyBatis-Plus的项目:BigData-KongGuan
- 配置Maven插件,点击菜单“File”-> “Settings”,然后在打开的窗口按如下图所示配置Maven
- 在com.qrsoft包上点击右键,创建Package包
- 在项目中需要创建如下图所示的包目录结构和配置文件
- 项目的包目录结构和配置文件的说明如下
目录 | 子目录/文件 | 目录/文件(代码)的说明 |
---|---|---|
src/main/java/com/qrsoft/ | common/ | 存放一些通用类 |
src/main/java/com/qrsoft/ | config/ | 存放SpringBoot的Configuration配置类 |
src/main/java/com/qrsoft/ | controller/ | 存放Controller控制器类 |
src/main/java/com/qrsoft/ | entity/ | 存放实体类 |
src/main/java/com/qrsoft/ | filter/ | 存放过滤器类 |
src/main/java/com/qrsoft/ | mapper/ | 存放MyBatis数据访问类 |
src/main/java/com/qrsoft/ | service/ | 存放Service接口类 |
src/main/java/com/qrsoft/ | util/ | 存放工具类 |
src/main/java/com/qrsoft/ | ApplicationStarter.java(BigDataKongGuanApplication.java) | SpringBoot的启动程序 |
src/main/resources/ | application.yml(application.properties) | SpringBoot的配置文件 |
src/main/resources/ | hbase.properties | HBase的配置文件 |
src/main/resources/ | kafka.properties | Kafka的配置文件 |
src/main/resources/ | log4j.properties | Log4j日志的配置文件 |
- 在项目的pom.xml中添加依赖,最终pom.xml文件如下
<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0"xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd"><modelVersion>4.0.0</modelVersion><groupId>com.qrsoft</groupId><artifactId>BigData-KongGuan</artifactId><version>1.0</version><parent><groupId>org.springframework.boot</groupId><version>2.4.5</version><artifactId>spring-boot-starter-parent</artifactId></parent><dependencies><dependency><groupId>org.springframework.boot</groupId><artifactId>spring-boot-starter-web</artifactId></dependency><dependency><groupId>com.baomidou</groupId><artifactId>mybatis-plus-boot-starter</artifactId><version>3.4.1</version></dependency><dependency><groupId>mysql</groupId><artifactId>mysql-connector-java</artifactId></dependency><!-- JWT --><dependency><groupId>io.jsonwebtoken</groupId><artifactId>jjwt-impl</artifactId><version>0.11.2</version><scope>runtime</scope></dependency><dependency><groupId>io.jsonwebtoken</groupId><artifactId>jjwt-jackson</artifactId><version>0.11.2</version><scope>runtime</scope></dependency><dependency><groupId>io.jsonwebtoken</groupId><artifactId>jjwt-api</artifactId><version>0.11.2</version></dependency><!-- Spring Boot Security --><dependency><groupId>org.springframework.boot</groupId><artifactId>spring-boot-starter-security</artifactId></dependency><!-- spring-redis --><dependency><groupId>org.springframework.boot</groupId><artifactId>spring-boot-starter-data-redis</artifactId><exclusions><exclusion><groupId>io.lettuce</groupId><artifactId>lettuce-core</artifactId></exclusion></exclusions></dependency><dependency><groupId>redis.clients</groupId><artifactId>jedis</artifactId></dependency><dependency><groupId>org.projectlombok</groupId><artifactId>lombok</artifactId></dependency><dependency><groupId>com.github.xiaoymin</groupId><artifactId>swagger-bootstrap-ui</artifactId><version>1.9.6</version></dependency><dependency><groupId>io.springfox</groupId><artifactId>springfox-swagger2</artifactId><version>2.9.2</version></dependency><dependency><groupId>io.springfox</groupId><artifactId>springfox-swagger-ui</artifactId><version>2.9.2</version></dependency><dependency><groupId>log4j</groupId><artifactId>log4j</artifactId><version>1.2.17</version></dependency><dependency><groupId>org.apache.logging.log4j</groupId><artifactId>log4j-slf4j-impl</artifactId><version>2.9.1</version></dependency><dependency><groupId>org.apache.kafka</groupId><artifactId>kafka_2.10</artifactId><version>0.10.0.0</version><exclusions><exclusion><artifactId>slf4j-log4j12</artifactId><groupId>org.slf4j</groupId></exclusion><exclusion><artifactId>log4j</artifactId><groupId>log4j</groupId></exclusion><exclusion><artifactId>slf4j-api</artifactId><groupId>org.slf4j</groupId></exclusion><exclusion><artifactId>log4j-slf4j-impl</artifactId><groupId>org.apache.logging.log4j</groupId></exclusion><exclusion><artifactId>kafka-clients</artifactId><groupId>org.apache.kafka</groupId></exclusion></exclusions></dependency><dependency><groupId>org.apache.kafka</groupId><artifactId>kafka-clients</artifactId><version>0.10.0.1</version></dependency><dependency><groupId>org.apache.hbase</groupId><artifactId>hbase-client</artifactId><version>1.3.1</version><exclusions><exclusion><artifactId>log4j</artifactId><groupId>log4j</groupId></exclusion><exclusion><artifactId>slf4j-log4j12</artifactId><groupId>org.slf4j</groupId></exclusion><exclusion><artifactId>slf4j-api</artifactId><groupId>org.slf4j</groupId></exclusion><exclusion><artifactId>log4j-slf4j-impl</artifactId><groupId>org.apache.logging.log4j</groupId></exclusion><exclusion><groupId>com.google.guava</groupId><artifactId>guava</artifactId></exclusion></exclusions></dependency><dependency><groupId>com.alibaba</groupId><artifactId>fastjson</artifactId><version>1.2.75</version></dependency><dependency><groupId>com.google.code.gson</groupId><artifactId>gson</artifactId><version>2.8.6</version></dependency><dependency><groupId>cn.hutool</groupId><artifactId>hutool-all</artifactId><version>5.6.3</version></dependency></dependencies><build><plugins><plugin><groupId>org.springframework.boot</groupId><artifactId>spring-boot-maven-plugin</artifactId></plugin></plugins></build>
</project>
- 更新Maven依赖
- 最终项目的目录结构如下所示
2、创建项目数据清洗端 BigData-Etl-KongGuan
BigData-Etl-KongGuan项目是当前民航项目的数据清洗程序,负责数据的ETL等工作。
- 创建一个基于SpringBoot的项目:BigData-Etl-KongGuan
- 配置Maven插件,点击菜单“File”-> “Settings”,然后在打开的窗口按如下图所示配置Maven
- 在com.qrsoft包上点击右键,创建Package包,名为 etl
- 在com.qrsoft.etl包上点击右键,创建如下所示的包目录结构和配置文件
- 项目的目录结构说明
目录 | 子目录/文件 | 目录/文件(代码)的说明 |
---|---|---|
src/main/java/com/qrsoft/etl/ | common/ | 存放一些通用类 |
src/main/java/com/qrsoft/etl/ | common/db | 存放数据库连接管理类 |
src/main/java/com/qrsoft/etl/ | dao/ | 存放数据访问类 |
src/main/java/com/qrsoft/etl/ | dao/entity | 存放实体类 |
src/main/java/com/qrsoft/etl/ | spark/ | 存放Spark数据清洗任务类 |
src/main/java/com/qrsoft/etl/ | Task/ | 存放SpringBoot定时任务类 |
src/main/java/com/qrsoft/etl/ | util/ | 存放工具类 |
src/main/java/com/qrsoft/ | BigDataEtlKongGuanApplication.java | SpringBoot的启动程序 |
src/main/resources/ | application.yml | SpringBoot的配置文件 |
src/main/resources/ | config.properties | mysql的配置文件 |
src/main/resources/ | myconfig.properties | Zookeeper的配置文件 |
src/main/resources/ | log4j.properties | Log4j日志的配置文件 |
- 在项目的pom.xml中添加依赖,最终pom.xml文件如下
<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 https://maven.apache.org/xsd/maven-4.0.0.xsd"><modelVersion>4.0.0</modelVersion><parent><groupId>org.springframework.boot</groupId><artifactId>spring-boot-starter-parent</artifactId><version>2.4.5</version><relativePath/> <!-- lookup parent from repository --></parent><groupId>com.qrsoft</groupId><artifactId>etl</artifactId><version>0.0.1-SNAPSHOT</version><name>BigData-Etl-KongGuan</name><description>BigData-Etl-KongGuan</description><properties><java.version>1.8</java.version><project.build.sourceEncoding>UTF-8</project.build.sourceEncoding><scala-version>2.10</scala-version><spark-version>2.0.2</spark-version></properties><dependencies><dependency><groupId>org.springframework.boot</groupId><artifactId>spring-boot-starter-web-services</artifactId></dependency><dependency><groupId>org.springframework.boot</groupId><artifactId>spring-boot-starter-test</artifactId><scope>test</scope></dependency><!--spark--><dependency><groupId>org.apache.spark</groupId><artifactId>spark-core_${scala-version}</artifactId><version>${spark-version}</version></dependency><dependency><groupId>org.apache.spark</groupId><artifactId>spark-streaming_${scala-version}</artifactId><version>${spark-version}</version></dependency><dependency><groupId>org.apache.spark</groupId><artifactId>spark-streaming-kafka-0-10_${scala-version}</artifactId><version>${spark-version}</version></dependency><!--Spark--><dependency><groupId>io.netty</groupId><artifactId>netty-all</artifactId><version>4.0.29.Final</version></dependency><!--Kafka--><dependency><groupId>org.apache.kafka</groupId><artifactId>kafka_2.10</artifactId><version>0.10.0.0</version><exclusions><exclusion><artifactId>slf4j-log4j12</artifactId><groupId>org.slf4j</groupId></exclusion><exclusion><artifactId>log4j</artifactId><groupId>log4j</groupId></exclusion><exclusion><artifactId>slf4j-api</artifactId><groupId>org.slf4j</groupId></exclusion><exclusion><artifactId>log4j-slf4j-impl</artifactId><groupId>org.apache.logging.log4j</groupId></exclusion><exclusion><artifactId>kafka-clients</artifactId><groupId>org.apache.kafka</groupId></exclusion></exclusions></dependency><dependency><groupId>org.apache.kafka</groupId><artifactId>kafka-clients</artifactId><version>0.10.0.1</version></dependency><!--Kafka--><dependency><groupId>org.apache.hbase</groupId><artifactId>hbase-client</artifactId><version>1.3.1</version><exclusions><exclusion><artifactId>log4j</artifactId><groupId>log4j</groupId></exclusion><exclusion><artifactId>slf4j-log4j12</artifactId><groupId>org.slf4j</groupId></exclusion><exclusion><artifactId>slf4j-api</artifactId><groupId>org.slf4j</groupId></exclusion><exclusion><artifactId>log4j-slf4j-impl</artifactId><groupId>org.apache.logging.log4j</groupId></exclusion><exclusion><groupId>com.google.guava</groupId><artifactId>guava</artifactId></exclusion></exclusions></dependency><dependency><groupId>com.alibaba</groupId><artifactId>fastjson</artifactId><version>1.2.75</version></dependency><dependency><groupId>com.google.code.gson</groupId><artifactId>gson</artifactId><version>2.8.6</version></dependency><dependency><groupId>net.sf.json-lib</groupId><artifactId>json-lib</artifactId><version>2.4</version><classifier>jdk15</classifier></dependency><dependency><groupId>org.projectlombok</groupId><artifactId>lombok</artifactId></dependency><dependency><groupId>com.baomidou</groupId><artifactId>mybatis-plus-boot-starter</artifactId><version>3.4.1</version></dependency><dependency><groupId>mysql</groupId><artifactId>mysql-connector-java</artifactId></dependency></dependencies><build><plugins><plugin><groupId>org.springframework.boot</groupId><artifactId>spring-boot-maven-plugin</artifactId></plugin></plugins></build>
</project>
- 更新Maven依赖
- 最终项目的目录结构如下所示