spark 写入mysql 中文数据 显示?? 或者 乱码

目录

前言

Spark报错:

解决办法:

总结一下:

报错:

解决:


前言

用spark写入mysql中,查看中文数据 显示?? 或者 乱码

Spark报错:

Sat Nov 23 19:15:59 CST 2024 WARN: Establishing SSL connection without server's identity verification is not recommended. According to MySQL 5.5.45+, 5.6.26+ and 5.7.6+ requirements SSL connection must be established by default if explicit option isn't set. For compliance with existing applications not using SSL the verifyServerCertificate property is set to 'false'. You need either to explicitly disable SSL by setting useSSL=false, or set useSSL=true and provide truststore for server certificate verification.
Sat Nov 23 19:16:02 CST 2024 WARN: Establishing SSL connection without server's identity verification is not recommended. According to MySQL 5.5.45+, 5.6.26+ and 5.7.6+ requirements SSL connection must be established by default if explicit option isn't set. For compliance with existing applications not using SSL the verifyServerCertificate property is set to 'false'. You need either to explicitly disable SSL by setting useSSL=false, or set useSSL=true and provide truststore for server certificate verification.
Sat Nov 23 19:16:02 CST 2024 WARN: Establishing SSL connection without server's identity verification is not recommended. According to MySQL 5.5.45+, 5.6.26+ and 5.7.6+ requirements SSL connection must be established by default if explicit option isn't set. For compliance with existing applications not using SSL the verifyServerCertificate property is set to 'false'. You need either to explicitly disable SSL by setting useSSL=false, or set useSSL=true and provide truststore for server certificate verification.
2024-11-23 19:16:02,903{yy/MM/dd java.sql.BatchUpdateException: Incorrect string value: '\xE5\xBE\x85\xE6\x9C\xBA' for column 'ChangeRecordState' at row 1at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)at java.lang.reflect.Constructor.newInstance(Constructor.java:423)at com.mysql.jdbc.Util.handleNewInstance(Util.java:425)at com.mysql.jdbc.Util.getInstance(Util.java:408)at com.mysql.jdbc.SQLError.createBatchUpdateException(SQLError.java:1163)at com.mysql.jdbc.PreparedStatement.executeBatchSerially(PreparedStatement.java:1823)at com.mysql.jdbc.PreparedStatement.executeBatchInternal(PreparedStatement.java:1307)at com.mysql.jdbc.StatementImpl.executeBatch(StatementImpl.java:970)at org.apache.spark.sql.execution.datasources.jdbc.JdbcUtils$.savePartition(JdbcUtils.scala:692)at org.apache.spark.sql.execution.datasources.jdbc.JdbcUtils$.$anonfun$saveTable$1(JdbcUtils.scala:856)at org.apache.spark.sql.execution.datasources.jdbc.JdbcUtils$.$anonfun$saveTable$1$adapted(JdbcUtils.scala:854)at org.apache.spark.rdd.RDD.$anonfun$foreachPartition$2(RDD.scala:1020)at org.apache.spark.rdd.RDD.$anonfun$foreachPartition$2$adapted(RDD.scala:1020)at org.apache.spark.SparkContext.$anonfun$runJob$5(SparkContext.scala:2242)at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:90)at org.apache.spark.scheduler.Task.run(Task.scala:131)at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$3(Executor.scala:497)at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1439)at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:500)at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)at java.lang.Thread.run(Thread.java:745)
Caused by: java.sql.SQLException: Incorrect string value: '\xE5\xBE\x85\xE6\x9C\xBA' for column 'ChangeRecordState' at row 1at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:965)at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3978)at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3914)at com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:2530)at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:2683)at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2495)at com.mysql.jdbc.PreparedStatement.executeInternal(PreparedStatement.java:1903)at com.mysql.jdbc.PreparedStatement.executeUpdateInternal(PreparedStatement.java:2124)at com.mysql.jdbc.PreparedStatement.executeBatchSerially(PreparedStatement.java:1801)... 16 more
2024-11-23 19:16:02,921{yy/MM/dd Sat Nov 23 19:16:02 CST 2024 WARN: Establishing SSL connection without server's identity verification is not recommended. According to MySQL 5.5.45+, 5.6.26+ and 5.7.6+ requirements SSL connection must be established by default if explicit option isn't set. For compliance with existing applications not using SSL the verifyServerCertificate property is set to 'false'. You need either to explicitly disable SSL by setting useSSL=false, or set useSSL=true and provide truststore for server certificate verification.
Exception in thread "main" org.apache.spark.SparkException: Job aborted due to stage failure: Task 76 in stage 12.0 failed 1 times, most recent failure: Lost task 76.0 in stage 12.0 (TID 242) (LAPTOP-P70BBG66 executor driver): java.sql.BatchUpdateException: Incorrect string value: '\xE5\xBE\x85\xE6\x9C\xBA' for column 'ChangeRecordState' at row 1at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)at java.lang.reflect.Constructor.newInstance(Constructor.java:423)at com.mysql.jdbc.Util.handleNewInstance(Util.java:425)at com.mysql.jdbc.Util.getInstance(Util.java:408)at com.mysql.jdbc.SQLError.createBatchUpdateException(SQLError.java:1163)at com.mysql.jdbc.PreparedStatement.executeBatchSerially(PreparedStatement.java:1823)at com.mysql.jdbc.PreparedStatement.executeBatchInternal(PreparedStatement.java:1307)at com.mysql.jdbc.StatementImpl.executeBatch(StatementImpl.java:970)at org.apache.spark.sql.execution.datasources.jdbc.JdbcUtils$.savePartition(JdbcUtils.scala:692)at org.apache.spark.sql.execution.datasources.jdbc.JdbcUtils$.$anonfun$saveTable$1(JdbcUtils.scala:856)at org.apache.spark.sql.execution.datasources.jdbc.JdbcUtils$.$anonfun$saveTable$1$adapted(JdbcUtils.scala:854)at org.apache.spark.rdd.RDD.$anonfun$foreachPartition$2(RDD.scala:1020)at org.apache.spark.rdd.RDD.$anonfun$foreachPartition$2$adapted(RDD.scala:1020)at org.apache.spark.SparkContext.$anonfun$runJob$5(SparkContext.scala:2242)at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:90)at org.apache.spark.scheduler.Task.run(Task.scala:131)at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$3(Executor.scala:497)at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1439)at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:500)at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)at java.lang.Thread.run(Thread.java:745)
Caused by: java.sql.SQLException: Incorrect string value: '\xE5\xBE\x85\xE6\x9C\xBA' for column 'ChangeRecordState' at row 1at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:965)at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3978)at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3914)at com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:2530)at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:2683)at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2495)at com.mysql.jdbc.PreparedStatement.executeInternal(PreparedStatement.java:1903)at com.mysql.jdbc.PreparedStatement.executeUpdateInternal(PreparedStatement.java:2124)at com.mysql.jdbc.PreparedStatement.executeBatchSerially(PreparedStatement.java:1801)... 16 moreDriver stacktrace:at org.apache.spark.scheduler.DAGScheduler.failJobAndIndependentStages(DAGScheduler.scala:2253)at org.apache.spark.scheduler.DAGScheduler.$anonfun$abortStage$2(DAGScheduler.scala:2202)at org.apache.spark.scheduler.DAGScheduler.$anonfun$abortStage$2$adapted(DAGScheduler.scala:2201)at scala.collection.mutable.ResizableArray.foreach(ResizableArray.scala:62)at scala.collection.mutable.ResizableArray.foreach$(ResizableArray.scala:55)at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:49)at org.apache.spark.scheduler.DAGScheduler.abortStage(DAGScheduler.scala:2201)at org.apache.spark.scheduler.DAGScheduler.$anonfun$handleTaskSetFailed$1(DAGScheduler.scala:1078)at org.apache.spark.scheduler.DAGScheduler.$anonfun$handleTaskSetFailed$1$adapted(DAGScheduler.scala:1078)at scala.Option.foreach(Option.scala:274)at org.apache.spark.scheduler.DAGScheduler.handleTaskSetFailed(DAGScheduler.scala:1078)at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.doOnReceive(DAGScheduler.scala:2440)at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:2382)at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:2371)at org.apache.spark.util.EventLoop$$anon$1.run(EventLoop.scala:49)at org.apache.spark.scheduler.DAGScheduler.runJob(DAGScheduler.scala:868)at org.apache.spark.SparkContext.runJob(SparkContext.scala:2202)at org.apache.spark.SparkContext.runJob(SparkContext.scala:2223)at org.apache.spark.SparkContext.runJob(SparkContext.scala:2242)at org.apache.spark.SparkContext.runJob(SparkContext.scala:2267)at org.apache.spark.rdd.RDD.$anonfun$foreachPartition$1(RDD.scala:1020)at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151)at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:112)at org.apache.spark.rdd.RDD.withScope(RDD.scala:414)at org.apache.spark.rdd.RDD.foreachPartition(RDD.scala:1018)at org.apache.spark.sql.execution.datasources.jdbc.JdbcUtils$.saveTable(JdbcUtils.scala:854)at org.apache.spark.sql.execution.datasources.jdbc.JdbcRelationProvider.createRelation(JdbcRelationProvider.scala:63)at org.apache.spark.sql.execution.datasources.SaveIntoDataSourceCommand.run(SaveIntoDataSourceCommand.scala:46)at org.apache.spark.sql.execution.command.ExecutedCommandExec.sideEffectResult$lzycompute(commands.scala:70)at org.apache.spark.sql.execution.command.ExecutedCommandExec.sideEffectResult(commands.scala:68)at org.apache.spark.sql.execution.command.ExecutedCommandExec.doExecute(commands.scala:90)at org.apache.spark.sql.execution.SparkPlan.$anonfun$execute$1(SparkPlan.scala:180)at org.apache.spark.sql.execution.SparkPlan.$anonfun$executeQuery$1(SparkPlan.scala:218)at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151)at org.apache.spark.sql.execution.SparkPlan.executeQuery(SparkPlan.scala:215)at org.apache.spark.sql.execution.SparkPlan.execute(SparkPlan.scala:176)at org.apache.spark.sql.execution.QueryExecution.toRdd$lzycompute(QueryExecution.scala:132)at org.apache.spark.sql.execution.QueryExecution.toRdd(QueryExecution.scala:131)at org.apache.spark.sql.DataFrameWriter.$anonfun$runCommand$1(DataFrameWriter.scala:989)at org.apache.spark.sql.execution.SQLExecution$.$anonfun$withNewExecutionId$5(SQLExecution.scala:103)at org.apache.spark.sql.execution.SQLExecution$.withSQLConfPropagated(SQLExecution.scala:163)at org.apache.spark.sql.execution.SQLExecution$.$anonfun$withNewExecutionId$1(SQLExecution.scala:90)at org.apache.spark.sql.SparkSession.withActive(SparkSession.scala:772)at org.apache.spark.sql.execution.SQLExecution$.withNewExecutionId(SQLExecution.scala:64)at org.apache.spark.sql.DataFrameWriter.runCommand(DataFrameWriter.scala:989)at org.apache.spark.sql.DataFrameWriter.saveToV1Source(DataFrameWriter.scala:438)at org.apache.spark.sql.DataFrameWriter.saveInternal(DataFrameWriter.scala:415)at org.apache.spark.sql.DataFrameWriter.save(DataFrameWriter.scala:301)at org.apache.spark.sql.DataFrameWriter.jdbc(DataFrameWriter.scala:817)at com._02.gy.GZ.zhibiao.Test01$.main(Test01.scala:61)at com._02.gy.GZ.zhibiao.Test01.main(Test01.scala)
Caused by: java.sql.BatchUpdateException: Incorrect string value: '\xE5\xBE\x85\xE6\x9C\xBA' for column 'ChangeRecordState' at row 1at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)at java.lang.reflect.Constructor.newInstance(Constructor.java:423)at com.mysql.jdbc.Util.handleNewInstance(Util.java:425)at com.mysql.jdbc.Util.getInstance(Util.java:408)at com.mysql.jdbc.SQLError.createBatchUpdateException(SQLError.java:1163)at com.mysql.jdbc.PreparedStatement.executeBatchSerially(PreparedStatement.java:1823)at com.mysql.jdbc.PreparedStatement.executeBatchInternal(PreparedStatement.java:1307)at com.mysql.jdbc.StatementImpl.executeBatch(StatementImpl.java:970)at org.apache.spark.sql.execution.datasources.jdbc.JdbcUtils$.savePartition(JdbcUtils.scala:692)at org.apache.spark.sql.execution.datasources.jdbc.JdbcUtils$.$anonfun$saveTable$1(JdbcUtils.scala:856)at org.apache.spark.sql.execution.datasources.jdbc.JdbcUtils$.$anonfun$saveTable$1$adapted(JdbcUtils.scala:854)at org.apache.spark.rdd.RDD.$anonfun$foreachPartition$2(RDD.scala:1020)at org.apache.spark.rdd.RDD.$anonfun$foreachPartition$2$adapted(RDD.scala:1020)at org.apache.spark.SparkContext.$anonfun$runJob$5(SparkContext.scala:2242)at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:90)at org.apache.spark.scheduler.Task.run(Task.scala:131)at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$3(Executor.scala:497)at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1439)at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:500)at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)at java.lang.Thread.run(Thread.java:745)
Caused by: java.sql.SQLException: Incorrect string value: '\xE5\xBE\x85\xE6\x9C\xBA' for column 'ChangeRecordState' at row 1at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:965)at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3978)at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3914)at com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:2530)at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:2683)at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2495)at com.mysql.jdbc.PreparedStatement.executeInternal(PreparedStatement.java:1903)at com.mysql.jdbc.PreparedStatement.executeUpdateInternal(PreparedStatement.java:2124)at com.mysql.jdbc.PreparedStatement.executeBatchSerially(PreparedStatement.java:1801)... 16 more
Sat Nov 23 19:16:03 CST 2024 WARN: Establishing SSL connection without server's identity verification is not recommended. According to MySQL 5.5.45+, 5.6.26+ and 5.7.6+ requirements SSL connection must be established by default if explicit option isn't set. For compliance with existing applications not using SSL the verifyServerCertificate property is set to 'false'. You need either to explicitly disable SSL by setting useSSL=false, or set useSSL=true and provide truststore for server certificate verification.
Sat Nov 23 19:16:03 CST 2024 WARN: Establishing SSL connection without server's identity verification is not recommended. According to MySQL 5.5.45+, 5.6.26+ and 5.7.6+ requirements SSL connection must be established by default if explicit option isn't set. For compliance with existing applications not using SSL the verifyServerCertificate property is set to 'false'. You need either to explicitly disable SSL by setting useSSL=false, or set useSSL=true and provide truststore for server certificate verification.
Sat Nov 23 19:16:03 CST 2024 WARN: Establishing SSL connection without server's identity verification is not recommended. According to MySQL 5.5.45+, 5.6.26+ and 5.7.6+ requirements SSL connection must be established by default if explicit option isn't set. For compliance with existing applications not using SSL the verifyServerCertificate property is set to 'false'. You need either to explicitly disable SSL by setting useSSL=false, or set useSSL=true and provide truststore for server certificate verification.
Sat Nov 23 19:16:03 CST 2024 WARN: Establishing SSL connection without server's identity verification is not recommended. According to MySQL 5.5.45+, 5.6.26+ and 5.7.6+ requirements SSL connection must be established by default if explicit option isn't set. For compliance with existing applications not using SSL the verifyServerCertificate property is set to 'false'. You need either to explicitly disable SSL by setting useSSL=false, or set useSSL=true and provide truststore for server certificate verification.
2024-11-23 19:16:03,282{yy/MM/dd java.lang.NullPointerExceptionat org.apache.spark.scheduler.Task.$anonfun$run$2(Task.scala:152)at org.apache.spark.util.Utils$.tryLogNonFatalError(Utils.scala:1419)at org.apache.spark.scheduler.Task.run(Task.scala:150)at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$3(Executor.scala:497)at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1439)at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:500)at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)at java.lang.Thread.run(Thread.java:745)
Sat Nov 23 19:16:03 CST 2024 WARN: Establishing SSL connection without server's identity verification is not recommended. According to MySQL 5.5.45+, 5.6.26+ and 5.7.6+ requirements SSL connection must be established by default if explicit option isn't set. For compliance with existing applications not using SSL the verifyServerCertificate property is set to 'false'. You need either to explicitly disable SSL by setting useSSL=false, or set useSSL=true and provide truststore for server certificate verification.
2024-11-23 19:16:03,330{yy/MM/dd java.lang.NullPointerExceptionat org.apache.spark.scheduler.Task.$anonfun$run$2(Task.scala:152)at org.apache.spark.util.Utils$.tryLogNonFatalError(Utils.scala:1419)at org.apache.spark.scheduler.Task.run(Task.scala:150)at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$3(Executor.scala:497)at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1439)at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:500)at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)at java.lang.Thread.run(Thread.java:745)
Sat Nov 23 19:16:03 CST 2024 WARN: Establishing SSL connection without server's identity verification is not recommended. According to MySQL 5.5.45+, 5.6.26+ and 5.7.6+ requirements SSL connection must be established by default if explicit option isn't set. For compliance with existing applications not using SSL the verifyServerCertificate property is set to 'false'. You need either to explicitly disable SSL by setting useSSL=false, or set useSSL=true and provide truststore for server certificate verification.
2024-11-23 19:16:03,386{yy/MM/dd java.lang.NullPointerExceptionat org.apache.spark.scheduler.Task.$anonfun$run$2(Task.scala:152)at org.apache.spark.util.Utils$.tryLogNonFatalError(Utils.scala:1419)at org.apache.spark.scheduler.Task.run(Task.scala:150)at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$3(Executor.scala:497)at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1439)at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:500)at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)at java.lang.Thread.run(Thread.java:745)Process finished with exit code 1

解决办法:

我试了网上的很多办法:在saprk里面更改编码,直接把这一列更改为UTF-8

.withColumn("ChangeRecordState", decode($"ChangeRecordState", "UTF-8"))

使用新的driver,然后更改maven里面的jdbc链接版本

properties.setProperty("driver", "com.mysql.cj.jdbc.Driver")

等等....都没有解决问题

所以我开始着手自己解决,问题的核心无非就是编码冲突问题,因为已经写入进去了,所以连接配置啥的都没问题,主要解决编码问题

spark中可以打印出中文字段所以spark基本没有什么问题,我还是用了老方法

mysqlURL的配置项上加了?useUnicode=true&characterEncoding=UTF-8

val mysqlUrl = "jdbc:mysql://bigdata1:3306/shtd_industry?useUnicode=true&characterEncoding=UTF-8"

主要解决的是mysql,所以我开始查询mysql的默认编码是什么,我找到了命令

SHOW TABLE STATUS LIKE 'machine_state_time';

找到Collation字段对应的就是默认的编码  --  latin1_swedish_ci

找到了默认编码不是UTF-8,所以这里问题很明显了,就是把 latin1_swedish_ci 变成 UTF-8

所以我又接着查询,把表转换编码的命令

alter table machine_state_time character set utf8;

结果还是不行,当我试着找出问题,才发现我只把表改成了UTF-8,没有把字段改成UTF-8

alter table machine_state_time convert to character set utf8;

这样mysql的问题也解决了,运行一下还是报错

找到问题,scala语言也需要设置编码方式,所以我又加了一条配置项

System.setProperty("file.encoding", "UTF-8")

这样就可以了,运行成功,然后查看一下mysql中的内容

总结一下:

报错:

主要原因是因为编码冲突,scala,spark,mysql-connect连接,mysql三个编码不一致

解决:

设置scala的编码:

System.setProperty("file.encoding", "UTF-8")

设置spark的编码:

saprk本身就是UTF-8 不用设置

设置mysqlURL的编码:

val mysqlUrl = "jdbc:mysql://bigdata1:3306/shtd_industry?useUnicode=true&characterEncoding=UTF-8"

设置mysql本身的编码:

用两条命令,分别是:

//修改目标表所有字段编码格式为utf8
alter table machine_state_time convert to character set utf8;
//修改目标表表编码格式为utf8
alter table machine_state_time character set utf8;

好的,今天的分享就到这里了,希望对你有帮助,有帮助可以点点赞,评论评论

本文来自互联网用户投稿,该文观点仅代表作者本人,不代表本站立场。本站仅提供信息存储空间服务,不拥有所有权,不承担相关法律责任。如若转载,请注明出处:http://www.mzph.cn/bicheng/61762.shtml

如若内容造成侵权/违法违规/事实不符,请联系多彩编程网进行投诉反馈email:809451989@qq.com,一经查实,立即删除!

相关文章

电子应用设计方案-20:智能电冰箱系统方案设计

智能电冰箱系统方案设计 一、系统概述 本智能电冰箱系统旨在提供更便捷、高效、智能化的食品存储和管理解决方案,通过集成多种传感器、智能控制技术和联网功能,实现对冰箱内部环境的精确监测和控制,以及与用户的互动和远程管理。 二、系统组成…

实验四:构建园区网(OSPF 动态路由)

目录 一、实验简介 二、实验目的 三、实验需求 四、实验拓扑 五、实验步骤 1、在 eNSP 中部署网络 2、设计全网 IP 地址 3、配置二层交换机 4、配置路由交换机并测试通信 5、配置路由接口地址 6、配置 OSPF 动态路由,实现全网互通 一、实验简介 使用路由…

【大数据学习 | Spark-Core】yarn-client与yarn-cluster的区别

1. yarn的提交命令 # yarn的提交命令参数 --master yarn #执行集群 --deploy-mode # 部署模式 --class #指定运行的类 --executor-memory #指定executor的内存 --executor-cores # 指定核数 --num-executors # 直接指定executor的数量 --queue # 指定队列 2. yarn-client模式…

WEB攻防-通用漏洞文件上传二次渲染.htaccess变异免杀

知识点: 1、文件上传-二次渲染 2、文件上传-简单免杀变异 3、文件上传-.htaccess妙用 4、文件上传-PHP语言特性 1、上传后门时,文件内容带.就不行 这时可以上传一个转换后的ip地址,ip地址对应网站包含后门代码 转换后的int会在访问的时候…

【汽车制动】汽车制动相关控制系统

目录 1.ABS (Anti-lock Brake System,防抱死制动系统) 2.EBD(Electronic Brake-force Distribution,电子制动力分配系统) 3.TCS(Traction Control System,牵引力控制系统) 4.VDC&#xff08…

Tri Mode Ethernet MAC IP核详解

本文对 Vivado 的三速 MAC IP 核(Tri Mode Ethernet MAC,TEMAC)进行介绍。 在自行实现三速以太网 MAC 控制器时,GMII/RGMII 接口可以通过 IDDR、ODDR 原语实现,然而实际使用中自己实现的模块性能不是很稳定&#xff08…

项目实战:基于Vue3实现一个小相册

相册的示例效果图 注意看注释... CSS部分&#xff1a; <style>/* 伪元素选择器&#xff0c;用于在具有clear_ele类的元素内部的末尾添加一个新的元素 */.clear_ele::after{ content: ""; /* 这个伪元素的内容属性必须有 *//* 块级元素&#xff0c;便于占据一…

HTMLCSS:比赛记分卡

效果演示 这段 HTML 和 CSS 代码创建了一个卡片式的体育比赛信息展示组件&#xff0c;用于显示篮球比赛的两个队伍名称、比赛时间、比分以及一些装饰性的视觉元素。 HTML <div class"card"><div data-status"inprogress" class"teams"…

Paper -- 3D建筑模型生成 -- GABLE: 基于高分辨率卫星影像的全国尺度精细3D建筑模型

基本信息 论文题目: GABLE: A first fine-grained 3D building model of China on a national scale from very high resolution satellite imagery 中文题目: GABLE: 基于高分辨率卫星影像的全国尺度精细3D建筑模型 作者及单位: – 多数作者来自中国科学院空间信息研究所&…

蓝桥杯嵌入式再学习(2)基础框架的构建

1&#xff1a;打开工程以后我们需要进行一些配置如图&#xff1a; 第一步&#xff1a;将core里面的对勾取消掉 勾选Microlib 优化等级选择level 0 将我们自己的文件夹添加到路径里面去 这个样子我们就基本将框架搭建好了我们然后需要编写各个底层的驱动了

矩阵/矩阵乘法/特征征/特征向量的讲解

线性代数里有很多的概念&#xff0c;很多概念是有几何意义的&#xff0c;了解了几何意义可能会更好的理解各种概念及其相互之间的关系。 矩阵&#xff1a; 矩阵是一个变换&#xff0c;一个坐标系到另一个坐标第的变换。矩阵里的各个参数&#xff0c;代表了如何进行变换。 矩阵…

Easyexcel(5-自定义列宽)

相关文章链接 Easyexcel&#xff08;1-注解使用&#xff09;Easyexcel&#xff08;2-文件读取&#xff09;Easyexcel&#xff08;3-文件导出&#xff09;Easyexcel&#xff08;4-模板文件&#xff09;Easyexcel&#xff08;5-自定义列宽&#xff09; 注解 ColumnWidth Data…

1+X应急响应(网络)文件包含漏洞:

常见网络攻击-文件包含漏洞&命令执行漏洞&#xff1a; 文件包含漏洞简介&#xff1a; 分析漏洞产生的原因&#xff1a; 四个函数&#xff1a; 产生漏洞的原因&#xff1a; 漏洞利用条件&#xff1a; 文件包含&#xff1a; 漏洞分类&#xff1a; 本地文件包含&#xff1a; …

Flutter:SlideTransition位移动画,Interval动画延迟

配置vsync&#xff0c;需要实现一下with SingleTickerProviderStateMixinclass _MyHomePageState extends State<MyHomePage> with SingleTickerProviderStateMixin{// 定义 AnimationControllerlate AnimationController _controller;overridevoid initState() {super.…

【Android】android compat理解

1&#xff0c;前提 即便是在同一手机上安装的不同apk&#xff0c;其编译的apk不同&#xff0c;也会导致行为上的差异。如SDK34有限制后台启动&#xff0c;但如果安装的apk所依赖的sdk是33&#xff0c;则不会表现出此差异。这是如何实现的呢&#xff1f;其实&#xff0c;本质是…

电脑自动关机时间如何定?Wise Auto Shutdown 设置关机教程

在日常使用电脑的过程中&#xff0c;有时我们需要让电脑在特定的时间自动关机&#xff0c;比如在下载大文件完成后、执行长时间的任务结束时&#xff0c;或者只是单纯想在某个预定时间让电脑自动关闭以节省能源。这时候&#xff0c;Wise Auto Shutdown 这款软件就能派上大用场了…

算法专题十一: 基础递归

目录 1. 汉诺塔2. 合并两个有序链表3. 反转链表4. 两两交换链表中的节点5. Pow(x, n) 1. 汉诺塔 题目链接&#xff1a; Leetcode汉诺塔 算法原理&#xff1a; 递归&#xff1a;宏观视角理解递归 本道题为什么能用递归&#xff1f; 让我们逐一分析 首先思考我们如何来解决汉诺…

Cmakelist.txt之win-c-udp-client

1.cmakelist.txt cmake_minimum_required(VERSION 3.16) ​ project(c_udp_client LANGUAGES C) ​ add_executable(c_udp_client main.c) ​ target_link_libraries(c_udp_client wsock32) ​ ​ include(GNUInstallDirs) install(TARGETS c_udp_clientLIBRARY DESTINATION $…

Git错误:gnutls_handshake() failed: The TLS connection was non-properly terminated

最终我通过这个博客解决了问题&#xff1a;解决Git错误&#xff1a;gnutls_handshake() failed: The TLS connection was non-properly terminated 解决方案 1. 检查网络连接 首先&#xff0c;确保你的网络连接是稳定的。尝试访问其他HTTPS网站或服务&#xff0c;以排除网络问…

Lua如何连接MySQL数据库?

大家好&#xff0c;我是袁庭新。使用Lua语言如何来连接数据库呢&#xff1f;新哥这篇文章给你安排上。 1 LuaSQL概述 LuaSQL是一个轻量级的Lua到数据库管理系统&#xff08;DBMS&#xff09;的接口库&#xff0c;由Kepler Project维护&#xff0c;且是开源的。它提供了一个简…