OB删除1.5亿数据耗费2小时

目录

回顾：mysql是怎么删除数据的？

删除方案

代码实现

执行结果

结论

本篇是实际操作 批量处理数据以及线程池线程数设置 记录学习

背景：有一张用户标签表，存储数据量达4个亿，使用OceanBase存储，由于其中一种标签类型tagType1刷数时逻辑错误，所以需要将该类型数据全部删掉

回顾：mysql是怎么删除数据的？

首先我们得知道mysql是怎么删除数据的？

MySQL 删除操作使用延迟清理的方式。当使用 DELETE 语句删除数据时，MySQL 会将删除的数据标记为已删除，并在需要时将其从磁盘上清理掉。

如果我们要删除id=2这行数据，mysql执行流程如下

所以删除一行数据，需要先查询数据，然后再将数据标记已删除更新

删除方案

批量删除数据，按照以往经验，按照自增ID分页并发删除最快：1. 查询不需要回表；2. 更新可以避免数据库行锁

由于写数据时，是多个标签任务同时写入的，所以需要删掉的tagType1数据并不是连续的，意味着删除语句必须筛选tag_type = "tagType1"

可以查询出tagType1的最小ID和最大ID，然后进行分页查询

代码实现

public static final int endId = 237655461;
public static final int batchSize = 1000;
public static final int startId = 85427001;
public static final int threadNum = 6;@Override
public void tmpDeleteClueTag() {log.info("deleteUserTag start");for (long i = startId; i < endId; i += batchSize * threadNum) {for (int j = 0; j < threadNum; j++) {final long from = i + j * batchSize;final long to = from + batchSize;executorService.submit(() -> {long l = System.currentTimeMillis();int result = clueUserTagGateway.deleteUserTag(from, to);log.info("deleteUserTag,{},{}->{},res:{},time:{}", Thread.currentThread().getName(), from, to,result, System.currentTimeMillis() - l);});}}log.info("deleteUserTag success");
}// 执行语句
DELETE FROM clue_user_tag where tag_type = 'spu' and id >= #{from} and id &lt; #{to}// 线程池定义
public class CustomizeThreadPoolConfig {public static final int CORE_SIZE = 6;public static final int MAX_SIZE = 6;public static final int KEEP_ALIVE_TIME = 5;public static final int QUEUE_SIZE = 200;@Bean(name = "customerTagExecutor", destroyMethod = "shutdown")public ExecutorService CustomerTagExecutor() {return new TraceableThreadPoolExecutor(CORE_SIZE, MAX_SIZE, KEEP_ALIVE_TIME, TimeUnit.MINUTES,new LinkedBlockingQueue<>(QUEUE_SIZE), new CustomizableThreadFactory("pushExecutor-"),new ThreadPoolExecutor.CallerRunsPolicy());}public static class TraceableThreadPoolExecutor extends ThreadPoolExecutor {public TraceableThreadPoolExecutor(int corePoolSize, int maximumPoolSize, long keepAliveTime, TimeUnit unit,BlockingQueue<Runnable> workQueue, ThreadFactory threadFactory, RejectedExecutionHandler handler) {super(corePoolSize, maximumPoolSize, keepAliveTime, unit, workQueue, threadFactory, handler);}}
}
  • 一次处理1000条，将id拆分开扔给线程池执行删除，可以避免数据库表行锁
  • 使用tag_type标签where删除无伤大雅，执行计划会根据id去优先查询的，查询出来后再从1000条中筛选出有tag_type匹配的情况，内存匹配是非常快的
  • 开启6个线程是因为我们数据平台只有6个线程去执行数据，如果线程数太多会造成数据平台侧资源争抢，一次删除请求会变慢

执行结果

刚开始，并不知道数据平台6个线程的限制，所以设置了20个线程

2024-12-16 15:47:46,478|INFO |796ffea717b93262a1e1fb8799631878|pushExecutor-6|c.x.n.x.c.s.impl.CustomerClueServiceImpl|deleteUserTag,threadName=pushExecutor-6,res:0,time:4
2024-12-16 15:47:46,481|INFO |796ffea717b93262a1e1fb8799631878|pushExecutor-12|c.x.n.x.c.s.impl.CustomerClueServiceImpl|deleteUserTag,threadName=pushExecutor-12,res:0,time:5
2024-12-16 15:47:46,481|INFO |796ffea717b93262a1e1fb8799631878|pushExecutor-14|c.x.n.x.c.s.impl.CustomerClueServiceImpl|deleteUserTag,threadName=pushExecutor-14,res:0,time:3
2024-12-16 15:47:46,482|INFO |796ffea717b93262a1e1fb8799631878|pushExecutor-6|c.x.n.x.c.s.impl.CustomerClueServiceImpl|deleteUserTag,threadName=pushExecutor-6,res:0,time:4
2024-12-16 15:47:46,483|INFO |796ffea717b93262a1e1fb8799631878|pushExecutor-12|c.x.n.x.c.s.impl.CustomerClueServiceImpl|deleteUserTag,threadName=pushExecutor-12,res:0,time:2
2024-12-16 15:47:46,573|INFO |796ffea717b93262a1e1fb8799631878|pushExecutor-19|c.x.n.x.c.s.impl.CustomerClueServiceImpl|deleteUserTag,threadName=pushExecutor-19,res:0,time:3098
2024-12-16 15:47:46,575|INFO |796ffea717b93262a1e1fb8799631878|pushExecutor-14|c.x.n.x.c.s.impl.CustomerClueServiceImpl|deleteUserTag,threadName=pushExecutor-14,res:0,time:94
2024-12-16 15:47:46,577|INFO |796ffea717b93262a1e1fb8799631878|pushExecutor-6|c.x.n.x.c.s.impl.CustomerClueServiceImpl|deleteUserTag,threadName=pushExecutor-6,res:0,time:95
2024-12-16 15:47:46,579|INFO |796ffea717b93262a1e1fb8799631878|pushExecutor-15|c.x.n.x.c.s.impl.CustomerClueServiceImpl|deleteUserTag,threadName=pushExecutor-15,res:0,time:3295
2024-12-16 15:47:46,580|INFO |796ffea717b93262a1e1fb8799631878|pushExecutor-19|c.x.n.x.c.s.impl.CustomerClueServiceImpl|deleteUserTag,threadName=pushExecutor-19,res:0,time:7
2024-12-16 15:47:46,581|INFO |796ffea717b93262a1e1fb8799631878|pushExecutor-12|c.x.n.x.c.s.impl.CustomerClueServiceImpl|deleteUserTag,threadName=pushExecutor-12,res:0,time:98
2024-12-16 15:47:46,583|INFO |796ffea717b93262a1e1fb8799631878|pushExecutor-6|c.x.n.x.c.s.impl.CustomerClueServiceImpl|deleteUserTag,threadName=pushExecutor-6,res:0,time:6
2024-12-16 15:47:46,584|INFO |796ffea717b93262a1e1fb8799631878|pushExecutor-15|c.x.n.x.c.s.impl.CustomerClueServiceImpl|deleteUserTag,threadName=pushExecutor-15,res:0,time:5
2024-12-16 15:47:46,584|INFO |796ffea717b93262a1e1fb8799631878||c.x.n.x.c.s.impl.CustomerClueServiceImpl|deleteUserTag,threadName=,res:1000,time:3404
2024-12-16 15:47:46,589|INFO |796ffea717b93262a1e1fb8799631878|pushExecutor-8|c.x.n.x.c.s.impl.CustomerClueServiceImpl|deleteUserTag,threadName=pushExecutor-8,res:1000,time:3008
2024-12-16 15:47:46,672|INFO |796ffea717b93262a1e1fb8799631878|pushExecutor-19|c.x.n.x.c.s.impl.CustomerClueServiceImpl|deleteUserTag,threadName=pushExecutor-19,res:0,time:92
2024-12-16 15:47:46,673|INFO |796ffea717b93262a1e1fb8799631878|pushExecutor-14|c.x.n.x.c.s.impl.CustomerClueServiceImpl|deleteUserTag,threadName=pushExecutor-14,res:0,time:98
2024-12-16 15:47:46,675|INFO |796ffea717b93262a1e1fb8799631878|pushExecutor-12|c.x.n.x.c.s.impl.CustomerClueServiceImpl|deleteUserTag,threadName=pushExecutor-12,res:0,time:94
2024-12-16 15:47:46,675|INFO |796ffea717b93262a1e1fb8799631878|pushExecutor-17|c.x.n.x.c.s.impl.CustomerClueServiceImpl|deleteUserTag,threadName=pushExecutor-17,res:0,time:3390
2024-12-16 15:47:46,676|INFO |796ffea717b93262a1e1fb8799631878|pushExecutor-6|c.x.n.x.c.s.impl.CustomerClueServiceImpl|deleteUserTag,threadName=pushExecutor-6,res:0,time:93
2024-12-16 15:47:46,677|INFO |796ffea717b93262a1e1fb8799631878|pushExecutor-8|c.x.n.x.c.s.impl.CustomerClueServiceImpl|deleteUserTag,threadName=pushExecutor-8,res:0,time:88
2024-12-16 15:47:46,677|INFO |796ffea717b93262a1e1fb8799631878|pushExecutor-14|c.x.n.x.c.s.impl.CustomerClueServiceImpl|deleteUserTag,threadName=pushExecutor-14,res:0,time:4
2024-12-16 15:47:46,679|INFO |796ffea717b93262a1e1fb8799631878|pushExecutor-15|c.x.n.x.c.s.impl.CustomerClueServiceImpl|deleteUserTag,threadName=pushExecutor-15,res:0,time:95
2024-12-16 15:47:46,679|INFO |796ffea717b93262a1e1fb8799631878|pushExecutor-3|c.x.n.x.c.s.impl.CustomerClueServiceImpl|deleteUserTag,threadName=pushExecutor-3,res:0,time:3306
2024-12-16 15:47:46,679|INFO |796ffea717b93262a1e1fb8799631878|pushExecutor-5|c.x.n.x.c.s.impl.CustomerClueServiceImpl|deleteUserTag,threadName=pushExecutor-5,res:0,time:3372
2024-12-16 15:47:46,680|INFO |796ffea717b93262a1e1fb8799631878|pushExecutor-19|c.x.n.x.c.s.impl.CustomerClueServiceImpl|deleteUserTag,threadName=pushExecutor-19,res:0,time:8
2024-12-16 15:47:46,680|INFO |796ffea717b93262a1e1fb8799631878|pushExecutor-12|c.x.n.x.c.s.impl.CustomerClueServiceImpl|deleteUserTag,threadName=pushExecutor-12,res:0,time:5
2024-12-16 15:47:46,681|INFO |796ffea717b93262a1e1fb8799631878|pushExecutor-6|c.x.n.x.c.s.impl.CustomerClueServiceImpl|deleteUserTag,threadName=pushExecutor-6,res:0,time:5
2024-12-16 15:47:46,682|INFO |796ffea717b93262a1e1fb8799631878|pushExecutor-17|c.x.n.x.c.s.impl.CustomerClueServiceImpl|deleteUserTag,threadName=pushExecutor-17,res:0,time:7
2024-12-16 15:47:46,682|INFO |796ffea717b93262a1e1fb8799631878|pushExecutor-8|c.x.n.x.c.s.impl.CustomerClueServiceImpl|deleteUserTag,threadName=pushExecutor-8,res:0,time:5
2024-12-16 15:47:46,774|INFO |796ffea717b93262a1e1fb8799631878|pushExecutor-3|c.x.n.x.c.s.impl.CustomerClueServiceImpl|deleteUserTag,threadName=pushExecutor-3,res:0,time:95
2024-12-16 15:47:46,776|INFO |796ffea717b93262a1e1fb8799631878|pushExecutor-5|c.x.n.x.c.s.impl.CustomerClueServiceImpl|deleteUserTag,threadName=pushExecutor-5,res:0,time:97
2024-12-16 15:47:46,778|INFO |796ffea717b93262a1e1fb8799631878|pushExecutor-19|c.x.n.x.c.s.impl.CustomerClueServiceImpl|deleteUserTag,threadName=pushExecutor-19,res:0,time:98
2024-12-16 15:47:46,780|INFO |796ffea717b93262a1e1fb8799631878|pushExecutor-12|c.x.n.x.c.s.impl.CustomerClueServiceImpl|deleteUserTag,threadName=pushExecutor-12,res:0,time:100
2024-12-16 15:47:46,781|INFO |796ffea717b93262a1e1fb8799631878|pushExecutor-6|c.x.n.x.c.s.impl.CustomerClueServiceImpl|deleteUserTag,threadName=pushExecutor-6,res:0,time:100
2024-12-16 15:47:46,782|INFO |796ffea717b93262a1e1fb8799631878|pushExecutor-17|c.x.n.x.c.s.impl.CustomerClueServiceImpl|deleteUserTag,threadName=pushExecutor-17,res:0,time:100
2024-12-16 15:47:46,784|INFO |796ffea717b93262a1e1fb8799631878|pushExecutor-8|c.x.n.x.c.s.impl.CustomerClueServiceImpl|deleteUserTag,threadName=pushExecutor-8,res:0,time:102
2024-12-16 15:47:46,785|INFO |796ffea717b93262a1e1fb8799631878|pushExecutor-3|c.x.n.x.c.s.impl.CustomerClueServiceImpl|deleteUserTag,threadName=pushExecutor-3,res:0,time:11
2024-12-16 15:47:47,079|INFO |796ffea717b93262a1e1fb8799631878|pushExecutor-5|c.x.n.x.c.s.impl.CustomerClueServiceImpl|deleteUserTag,threadName=pushExecutor-5,res:0,time:303
2024-12-16 15:47:47,082|INFO |796ffea717b93262a1e1fb8799631878|pushExecutor-12|c.x.n.x.c.s.impl.CustomerClueServiceImpl|deleteUserTag,threadName=pushExecutor-12,res:0,time:302
2024-12-16 15:47:47,082|INFO |796ffea717b93262a1e1fb8799631878|pushExecutor-19|c.x.n.x.c.s.impl.CustomerClueServiceImpl|deleteUserTag,threadName=pushExecutor-19,res:0,time:304
2024-12-16 15:47:47,083|INFO |796ffea717b93262a1e1fb8799631878|pushExecutor-17|c.x.n.x.c.s.impl.CustomerClueServiceImpl|deleteUserTag,threadName=pushExecutor-17,res:0,time:301
2024-12-16 15:47:47,084|INFO |796ffea717b93262a1e1fb8799631878|pushExecutor-6|c.x.n.x.c.s.impl.CustomerClueServiceImpl|deleteUserTag,threadName=pushExecutor-6,res:0,time:303
2024-12-16 15:47:47,084|INFO |796ffea717b93262a1e1fb8799631878|pushExecutor-1|c.x.n.x.c.s.impl.CustomerClueServiceImpl|deleteUserTag,threadName=pushExecutor-1,res:1000,time:3307
2024-12-16 15:47:47,085|INFO |796ffea717b93262a1e1fb8799631878|pushExecutor-3|c.x.n.x.c.s.impl.CustomerClueServiceImpl|deleteUserTag,threadName=pushExecutor-3,res:0,time:300
2024-12-16 15:47:47,085|INFO |796ffea717b93262a1e1fb8799631878|pushExecutor-8|c.x.n.x.c.s.impl.CustomerClueServiceImpl|deleteUserTag,threadName=pushExecutor-8,res:0,time:301
2024-12-16 15:47:47,184|INFO |796ffea717b93262a1e1fb8799631878|pushExecutor-5|c.x.n.x.c.s.impl.CustomerClueServiceImpl|deleteUserTag,threadName=pushExecutor-5,res:0,time:105
2024-12-16 15:47:47,185|INFO |796ffea717b93262a1e1fb8799631878|pushExecutor-2|c.x.n.x.c.s.impl.CustomerClueServiceImpl|deleteUserTag,threadName=pushExecutor-2,res:1000,time:3409
2024-12-16 15:47:47,185|INFO |796ffea717b93262a1e1fb8799631878|pushExecutor-12|c.x.n.x.c.s.impl.CustomerClueServiceImpl|deleteUserTag,threadName=pushExecutor-12,res:0,time:103
2024-12-16 15:47:47,187|INFO |796ffea717b93262a1e1fb8799631878|pushExecutor-6|c.x.n.x.c.s.impl.CustomerClueServiceImpl|deleteUserTag,threadName=pushExecutor-6,res:0,time:103
2024-12-16 15:47:47,188|INFO |796ffea717b93262a1e1fb8799631878|pushExecutor-17|c.x.n.x.c.s.impl.CustomerClueServiceImpl|deleteUserTag,threadName=pushExecutor-17,res:0,time:105
2024-12-16 15:47:47,189|INFO |796ffea717b93262a1e1fb8799631878|pushExecutor-19|c.x.n.x.c.s.impl.CustomerClueServiceImpl|deleteUserTag,threadName=pushExecutor-19,res:0,time:107
2024-12-16 15:47:47,273|INFO |796ffea717b93262a1e1fb8799631878|pushExecutor-3|c.x.n.x.c.s.impl.CustomerClueServiceImpl|deleteUserTag,threadName=pushExecutor-3,res:0,time:188
2024-12-16 15:47:47,275|INFO |796ffea717b93262a1e1fb8799631878|pushExecutor-1|c.x.n.x.c.s.impl.CustomerClueServiceImpl|deleteUserTag,threadName=pushExecutor-1,res:0,time:191
2024-12-16 15:47:47,277|INFO |796ffea717b93262a1e1fb8799631878|pushExecutor-8|c.x.n.x.c.s.impl.CustomerClueServiceImpl|deleteUserTag,threadName=pushExecutor-8,res:0,time:192
2024-12-16 15:47:47,280|INFO |796ffea717b93262a1e1fb8799631878|pushExecutor-5|c.x.n.x.c.s.impl.CustomerClueServiceImpl|deleteUserTag,threadName=pushExecutor-5,res:0,time:96
2024-12-16 15:47:47,281|INFO |796ffea717b93262a1e1fb8799631878|pushExecutor-12|c.x.n.x.c.s.impl.CustomerClueServiceImpl|deleteUserTag,threadName=pushExecutor-12,res:0,time:96
2024-12-16 15:47:47,283|INFO |796ffea717b93262a1e1fb8799631878|pushExecutor-2|c.x.n.x.c.s.impl.CustomerClueServiceImpl|deleteUserTag,threadName=pushExecutor-2,res:0,time:98
2024-12-16 15:47:47,285|INFO |796ffea717b93262a1e1fb8799631878|pushExecutor-6|c.x.n.x.c.s.impl.CustomerClueServiceImpl|deleteUserTag,threadName=pushExecutor-6,res:0,time:98
2024-12-16 15:47:47,286|INFO |796ffea717b93262a1e1fb8799631878|pushExecutor-17|c.x.n.x.c.s.impl.CustomerClueServiceImpl|deleteUserTag,threadName=pushExecutor-17,res:0,time:98
2024-12-16 15:47:47,288|INFO |796ffea717b93262a1e1fb8799631878|pushExecutor-19|c.x.n.x.c.s.impl.CustomerClueServiceImpl|deleteUserTag,threadName=pushExecutor-19,res:0,time:99
2024-12-16 15:47:47,289|INFO |796ffea717b93262a1e1fb8799631878|pushExecutor-3|c.x.n.x.c.s.impl.CustomerClueServiceImpl|deleteUserTag,threadName=pushExecutor-3,res:0,time:15
2024-12-16 15:47:47,374|INFO |796ffea717b93262a1e1fb8799631878|pushExecutor-1|c.x.n.x.c.s.impl.CustomerClueServiceImpl|deleteUserTag,threadName=pushExecutor-1,res:0,time:99
2024-12-16 15:47:47,375|INFO |796ffea717b93262a1e1fb8799631878|pushExecutor-8|c.x.n.x.c.s.impl.CustomerClueServiceImpl|deleteUserTag,threadName=pushExecutor-8,res:0,time:98
2024-12-16 15:47:47,485|INFO |796ffea717b93262a1e1fb8799631878|pushExecutor-2|c.x.n.x.c.s.impl.CustomerClueServiceImpl|deleteUserTag,threadName=pushExecutor-2,res:0,time:202
2024-12-16 15:47:47,573|INFO |796ffea717b93262a1e1fb8799631878|pushExecutor-12|c.x.n.x.c.s.impl.CustomerClueServiceImpl|deleteUserTag,threadName=pushExecutor-12,res:0,time:292
2024-12-16 15:47:47,575|INFO |796ffea717b93262a1e1fb8799631878|pushExecutor-5|c.x.n.x.c.s.impl.CustomerClueServiceImpl|deleteUserTag,threadName=pushExecutor-5,res:0,time:295
2024-12-16 15:47:47,577|INFO |796ffea717b93262a1e1fb8799631878|pushExecutor-6|c.x.n.x.c.s.impl.CustomerClueServiceImpl|deleteUserTag,threadName=pushExecutor-6,res:0,time:292
2024-12-16 15:47:47,578|INFO |796ffea717b93262a1e1fb8799631878|pushExecutor-17|c.x.n.x.c.s.impl.CustomerClueServiceImpl|deleteUserTag,threadName=pushExecutor-17,res:0,time:292
2024-12-16 15:47:47,580|INFO |796ffea717b93262a1e1fb8799631878|pushExecutor-19|c.x.n.x.c.s.impl.CustomerClueServiceImpl|deleteUserTag,threadName=pushExecutor-19,res:0,time:291
2024-12-16 15:47:47,581|INFO |796ffea717b93262a1e1fb8799631878|pushExecutor-3|c.x.n.x.c.s.impl.CustomerClueServiceImpl|deleteUserTag,threadName=pushExecutor-3,res:0,time:292
2024-12-16 15:47:47,583|INFO |796ffea717b93262a1e1fb8799631878|pushExecutor-1|c.x.n.x.c.s.impl.CustomerClueServiceImpl|deleteUserTag,threadName=pushExecutor-1,res:0,time:209
2024-12-16 15:47:47,584|INFO |796ffea717b93262a1e1fb8799631878|pushExecutor-8|c.x.n.x.c.s.impl.CustomerClueServiceImpl|deleteUserTag,threadName=pushExecutor-8,res:0,time:209
2024-12-16 15:47:47,883|INFO |796ffea717b93262a1e1fb8799631878|pushExecutor-20|c.x.n.x.c.s.impl.CustomerClueServiceImpl|deleteUserTag,threadName=pushExecutor-20,res:1000,time:3405
2024-12-16 15:47:47,987|INFO |796ffea717b93262a1e1fb8799631878|pushExecutor-8|c.x.n.x.c.s.impl.CustomerClueServiceImpl|deleteUserTag,threadName=pushExecutor-8,res:0,time:403
2024-12-16 15:47:48,881|INFO |796ffea717b93262a1e1fb8799631878|pushExecutor-13|c.x.n.x.c.s.impl.CustomerClueServiceImpl|deleteUserTag,threadName=pushExecutor-13,res:1000,time:3194
2024-12-16 15:47:49,184|INFO |796ffea717b93262a1e1fb8799631878|pushExecutor-7|c.x.n.x.c.s.impl.CustomerClueServiceImpl|deleteUserTag,threadName=pushExecutor-7,res:1000,time:3211
2024-12-16 15:47:49,278|INFO |796ffea717b93262a1e1fb8799631878|pushExecutor-16|c.x.n.x.c.s.impl.CustomerClueServiceImpl|deleteUserTag,threadName=pushExecutor-16,res:1000,time:3193
2024-12-16 15:47:49,278|INFO |796ffea717b93262a1e1fb8799631878|pushExecutor-4|c.x.n.x.c.s.impl.CustomerClueServiceImpl|deleteUserTag,threadName=pushExecutor-4,res:0,time:3390
2024-12-16 15:47:49,286|INFO |796ffea717b93262a1e1fb8799631878|pushExecutor-11|c.x.n.x.c.s.impl.CustomerClueServiceImpl|deleteUserTag,threadName=pushExecutor-11,res:1000,time:3312
2024-12-16 15:47:49,484|INFO |796ffea717b93262a1e1fb8799631878|pushExecutor-9|c.x.n.x.c.s.impl.CustomerClueServiceImpl|deleteUserTag,threadName=pushExecutor-9,res:0,time:3112
2024-12-16 15:47:49,488|INFO |796ffea717b93262a1e1fb8799631878|pushExecutor-16|c.x.n.x.c.s.impl.CustomerClueServiceImpl|deleteUserTag,threadName=pushExecutor-16,res:0,time:210
2024-12-16 15:47:49,580|INFO |796ffea717b93262a1e1fb8799631878|pushExecutor-10|c.x.n.x.c.s.impl.CustomerClueServiceImpl|deleteUserTag,threadName=pushExecutor-10,res:1000,time:3205
2024-12-16 15:47:49,584|INFO |796ffea717b93262a1e1fb8799631878|pushExecutor-18|c.x.n.x.c.s.impl.CustomerClueServiceImpl|deleteUserTag,threadName=pushExecutor-18,res:1000,time:3204

观察日志可见，刚开始执行，是个位数毫秒，耗时在逐步增加 ，询问数据平台得知，仅有6个线程在处理数据，所以造成资源争抢，产生请求堆积。

优化：修改线程池数量为6个（最终方案）

执行时间段：17:04 ~ 19:00

刚开始执行日志

2024-12-16 17:04:08,526|INFO |5e7bda947e8bce91351c806dcbe821da|pushExecutor-2|c.x.n.x.c.s.impl.CustomerClueServiceImpl|deleteUserTag,pushExecutor-2,85452001->85453001,res:0,time:5
2024-12-16 17:04:08,526|INFO |5e7bda947e8bce91351c806dcbe821da|pushExecutor-6|c.x.n.x.c.s.impl.CustomerClueServiceImpl|deleteUserTag,pushExecutor-6,85454001->85455001,res:0,time:3
2024-12-16 17:04:08,526|INFO |5e7bda947e8bce91351c806dcbe821da|pushExecutor-5|c.x.n.x.c.s.impl.CustomerClueServiceImpl|deleteUserTag,pushExecutor-5,85455001->85456001,res:0,time:3
2024-12-16 17:04:08,527|INFO |5e7bda947e8bce91351c806dcbe821da|pushExecutor-1|c.x.n.x.c.s.impl.CustomerClueServiceImpl|deleteUserTag,pushExecutor-1,85457001->85458001,res:0,time:3
2024-12-16 17:04:08,527|INFO |5e7bda947e8bce91351c806dcbe821da|pushExecutor-4|c.x.n.x.c.s.impl.CustomerClueServiceImpl|deleteUserTag,pushExecutor-4,85456001->85457001,res:0,time:3
2024-12-16 17:04:08,579|INFO |5e7bda947e8bce91351c806dcbe821da||c.x.n.x.c.s.impl.CustomerClueServiceImpl|deleteUserTag,,85633001->85634001,res:0,time:75
2024-12-16 17:04:08,986|INFO |5e7bda947e8bce91351c806dcbe821da|pushExecutor-3|c.x.n.x.c.s.impl.CustomerClueServiceImpl|deleteUserTag,pushExecutor-3,85458001->85459001,res:0,time:461
2024-12-16 17:04:09,011|INFO |5e7bda947e8bce91351c806dcbe821da|pushExecutor-5|c.x.n.x.c.s.impl.CustomerClueServiceImpl|deleteUserTag,pushExecutor-5,85461001->85462001,res:0,time:485
2024-12-16 17:04:09,079|INFO |5e7bda947e8bce91351c806dcbe821da|pushExecutor-2|c.x.n.x.c.s.impl.CustomerClueServiceImpl|deleteUserTag,pushExecutor-2,85459001->85460001,res:0,time:553
2024-12-16 17:04:09,083|INFO |5e7bda947e8bce91351c806dcbe821da|pushExecutor-2|c.x.n.x.c.s.impl.CustomerClueServiceImpl|deleteUserTag,pushExecutor-2,85466001->85467001,res:0,time:4
2024-12-16 17:04:09,084|INFO |5e7bda947e8bce91351c806dcbe821da|pushExecutor-6|c.x.n.x.c.s.impl.CustomerClueServiceImpl|deleteUserTag,pushExecutor-6,85460001->85461001,res:0,time:558
2024-12-16 17:04:09,089|INFO |5e7bda947e8bce91351c806dcbe821da|pushExecutor-1|c.x.n.x.c.s.impl.CustomerClueServiceImpl|deleteUserTag,pushExecutor-1,85462001->85463001,res:0,time:562
2024-12-16 17:04:09,102|INFO |5e7bda947e8bce91351c806dcbe821da|pushExecutor-4|c.x.n.x.c.s.impl.CustomerClueServiceImpl|deleteUserTag,pushExecutor-4,85463001->85464001,res:0,time:575
2024-12-16 17:04:09,582|INFO |5e7bda947e8bce91351c806dcbe821da||c.x.n.x.c.s.impl.CustomerClueServiceImpl|deleteUserTag,,85665001->85666001,res:1000,time:1003
2024-12-16 17:04:09,893|INFO |5e7bda947e8bce91351c806dcbe821da|pushExecutor-3|c.x.n.x.c.s.impl.CustomerClueServiceImpl|deleteUserTag,pushExecutor-3,85464001->85465001,res:1000,time:907
2024-12-16 17:04:09,972|INFO |5e7bda947e8bce91351c806dcbe821da|pushExecutor-5|c.x.n.x.c.s.impl.CustomerClueServiceImpl|deleteUserTag,pushExecutor-5,85465001->85466001,res:1000,time:961
2024-12-16 17:04:10,002|INFO |5e7bda947e8bce91351c806dcbe821da|pushExecutor-1|c.x.n.x.c.s.impl.CustomerClueServiceImpl|deleteUserTag,pushExecutor-1,85469001->85470001,res:1000,time:913
2024-12-16 17:04:10,006|INFO |5e7bda947e8bce91351c806dcbe821da|pushExecutor-2|c.x.n.x.c.s.impl.CustomerClueServiceImpl|deleteUserTag,pushExecutor-2,85467001->85468001,res:1000,time:923
2024-12-16 17:04:10,009|INFO |5e7bda947e8bce91351c806dcbe821da|pushExecutor-4|c.x.n.x.c.s.impl.CustomerClueServiceImpl|deleteUserTag,pushExecutor-4,85470001->85471001,res:1000,time:907
2024-12-16 17:04:10,011|INFO |5e7bda947e8bce91351c806dcbe821da|pushExecutor-6|c.x.n.x.c.s.impl.CustomerClueServiceImpl|deleteUserTag,pushExecutor-6,85468001->85469001,res:1000,time:927
2024-12-16 17:04:10,518|INFO |5e7bda947e8bce91351c806dcbe821da||c.x.n.x.c.s.impl.CustomerClueServiceImpl|deleteUserTag,,85673001->85674001,res:1000,time:936

执行一个多小时后日志

2024-12-16 18:59:58,076|INFO |5e7bda947e8bce91351c806dcbe821da|pushExecutor-6|c.x.n.x.c.s.impl.CustomerClueServiceImpl|deleteUserTag,pushExecutor-6,134119001->134120001,res:0,time:2
2024-12-16 18:59:58,190|INFO |5e7bda947e8bce91351c806dcbe821da|pushExecutor-3|c.x.n.x.c.s.impl.CustomerClueServiceImpl|deleteUserTag,pushExecutor-3,134115001->134116001,res:0,time:966
2024-12-16 18:59:58,192|INFO |5e7bda947e8bce91351c806dcbe821da|pushExecutor-2|c.x.n.x.c.s.impl.CustomerClueServiceImpl|deleteUserTag,pushExecutor-2,134114001->134115001,res:0,time:974
2024-12-16 18:59:58,217|INFO |5e7bda947e8bce91351c806dcbe821da|pushExecutor-5|c.x.n.x.c.s.impl.CustomerClueServiceImpl|deleteUserTag,pushExecutor-5,134116001->134117001,res:0,time:918
2024-12-16 18:59:58,299|INFO |5e7bda947e8bce91351c806dcbe821da|pushExecutor-1|c.x.n.x.c.s.impl.CustomerClueServiceImpl|deleteUserTag,pushExecutor-1,134117001->134118001,res:0,time:887
2024-12-16 18:59:58,300|INFO |5e7bda947e8bce91351c806dcbe821da||c.x.n.x.c.s.impl.CustomerClueServiceImpl|deleteUserTag,,134117001->134118001,res:0,time:684
2024-12-16 18:59:58,540|INFO |5e7bda947e8bce91351c806dcbe821da|pushExecutor-4|c.x.n.x.c.s.impl.CustomerClueServiceImpl|deleteUserTag,pushExecutor-4,134118001->134119001,res:1000,time:925
2024-12-16 18:59:58,629|INFO |5e7bda947e8bce91351c806dcbe821da||c.x.n.x.c.s.impl.CustomerClueServiceImpl|deleteUserTag,,134205001->134206001,res:1000,time:1006
2024-12-16 18:59:58,632|INFO |5e7bda947e8bce91351c806dcbe821da||c.x.n.x.c.s.impl.CustomerClueServiceImpl|deleteUserTag,,134207001->134208001,res:0,time:3
2024-12-16 18:59:59,000|INFO |5e7bda947e8bce91351c806dcbe821da|pushExecutor-6|c.x.n.x.c.s.impl.CustomerClueServiceImpl|deleteUserTag,pushExecutor-6,134120001->134121001,res:0,time:924
2024-12-16 18:59:59,183|INFO |5e7bda947e8bce91351c806dcbe821da|pushExecutor-2|c.x.n.x.c.s.impl.CustomerClueServiceImpl|deleteUserTag,pushExecutor-2,134122001->134123001,res:1000,time:991
2024-12-16 18:59:59,185|INFO |5e7bda947e8bce91351c806dcbe821da|pushExecutor-3|c.x.n.x.c.s.impl.CustomerClueServiceImpl|deleteUserTag,pushExecutor-3,134121001->134122001,res:1000,time:995
2024-12-16 18:59:59,188|INFO |5e7bda947e8bce91351c806dcbe821da|pushExecutor-3|c.x.n.x.c.s.impl.CustomerClueServiceImpl|deleteUserTag,pushExecutor-3,134128001->134129001,res:0,time:3
2024-12-16 18:59:59,217|INFO |5e7bda947e8bce91351c806dcbe821da|pushExecutor-5|c.x.n.x.c.s.impl.CustomerClueServiceImpl|deleteUserTag,pushExecutor-5,134123001->134124001,res:1000,time:1000
2024-12-16 18:59:59,220|INFO |5e7bda947e8bce91351c806dcbe821da||c.x.n.x.c.s.impl.CustomerClueServiceImpl|deleteUserTag,,134124001->134125001,res:0,time:920
2024-12-16 18:59:59,273|INFO |5e7bda947e8bce91351c806dcbe821da|pushExecutor-1|c.x.n.x.c.s.impl.CustomerClueServiceImpl|deleteUserTag,pushExecutor-1,134124001->134125001,res:1000,time:974
2024-12-16 18:59:59,573|INFO |5e7bda947e8bce91351c806dcbe821da|pushExecutor-4|c.x.n.x.c.s.impl.CustomerClueServiceImpl|deleteUserTag,pushExecutor-4,134125001->134126001,res:1000,time:1033
2024-12-16 18:59:59,632|INFO |5e7bda947e8bce91351c806dcbe821da||c.x.n.x.c.s.impl.CustomerClueServiceImpl|deleteUserTag,,134208001->134209001,res:1000,time:1000
2024-12-16 18:59:59,975|INFO |5e7bda947e8bce91351c806dcbe821da|pushExecutor-6|c.x.n.x.c.s.impl.CustomerClueServiceImpl|deleteUserTag,pushExecutor-6,134126001->134127001,res:1000,time:975

可以看到，请求耗时基本是平稳到 1s左右的，说明数据平台侧完全可以处理当前的并发量，资源是足够的

结论

由于我们对接OB没有监控平台，所以观察不了，本例通过最简单的日志打印观察执行情况

  • 如果处理耗时较一开始慢慢增大，表示并发量太大，处理不过来了，请求开始堆积了，那么就要减小并发量；
  • 如果处理耗时很平稳一直很快，可以尝试增加并发量观察，直到达到最大并发量即可。

本例没有其他操作，只是一直循环执行删除操作，那么业务并发量就与数据侧的处理线程持平上下浮动，对标CPU密集型任务；如果业务还需要查询第三方之类的操作再去删除，那么业务并发量就要先大设置观察，对标IO密集型任务

本文来自互联网用户投稿，该文观点仅代表作者本人，不代表本站立场。本站仅提供信息存储空间服务，不拥有所有权，不承担相关法律责任。如若转载，请注明出处：http://www.mzph.cn/pingmian/64703.shtml

如若内容造成侵权/违法违规/事实不符，请联系多彩编程网进行投诉反馈email:809451989@qq.com，一经查实，立即删除！

相关文章

【2025最新计算机毕业设计】基于SSM框架的宠物领养系统【提供源码+答辩PPT+文档+项目部署】

【2025最新计算机毕业设计】基于SSM框架的宠物领养系统【提供源码+答辩PPT+文档+项目部署】

作者简介&#xff1a;✌CSDN新星计划导师、Java领域优质创作者、掘金/华为云/阿里云/InfoQ等平台优质作者、专注于Java技术领域和学生毕业项目实战,高校老师/讲师/同行前辈交流。✌ 主要内容&#xff1a;&#x1f31f;Java项目、Python项目、前端项目、PHP、ASP.NET、人工智能…
阅读更多...
【专题】2024抖音电商母婴行业分析报告汇总PDF洞察（附原数据表）

【专题】2024抖音电商母婴行业分析报告汇总PDF洞察（附原数据表）

原文链接&#xff1a;https://tecdat.cn/?p38651 在数字化浪潮的席卷下&#xff0c;抖音电商母婴行业正经历着深刻变革。当下&#xff0c;年轻一代父母崛起&#xff0c;特别是 24 至 30 岁以及 18 至 23 岁的群体成为抖音母婴行业兴趣人群的主力军。他们带来全新育儿理念&…
阅读更多...
免费开源！推荐一款网页版数据库管理工具！

免费开源！推荐一款网页版数据库管理工具！

免费开源&#xff01;推荐一款网页版数据库管理工具&#xff01; DBGate 是一个开源的数据库管理工具&#xff0c;DBGate 的最大特点是可以 Web 访问&#xff01;&#xff0c;轻松实现一台机器部署&#xff0c;所有人使用&#xff01; 无论是 MySQL、PostgreSQL、SQLite 还是…
阅读更多...
使用 UniApp 在微信小程序中实现 SSE 流式响应

使用 UniApp 在微信小程序中实现 SSE 流式响应

概述 服务端发送事件（Server-Sent Events, SSE）是一种允许服务器向客户端推送实时更新的技术。SSE 提供了一种单向的通信通道，服务器可以持续地向客户端发送数据，而不需要客户端频繁发起请求。这对于需要实时更新的应用场景非常有用。 流式传输的特点是将数据逐步传输给客…
阅读更多...
【Java项目】基于SpringBoot的【旅游管理系统 】

【Java项目】基于SpringBoot的【旅游管理系统 】

【Java项目】基于SpringBoot的【旅游管理系统 】 技术简介&#xff1a;本系统使用JAVA语言开发&#xff0c;采用B/S架构、Spring Boot框架、MYSQL数据库进行开发设计。 系统简介&#xff1a;&#xff08;1&#xff09;管理员功能&#xff1a;可以管理个人中心、用户管理、景区分…
阅读更多...
【系统】Windows11更新解决办法，一键暂停

【系统】Windows11更新解决办法，一键暂停

最近的windows更新整的我是措不及防&#xff0c;干啥都要关注一下更新的问题&#xff0c;有的时候还关不掉&#xff0c;我的强迫症就来了&#xff0c;非得关了你不可&#xff01; 经过了九九八十一难的研究之后&#xff0c;终于找到了一个算是比较靠谱的暂停更新的方法&#x…
阅读更多...
Canoe E2E校验自定义Checksum算法

Canoe E2E校验自定义Checksum算法

文章目录 一、添加 DBC文件二、导入要仿真的ECU节点三、编写 CAPL脚本1. 创建 .can 文件2. 设置counter递增3. 设置 CRC 算法&#xff0c;以profile01 8-bit SAE J1850 CRC校验为例 四、开始仿真五、运行结果CRC在线校验 当E2E的 CRC算法非常规算法&#xff0c;则需要自己编写代…
阅读更多...
穷举vs暴搜vs深搜vs回溯vs剪枝系列一＞找出所有子集的异或总和再求和

穷举vs暴搜vs深搜vs回溯vs剪枝系列一＞找出所有子集的异或总和再求和

题目&#xff1a; 解析&#xff1a; 代码&#xff1a; private int ret;//返回周结果private int path;//枚举一个元素就异或进去public int subsetXORSum(int[] nums) {dfs(nums, 0);return ret;} private void dfs(int[] nums, int pos){ret path;for(int i pos; i <…
阅读更多...
python使用Flask框架创建一个简单的动态日历

python使用Flask框架创建一个简单的动态日历

0. 运行效果 运行代码&#xff0c;然后在浏览器中访问 http://127.0.0.1:5000/&#xff0c;将看到一个动态日历&#xff0c;能够通过点击按钮切换月份。 1. 安装 Flask 首先&#xff0c;确保你已经安装了Flask。如果没有&#xff0c;可以使用以下命令安装&#xff1a; pip i…
阅读更多...
windows openssl编译x64版libssl.lib，编译x64版本libcurl.lib，支持https，vs2015编译器

windows openssl编译x64版libssl.lib，编译x64版本libcurl.lib，支持https，vs2015编译器

不要纠结&#xff0c;直接选择用perl编译&#xff01; 告诫想要用弄成vs编译版的&#xff0c;暂时先别给自己增加麻烦 告诫&#xff0c;以下执行的每一步&#xff0c;都不要纠结 先安装环境 nasm 64位版本 https://www.nasm.us/pub/nasm/releasebuilds/2.16.01/win64/nasm-…
阅读更多...
【WRF-Urban】输入空间分布人为热排放数据的WRF运行全过程总结

【WRF-Urban】输入空间分布人为热排放数据的WRF运行全过程总结

目录 数据准备检查新增变量配置（如果有）WPS预处理修改namelist.wpsStep1: geogridStep2: ungribStep3: metgridWRF运行修改namelist.input调试namelist.input运行./real.exe运行./wrf.exe参考WRF模型的基本流程如下： 数据准备 空间分布热排放数据下载及制备可参见另一博客…
阅读更多...
JavaSE——绘图入门

JavaSE——绘图入门

一、Java绘图坐标体系 下图说明了Java坐标系&#xff0c;坐标原地位于左上角&#xff0c;以像素为单位。在Java坐标系中&#xff0c;第一个是x坐标&#xff0c;表示当前位置为水平方向&#xff0c;距离坐标原点x个像素&#xff1b;第二个是y坐标&#xff0c;表示当前位置为垂直…
阅读更多...
初学stm32 --- 系统时钟配置

初学stm32 --- 系统时钟配置

众所周知&#xff0c;时钟系统是 CPU 的脉搏&#xff0c;就像人的心跳一样。所以时钟系统的重要性就不言而喻了。 STM32 的时钟系统比较复杂&#xff0c;不像简单的 51 单片机一个系统时钟就可以解决一切。于是有人要问&#xff0c;采用一个系统时钟不是很简单吗&#xff1f;为…
阅读更多...
FM-CLIP: Flexible Modal CLIP for Face Anti-Spoofing

FM-CLIP: Flexible Modal CLIP for Face Anti-Spoofing

“FM-CLIP: Flexible Modal CLIP for Face Anti-Spoofing”由澳门科技大学、中国科学院等单位的Ajian Liu、Hui Ma等人撰写。文章提出了一种名为FM-CLIP的新型灵活模态人脸反欺诈（FAS）方法，通过利用文本特征动态调整视觉特征以实现模态独立性，在多模态数据集上取得了优于现…
阅读更多...
进网许可认证、交换路由设备检测项目更新25年1月起

进网许可认证、交换路由设备检测项目更新25年1月起

实施时间 2025年1月1日起实施 涉及设备范围 核心路由器、边缘路由器、以太网交换机、三层交换机、宽带网络接入服务器&#xff08;BNAS&#xff09; 新增检测依据 GBT41266-2022网络关键设备安全检测方法交换机设备 GBT41267-2022网络关键设备安全技术要求交换机设备 GB/…
阅读更多...
白嫖内网穿透之神卓互联Linux安装教程（树莓派）

白嫖内网穿透之神卓互联Linux安装教程（树莓派）

最近家里有一个树莓派&#xff0c;捣鼓来去不知道干嘛&#xff0c;于是打算作为内网穿透盒子用&#xff0c;于是百度了一下&#xff0c;发现神卓互联还不错&#xff0c;可以让外网请求通过各种复杂的路由和防火墙访问到内网的服务。 以下是在Linux树莓派系统上安装神卓互联客户…
阅读更多...
Ansible 批量管理华为 CE 交换机

Ansible 批量管理华为 CE 交换机

注&#xff1a;本文为 “Ansible 管理华为 CE 交换机” 相关文章合辑。 使用 CloudEngine - Ansible 批量管理华为 CE 交换机 wsf535 IP 属地&#xff1a;贵州 2018.02.05 15:26:05 总体介绍 Ansible 是一个开源的自动化运维工具&#xff0c;AnsibleWorks 成立于 2012 年&a…
阅读更多...
防火墙(RHCE)

防火墙(RHCE)

1、什么是防火墙 防火墙&#xff1a;防火墙是位于内部网和外部网之间的屏障&#xff0c;它按照系统管理员预先定义好的规则来控制数据包 的进出。 防火墙又可以分为硬件防火墙与软件防火墙。硬件防火墙是由厂商设计好的主机硬件&#xff0c;这台硬件防火墙 的操作系统主要以提…
阅读更多...
线性代数基础与应用：基底 (Basis) 与现金流及单期贷款模型（中英双语）

线性代数基础与应用：基底 (Basis) 与现金流及单期贷款模型（中英双语）

具体请参考&#xff1a;https://web.stanford.edu/~boyd/vmls/ 下面的例子来源于这本书。 线性代数基础与应用&#xff1a;基底 (Basis) 与现金流及单期贷款模型 在线性代数中&#xff0c;基底&#xff08;Basis&#xff09;是一个重要的概念&#xff0c;广泛应用于信号处理、…
阅读更多...
【安当产品应用案例100集】032-重塑企业SaaS平台的PostgreSQL凭据管理体系

【安当产品应用案例100集】032-重塑企业SaaS平台的PostgreSQL凭据管理体系

一、案例背景 在本次案例分享中&#xff0c;一家为旅行社提供SaaS服务的技术服务商&#xff0c;其依赖PostgreSQL作为其核心数据存储解决方案&#xff0c;并且在阿里云和内网环境中均部署了相关服务与数据库实例。随着业务的发展和技术团队规模的扩大&#xff0c;当前的数据库…
阅读更多...
推荐文章
最新文章

Copyright @ 2022~2023