redis 重建主从变化

说明：监控某个从节点的变化，从节点清空了 appendonlydir/ 目录,删除了 dump.rdb，重启该从节点，监测变化如下。

第一阶段

重新启动该节点期间，监测数据目录下文件变化，首先生成 temp-xxx.rdb 文件，并且该文件一直增大,在此期间日志输出如下

Defaulted container "redis" out of: redis, sentinel, split-brain-fix, redis-exporter, config-init (init)
1:C 20 Nov 2023 11:10:14.899 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
1:C 20 Nov 2023 11:10:14.899 # Redis version=7.0.9, bits=64, commit=00000000, modified=0, pid=1, just started
1:C 20 Nov 2023 11:10:14.899 # Configuration loaded
1:S 20 Nov 2023 11:10:14.899 * monotonic clock: POSIX clock_gettime
1:S 20 Nov 2023 11:10:14.900 * Running mode=standalone, port=6379.
1:S 20 Nov 2023 11:10:14.900 # Server initialized
1:S 20 Nov 2023 11:10:14.901 * Creating AOF base file appendonly.aof.1.base.rdb on server start
1:S 20 Nov 2023 11:10:14.901 * Creating AOF incr file appendonly.aof.1.incr.aof on server start
1:S 20 Nov 2023 11:10:14.901 * Ready to accept connections
1:S 20 Nov 2023 11:10:14.901 * Connecting to MASTER 10.97.31.186:6379
1:S 20 Nov 2023 11:10:14.901 * MASTER <-> REPLICA sync started
1:S 20 Nov 2023 11:10:14.901 * Non blocking connect for SYNC fired the event.
1:S 20 Nov 2023 11:10:14.902 * Master replied to PING, replication can continue...
1:S 20 Nov 2023 11:10:14.902 * Partial resynchronization not possible (no cached master)
1:S 20 Nov 2023 11:10:19.796 * Full resync from master: 1db245adaf12b98d4dea329679a4b03b0a310272:1019321
1:S 20 Nov 2023 11:10:21.616 * MASTER <-> REPLICA sync: receiving streamed RDB from master with EOF to disk

kl logs -f redis-server-2
1:S 20 Nov 2023 10:20:02.717 * MASTER <-> REPLICA sync: Flushing old data
1:S 20 Nov 2023 10:20:02.717 * MASTER <-> REPLICA sync: Loading DB in memory
1:S 20 Nov 2023 10:20:02.718 * Loading RDB produced by version 7.0.9
1:S 20 Nov 2023 10:20:02.718 * RDB age 354 seconds
1:S 20 Nov 2023 10:20:02.718 * RDB memory usage when created 52476.26 Mb

文件变化，先产生 temp-xxx.rdb 文件，并且该文件一直增大

total 22G
drwxrwxrwx 4 root      root 4.0K Nov 20 18:14 ./
drwxr-xr-x 3 root      root 4.0K Nov 20 17:50 ../
drwxr-xr-x 2 openbayes root 4.0K Nov 20 18:14 appendonlydir/
drwxr-xr-x 2 openbayes root 4.0K Nov 20 18:14 conf/
-rw-r--r-- 1 openbayes root  22G Nov 20 18:20 temp-1700475247.1.rdb

等到跟主库的 dump.rdb 文件一样大的时候，瞬间生成 dump.rdb 文件

total 22G
drwxrwxrwx 4 root      root 4.0K Nov 20 18:20 ./
drwxr-xr-x 3 root      root 4.0K Nov 20 17:50 ../
drwxr-xr-x 2 openbayes root 4.0K Nov 20 18:14 appendonlydir/
drwxr-xr-x 2 openbayes root 4.0K Nov 20 18:14 conf/
-rw-r--r-- 1 openbayes root  22G Nov 20 18:20 dump.rdb

第二阶段

生成 dump.rbd 文件后内存用量会增加,这期间日志输出如下，文件不会有变化。

1:S 20 Nov 2023 11:16:16.342 * MASTER <-> REPLICA sync: Flushing old data
1:S 20 Nov 2023 11:16:16.342 * MASTER <-> REPLICA sync: Loading DB in memory
1:S 20 Nov 2023 11:16:16.343 * Loading RDB produced by version 7.0.9
1:S 20 Nov 2023 11:16:16.343 * RDB age 356 seconds
1:S 20 Nov 2023 11:16:16.343 * RDB memory usage when created 52477.00 Mb

第三阶段

生成 dump.rbd 文件后内存用量会增加，当内存用量增加到一定极值后,日志输出如下，开始第三阶段

1:S 20 Nov 2023 11:20:33.582 * Done loading RDB, keys loaded: 136197566, keys expired: 0.
1:S 20 Nov 2023 11:20:33.582 * MASTER <-> REPLICA sync: Finished with success
1:S 20 Nov 2023 11:20:33.582 * Creating AOF incr file temp-appendonly.aof.incr on background rewrite
1:S 20 Nov 2023 11:20:34.166 * Background append only file rewriting started by pid 579

生成 dump.rbd 文件后内存用量会增加，当内存用量增加到一定极值后开始在 appendonlydir 目录下产生 temp-appendonly.aof.incr 文件，该文件不增加

ll -h appendonlydir/
total 16K
drwxr-xr-x 2 openbayes root 4.0K Nov 20 18:24 ./
drwxrwxrwx 4 root      root 4.0K Nov 20 18:24 ../
-rw-r--r-- 1 openbayes root   88 Nov 20 18:14 appendonly.aof.1.base.rdb
-rw-r--r-- 1 openbayes root    0 Nov 20 18:14 appendonly.aof.1.incr.aof
-rw-r--r-- 1 openbayes root   88 Nov 20 18:14 appendonly.aof.manifest
-rw-r--r-- 1 openbayes root    0 Nov 20 18:24 temp-appendonly.aof.incr

同时在 appendonlydir 同级目录下产生 temp-rewriteaof-578.aof 文件，该文件一直增大

total 28G
drwxrwxrwx 4 root      root 4.0K Nov 20 18:24 ./
drwxr-xr-x 3 root      root 4.0K Nov 20 17:50 ../
drwxr-xr-x 2 openbayes root 4.0K Nov 20 18:24 appendonlydir/
drwxr-xr-x 2 openbayes root 4.0K Nov 20 18:14 conf/
-rw-r--r-- 1 openbayes root  22G Nov 20 18:20 dump.rdb
-rw-r--r-- 1 openbayes root 6.3G Nov 20 18:26 temp-rewriteaof-578.aof

第四阶段

当 temp-rewriteaof-578.aof 文件达到跟 dump.rbd 文件大一样大的时候， temp-rewriteaof-578.aof 文件消失,然后在 appendonlydir 同级目录下生成一个 temp-rewriteaof-bg-579.aof 文件

ll -h appendonlydir/
total 16
drwxr-xr-x 2 openbayes root 4096 Nov 20 19:20 ./
drwxrwxrwx 4 root      root 4096 Nov 20 19:27 ../
-rw-r--r-- 1 openbayes root   88 Nov 20 19:10 appendonly.aof.1.base.rdb
-rw-r--r-- 1 openbayes root    0 Nov 20 19:10 appendonly.aof.1.incr.aof
-rw-r--r-- 1 openbayes root   88 Nov 20 19:10 appendonly.aof.manifest
-rw-r--r-- 1 openbayes root    0 Nov 20 19:20 temp-appendonly.aof.incr

total 43G
drwxrwxrwx 4 root      root 4.0K Nov 20 19:27 ./
drwxr-xr-x 3 root      root 4.0K Nov 20 17:50 ../
drwxr-xr-x 2 openbayes root 4.0K Nov 20 19:20 appendonlydir/
drwxr-xr-x 2 openbayes root 4.0K Nov 20 19:10 conf/
-rw-r--r-- 1 openbayes root  22G Nov 20 19:16 dump.rdb
-rw-r--r-- 1 openbayes root  22G Nov 20 19:27 temp-rewriteaof-bg-579.aof

生成 temp-rewriteaof-bg-579.aof 文件的日志输出

579:C 20 Nov 2023 11:27:04.566 * Successfully created the temporary AOF base file temp-rewriteaof-bg-579.aof
579:C 20 Nov 2023 11:27:05.051 * Fork CoW for AOF rewrite: current 2 MB, peak 2 MB, average 2 MB
1:S 20 Nov 2023 11:27:05.753 * Background AOF rewrite terminated with success

片刻间，temp-rewriteaof-bg-579.aof 文件消失变成 appendonlydir 目录下 appendonly.aof.2.base.rdb 文件，temp-appendonly.aof.incr 变成 appendonly.aof.2.incr.aof 文件。并且清理原来的 appendonly.aof.1.base.rdb 和 appendonly.aof.1.incr.aof 文件

drwxrwxrwx 4 root      root 4.0K Nov 20 18:05 ./
drwxr-xr-x 3 root      root 4.0K Nov 20 17:28 ../
drwxr-xr-x 2 openbayes root 4.0K Nov 20 17:34 appendonlydir/
drwxr-xr-x 2 openbayes root 4.0K Nov 20 17:48 conf/
-rw-r--r-- 1 openbayes root  22G Nov 20 18:05 dump.rdb

ll -h appendonlydir/
total 22G
drwxr-xr-x 2 openbayes root 4.0K Nov 20 19:27 ./
drwxrwxrwx 4 root      root 4.0K Nov 20 19:27 ../
-rw-r--r-- 1 openbayes root  22G Nov 20 19:27 appendonly.aof.2.base.rdb
-rw-r--r-- 1 openbayes root    0 Nov 20 19:20 appendonly.aof.2.incr.aof
-rw-r--r-- 1 openbayes root   88 Nov 20 19:27 appendonly.aof.manifest

文件变化日志

1:S 20 Nov 2023 11:27:05.753 * Successfully renamed the temporary AOF base file temp-rewriteaof-bg-579.aof into appendonly.aof.2.base.rdb
1:S 20 Nov 2023 11:27:05.753 * Successfully renamed the temporary AOF incr file temp-appendonly.aof.incr into appendonly.aof.2.incr.aof

清理原来的 appendonly.aof.1.base.rdb 和 appendonly.aof.1.incr.aof 文件日志输出

1:S 20 Nov 2023 11:27:05.753 * Removing the history file appendonly.aof.1.incr.aof in the background
1:S 20 Nov 2023 11:27:05.754 * Removing the history file appendonly.aof.1.base.rdb in the background

该阶段日志输出如下

579:C 20 Nov 2023 11:27:04.566 * Successfully created the temporary AOF base file temp-rewriteaof-bg-579.aof
579:C 20 Nov 2023 11:27:05.051 * Fork CoW for AOF rewrite: current 2 MB, peak 2 MB, average 2 MB
1:S 20 Nov 2023 11:27:05.753 * Background AOF rewrite terminated with success
1:S 20 Nov 2023 11:27:05.753 * Successfully renamed the temporary AOF base file temp-rewriteaof-bg-579.aof into appendonly.aof.2.base.rdb
1:S 20 Nov 2023 11:27:05.753 * Successfully renamed the temporary AOF incr file temp-appendonly.aof.incr into appendonly.aof.2.incr.aof
1:S 20 Nov 2023 11:27:05.753 * Removing the history file appendonly.aof.1.incr.aof in the background
1:S 20 Nov 2023 11:27:05.754 * Removing the history file appendonly.aof.1.base.rdb in the background
1:S 20 Nov 2023 11:27:05.754 * Background AOF rewrite finished successfully

从库恢复期间，完整的日志输出如下

578:C 20 Nov 2023 10:30:50.987 * Successfully created the temporary AOF base file temp-rewriteaof-bg-578.aof
578:C 20 Nov 2023 10:30:51.486 * Fork CoW for AOF rewrite: current 2 MB, peak 2 MB, average 2 MB
1:S 20 Nov 2023 10:30:52.228 * Background AOF rewrite terminated with success
1:S 20 Nov 2023 10:30:52.228 * Successfully renamed the temporary AOF base file temp-rewriteaof-bg-578.aof into appendonly.aof.2.base.rdb
1:S 20 Nov 2023 10:30:52.228 * Successfully renamed the temporary AOF incr file temp-appendonly.aof.incr into appendonly.aof.2.incr.aof
1:S 20 Nov 2023 10:30:52.229 * Removing the history file appendonly.aof.1.incr.aof in the background
1:S 20 Nov 2023 10:30:52.229 * Removing the history file appendonly.aof.1.base.rdb in the background
1:S 20 Nov 2023 10:30:52.229 * Background AOF rewrite finished successfully

重建主从期间日志变化

kl logs -f redis-server-2
Defaulted container "redis" out of: redis, sentinel, split-brain-fix, redis-exporter, config-init (init)
1:C 20 Nov 2023 11:10:14.899 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
1:C 20 Nov 2023 11:10:14.899 # Redis version=7.0.9, bits=64, commit=00000000, modified=0, pid=1, just started
1:C 20 Nov 2023 11:10:14.899 # Configuration loaded
1:S 20 Nov 2023 11:10:14.899 * monotonic clock: POSIX clock_gettime
1:S 20 Nov 2023 11:10:14.900 * Running mode=standalone, port=6379.
1:S 20 Nov 2023 11:10:14.900 # Server initialized
1:S 20 Nov 2023 11:10:14.901 * Creating AOF base file appendonly.aof.1.base.rdb on server start
1:S 20 Nov 2023 11:10:14.901 * Creating AOF incr file appendonly.aof.1.incr.aof on server start
1:S 20 Nov 2023 11:10:14.901 * Ready to accept connections
1:S 20 Nov 2023 11:10:14.901 * Connecting to MASTER 10.97.31.186:6379
1:S 20 Nov 2023 11:10:14.901 * MASTER <-> REPLICA sync started
1:S 20 Nov 2023 11:10:14.901 * Non blocking connect for SYNC fired the event.
1:S 20 Nov 2023 11:10:14.902 * Master replied to PING, replication can continue...
1:S 20 Nov 2023 11:10:14.902 * Partial resynchronization not possible (no cached master)
1:S 20 Nov 2023 11:10:19.796 * Full resync from master: 1db245adaf12b98d4dea329679a4b03b0a310272:1019321
1:S 20 Nov 2023 11:10:21.616 * MASTER <-> REPLICA sync: receiving streamed RDB from master with EOF to disk1:S 20 Nov 2023 11:16:16.342 * MASTER <-> REPLICA sync: Flushing old data
1:S 20 Nov 2023 11:16:16.342 * MASTER <-> REPLICA sync: Loading DB in memory
1:S 20 Nov 2023 11:16:16.343 * Loading RDB produced by version 7.0.9
1:S 20 Nov 2023 11:16:16.343 * RDB age 356 seconds
1:S 20 Nov 2023 11:16:16.343 * RDB memory usage when created 52477.00 Mb1:S 20 Nov 2023 11:20:33.582 * Done loading RDB, keys loaded: 136197566, keys expired: 0.
1:S 20 Nov 2023 11:20:33.582 * MASTER <-> REPLICA sync: Finished with success
1:S 20 Nov 2023 11:20:33.582 * Creating AOF incr file temp-appendonly.aof.incr on background rewrite
1:S 20 Nov 2023 11:20:34.166 * Background append only file rewriting started by pid 579579:C 20 Nov 2023 11:27:04.566 * Successfully created the temporary AOF base file temp-rewriteaof-bg-579.aof579:C 20 Nov 2023 11:27:05.051 * Fork CoW for AOF rewrite: current 2 MB, peak 2 MB, average 2 MB
1:S 20 Nov 2023 11:27:05.753 * Background AOF rewrite terminated with success1:S 20 Nov 2023 11:27:05.753 * Successfully renamed the temporary AOF base file temp-rewriteaof-bg-579.aof into appendonly.aof.2.base.rdb
1:S 20 Nov 2023 11:27:05.753 * Successfully renamed the temporary AOF incr file temp-appendonly.aof.incr into appendonly.aof.2.incr.aof1:S 20 Nov 2023 11:27:05.753 * Removing the history file appendonly.aof.1.incr.aof in the background
1:S 20 Nov 2023 11:27:05.754 * Removing the history file appendonly.aof.1.base.rdb in the background
1:S 20 Nov 2023 11:27:05.754 * Background AOF rewrite finished successfully