PostgreSQL pg_repack 重新组织表并释放表空间

pg_repack

pg_repack是 PostgreSQL 的一个扩展，它允许您从表和索引中删除膨胀，并可选择恢复聚集索引的物理顺序。与CLUSTER和VACUUM FULL不同，它可以在线工作，在处理过程中无需对已处理的表保持独占锁定。pg_repack 启动效率高，性能可与直接使用 CLUSTER 相媲美。

要求

PostgreSQL 版本
PostgreSQL 9.5、9.6、10、11、12、13、14、15、16、17。

PostgreSQL 9.4 及之前版本不受支持。

磁盘
执行全表重新打包需要的可用磁盘空间大约是目标表及其索引的两倍。
例如，如果要重组的表和索引的总大小为 1GB，则需要额外的 2GB 磁盘空间。

安装

pg_repack 可以在 UNIX 或 Linux 上使用make构建。在构建之前，您可能需要安装 PostgreSQL 开发包（postgresql-devel等）并将包含pg_config的目录添加到您的$PATH。然后您可以运行：

dnf install -y lz4-develdnf install -y readline-develdnf install -y clang sudo dnf install -y llvm llvm-develwget -c 'https://github.com/reorg/pg_repack/archive/refs/tags/ver_1.5.2.zip'cd /opt unzip  pg_repack-ver_1.5.2.zip make PG_CONFIG={your pg_config path}make PG_CONFIG={your pg_config path}  install #　安装后，在你要处理的数据库中加载 pg_repack 扩展。pg_repack 被打包为一个扩展，因此你可以执行：$ psql -c "CREATE EXTENSION pg_repack" -d your_databasetest=# select repack.version(), repack.version_sql();version     |   version_sql   
-----------------+-----------------pg_repack 1.5.2 | pg_repack 1.5.2
(1 row)

pgstattuple

pgstattuple 提供了一个非常详细的表和索引空间使用统计信息，尤其是在处理表膨胀和空间碎片时非常有用。通过这些信息，您可以有效地评估表的膨胀率，并决定是否需要执行 VACUUM FULL 或 pg_repack 等操作来回收空间并优化数据库性能。

test=# CREATE EXTENSION IF NOT EXISTS pgstattuple;
CREATE EXTENSIONtest=# select * from pg_extension ;oid  |     extname      | extowner | extnamespace | extrelocatable | extversion | extconfig | extcondition 
-------+------------------+----------+--------------+----------------+------------+-----------+--------------14528 | plpgsql          |       10 |           11 | f              | 1.0        |           | 16397 | pg_repack        |       10 |         2200 | f              | 1.5.2      |           | 16851 | pgstattuple      |       10 |         2200 | t              | 1.5        |           | 
(3 rows)

测试


### 关闭 autovacuum -- 仅测试 test=# alter system set autovacuum=off; 
ALTER SYSTEMtest=# select pg_reload_conf();pg_reload_conf 
----------------t
(1 row)test=# show autovacuum;autovacuum 
------------off
(1 row)#### 创建数据库表CREATE TABLE big_table (id SERIAL PRIMARY KEY,name varchar(100),value INT
);CREATE INDEX idx_name on big_table(name);#### 插入测试数据#  插入 300 万行数据
# -- 生成 Name 1, Name 2, Name 3... 的格式
# -- 生成一个在 1 到 100 之间的值INSERT INTO big_table (name, value)
SELECT'Name ' || gs,         (gs % 100) + 1        
FROM generate_series(1, 3000000) gs;# 查看表的大小和死行数量test=# SELECT * FROM pg_stat_user_tables WHERE relname = 'big_table'\gx
-[ RECORD 1 ]-------+----------
relid               | 16929
schemaname          | public
relname             | big_table
seq_scan            | 2
seq_tup_read        | 0
idx_scan            | 0
idx_tup_fetch       | 0
n_tup_ins           | 3000000
n_tup_upd           | 0
n_tup_del           | 0
n_tup_hot_upd       | 0
n_live_tup          | 3000000
n_dead_tup          | 0
n_mod_since_analyze | 3000000
n_ins_since_vacuum  | 3000000
last_vacuum         | 
last_autovacuum     | 
last_analyze        | 
last_autoanalyze    | 
vacuum_count        | 0
autovacuum_count    | 0
analyze_count       | 0
autoanalyze_count   | 0# 查看表的空间大小、索引大小，总大小
test=# SELECT
test-#      pg_size_pretty(pg_table_size('big_table')) AS table_size,
test-#      pg_size_pretty(pg_indexes_size('big_table')) AS indexes_size,
test-#      pg_size_pretty(pg_total_relation_size('big_table')) AS total_size;table_size | indexes_size | total_size 
------------+--------------+------------149 MB     | 219 MB       | 369 MB# 查询默认表空间路径（pg_default）中的物理文件test=# SELECT oid, datname FROM pg_database WHERE datname = 'test';oid  | datname 
-------+---------16389 | test
(1 row)test=# SELECT pg_relation_filepath('big_table');pg_relation_filepath 
----------------------base/16389/16929
(1 row)# 物理文件空间大小ls -lth |grep 16929
-rw-------. 1 postgres postgres 150M Feb 14 16:23 16929
-rw-------. 1 postgres postgres 56K Feb 14 16:22 16929_fsm# 这个查询会返回有关指定表的空间使用统计信息。test=# SELECT * FROM pgstattuple('big_table');table_len | tuple_count | tuple_len | tuple_percent | dead_tuple_count | dead_tuple_len | dead_tuple_percent | free_space | free_percent 
-----------+-------------+-----------+---------------+------------------+----------------+--------------------+------------+--------------156540928 |     3000000 | 139999608 |         89.43 |                0 |              0 |                  0 |       6668 |            0
(1 row)#### 执行一些删除和更新操作#为了模拟数据库的实际负载，删除和更新一些行，产生死行和空洞：# 删除一些数据 33%
test=# DELETE FROM big_table WHERE id%3 = 0; 
DELETE 1000000test=# select count(*) from big_table;count  
---------2000000
(1 row)test=# SELECT * FROM pg_stat_user_tables WHERE relname = 'big_table'\gx
-[ RECORD 1 ]-------+----------
relid               | 16929
schemaname          | public
relname             | big_table
seq_scan            | 8
seq_tup_read        | 11000000
idx_scan            | 0
idx_tup_fetch       | 0
n_tup_ins           | 3000000
n_tup_upd           | 0
n_tup_del           | 1000000   # 被删除的行数 
n_tup_hot_upd       | 0
n_live_tup          | 2000000
n_dead_tup          | 1000000   # 产生死行的数量
n_mod_since_analyze | 4000000
n_ins_since_vacuum  | 3000000
last_vacuum         | 
last_autovacuum     | 
last_analyze        | 
last_autoanalyze    | 
vacuum_count        | 0
autovacuum_count    | 0
analyze_count       | 0
autoanalyze_count   | 0test=# SELECT * FROM pgstattuple('big_table');table_len | tuple_count | tuple_len | tuple_percent | dead_tuple_count | dead_tuple_len | dead_tuple_percent | free_space | free_percent 
-----------+-------------+-----------+---------------+------------------+----------------+--------------------+------------+--------------156540928 |     2000000 |  93333072 |         59.62 |               10 |            480 |                  0 |   48005924 |        30.67
(1 row)# 查询 物理文件是否发生变化ls -lth |grep 16929
-rw-------. 1 postgres postgres 150M Feb 14 16:37 16929
-rw-------. 1 postgres postgres   56K Feb 14 16:22 16929_fsm# 更新一些数据test=# UPDATE big_table SET name = name || 'updated' WHERE id%3 = 1; 
UPDATE 1000000test=# select count(*) from big_table;count  
---------2000000
(1 row)test=# SELECT * FROM pg_stat_user_tables WHERE relname = 'big_table'\gx
-[ RECORD 1 ]-------+----------
relid               | 16929
schemaname          | public
relname             | big_table
seq_scan            | 15
seq_tup_read        | 21000020
idx_scan            | 0
idx_tup_fetch       | 0
n_tup_ins           | 3000000
n_tup_upd           | 1000000  # 已更新的行
n_tup_del           | 1000000
n_tup_hot_upd       | 0
n_live_tup          | 2000000
n_dead_tup          | 2000000  # 产生死行的数量 1000000+1000000=2000000 ( n_tup_del + n_tup_upd) 
n_mod_since_analyze | 5000000
n_ins_since_vacuum  | 3000000
last_vacuum         | 
last_autovacuum     | 
last_analyze        | 
last_autoanalyze    | 
vacuum_count        | 0
autovacuum_count    | 0
analyze_count       | 0
autoanalyze_count   | 0# 检查数据、索引文件大小 test=# SELECT * FROM pgstattuple('big_table');table_len | tuple_count | tuple_len | tuple_percent | dead_tuple_count | dead_tuple_len | dead_tuple_percent | free_space | free_percent 
-----------+-------------+-----------+---------------+------------------+----------------+--------------------+------------+--------------169197568 |     2000000 |  98665204 |         58.31 |                0 |              0 |                  0 |   48622184 |        28.74
(1 row)# 查询 物理文件是否发生变化ls -lth |grep 16929
-rw-------. 1 postgres postgres 162M Feb 14 16:44 16929
-rw-------. 1 postgres postgres  64K Feb 14 16:39 16929_fsm#运行 pg_repack 来重建表。这将移除死行并优化表：pg_repack -d test --table public.big_table --echo-------------------  日志如下  ------------------
LOG: (query) SET search_path TO pg_catalog, pg_temp, public
LOG: (query) SET search_path TO pg_catalog, pg_temp, public
LOG: (query) select repack.version(), repack.version_sql()
LOG: (query) SET statement_timeout = 0
LOG: (query) SET search_path = pg_catalog, pg_temp, public
LOG: (query) SET client_min_messages = warning
LOG: (query) SELECT r FROM (VALUES ($1, 'r')) AS given_t(r,kind) WHERE NOT EXISTS(  SELECT FROM repack.tables WHERE relid=to_regclass(given_t.r)) AND NOT EXISTS(  SELECT FROM pg_catalog.pg_class c WHERE c.oid=to_regclass(given_t.r) AND c.relkind = given_t.kind AND given_t.kind = 'p')
LOG:    (param:0) = public.big_table
LOG: (query) SELECT t.*, coalesce(v.tablespace, t.tablespace_orig) as tablespace_dest FROM repack.tables t,  (VALUES ($1::text)) as v (tablespace) WHERE (relid = $2::regclass) ORDER BY t.relname, t.schemaname
LOG:    (param:0) = (null)
LOG:    (param:1) = public.big_table
INFO: repacking table "public.big_table"
LOG: (query) SELECT pg_try_advisory_lock($1, CAST(-2147483648 + $2::bigint AS integer))
LOG:    (param:0) = 16185446
LOG:    (param:1) = 16929
LOG: (query) BEGIN ISOLATION LEVEL READ COMMITTED
LOG: (query) SET LOCAL lock_timeout = 100
LOG: (query) LOCK TABLE public.big_table IN ACCESS EXCLUSIVE MODE
LOG: (query) RESET lock_timeout
LOG: (query) SELECT pg_get_indexdef(indexrelid) FROM pg_index WHERE indrelid = $1 AND NOT indisvalid
LOG:    (param:0) = 16929
LOG: (query) SELECT indexrelid, repack.repack_indexdef(indexrelid, indrelid, $2, FALSE)  FROM pg_index WHERE indrelid = $1 AND indisvalid
LOG:    (param:0) = 16929
LOG:    (param:1) = (null)
LOG: (query) SELECT repack.conflicted_triggers($1)
LOG:    (param:0) = 16929
LOG: (query) SELECT repack.create_index_type(16933,16929)
LOG: (query) SELECT repack.create_log_table(16929)
LOG: (query) CREATE TRIGGER repack_trigger AFTER INSERT OR DELETE OR UPDATE ON public.big_table FOR EACH ROW EXECUTE PROCEDURE repack.repack_trigger('id')
LOG: (query) ALTER TABLE public.big_table ENABLE ALWAYS TRIGGER repack_trigger
LOG: (query) SELECT repack.disable_autovacuum('repack.log_16929')
LOG: (query) BEGIN ISOLATION LEVEL READ COMMITTED
LOG: (query) SELECT pg_backend_pid()
LOG: (query) SELECT pid FROM pg_locks WHERE locktype = 'relation' AND granted = false AND relation = 16929 AND mode = 'AccessExclusiveLock' AND pid <> pg_backend_pid()
LOG: (query) COMMIT
LOG: (query) BEGIN ISOLATION LEVEL SERIALIZABLE
LOG: (query) SELECT set_config('work_mem', current_setting('maintenance_work_mem'), true)
LOG: (query) SELECT coalesce(array_agg(l.virtualtransaction), '{}')   FROM pg_locks AS l   LEFT JOIN pg_stat_activity AS a     ON l.pid = a.pid   LEFT JOIN pg_database AS d     ON a.datid = d.oid   WHERE l.locktype = 'virtualxid'   AND l.pid NOT IN (pg_backend_pid(), $1)   AND (l.virtualxid, l.virtualtransaction) <> ('1/1', '-1/0')   AND (a.application_name IS NULL OR a.application_name <> $2)  AND a.query !~* E'^\\s*vacuum\\s+'   AND a.query !~ E'^autovacuum: '   AND ((d.datname IS NULL OR d.datname = current_database()) OR l.database = 0)
LOG:    (param:0) = 12616
LOG:    (param:1) = pg_repack
LOG: (query) DELETE FROM repack.log_16929
LOG: (query) SAVEPOINT repack_sp1
LOG: (query) SELECT pid FROM pg_locks WHERE locktype = 'relation' AND granted = false AND relation = 16929 AND mode = 'AccessExclusiveLock' AND pid <> pg_backend_pid()
LOG: (query) SET LOCAL lock_timeout = 100
LOG: (query) LOCK TABLE public.big_table IN ACCESS SHARE MODE
LOG: (query) RESET lock_timeout
LOG: (query) SELECT repack.create_table($1, $2)
LOG:    (param:0) = 16929
LOG:    (param:1) = pg_default
LOG: (query) INSERT INTO repack.table_16929 SELECT id,name,value FROM ONLY public.big_table
LOG: (query) SELECT repack.disable_autovacuum('repack.table_16929')
LOG: (query) COMMIT
LOG: (query) SELECT 'repack.table_16929'::regclass::oid
LOG: (query) CREATE UNIQUE INDEX index_16933 ON repack.table_16929 USING btree (id)
LOG: (query) CREATE INDEX index_16935 ON repack.table_16929 USING btree (name)
LOG: (query) SELECT repack.repack_apply($1, $2, $3, $4, $5, $6)
LOG:    (param:0) = SELECT * FROM repack.log_16929 ORDER BY id LIMIT $1
LOG:    (param:1) = INSERT INTO repack.table_16929 VALUES ($1.*)
LOG:    (param:2) = DELETE FROM repack.table_16929 WHERE (id) = ($1.id)
LOG:    (param:3) = UPDATE repack.table_16929 SET (id, name, value) = ($2.id, $2.name, $2.value) WHERE (id) = ($1.id)
LOG:    (param:4) = DELETE FROM repack.log_16929 WHERE id IN (
LOG:    (param:5) = 1000
LOG: (query) SELECT pid FROM pg_locks WHERE locktype = 'virtualxid' AND pid <> pg_backend_pid() AND virtualtransaction = ANY($1)
LOG:    (param:0) = {}
LOG: (query) SAVEPOINT repack_sp1
LOG: (query) SET LOCAL lock_timeout = 100
LOG: (query) LOCK TABLE public.big_table IN ACCESS EXCLUSIVE MODE
LOG: (query) RESET lock_timeout
LOG: (query) SAVEPOINT repack_sp1
LOG: (query) SET LOCAL lock_timeout = 100
LOG: (query) LOCK TABLE repack.table_16929 IN ACCESS EXCLUSIVE MODE
LOG: (query) RESET lock_timeout
LOG: (query) SELECT repack.repack_apply($1, $2, $3, $4, $5, $6)
LOG:    (param:0) = SELECT * FROM repack.log_16929 ORDER BY id LIMIT $1
LOG:    (param:1) = INSERT INTO repack.table_16929 VALUES ($1.*)
LOG:    (param:2) = DELETE FROM repack.table_16929 WHERE (id) = ($1.id)
LOG:    (param:3) = UPDATE repack.table_16929 SET (id, name, value) = ($2.id, $2.name, $2.value) WHERE (id) = ($1.id)
LOG:    (param:4) = DELETE FROM repack.log_16929 WHERE id IN (
LOG:    (param:5) = 0
LOG: (query) SELECT repack.repack_swap($1)
LOG:    (param:0) = 16929
LOG: (query) COMMIT
LOG: (query) BEGIN ISOLATION LEVEL READ COMMITTED
LOG: (query) SAVEPOINT repack_sp1
LOG: (query) SET LOCAL lock_timeout = 100
LOG: (query) LOCK TABLE public.big_table IN ACCESS EXCLUSIVE MODE
LOG: (query) RESET lock_timeout
LOG: (query) SELECT repack.repack_drop($1, $2)
LOG:    (param:0) = 16929
LOG:    (param:1) = 4
LOG: (query) COMMIT
LOG: (query) BEGIN ISOLATION LEVEL READ COMMITTED
LOG: (query) ANALYZE public.big_table
LOG: (query) COMMIT
LOG: (query) SELECT pg_advisory_unlock($1, CAST(-2147483648 + $2::bigint AS integer))
LOG:    (param:0) = 16185446
LOG:    (param:1) = 16929# 创建触发器跟踪新表--旧表test=# \d big_tableTable "public.big_table"Column |          Type          | Collation | Nullable |                Default                
--------+------------------------+-----------+----------+---------------------------------------id     | integer                |           | not null | nextval('big_table_id_seq'::regclass)name   | character varying(100) |           |          | value  | integer                |           |          | 
Indexes:"big_table_pkey" PRIMARY KEY, btree (id)"idx_name" btree (name)
Triggers firing always:repack_trigger AFTER INSERT OR DELETE OR UPDATE ON big_table FOR EACH ROW EXECUTE FUNCTION repack.repack_trigger('id')# 创建新表
test=# set search_path=public,repack;
SETtest=# \dList of relationsSchema |       Name        |   Type   |    Owner     
--------+-------------------+----------+--------------public | big_table         | table    | postgrespublic | big_table_id_seq  | sequence | postgrespublic | student           | table    | postgrespublic | test_table        | table    | postgrespublic | test_table_id_seq | sequence | postgresrepack | log_16929         | table    | postgres   # 创建日志表：  log_[relid], 新数据记录到这张表 repack | log_16929_id_seq  | sequence | postgresrepack | primary_keys      | view     | postgresrepack | table_16929       | table    | postgres   # 创建临时表：  表名_[relid]repack | tables            | view     | postgres
(10 rows)# 产生新的数据test=# select * from log_16929;id |    pk     |               row                
----+-----------+----------------------------------1 |           | (3000004,"name inserted",99)2 | (367871)  | (367871,"Name 367871updated",72)3 | (3000004) | 4 |           | (3000005,"name inserted",99)5 | (3000005) | 
(5 rows)test=# SELECT pg_relation_filepath('big_table');pg_relation_filepath 
----------------------base/16389/16949
(1 row)# 查询 物理文件是否发生变化ls -lth |grep 16949
-rw-------. 1 postgres postgres 108M Feb 14 16:52 16949
-rw-------. 1 postgres postgres  48K Feb 14 16:52 16949_fsm# 检查膨胀率  test=# SELECT * FROM pgstattuple('big_table');table_len | tuple_count | tuple_len | tuple_percent | dead_tuple_count | dead_tuple_len | dead_tuple_percent | free_space | free_percent 
-----------+-------------+-----------+---------------+------------------+----------------+--------------------+------------+--------------112607232 |     2000000 |  98665260 |         87.62 |                8 |            384 |                  0 |     229804 |          0.2
(1 row)

重建表总结：

原理介绍

pg_repack插件支持对全表和索引进行repack操作。
对全表进行repack的实现原理如下：

1.创建日志表，记录repack期间对原表的变更。
2.在原表上创建触发器，将原表的INSERT、UPDATE和DELETE操作记录到日志表中。
3.创建原表结构相同的新表并将原表数据导入其中。
4.在新表中创建与原表相同的索引。
5.将日志表里的变更（即repack期间表上产生的增量数据）应用到新表。
6.在系统catalog交换新旧表。
7.删除旧表。