asm磁盘组无法写入问题-处理中

有个11204的rac环境，没应用补丁，5号突然报归档满，登录环境后发现奇怪，一个1T磁盘建成的DATA磁盘组使用了近800G，读写正常，一个1.5T磁盘建成的FRA磁盘组，目前还剩余729551M，无法写入归档

手动启停实例，报错如下

SQL> startup;
ORACLE instance started.

Total System Global Area 8.5516E+10 bytes
Fixed Size            2262656 bytes
Variable Size       1.8522E+10 bytes
Database Buffers   6.6840E+10 bytes
Redo Buffers          151117824 bytes
Database mounted.
ORA-16038: log 8 sequence# 10572 cannot be archived
ORA-19502: write error on file "", block number (block size=)
ORA-00312: online log 8 thread 2:
'+DATA/orcl/onlinelog/group_8.358.1128304555'

SQL>

alert日志报错

ORA-27061: waiting for async I/Os failed
Linux-x86_64 Error: 5: Input/output error
Additional information: -1
Additional information: 1048576
ORA-19502: write error on file "+FRA/orcl/archivelog/2024_04_04/thread_2_seq_10575.128205.1165411713", block number 67584 (block size=512)
WARNING: failed to write mirror side 1 of virtual extent 179 logical extent 0 of file 128205 in group 1 on disk 0 allocation unit 845657
Errors in file /u01/app/oracle/diag/rdbms/orcl/orcl2/trace/orcl2_arc0_456667.trc:
ORA-15080: synchronous I/O operation to a disk failed
ORA-27061: waiting for async I/Os failed
Linux-x86_64 Error: 5: Input/output error
Additional information: -1
Additional information: 1048576
ORA-19502: write error on file "+FRA/orcl/archivelog/2024_04_04/thread_2_seq_10575.128205.1165411713", block number 67584 (block size=512)
WARNING: failed to write mirror side 1 of virtual extent 164 logical extent 0 of file 128205 in group 1 on disk 0 allocation unit 845642
Errors in file /u01/app/oracle/diag/rdbms/orcl/orcl2/trace/orcl2_arc0_456667.trc:
ORA-15080: synchronous I/O operation to a disk failed
ORA-27061: waiting for async I/Os failed
Linux-x86_64 Error: 5: Input/output error
Additional information: -1
Additional information: 1048576
ORA-19502: write error on file "+FRA/orcl/archivelog/2024_04_04/thread_2_seq_10575.128205.1165411713", block number 67584 (block size=512)
WARNING: failed to write mirror side 1 of virtual extent 110 logical extent 0 of file 128205 in group 1 on disk 0 allocation unit 845588
Errors in file /u01/app/oracle/diag/rdbms/orcl/orcl2/trace/orcl2_arc0_456667.trc:
ORA-15080: synchronous I/O operation to a disk failed
ORA-27061: waiting for async I/Os failed
Linux-x86_64 Error: 5: Input/output error
Additional information: -1
Additional information: 1048576
ORA-19502: write error on file "+FRA/orcl/archivelog/2024_04_04/thread_2_seq_10575.128205.1165411713", block number 67584 (block size=512)
WARNING: failed to write mirror side 1 of virtual extent 95 logical extent 0 of file 128205 in group 1 on disk 0 allocation unit 845573
Errors in file /u01/app/oracle/diag/rdbms/orcl/orcl2/trace/orcl2_arc0_456667.trc:
ORA-15080: synchronous I/O operation to a disk failed
ORA-27061: waiting for async I/Os failed
Linux-x86_64 Error: 5: Input/output error
Additional information: -1
Additional information: 1048576
ORA-19502: write error on file "+FRA/orcl/archivelog/2024_04_04/thread_2_seq_10575.128205.1165411713", block number 67584 (block size=512)
WARNING: failed to write mirror side 1 of virtual extent 48 logical extent 0 of file 128205 in group 1 on disk 0 allocation unit 845525
ARC0: I/O error 19502 archiving log 7 to '+FRA/orcl/archivelog/2024_04_04/thread_2_seq_10575.128205.1165411713'
ARCH: Archival stopped, error occurred. Will continue retrying
ORACLE Instance orcl2 - Archival Error
ORA-16038: log 7 sequence# 10575 cannot be archived
ORA-19502: write error on file "", block number (block size=)

Thu Apr 04 12:47:37 2024
WARNING: Write Failed. group:1 disk:0 AU:831879 offset:0 size:1048576
Errors in file /u01/app/oracle/diag/rdbms/orcl/orcl1/trace/orcl1_arc1_318526.trc:
ORA-15080: synchronous I/O operation to a disk failed
ORA-27061: waiting for async I/Os failed
Linux-x86_64 Error: 5: Input/output error
Additional information: -1
Additional information: 1048576
WARNING: failed to write mirror side 1 of virtual extent 166 logical extent 0 of file 129834 in group 1 on disk 0 allocation unit 831879
ARC1: Encountered disk I/O error 19502
ARC1: Closing local archive destination LOG_ARCHIVE_DEST_1: '+FRA/orcl/archivelog/2024_04_04/thread_1_seq_14255.129834.1165409257' (error 19502) (orcl1)
ARC1: I/O error 19502 archiving log 3 to '+FRA/orcl/archivelog/2024_04_04/thread_1_seq_14255.129834.1165409257'
ARCH: Archival stopped, error occurred. Will continue retrying
ORACLE Instance orcl1 - Archival Error
ORA-16038: log 3 sequence# 14255 cannot be archived
ORA-19502: write error on file "", block number (block size=)
ORA-00312: online log 3 thread 1: '+DATA/orcl/onlinelog/group_3.353.1128304551'
Thu Apr 04 12:47:37 2024
ARCH: Archival stopped, error occurred. Will continue retrying
ORACLE Instance orcl1 - Archival Error
ORA-16014: log 3 sequence# 14255 not archived, no available destinations
-----------------------------------------------------
ARC3: Archival started
ARC0: STARTING ARCH PROCESSES COMPLETE
WARNING: Write Failed. group:1 disk:0 AU:831879 offset:0 size:1048576
Errors in file /u01/app/oracle/diag/rdbms/orcl/orcl1/trace/orcl1_ora_211812.trc:
ORA-15080: synchronous I/O operation to a disk failed
ORA-27061: waiting for async I/Os failed
Linux-x86_64 Error: 5: Input/output error
Additional information: -1
Additional information: 1048576
WARNING: failed to write mirror side 1 of virtual extent 166 logical extent 0 of file 129834 in group 1 on disk 0 allocation unit 831879
ARCH: Encountered disk I/O error 19502
ARCH: Closing local archive destination LOG_ARCHIVE_DEST_1: '+FRA/orcl/archivelog/2024_04_04/thread_2_seq_10572.129834.1165409485' (error 19502) (orcl1)
ARCH: I/O error 19502 archiving log 8 to '+FRA/orcl/archivelog/2024_04_04/thread_2_seq_10572.129834.1165409485'
Abort recovery for domain 0
WARNING: Write Failed. group:1 disk:0 AU:844803 offset:0 size:1048576
Errors in file /u01/app/oracle/diag/rdbms/orcl/orcl1/trace/orcl1_arc2_212060.trc:
ORA-15080: synchronous I/O operation to a disk failed
ORA-27061: waiting for async I/Os failed
Linux-x86_64 Error: 5: Input/output error
Additional information: -1
Additional information: 1048576
WARNING: failed to write mirror side 1 of virtual extent 5 logical extent 0 of file 129836 in group 1 on disk 0 allocation unit 844803
ARC2: Encountered disk I/O error 19502
ARC2: Closing local archive destination LOG_ARCHIVE_DEST_1: '+FRA/orcl/archivelog/2024_04_04/thread_2_seq_10573.129836.1165409485' (error 19502) (orcl1)
Errors in file /u01/app/oracle/diag/rdbms/orcl/orcl1/trace/orcl1_ora_211812.trc:
ORA-16038: log 8 sequence# 10572 cannot be archived
ORA-19502: write error on file "", block number (block size=)
ORA-00312: online log 8 thread 2: '+DATA/orcl/onlinelog/group_8.358.1128304555'
ORA-16038 signalled during: ALTER DATABASE OPEN /* db agent *//* {1:33866:2} */...

[root@oracle1 trace]# strings /u01/app/oracle/diag/rdbms/orcl/orcl1/trace/orcl1_arc2_212060.trc |more
Trace file /u01/app/oracle/diag/rdbms/orcl/orcl1/trace/orcl1_arc2_212060.trc
Oracle Database 11g Enterprise Edition Release 11.2.0.4.0 - 64bit Production
With the Partitioning, Real Application Clusters, Automatic Storage Management, OLAP,
Data Mining and Real Application Testing options
ORACLE_HOME = /u01/app/oracle/product/11.2.0/db_1
System name:   Linux
Node name:   oracle1
Release:   3.10.0-957.el7.x86_64
Version:   #1 SMP Thu Nov 8 23:39:32 UTC 2018
Machine:   x86_64
Instance name: orcl1
Redo thread mounted by this instance: 1
Oracle process number: 49
Unix process pid: 212060, image: oracle@oracle1 (ARC2)
*** 2024-04-04 12:51:24.838
*** SESSION ID:(9458.19) 2024-04-04 12:51:24.838
*** CLIENT ID:() 2024-04-04 12:51:24.838
*** SERVICE NAME:() 2024-04-04 12:51:24.838
*** MODULE NAME:() 2024-04-04 12:51:24.838
*** ACTION NAME:() 2024-04-04 12:51:24.838
Initial buffer sizes: read 1024K, overflow 832K, change 805K
*** 2024-04-04 12:51:25.696
WARNING: Write Failed. group:1 disk:0 AU:844803 offset:0 size:1048576
path:/dev/asmfra1
   incarnation:0xd36c3ae2 asynchronous result:'I/O error'
   subsys:System iop:0x7fd3d9ae1178 bufp:0x7fd3d95de000 osderr:0x0 osderr1:0x0
ORA-15080: synchronous I/O operation to a disk failed
ORA-27061: waiting for async I/Os failed
Linux-x86_64 Error: 5: Input/output error
Additional information: -1
Additional information: 1048576
WARNING: failed to write mirror side 1 of virtual extent 5 logical extent 0 of file 129836 in group 1 on disk 0 allocation unit 844803
*** 2024-04-04 12:51:25.699 4329 krsh.c
ARC2: Encountered disk I/O error 19502
[root@oracle1 trace]#