本帖最后由 ccton 于 2014-2-18 12:08 编辑
[root@**** hydata]# cat /etc/redhat-release
Red Hat Enterprise Linux Server release 5.6 (Tikanga)
[root@**** hydata]# uname -a
Linux gywsj.hyb210 2.6.18-238.el5 #1 SMP Sun Dec 19 14:22:44 EST 2010 x86_64 x86_64 x86_64 GNU/Linux
数据库版本：
Oracle Database 11g Enterprise Edition Release 11.2.0.1.0 - 64bit Production
问题描述，运行一段时间后出现挂载阵列的文件系统会逻辑错误，自动变为只读
另外，曾经用循环批量写入大文件，将磁盘写满也未报过错误，重新mount后写文件也正常
我怀疑是阵列的电压不稳定导致磁盘逻辑块错误，或者是ORACLE bug，但未找到相关资料证明
请各位高手帮忙诊断下
下面是相关日志
数据库日志：
Tue Feb 18 09:53:28 2014
Archived Log entry 18018 added for thread 1 sequence 565 ID 0x51475291 dest 1:
Tue Feb 18 10:19:41 2014
Errors in file /u01/app/oracle/diag/rdbms/orcl/orcl/trace/orcl_ckpt_30996.trc:
ORA-00206: 写入控制文件时出错 (块 3, # 块 1)
ORA-00202: 控制文件: ''/hydata/flash_recovery_area/orcl/control02.ctl''
ORA-27072: 文件 I/O 错误
Linux-x86_64 Error: 30: Read-only file system
Additional information: 4
Additional information: 3
Additional information: -1
Tue Feb 18 10:19:41 2014
KCF: read, write or open error, block=0xaa13a online=1
Tue Feb 18 10:19:41 2014
KCF: read, write or open error, block=0xa5dfd online=1
file=5 '/hydata/tablespaces/cmsservergy.dat'
Errors in file /u01/app/oracle/diag/rdbms/orcl/orcl/trace/orcl_ckpt_30996.trc:
ORA-00221: 写入控制文件时出错
ORA-00206: 写入控制文件时出错 (块 3, # 块 1)
ORA-00202: 控制文件: ''/hydata/flash_recovery_area/orcl/control02.ctl''
ORA-27072: 文件 I/O 错误
Linux-x86_64 Error: 30: Read-only file system
Additional information: 4
Additional information: 3
Additional information: -1
file=5 '/hydata/tablespaces/cmsservergy.dat'
Tue Feb 18 10:19:41 2014
KCF: read, write or open error, block=0x21593e online=1
error=27072 txt: 'Linux-x86_64 Error: 30: Read-only file system
CKPT (ospid: 30996): terminating the instance due to error 221
file=10 '/hydata/tablespaces/cmsservergy4.dat'
error=27072 txt: 'Linux-x86_64 Error: 30: Read-only file system
Additional information: 4
error=27072 txt: 'Linux-x86_64 Error: 30: Read-only file system
Additional information: 4
Tue Feb 18 10:19:41 2014
KCF: read, write or open error, block=0x153877 online=1
Additional information: 4
Additional information: 696634
file=10 '/hydata/tablespaces/cmsservergy4.dat'
Additional information: 2185534
Additional information: 679421
Additional information: -1'
error=27072 txt: 'Linux-x86_64 Error: 30: Read-only file system
Additional information: -1'
Additional information: -1'
Additional information: 4
Additional information: 1390711
Additional information: -1'
Tue Feb 18 10:19:41 2014
Some DDE async actions failed or were cancelled
Errors in file /u01/app/oracle/diag/rdbms/orcl/orcl/trace/orcl_lgwr_30992.trc:
ORA-00345: 重做日志写入块 193051 计数 13 出错
ORA-00312: 联机日志 2 线程 1: '/hydata/orcl/redo02.log'
ORA-27072: 文件 I/O 错误
Linux-x86_64 Error: 5: Input/output error
Additional information: 4
Additional information: 193051
Additional information: -1
Tue Feb 18 10:19:42 2014
opiodr aborting process unknown ospid (11121) as a result of ORA-1092
Tue Feb 18 10:19:42 2014
ORA-1092 : opitsk aborting process
Instance terminated by CKPT, pid = 30996
操作系统日志：
Feb 18 10:19:05 gywsj kernel: INFO: task extract:32604 blocked for more than 120 seconds.
Feb 18 10:19:05 gywsj kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Feb 18 10:19:05 gywsj kernel: extract D ffffffff80153806 0 32604 9994 32577 (NOTLB)
Feb 18 10:19:05 gywsj kernel: ffff8101d9957b78 0000000000000082 ffff810001059800 0000000000000000
Feb 18 10:19:05 gywsj kernel: ffffffff804d3480 000000000000000a ffff8102c9a5e080 ffff81087fffb080
Feb 18 10:19:05 gywsj kernel: 00067b698e9d15a4 000000000000322f ffff8102c9a5e268 0000001a00000000
Feb 18 10:19:05 gywsj kernel: Call Trace:
Feb 18 10:19:05 gywsj kernel: [] do_gettimeofday+0x40/0x90
Feb 18 10:19:05 gywsj kernel: [] sync_page+0x0/0x43
Feb 18 10:19:05 gywsj kernel: [] io_schedule+0x3f/0x67
Feb 18 10:19:05 gywsj kernel: [] sync_page+0x3e/0x43
Feb 18 10:19:05 gywsj kernel: [] __wait_on_bit+0x40/0x6e
Feb 18 10:19:05 gywsj kernel: [] wait_on_page_bit+0x6c/0x72
Feb 18 10:19:05 gywsj kernel: [] wake_bit_function+0x0/0x23
Feb 18 10:19:05 gywsj kernel: [] pagevec_lookup_tag+0x1a/0x21
Feb 18 10:19:05 gywsj kernel: [] wait_on_page_writeback_range+0x62/0x12e
Feb 18 10:19:05 gywsj kernel: [] do_writepages+0x29/0x2f
Feb 18 10:19:05 gywsj kernel: [] __filemap_fdatawrite_range+0x50/0x5b
Feb 18 10:19:05 gywsj kernel: [] filemap_write_and_wait+0x26/0x31
Feb 18 10:19:05 gywsj kernel: [] generic_file_direct_IO+0x81/0x122
Feb 18 10:19:05 gywsj kernel: [] __generic_file_aio_read+0xb8/0x198
Feb 18 10:19:05 gywsj kernel: [] generic_file_aio_read+0x34/0x39
Feb 18 10:19:05 gywsj kernel: [] do_sync_read+0xc7/0x104
Feb 18 10:19:05 gywsj kernel: [] autoremove_wake_function+0x0/0x2e
Feb 18 10:19:05 gywsj kernel: [] hrtimer_cancel+0xc/0x16
Feb 18 10:19:05 gywsj kernel: [] hrtimer_nanosleep+0x58/0x118
Feb 18 10:19:05 gywsj kernel: [] vfs_read+0xcb/0x171
Feb 18 10:19:05 gywsj kernel: [] sys_read+0x45/0x6e
Feb 18 10:19:05 gywsj kernel: [] tracesys+0xd5/0xe0
Feb 18 10:19:05 gywsj kernel:
Feb 18 10:19:41 gywsj kernel: sd 3:0:0:1: timing out command, waited 360s
Feb 18 10:19:41 gywsj kernel: sd 3:0:0:1: SCSI error: return code = 0x060d0000
Feb 18 10:19:41 gywsj kernel: end_request: I/O error, dev sdc, sector 662064886
Feb 18 10:19:41 gywsj kernel: Buffer I/O error on device sdc5, logical block 82758095
Feb 18 10:19:41 gywsj kernel: lost page write due to I/O error on sdc5
Feb 18 10:19:41 gywsj kernel: Buffer I/O error on device sdc5, logical block 82758096
Feb 18 10:19:41 gywsj kernel: lost page write due to I/O error on sdc5
Feb 18 10:19:41 gywsj kernel: Aborting journal on device sdc5.
Feb 18 10:19:41 gywsj kernel: ext3_abort called.
Feb 18 10:19:41 gywsj kernel: EXT3-fs error (device sdc5): ext3_journal_start_sb: Detected aborted journal
Feb 18 10:19:41 gywsj kernel: Remounting filesystem read-only
--重新mount后可以写入文件
操作系统日志：
Feb 18 10:47:35 gywsj kernel: __journal_remove_journal_head: freeing b_frozen_data
Feb 18 10:47:35 gywsj last message repeated 2 times
Feb 18 10:47:35 gywsj kernel: ext3_abort called.
Feb 18 10:47:35 gywsj kernel: EXT3-fs error (device sdc5): ext3_put_super: Couldn't clean up the journal
Feb 18 10:47:51 gywsj kernel: kjournald starting. Commit interval 5 seconds
Feb 18 10:47:51 gywsj kernel: EXT3-fs warning (device sdc5): ext3_clear_journal_err: Filesystem error recorded from previous mount: IO failure
Feb 18 10:47:51 gywsj kernel: EXT3-fs warning (device sdc5): ext3_clear_journal_err: Marking fs in need of filesystem check.
Feb 18 10:47:51 gywsj kernel: EXT3-fs warning: mounting fs with errors, running e2fsck is recommended
Feb 18 10:47:51 gywsj kernel: EXT3 FS on sdc5, internal journal
Feb 18 10:47:51 gywsj kernel: EXT3-fs: recovery complete.
Feb 18 10:47:51 gywsj kernel: EXT3-fs: mounted filesystem with ordered data mode.
)
