Oracle Exadata X7-2掉电宕机导致集群无法启动处理过程

文章目录

前言
一、当前的状态是什么？
二、集群启动异常怀疑对象
- 1.排查心跳网络异常
- - ping自己私有IP延迟高
  - ping其它主机私有IP不通
- 2.是否发生过重启
三、日志信息收集
- ocssd.trc
- 集群crs日志
- cell的griddisk状态及报错
四、IB交换机的问题排查处理
五、紧急恢复业务
- 在IB完成正常重启后，重新启动所有cell服务
- 拉起集群：
六、收尾工作
- check修复第二台IB交换机
- 重新挂载nfs共享目录
- 检查PDU，确实已掉电
七、原因调查
- PDU问题由于29日晚操作切电操作导致UPS路跳闸
- 主机等log显示电源切换
- 29日有检测到FAN0风扇数值是0
总结

前言

客户突然联系说应用无法连接数据库，报错如下：

[ERROR]-[Thread: Druid-ConnectionPool-Create-26728049]-[com.alibaba.druid.pool.DruidDataSource$CreateConnectionThread.run()]: create connection error, url: jdbc:oracle:thin:@x.x.x.93:1521:empdb011, errorCode 17002, state 08006
java.sql.SQLRecoverableException: IO 错误: The Network Adapter could not establish the connectionat oracle.jdbc.driver.T4CConnection.logon(T4CConnection.java:774)at oracle.jdbc.driver.PhysicalConnection.connect(PhysicalConnection.java:688)at oracle.jdbc.driver.T4CDriverExtension.getConnection(T4CDriverExtension.java:39)at oracle.jdbc.driver.OracleDriver.connect(OracleDriver.java:691)at com.alibaba.druid.filter.FilterChainImpl.connection_connect(FilterChainImpl.java:148)at com.alibaba.druid.filter.stat.StatFilter.connection_connect(StatFilter.java:220)at com.alibaba.druid.filter.FilterChainImpl.connection_connect(FilterChainImpl.java:142)at com.alibaba.druid.filter.FilterAdapter.connection_connect(FilterAdapter.java:785)at com.alibaba.druid.filter.FilterChainImpl.connection_connect(FilterChainImpl.java:142)at com.alibaba.druid.pool.DruidAbstractDataSource.createPhysicalConnection(DruidAbstractDataSource.java:1463)at com.alibaba.druid.pool.DruidAbstractDataSource.createPhysicalConnection(DruidAbstractDataSource.java:1525)at com.alibaba.druid.pool.DruidDataSource$CreateConnectionThread.run(DruidDataSource.java:2100)
Caused by: oracle.net.ns.NetException: The Network Adapter could not establish the connectionat oracle.net.nt.ConnStrategy.execute(ConnStrategy.java:523)at oracle.net.resolver.AddrResolution.resolveAndExecute(AddrResolution.java:521)at oracle.net.ns.NSProtocol.establishConnection(NSProtocol.java:660)at oracle.net.ns.NSProtocol.connect(NSProtocol.java:286)at oracle.jdbc.driver.T4CConnection.connect(T4CConnection.java:1438)at oracle.jdbc.driver.T4CConnection.logon(T4CConnection.java:518)... 11 more
Caused by: java.io.IOException: Connection timed out: connect, socket connect lapse 20998 ms. /x.x.x.93 1521 0 1 trueat ora