linux oracle10.2.0.1 lsnrctl无法启动

故障现象：lsnrctl stat后即无任何反应，而SQLPLUS正常：

[oracle@wlserver3 ~]$ lsnrctl stat

LSNRCTL for Linux: Version 10.2.0.1.0 - Production on 13-SEP-2011 23:16:11

Connecting to (DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=localhost.localdomain)(PORT=1521)))

尝试ps -ef|grep lsnrctl后kill -9 杀进程，但杀后重启时也碰到同样问题

尝试ps -ef|grep lsnr（注意和上述命令的区别少了ctl）后kill -9 杀进程，后重启成功

[root@wlserver3 admin]# ps -ef|grep lsnr

oracle 3067 16807 0 Sep10 ? 00:00:01 /oracle/product/10.2.0/db_1/bin/tnslsnr LISTENER -inherit

oracle 16807 1 0 Jul08 ? 00:02:07 /oracle/product/10.2.0/db_1/bin/tnslsnr LISTENER -inherit

oracle 18378 18350 0 Sep13 pts/1 00:00:00 lsnrctl stat

oracle 18386 18350 0 Sep13 pts/1 00:00:00 lsnrctl stat

oracle 18423 18350 0 Sep13 pts/1 00:00:00 lsnrctl stat

root 19686 18480 0 01:18 pts/3 00:00:00 grep lsnr

[root@wlserver3 admin]# kill -9 3067

[root@wlserver3 admin]# kill -9 16807

重启成功:

oracle@wlserver3 bdump]$ lsnrctl start

LSNRCTL for Linux: Version 10.2.0.1.0 - Production on 14-SEP-2011 01:21:35

Starting /oracle/product/10.2.0/db_1/bin/tnslsnr: please wait...

TNSLSNR for Linux: Version 10.2.0.1.0 - Production

System parameter file is /oracle/product/10.2.0/db_1/network/admin/listener.ora

Log messages written to /oracle/product/10.2.0/db_1/network/log/listener.log

Listening on: (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=localhost.localdomain)(PORT=1521)))

Listening on: (DESCRIPTION=(ADDRESS=(PROTOCOL=ipc)(KEY=EXTPROC0)))

Connecting to (DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=localhost.localdomain)(PORT=1521)))

STATUS of the LISTENER

------------------------

Alias LISTENER

Version TNSLSNR for Linux: Version 10.2.0.1.0 - Production

Start Date 14-SEP-2011 01:21:37

Uptime 0 days 0 hr. 0 min. 0 sec

Trace Level off

Security ON: Local OS Authentication

SNMP OFF

Listener Parameter File /oracle/product/10.2.0/db_1/network/admin/listener.ora

Listener Log File /oracle/product/10.2.0/db_1/network/log/listener.log

Listening Endpoints Summary...

(DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=localhost.localdomain)(PORT=1521)))

(DESCRIPTION=(ADDRESS=(PROTOCOL=ipc)(KEY=EXTPROC0)))

Services Summary...

Service "PLSExtProc" has 1 instance(s).

Instance "PLSExtProc", status UNKNOWN, has 1 handler(s) for this service...

The command completed successfully

[oracle@wlserver3 bdump]$

补充:处理过程中看到过一个10.2.0.1环境下的BUG,也一并修复掉了

10g Listener: High CPU Utilization - Listener May Hang [ID 284602.1]

Symptoms
1) If the listener is hanging, a sqlnet trace (level 16) will show that the last operation was a fork.

[000068 28-SEP-2004 14:16:17:398] sntpcall: entry
[000068 28-SEP-2004 14:16:17:401] sntpcall: detaching from parent with additional fork 68 bytes to NS buffer

2) netstat -na reports numerous (maybe thousands) of sockets associated with port 6100 in a TIME_WAIT status

127.0.0.1.6100 127.0.0.1.55228 49152 0 49152 0 TIME_WAIT
127.0.0.1.6100 127.0.0.1.55229 49152 0 49152 0 TIME_WAIT
127.0.0.1.6100 127.0.0.1.55230 49152 0 49152 0 TIME_WAIT
127.0.0.1.6100 127.0.0.1.55231 49152 0 49152 0 TIME_WAIT
127.0.0.1.6100 127.0.0.1.55232 49152 0 49152 0 TIME_WAIT

3) "sdtprocess" (Solaris), or "top" show that the listener process (tnslsnr) is using a large amount of cpu.

ID Name Owner CPU% RAM Size Started Parent
-----------------------------------------------------------------------------------------------
21145 tnslsnr mseibt 44 9440 22368 13:36:03 1 /u01/app/oracle/product/10.1.0/bin/tnslsnr LISTENER -inherit

4) The 10g(iAS) $ORACLE_HOME/opmn/log on the same system shows the following repeating error in the ons logs:

04/10/01 13:30:43 [4] Local connection 0,127.0.0.1,6100 missing form factor
04/10/01 13:30:43 [4] Local connection 0,127.0.0.1,6100 missing form factor
04/10/01 13:30:43 [4] Local connection 0,127.0.0.1,6100 missing form factor
04/10/01 13:30:43 [4] Local connection 0,127.0.0.1,6100 missing form factor
04/10/01 13:30:43 [4] Local connection 0,127.0.0.1,6100 missing form factor

Changes
- Both 10g(iAS) and Oracle 10g(rdbms) are installed on the same system.
- Oracle 10g (non-RAC) is installed on a system with or without other Oracle installations.

Cause
The Oracle Notification Server (ONS) running in the 10g(iAS) home uses the ports defined in $ORACLE_HOME/opmn/conf/ons.config. The database listener, which is an ONS client, also utilizes the ONS ports identified in its own $ORACLE_HOME/opmn/conf/ons.config. When 10g(iAS) and 10g(rdbms) are installed on the same box, the installer will mistakenly configure identical ONS ports in both homes which creates an operational conflict when both the 10g(iAS) ONS and 10g(rdbms) listener services are running.

In non-RAC environments where Oracle 10g is the only Oracle product on the system, the existence of the ons.config may still cause a hang condition in the listener.

Solution
This issue is corrected in versions 10.2.0.3 and higher.
(or apply backport patch <<4518443>>)

You may also use either of the following 2 workarounds to address this problem:

1)  In the 10g(rdbms) home, rename the $ORACLE_HOME/opmn/conf/ons.config file (so that the listener does not find or use it).

For example:

cd $ORACLE_HOME/opmn/conf
mv ons.config ons.config.orig

(then restart the listener)


      - or -

2)  Explicitly disable the 10g TNS Listener from subscribing to the ONS.

Edit the active 10g  listener.ora file and add the parameter below:

SUBSCRIBE_FOR_NODE_DOWN_EVENT_<listener_name>=OFF

where <listener_name> would be replaced with the actual listener name configured in the listener.ora file.

For example, if the listener name is LISTENER (default), the parameter would be:

SUBSCRIBE_FOR_NODE_DOWN_EVENT_LISTENER=OFF

(then restart the listener)

References
BUG:3881276 - LISTENER HANGS INTERMITTANTLY AFTER APPLYING 10.1.0.3 PATCHSET
BUG:6017068 - PMON SPINNING IN NETWORKING CODE