oracle切换ADG后JVM组件查询报错ORA-29516处理

近期，某用户将数据库系统从EXADATA切换到普通X86 LINUX架构服务器上运行时，使用JAVA组件时报错ORA-29516: Aurora assertion failure: Assertion failure at jol.c:11157
joez mt-index botch; mt_index = 65535, vtbl_len = 12, static_len = 2

对于此报错、网络上以及ORACLE MOS，涉及的信息很少；结合实际情况分析对比，EXADATA上的JAVA版本为：java version "1.8.0_241" ；而普通X86 LINUX架构服务器上JAVA版本为openjdk version "1.8.0_242"；推测为系统JAVA版本导致的问题。

此时，我们将X86 LINUX架构服务器上JAVA版本更新为一致的非OPENJDK版本的JAVA:java version "1.8.0_241"，使变量生效，之后重启数据库，仍然未能解决问题。按照报错号ORA-29516搜索，很多文档提到设置JAVA_JIT_ENABLED参数，进行设置后alter system set JAVA_JIT_ENABLED=FALSE scope=both;并不能解决问题。

从逻辑上分析，在EXADATA上是使用DBCA监控，然后DG方式同步到X86架构的LINUX环境；DBCA在监控过程充安装组件会用到本地的一些环境、变量，而X86架构的LINUX环境存在一些差异导致无法正常运行JAVA组件；前面的人工安装JAVA来实现X86架构的LINUX环境一致的方式已经验证不可行；

按经验来看，可以考虑重装一下JVM组件；首先将数据库切回EXADATA；将ADG备库强制打开后，重装JVM组件，后续应用测试可以正常运行。

报错信息及CALL STACK：

Errors in file /u01/app/oracle/diag/rdbms/TEST1/trace/oa1_m000_38649.trc:
ORA-29516: Aurora assertion failure: Assertion failure at jol.c:11157
joez mt-index botch; mt_index = 65535, vtbl_len = 12, static_len = 2
TRACE:
Oracle Database 11g Enterprise Edition Release 11.2.0.4.0 - 64bit Production
With the Partitioning, Automatic Storage Management, OLAP, Data Mining
and Real Application Testing options
ORACLE_HOME = /u01/app/oracle/product/11.2.0.4/dbhome_1
System name:	Linux
Node name:	testadg1
Release:	4.14.35-1902.300.11.el7uek.x86_64
Version:	#2 SMP Tue Mar 17 17:11:47 PDT 2020
Machine:	x86_64
Instance name: oa1
Redo thread mounted by this instance: 1
Oracle process number: 38
Unix process pid: 38649, image: oracle@testadg1 (M000)*** 2024-03-20 20:49:16.780
*** SESSION ID:(4485.3) 2024-03-20 20:49:16.780
*** CLIENT ID:() 2024-03-20 20:49:16.780
*** SERVICE NAME:(SYS$BACKGROUND) 2024-03-20 20:49:16.780
*** MODULE NAME:(MMON_SLAVE) 2024-03-20 20:49:16.780
*** ACTION NAME:(JAVAVM JIT slave action) 2024-03-20 20:49:16.780Assertion failure at jol.c:11157
joez mt-index botch; mt_index = 65535, vtbl_len = 12, static_len = 2
----- Call Stack Trace -----
calling              call     entry                argument values in hex      
location             type     point                (? means dubious value)     
-------------------- -------- -------------------- ----------------------------
skdstdst()+41        call     kgdsdst()            000000000 ? 000000000 ?7FFC37DA6010 ? 7FFC37DA60E8 ?7FFC37DAAB90 ? 000000002 ?
ksedst1()+103        call     skdstdst()           000000000 ? 000000000 ?7FFC37DA6010 ? 7FFC37DA60E8 ?7FFC37DAAB90 ? 000000002 ?
ksedst()+39          call     ksedst1()            000000000 ? 000000001 ?7FFC37DA6010 ? 7FFC37DA60E8 ?7FFC37DAAB90 ? 000000002 ?
ioc_dump_stacktrace  call     ksedst()             000000000 ? 000000001 ?
()+25                                              7FFC37DA6010 ? 7FFC37DA60E8 ?7FFC37DAAB90 ? 000000002 ?
iocbf_dump_stacktra  call     ioc_dump_stacktrace  000000000 ? 000000001 ?
ce()+9                        ()                   7FFC37DA6010 ? 7FFC37DA60E8 ?7FFC37DAAB90 ? 000000002 ?
ioc_do_call()+278    call     iocbf_dump_stacktra  000000000 ? 000000001 ?ce()                 7FFC37DA6010 ? 7FFC37DA60E8 ?7FFC37DAAB90 ? 000000002 ?
joet_switched_env_c  call     ioc_do_call()        7FFC37DAC9E0 ? 003CDBEC8 ?
allback()+436                                      7FFC37DAB740 ? 000000000 ?7FFC37DAB2D4 ? 7FFC37DAB6C0 ?
ioct_dump_stacktrac  call     joet_switched_env_c  7FFC37DAC9E0 ? 7F6669C7BD1F ?
e()+30                        allback()            7FFC37DAB740 ? 000000000 ?7FFC37DAB2D4 ? 7FFC37DAB6C0 ?
eoc_assert_failure(  call     ioct_dump_stacktrac  7FFC37DAC9E0 ? 7F6669C7BD1F ?
)+292                         e()                  7FFC37DAB740 ? 000000000 ?7FFC37DAB2D4 ? 7FFC37DAB6C0 ?
ioei_load_IU_compil  call     eoc_assert_failure(  7FFC37DAC9E0 ? 7F6669C7BD1F ?
ed_code()+1921                )                    7FFC37DAB740 ? 000000000 ?7FFC37DAB2D4 ? 00000FFFF ?
ioesub_load_IU_comp  call     ioei_load_IU_compil  7F6669C7BD1F ? 7FFC37DAC9E0 ?
iled_code()+466               ed_code()            9FF6874F0 ?FFFFFFFFFFFFFFFF ?7FFC37DAB2D4 ? 00000FFFF ?
ioe_load_IU_compile  call     ioesub_load_IU_comp  7FFC37DAC840 ? 7FFC37DAC9E0 ?
d_code()+121                  iled_code()          9FF6874F0 ?FFFFFFFFFFFFFFFF ?7FFC37DAB2D4 ? 00000FFFF ?
joxlod_compiled_cod  call     ioe_load_IU_compile  7FFC37DAC9E0 ? 9FF6874F0 ?
e()+1710                      d_code()             0FFFFFFFF ? 002F1B882 ?7F66698E57C8 ? 000000000 ?
joxlod_()+5980       call     joxlod_compiled_cod  00C1045E0 ? 9FF6874F0 ?e()                  0FFFFFFFF ? 002F1B882 ?7F66698E57C8 ? 000000000 ?
joxldn_()+714        call     joxlod_()            00C1045E0 ? 9FF6874F0 ?0FFFFFFFF ? 002F1B882 ?000000009 ? 7F6669DF01E0 ?
joxldn()+9           call     joxldn_()            00C1045E0 ? 9FF6874F0 ?000000004 ? 000000000 ?000000009 ? 7F6669DF01E0 ?
kqlmcdlod()+759      call     joxldn()             00C1045E0 ? 9FF6874F0 ?000000004 ? 000000000 ?000000009 ? 7F6669DF01E0 ?
kqllod()+5647        call     kqlmcdlod()          9FF6874F0 ? 000000000 ?000000004 ? 000000000 ?000000009 ? 7F6669DF01E0 ?
kglobld()+1179       call     kqllod()             00C1045E0 ? 000000000 ?000000004 ? 000000000 ?000000009 ? 7F6669DF01E0 ?
kglobpn()+2431       call     kglobld()            00C1045E0 ? 9FF686050 ?7FFC37DAFDF0 ? 000000000 ?000000009 ? 7F6669DF01E0 ?
kglpim()+450         call     kglobpn()            00C1045E0 ? A1F1DC598 ?7FFC37DAFDF0 ? 000000001 ?000000009 ? 7F6669DF01E0 ?
ioc_access_IU()+231  call     kglpim()             00C1045E0 ? 7FFC37DAFDF0 ?
2                                                  7FFC37DAFDF0 ? 000000001 ?000000009 ? 7F6669DF01E0 ?
iocbf_access_IU()+4  call     ioc_access_IU()      7FFC37DB2FC0 ? 7FFC37DB0970 ?
6                                                  9FF6874F0 ? 000000001 ?000000009 ? 7F6669DF01E0 ?
ioc_do_call()+278    call     iocbf_access_IU()    7FFC37DB2FC0 ? 7FFC37DB0780 ?9FF6874F0 ? 000000001 ?000000009 ? 7F6669DF01E0 ?
joet_switched_env_c  call     ioc_do_call()        7FFC37DB2FC0 ? 003CDC9F4 ?
allback()+436                                      7FFC37DB0780 ? 000000000 ?7FFC37DB0314 ? 7FFC37DB0700 ?
ioct_access_IU()+62  call     joet_switched_env_c  7FFC37DB2FC0 ? 7F6669C7BD1F ?allback()            7FFC37DB0780 ? 000000000 ?7FFC37DB0314 ? 7FFC37DB0700 ?
eoaccess_IU()+854    call     ioct_access_IU()     7FFC37DB2FC0 ? 7F6669C7BD1F ?7FFC37DB0970 ? 9FF6874F0 ?000000000 ? 000000001 ?
jom_access_IU()+288  call     eoaccess_IU()        7F6669C7BD1F ? 7FFC37DB09F0 ?9FF6874F0 ? 000000000 ?000000000 ? 000000001 ?
jom_realize_shared_  call     jom_access_IU()      7F6669C7BD1F ? 9FF6874F0 ?
clint()+322                                        7FFC37DB09F0 ? 000000001 ?000000000 ? 000000001 ?
jom_realize_clint_r  call     jom_realize_shared_  7F6669C7BD1F ? 9FF6874F0 ?
ef()+45                       clint()              000000001 ? 000000000 ?000000000 ? 000000001 ?
jol_realize_clint_r  call     jom_realize_clint_r  7F6669C7BD1F ? 9FF6874F0 ?
ef()+43                       ef()                 000000001 ? 000000000 ?000000000 ? 000000001 ?
jol_load_super_clin  call     jol_realize_clint_r  7FFC37DB16F0 ? 9FF6874F0 ?
t()+760                       ef()                 000000001 ? 000000000 ?000000000 ? 000000001 ?
jol_load_for_execut  call     jol_load_super_clin  7FFC37DB16F0 ? 9FF6874F0 ?
ion()+23009                   t()                  000000001 ? 000000000 ?000000000 ? 000000001 ?
ioei_load_IU()+503   call     jol_load_for_execut  7FFC37DB2FC0 ? 7F6669C7BD1F ?ion()                000000000 ? 002F1B882 ?9FF689CF0 ? 000000001 ?
ioesub_load_IU()+42  call     ioei_load_IU()       7F6669C7BD1F ? 7FFC37DB2FC0 ?
3                                                  9FF689CF0 ? 000000071 ?002F1B882 ? 000000001 ?
ioe_load_IU()+97     call     ioesub_load_IU()     7FFC37DB2B60 ? 7FFC37DBA480 ?9FF689CF0 ? 000000071 ?002F1B882 ? 000000001 ?
jox00_call_loader()  call     ioe_load_IU()        7FFC37DB2FC0 ? 9FF689CF0 ?
+1031                                              000000071 ? 002F1B882 ?002F1B882 ? 000000001 ?
joxlod_()+5822       call     jox00_call_loader()  7FFC37DB2FC0 ? 00C1045E0 ?9FF689CF0 ? 9FF6830D0 ?002F1B882 ? 000000001 ?
joxldn_()+714        call     joxlod_()            00C1045E0 ? 9FF689CF0 ?9FF689CF0 ? 9FF6830D0 ?00000002A ? 7F66698661C0 ?
joxldn()+9           call     joxldn_()            00C1045E0 ? 9FF689CF0 ?000000004 ? 000000000 ?00000002A ? 7F66698661C0 ?
kqlmcdlod()+759      call     joxldn()             00C1045E0 ? 9FF689CF0 ?000000004 ? 000000000 ?00000002A ? 7F66698661C0 ?
kqllod()+5647        call     kqlmcdlod()          9FF689CF0 ? 000000000 ?000000004 ? 000000000 ?00000002A ? 7F66698661C0 ?
kglobld()+1179       call     kqllod()             00C1045E0 ? 000000000 ?000000004 ? 000000000 ?00000002A ? 7F66698661C0 ?
kglobpn()+2431       call     kglobld()            00C1045E0 ? 9FF688420 ?7FFC37DB62F0 ? 000000000 ?00000002A ? 7F66698661C0 ?
kglpim()+450         call     kglobpn()            00C1045E0 ? A1F1E0C50 ?7FFC37DB62F0 ? 000000001 ?00000002A ? 7F66698661C0 ?
ioc_access_IU()+231  call     kglpim()             00C1045E0 ? 7FFC37DB62F0 ?
2                                                  7FFC37DB62F0 ? 000000001 ?00000002A ? 7F66698661C0 ?
iocbf_access_IU()+4  call     ioc_access_IU()      7FFC37DBB1C8 ? 7FFC37DB6E70 ?
6                                                  9FF689CF0 ? 000000001 ?00000002A ? 7F66698661C0 ?
ioc_do_call()+278    call     iocbf_access_IU()    7FFC37DBB1C8 ? 7FFC37DB6C80 ?9FF689CF0 ? 000000001 ?00000002A ? 7F66698661C0 ?
joet_switched_env_c  call     ioc_do_call()        7FFC37DBB1C8 ? 003CDC9F4 ?
allback()+436                                      7FFC37DB6C80 ? 000000000 ?7FFC37DB6814 ? 7FFC37DB6C00 ?
ioct_access_IU()+62  call     joet_switched_env_c  7FFC37DBB1C8 ? 7F6669C7BD1F ?allback()            7FFC37DB6C80 ? 000000000 ?7FFC37DB6814 ? 7FFC37DB6C00 ?
eoaccess_IU()+854    call     ioct_access_IU()     7FFC37DBB1C8 ? 7F6669C7BD1F ?7FFC37DB6E70 ? 9FF689CF0 ?000000000 ? 000000001 ?
jom_access_IU()+288  call     eoaccess_IU()        7F6669C7BD1F ? 7FFC37DB6F30 ?9FF689CF0 ? 000000000 ?000000000 ? 000000001 ?
jom_active_clint_fr  call     jom_access_IU()      7F6669C7BD1F ? 9FF689CF0 ?
om_IU()+75                                         7FFC37DB6F30 ? 000000001 ?000000000 ? 000000001 ?
ioei_execute()+528   call     jom_active_clint_fr  7F6669C7BD1F ? 7F6669C7BD1F ?om_IU()              7FFC37DB6F30 ? 000000001 ?000000000 ? 000000001 ?
ioesub_EXECUTE()+60  call     ioei_execute()       7F6669C7BD1F ? 7FFC37DBB1C8 ?
1                                                  7F6669DD46A0 ? 000000001 ?000000000 ? 000000000 ?
seoa_note_stack_out  call     ioesub_EXECUTE()     7FFC37DBA4A0 ? 7FFC37DBA480 ?
side()+26                                          7F6669DD46A0 ? 000000001 ?000000000 ? 000000000 ?
ioe_execute()+259    call     seoa_note_stack_out  7FFC37DBA4A0 ? 7FFC37DBA480 ?side()               7F6669DD46A0 ? 000000001 ?000000000 ? 000000000 ?
jox_ioe_execute()+3  call     ioe_execute()        7FFC37DBB1C8 ? 7F6669DD46A0 ?
55                                                 7FFC37DBAF38 ? 7FFC37DBAC38 ?000000050 ? 000000004 ?
joxgrovel_jitsomeme  call     jox_ioe_execute()    7FFC37DBB1C8 ? 7F6669DD46A0 ?
thods()+374                                        00AA92094 ? 00000000E ?00AA9208C ? 000000004 ?
joxgroveldrv()+778   call     joxgrovel_jitsomeme  00C1045E0 ? 7FFC37DBAF40 ?thods()              00AA92094 ? 00000000E ?00AA9208C ? 000000004 ?
joxjitexe()+577      call     joxgroveldrv()       00C1045E0 ? 7FFC37DBAF40 ?00AA92094 ? 00000000E ?00AA9208C ? 000000004 ?
joxrdp_()+31         call     joxjitexe()          00C1045E0 ? 7FFC37DBAF40 ?00AA92094 ? 00000000E ?00AA9208C ? 000000004 ?
joxrdp()+19          call     joxrdp_()            00C1045E0 ? 7FFC37DBAF40 ?00AA92094 ? 00000000E ?00AA9208C ? 000000004 ?
kebm_slave_main()+5  call     joxrdp()             00C1045E0 ? 7FFC37DBAF40 ?
86                                                 00AA92094 ? 00000000E ?00AA9208C ? 000000004 ?
ksvrdp()+1766        call     kebm_slave_main()    000000000 ? A1CB8B0F0 ?000000000 ? 00000000E ?00AA9208C ? 000000004 ?
opirip()+674         call     ksvrdp()             000000000 ? A1CB8B0F0 ?000000000 ? 00000000E ?00AA9208C ? 000000004 ?
opidrv()+603         call     opirip()             000000032 ? 000000004 ?7FFC37DC0528 ? 00000000E ?00AA9208C ? 000000004 ?
sou2o()+103          call     opidrv()             000000032 ? 000000004 ?7FFC37DC0528 ? 00000000E ?00AA9208C ? 000000004 ?
opimai_real()+250    call     sou2o()              7FFC37DC0500 ? 000000032 ?000000004 ? 7FFC37DC0528 ?00AA9208C ? 000000004 ?
ssthrdmain()+265     call     opimai_real()        000000000 ? 7FFC37DC06F0 ?000000004 ? 7FFC37DC0528 ?00AA9208C ? 000000004 ?
main()+201           call     ssthrdmain()         000000003 ? 7FFC37DC06F0 ?000000001 ? 000000000 ?00AA9208C ? 000000004 ?
__libc_start_main()  call     main()               000000003 ? 7FFC37DC0890 ?
+245                                               000000001 ? 000000000 ?00AA9208C ? 000000004 ?
_start()+41          call     __libc_start_main()  000A17680 ? 000000001 ?7FFC37DC0888 ? 000000000 ?00AA9208C ? 000000004 ?--------------------- Binary Stack Dump ---------------------

2、重建JVM的步骤

参考MOS文档How to Un-install JVM Components from Database (Doc ID 3000161.1)，步骤如下：

1. Download the script moddbj.sql and remove_java.sql attached below and keep the files in a directory of the database server which will be referred as"removal working directory" later in this note.2. Create the following four temporary files inside the ORACLE_HOME using which the database is running :A. mddins_temp.sql:----------------------------a. Copy the file $ORACLE_HOME/md/admin/mddins.sql into $ORACLE_HOME/md/admin/mddins_temp.sqlb. Edit $ORACLE_HOME/md/admin/mddins_temp.sql and remove the following line, if it is present.      exec SYS.UTL_RECOMP.recomp_serial('MDSYS');Otherwise, mddins_temp.sql can remain as an unmodified copy. However, it needs to be present because other scripts refer to it.B. catnojav_temp.sql:-----------------------------a. Copy the file $ORACLE_HOME/rdbms/admin/catnojav.sql into $ORACLE_HOME/rdbms/admin/catnojav_temp.sqlb. Edit catnojav_temp.sql to remove the two linesDROP JAVA SOURCE "schedFileWatcherJava";DROP JAVA SOURCE "dbFWTrace";C. jvmrm_temp.sql--------------------------a. Copy the file $ORACLE_HOME/javavm/install/jvmrm.sql into $ORACLE_HOME/javavm/install/jvmrm_temp.sqlb. Edit jvmrm_temp.sql, identify the following block in it and move it to the end of the file.beginif :jvmrmaction = 'FULL_REMOVAL' or(:jvmrmaction = 'DOWNGRADE' andinitjvmaux.compare_releases(:jvmdownto, '12.2') = 'FIRST IS OLDER')thenexecute immediate 'drop table javanamestatus$';execute immediate 'drop table javasnm_tmp$';end if;end;/c. Identify another large block that begins withbegin if :jvmrmaction = 'FULL_REMOVAL' or(:jvmrmaction = 'DOWNGRADE' andinitjvmaux.compare_releases(:jvmdownto, '12.1') = 'FIRST IS OLDER')theninitjvmaux.drp('declare trash varchar2(100):=dbms_java.endsession_and_related_state;begin null;end;');initjvmaux.drp('drop user ojvmsys cascade');initjvmaux.drp('declare trash varchar2(100):=dbms_java.endsession_and_related_state;begin null;end;');initjvmaux.drp('drop java resource "lib/security/java.security.alt"'); Replace the last three lines shown above with the following two lines by moving them from their original position later in the block into the position where the lines to be replaced had been.-- remove entry for drop user cascadeinitjvmaux.exec('delete from duc$ where OWNER = ''SYS'' and PACK = ''OJDS_CONTEXT'' and PROC = ''USER_DROPPED'''); NOTE: The files sample_jvmrm.sql and sample_jvmrm_temp.sql attached below are an example using the 19.17 OJVMRU version of jvmrm.sql demonstrating the modifications specified above.These should not be used for anything else other than the illustration.D. rmjvm_temp.sql--------------------------a. Copy the file $ORACLE_HOME/javavm/install/rmjvm.sql into $ORACLE_HOME/javavm/install/rmjvm_temp.sqlb. Edit rmjvm_temp.sql, insert the following two linesdrop table java$jvm$steps$done;drop table java$jvm$status;immediately before the line@@initjvmac. Replace the final portion of the file starting at@@jvmrmwith@@jvmrm_tempexec initjvmaux.drp('drop package dbms_java')exec initjvmaux.drp('drop PUBLIC SYNONYM dbms_java')exec initjvmaux.drp('drop FUNCTION dbj_long_name')exec initjvmaux.drp('drop function "NameFromLastDDL"')exec initjvmaux.drp('drop PUBLIC SYNONYM "NameFromLastDDL"')exec initjvmaux.drp('drop FUNCTION dbj_short_name')exec initjvmaux.drp('drop PUBLIC SYNONYM dbj_short_name')exec initjvmaux.drp('drop package dbms_javascript')exec initjvmaux.drp('drop PUBLIC SYNONYM dbms_javascript')exec initjvmaux.drp('drop user mdsys cascade')exec initjvmaux.drp('drop user mddata cascade')drop table java$jvm$steps$done;drop table java$jvm$status;drop user ojvmsys cascade;drop package rmjvm;drop package initjvmaux;REMREM end java object removalREMNOTE: The repetition of the drop table java$jvm$* lines are intentional. The files sample_rmjvm.sql and sample_rmjvm_temp.sql attached below are an example using the 19.17 OJVMRUversion of jvmrm.sql demonstrating the modifications specified above. These should not be used for anything else other than the illustration.
3. Create the file commands.txt in the same directory where moddbj.sql and and remove_java.sql were kept in Step 1 with the following content based onthe type of database :For multitenant database
-----------------------------------$ORACLE_HOME/perl/bin/perl $ORACLE_HOME/rdbms/admin/catcon.pl -n 1 -e -b moddbj -d $REMOVAL_WORK_DIR $REMOVAL_WORK_DIR/moddbj.sql$ORACLE_HOME/perl/bin/perl $ORACLE_HOME/rdbms/admin/catcon.pl -n 1 -C 'CDB$ROOT' -e -b remove_java_pdbs -d $REMOVAL_WORK_DIR $REMOVAL_WORK_DIR/remove_java.sql$ORACLE_HOME/perl/bin/perl $ORACLE_HOME/rdbms/admin/catcon.pl -n 1 -c 'CDB$ROOT' -e -b remove_java_cdb -d $REMOVAL_WORK_DIR $REMOVAL_WORK_DIR/remove_java.sql$ORACLE_HOME/perl/bin/perl $ORACLE_HOME/rdbms/admin/catcon.pl -n 1 -e -b utlrp -d $REMOVAL_WORK_DIR $ORACLE_HOME/rdbms/admin/utlrp.sqlFor non-multitenant database
-----------------------------------------sqlplus / as sysdbaspool $REMOVAL_WORK_DIR/jvm_removal_log.txt
@$REMOVAL_WORK_DIR/moddbj.sql
@$REMOVAL_WORK_DIR/remove_java.sql
@$ORACLE_HOME/rdbms/admin/utlrp.sql
spool offexit4. Follow the instructions below that are Unix specific but should be readily adaptable to other platforms as well.A. Start a shell of one's choice such as bash or csh. The environment in this shell should be one suitable for connecting to the database one wants to modify. In particular ORACLE_HOME and PATHshould be set accordingly.B. Make sure that the database is up and running. Also check the SYS credential used in commands.txt is correct, otherwise rectify it.C. Execution of the removal commands will create log files in the removal working directory. Ensure that this directory is writable by the current user.D. Define the environment variable REMOVAL_WORK_DIR to have the value that identifies the removal working directory, either as an absolute path or as a path relative to the current workingdirectory. For example, if the removal working directory is /tmp/removejvm and one is using csh, then one would do    setenv REMOVAL_WORK_DIR /tmp/removejvmE. Execute the following to run the remove scripts and thereby do the removal:source $REMOVAL_WORK_DIR/commands.txt
NOTE: This step will take a few minutes. When it completes, the spool log should be found in the removal working directory. The log should be inspected for errors. Certain errors are expected in some situations depending on the configuration of the database. In particular any ofORA-00942
ORA-01432
ORA-02289
ORA-04043
ORA-04080
ORA-29807
ORA-29833are ignorable as long as they happen during drop commands. These errors basically report the objects to be dropped do not exist where the error number varies depending on the type of the object. Any other errors are cause for concern and Oracle support should be contacted.5. Clean UpOnce satisfied that removal has been successful, delete the temporary files created in section 1, 2 and 3.