AutoPatch error: The worker should not have status 'Running' or 'Restarted' at this point.

Problem:
Recently while applying a patch through apatch in Oracle EBS R12 version, the adpatch session got terminated due to network issues. 
So I thought to re-start the patch from another session but before doing that I jumped in $APPL_TOP/admin/appl/<CONTEXT_NAME>/log directory to review the log file created by adpatch session to look for any AutoPatch errors and warnings.
I see the below error in adpatch.log
AutoPatch error:
The worker should not have status 'Running' or 'Restarted' at this point.
Telling workers to quit...
All workers have quit.
AutoPatch error:
Error running SQL and EXEC commands in parallel

Cause:
Lost connectivity, did not use screen command before applying patch through adpatch.

Solution:
Before starting another adpatch session or continuing with old session, we need to address all manager that a worker failed its job or we need to clean up all the previous workers and make sure the status of workers shows quit while checking through adrctl utility.

• Start adctrl and look at the worker status. Are workers running or started? If so, from within adctrl, tell workers to quit

• Select option: Show Worker Status, tell adctrl that the worker failed and to restart

• Start applying the patch from the beginning and choose not run the previous adpatch session 

There are times when the workers appears in hung state, in such scenarios you may need to terminate the process manually as follows:

• Get the worker’s Process ID

 Unix: Issue " ps -efa | grep adworker "
After killing all the workers process which appeared to hang, you can start the adpatch again.

• Start applying the patch from the beginning and choose NOT to run the previous adpatch session

• When it prompts to delete the existing FND_INSTALL_PROCESSES table, type "Yes" 

Note: adpatch creates the FND_INSTALL_PROCESSES table, assigns each worker a unique ID, and inserts a row for each worker. This table serves as a staging area for the job information, and as a way for the manager and the worker to communicate. Once all jobs are complete, the manager tells the workers to shut down, and then drops the FND_INSTALL_PROCESSES table.
Please make sure FND_INSTALL_PROCESS should not be present before applying a patch through adpatch.

ORA-00494: enqueue [CF] held for too long (more than 900 seconds)

Problem:

For one of our client we have a backup strategy where we take a RMAN hot backup every night for all the EBS enviroment with Oracle Database 11.2.0.3.

But last night RMAN hot backup for our test environment failed or I would say got killed with the below error messages:

RMAN-03009: failure of backup command on dsk_9 channel at 03/27/2014 03:19:39 ORA-00028: your session has been killed ORA-00028: your session has been killed continuing other job steps, job failed will not be re-run

At the same time we found below error message in the alert alert log:

Thu Mar 27 03:18:11 2014 Errors in file /mnt/TSTapps/oracle/trctstdb/11.2.0/dbhome_1/log/diag/rdbms/trctst/TRCTST/trace/TRCTST_ora_70736.trc  (incident=234835): ORA-00494: enqueue [CF] held for too long (more than 900 seconds) by 'inst 1, osid 66183' Incident details in: /mnt/TSTapps/oracle/trctstdb/11.2.0/dbhome_1/log/diag/rdbms/trctst/TRCTST/incident/incdir_234835/TRCTST_ora_70736_i234835.trc

Cause:

The reason for above behavior could be high load on server or high concurrency on resources, IO waits , which keep the Oracle background processes from receiving the necessary resources.

We verified and found high IO waits on our server with sar -u 2 5 command, this probably could be the reason why kill blocker interface killed our rman hot backup process.

Solutions:

This kill blocker interface / ORA-494 was introduced in 10.2.0.4. This new mechanism will kill *any* kind of blocking process, non-background or background. If that blocking process is background process, then the instances crashes too.

 The following situation can be avoided in two ways:

1. If you want to avoid the kill of the blocker (background or non-background process), you can set.

•          _kill_controlfile_enqueue_blocker=false.    

             This means that no type of blocker will be killed anymore

2.   In order to prevent a background blocker from being killed, you can set the following init.ora parameter to 1 (default is 3).    

•          _kill_enqueue_blocker=1      

            With this parameter, if the enqueue holder is a background process, then it will not be killed, therefore the instance will not crash.

NOTE:A common root cause of an ORA-494 are very frequent loq switches as can be seen in the alert.log. In those cases the enqueue is normally held by the CKPT process.

References:

Ora-00494: Enqueue [Cf] Held For Too Long Causing Database To Crash [ID 1101862.1]
PHYSICAL: ORA-00494: enqueue [CF] held for too long after Node Crash [ID 747071.1] 

ORA-00494 Or ORA-600 [2103] During High Load After 10.2.0.4 Upgrade (Doc ID 779552.1)

Database Crashes With ORA-00494 (Doc ID 753290.1)

OPatch failed with error code 73

Problem:
While checking the inventory details in my database, I got the error below:

$ cd $ORACLE_HOME/OPatch

$ ./opatch lsinventory
Invoking OPatch 11.2.0.1.7
Oracle Interim Patch Installer version 11.2.0.1.7
Copyright (c) 2011, Oracle Corporation.  All rights reserved.

Oracle Home       : /mnt/TSTapps/oracle/trctstdb/11.2.0/dbhome_1

Central Inventory : /etc/oracle/oraInventory
   from           : /etc/oraInst.loc
OPatch version    : 11.2.0.1.7
OUI version       : 11.2.0.3.0
Log file location : /mnt/TSTapps/oracle/trctstdb/11.2.0/dbhome_1/cfgtoollogs/opatch/opatch2014-03-24_14-53-27PM.log

OPatch failed to locate Central Inventory.

Possible causes are:
    The Central Inventory is corrupted
    The oraInst.loc file specified is not valid.
LsInventorySession failed: OPatch failed to locate Central Inventory.
Possible causes are:
    The Central Inventory is corrupted
    The oraInst.loc file specified is not valid.

OPatch failed with error code 73

Possibilities: 

• The Central Inventory is corrupted
• The oraInst.loc file specified is not valid

Workaround:

·         With the help of Doc ID 295185.1, I tried to recreate the global inventory.

• Checked oraInst.loc to see the location of global inventory,

# cat /etc/oraInst.loc

inventory_loc=/etc/oracle/oraInventory

inst_group=dba

• Rename the corrupted global oraInventory with a different name

cd /etc/oracle

mv oraInventory oraInventory.ori

• Download patch p5035661_11i_LINUX.zip, while trying to apply the patch it gives me this error:

$ cd appsoui/setup/
gdtstdb1.tireco.com:/mnt/TSTapps/patches/appsoui/setup>ls -lrt
total 28
-rw-r--r-- 1 oraTST dba  6361 Nov 30  2004 OUIsetup.pl
-rw-r--r-- 1 oraTST dba 13464 May 12  2005 OUISetup.jar
-rw-r--r-- 1 oraTST dba   459 Mar 24 14:46 OUIsetup.log
gdtstdb1.tireco.com:/mnt/TSTapps/patches/appsoui/setup>perl OUIsetup.pl
Derived ORACLE_HOME(CWD) = /mnt/TSTapps/patches
ENV RDBMS ORACLE_HOME    = /mnt/TSTapps/oracle/trctstdb/11.2.0/dbhome_1

 Using ORACLE_HOME from ENV settings

ERROR: Could not find XML Parser java at:
        /mnt/TSTapps/oracle/trctstdb/11.2.0/dbhome_1/appsoui/oui/lib/xmlparserv2.zip

 Download the OUI patch again and unzip if the file is missing.

  

Note: I have downloaded this patch thrice and tried the same thing, but same error occur always.

•  I just tried one more time and got a different error this time

$ cd $ORACLE_HOME/appsoui/setup
$ perl OUIsetup.pl
        ORACLE_HOME  : /mnt/TSTapps/oracle/trctstdb/11.2.0/dbhome_1 (APPSDB_TRCTST)
        OUI Location : /mnt/TSTapps/oracle/trctstdb/11.2.0/dbhome_1/appsoui/oui
        CLASSPATH    : /mnt/TSTapps/oracle/trctstdb/11.2.0/dbhome_1/appsoui/setup/OUISetup.jar:/mnt/TSTapps/oracle/trctstdb/11.2.0/dbhome_1/appsoui/oui/lib/OraInstaller.jar:/mnt/TSTapps/oracle/trctstdb/11.2.0/dbhome_1/appsoui/oui/lib/xmlparserv2.jar
        JRE_INST     : /mnt/TSTapps/oracle/trctstdb/11.2.0/dbhome_1/appsoui/jre/1.1.8

 Executing OUI setup now:

/mnt/TSTapps/oracle/trctstdb/11.2.0/dbhome_1/appsoui/jre/1.3.1/bin/java -classpath /mnt/TSTapps/oracle/trctstdb/11.2.0/dbhome_1/appsoui/setup/OUISetup.jar:/mnt/TSTapps/oracle/trctstdb/11.2.0/dbhome_1/appsoui/oui/lib/OraInstaller.jar:/mnt/TSTapps/oracle/trctstdb/11.2.0/dbhome_1/appsoui/oui/lib/xmlparserv2.jar -DDEBUG_MODE=false oracle.apps.ad.util.OUISetup /mnt/TSTapps/oracle/trctstdb/11.2.0/dbhome_1 /mnt/TSTapps/oracle/trctstdb/11.2.0/dbhome_1/appsoui/oui /mnt/TSTapps/oracle/trctstdb/11.2.0/dbhome_1/appsoui/jre/1.1.8 APPSDB_TRCTST

 /mnt/TSTapps/oracle/trctstdb/11.2.0/dbhome_1/appsoui/jre/1.3.1/bin/x86_64/native_threads/java: error while loading shared libraries: libstdc++-libc6.1-1.so.2: cannot open shared object file: No such file or directory

• Then I verified if the following RPM is installed or not

            [root@gdtstdb1 ~]# rpm -qa libstdc++
            libstdc++-4.1.2-52.el5_8.1
            libstdc++-4.1.2-52.el5_8.1 

• Created a soft link to fix the issue

            [root@gdtstdb1 ~]# cd /usr/bin
            ln -s libstdc++-libc6.2-2.so.3 libstdc++-libc6.1-1.so.2

• Also, ran ldconfig command as root ( which sets up the links to the shared libraries) 

• Now applied the patch one more time, and it went through this time.

$ cd $ORACLE_HOME/appsoui/setup
$ perl OUIsetup.pl
  ORACLE_HOME : /mnt/TSTapps/oracle/trctstdb/11.2.0/dbhome_1 (APPSDB_TRCTST)
  OUI Location : /mnt/TSTapps/oracle/trctstdb/11.2.0/dbhome_1/appsoui/oui
CLASSPATH : /mnt/TSTapps/oracle/trctstdb/11.2.0/dbhome_1/appsoui/setup/OUISetup.jar:/mnt/TSTapps/oracle/trctstdb/11.2.0/dbhome_1/appsoui/oui/lib/OraInstaller.jar:/mnt/TSTapps/oracle/trctstdb/11.2.0/dbhome_1/appsoui/oui/lib/xmlparserv2.jar
  JRE_INST : /mnt/TSTapps/oracle/trctstdb/11.2.0/dbhome_1/appsoui/jre/1.1.8

Executing OUI setup now:
/mnt/TSTapps/oracle/trctstdb/11.2.0/dbhome_1/appsoui/jre/1.3.1/bin/java -classpath /mnt/TSTapps/oracle/trctstdb/11.2.0/dbhome_1/appsoui/setup/OUISetup.jar:/mnt/TSTapps/oracle/trctstdb/11.2.0/dbhome_1/appsoui/oui/lib/OraInstaller.jar:/mnt/TSTapps/oracle/trctstdb/11.2.0/dbhome_1/appsoui/oui/lib/xmlparserv2.jar -DDEBUG_MODE=false oracle.apps.ad.util.OUISetup /mnt/TSTapps/oracle/trctstdb/11.2.0/dbhome_1 /mnt/TSTapps/oracle/trctstdb/11.2.0/dbhome_1/appsoui/oui /mnt/TSTapps/oracle/trctstdb/11.2.0/dbhome_1/appsoui/jre/1.1.8 APPSDB_TRCTST

Log file : /mnt/TSTapps/oracle/trctstdb/11.2.0/dbhome_1/appsoui/setup/OUIsetup.log
Checking inventory ...
Updating registration of ORACLE_HOME in global inventory...
Oracle home registration completed.
Removing OUI entries from the global inventory: /etc/oracle/oraInventory/ContentsXML/comps.xml
Registering the OUI component with local inventory...
OUI 2.2 registration completed.
$./opatch lsinventory
Invoking OPatch 11.2.0.1.7

 Oracle Interim Patch Installer version 11.2.0.1.7

Copyright (c) 2011, Oracle Corporation.  All rights reserved.

Oracle Home       : /mnt/TSTapps/oracle/trctstdb/11.2.0/dbhome_1

Central Inventory : /etc/oracle/oraInventory
   from           : /etc/oraInst.loc
OPatch version    : 11.2.0.1.7
OUI version       : 11.2.0.3.0
Log file location : /mnt/TSTapps/oracle/trctstdb/11.2.0/dbhome_1/cfgtoollogs/opatch/opatch2014-03-24_15-34-00PM.log

Lsinventory Output file location : /mnt/TSTapps/oracle/trctstdb/11.2.0/dbhome_1/cfgtoollogs/opatch/lsinv/lsinventory2014-03-24_15-34-00PM.txt

--------------------------------------------------------------------------------

Installed Top-level Products (2):

Oracle Database 11g                                                  11.2.0.3.0

Oracle Database 11g Examples                                         11.2.0.3.0
There are 2 products installed in this Oracle Home.

Interim patches (7) :

Patch  13366268     : applied on Wed Aug 08 14:48:42 PDT 2012

Unique Patch ID:  14394336
   Created on 21 Dec 2011, 08:14:06 hrs PST8PDT
   Bugs fixed:
    13366268

Patch  13258936     : applied on Wed Aug 08 14:48:21 PDT 2012

Unique Patch ID:  14465000
   Created on 5 Dec 2011, 02:41:32 hrs PST8PDT
   Bugs fixed:
     13258936

Patch  13004894     : applied on Wed Aug 08 14:47:50 PDT 2012

Unique Patch ID:  14175139
   Created on 14 Oct 2011, 03:35:11 hrs PST8PDT
   Bugs fixed:
     13004894

Patch  13001379     : applied on Wed Aug 08 14:47:15 PDT 2012

Unique Patch ID:  14706905.1
   Created on 28 Mar 2012, 11:30:01 hrs PST8PDT
   Bugs fixed:
     13001379

Patch  12985184     : applied on Wed Aug 08 14:46:51 PDT 2012

Unique Patch ID:  14296756
   Created on 30 Nov 2011, 15:18:06 hrs PST8PDT
   Bugs fixed:
     12985184

Patch  12960302     : applied on Wed Aug 08 14:46:17 PDT 2012

Unique Patch ID:  14254250
   Created on 10 Nov 2011, 02:40:18 hrs PST8PDT
   Bugs fixed:
     12960302

Patch  9858539      : applied on Wed Aug 08 14:45:06 PDT 2012

Unique Patch ID:  14308099
   Created on 28 Nov 2011, 01:07:44 hrs PST8PDT
   Bugs fixed:
     9858539

--------------------------------------------------------------------------------

OPatch succeeded. 

• If the issue still persists, you can perform detachHome and attachHome command to reattch your home to central inventory, that will resolve this issue.

./runInstaller -silent -detachHome ORACLE_HOME="<Oracle_Home_Location>" ORACLE_HOME_NAME="<Name_Of _Oracle_Home>"

./runInstaller -silent -ignoreSysPrereqs -attachHome ORACLE_HOME="<Oracle_Home_Location>" ORACLE_HOME_NAME="<Name_Of_Oracle_Home>"

References:
ORACLE_HOME could not be registered in the Global Inventory [Article ID 298993.1]

How to Recreate the Global oraInventory [Article ID 295185.1]

HTH
Thanks for reading..!!