Unable to find 'kshell' in path

Problem:
For a Clone of an E-Business Suite Instance Release 11.5.10.2 we receive following error on the target for the execution of the script 'perl adcfgclone.pl dbTier' :
...
Checking for ksh...
Unable to find 'kshell' in path
...

Cause:
The Util 'adchkutil.sh' called by the adcfgclone procedure is checking for 'pdksh'. In the past this check made sense for older Linux Releases, but nowadays the 'pdksh' is not needed anymore on the DB-Tier and so this check became obsolete.

Solution:
To get this failure fixed execute following steps :

•  Download and apply Patch 5972212 - 'ADCHKUTL.SH FAILS IN SLES 10 DUE TO PDKSH REQUIREMENT'

This Patch is available for Release 11i and R12.0.x - it brings the script 'adchkutl.sh' in Release 11.5.6, which no longer checks for 'pdksh'.

Following Workaround can be used as an immediate fix :

•  Before starting the adcfgclone.pl script set in your Environment the KSH_VERSION. The following is applicable to the Linux Red Hat 5 platform :

export KSH_VERSION='@(#)PD KSH v5.2.14 99/07/13.2'

• Execute 'perl adcfgclone.pl dbTier' and it should now run successfully.

Upgrade Oracle EBS R12.1.1 to 12.1.3

Document Reference:

Oracle E-Business Suite Release 12.1.3 Readme [ID 1080973.1]

Applies to: 12.1

Our Application version :12.1.1

Our Database version :11.2.0.3

Oracle Apps 12.1.3 upgrade is only possible if your system is in 12.1.1 or upper version.

Prerequisite Steps

1.   It is advisable to complete all of the following four steps before attempting to apply the 12.1.3       maintenance pack.

·         Upgrade database version to 11.2.0.3 (upgraded)

·          Upgrade to the latest Java versions

·         Upgrade iAS 10.1.3 to the latest version

·          Upgrade forms 10.1.2 to the latest version

As we are already on 12.1.1 version and upgraded to 11.2.0.3, we  have already upgraded form tier and web tier. So, no action is required for form and web tier upgrade

2.  At first make some changes in the database parameters for upgrade. You need to set the following parameters using initialization file.

recyclebin=false

_pga_max_size=104857600

_disable_fast_validate = TRUE

Use the following kind of comand to changes the mentioned three parameters.

SQL> alter system set "_disable_fast_validate"=TRUE SCOPE=BOTH;

System altered.

SQL> alter system set "_pga_max_size"=104857600 scope=both;

System altered.

                                    Or

edit the init parameter file in the database

add (_disable_fast_validate=TRUE).

pga_aggregate_target=2G

set the _pga_max_size initialization parameter to a larger value as follows: _pga_max_size=104857600
and restart your database.)

SQL> startup pfile='/u01/db_top/db/tech_st/11.2.0/dbs/initPROD.ora';

ORACLE instance started.

Total System Global Area 1068937216 bytes

Fixed Size                  2235208 bytes

Variable Size             427820216 bytes

Database Buffers          624951296 bytes

Redo Buffers               13930496 bytes

Database mounted.

Database opened.

 

Main Upgrade Patch Application

• Download the following two patches from metalink 9239089, 9239090.

•  Using autopatch utility first apply R12.AD.B.DELTA.3 Patch 9239089

Pre Install Steps as per Patch readmefile & Doc.1077769.1

(i) create a dir on $ORACLE_HOME/appustil/admin

(ii) Copy adgrants.sql from this patch directory to $ORACLE_HOME/appsutil/admin

[oracle@oracleupk admin]$ cp /u01/upgrade_12.1.3_patches/9239089/admin/adgrants.sql .

[oracle@oracleupk admin]$ ls -lrt

total 52

-rwxr-xr-x 1 oracle oinstall 53060 May  2 10:10 adgrants.sql

(iii) Set the database environment

(iv) logon to sqlplus and run adgrants.sql with apps schema password as below

sqlplus /nolog
sql> connect as sysdba
sql> @$ORACLE_HOME/appsutil/admin/adgrants.sql apps

• Disable maintanance mode using adadmin utility

• Set application environment.

• Run Patch 9239089 using adpatch

• Compile APPS schema

• Run adadmin > Compile/Reload Applications Database Entities menu >Compile APPS schema.

•  Apply patch 9239090 using adpatch

(When adrelink error chose "Yes" to continue)

• Use AutoPatch to apply the latest consolidated online help Patch 9239095 and follow the instructions in the patch readme file.

Post upgrade steps

•  Apply post-install Oracle E-Business Suite Applications Technology patches. (Required)

• Apply mandatory Patch 9817770 (9817770:R12.ATG_PF.B [POST-R12.ATG_PF.B.DELTA.3 CONSOLIDATED PATCH].)

• To not get the error :Relink of module "MSCCPP" failed, "FEMCCE","MSCNEW","MSCMON" please see if (R12: MSC Modules Error Out When Relinking with Adrelink [ID 1345788.1]) helps.

• This patch requires additional code levels to have been successfully

applied to your system before it can be applied.  These code levels,

as at the time that this patch was built, are listed below.  Please check

'My Oracle Support' to determine if any of these have been superseded

§     * R12.AD.B.delta.3

§     * R12.ATG_PF.B.delta.3

• Both the patches are already applied.

• Apply mandatory Patch 9966055 (9966055:R12.FND.B [TRANSLATED VERSION OF FNDSCSGN NOT LAUNCHED].) 

•  After you have upgraded to Oracle E-Business Suite Release 12.1.3, perform the following post-update steps to update all database tier nodes with the code level, provided by Oracle E-Business Suite Release 12.1.3

Application tier:

• As the APPLMGR user, run the environment (UNIX) or the command (WINDOWS) file for the current APPL_TOP.

[applmgr@oracleupk appl]$ . ./APPSPROD_oracleupk.env

• Run AutoConfig on the APPL_TOP

• Run the admkappsutil.pl utility to create the file appsutil.zip in the <INST_TOP>/admin/out directory.

            [applmgr@oracleupk bin]$ pwd

            /u01/appl_top/apps/apps_st/appl/ad/12.0.0/bin

            [applmgr@oracleupk bin]$ ls admkappsutil.pl

            admkappsutil.pl

            [applmgr@oracleupk out]$ pwd

            /u01/appl_top/inst/apps/PROD_oracleupk/admin/out

            [applmgr@oracleupk out]$ ls appsutil.zip

            appsutil.zip

Database tier:

• As the ORACLE user, run the environment (UNIX) or the command (WINDOWS) file for the current ORACLE_HOME.

cd <RDBMS ORACLE_HOME>

[oracle@oracleupk 11.2.0]$ . ./PROD_oracleupk.env

• Copy or FTP the appsutil.zip file to the <RDBMS ORACLE_HOME>

[oracle@oracleupk PROD_oracleupk]$ cp /u01/appl_top/inst/apps/PROD_oracleupk/admin/out/appsutil.zip .

• Uncompress appsutil.zip under the <RDBMS ORACLE_HOME>.

            [oracle@oracleupk PROD_oracleupk]$ unzip -o appsutil.zip

• Run AutoConfig on the <RDBMS ORACLE_HOME>.           

[oracle@oracleupk PROD_oracleupk]$ ./adautocfg.sh

• Modify Initialization Parameters and bounce database

1. remove entry _disable_fast_validate=TRUE
2. set recyclebin=ON 

SQL> alter system set "_disable_fast_validate"=FALSE SCOPE=BOTH;

System altered.

SQL> ALTER SYSTEM SET recyclebin = ON DEFERRED;

System altered.

 Database tier and application tier:

• Re-run adpreclone.pl on the database tier and the application tier. Since updates included in Oracle E-Business Suite Release 12.1.3 were applied to the system, adpreclone must be run again to apply the updates to the clone directory structures that were used during cloning.

• Now run preclone in db tier first, then in apps tier.

[oracle@oracleupk PROD_oracleupk]$ perl adpreclone.pl dbTier

 

[applmgr@oracleupk scripts]$ perl adpreclone.pl appsTier

•          Apply the language patch of the main patch 9239090 if applicable in your environment. - Not required in our case.

•        Disable Maintenance Mode using adadmin and check the version using following sql statement.

SQL> select release_name from fnd_product_groups;
RELEASE_NAME
————————————————–
12.1.3

•       Restart the while Apps Tier and Database Tier.

•      Login to the Oracle EBS and perform a health check of the instance. Now, start testing individual product.

APP-FND-01564: ORACLE error 1041 in fdudat

Problem Description:
Recently I had this issue in a two node Oracle EBS 11i (11.5.10.2) non production environment on a RHEL5 (32-bit).
While submitting any concurrent request, it is showing Phase=INACTIVE and Status=No manager. After checking the CM log files, we found out that the internal concurrent manager got terminated with the below errors:

The Internal Concurrent Manager has encountered an error. Review concurrent manager log file for more detailed information. : 21-DEC-2013 02:19:13 - Shutting down Internal Concurrent Manager : 21-DEC-2013 02:19:14 Reviver is not enabled, not spawning a reviver process. List of errors encountered: ............................................................................. _ 1 _ Routine AFPCMT encountered an ORACLE error. ORA-01041: internal error. hostdef extension doesn't exist . Review your error messages for the cause of the error. (=<POINTER>) _ 2 _ Routine AFPSMG encountered an ORACLE error. ORA-03114: not connected to ORACLE . Review your error messages for the cause of the error. (=<POINTER>) _ 3 _ Routine AFPCMT encountered an ORACLE error. ORA-03113: end-of-file on communication channel . Review your error messages for the cause of the error. (=<POINTER>) ............................................................................. APP-FND-01564: ORACLE error 1041 in fdudat Cause:  fdudat failed due to ORA-01041: internal error. hostdef extension doesn't exist. The SQL statement being executed at the time of the error was: &SQLSTMT and was executed from the file &ERRFILE.   List of errors encountered: ............................................................................. _ 1 _ Routine AFPCAL received failure code while parsing or running your concurrent program CPMGR Review your concurrent request log file for more detailed information. Make sure you are passing arguments in the correct format. ............................................................................. The TRCDEV_1218@TRCDEV internal concurrent manager has terminated with status 1 - giving up.

This could happen due to possible reasons:

• Database going down
• Listener is down
• Packet loss during the communication between the CM and the DB

In our case, the following error indicates that the database went down.
fdudat failed due to ORA-01041: internal error. hostdef extension doesn't exist.

If ORA-01041 is thrown, an internal error has occurred in which the pointer to the hostdef extension in hostdef is null. 

 

Solution:

To resolve ORA-01041, you should report the error as a bug. In the case of ORA-01041 as a response to performing shutdown and startup, you can use the ipcrm command and remove allocated shared memory segments.

 

So the solution was to bounce the Database and start the concurrent managers again.
 

Also, see Note: 811093.1 for details.

You may log a SR and see if there are any other parameter/option which can be set to overcome this issue.

ORA-00494: enqueue [CF] held for too long (more than 900 seconds)

Problem: 
Today morning one of our development instance crashed due to following error found in alert log file.

Contents of alert log file:-

Sat Dec 21 02:17:04 2013
Errors in file /mnt/DEVapps/oracle/trcdevdb/11.2.0/dbhome_1/log/diag/rdbms/trcdev/TRCDEV/trace/TRCDEV_ora_5707.trc  (incident=81065):
ORA-00494: enqueue [CF] held for too long (more than 900 seconds) by 'inst 1, osid 24599'
Sat Dec 21 02:17:14 2013
minact-scn: useg scan erroring out with error e:12751
Suspending MMON action 'Block Cleanout Optim, Undo Segment Scan' for 82800 seconds
Incident details in: /mnt/DEVapps/oracle/trcdevdb/11.2.0/dbhome_1/log/diag/rdbms/trcdev/TRCDEV/incident/incdir_81065/TRCDEV_ora_5707_i81065.trc
Sat Dec 21 02:18:33 2013
Killing enqueue blocker (pid=24599) on resource CF-00000000-00000000 by (pid=5707)
 by killing session 3.1
Killing enqueue blocker (pid=24599) on resource CF-00000000-00000000 by (pid=5707)
 by terminating the process
USER (ospid: 5707): terminating the instance due to error 2103
Sat Dec 21 02:18:49 2013
Termination issued to instance processes. Waiting for the processes to exit
Instance termination failed to kill one or more processes
Instance terminated by USER, pid = 5707

Cause:-
The Possible reasons for this sudden instance crash are:

• Database running out of memory
• Due to high load,server became unresponsive
• Server is running out of memory and/or swap.
• High concurrency on resources, IO waits and contention 

Solution:-
In our development database, we reviewed the alert log and trace files related to this ORA error and we found out that there was high load on the Linux server due to RMAN hot backup running, export of a  huge schema going on simultaneously.

At the time of this instance crash, there was very high load at the server  and that's why the kill blocker interface killed the background process and so the instance crashed.

• In order to prevent a background blocker from being killed, you can set the following init.ora parameter to 1 (default is 3).

_kill_enqueue_blocker=1

• If you want to avoid the kill of the blocker (background or non-background process) you can set _kill_controlfile_enqueue_blocker=false.

With these parameter, if the enqueue holder is a background process, then it will not be killed, therefore the instance will not crash 



Note:- Don't set undocumented hidden parameters until advise by oracle support.

References:-
Database Crashes With ORA-00494 [ID 753290.1]
Ora-00494: Enqueue [Cf] Held For Too Long Causing Database To Crash [ID 1101862.1]  
PHYSICAL: ORA-00494: enqueue [CF] held for too long after Node Crash [ID 747071.1]

ORA-00494 Or ORA-600 [2103] During High Load After 10.2.0.4 Upgrade (Doc ID 779552.1)

Could not find service instance context for service instance number

Problem:-
I was working on Oracle application 11i cloning  lately and after completing the clone, my conc. manager output post manager (OPP) is not coming up in target system.

I  tried to fix the issue by running "cmclean.sql" and thn ran autoconfig on application node, but still the issue persists

The logfile for FNDOPP(outout post manager) says:-
[12/17/13 5:24:30 PM] [main] Starting GSF service with concurrent process id = 154919.
[12/17/13 5:24:30 PM] [main] Initialization Parameters: oracle.apps.fnd.cp.opp.OPPServiceThread:2:0:max_threads=5
[12/17/13 5:24:31 PM] [Thread-23] Service thread starting up.
[12/17/13 5:24:31 PM] [Thread-24] Service thread starting up.
[12/17/13 6:52:55 PM] [GSMServiceController:154919] Received shutdown request.
[12/17/13 6:52:55 PM] [GSMServiceController:154919] Preparing to shutdown service.
[12/17/13 6:52:55 PM] [GSMServiceController:154919] Stopping all Service Threads.
[12/17/13 6:52:55 PM] [OPPServiceThread1] Preparing to shut down service thread.
[12/17/13 6:52:55 PM] [OPPServiceThread0] Preparing to shut down service thread.
[12/17/13 6:53:55 PM] [OPPServiceThread0] Service thread has shutdown.
[12/17/13 6:54:00 PM] [OPPServiceThread1] Service thread has shutdown.
[12/17/13 6:54:00 PM] [GSMServiceController:154919] All Service Threads have completed.
[12/17/13 6:54:00 PM] [GSMServiceController:154919] Service has shut down.

FYI, The below metalink note did not help
Concurrent Processing - R12 Output Post Processor Service Not Coming Up. (Doc ID 460578.1)

Cause:-
In ICM log file, I see occurrence of these error messages in bulk.It looks Mangers are wrongly defined or fnd_nodes having some incorrect values.

Process monitor session started : 17-DEC-2013 21:29:02

Could not find service instance context for service instance number 1072
Could not find service instance context for service instance number 1092
Could not find service instance context for service instance number 1089
Could not find service instance context for service instance number 1090
Could not find service instance context for service instance number 1091
Could not find service instance context for service instance number 1043


 It looks Mangers are wrongly defined or fnd_nodes having some incorrect values.

Solution:
- Shtudown Application Services
- Run cmclean.sql script
- Run following query as apps user:
     EXEC FND_CONC_CLONE.SETUP_CLEAN;
     COMMIT;
     EXIT
- Run autoconfig for Apps Tier
- Start Application Services

Do the below, and check the output of below queries:

select CONCURRENT_QUEUE_NAME, TARGET_NODE
from fnd_concurrent_queues;

select decode(CONCURRENT_QUEUE_NAME,'FNDICM','Internal Manager','FNDCRM','Conflict Resolution Manager','AMSDMIN','Marketing Data Mining Manager','C_AQCT_SVC','C AQCART Service','FFTM','FastFormula Transaction Manager','FNDCPOPP','Output Post Processor','FNDSCH','Scheduler/Prereleaser Manager','FNDSM_AQHERP','Service Manager: AQHERP','FTE_TXN_MANAGER','Transportation Manager','IEU_SH_CS','Session History Cleanup','IEU_WL_CS','UWQ Worklist Items Release for Crashed session','INVMGR','Inventory Manager','INVTMRPM','INV Remote Procedure Manager','OAMCOLMGR','OAM Metrics Collection Manager','PASMGR','PA Streamline Manager','PODAMGR','PO Document Approval Manager','RCVOLTM','Receiving Transaction Manager','STANDARD','Standard Manager','WFALSNRSVC','Workflow Agent Listener Service','WFMLRSVC','Workflow Mailer Service','WFWSSVC','Workflow Document Web Services Service','WMSTAMGR','WMS Task Archiving Manager','XDP_APPL_SVC','SFM Application Monitoring Service','XDP_CTRL_SVC','SFM Controller Service','XDP_Q_EVENT_SVC','SFM Event Manager Queue Service','XDP_Q_FA_SVC','SFM Fulfillment Actions Queue Service','XDP_Q_FE_READY_SVC','SFM Fulfillment Element Ready Queue Service','XDP_Q_IN_MSG_SVC','SFM Inbound Messages Queue Service','XDP_Q_ORDER_SVC','SFM Order Queue Service','XDP_Q_TIMER_SVC','SFM Timer Queue Service','XDP_Q_WI_SVC','SFM Work Item Queue Service','XDP_SMIT_SVC','SFM SM Interface Test Service') as "Concurrent Manager's Name", max_processes as "TARGET Processes", running_processes as "ACTUAL Processes"
from apps.fnd_concurrent_queues
where CONCURRENT_QUEUE_NAME in ('FNDICM','FNDCRM','AMSDMIN','C_AQCT_SVC','FFTM','FNDCPOPP','FNDSCH','FNDSM_AQHERP','FTE_TXN_MANAGER','IEU_SH_CS','IEU_WL_CS','INVMGR','INVTMRPM','OAMCOLMGR','PASMGR','PODAMGR','RCVOLTM','STANDARD','WFALSNRSVC','WFMLRSVC','WFWSSVC','WMSTAMGR','XDP_APPL_SVC','XDP_CTRL_SVC','XDP_Q_EVENT_SVC','XDP_Q_FA_SVC','XDP_Q_FE_READY_SVC','XDP_Q_IN_MSG_SVC','XDP_Q_ORDER_SVC','XDP_Q_TIMER_SVC','XDP_Q_WI_SVC','XDP_SMIT_SVC');


Reference Docs:-
305986.1
368380.1