APP-FND-01564: ORACLE error 1041 in fdudat

Problem Description:
Recently I had this issue in a two node Oracle EBS 11i (11.5.10.2) non production environment on a RHEL5 (32-bit).
While submitting any concurrent request, it is showing Phase=INACTIVE and Status=No manager. After checking the CM log files, we found out that the internal concurrent manager got terminated with the below errors:

The Internal Concurrent Manager has encountered an error. Review concurrent manager log file for more detailed information. : 21-DEC-2013 02:19:13 - Shutting down Internal Concurrent Manager : 21-DEC-2013 02:19:14 Reviver is not enabled, not spawning a reviver process. List of errors encountered: ............................................................................. _ 1 _ Routine AFPCMT encountered an ORACLE error. ORA-01041: internal error. hostdef extension doesn't exist . Review your error messages for the cause of the error. (=<POINTER>) _ 2 _ Routine AFPSMG encountered an ORACLE error. ORA-03114: not connected to ORACLE . Review your error messages for the cause of the error. (=<POINTER>) _ 3 _ Routine AFPCMT encountered an ORACLE error. ORA-03113: end-of-file on communication channel . Review your error messages for the cause of the error. (=<POINTER>) ............................................................................. APP-FND-01564: ORACLE error 1041 in fdudat Cause:  fdudat failed due to ORA-01041: internal error. hostdef extension doesn't exist. The SQL statement being executed at the time of the error was: &SQLSTMT and was executed from the file &ERRFILE.   List of errors encountered: ............................................................................. _ 1 _ Routine AFPCAL received failure code while parsing or running your concurrent program CPMGR Review your concurrent request log file for more detailed information. Make sure you are passing arguments in the correct format. ............................................................................. The TRCDEV_1218@TRCDEV internal concurrent manager has terminated with status 1 - giving up.

This could happen due to possible reasons:

• Database going down
• Listener is down
• Packet loss during the communication between the CM and the DB

In our case, the following error indicates that the database went down.
fdudat failed due to ORA-01041: internal error. hostdef extension doesn't exist.

If ORA-01041 is thrown, an internal error has occurred in which the pointer to the hostdef extension in hostdef is null. 

 

Solution:

To resolve ORA-01041, you should report the error as a bug. In the case of ORA-01041 as a response to performing shutdown and startup, you can use the ipcrm command and remove allocated shared memory segments.

 

So the solution was to bounce the Database and start the concurrent managers again.
 

Also, see Note: 811093.1 for details.

You may log a SR and see if there are any other parameter/option which can be set to overcome this issue.