Unable to start Workflow Notification Mailer in 11i

Problem:
Recently, in our two node Oracle EBS(11.5.10.2) development environment on RHEL5 (32-bit) with 11gR2 database, the workflow notification mailer went down. We tried to start the WFM services from OAM but unable to start them.

Then we looked in the workflow mailer log's: FNDCPGSC*.txt under $APPLCSF/$APPLOG directory

Log file contents:-
LOG_ID_UNKNOWN : oracle.apps.fnd.cp.gsc.Logger.Logger(String, int) : Logging to System.out until necessary parameters are retrieved for Logger to be properly started.

LOG_ID_UNKNOWN : oracle.apps.fnd.cp.gsm.GSMSvcComponentContainer.initializeStateMachine() : BEGIN

LOG_ID_UNKNOWN : oracle.apps.fnd.cp.gsc.SvcComponentContainer.getNewWorkflowContext() : BEGIN

LOG_ID_UNKNOWN : oracle.apps.fnd.cp.gsc.SvcComponentContainer.getNewWorkflowContext() : BEGIN

LOG_ID_UNKNOWN : oracle.apps.fnd.cp.gsc.SvcComponentContainer.loadGlobalParameters() : BEGIN

LOG_ID_UNKNOWN : oracle.apps.fnd.cp.gsm.GSMSvcComponentContainer.loadContainerParameters() : BEGIN

LOG_ID_UNKNOWN : oracle.apps.fnd.cp.gsc.SvcComponentContainer.start() : Successfully retrieved global and container parameters

LOG_ID_UNKNOWN : oracle.apps.fnd.cp.gsc.SvcComponentContainer.validateParameterValues(Properties) : BEGIN

LOG_ID_UNKNOWN : oracle.apps.fnd.cp.gsc.SvcComponentContainer.validateParameterValues(Properties) : ({SVC_COMP_MAX_ERROR_COUNT=5, SVC_COMP_MONITOR_LOOP_SLEEP=60, SVC_CONTAINER_LOOP_SLEEP=10, SVC_CONTAINER_READ_TIMEOUT=10, SVC_COMP_MONITOR_ONDEMAND_FREQ=300, SVC_CONTAINER_LOG_LEVEL=5, SVC_WRITE_DIAG_TO_GSM_LOG=Y})

LOG_ID_UNKNOWN : oracle.apps.fnd.cp.gsc.SvcComponentContainer.start() : Successfully validated container parameters

LOG_ID_UNKNOWN : oracle.apps.fnd.cp.gsm.GSMSvcComponentContainer.loadDetails(Connection) : BEGIN (oracle.jdbc.driver.OracleConnection@119dc16)

LOG_ID_UNKNOWN : oracle.apps.fnd.cp.gsc.SvcComponentContainer.start() : Successfully retrieved container details

LOG_ID_UNKNOWN : oracle.apps.fnd.cp.gsm.GSMSvcComponentContainer.startLogger() : BEGIN

LOG_ID_UNKNOWN : oracle.apps.fnd.cp.gsc.SvcComponentContainer.startLogger() : BEGIN [default implementation]

[Jan 3, 2014 9:45:37 PM PST]:1388814337257:Thread[GSMQueueProcessor,5,main]:0:-1:oracle.xxxxxxcom:192.xxx.x.xx:-1:-1:ERROR:[SVC-GSM-WFWSSVC-154930 : oracle.apps.fnd.cp.gsm.GSMQueueProcessor.read()]:Could not read from the GSM queue -> oracle.apps.fnd.cp.gsm.GenCartGetMessageException: Io exception: No route to host

Possibilities:
You need to troubleshoot to find out the root cause,

• Was this working before? If yes, any changes have been done recently?
• Can you find any errors in the CM/Database log file (not just the WFM log file)?
• Can you ping the database server from the apps server and connect as apps ?

Solution:

• I reconfigured the entries from adovars.env file with the correct info for the following:

AF_JRE_TOP
AFJVAPRG
AF_CLASSPATH

• After that I scheduled Workflow Control Queue Cleanup program.
• Ran cmcclean.sql and restart the concurrent manager.
• Please run AutoConfig on the apps tier node and make it completes with no errors.

References:

• See the troubleshooting steps in MOS 242941.1 - also run Diagnostics as outlined in 179661.1
• How to Troubleshoot when the Workflow Services Go Down (Doc ID 564394.1) 
• OWF H Diagnostics, Solutions and Information (Doc ID 332152.1 
• 11i - 12 How to Perform a Meaningful SMTP Telnet Test to Troubleshoot Java Mailer For Sending Email Notifications (Doc ID 753845.1)