Prerequisite Checks Failed while agent deployment EM12c

Problem:

Recently when I was trying to deploy management agent in Enterprise Manager  12c using "Add Targets Manually" , I was getting the below error.

Remote Prerequisite Checks Failed.

 

Possible Cause:

• It seems I was trying to install agent on same OMS ( OEM12c R2 (12.1.0.2)) machine.
• An EM agent home already exists on the host.
• 11g and 12c Agents are already installed on the same server where we are trying to install 12c agent.

 Solution:

• This is just a Warning Message, you can ignore and proceed with the new 12c Agent Installation; if the existing Agent is communicating with a different OMS. 
• But as the  existing 12c agent is communicating with the same 12c OMS, so now I'm trying to deinstall the Oracle Management Agent home and retry the agent deployment part.

Prerequisites

Before you deinstall a Management Agent, do the following:

• Shut down the Management Agent by running the following command from its home. If it is already shut down, then skip this step.

             $<AGENT_HOME>/bin/emctl stop agent

•     Wait for the Management Agent to go to the unreachable state in the Cloud Control console. If it is already in the unreachable state, then go to the next step.

•     Delete the Management Agent targets and their monitored targets (from any host where EM CLI is installed):

    emcli delete_target
    -name="example.com:1836"
    -type="oracle_emd"
    -delete_monitored_targets

There are 4 ways to deinstall Oracle Management agent, I will be discussing the one which is most recommended and highly used:

 

1. Using the AgentDeinstall.pl Script
2. Using the Installation Wizard in Graphical Mode
3. Using the Installation Wizard in Silent Mode
4. Using an RPM File

Deinstalling Oracle Management Agents Using the Installation Wizard in Graphical Mode 

          1. Go to the agent home directory and invoke OUI(installer)

                cd $AGENT_HOME/oui/bin
./runInstaller.sh -deinstall ORACLE_HOME=<absolute_path_to_agent_home> [-removeallfiles] 

          2. In the installation wizard, click Installed Products.  

          3. On the Inventory screen, select the plug-in homes under the required Management 

              Agent home, then click Remove. 

          4. On the Inventory screen, select the sbin12cX home, and click Remove. 

          5. On the Inventory screen, select the Management Agent, and click Remove. 

          6. Manually delete the agent base directory. For information on installation base directory.

                     For UNIX platforms: 

             rm -rf <absolute_path_to_agent_base_dir>

After deinstalling the Oracle Management Agent I was able to deploy the agent successfully.

References:

EM 12c: Troubleshooting 12c Management Agent Installation issues (Doc ID 1396675.1)

EM Configuration issue. $ORACLE_HOME/rhel.localdomain.net_orcl not found.

Problem:

Recently I installed RHEL 5 and Oracle Database 11gR2 (11.2.0.3) on a machine. After that I installed OEM12c R2 (12.1.0.2) without any issues.

Right after the Oracle installation, without having made any changes, I typed:
cd $OMS_HOME/bin
$ ./emctl stop oms -all
 
and received the following error:

EM Configuration issue. /u01/app/oracle/product/11.2.0/db_1/rhel.localdomain.net_orcl not found.

I was able to correct the problem by doing the following:

cd $ORACLE_HOME
ln -s localhost_orcl rhel.localdomain.net_orcl
cd $ORACLE_HOME/oc4j/j2ee
ln -s OC4J_DBConsole_localhost_orcl OC4J_DBConsole_rhel.localdomain.net_orcl

Note: If the folder localhost_orcl doesn't lie in ORACLE_HOME location, you can simply create the folder with user who owns the database.

I can now use emctl to start and stop the OMS without trouble.

But the strange question is why did the Oracle installer use 'localhost_orcl' instead of 'rhel.localdomain.net_orcl'.
The service name I specified at the 'Desktop Class' installation in the OUI was 'rhel.localdomain.net' - it was printed already.
 I definitely did not change the host or domain name or made any changes to the hosts file, or any other changes at all

I tried to stop database and listner, than start database and listner and retest issue but still the issue persisted.

Solution:
So I would recommend in environment variable file set  ORACLE_HOSTNAME variable before invoking the OUI or when you run OUI environment variable, you can specify this hostname to avoid any problem.

./runInstaller -ORACLE_HOSTNAME=myhostname.mydomain

Note:Please be sure DO NOT SET ORACLE_HOME or ORACLE_SID before starting EM12c installation.
Now you should not see this issue.

++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

OC4J Configuration issue. /u01/app/oracle/product/11.2.0/db_1/oc4j/j2ee/OC4J_DBConsole_rhel.localdomain.net_orcl not found

For this error please try below command:

./emca -config dbcontrol db << it will ask u some questions liks sys user pass, lsnr port etc. please give correct value and retest issue
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

Hope this helps..!!

11.2.0.3.9 (Jan 2014) Database Patch Set Update (PSU)

This summary is not available. Please click here to view the post.

Could not contact Service Manager FNDSM_xxxx_xxx . The TNS alias could not be located

Problem:

When attempting to start concurrent manager, below message occurs in internal manager log file for every non concurrent processing node but concurrent manager is up and running as expected.

Due to this issue, internal manager log file is having irrelevant messages related to concurrent processing which is creating confusion.

Contents of ICM log file:

 Could not contact Service Manager FNDSM_<hostname>_XXXX . The TNS alias could not be located, the listener process could not be contacted, or the listener failed to spawn the Service Manager process.
                     Process monitor session ended : 03-JAN-2014 20:46:26

Cause:

As per the log file, FNDSM listener was trying to connect to server   FNDSM_<hostname>_XXXX and we do not have concurrent manager running on <hostname> server.

For every non concurrent processing node you can see below messages in internal manager log file.
========================
Could not contact Service Manager FNDSM_<HOSTNAME>_<SID>. The TNS alias could not be located, the listener process on <HOSTNAME> could not be contacted, or the listener failed to spawn the Service Manager process.
========================

If the Service Manager is not present/defined for a particular node, then this causes all the services provided by Service Manager like OPP,WF etc.. not to work. 

Either Start APPS/FNDFS listener for each non concurrent processing node manually or upgrade oracle application to Release 12.0.6.

Solution:

This issue was fixed as part of a number of enhancement requests and was fixed in Release 12.0.5 but released in 12.0.6.

Service Manager: <HostName> on any node will start if node will be a concurrent processing node.

For a Concurrent Processing node :
========================
Service Manager: <HostName> => Actual Process = Target Process = 1

Navigate to System Administrator => Concurrent => Manager => Administer => Query for Service Manager: <HostName> => Click on "Processes" => You will see an "Active" process.

But for a Non Concurrent Processing node :
========================
Service Manager: <HostName> => Actual Process = 1, Target Process = 0  

In our case, we deactivated the Service Manager:<hostname> to fix the issue.

Note: <hostname> of the Non Concurrent Processing node

References:

Concurrent Manager Fails To Start With Error 'Could not start Service Manager FNDSM_<server>_<SID>' Error (Doc ID 423170.1)

Concurrent Processing - Internal Manager Log File contains messages "Could Not Initialize the Service Manager FNDSM_<HOSTNAME>_<SID>" for Non Concurrent Processing Nodes (Doc ID 754698.1)

Unable to Bring Up Concurrent Managers after Running Autoconfig (Doc ID 736024.1)

Apps Listener Starts But Fails to Spawn The Service Manager FNDSM (Doc ID 1073677.1)

Concurrent Processing - R12 Output Post Processor Service Not Coming Up. (Doc ID 460578.1)

Unable to start Workflow Notification Mailer in 11i

Problem:
Recently, in our two node Oracle EBS(11.5.10.2) development environment on RHEL5 (32-bit) with 11gR2 database, the workflow notification mailer went down. We tried to start the WFM services from OAM but unable to start them.

Then we looked in the workflow mailer log's: FNDCPGSC*.txt under $APPLCSF/$APPLOG directory

Log file contents:-
LOG_ID_UNKNOWN : oracle.apps.fnd.cp.gsc.Logger.Logger(String, int) : Logging to System.out until necessary parameters are retrieved for Logger to be properly started.

LOG_ID_UNKNOWN : oracle.apps.fnd.cp.gsm.GSMSvcComponentContainer.initializeStateMachine() : BEGIN

LOG_ID_UNKNOWN : oracle.apps.fnd.cp.gsc.SvcComponentContainer.getNewWorkflowContext() : BEGIN

LOG_ID_UNKNOWN : oracle.apps.fnd.cp.gsc.SvcComponentContainer.getNewWorkflowContext() : BEGIN

LOG_ID_UNKNOWN : oracle.apps.fnd.cp.gsc.SvcComponentContainer.loadGlobalParameters() : BEGIN

LOG_ID_UNKNOWN : oracle.apps.fnd.cp.gsm.GSMSvcComponentContainer.loadContainerParameters() : BEGIN

LOG_ID_UNKNOWN : oracle.apps.fnd.cp.gsc.SvcComponentContainer.start() : Successfully retrieved global and container parameters

LOG_ID_UNKNOWN : oracle.apps.fnd.cp.gsc.SvcComponentContainer.validateParameterValues(Properties) : BEGIN

LOG_ID_UNKNOWN : oracle.apps.fnd.cp.gsc.SvcComponentContainer.validateParameterValues(Properties) : ({SVC_COMP_MAX_ERROR_COUNT=5, SVC_COMP_MONITOR_LOOP_SLEEP=60, SVC_CONTAINER_LOOP_SLEEP=10, SVC_CONTAINER_READ_TIMEOUT=10, SVC_COMP_MONITOR_ONDEMAND_FREQ=300, SVC_CONTAINER_LOG_LEVEL=5, SVC_WRITE_DIAG_TO_GSM_LOG=Y})

LOG_ID_UNKNOWN : oracle.apps.fnd.cp.gsc.SvcComponentContainer.start() : Successfully validated container parameters

LOG_ID_UNKNOWN : oracle.apps.fnd.cp.gsm.GSMSvcComponentContainer.loadDetails(Connection) : BEGIN (oracle.jdbc.driver.OracleConnection@119dc16)

LOG_ID_UNKNOWN : oracle.apps.fnd.cp.gsc.SvcComponentContainer.start() : Successfully retrieved container details

LOG_ID_UNKNOWN : oracle.apps.fnd.cp.gsm.GSMSvcComponentContainer.startLogger() : BEGIN

LOG_ID_UNKNOWN : oracle.apps.fnd.cp.gsc.SvcComponentContainer.startLogger() : BEGIN [default implementation]

[Jan 3, 2014 9:45:37 PM PST]:1388814337257:Thread[GSMQueueProcessor,5,main]:0:-1:oracle.xxxxxxcom:192.xxx.x.xx:-1:-1:ERROR:[SVC-GSM-WFWSSVC-154930 : oracle.apps.fnd.cp.gsm.GSMQueueProcessor.read()]:Could not read from the GSM queue -> oracle.apps.fnd.cp.gsm.GenCartGetMessageException: Io exception: No route to host

Possibilities:
You need to troubleshoot to find out the root cause,

• Was this working before? If yes, any changes have been done recently?
• Can you find any errors in the CM/Database log file (not just the WFM log file)?
• Can you ping the database server from the apps server and connect as apps ?

Solution:

• I reconfigured the entries from adovars.env file with the correct info for the following:

AF_JRE_TOP
AFJVAPRG
AF_CLASSPATH

• After that I scheduled Workflow Control Queue Cleanup program.
• Ran cmcclean.sql and restart the concurrent manager.
• Please run AutoConfig on the apps tier node and make it completes with no errors.

References:

• See the troubleshooting steps in MOS 242941.1 - also run Diagnostics as outlined in 179661.1
• How to Troubleshoot when the Workflow Services Go Down (Doc ID 564394.1) 
• OWF H Diagnostics, Solutions and Information (Doc ID 332152.1 
• 11i - 12 How to Perform a Meaningful SMTP Telnet Test to Troubleshoot Java Mailer For Sending Email Notifications (Doc ID 753845.1)