CONC-SM TNS FAIL Call to PingProcess failed for FNDCPGSC

End users begin to see issues running requests, checking the ICM logfile shows the following errors:

CONC-SM TNS FAIL
Call to PingProcess failed for FNDCPGSC
CONC-SM TNS FAIL
Call to StopProcess failed for FNDCPGSC
CONC-SM TNS FAIL
Call to PingProcess failed for FNDOPP
CONC-SM TNS FAIL
Call to StopProcess failed for FNDOPP


Changes

Issue can occur after fresh installs, after patching/maintenance, or after other configuration changes
Cause

To manage non concurrent processing middle tier services, Service Managers calls into control cartridges (shared object libraries) to Start, Stop, Ping, Suspend, Resume, Verify GSM service processes. Following are the two control cartridges supplied with GSM.

1. AQCART (for Java services)
 Examples:
 Service Fulfillment Manager Adapters
 Workflow Notification Mailer
2. CMDCART (for C services)
Examples:
Forms Listener
Metrics Server
Metrics Client
Reports Server
Apache Listener
Each service may have one or more active OS processes. These processes are monitored by the ICM and controlled via the Service Manager acting as the ICM's agent. All communication between Concurrent Processing and the integrated service will take place through the  StartProcess,PingProcess and StopProcess functions.

For example to check the status of generic services, the ICM will call the PingProcess function from the appropriate control cartridge. Following messages are expected when the control cartridge is unable to communicate with generic services.

Call to PingProcess failed for FNDCPGSC
Call to PingProcess failed for FNDOPP

Hence issue can have multiple causes, usually due to instance specific issues during installs,patching/maintenance, or other configuration changes.
Solution

Potential Solutions

Please perform the following steps:

1. Stop all middle tier services including the concurrent managers. Make sure no FNDLIBR, FNDSM, or other dead processes are running.

2. Stop and then restart the database.

3. Connect to SQLPLUS as APPS user and run the following :

EXEC FND_CONC_CLONE.SETUP_CLEAN;
COMMIT;
EXIT;

4.  Reference Note.260887.1 regarding the Steps to Clean Nonexistent Nodes or IP Addresses from FND_NODES.

5.. Run AutoConfig on all tiers, firstly on the DB tier and then the APPS tiers and Web tiers to repopulate the required system tables.

6. Connect to SQLPLUS as APPS user and run the following statement :

select CONCURRENT_QUEUE_NAME
from FND_CONCURRENT_QUEUES
where CONCURRENT_QUEUE_NAME like 'FNDSM%';

If the above SQL does not return any value please do the following:

cd $FND_TOP/patch/115/sql

Connect to SQLPLUS as APPS user and run the following script :

SQL> @afdcm037.sql;

Reference Note 218893.1 How to Create The Service Manager 'FNDSM' on Oracle Applications
to create libraries for FNDSM and create managers for preexisting nodes.

Check again that FNDSM entries now exist:

select CONCURRENT_QUEUE_NAME
from FND_CONCURRENT_QUEUES
where CONCURRENT_QUEUE_NAME like 'FNDSM%';

7. . Run the CMCLEAN.SQL script from the referenced note below (don't forget to
commit). Reference Note.134007.1 CMCLEAN.SQL - Non Destructive Script to Clean Concurrent Manager Tables

8. Start the middle tier services including your concurrent manager.

9. Retest the issue.

If the above potential solutions do not resolve the issue, please seek support assistance to reference limited distribution Note.1312980.1 to generate diagnostics and debugging to further investigate the issue.