Oracle EBS Concurrent Request Stuck in Pending Standby – Conflict Resolution Manager Troubleshooting
In Oracle E-Business Suite 12.2, a concurrent request may sometimes remain in Pending / Standby status even when the Standard Concurrent Manager has sufficient available processes.
This does not necessarily indicate a problem with the Standard Manager. A request in Pending / Standby is normally being evaluated by the Conflict Resolution Manager (CRM) because the concurrent program has incompatibility or serialization rules associated with it.
This post provides a generic troubleshooting approach for identifying whether the request is waiting normally because of an incompatible program or whether there is an issue with the Conflict Resolution Manager itself.
1. Typical Symptom
A concurrent request may appear as follows:
Request ID : 123456789 Phase : Pending Status : Standby Manager : Conflict Resolution Manager
The Request Diagnostics window may display a message similar to:
This request is waiting to be processed by the Conflict Resolution Manager. This request cannot yet begin execution because other requests may conflict with it. The Conflict Resolution Manager will determine when this request may begin execution. No action required. This is a normal condition.
The important point is that the request has not yet been released to a normal Concurrent Manager.
2. What Does Pending / Standby Mean?
Internally the concurrent request normally has:
PHASE_CODE = P STATUS_CODE = Q
Which represents:
P = Pending Q = Standby
A request in this state is generally considered a constrained concurrent request.
The processing flow is approximately:
Concurrent Request Submitted
|
v
Is Program Constrained?
|
+----+----+
| |
No Yes
| |
v v
Concurrent Conflict Resolution
Manager Manager
|
v
Check Incompatibilities
|
+------+------+
| |
Conflict No Conflict
Exists
| |
v v
Standby Release Request
|
v
Concurrent Manager
|
v
Running
3. Check the Concurrent Request
Always start by checking the database status of the affected request. Replace the example Request ID with the actual Request ID.
set lines 220
set pages 100
column request_date format a20
column requested_start_date format a20
column actual_start_date format a20
SELECT request_id,
program_application_id,
concurrent_program_id,
phase_code,
status_code,
hold_flag,
TO_CHAR(request_date,
'DD-MON-YYYY HH24:MI:SS') request_date,
TO_CHAR(requested_start_date,
'DD-MON-YYYY HH24:MI:SS') requested_start_date,
TO_CHAR(actual_start_date,
'DD-MON-YYYY HH24:MI:SS') actual_start_date,
controlling_manager,
parent_request_id
FROM fnd_concurrent_requests
WHERE request_id = 123456789;
Typical output:
REQUEST_ID PHASE_CODE STATUS_CODE ---------- ---------- ----------- 123456789 P Q
This confirms:
Pending / Standby
4. Identify the Concurrent Program
Determine which concurrent program is associated with the request and whether it has special queue or Run Alone characteristics.
set lines 220
set pages 100
column application_short_name format a20
column concurrent_program_name format a35
column user_concurrent_program_name format a60
SELECT fa.application_short_name,
cp.concurrent_program_name,
cpt.user_concurrent_program_name,
cp.enabled_flag,
cp.run_alone_flag,
cp.queue_method_code
FROM fnd_concurrent_programs cp,
fnd_concurrent_programs_tl cpt,
fnd_application fa,
fnd_concurrent_requests r
WHERE r.request_id = 123456789
AND cp.application_id = r.program_application_id
AND cp.concurrent_program_id = r.concurrent_program_id
AND cpt.application_id = cp.application_id
AND cpt.concurrent_program_id = cp.concurrent_program_id
AND cpt.language = USERENV('LANG')
AND fa.application_id = cp.application_id;
Pay particular attention to:
RUN_ALONE_FLAG QUEUE_METHOD_CODE
If the program is constrained, the Conflict Resolution Manager must evaluate its incompatibility rules before releasing it.
5. Check Conflict Resolution Manager Status
The next important step is to verify that the Conflict Resolution Manager is actually running.
set lines 200
set pages 100
column concurrent_queue_name format a20
column user_concurrent_queue_name format a40
column target_node format a30
SELECT concurrent_queue_name,
user_concurrent_queue_name,
running_processes,
max_processes,
target_node,
control_code,
enabled_flag
FROM fnd_concurrent_queues_vl
WHERE concurrent_queue_name = 'FNDCRM';
A healthy CRM would normally show something similar to:
CONCURRENT_QUEUE_NAME : FNDCRM RUNNING_PROCESSES : 1 MAX_PROCESSES : 1 TARGET_NODE : APPNODE01
If RUNNING_PROCESSES = 0, the Conflict Resolution Manager itself should be investigated.
6. Check How Many Requests Are in Pending / Standby
Determine whether the problem affects only one request or many concurrent requests.
SELECT COUNT(*) standby_requests
FROM fnd_concurrent_requests
WHERE phase_code = 'P'
AND status_code = 'Q';
If only a small number of requests are in Standby, they may legitimately be waiting for incompatible programs.
If hundreds or thousands of requests are accumulating in Standby, investigate the Conflict Resolution Manager immediately.
7. List All Pending / Standby Requests
set lines 220
set pages 500
column user_name format a20
column user_concurrent_program_name format a60
column request_date format a20
column requested_start_date format a20
SELECT r.request_id,
u.user_name,
cp.user_concurrent_program_name,
TO_CHAR(r.request_date,
'DD-MON-YYYY HH24:MI:SS') request_date,
TO_CHAR(r.requested_start_date,
'DD-MON-YYYY HH24:MI:SS') requested_start_date,
ROUND((SYSDATE-r.request_date)*24*60,1) pending_minutes
FROM fnd_concurrent_requests r,
fnd_concurrent_programs_tl cp,
fnd_user u
WHERE r.phase_code = 'P'
AND r.status_code = 'Q'
AND cp.concurrent_program_id = r.concurrent_program_id
AND cp.application_id = r.program_application_id
AND cp.language = USERENV('LANG')
AND u.user_id = r.requested_by
ORDER BY r.request_date;
This query is useful for identifying the oldest requests waiting on the Conflict Resolution Manager.
8. Find the Oldest Standby Request
SELECT COUNT(*) standby_count,
TO_CHAR(MIN(request_date),
'DD-MON-YYYY HH24:MI:SS') oldest_standby
FROM fnd_concurrent_requests
WHERE phase_code = 'P'
AND status_code = 'Q';
An old Standby request may indicate either:
- A legitimate long-running incompatibility
- A request waiting for another request set stage
- A Run Alone restriction
- A stale or abnormal CRM condition
- A Conflict Resolution Manager processing problem
9. Check Concurrent Program Incompatibilities
Oracle EBS stores concurrent program incompatibility definitions in FND_CONCURRENT_PROGRAM_SERIAL.
The following query identifies programs configured as incompatible with the program associated with the sample request.
set lines 220
set pages 200
column target_program format a60
column incompatible_program format a60
column incompatibility_type format a10
SELECT cp1.user_concurrent_program_name target_program,
cp2.user_concurrent_program_name incompatible_program,
s.incompatibility_type
FROM fnd_concurrent_program_serial s,
fnd_concurrent_programs_tl cp1,
fnd_concurrent_programs_tl cp2
WHERE cp1.application_id =
s.to_run_application_id
AND cp1.concurrent_program_id =
s.to_run_concurrent_program_id
AND cp2.application_id =
s.running_application_id
AND cp2.concurrent_program_id =
s.running_concurrent_program_id
AND cp1.language = USERENV('LANG')
AND cp2.language = USERENV('LANG')
AND (
(s.to_run_application_id,
s.to_run_concurrent_program_id)
=
(SELECT program_application_id,
concurrent_program_id
FROM fnd_concurrent_requests
WHERE request_id = 123456789)
OR
(s.running_application_id,
s.running_concurrent_program_id)
=
(SELECT program_application_id,
concurrent_program_id
FROM fnd_concurrent_requests
WHERE request_id = 123456789)
);
10. Find Currently Running Incompatible Requests
The following SQL attempts to identify currently running requests that have an incompatibility relationship with the program waiting in Standby.
set lines 220
set pages 200
column blocker_program format a60
column actual_start_date format a20
column argument_text format a70
WITH target_req AS
(
SELECT request_id,
program_application_id,
concurrent_program_id
FROM fnd_concurrent_requests
WHERE request_id = 123456789
),
incompat AS
(
SELECT s.running_application_id blocker_app_id,
s.running_concurrent_program_id blocker_program_id,
s.incompatibility_type
FROM fnd_concurrent_program_serial s,
target_req t
WHERE s.to_run_application_id =
t.program_application_id
AND s.to_run_concurrent_program_id =
t.concurrent_program_id
UNION
SELECT s.to_run_application_id,
s.to_run_concurrent_program_id,
s.incompatibility_type
FROM fnd_concurrent_program_serial s,
target_req t
WHERE s.running_application_id =
t.program_application_id
AND s.running_concurrent_program_id =
t.concurrent_program_id
)
SELECT r.request_id blocker_request_id,
cp.user_concurrent_program_name blocker_program,
r.phase_code,
r.status_code,
TO_CHAR(r.actual_start_date,
'DD-MON-YYYY HH24:MI:SS') actual_start_date,
ROUND((SYSDATE-r.actual_start_date)*24*60,2)
running_minutes,
i.incompatibility_type,
r.argument_text
FROM incompat i,
fnd_concurrent_requests r,
fnd_concurrent_programs_tl cp
WHERE r.program_application_id =
i.blocker_app_id
AND r.concurrent_program_id =
i.blocker_program_id
AND r.phase_code = 'R'
AND cp.application_id =
r.program_application_id
AND cp.concurrent_program_id =
r.concurrent_program_id
AND cp.language = USERENV('LANG')
ORDER BY r.actual_start_date;
If this query returns a running request, the Standby condition may be completely normal.
11. Check Previous Executions of the Same Program
It is useful to determine whether the same concurrent program regularly goes into Standby or whether the behavior is new.
set lines 220
set pages 100
SELECT r.request_id,
r.phase_code,
r.status_code,
TO_CHAR(r.request_date,
'DD-MON-YYYY HH24:MI:SS') request_date,
TO_CHAR(r.actual_start_date,
'DD-MON-YYYY HH24:MI:SS') actual_start_date,
TO_CHAR(r.actual_completion_date,
'DD-MON-YYYY HH24:MI:SS') completion_date
FROM fnd_concurrent_requests r
WHERE r.program_application_id =
(
SELECT program_application_id
FROM fnd_concurrent_requests
WHERE request_id = 123456789
)
AND r.concurrent_program_id =
(
SELECT concurrent_program_id
FROM fnd_concurrent_requests
WHERE request_id = 123456789
)
ORDER BY r.request_id DESC
FETCH FIRST 50 ROWS ONLY;
This helps answer:
Is only one submission stuck? OR Does every execution of this program enter Pending / Standby?
12. Check Whether the Request Is on Hold
SELECT request_id,
phase_code,
status_code,
hold_flag
FROM fnd_concurrent_requests
WHERE request_id = 123456789;
Normally:
HOLD_FLAG = N
If the request is explicitly placed on hold, that should be investigated separately from CRM processing.
13. Check Concurrent Manager Capacity
A Pending / Standby request is normally controlled by CRM rather than worker availability. However, manager availability should still be checked after CRM releases the request.
set lines 220
set pages 200
column manager_name format a50
column target_node format a30
SELECT user_concurrent_queue_name manager_name,
concurrent_queue_name,
target_node,
running_processes,
max_processes,
control_code,
enabled_flag
FROM fnd_concurrent_queues_vl
ORDER BY user_concurrent_queue_name;
For example, a Standard Manager may have:
Standard Manager Maximum Processes : 40 Running Processes : 40
Even if all workers are busy, that normally results in a request waiting for a manager worker after CRM processing. It does not by itself explain why the request remains under Conflict Resolution Manager control.
14. Check Conflict Resolution Manager at OS Level
Source the Oracle EBS application environment first.
. ./EBSapps.env run
Then check the application processes:
ps -ef | grep FNDLIBR | grep -v grep
Also check Service Manager processes:
ps -ef | grep FNDSM | grep -v grep
15. Locate the CRM Log
Concurrent Manager logs are normally available below:
$APPLCSF/$APPLLOG
For example:
cd $APPLCSF/$APPLLOG
ls -ltr | grep -i FNDCRM
If the exact file name is unknown:
find $APPLCSF/$APPLLOG -type f -mtime -1 -ls
Review the latest CRM log for database errors, manager communication problems, or repeated processing failures.
16. Check ICM and CRM Together
set lines 200
set pages 100
column concurrent_queue_name format a20
column user_concurrent_queue_name format a40
column target_node format a30
SELECT concurrent_queue_name,
user_concurrent_queue_name,
running_processes,
max_processes,
target_node,
control_code,
enabled_flag
FROM fnd_concurrent_queues_vl
WHERE concurrent_queue_name IN
('FNDICM','FNDCRM');
This quickly confirms whether both the Internal Concurrent Manager and Conflict Resolution Manager are operational.
17. Useful Summary Query
The following query provides a quick summary of all requests currently waiting in Standby.
set lines 220
set pages 500
column program_name format a60
column submitted_by format a20
column request_date format a20
SELECT r.request_id,
cp.user_concurrent_program_name program_name,
fu.user_name submitted_by,
TO_CHAR(r.request_date,
'DD-MON-YYYY HH24:MI:SS') request_date,
ROUND((SYSDATE-r.request_date)*24*60,2)
waiting_minutes,
r.hold_flag,
r.parent_request_id
FROM fnd_concurrent_requests r,
fnd_concurrent_programs_tl cp,
fnd_user fu
WHERE r.phase_code = 'P'
AND r.status_code = 'Q'
AND cp.application_id =
r.program_application_id
AND cp.concurrent_program_id =
r.concurrent_program_id
AND cp.language = USERENV('LANG')
AND fu.user_id = r.requested_by
ORDER BY r.request_date;
18. Troubleshooting Decision Tree
Concurrent Request
|
v
Pending / Standby
|
v
Check FNDCRM
|
+-------------------------+
| |
FNDCRM Running? FNDCRM Down?
| |
Yes No
| |
v v
Check Program Investigate CRM
Incompatibility / ICM / Node
|
v
Running Incompatible Request?
|
+---+---+
| |
Yes No
| |
v v
Normal Check:
Wait - CRM backlog
- CRM logs
- Run Alone flag
- Request Set
- Parent request
- Stale Standby requests
- Manager state
19. Scenario 1 – Running Incompatible Request Found
Example:
Request 123456789 Pending / Standby Waiting because: Request 123450001 Program: Payables Purge Phase : Running
In this situation the CRM is functioning correctly.
Once the incompatible request finishes, CRM should reevaluate the waiting request and release it.
20. Scenario 2 – No Incompatible Request Found
Suppose:
Request 123456789 Pending / Standby CRM Running : YES Incompatible Requests : NONE Standby Duration : Several Hours
This situation requires deeper investigation.
Check:
- Conflict Resolution Manager log
- ICM log
- CRM process health
- Concurrent Manager database state
- Run Alone configuration
- Parent request or request set dependencies
- Whether many requests are accumulating in P/Q status
21. Scenario 3 – Many Requests in Pending / Standby
For example:
Standby Requests : 1500 Oldest Request : Several Hours Old FNDCRM Processes : 0
This strongly suggests a Conflict Resolution Manager problem rather than an individual concurrent program issue.
The CRM/ICM logs and manager processes should be investigated before taking any corrective action.
22. Scenario 4 – Only a Few Standby Requests
For example:
Standby Requests : 3 FNDCRM Processes : 1
