Thursday, August 20, 2026

How to Find Which Concurrent Program Is Executed by Which Custom Concurrent Manager in Oracle EBS 12.2

 

How to Find Which Concurrent Program Is Executed by Which Custom Concurrent Manager in Oracle EBS 12.2

In Oracle E-Business Suite 12.2, concurrent programs may be eligible to run under the Standard Manager or one or more custom Concurrent Managers based on specialization rules.

However, eligibility does not necessarily prove which manager actually executed a request. To identify the manager that processed a concurrent request, use the CONTROLLING_MANAGER column in FND_CONCURRENT_REQUESTS.

The following production-safe, read-only SQL queries provide:

  • Concurrent programs executed by each custom manager
  • Request-level execution details
  • Program execution counts
  • First and last execution times
  • Average runtime
  • Manager name and short name

Important distinction

There are two different requirements:

  1. Actual execution history: Which manager actually executed a request?
  2. Manager eligibility: Which managers are allowed to execute a program based on specialization rules?

The queries in this article show the actual execution history.

A program can be eligible for multiple managers, but each individual request is processed by one Concurrent Manager process. The relationship is captured through:

FND_CONCURRENT_REQUESTS.CONTROLLING_MANAGER
        ↓
FND_CONCURRENT_PROCESSES.CONCURRENT_PROCESS_ID
        ↓
FND_CONCURRENT_QUEUES

Query 1: Programs executed by each custom Concurrent Manager

The following query returns a distinct list of programs historically executed by custom Concurrent Managers:

SELECT DISTINCT
       fcq.user_concurrent_queue_name manager_name,
       fcq.concurrent_queue_name manager_short_name,
       fav.application_name,
       fcp.concurrent_program_name program_short_name,
       fcp.user_concurrent_program_name program_name
FROM apps.fnd_concurrent_requests fcr
JOIN apps.fnd_concurrent_processes fpr
  ON fpr.concurrent_process_id = fcr.controlling_manager
JOIN apps.fnd_concurrent_queues_vl fcq
  ON fcq.application_id = fpr.queue_application_id
 AND fcq.concurrent_queue_id = fpr.concurrent_queue_id
JOIN apps.fnd_concurrent_programs_vl fcp
  ON fcp.application_id = fcr.program_application_id
 AND fcp.concurrent_program_id = fcr.concurrent_program_id
JOIN apps.fnd_application_vl fav
  ON fav.application_id = fcr.program_application_id
WHERE fcr.actual_start_date IS NOT NULL
  AND fcq.concurrent_queue_name NOT IN
      ('FNDICM', 'STANDARD', 'FNDCRM', 'FNDIM', 'FNDSM')
ORDER BY
       fcq.user_concurrent_queue_name,
       fav.application_name,
       fcp.user_concurrent_program_name;

Output columns

  • MANAGER_NAME: User-friendly Concurrent Manager name
  • MANAGER_SHORT_NAME: Internal Concurrent Manager short name
  • APPLICATION_NAME: Application that owns the program
  • PROGRAM_SHORT_NAME: Internal concurrent program name
  • PROGRAM_NAME: User concurrent program name

The standard Oracle managers are excluded so that the output focuses on custom managers.

Query 2: Detailed request execution history

Use the following query to display the request ID, manager, program, phase, status, start time, completion time, and runtime:

SELECT fcr.request_id,
       fcq.user_concurrent_queue_name manager_name,
       fcq.concurrent_queue_name manager_short_name,
       fav.application_name,
       fcp.concurrent_program_name program_short_name,
       fcp.user_concurrent_program_name program_name,
       fcr.actual_start_date,
       fcr.actual_completion_date,
       ROUND(
           (NVL(fcr.actual_completion_date, SYSDATE) -
            fcr.actual_start_date) * 24 * 60,
           2
       ) runtime_minutes,
       DECODE(fcr.phase_code,
              'P', 'Pending',
              'R', 'Running',
              'C', 'Completed',
              'I', 'Inactive',
              fcr.phase_code) phase,
       DECODE(fcr.status_code,
              'A', 'Waiting',
              'B', 'Resuming',
              'C', 'Normal',
              'D', 'Cancelled',
              'E', 'Error',
              'G', 'Warning',
              'H', 'On Hold',
              'I', 'Normal',
              'M', 'No Manager',
              'Q', 'Standby',
              'R', 'Normal',
              'S', 'Suspended',
              'T', 'Terminating',
              'U', 'Disabled',
              'W', 'Paused',
              'X', 'Terminated',
              'Z', 'Waiting',
              fcr.status_code) status
FROM apps.fnd_concurrent_requests fcr
JOIN apps.fnd_concurrent_processes fpr
  ON fpr.concurrent_process_id = fcr.controlling_manager
JOIN apps.fnd_concurrent_queues_vl fcq
  ON fcq.application_id = fpr.queue_application_id
 AND fcq.concurrent_queue_id = fpr.concurrent_queue_id
JOIN apps.fnd_concurrent_programs_vl fcp
  ON fcp.application_id = fcr.program_application_id
 AND fcp.concurrent_program_id = fcr.concurrent_program_id
JOIN apps.fnd_application_vl fav
  ON fav.application_id = fcr.program_application_id
WHERE fcr.actual_start_date >= SYSDATE - 30
  AND fcq.concurrent_queue_name NOT IN
      ('FNDICM', 'STANDARD', 'FNDCRM', 'FNDIM', 'FNDSM')
ORDER BY fcr.actual_start_date DESC;

This query displays the executions from the last 30 days.

Changing the reporting period

Last 24 hours

WHERE fcr.actual_start_date >= SYSDATE - 1

Last seven days

WHERE fcr.actual_start_date >= SYSDATE - 7

Last 30 days

WHERE fcr.actual_start_date >= SYSDATE - 30

Last 90 days

WHERE fcr.actual_start_date >= SYSDATE - 90

Last year

WHERE fcr.actual_start_date >= ADD_MONTHS(SYSDATE, -12)

Query 3: Manager-wise program execution count

The following query shows how many times each program was executed by each custom manager during the last 90 days:

SELECT fcq.user_concurrent_queue_name manager_name,
       fcq.concurrent_queue_name manager_short_name,
       fav.application_name,
       fcp.concurrent_program_name program_short_name,
       fcp.user_concurrent_program_name program_name,
       COUNT(*) execution_count,
       MIN(fcr.actual_start_date) first_execution,
       MAX(fcr.actual_start_date) last_execution,
       ROUND(
           AVG(
               (NVL(fcr.actual_completion_date, SYSDATE) -
                fcr.actual_start_date) * 24 * 60
           ),
           2
       ) average_runtime_minutes
FROM apps.fnd_concurrent_requests fcr
JOIN apps.fnd_concurrent_processes fpr
  ON fpr.concurrent_process_id = fcr.controlling_manager
JOIN apps.fnd_concurrent_queues_vl fcq
  ON fcq.application_id = fpr.queue_application_id
 AND fcq.concurrent_queue_id = fpr.concurrent_queue_id
JOIN apps.fnd_concurrent_programs_vl fcp
  ON fcp.application_id = fcr.program_application_id
 AND fcp.concurrent_program_id = fcr.concurrent_program_id
JOIN apps.fnd_application_vl fav
  ON fav.application_id = fcr.program_application_id
WHERE fcr.actual_start_date >= SYSDATE - 90
  AND fcq.concurrent_queue_name NOT IN
      ('FNDICM', 'STANDARD', 'FNDCRM', 'FNDIM', 'FNDSM')
GROUP BY
       fcq.user_concurrent_queue_name,
       fcq.concurrent_queue_name,
       fav.application_name,
       fcp.concurrent_program_name,
       fcp.user_concurrent_program_name
ORDER BY
       fcq.user_concurrent_queue_name,
       execution_count DESC;

This report is useful for:

  • Identifying which programs are processed by each custom manager
  • Understanding manager workload
  • Finding frequently executed programs
  • Reviewing average program runtime
  • Supporting Concurrent Manager capacity planning
  • Validating custom manager utilization before migration or cloning

Query 4: Show all managers, including the Standard Manager

Remove the custom-manager exclusion condition when you need the complete manager mapping.

Remove:

AND fcq.concurrent_queue_name NOT IN
    ('FNDICM', 'STANDARD', 'FNDCRM', 'FNDIM', 'FNDSM')

The output will then include requests processed by both seeded and custom Concurrent Managers.

Query 5: Find the manager for a specific request ID

When troubleshooting an individual concurrent request, use:

SELECT fcr.request_id,
       fcq.user_concurrent_queue_name manager_name,
       fcq.concurrent_queue_name manager_short_name,
       fpr.concurrent_process_id,
       fpr.os_process_id,
       fcp.user_concurrent_program_name program_name,
       fcr.actual_start_date,
       fcr.actual_completion_date
FROM apps.fnd_concurrent_requests fcr
JOIN apps.fnd_concurrent_processes fpr
  ON fpr.concurrent_process_id = fcr.controlling_manager
JOIN apps.fnd_concurrent_queues_vl fcq
  ON fcq.application_id = fpr.queue_application_id
 AND fcq.concurrent_queue_id = fpr.concurrent_queue_id
JOIN apps.fnd_concurrent_programs_vl fcp
  ON fcp.application_id = fcr.program_application_id
 AND fcp.concurrent_program_id = fcr.concurrent_program_id
WHERE fcr.request_id = &request_id;

The query prompts for the concurrent request ID and returns the manager and operating-system process information.

Why pending requests may not appear

Pending requests may not yet have a value in CONTROLLING_MANAGER. The manager is normally identified after the request is selected and processed.

Therefore, these queries are intended primarily for:

  • Running requests
  • Completed requests
  • Historical execution analysis

To determine which managers could potentially execute a pending request, the Concurrent Manager specialization rules must be evaluated separately.

Production safety

All queries in this article are read-only. They do not update Concurrent Manager definitions, concurrent requests, or specialization rules.

Before running a large historical query in production:

  • Restrict the query using ACTUAL_START_DATE.
  • Start with the last one, seven, or 30 days.
  • Avoid querying the complete request history during peak hours.
  • Export large results using SQL Developer, SQLcl, SQL*Plus, or an approved reporting tool.
  • Review the execution plan if the request tables contain substantial historical data.

Conclusion

It is possible to identify which concurrent program was executed by which custom Concurrent Manager in Oracle EBS 12.2.

The most reliable historical relationship is:

Concurrent Request
→ Controlling Manager Process
→ Concurrent Manager Queue
→ Concurrent Program

These queries show the manager that actually executed each request. They should not be interpreted as the complete specialization-rule configuration because a program can be eligible for multiple managers while its individual request is executed by only one manager.

How to Export Completed Oracle EBS Concurrent Programs by Daily, Weekly, Monthly, Quarterly, Half-Yearly, and Yearly Periods

How to Export Completed Oracle EBS Concurrent Programs by Daily, Weekly, Monthly, Quarterly, Half-Yearly, and Yearly Periods

Oracle E-Business Suite stores concurrent request execution details in the FND_CONCURRENT_REQUESTS table. Using a read-only SQL query, an Apps DBA can extract completed concurrent programs for a selected reporting period and export the results to Excel.

The report includes:

  • Request ID
  • Application and concurrent program
  • Requested-by user
  • Request and start times
  • Completion time
  • Execution duration
  • Completion status
  • Request arguments
  • Oracle session and operating system process IDs
  • Log and output file locations

Important distinction

This report returns concurrent requests completed during a selected period.

For example:

  • DAILY returns requests completed today.
  • WEEKLY returns requests completed during the current ISO week.
  • MONTHLY returns requests completed during the current month.
  • QUARTERLY returns requests completed during the current quarter.
  • HALFYEARLY returns requests completed during the current six-month period.
  • YEARLY returns requests completed during the current year.

It does not determine whether a concurrent program is itself scheduled to run daily, weekly, or monthly. Schedule-frequency analysis requires a separate query using request scheduling information and execution history.

Completed Concurrent Requests Query

Run the following query as the Oracle EBS APPS user:

DEFINE p_period = 'DAILY';

WITH period_dates AS
(
    SELECT
        CASE UPPER('&p_period')
            WHEN 'DAILY' THEN
                TRUNC(SYSDATE)
            WHEN 'WEEKLY' THEN
                TRUNC(SYSDATE, 'IW')
            WHEN 'MONTHLY' THEN
                TRUNC(SYSDATE, 'MM')
            WHEN 'QUARTERLY' THEN
                TRUNC(SYSDATE, 'Q')
            WHEN 'HALFYEARLY' THEN
                ADD_MONTHS(
                    TRUNC(SYSDATE, 'YYYY'),
                    CASE
                        WHEN TO_NUMBER(TO_CHAR(SYSDATE, 'MM')) <= 6
                        THEN 0
                        ELSE 6
                    END
                )
            WHEN 'YEARLY' THEN
                TRUNC(SYSDATE, 'YYYY')
        END AS start_date,
        SYSDATE AS end_date
    FROM dual
)
SELECT
    UPPER('&p_period') AS report_period,
    d.start_date AS period_start,
    d.end_date AS period_end,
    r.request_id,
    a.application_name,
    p.concurrent_program_name AS program_short_name,
    p.user_concurrent_program_name,
    u.user_name AS requested_by,
    r.request_date,
    r.requested_start_date,
    r.actual_start_date,
    r.actual_completion_date,
    ROUND(
        (r.actual_completion_date - r.actual_start_date) * 24,
        2
    ) AS elapsed_hours,
    TRUNC(
        (r.actual_completion_date - r.actual_start_date) * 24
    ) || ':' ||
    LPAD(
        TRUNC(
            MOD(
                (r.actual_completion_date - r.actual_start_date) * 1440,
                60
            )
        ),
        2,
        '0'
    ) || ':' ||
    LPAD(
        TRUNC(
            MOD(
                (r.actual_completion_date - r.actual_start_date) * 86400,
                60
            )
        ),
        2,
        '0'
    ) AS elapsed_hh_mm_ss,
    DECODE(
        r.status_code,
        'C', 'Normal',
        'G', 'Warning',
        'E', 'Error',
        'X', 'Terminated',
        'D', 'Cancelled',
        r.status_code
    ) AS completion_status,
    r.argument_text,
    r.oracle_process_id AS os_process_id,
    r.oracle_session_id,
    r.logfile_name,
    r.outfile_name
FROM fnd_concurrent_requests r
JOIN fnd_concurrent_programs_vl p
  ON p.application_id = r.program_application_id
 AND p.concurrent_program_id = r.concurrent_program_id
JOIN fnd_application_vl a
  ON a.application_id = r.program_application_id
JOIN fnd_user u
  ON u.user_id = r.requested_by
CROSS JOIN period_dates d
WHERE r.phase_code = 'C'
  AND r.actual_completion_date >= d.start_date
  AND r.actual_completion_date <= d.end_date
ORDER BY r.actual_completion_date DESC;

Selecting the Reporting Period

Change the value of p_period before executing the query.

Daily report

DEFINE p_period = 'DAILY';

This returns requests completed from midnight today until the current time.

Weekly report

DEFINE p_period = 'WEEKLY';

This uses the ISO week, starting on Monday.

Monthly report

DEFINE p_period = 'MONTHLY';

This returns requests completed from the first day of the current month.

Quarterly report

DEFINE p_period = 'QUARTERLY';

This returns requests completed from the beginning of the current calendar quarter.

Half-yearly report

DEFINE p_period = 'HALFYEARLY';

The reporting periods are:

  • January through June
  • July through December

Yearly report

DEFINE p_period = 'YEARLY';

This returns requests completed from January 1 of the current year.

Completion Status Filters

The condition below includes every request whose phase is Completed:

WHERE r.phase_code = 'C'

A completed phase can contain requests that ended with Normal, Warning, Error, Terminated, or Cancelled status.

Successfully completed requests only

Add the following condition:

AND r.status_code = 'C'

Normal, Warning, and Error requests

Use:

AND r.status_code IN ('C', 'G', 'E')

Error requests only

Use:

AND r.status_code = 'E'

Warning requests only

Use:

AND r.status_code = 'G'

Understanding the Execution Duration

The report provides duration in two formats.

Decimal hours

The ELAPSED_HOURS column displays the runtime in hours:

2.50

This represents two hours and thirty minutes.

Hours, minutes, and seconds

The ELAPSED_HH_MM_SS column displays duration as:

02:30:00

This format is useful when reviewing long-running concurrent requests.

Exporting the Results to Excel

In Oracle SQL Developer:

  1. Execute the query.
  2. Right-click anywhere in the query result grid.
  3. Select Export.
  4. Select Excel 2007+ (.xlsx) as the output format.
  5. Enable Include Column Headers.
  6. Select the destination file.
  7. Click Next, followed by Finish.

The results can be maintained in separate Excel worksheets:

  • Daily
  • Weekly
  • Monthly
  • Quarterly
  • Half-Yearly
  • Yearly

Production Safety

The query is read-only and does not update Oracle EBS data.

However, yearly reports may retrieve a large number of records from a busy production environment. Consider the following precautions:

  • Run large reports outside peak business hours.
  • Test the query in a non-production environment first.
  • Use a specific date range when the request history is extensive.
  • Avoid opening millions of rows directly in Excel.
  • Export large results to CSV when necessary.

An Excel worksheet supports a maximum of 1,048,576 rows. If the report exceeds this limit, use CSV files or divide the report into smaller date ranges.

Conclusion

This query provides a production-safe method to extract completed Oracle EBS concurrent requests for daily, weekly, monthly, quarterly, half-yearly, and yearly reporting periods. The output can be exported directly to Excel for operational reporting, performance analysis, audit review, and identification of long-running or failed concurrent requests.

Wednesday, August 19, 2026

Oracle EBS R12.2 Workflow Notification Mailer Stuck in Starting – Troubleshooting and RCA Guide

 

Oracle EBS R12.2 Workflow Notification Mailer Stuck in Starting – Troubleshooting and RCA Guide

In Oracle E-Business Suite R12.2, the Workflow Notification Mailer is responsible for sending and receiving workflow email notifications.

A common issue is that the mailer remains in:

Starting

instead of moving to:

Running

If the status remains in Starting for several minutes, the mailer has usually encountered an initialization problem.

This article provides a generic troubleshooting approach to identify the root cause before making configuration changes.


1. Check Workflow Mailer Status

Navigate to:

Oracle Applications Manager
   ↓
Workflow Manager
   ↓
Service Components

Filter using:

Type (Internal) = WF_MAILER

Check the status of:

Workflow Notification Mailer

Typical statuses include:

Running
Stopped
Starting
Stopping
Error

If the mailer remains in Starting, continue with the following checks.


2. Check View Log

Select:

Workflow Notification Mailer

and click:

View Log

Look at the log entries corresponding to the exact time when the mailer was started.

Search for errors such as:

ERROR
Exception
Failed
Authentication
Connection
Timeout
SSL
PKIX
UnknownHost
OutOfMemory

The first exception generated during startup is usually more useful for RCA than the final generic error.


3. Check View Event History

From the same Service Components page, select the Workflow Notification Mailer and click:

View Event History

Review events such as:

Start requested
Component starting
Component stopped
Component error
Unexpected termination

Event history helps determine whether the component:

  • never started

  • started and immediately failed

  • repeatedly restarted

  • was manually stopped

  • was terminated by the service container


4. Locate Workflow Mailer Log Files

Source the Oracle EBS RUN filesystem environment.

For example:

. EBSapps.env run

Check the log location:

echo $APPLCSF
echo $APPLLOG

Workflow Java/GSM component logs commonly appear as:

$APPLCSF/$APPLLOG/FNDCPGSC*.txt

List the latest files:

ls -ltr $APPLCSF/$APPLLOG/FNDCPGSC*.txt | tail

Check the latest log:

tail -300 <latest_FNDCPGSC_file>

Search for errors:

egrep -i "error|exception|failed|authentication|ssl|connect|timeout|unknownhost|outofmemory|pkix" \
<latest_FNDCPGSC_file>

5. Check Workflow Mailer Component Status from Database

Connect as the APPS user.

set lines 200

col component_name format a40
col component_status format a20
col startup_mode format a15

select component_id,
       component_name,
       component_status,
       startup_mode
from fnd_svc_components
where component_type = 'WF_MAILER';

Example:

COMPONENT_ID   COMPONENT_NAME                  COMPONENT_STATUS
------------   ------------------------------  ----------------
10006          Workflow Notification Mailer    STARTING

Record the:

COMPONENT_ID

for additional investigation.


6. Check Mailer Configuration

Review the mailer configuration from:

Workflow Manager
   ↓
Service Components
   ↓
Workflow Notification Mailer
   ↓
Edit

Verify parameters related to:

SMTP Server
SMTP Port
IMAP Server
IMAP Port
Mailbox Username
Reply-To Address
Inbound Processing
Outbound Processing
SSL/TLS configuration

Be particularly careful after:

Clone
Refresh
Migration
DR activation
Environment build
Network migration
Mail-server migration

A cloned environment may still contain configuration inherited from another environment.


7. Verify Non-Encrypted Parameters from Database

The following query can be used to review component parameters that are not encrypted.

Replace the component ID with the value obtained earlier.

set lines 220

col parameter_name format a40
col parameter_display_name format a45
col parameter_value format a60

select p.parameter_name,
       v.parameter_display_name,
       v.parameter_value
from fnd_svc_comp_param_vals v,
     fnd_svc_comp_params_b p
where v.parameter_id = p.parameter_id
and v.component_id = <COMPONENT_ID>
and nvl(p.encrypted_flag,'N') = 'N'
order by p.parameter_name;

Check whether the configuration matches the intended environment.


8. Verify Concurrent Processing and GSM

Workflow Notification Mailer is managed through Oracle EBS Generic Service Management.

Check the relevant operating-system processes:

ps -ef | grep FNDLIBR | grep -v grep

Check Service Manager:

ps -ef | grep FNDSM | grep -v grep

The following components should also be verified from Oracle EBS:

Internal Concurrent Manager
Service Manager
Workflow Mailer Service
Workflow Agent Listener Service

Navigate to:

System Administrator
   ↓
Concurrent
   ↓
Manager
   ↓
Administer

If the Service Manager or GSM infrastructure is unavailable, the Workflow Mailer may not start correctly.


9. Check DNS Resolution

From the application server where the mailer is running, verify that the mail servers resolve correctly.

host <smtp_server>

and:

host <imap_server>

Alternatively:

nslookup <smtp_server>
nslookup <imap_server>

If the hostname cannot be resolved, investigate:

DNS
/etc/hosts
Network configuration
Incorrect mail server hostname

10. Test SMTP Connectivity

For traditional SMTP:

telnet <smtp_server> 25

For SMTP using STARTTLS:

openssl s_client -starttls smtp \
-connect <smtp_server>:587 \
-servername <smtp_server>

A successful connection indicates that the application server can reach the SMTP service.


11. Test IMAP Connectivity

For IMAPS:

openssl s_client \
-connect <imap_server>:993 \
-servername <imap_server>

For standard IMAP:

telnet <imap_server> 143

If the connection times out, investigate the network before changing Workflow Mailer configuration.


12. Understand Common Errors

Authentication Failure

Example:

AuthenticationFailedException

or:

535 Authentication failed

Possible causes:

Incorrect mailbox password
Password expired
Mailbox account locked
SMTP authentication disabled
Authentication policy changed

Connection Timeout

Example:

Connection timed out

Possible causes:

Firewall blocking the port
Network routing issue
Incorrect hostname
Incorrect port
Mail server unreachable

Connection Refused

Example:

Connection refused

Possible causes:

Incorrect port
SMTP/IMAP service stopped
Mail server not listening on the configured port
Load balancer issue

Unknown Host

Example:

UnknownHostException

Possible causes:

DNS resolution failure
Incorrect hostname
Missing DNS entry
Incorrect /etc/hosts entry

SSL Handshake Failure

Example:

SSLHandshakeException

Possible causes:

Certificate problem
Unsupported TLS protocol
Cipher mismatch
Expired certificate
Missing certificate chain

PKIX Error

Example:

PKIX path building failed

This normally indicates that Java cannot establish trust with the certificate presented by the mail server.

Check:

Mail server certificate
Intermediate certificates
Root certificate
Java trust store
Oracle EBS certificate configuration

JVM Memory Problem

Example:

java.lang.OutOfMemoryError

Investigate:

Mailer JVM memory
Large email attachments
Very large notification messages
Abnormal mail queue
JVM configuration

13. Check Workflow Agent Listener

The Workflow Agent Listener should also be checked because Workflow Mailer processing depends on Workflow event processing.

Query service components:

set lines 200

select component_id,
       component_name,
       component_status,
       startup_mode
from fnd_svc_components
order by component_name;

Look for components related to:

Workflow Mailer
Workflow Agent Listener

Verify that required Workflow services are operational.


14. Check Workflow Queues

Check whether Workflow queues are accumulating messages.

Example:

select count(*)
from wf_deferred;

Check notification status:

select status,
       mail_status,
       count(*)
from wf_notifications
group by status,
         mail_status
order by status,
         mail_status;

A large number of pending notifications can help identify whether the problem is:

Mailer startup
Outbound email processing
Workflow queue processing
SMTP delivery

15. Check Pending Notifications

For a more detailed view:

select notification_id,
       message_type,
       message_name,
       recipient_role,
       status,
       mail_status,
       begin_date
from wf_notifications
where mail_status = 'MAIL'
order by begin_date;

Avoid changing Workflow notification data directly unless instructed by Oracle Support or supported documentation.


16. Restart Only After Identifying the Error

Repeatedly restarting the Workflow Mailer without checking the logs normally does not resolve the underlying issue.

Use the supported Workflow Manager interface:

Workflow Manager
   ↓
Service Components
   ↓
Workflow Notification Mailer

Perform:

Stop

Wait until the component reaches:

Stopped

Then:

Start

Monitor:

View Log
View Event History
FNDCPGSC log

17. Do Not Force the Status from Database

Avoid directly updating tables such as:

FND_SVC_COMPONENTS

to artificially change:

STARTING

to:

RUNNING

Changing the database status does not start the underlying Java component and can make troubleshooting more difficult.

Use supported Workflow Service Component controls.


18. Run Workflow Mailer Diagnostic Tests

Oracle EBS provides Workflow diagnostic utilities.

Run the relevant Workflow Mailer diagnostic and validate:

Mailer configuration
SMTP configuration
IMAP configuration
Service component configuration
Workflow services

Diagnostics are particularly useful when the configuration looks correct but the component still fails during initialization.


19. Send a Test Notification

Once the mailer reaches:

Running

send a test notification.

From Workflow Manager use the available mailer test functionality.

Validate:

Notification generated
Notification dequeued
SMTP connection established
Mail delivered
Recipient received email

For inbound processing, also reply to the notification and validate the inbound mail path.


Recommended Troubleshooting Sequence

The following sequence normally provides the fastest RCA:

1. Check Mailer status
        ↓
2. View Log
        ↓
3. View Event History
        ↓
4. Find latest FNDCPGSC log
        ↓
5. Identify first Java exception
        ↓
6. Check GSM / Service Manager
        ↓
7. Verify SMTP/IMAP configuration
        ↓
8. Verify DNS
        ↓
9. Test SMTP/IMAP connectivity
        ↓
10. Check SSL/TLS certificates
        ↓
11. Check authentication
        ↓
12. Check Workflow queues
        ↓
13. Correct root cause
        ↓
14. Restart Workflow Mailer
        ↓
15. Send test notification

Quick RCA Matrix

Symptom / ErrorLikely Root CauseArea to Check
Mailer remains StartingInitialization failureFNDCPGSC log
AuthenticationFailedExceptionInvalid mailbox credentialsMail account
535 Authentication FailedSMTP authenticationSMTP server/account
Connection timed outFirewall/networkNetwork
Connection refusedService/port unavailableMail server
UnknownHostExceptionDNS problemDNS
SSLHandshakeExceptionTLS/certificate issueCertificates
PKIX path building failedCertificate not trustedJava trust store
OutOfMemoryErrorMailer JVM memoryJVM
No mailer Java processGSM/Service ManagerConcurrent Processing
Mailer Running but no emailsQueue/SMTP issueWF queues/SMTP
Outbound works, inbound failsIMAP configurationIMAP/mailbox
Problem starts after cloneSource configuration retainedPost-clone configuration

Useful Commands

Check latest GSM logs

ls -ltr $APPLCSF/$APPLLOG/FNDCPGSC*.txt | tail

Check errors

egrep -i "error|exception|failed|authentication|ssl|connect|timeout|unknownhost|outofmemory|pkix" \
$APPLCSF/$APPLLOG/FNDCPGSC*.txt | tail -100

Check FNDLIBR

ps -ef | grep FNDLIBR | grep -v grep

Check Service Manager

ps -ef | grep FNDSM | grep -v grep

Check SMTP

openssl s_client -starttls smtp \
-connect <smtp_server>:587 \
-servername <smtp_server>

Check IMAPS

openssl s_client \
-connect <imap_server>:993 \
-servername <imap_server>

RCA Example

A proper RCA should identify the complete failure chain.

For example:

Issue:
Workflow Notification Mailer remained in STARTING status.

Observation:
FNDCPGSC log reported connection timeout while establishing
connection to the configured SMTP server.

Root Cause:
The application server was unable to establish connectivity to
the SMTP server on the configured port because the network
firewall rule was missing.

Resolution:
The required network connectivity was enabled between the
Oracle EBS application server and SMTP server.

Validation:
SMTP connectivity was successfully tested from the application
server. Workflow Notification Mailer was restarted and changed
to RUNNING status. A test notification was successfully delivered.

Preventive Action:
SMTP and IMAP connectivity checks were added to the
post-clone/environment validation checklist.

Conclusion

When an Oracle EBS Workflow Notification Mailer is stuck in Starting, avoid immediately restarting services or modifying component tables.

The most effective troubleshooting approach is:

FNDCPGSC log
        ↓
First exception
        ↓
GSM status
        ↓
SMTP/IMAP configuration
        ↓
Network/DNS
        ↓
Authentication
        ↓
SSL/TLS
        ↓
Workflow queues

In most cases, the first meaningful exception in the Workflow Mailer log provides the direction for the RCA.

This method can be used across Oracle E-Business Suite R12.2 environments including Development, Test, UAT, Production, DR and post-clone environments.

Tuesday, August 18, 2026

CM

SET LINESIZE 220

SET PAGESIZE 200


COLUMN concurrent_queue_name      FORMAT A35

COLUMN user_concurrent_queue_name FORMAT A50

COLUMN node_name                  FORMAT A25

COLUMN target_node                FORMAT A25


SELECT concurrent_queue_name,

       user_concurrent_queue_name,

       enabled_flag,

       node_name,

       target_node,

       max_processes,

       running_processes

FROM apps.fnd_concurrent_queues_vl

WHERE manager_type = '0'

ORDER BY user_concurrent_queue_name;

Oracle EBS: SQL Script to Check Concurrent Manager Status

 

Oracle EBS: SQL Script to Check Concurrent Manager Status

Oracle E-Business Suite stores Concurrent Manager configuration and process information in the FND_CONCURRENT_QUEUES_VL view.

The following SQL queries can be used to check Concurrent Manager status, target processes, running processes, assigned nodes and control status.

Check All Concurrent Managers

SET LINESIZE 220
SET PAGESIZE 100

COLUMN concurrent_queue_name      FORMAT A30
COLUMN user_concurrent_queue_name FORMAT A45
COLUMN node_name                  FORMAT A25
COLUMN target_node                FORMAT A25
COLUMN status                     FORMAT A15

SELECT q.concurrent_queue_name,
       q.user_concurrent_queue_name,
       q.node_name,
       q.target_node,
       q.max_processes,
       q.running_processes,
       CASE
           WHEN q.enabled_flag = 'N' THEN 'Disabled'
           WHEN q.control_code = 'A' THEN 'Activating'
           WHEN q.control_code = 'D' THEN 'Deactivating'
           WHEN q.control_code = 'T' THEN 'Terminating'
           WHEN q.control_code = 'V' THEN 'Verifying'
           WHEN q.control_code = 'X' THEN 'Terminated'
           WHEN q.running_processes > 0 THEN 'Running'
           ELSE 'Inactive'
       END status
FROM fnd_concurrent_queues_vl q
WHERE q.manager_type = '0'
ORDER BY q.user_concurrent_queue_name;

Concurrent Manager Status Summary

This query provides a summary of running, inactive and disabled Concurrent Managers.

SELECT CASE
           WHEN enabled_flag = 'N' THEN 'Disabled'
           WHEN running_processes > 0 THEN 'Running'
           ELSE 'Inactive'
       END status,
       COUNT(*) managers,
       SUM(max_processes) target_processes,
       SUM(running_processes) running_processes
FROM fnd_concurrent_queues_vl
WHERE manager_type = '0'
GROUP BY CASE
             WHEN enabled_flag = 'N' THEN 'Disabled'
             WHEN running_processes > 0 THEN 'Running'
             ELSE 'Inactive'
         END
ORDER BY status;

Check a Specific Concurrent Manager

The following example checks the Standard Manager. Replace STANDARD with any part of the required Concurrent Manager name.

SET LINESIZE 200

COLUMN concurrent_queue_name      FORMAT A30
COLUMN user_concurrent_queue_name FORMAT A45
COLUMN node_name                  FORMAT A25
COLUMN target_node                FORMAT A25

SELECT concurrent_queue_name,
       user_concurrent_queue_name,
       node_name,
       target_node,
       max_processes,
       running_processes,
       enabled_flag,
       control_code
FROM fnd_concurrent_queues_vl
WHERE UPPER(user_concurrent_queue_name) LIKE UPPER('%STANDARD%');

Check Internal Concurrent Manager

SELECT concurrent_queue_name,
       user_concurrent_queue_name,
       node_name,
       target_node,
       max_processes,
       running_processes,
       enabled_flag,
       control_code
FROM fnd_concurrent_queues_vl
WHERE concurrent_queue_name = 'FNDICM';

Important Columns

  • CONCURRENT_QUEUE_NAME: Internal name of the Concurrent Manager.

  • USER_CONCURRENT_QUEUE_NAME: Display name shown in Oracle EBS.

  • NODE_NAME: Node on which the manager is currently running.

  • TARGET_NODE: Node assigned to run the manager.

  • MAX_PROCESSES: Configured number of target processes.

  • RUNNING_PROCESSES: Number of processes currently running.

  • ENABLED_FLAG: Indicates whether the manager is enabled.

  • CONTROL_CODE: Current control action or state.

Common Control Codes

Control codeMeaning
AActivating
DDeactivating
TTerminating
VVerifying
XTerminated
NULLNo pending control action

Conclusion

The FND_CONCURRENT_QUEUES_VL view provides a quick way to monitor Oracle EBS Concurrent Managers from the database. It helps identify disabled managers, managers with zero running processes, incorrect node assignments and differences between configured and running processes.

These queries should normally be executed using the APPS database user.

Keywords: Oracle EBS Concurrent Manager, FND_CONCURRENT_QUEUES_VL, Concurrent Manager status query, Standard Manager, Internal Concurrent Manager, Oracle Apps DBA

Monday, August 17, 2026

How to Update Oracle EBS Workflow Mailer Parameters from the Backend

How to Update Oracle EBS Workflow Mailer Parameters from the Backend

Oracle E-Business Suite provides the afsvcpup.sql script to update Workflow Notification Mailer and Workflow Agent Listener parameters directly from SQL*Plus. This is useful when the Oracle EBS application login page or Oracle Applications Manager is unavailable.

Run this procedure during an approved maintenance window and record the existing parameter value before making any change.

1. Identify the Workflow Mailer Component ID

Connect to the Oracle EBS database as the APPS user:

sqlplus apps

Enter the APPS password when prompted.

Run the following query:

SET LINESIZE 200
SET PAGESIZE 100

COLUMN component_name FORMAT A50

SELECT component_id,
       component_name
FROM   fnd_svc_components
WHERE  component_type = 'WF_MAILER'
ORDER BY component_id;

Example output:

COMPONENT_ID  COMPONENT_NAME
------------  ----------------------------------------
10006         Workflow Notification Mailer

If multiple Workflow Notification Mailers exist, select the correct component carefully.

2. Review the Existing Parameter Values

Before changing anything, capture the current configuration:

SET LINESIZE 250
SET PAGESIZE 1000

COLUMN component_name         FORMAT A35
COLUMN component_status       FORMAT A18
COLUMN parameter_name         FORMAT A35
COLUMN parameter_display_name FORMAT A45
COLUMN parameter_value        FORMAT A70
COLUMN startup_mode           FORMAT A15

SELECT sc.component_id,
       sc.component_name,
       sc.component_status,
       sc.correlation_id AS corrid,
       v.parameter_id,
       p.parameter_name,
       v.parameter_display_name,
       v.parameter_value,
       sc.startup_mode
FROM   fnd_svc_comp_param_vals_v v,
       fnd_svc_components sc,
       fnd_svc_comp_params_b p
WHERE  v.component_id = sc.component_id
AND    sc.component_type = 'WF_MAILER'
AND    v.parameter_id = p.parameter_id
ORDER BY sc.component_id,
         v.parameter_display_name;

Record the following information for the parameter being changed:

  • Component ID

  • Parameter ID

  • Parameter name

  • Existing parameter value

  • New parameter value

3. Run the Parameter Update Script

Ensure that the Oracle EBS application environment is sourced and confirm that $FND_TOP is set:

echo $FND_TOP

Start SQL*Plus as the APPS user:

sqlplus apps

Run the seeded Oracle EBS script:

@$FND_TOP/sql/afsvcpup.sql

The script will display the parameters and prompt for the required information.

Prompt 1: Component ID

Enter Component Id:

Enter the Workflow Notification Mailer component ID identified earlier.

Example:

10006

Prompt 2: Component Parameter ID

Enter the Comp Param Id to update:

Enter the parameter ID corresponding to the parameter you want to modify.

Prompt 3: Parameter Value

Enter a value for the parameter:

Enter the new value for the selected parameter.

Review the script output carefully and confirm that it completes without errors.

4. Verify the Updated Value

After running the script, execute the following query:

SET LINESIZE 250
SET PAGESIZE 1000

COLUMN component_name         FORMAT A35
COLUMN component_status       FORMAT A18
COLUMN parameter_name         FORMAT A35
COLUMN parameter_display_name FORMAT A45
COLUMN parameter_value        FORMAT A70
COLUMN startup_mode           FORMAT A15

SELECT sc.component_id,
       sc.component_name,
       sc.component_status,
       sc.correlation_id AS corrid,
       v.parameter_id,
       p.parameter_name,
       v.parameter_display_name,
       v.parameter_value,
       sc.startup_mode
FROM   fnd_svc_comp_param_vals_v v,
       fnd_svc_components sc,
       fnd_svc_comp_params_b p
WHERE  v.component_id = sc.component_id
AND    sc.component_type = 'WF_MAILER'
AND    v.parameter_id = p.parameter_id
ORDER BY sc.component_id,
         v.parameter_display_name;

To verify only the updated parameter, add the relevant component and parameter IDs:

AND sc.component_id = 10006
AND v.parameter_id = <parameter_id>

Place these conditions before the ORDER BY clause.

5. Restart the Workflow Mailer if Required

Depending on the parameter changed, the Workflow Notification Mailer may need to be restarted for the new value to take effect.

Use Oracle Applications Manager to stop and start the Workflow Notification Mailer. If the application interface is unavailable, use the approved administrative procedure for your environment.

After restarting, verify:

  • Workflow Mailer component status

  • Inbound and outbound mail processing

  • SMTP and IMAP connectivity

  • Workflow Mailer log files

  • Pending or failed notifications

  • Test notification delivery

Important Precautions

  • Test the change in a non-production environment first.

  • Do not pass the APPS password directly on the command line because it may appear in shell history or the operating-system process list.

  • Confirm the correct component ID when multiple mailers exist.

  • Capture the original value before updating the parameter.

  • Do not update the underlying Workflow tables directly.

  • Ensure that a rollback value and validation plan are available.

  • Avoid changing passwords or sensitive values while terminal logging or screen recording is enabled.

Conclusion

The seeded $FND_TOP/sql/afsvcpup.sql script provides a controlled method for changing Workflow Notification Mailer and Workflow Agent Listener parameters without logging in to Oracle E-Business Suite. Always capture the existing configuration, select the correct component and parameter IDs, verify the updated value, and restart the affected service when required.

Thursday, August 6, 2026

Troubleshooting an Idle or Stuck Oracle E-Business Suite Forms Session

Troubleshooting an Idle or Stuck Oracle E-Business Suite Forms Session

Oracle Enterprise Manager may show an Oracle E-Business Suite Forms session as idle, long-running, or potentially stuck. Before terminating it, identify the exact database session, map it to the EBS application user, examine its recent activity, and confirm that it is safe to disconnect.

All identifiers in the examples below are represented by placeholders to protect user and environment information:

  • <SERIAL_NUMBER>

  • <SID>

  • <INSTANCE_ID>

  • <FORM_MODULE>

  • <CLIENT_IDENTIFIER>

  • <SQL_ID>

  • <INDEX_OWNER>

  • <INDEX_NAME>

A serial number alone does not uniquely identify a database session. Always confirm the SID, serial number, instance, username, module, client identifier, and logon time before taking action.

1. Identify the Exact Database Session

Use the serial number and Forms module captured from the monitoring tool:

SELECT s.inst_id,
       s.sid,
       s.serial#,
       s.username,
       s.status,
       s.logon_time,
       s.last_call_et,
       s.blocking_instance,
       s.blocking_session,
       s.event,
       s.wait_class,
       s.sql_id,
       s.prev_sql_id,
       s.module,
       s.action,
       s.client_identifier,
       s.process,
       s.paddr
FROM   gv$session s
WHERE  s.serial# = <SERIAL_NUMBER>
AND    s.module LIKE '%<FORM_MODULE>%';

GV$SESSION works for both Oracle RAC and single-instance databases. For a non-RAC database, V$SESSION can also be used.

Validate:

  • SID and serial number

  • Instance ID

  • Database username

  • Module and action

  • Client identifier

  • Logon time

  • Current and previous SQL IDs

  • Current wait event

  • Blocking-session details

LAST_CALL_ET is measured in seconds:

  • For an ACTIVE session, it shows how long the current call has been active.

  • For an INACTIVE session, it shows how long the session has been inactive.

An inactive session is not automatically a problem.

2. Identify the EBS Application User

Oracle EBS commonly stores the ICX session ID in CLIENT_IDENTIFIER. It can be used to identify the application user and responsibility:

SELECT s.inst_id,
       s.sid,
       s.serial#,
       s.client_identifier,
       ic.session_id,
       fu.user_name,
       fr.responsibility_name,
       ic.disable_date
FROM   gv$session s
JOIN   apps.icx_sessions ic
       ON s.client_identifier = TO_CHAR(ic.session_id)
JOIN   apps.fnd_user fu
       ON ic.user_id = fu.user_id
LEFT JOIN apps.fnd_responsibility_tl fr
       ON ic.responsibility_id = fr.responsibility_id
      AND fr.language = 'US'
WHERE  s.serial# = <SERIAL_NUMBER>
AND    s.module LIKE '%<FORM_MODULE>%';

This establishes:

  • The named EBS user

  • The responsibility used

  • The associated ICX session

  • Whether the application session has been disabled

Change the language condition if the environment uses a language other than US English.

This mapping is valid only when CLIENT_IDENTIFIER contains the EBS ICX session ID. Confirm the implementation in your environment.

3. Check Whether the Session Is Blocking Others

Display all currently blocked sessions:

SELECT w.inst_id,
       w.sid,
       w.serial#,
       w.username,
       w.module,
       w.blocking_instance,
       w.blocking_session,
       w.wait_class,
       w.event,
       w.seconds_in_wait
FROM   gv$session w
WHERE  w.blocking_session IS NOT NULL
ORDER BY w.blocking_instance,
         w.blocking_session,
         w.inst_id,
         w.sid;

Check specifically whether the identified session is blocking another session:

SELECT w.inst_id,
       w.sid,
       w.serial#,
       w.username,
       w.module,
       w.event,
       w.wait_class,
       w.seconds_in_wait
FROM   gv$session w
WHERE  w.blocking_session  = <SID>
AND    w.blocking_instance = <INSTANCE_ID>;

No rows means Oracle does not currently report another session as directly blocked by this session. Historical or intermittent blocking may still require ASH or application-log analysis.

4. Review the Current and Previous SQL

The session query returns SQL_ID and PREV_SQL_ID. For an inactive session, PREV_SQL_ID is often more useful because no statement may currently be executing.

SELECT inst_id,
       sql_id,
       plan_hash_value,
       sql_text,
       executions,
       elapsed_time / 1e6 AS elapsed_sec,
       cpu_time / 1e6     AS cpu_sec,
       disk_reads,
       buffer_gets,
       rows_processed,
       last_active_time
FROM   gv$sqlarea
WHERE  sql_id = '<SQL_ID>'
ORDER BY inst_id,
         last_active_time DESC;

Review:

  • SQL text

  • Execution count

  • Elapsed and CPU time

  • Disk reads and buffer gets

  • Rows processed

  • Plan hash value

  • Last active time

The values in GV$SQLAREA are cumulative for the shared cursor. They may include executions by multiple sessions and should not be attributed entirely to one user.

Retrieve Aged-Out SQL from AWR

If the SQL is no longer in the shared pool and the organization is licensed for Oracle Diagnostics Pack:

SELECT st.dbid,
       st.sql_id,
       st.sql_text
FROM   dba_hist_sqltext st
WHERE  st.sql_id = '<SQL_ID>';

Retrieve its historical performance:

SELECT sn.begin_interval_time,
       ss.instance_number,
       ss.snap_id,
       ss.plan_hash_value,
       ss.executions_delta,
       ss.elapsed_time_delta / 1e6 AS elapsed_sec,
       ss.cpu_time_delta / 1e6     AS cpu_sec,
       ss.buffer_gets_delta,
       ss.disk_reads_delta,
       ss.rows_processed_delta
FROM   dba_hist_sqlstat ss
JOIN   dba_hist_snapshot sn
       ON sn.dbid = ss.dbid
      AND sn.instance_number = ss.instance_number
      AND sn.snap_id = ss.snap_id
WHERE  ss.sql_id = '<SQL_ID>'
ORDER BY sn.begin_interval_time DESC,
         ss.instance_number;

AWR and DBA_HIST_* views require the appropriate Oracle Diagnostics Pack licence.

5. Review Recent Wait History

Use the SID and instance ID identified earlier:

SELECT inst_id,
       sid,
       seq#,
       event,
       wait_time,
       p1,
       p2,
       p3
FROM   gv$session_wait_history
WHERE  sid     = <SID>
AND    inst_id = <INSTANCE_ID>
ORDER BY seq#;

This view contains only a small, recent in-memory history. It may show what the session was waiting for immediately before entering its current state.

Common idle events include:

  • SQL*Net message from client

  • SQL*Net message to client

An idle network wait usually means the database is waiting for the application or user to submit another request. It does not, by itself, prove that the session is stuck.

6. Determine Whether the Issue Is Isolated or Systemic

Review all sessions using the same Oracle Forms module:

SELECT inst_id,
       sid,
       serial#,
       username,
       client_identifier,
       status,
       last_call_et,
       sql_id,
       prev_sql_id,
       event,
       wait_class,
       blocking_instance,
       blocking_session,
       logon_time
FROM   gv$session
WHERE  module LIKE '%<FORM_MODULE>%'
ORDER BY last_call_et DESC;

This helps determine:

  • Whether only one user is affected

  • Whether multiple users on the same form are inactive

  • Whether multiple sessions have the same wait event

  • Whether the sessions executed the same SQL

  • Whether they share a common blocker

  • Whether the problem began around the same time

If multiple sessions have the same non-idle wait or SQL ID, the issue may be systemic.

7. Check the Relevant Database Object

If SQL analysis identifies an index involved in the activity, review its condition and statistics:

SELECT owner,
       index_name,
       table_owner,
       table_name,
       status,
       visibility,
       num_rows,
       distinct_keys,
       last_analyzed,
       blevel,
       leaf_blocks,
       clustering_factor
FROM   dba_indexes
WHERE  owner      = '<INDEX_OWNER>'
AND    index_name = '<INDEX_NAME>';

Do not rebuild or modify an index simply because:

  • Its statistics are old

  • Its clustering factor is high

  • It appeared in an execution plan

  • It was the last object observed in a monitoring tool

First establish evidence of an actual access-path, corruption, usability, or performance problem.

For Oracle EBS objects, use Oracle-supported maintenance procedures and the appropriate EBS statistics-gathering programs.

8. Build an ASH Activity Timeline

If Oracle Diagnostics Pack is licensed, query Active Session History:

SELECT sample_time,
       inst_id,
       session_id,
       session_serial#,
       sql_id,
       sql_plan_hash_value,
       event,
       wait_class,
       blocking_inst_id,
       blocking_session,
       session_state,
       module,
       action
FROM   gv$active_session_history
WHERE  session_id      = <SID>
AND    session_serial# = <SERIAL_NUMBER>
AND    inst_id         = <INSTANCE_ID>
ORDER BY sample_time DESC;

ASH samples active sessions. A session that has remained idle may therefore have few or no recent samples.

For activity outside the in-memory ASH retention period, licensed environments can query DBA_HIST_ACTIVE_SESS_HISTORY.

ASH, historical ASH, and AWR require the appropriate Oracle Diagnostics Pack licence.

9. Check for an Open Transaction

An Oracle session can be inactive while still holding an uncommitted transaction:

SELECT s.inst_id,
       s.sid,
       s.serial#,
       s.username,
       t.start_time,
       t.used_ublk,
       t.used_urec
FROM   gv$session s
JOIN   gv$transaction t
       ON t.inst_id = s.inst_id
      AND t.addr    = s.taddr
WHERE  s.sid     = <SID>
AND    s.serial# = <SERIAL_NUMBER>
AND    s.inst_id = <INSTANCE_ID>;

If the query returns a row, the session has an active transaction. Terminating it will cause Oracle to roll back that transaction. The rollback may take considerable time depending on the amount of undo generated.

10. Terminate the Session Only After Validation

Terminate a session only after confirming:

  • The SID, serial number, and instance are correct.

  • The session still belongs to the expected EBS user and module.

  • It is stale, orphaned, or causing a confirmed problem.

  • The user has confirmed there is no unsaved work.

  • Application and business approvals have been obtained.

  • The impact of rolling back any open transaction has been assessed.

  • Diagnostic evidence has been saved.

For a single-instance database:

ALTER SYSTEM KILL SESSION '<SID>,<SERIAL_NUMBER>' IMMEDIATE;

For Oracle RAC:

ALTER SYSTEM KILL SESSION '<SID>,<SERIAL_NUMBER>,@<INSTANCE_ID>' IMMEDIATE;

Immediately before executing the command, rerun the session-identification query. This prevents terminating the wrong session if the original session disconnected and its SID was reused.

Confirm the result:

SELECT inst_id,
       sid,
       serial#,
       username,
       status,
       server,
       event,
       module,
       client_identifier
FROM   gv$session
WHERE  sid     = <SID>
AND    serial# = <SERIAL_NUMBER>
AND    inst_id = <INSTANCE_ID>;

The session may temporarily remain visible with a KILLED status while Oracle completes cleanup or transaction rollback.

Recommended Troubleshooting Sequence

  1. Identify the exact database session.

  2. Map it to the EBS application user.

  3. Check for blocked sessions.

  4. Check for an open transaction.

  5. Review the current and previous SQL.

  6. Examine recent waits and historical activity.

  7. Compare other sessions using the same form.

  8. Assess the application and business impact.

  9. Capture evidence and obtain approval.

  10. Revalidate the session identity.

  11. Terminate it only when necessary and safe.

A long-running or inactive Oracle EBS Forms session should not be killed solely because it has remained connected for an extended period. An evidence-based investigation reduces the risk of disconnecting the wrong user, losing unsaved application work, or initiating a large transaction rollback.

Wednesday, August 5, 2026

Oracle Apps DBA Cookbook

 

Oracle Apps DBA Cookbook — Volume 1 | 572 Practical Recipes for EBS 12.2

Oracle Apps DBA Cookbook

Volume 1 — A day-to-day operational guide
572 Recipes EBS 12.2 Oracle Database 19c

Practical procedures for Oracle E-Business Suite 12.2 on Oracle Database 19c.

Compiled from field practice on large-scale EBS estates. Every recipe is designed to be readable, adaptable, and safe when used with proper change control.

1. Scope & Target Environment

Dimension Assumed Baseline Notes
EBS release 12.2.x (12.2.9 / 12.2.10 / 12.2.11+) Online patching (ADOP), dual filesystem fs1/fs2
Database 19c (19.3+), Enterprise Edition Non-CDB and single-PDB both covered
Middle tier WebLogic Server 10.3.6 / FMW 11g oacore, forms, oafm, forms-c4ws managed servers
OS IBM AIX 7.x (primary), Oracle Linux 7/8 AIX-specific recipes flagged [AIX]
HA Data Guard physical standby Broker-managed, Active Data Guard optional
Backup RMAN with catalog or controlfile-only Disk + tape (TSM/SBT) variants

2. Recipe Numbering

Every procedure has a stable ID: R<chapter>.<sequence> — for example R06.23.

IDs are permanent. If a recipe is retired it is marked [RETIRED] rather than renumbered, so cross-references, runbooks, and your own notes never break.

3. Recipe Anatomy

### R02.14 — Short imperative title
Use: One line — when you reach for this.
<script / SQL block>
> Note: gotchas, prerequisites, version differences, destructive warnings.

4. Risk Flags

FlagMeaning
[READ]Read-only. Safe on production at any time.
[WRITE]Modifies data or configuration. Take a backup first.
[OUTAGE]Requires downtime or causes service interruption.
[AIX]AIX-specific syntax or command.
[ROOT]Needs root or a privileged OS account.

Unflagged recipes are [READ] by default.

5. Master Index — 572 Recipes

Ch Title Recipes IDs
01Environment & Foundations24R01.01–R01.24
02SQL Toolkit — EBS Data Dictionary45R02.01–R02.45
03Shell Scripts & Automation40R03.01–R03.40
04Startup, Shutdown & Service Control24R04.01–R04.24
05AutoConfig28R05.01–R05.28
06ADOP / Online Patching45R06.01–R06.45
07Cloning & Refresh38R07.01–R07.38
08WebLogic, FMW & Middle Tier40R08.01–R08.40
09Concurrent Processing35R09.01–R09.35
10RMAN Backup & Recovery40R10.01–R10.40
11Data Guard38R11.01–R11.38
12Health Checks & Monitoring35R12.01–R12.35
13Performance Tuning45R13.01–R13.45
14Troubleshooting Playbooks40R14.01–R14.40
15Security, Users & Access25R15.01–R15.25
16Utilities, Housekeeping & Space30R16.01–R16.30
Total572

Build Status

ChapterStatus
02 — SQL Toolkit Complete
01, 03–16 Catalogued — drafting in sequence

Chapter 02 — SQL Toolkit: The EBS Data Dictionary

45 recipes · R02.01 – R02.45

Run everything here as APPS unless stated otherwise. All recipes are [READ] unless flagged.

Recommended SQL*Plus preamble:
SET LINESIZE 300 PAGESIZE 200 TRIMSPOOL ON
ALTER SESSION SET NLS_DATE_FORMAT = 'DD-MON-YYYY HH24:MI:SS';
Section A — Release, Patches & Editions
R02.01 — Confirm EBS release from the database
Use: First question in every ticket, every Oracle SR.
SELECT release_name,
       applications_system_name,
       last_update_date
  FROM fnd_product_groups;
This is the authoritative release. Do not trust the login page banner — it can be stale after a patch until caches clear.
R02.02 — List product installation status and patch levels
Use: Determining which products are shared, installed or inactive before applying a family pack.
SELECT fa.application_short_name       AS prod,
       fat.application_name,
       fpi.patch_level,
       DECODE(fpi.status,'I','Installed','S','Shared','N','Not installed',fpi.status) AS status,
       fpi.db_status
  FROM fnd_product_installations fpi,
       fnd_application            fa,
       fnd_application_tl         fat
 WHERE fpi.application_id = fa.application_id
   AND fa.application_id  = fat.application_id
   AND fat.language       = USERENV('LANG')
 ORDER BY fa.application_short_name;
R02.03 — Check whether a specific patch has been applied
Use: Oracle Support asks "is patch 12345678 applied?" — answer in ten seconds.
SELECT bug_number,
       creation_date,
       last_update_date
  FROM ad_bugs
 WHERE bug_number = '&patch_number';
AD_BUGS records the bug/patch number. A row here means the patch was applied to this edition of the database. For 12.2, always confirm which edition you are connected to — see R02.08.
R02.04 — List patches applied in a date range
Use: "What changed last weekend?" during a post-change investigation.
SELECT ap.patch_name,
       ap.patch_type,
       apr.end_date,
       apr.success_flag,
       apr.appl_top_id
  FROM ad_applied_patches ap,
       ad_patch_drivers   apd,
       ad_patch_runs      apr
 WHERE ap.applied_patch_id = apd.applied_patch_id
   AND apd.patch_driver_id = apr.patch_driver_id
   AND apr.end_date BETWEEN TO_DATE('&from_date','DD-MON-YYYY')
                        AND TO_DATE('&to_date','DD-MON-YYYY') + 1
 ORDER BY apr.end_date DESC;
R02.05 — Show full patch run detail for one patch
Use: Confirming a patch completed on every node, not just the one you ran it from.
SELECT ap.patch_name,
       at.name              AS appl_top_name,
       apr.start_date,
       apr.end_date,
       apr.success_flag,
       apr.patchtop
  FROM ad_applied_patches ap,
       ad_patch_drivers   apd,
       ad_patch_runs      apr,
       ad_appl_tops       at
 WHERE ap.applied_patch_id = apd.applied_patch_id
   AND apd.patch_driver_id = apr.patch_driver_id
   AND apr.appl_top_id     = at.appl_top_id
   AND ap.patch_name       = '&patch_number'
 ORDER BY apr.start_date;
R02.06 — List all ADOP sessions and their phase status
Use: The single most useful ADOP query. Run it before you run any adop command.
SELECT adop_session_id      AS session_id,
       prepare_status       AS prep,
       apply_status         AS appl,
       finalize_status      AS fnl,
       cutover_status       AS cut,
       cleanup_status       AS clnup,
       abort_status         AS abrt,
       status               AS overall,
       node_name,
       TO_CHAR(prepare_phase_end_date,'DD-MON HH24:MI')  AS prep_end,
       TO_CHAR(cutover_phase_end_date,'DD-MON HH24:MI')  AS cut_end
  FROM ad_adop_sessions
 ORDER BY adop_session_id DESC
 FETCH FIRST 10 ROWS ONLY;
Status codes are Y (completed), N (not done), X (not applicable), F (failed), R (running). A session with status = 'C' is complete. Anything else at the top of this list means you have an open cycle.
R02.07 — List patches applied within an ADOP session
Use: Reconstructing exactly what a patch weekend delivered.
SELECT adop_session_id,
       bug_number,
       patch_file_name,
       node_name,
       applied_file_system_base,
       status,
       TO_CHAR(end_date,'DD-MON-YYYY HH24:MI') AS ended
  FROM ad_adop_session_patches
 WHERE adop_session_id = &session_id
 ORDER BY end_date;
R02.08 — Show current run and patch editions
Use: Knowing which edition your session is actually in. Get this wrong and every other query misleads you.
SELECT SYS_CONTEXT('USERENV','CURRENT_EDITION_NAME') AS my_edition,
       ad_zd.get_edition('RUN')                      AS run_edition,
       ad_zd.get_edition('PATCH')                    AS patch_edition
  FROM dual;

SELECT edition_name, parent_edition_name, usable
  FROM dba_editions
 ORDER BY edition_name;
PATCH returns null when no patch cycle is open. If my_edition is not the run edition and you did not intend that, disconnect and re-source your environment.
R02.09 — Count objects per edition
Use: Judging how much cleanup debt has accumulated across old editions.
SELECT o.edition_name,
       o.object_type,
       COUNT(*) AS obj_count
  FROM dba_objects_ae o
 WHERE o.owner = 'APPS'
   AND o.edition_name IS NOT NULL
 GROUP BY o.edition_name, o.object_type
 ORDER BY o.edition_name, obj_count DESC;
Old editions accumulate if cleanup is skipped. A long tail of editions is a strong signal that full cleanup is overdue.
R02.10 — List invalid objects by owner and type
Use: Standard post-patch and post-clone check.
SELECT owner, object_type, COUNT(*) AS invalid_count
  FROM dba_objects
 WHERE status = 'INVALID'
 GROUP BY owner, object_type
 ORDER BY invalid_count DESC;
R02.11 — Generate a targeted recompile script for invalids [WRITE]
Use: When utlrp is too blunt and you want to recompile a specific set.
SET HEADING OFF FEEDBACK OFF PAGESIZE 0
SPOOL recompile_invalids.sql
SELECT 'ALTER ' ||
       DECODE(object_type,'PACKAGE BODY','PACKAGE',object_type) || ' ' ||
       owner || '.' || object_name || ' COMPILE' ||
       DECODE(object_type,'PACKAGE BODY',' BODY','') || ';'
  FROM dba_objects
 WHERE status = 'INVALID'
   AND object_type IN ('PACKAGE','PACKAGE BODY','PROCEDURE','FUNCTION','TRIGGER','VIEW','SYNONYM')
 ORDER BY DECODE(object_type,'VIEW',1,'SYNONYM',2,'PACKAGE',3,'PACKAGE BODY',4,5);
SPOOL OFF
SET HEADING ON FEEDBACK ON PAGESIZE 200
Review the generated file before running it. In 12.2, prefer adadmin or adop compile options for APPS objects so edition rules are respected.
Section B — Topology, Nodes & Profiles
R02.12 — List registered application tier nodes
Use: Verifying FND_NODES matches physical reality — a frequent source of clone and ADOP failures.
SELECT node_name,
       node_id,
       server_address,
       platform_code,
       status,
       TO_CHAR(creation_date,'DD-MON-YYYY') AS registered
  FROM fnd_nodes
 ORDER BY node_name;
Rows for decommissioned hosts, or a stale AUTHENTICATION node, will break adop. Clean them with FND_CONC_CLONE.SETUP_CLEAN followed by AutoConfig on every tier — never by direct DELETE.
R02.13 — Show which services each node supports
Use: Confirming service distribution matches the intended topology.
SELECT node_name,
       support_cp     AS conc_proc,
       support_forms  AS forms,
       support_web    AS web,
       support_admin  AS admin,
       support_db     AS database,
       virtual_ip
  FROM fnd_nodes
 ORDER BY node_name;
R02.14 — Retrieve a profile option value at every level
Use: Understanding why a setting behaves differently for one user or responsibility.
SELECT po.profile_option_name        AS internal_name,
       pot.user_profile_option_name  AS display_name,
       DECODE(pov.level_id, 10001,'Site',
                            10002,'Application',
                            10003,'Responsibility',
                            10004,'User',
                            10005,'Server',
                            10006,'Organization',
                            10007,'Server+Resp', TO_CHAR(pov.level_id)) AS level_name,
       DECODE(pov.level_id, 10002, app.application_short_name,
                            10003, rsp.responsibility_name,
                            10004, usr.user_name,
                            10005, svr.node_name, 'Site')               AS level_value,
       pov.profile_option_value       AS value,
       pov.last_update_date
  FROM fnd_profile_options       po,
       fnd_profile_options_tl    pot,
       fnd_profile_option_values pov,
       fnd_application           app,
       fnd_responsibility_vl     rsp,
       fnd_user                  usr,
       fnd_nodes                 svr
 WHERE po.profile_option_id     = pov.profile_option_id
   AND po.application_id        = pov.application_id
   AND po.profile_option_name   = pot.profile_option_name
   AND pot.language             = USERENV('LANG')
   AND pov.level_value          = app.application_id (+)
   AND pov.level_value          = rsp.responsibility_id (+)
   AND pov.level_value          = usr.user_id (+)
   AND pov.level_value          = svr.node_id (+)
   AND UPPER(pot.user_profile_option_name) LIKE UPPER('%&profile_name%')
 ORDER BY pov.level_id;
Lower level wins. User beats Responsibility beats Application beats Site.
R02.15 — Find the effective profile value for a specific user
Use: Fastest possible answer when a user reports different behaviour to a colleague.
DECLARE
  l_value VARCHAR2(4000);
BEGIN
  fnd_global.apps_initialize(
     user_id      => (SELECT user_id FROM fnd_user WHERE user_name = UPPER('&user_name')),
     resp_id      => &resp_id,
     resp_appl_id => &resp_appl_id);
  l_value := fnd_profile.value('&profile_internal_name');
  dbms_output.put_line('Effective value: ' || NVL(l_value,'<null>'));
END;
/
Requires SET SERVEROUTPUT ON. Get resp_id and resp_appl_id from R02.18.
R02.16 — List recently changed profile options
Use: Post-incident: "did someone change a profile?"
SELECT pot.user_profile_option_name AS profile_name,
       pov.level_id,
       pov.profile_option_value     AS value,
       pov.last_update_date,
       u.user_name                  AS changed_by
  FROM fnd_profile_option_values pov,
       fnd_profile_options       po,
       fnd_profile_options_tl    pot,
       fnd_user                  u
 WHERE pov.profile_option_id   = po.profile_option_id
   AND pov.application_id      = po.application_id
   AND po.profile_option_name  = pot.profile_option_name
   AND pot.language            = USERENV('LANG')
   AND pov.last_updated_by     = u.user_id
   AND pov.last_update_date > SYSDATE - &days
 ORDER BY pov.last_update_date DESC;
Section C — Users, Responsibilities & Access
R02.17 — List application users and account status
Use: Access review, dormant account cleanup, licence counting.
SELECT user_name,
       description,
       TO_CHAR(start_date,'DD-MON-YYYY')       AS start_date,
       TO_CHAR(end_date,'DD-MON-YYYY')         AS end_date,
       CASE WHEN end_date IS NULL OR end_date > SYSDATE
            THEN 'ACTIVE' ELSE 'INACTIVE' END  AS status,
       TO_CHAR(last_logon_date,'DD-MON-YYYY')  AS last_logon,
       employee_id
  FROM fnd_user
 ORDER BY status, user_name;
R02.18 — Show all responsibilities assigned to a user
Use: "I can't see the menu I used yesterday."
SELECT u.user_name,
       r.responsibility_name,
       r.responsibility_id,
       r.application_id  AS resp_appl_id,
       TO_CHAR(urg.start_date,'DD-MON-YYYY') AS assigned_from,
       TO_CHAR(urg.end_date,'DD-MON-YYYY')   AS assigned_to,
       CASE WHEN urg.end_date IS NULL OR urg.end_date > SYSDATE
            THEN 'ACTIVE' ELSE 'ENDED' END   AS status
  FROM fnd_user                     u,
       fnd_user_resp_groups_direct  urg,
       fnd_responsibility_vl        r
 WHERE u.user_id                 = urg.user_id
   AND urg.responsibility_id     = r.responsibility_id
   AND urg.responsibility_application_id = r.application_id
   AND u.user_name = UPPER('&user_name')
 ORDER BY status, r.responsibility_name;
R02.19 — Show all users holding a given responsibility
Use: Segregation-of-duties review; finding who can approve payments.
SELECT r.responsibility_name,
       u.user_name,
       u.description,
       TO_CHAR(urg.start_date,'DD-MON-YYYY') AS assigned_from,
       TO_CHAR(u.last_logon_date,'DD-MON-YYYY') AS last_logon
  FROM fnd_user                     u,
       fnd_user_resp_groups_direct  urg,
       fnd_responsibility_vl        r
 WHERE u.user_id             = urg.user_id
   AND urg.responsibility_id = r.responsibility_id
   AND urg.responsibility_application_id = r.application_id
   AND (urg.end_date IS NULL OR urg.end_date > SYSDATE)
   AND UPPER(r.responsibility_name) LIKE UPPER('%&resp_name%')
 ORDER BY u.user_name;
R02.20 — Map responsibility → request group → concurrent programs
Use: "Why can't this user submit that report?" Ninety percent of the time, the answer is here.
SELECT r.responsibility_name,
       rg.request_group_name,
       cp.concurrent_program_name AS short_name,
       cpt.user_concurrent_program_name AS program_name
  FROM fnd_responsibility_vl        r,
       fnd_request_groups           rg,
       fnd_request_group_units      rgu,
       fnd_concurrent_programs      cp,
       fnd_concurrent_programs_tl   cpt
 WHERE r.request_group_id         = rg.request_group_id
   AND r.application_id           = rg.application_id
   AND rg.request_group_id        = rgu.request_group_id
   AND rg.application_id          = rgu.application_id
   AND rgu.request_unit_id        = cp.concurrent_program_id
   AND cp.concurrent_program_id   = cpt.concurrent_program_id
   AND cpt.language               = USERENV('LANG')
   AND UPPER(r.responsibility_name) LIKE UPPER('%&resp_name%')
 ORDER BY cpt.user_concurrent_program_name;
R02.21 — Find users with System Administrator access
Use: The first query any auditor asks for.
SELECT u.user_name,
       u.description,
       r.responsibility_name,
       TO_CHAR(u.last_logon_date,'DD-MON-YYYY') AS last_logon
  FROM fnd_user                    u,
       fnd_user_resp_groups_direct urg,
       fnd_responsibility_vl       r
 WHERE u.user_id             = urg.user_id
   AND urg.responsibility_id = r.responsibility_id
   AND urg.responsibility_application_id = r.application_id
   AND (urg.end_date IS NULL OR urg.end_date > SYSDATE)
   AND (u.end_date  IS NULL OR u.end_date  > SYSDATE)
   AND r.responsibility_name IN ('System Administrator','System Administration',
                                 'Application Developer','Functional Administrator')
 ORDER BY r.responsibility_name, u.user_name;
R02.22 — Check APPS/APPLSYS database account status and expiry
Use: Preventing the classic "everything died at 2am because the password expired".
SELECT username,
       account_status,
       TO_CHAR(lock_date,'DD-MON-YYYY')    AS locked_on,
       TO_CHAR(expiry_date,'DD-MON-YYYY')  AS expires_on,
       profile,
       TO_CHAR(created,'DD-MON-YYYY')      AS created
  FROM dba_users
 WHERE username IN ('APPS','APPLSYS','APPLSYSPUB','SYSTEM','SYS','APPS_NE','EBS_SYSTEM')
    OR username LIKE 'XX%'
 ORDER BY username;
Run as a DBA account. EBS_SYSTEM exists only on 12.2.10 and later after the EBS System Schema Migration.
Section D — Concurrent Processing
R02.23 — List currently running concurrent requests with elapsed time
Use: Every single morning, and the moment anyone says "the system is slow".
SELECT fcr.request_id,
       fcpt.user_concurrent_program_name AS program,
       fu.user_name                      AS submitted_by,
       TO_CHAR(fcr.actual_start_date,'DD-MON HH24:MI') AS started,
       ROUND((SYSDATE - fcr.actual_start_date)*24*60,1) AS mins_running,
       fcr.oracle_process_id  AS spid,
       fcr.os_process_id      AS os_pid,
       fcr.phase_code, fcr.status_code
  FROM fnd_concurrent_requests    fcr,
       fnd_concurrent_programs_tl fcpt,
       fnd_user                   fu
 WHERE fcr.concurrent_program_id = fcpt.concurrent_program_id
   AND fcr.program_application_id = fcpt.application_id
   AND fcpt.language = USERENV('LANG')
   AND fcr.requested_by = fu.user_id
   AND fcr.phase_code = 'R'
 ORDER BY fcr.actual_start_date;
phase_code: P=Pending, R=Running, C=Completed, I=Inactive. status_code: N=Normal, E=Error, G=Warning, W=Paused, Q=Standby, R=Normal-running.
R02.24 — Map a concurrent request to its database session and OS process
Use: You need to trace, kill, or explain a specific request.
SELECT fcr.request_id,
       fcpt.user_concurrent_program_name AS program,
       s.sid, s.serial#, s.status,
       p.spid                            AS os_pid,
       s.event                           AS current_wait,
       s.sql_id,
       ROUND(s.last_call_et/60,1)        AS mins_in_call
  FROM fnd_concurrent_requests    fcr,
       fnd_concurrent_programs_tl fcpt,
       v$session                  s,
       v$process                  p
 WHERE fcr.concurrent_program_id  = fcpt.concurrent_program_id
   AND fcr.program_application_id = fcpt.application_id
   AND fcpt.language = USERENV('LANG')
   AND fcr.oracle_process_id = p.spid
   AND s.paddr = p.addr
   AND fcr.request_id = &request_id;
R02.25 — Show concurrent manager status and process counts
Use: Target vs actual processes tells you instantly whether the ICM is healthy.
SELECT fcq.concurrent_queue_name       AS queue,
       fcqt.user_concurrent_queue_name AS manager,
       fcq.max_processes               AS target,
       fcq.running_processes           AS actual,
       DECODE(fcq.control_code,'D','Deactivating','E','Deactivated',
                               'N','Starting up','A','Activating',
                               'X','Terminated', 'R','Restarting',
                               NULL,'Running', fcq.control_code) AS control_state,
       fcq.enabled_flag,
       fcq.target_node
  FROM fnd_concurrent_queues    fcq,
       fnd_concurrent_queues_tl fcqt
 WHERE fcq.concurrent_queue_id = fcqt.concurrent_queue_id
   AND fcq.application_id      = fcqt.application_id
   AND fcqt.language = USERENV('LANG')
   AND fcq.enabled_flag = 'Y'
 ORDER BY fcq.concurrent_queue_name;
actual < target on the Standard Manager during working hours is a live problem. actual = 0 on the ICM means the whole subsystem is down.
R02.26 — Measure pending request backlog by manager
Use: Deciding whether to add processes or investigate a blockage.
SELECT NVL(fcqt.user_concurrent_queue_name,'<unassigned>') AS manager,
       COUNT(*)                                            AS pending_count,
       MIN(fcr.requested_start_date)                       AS oldest_queued,
       ROUND((SYSDATE - MIN(fcr.requested_start_date))*24*60,1) AS oldest_wait_mins
  FROM fnd_concurrent_requests    fcr,
       fnd_concurrent_queues      fcq,
       fnd_concurrent_queues_tl   fcqt
 WHERE fcr.concurrent_queue_id      = fcq.concurrent_queue_id (+)
   AND fcr.queue_application_id     = fcq.application_id (+)
   AND fcq.concurrent_queue_id      = fcqt.concurrent_queue_id (+)
   AND fcq.application_id           = fcqt.application_id (+)
   AND fcqt.language (+)            = USERENV('LANG')
   AND fcr.phase_code = 'P'
   AND fcr.requested_start_date <= SYSDATE
 GROUP BY fcqt.user_concurrent_queue_name
 ORDER BY pending_count DESC;
R02.27 — List requests completed in error in the last 24 hours
Use: Morning check. Ideally this returns nothing.
SELECT fcr.request_id,
       fcpt.user_concurrent_program_name AS program,
       fu.user_name                      AS submitted_by,
       TO_CHAR(fcr.actual_completion_date,'DD-MON HH24:MI') AS completed,
       fcr.status_code,
       SUBSTR(fcr.completion_text,1,120) AS completion_text
  FROM fnd_concurrent_requests    fcr,
       fnd_concurrent_programs_tl fcpt,
       fnd_user                   fu
 WHERE fcr.concurrent_program_id  = fcpt.concurrent_program_id
   AND fcr.program_application_id = fcpt.application_id
   AND fcpt.language = USERENV('LANG')
   AND fcr.requested_by = fu.user_id
   AND fcr.phase_code  = 'C'
   AND fcr.status_code IN ('E','G','T')
   AND fcr.actual_completion_date > SYSDATE - 1
 ORDER BY fcr.actual_completion_date DESC;
R02.28 — Find the longest-running programs over the last 7 days
Use: Building the tuning candidate list, and spotting runtime regression.
SELECT fcpt.user_concurrent_program_name AS program,
       COUNT(*)                          AS runs,
       ROUND(AVG((fcr.actual_completion_date - fcr.actual_start_date)*24*60),1) AS avg_mins,
       ROUND(MAX((fcr.actual_completion_date - fcr.actual_start_date)*24*60),1) AS max_mins,
       ROUND(SUM((fcr.actual_completion_date - fcr.actual_start_date)*24*60),1) AS total_mins
  FROM fnd_concurrent_requests    fcr,
       fnd_concurrent_programs_tl fcpt
 WHERE fcr.concurrent_program_id  = fcpt.concurrent_program_id
   AND fcr.program_application_id = fcpt.application_id
   AND fcpt.language = USERENV('LANG')
   AND fcr.phase_code = 'C'
   AND fcr.actual_start_date > SYSDATE - 7
   AND fcr.actual_completion_date IS NOT NULL
 GROUP BY fcpt.user_concurrent_program_name
 HAVING SUM((fcr.actual_completion_date - fcr.actual_start_date)*24*60) > 30
 ORDER BY total_mins DESC
 FETCH FIRST 25 ROWS ONLY;
Order by total_mins, not max_mins. A 90-second program running 4,000 times a day costs you more than one nightly 40-minute batch.
R02.29 — Look up a concurrent program definition by name
Use: Finding the short name, application and execution method before you trace or clone it.
SELECT cpt.user_concurrent_program_name AS program_name,
       cp.concurrent_program_name       AS short_name,
       fa.application_short_name        AS application,
       cp.enabled_flag,
       cp.execution_method_code,
       cp.enable_trace,
       cp.run_alone_flag
  FROM fnd_concurrent_programs    cp,
       fnd_concurrent_programs_tl cpt,
       fnd_application            fa
 WHERE cp.concurrent_program_id = cpt.concurrent_program_id
   AND cp.application_id        = cpt.application_id
   AND cp.application_id        = fa.application_id
   AND cpt.language = USERENV('LANG')
   AND UPPER(cpt.user_concurrent_program_name) LIKE UPPER('%&program_name%')
 ORDER BY cpt.user_concurrent_program_name;
R02.30 — Find the executable and file behind a concurrent program
Use: Locating the actual .sql, .prog, .rdf or Java class you need to inspect or patch.
SELECT cpt.user_concurrent_program_name AS program_name,
       cp.concurrent_program_name       AS short_name,
       fe.executable_name,
       fe.execution_file_name,
       DECODE(fe.execution_method_code,
              'I','PL/SQL Stored Procedure','P','Oracle Reports',
              'H','Host','S','Immediate','J','Java Concurrent Program',
              'K','Java Stored Procedure','L','SQL*Loader','Q','SQL*Plus',
              'B','Request Set Stage Function','A','Spawned',
              fe.execution_method_code) AS exec_method,
       fa.application_short_name        AS exec_application
  FROM fnd_concurrent_programs    cp,
       fnd_concurrent_programs_tl cpt,
       fnd_executables            fe,
       fnd_application            fa
 WHERE cp.concurrent_program_id = cpt.concurrent_program_id
   AND cp.application_id        = cpt.application_id
   AND cp.executable_id         = fe.executable_id
   AND cp.executable_application_id = fe.application_id
   AND fe.application_id        = fa.application_id
   AND cpt.language = USERENV('LANG')
   AND UPPER(cpt.user_concurrent_program_name) LIKE UPPER('%&program_name%');
R02.31 — Report request volume by user and responsibility
Use: Capacity planning, and identifying who is generating the load.
SELECT fu.user_name,
       fr.responsibility_name,
       COUNT(*) AS request_count,
       ROUND(SUM((fcr.actual_completion_date - fcr.actual_start_date)*24*60),1) AS total_mins
  FROM fnd_concurrent_requests fcr,
       fnd_user                fu,
       fnd_responsibility_vl   fr
 WHERE fcr.requested_by         = fu.user_id
   AND fcr.responsibility_id    = fr.responsibility_id (+)
   AND fcr.responsibility_application_id = fr.application_id (+)
   AND fcr.request_date > SYSDATE - &days
 GROUP BY fu.user_name, fr.responsibility_name
 ORDER BY request_count DESC
 FETCH FIRST 30 ROWS ONLY;
Section E — Sessions, Locks & Contention
R02.32 — Identify blocking sessions and the blocked chain
Use: "The screen is frozen." This is almost always the answer.
SELECT LPAD(' ', 2*(LEVEL-1)) || s.sid AS blocking_tree,
       s.sid, s.serial#, s.username,
       s.module, s.action,
       s.status,
       s.event                AS waiting_on,
       ROUND(s.seconds_in_wait/60,1) AS wait_mins,
       s.blocking_session,
       p.spid                 AS os_pid
  FROM v$session s, v$process p
 WHERE s.paddr = p.addr
 START WITH s.blocking_session IS NULL
        AND s.sid IN (SELECT blocking_session FROM v$session WHERE blocking_session IS NOT NULL)
 CONNECT BY PRIOR s.sid = s.blocking_session
 ORDER SIBLINGS BY s.sid;
The root of each tree is the true culprit. Killing a leaf achieves nothing.
R02.33 — Show active EBS sessions with module, action and client info
Use: Attributing a database session back to a real person or a request.
SELECT s.sid, s.serial#, s.username AS db_user,
       fu.user_name                 AS ebs_user,
       s.module, s.action,
       s.machine, s.program,
       s.status,
       s.sql_id,
       s.event,
       ROUND(s.last_call_et/60,1)   AS mins_in_call,
       p.spid                       AS os_pid
  FROM v$session s,
       v$process p,
       fnd_logins fl,
       fnd_user   fu
 WHERE s.paddr        = p.addr
   AND s.audsid       = fl.spid (+)
   AND fl.user_id     = fu.user_id (+)
   AND s.username     = 'APPS'
   AND s.status       = 'ACTIVE'
   AND s.type         = 'USER'
 ORDER BY s.last_call_et DESC;
FND_LOGINS linkage is only reliable when Sign-On Audit is enabled. MODULE set by EBS usually carries the form or program name and is the more dependable clue.
R02.34 — Generate a kill-session script for a request or session [WRITE]
Use: Terminating a runaway request cleanly, with an audit trail of exactly what you killed.
SELECT 'ALTER SYSTEM KILL SESSION ''' || s.sid || ',' || s.serial#
       || ',@' || s.inst_id || ''' IMMEDIATE;  -- req ' || fcr.request_id
       || ' user ' || fu.user_name AS kill_command
  FROM gv$session               s,
       gv$process               p,
       fnd_concurrent_requests  fcr,
       fnd_user                 fu
 WHERE s.paddr = p.addr
   AND s.inst_id = p.inst_id
   AND fcr.oracle_process_id = p.spid
   AND fcr.requested_by = fu.user_id
   AND fcr.request_id = &request_id;
Always cancel the request from the Concurrent Requests form first and give it a minute. Killing the session leaves the request in Running with no process, which then needs the manager cleanup procedure. Never kill the ICM's own session.
Section F — Space, Storage & Statistics
R02.35 — Summarise tablespace usage and headroom
Use: Daily. This and the alert log are the two checks you never skip.
SELECT df.tablespace_name,
       ROUND(df.alloc_mb)                                AS alloc_mb,
       ROUND(df.max_mb)                                  AS max_mb,
       ROUND(df.alloc_mb - NVL(fs.free_mb,0))            AS used_mb,
       ROUND(100*(df.alloc_mb - NVL(fs.free_mb,0))/df.alloc_mb,1)  AS pct_used_alloc,
       ROUND(100*(df.alloc_mb - NVL(fs.free_mb,0))/df.max_mb,1)    AS pct_used_max,
       df.file_count
  FROM (SELECT tablespace_name,
               SUM(bytes)/1024/1024                            AS alloc_mb,
               SUM(GREATEST(bytes, NVL(maxbytes,bytes)))/1024/1024 AS max_mb,
               COUNT(*)                                        AS file_count
          FROM dba_data_files GROUP BY tablespace_name) df,
       (SELECT tablespace_name, SUM(bytes)/1024/1024 AS free_mb
          FROM dba_free_space GROUP BY tablespace_name) fs
 WHERE df.tablespace_name = fs.tablespace_name (+)
 ORDER BY pct_used_max DESC;
pct_used_max is the number that matters. A tablespace at 98% allocated but 40% of maximum is fine; one at 60% allocated with autoextend off is not.
R02.36 — List the top segments by size
Use: Finding what is actually consuming your growth before you add another datafile.
SELECT owner, segment_name, segment_type, tablespace_name,
       ROUND(bytes/1024/1024/1024,2) AS size_gb,
       partition_name
  FROM dba_segments
 ORDER BY bytes DESC
 FETCH FIRST 30 ROWS ONLY;
R02.37 — Report datafile autoextend configuration and ceiling
Use: Catching the datafile with autoextend off before it catches you at 3am.
SELECT tablespace_name,
       file_name,
       ROUND(bytes/1024/1024)     AS current_mb,
       autoextensible,
       ROUND(increment_by * (SELECT value/1024/1024 FROM v$parameter WHERE name='db_block_size'),1) AS next_mb,
       ROUND(maxbytes/1024/1024)  AS max_mb
  FROM dba_data_files
 WHERE autoextensible = 'NO'
    OR maxbytes < bytes * 1.2
 ORDER BY tablespace_name, file_name;
R02.38 — Show temp tablespace usage by session
Use: Identifying the one query eating 200GB of temp.
SELECT s.sid, s.serial#, s.username, s.module,
       s.sql_id,
       ROUND(SUM(u.blocks) * (SELECT value FROM v$parameter WHERE name='db_block_size')
             /1024/1024/1024, 2) AS temp_gb,
       u.tablespace
  FROM v$session s, v$sort_usage u
 WHERE s.saddr = u.session_addr
 GROUP BY s.sid, s.serial#, s.username, s.module, s.sql_id, u.tablespace
 ORDER BY temp_gb DESC;
R02.39 — Report undo usage, retention and tuned retention
Use: Diagnosing ORA-01555 and sizing undo for long-running batch.
SELECT (SELECT value FROM v$parameter WHERE name='undo_retention')     AS undo_retention_s,
       (SELECT TO_CHAR(tuned_undoretention) FROM v$undostat
         WHERE ROWNUM = 1 ORDER BY begin_time DESC)                    AS tuned_retention_s,
       (SELECT ROUND(SUM(bytes)/1024/1024/1024,2) FROM dba_data_files
         WHERE tablespace_name = (SELECT value FROM v$parameter WHERE name='undo_tablespace')) AS undo_gb,
       (SELECT COUNT(*) FROM dba_undo_extents WHERE status='ACTIVE')   AS active_extents,
       (SELECT COUNT(*) FROM dba_undo_extents WHERE status='EXPIRED')  AS expired_extents
  FROM dual;
R02.40 — Check FND_STATS gather history by schema
Use: Confirming the "Gather Schema Statistics" request actually did what it claimed.
SELECT schema_name,
       COUNT(*)                       AS objects_gathered,
       MIN(last_gather_start_time)    AS first_start,
       MAX(last_gather_end_time)      AS last_end
  FROM fnd_stats_hist
 WHERE last_gather_start_time > SYSDATE - &days
 GROUP BY schema_name
 ORDER BY last_end DESC;
In EBS, always gather statistics through FND_STATS or the Gather Schema Statistics concurrent program — not raw DBMS_STATS. FND_STATS applies the EBS-specific settings and histogram handling.
R02.41 — Identify objects with stale or missing optimizer statistics
Use: Explaining a sudden plan change after a large data load.
SELECT owner, table_name, num_rows,
       TO_CHAR(last_analyzed,'DD-MON-YYYY HH24:MI') AS last_analyzed,
       stale_stats,
       partitioned
  FROM dba_tab_statistics
 WHERE owner IN ('APPS','APPLSYS','AP','AR','GL','INV','ONT','PO','WIP','XX_CUSTOM')
   AND (stale_stats = 'YES' OR last_analyzed IS NULL)
   AND object_type = 'TABLE'
   AND NVL(num_rows,0) > 10000
 ORDER BY num_rows DESC NULLS FIRST
 FETCH FIRST 40 ROWS ONLY;
R02.42 — Report FND_LOBS size and SecureFile conversion status
Use: FND_LOBS is routinely the largest object in an EBS database. Check it monthly.
SELECT l.owner, l.table_name, l.column_name,
       l.segment_name,
       l.securefile,
       l.compression, l.deduplication,
       ROUND(s.bytes/1024/1024/1024,2) AS lob_gb
  FROM dba_lobs l, dba_segments s
 WHERE l.segment_name = s.segment_name
   AND l.owner        = s.owner
   AND l.table_name   = 'FND_LOBS'
 ORDER BY s.bytes DESC;

-- Row count and age profile
SELECT TO_CHAR(upload_date,'YYYY-MM') AS month,
       COUNT(*)                       AS rows_loaded,
       ROUND(SUM(DBMS_LOB.GETLENGTH(file_data))/1024/1024/1024,2) AS gb
  FROM applsys.fnd_lobs
 GROUP BY TO_CHAR(upload_date,'YYYY-MM')
 ORDER BY month DESC
 FETCH FIRST 24 ROWS ONLY;
The second query reads every LOB and is expensive on a large table — run it off-hours. Conversion from BasicFile to SecureFile is covered in R16.07.
R02.43 — Inventory custom (XX*) schemas and objects
Use: Impact assessment before an upgrade, and handover documentation.
SELECT owner, object_type, COUNT(*) AS obj_count,
       MAX(last_ddl_time) AS most_recent_change
  FROM dba_objects
 WHERE owner LIKE 'XX%'
    OR (owner = 'APPS' AND object_name LIKE 'XX%')
 GROUP BY owner, object_type
 ORDER BY owner, obj_count DESC;
R02.44 — Check Workflow backlog: open items and notifications
Use: Approvals not arriving is nearly always visible here first.
-- Open workflow items by type
SELECT item_type,
       COUNT(*)                AS open_items,
       MIN(begin_date)         AS oldest,
       ROUND(SYSDATE - MIN(begin_date)) AS oldest_days
  FROM wf_items
 WHERE end_date IS NULL
 GROUP BY item_type
 ORDER BY open_items DESC;

-- Notification mailer backlog
SELECT status, mail_status, COUNT(*) AS notif_count,
       MIN(begin_date) AS oldest
  FROM wf_notifications
 WHERE status = 'OPEN'
 GROUP BY status, mail_status
 ORDER BY notif_count DESC;
A growing count with mail_status = 'MAIL' means the notification mailer is not sending. See Ch. 09 R09.32.
R02.45 — Verify database initialization parameters against EBS requirements
Use: After any DB patch, clone or parameter change. Deviations here cause the strangest bugs.
SELECT name,
       value,
       isdefault,
       ismodified,
       description
  FROM v$parameter
 WHERE name IN ('compatible','optimizer_features_enable','nls_length_semantics',
                'nls_comp','nls_sort','_system_trig_enabled','sga_target',
                'pga_aggregate_target','processes','sessions','session_cached_cursors',
                'open_cursors','db_block_size','db_files','undo_management',
                'plsql_code_type','plsql_optimize_level','optimizer_adaptive_plans',
                'parallel_max_servers','job_queue_processes','max_string_size',
                'shared_pool_size','result_cache_max_size','db_writer_processes')
 ORDER BY name;

-- Non-default hidden parameters (frequent source of surprise after a clone)
SELECT a.ksppinm AS parameter, b.ksppstvl AS value, b.ksppstdf AS is_default
  FROM x$ksppi a, x$ksppcv b
 WHERE a.indx = b.indx
   AND a.ksppinm LIKE '\_%' ESCAPE '\'
   AND b.ksppstdf = 'FALSE'
 ORDER BY a.ksppinm;
Run the second query as SYS. Compare the result against the EBS 12.2 database parameter note for your exact release before changing anything — some underscore parameters are mandatory for EBS and must not be removed.