Wednesday, August 13, 2025

pro

 You are an expert Oracle Apps DBA and Python developer. Write a complete Python script designed to run on a Solaris server to monitor and report on long-running Oracle EBS concurrent requests.

The script must meet the following requirements:

  • Database Connection: Use the cx_Oracle library. The connection must be simple, using only a username and password, assuming the full database connection details are already set in the shell environment by sourcing a file like EBSapps.env.

  • Monitoring Logic: Identify all concurrent requests that are in a 'Running' phase and have been running longer than a configurable time threshold (defaulting to 60 minutes).

  • Data to Capture: For each long-running request, retrieve essential details including the Request ID, Program Name, Submitter, Elapsed Time, Start Time, Database Session ID (SID), and the current SQL text being executed.

  • HTML Report: Format the collected data into a clean, professional HTML table. The SQL text should be wrapped in <pre> tags to preserve formatting.

  • Email Delivery: The script must not use SMTP libraries. Instead, it must send the generated HTML report as the body of an email using the system's mailx command. Ensure the command is constructed properly to send the email with an HTML content type so it renders correctly in email clients."

Sunday, August 10, 2025

Troubleshooting Oracle EBS Workflow Mailer: HTTPClientException for Framework Regions

Troubleshooting Oracle EBS Workflow Mailer: HTTPClientException for Framework Regions

Troubleshooting Oracle EBS Workflow Mailer: HTTPClientException for Framework Regions

If you're an Oracle Apps DBA, you've likely run into issues with the Workflow Mailer. A particularly tricky one is when email notifications with embedded OA Framework regions fail to send. This post will walk you through identifying the problem, understanding the cause, and applying a comprehensive solution.


Symptom

You'll notice that certain workflow email notifications just aren't going out. When you check the Workflow Mailer log file (e.g., FNDCPGSC*.txt), you'll find a distinct error message that points to a problem fetching HTML content.

Problem getting the HTML content -> oracle.apps.fnd.wf.mailer.NotificationFormatter$FormatterSAXException: Problem obtaining the HTML content -> oracle.apps.fnd.wf.common.HTTPClientException: Unable to invoke method HTTPClient.HTTPConnection.Get

Cause

This error is almost always caused by an incorrect configuration of the profile option "WF: Workflow Mailer Framework Web Agent" (internal name WF_MAIL_WEB_AGENT).

When a notification includes an embedded OA Framework region, the mailer needs to make an HTTP call to the web server to render that region into HTML for the email body. If this profile option is blank, or points to a load balancer address instead of a direct, physical web server URL, the mailer can't make the connection, and the notification fails.


Solution

Here’s the step-by-step guide to fix it. You can safely perform these steps in a production environment, but we always recommend testing in a non-production instance first.

  1. Stop the Workflow Mailer and Agent Listener services.
    1. Log in to Oracle Application Manager (OAM).
    2. Navigate to Workflow Manager -> Service Components.
    3. Select and stop the "Workflow Notification Mailer" and "Workflow Agent Listener Service".
    4. Wait for the services to show a status of "Deactivated" with 0 actual and 0 target processes.
    5. Confirm they are stopped with the following SQL query:
      SELECT component_name, component_status, component_status_info
      FROM fnd_svc_components_v
      WHERE component_name like 'Workflow%';
  2. Find the tablespace for the workflow queue indexes.

    You'll need this tablespace name in a later step. Run this query to find it:

    SELECT DISTINCT tablespace_name
    FROM dba_indexes, dba_queues
    WHERE index_name LIKE 'WF%N1'
    AND table_name = queue_table
    AND name LIKE 'WF%';
  3. Set the "WF: Workflow Mailer Framework Web Agent" profile option.

    This is the core of the fix. You need to set this profile option at the Site level to a URL pointing to one of your physical application web servers.

    To construct the correct URL, find these values in the context file ($CONTEXT_FILE) on one of your web nodes:

    $ grep -ie s_webhost $CONTEXT_FILE
    $ grep -ie s_domainname $CONTEXT_FILE
    $ grep -ie s_webport $CONTEXT_FILE

    Your URL will be in the format http://<s_webhost>.<s_domainname>:<s_webport>. For example: http://myhost.mydomain.com:8000.

    Note:

    • The protocol must be http, not https, as the connection is internal.
    • The port must be the web port (s_webport), not an SSL port.

    Validate the URL from any application tier node using wget. You should get a "200 OK" response.

    $ wget http://myhost.mydomain.com:8000
    ...
    HTTP request sent, awaiting response... 200 OK
    ...
  4. Rebuild the Workflow Mailer queue.

    Connect to the database as the APPS user and run the following script:

    @$FND_TOP/patch/115/sql/wfntfqup.sql APPS <APPS_SCHEMA_PASSWORD> APPLSYS
  5. Recreate the index on the CORRID column.

    Connect as the APPLSYS user and run the following SQL. When prompted, enter the tablespace name you found in step 2.

    CREATE INDEX WF_NOTIFICATION_OUT_N1
    ON WF_NOTIFICATION_OUT(CORRID)
    STORAGE (INITIAL 1M NEXT 1M MINEXTENTS 1 MAXEXTENTS 2147483645 PCTINCREASE 0)
    TABLESPACE &tbs;
  6. Start the Workflow Mailer and Agent Listener services.

    Go back to OAM (Workflow Manager -> Service Components) and start the services you stopped in step 1. Verify they become "Activated" with 1 actual and 1 target process.

  7. Wait and retest.

    Allow about 30 minutes for the mailer to process the backlog of notifications in the queue. Then, test the fix with a fresh notification that previously failed.

IMPORTANT: Prevent AutoConfig from Undoing Your Fix

Your hard work can be undone the next time you run AutoConfig. To prevent this, you must ensure the context file variable s_wfmail_agent is set to the same physical web server URL on ALL application nodes. If this variable is blank, AutoConfig will overwrite your profile option setting, and the issue will return.


Initial Diagnostic Information

Before you begin, it's helpful to gather some baseline information. Here are the queries and commands often used to diagnose mailer issues.

1. Check the Mailer Queue Status:

SELECT
    corr_id CORRID,
    wfno.user_data.GET_STRING_PROPERTY('BES_EVENT_KEY') NOTIFICATION_ID,
    wfno.user_data.GET_STRING_PROPERTY('ROLE') ROLE,
    wfno.msg_state STATE,
    to_char(enq_time, 'YYYY/MM/DD HH24:MI:SS') enq_time,
    to_char(deq_time, 'YYYY/MM/DD HH24:MI:SS') deq_time,
    to_char(sysdate, 'YYYY/MM/DD HH24:MI:SS') sys_date,
    retry_count RETRY
FROM applsys.aq$wf_notification_out wfno
WHERE wfno.user_data.GET_STRING_PROPERTY('BES_EVENT_NAME') <> 'oracle.apps.wf.notification.summary.send';

2. Get Workflow Version Information:

-- Navigate to $FND_TOP/sql
-- Connect to SQL*Plus as the APPS user

spool wfver.txt
sta wfver.sql
spool off

3. Find Active Mailer Log Files:

SELECT logfile_name
FROM fnd_concurrent_processes
WHERE process_status_code = 'A'
AND concurrent_queue_id IN
(SELECT concurrent_queue_id FROM fnd_concurrent_queues
 WHERE concurrent_queue_name IN ('WFMLRSVC','WFALSNRSVC'));

ASH & AWR Performance Tuning Pack

ASH & AWR Performance Tuning Pack (RAC-aware)

ASH & AWR Performance Tuning Pack

A RAC-aware script for Oracle DBAs (11gR2–19c+)

This script collection provides a powerful toolkit for diagnosing performance issues in Oracle databases using the Active Session History (ASH) and Automatic Workload Repository (AWR). It's designed to be RAC-aware and works on versions from 11gR2 to 19c and beyond.

Requirements: SELECT_CATALOG_ROLE. AWR queries require the Oracle Diagnostics Pack license.

set pages 200 lines 200 trimspool on long 100000 longchunksize 100000
col event format a50 trunc
col wait_class format a20 trunc
col username format a20
col module format a35 trunc
col sql_text format a80 trunc
col object_name format a40 trunc
col object_type format a18 trunc
col segment_name format a35 trunc
set verify off

-- ---- Parameters (adjust as you like)
DEF mins_back = 60
DEF days_back = 1
DEF top_n     = 5
DEF hour_from = 9
DEF hour_to   = 11
DEF start_ts  = '2012-11-14 22:00'
DEF end_ts    = '2012-11-14 23:00'
DEF sqlid     = '&&sqlid'      -- set when needed

1. Top Recent Wait Events (ASH)

Shows the top wait events from the last &mins_back minutes.

WITH ash AS (
  SELECT /*+ MATERIALIZE */
         event, wait_class,
         (wait_time + time_waited) AS wt_us
  FROM   gv$active_session_history
  WHERE  sample_time >= systimestamp - ( &mins_back / (24*60) )
    AND  event IS NOT NULL
)
SELECT * FROM (
  SELECT event,
         wait_class,
         ROUND(SUM(wt_us)/1e6,2) AS total_wait_s,
         ROUND(100*SUM(wt_us)/NULLIF(SUM(SUM(wt_us)) OVER (),0),1) AS pct
  FROM   ash
  GROUP  BY event, wait_class
  ORDER  BY total_wait_s DESC
)
FETCH FIRST &top_n ROWS ONLY;

2. Top Wait Class Since Startup

Aggregates wait times by class across the entire system since the last startup.

SELECT wait_class,
       SUM(time_waited) AS time_waited_cs,
       ROUND(SUM(time_waited)/100,1) AS time_waited_s
FROM   v$system_event e JOIN v$event_name n ON n.event_id = e.event_id
WHERE  n.wait_class <> 'Idle'
GROUP  BY wait_class
ORDER  BY time_waited_cs DESC;

3. Users Currently Waiting (non-Idle)

Lists active user sessions that are currently in a non-idle wait state.

SELECT s.inst_id, s.sid, s.serial#, s.username, s.module,
       s.event, s.wait_class, s.state, s.seconds_in_wait
FROM   gv$session s
WHERE  s.type = 'USER'
  AND  s.username IS NOT NULL
  AND  s.state = 'WAITING'
  AND  s.wait_class <> 'Idle'
ORDER  BY s.seconds_in_wait DESC;

4. Main DB Wait Events in a Time Interval (AWR)

First, find the snapshot range for your desired time window.

SELECT MIN(snap_id) begin_snap, MAX(snap_id) end_snap,
       TO_CHAR(MIN(begin_interval_time),'YYYY-MM-DD HH24:MI') begin_time,
       TO_CHAR(MAX(end_interval_time) ,'YYYY-MM-DD HH24:MI') end_time
FROM   dba_hist_snapshot
WHERE  end_interval_time BETWEEN TO_DATE('&&start_ts','YYYY-MM-DD HH24:MI')
                         AND TO_DATE('&&end_ts','YYYY-MM-DD HH24:MI');

Then, define begin_snap and end_snap and run the query below to see top events in that range.

-- DEF begin_snap = 12345
-- DEF end_snap   = 12350
-- SELECT * FROM (
--   SELECT h.event, h.wait_class,
--          ROUND(SUM(h.time_waited)/1e6,2) total_wait_s
--   FROM   dba_hist_active_sess_history h
--   WHERE  h.snap_id BETWEEN &begin_snap AND &end_snap
--     AND  h.event IS NOT NULL
--   GROUP  BY h.event, h.wait_class
--   ORDER  BY total_wait_s DESC
-- ) FETCH FIRST &top_n ROWS ONLY;

5. Top CPU-consuming SQL in a Time Window (AWR)

Identifies the most CPU-intensive SQL statements between specific hours over the last &days_back days.

SELECT * FROM (
  SELECT a.sql_id,
         ROUND(SUM(a.cpu_time_delta)/1e6,2) cpu_s,
         ROUND(SUM(a.elapsed_time_delta)/1e6,2) ela_s,
         SUM(a.executions_delta) execs
  FROM   dba_hist_sqlstat a
  JOIN   dba_hist_snapshot s ON s.snap_id = a.snap_id
  WHERE  s.begin_interval_time >= TRUNC(SYSDATE) - &days_back
    AND  EXTRACT(HOUR FROM s.end_interval_time) BETWEEN &hour_from AND &hour_to
  GROUP  BY a.sql_id
  ORDER  BY cpu_s DESC
) FETCH FIRST &top_n ROWS ONLY;

6. Objects with Most Waits in Past Hour (ASH)

Pinpoints which database objects (tables, indexes) have been sources of contention recently.

SELECT * FROM (
  SELECT o.owner||'.'||o.object_name AS object_name,
         o.object_type,
         a.event,
         ROUND(SUM(a.wait_time + a.time_waited)/1e6,2) AS total_wait_s
  FROM   gv$active_session_history a
  JOIN   dba_objects o ON o.object_id = a.current_obj#
  WHERE  a.sample_time BETWEEN SYSTIMESTAMP - (1/24) AND SYSTIMESTAMP
    AND  a.event IS NOT NULL
  GROUP  BY o.owner, o.object_name, o.object_type, a.event
  ORDER  BY total_wait_s DESC
) FETCH FIRST &top_n ROWS ONLY;

7. Top Segments by Physical Reads

Shows which segments are responsible for the most physical I/O.

SELECT * FROM (
  SELECT owner||'.'||object_name AS segment_name,
         object_type,
         value AS total_physical_reads
  FROM   v$segment_statistics
  WHERE  statistic_name = 'physical reads'
  ORDER  BY total_physical_reads DESC
) FETCH FIRST &top_n ROWS ONLY;

8. Top SQL (ASH) in Past Hour

Finds the SQL statements with the highest total wait time in the last hour.

SELECT * FROM (
  SELECT a.sql_id,
         u.username,
         ROUND(SUM(a.wait_time + a.time_waited)/1e6,2) AS total_wait_s,
         MIN(sa.sql_text) KEEP (DENSE_RANK FIRST ORDER BY SUM(a.wait_time + a.time_waited) DESC) AS sql_text
  FROM   gv$active_session_history a
  LEFT   JOIN v$sqlarea sa ON sa.sql_id = a.sql_id
  LEFT   JOIN dba_users u  ON u.user_id = a.user_id
  WHERE  a.sample_time >= SYSTIMESTAMP - (1/24)
    AND  a.sql_id IS NOT NULL
  GROUP  BY a.sql_id, u.username
  ORDER  BY total_wait_s DESC
) FETCH FIRST &top_n ROWS ONLY;

9. SQL with Highest I/O (reads) Past Day (AWR)

Identifies SQL that spent the most time on I/O wait events in the last 24 hours.

SELECT * FROM (
  SELECT h.sql_id,
         COUNT(*) AS ash_secs
  FROM   dba_hist_active_sess_history h
  JOIN   dba_hist_snapshot x
         ON x.snap_id = h.snap_id
        AND x.dbid = h.dbid
        AND x.instance_number = h.instance_number
  WHERE  x.begin_interval_time > SYSDATE - 1
    AND  h.event IN ('db file sequential read','db file scattered read')
  GROUP  BY h.sql_id
  ORDER  BY ash_secs DESC
) FETCH FIRST &top_n ROWS ONLY;

10. Top CPU since Past Day (AWR)

Shows the top CPU consumers from the last day, along with their disk reads and execution counts.

SELECT * FROM (
  SELECT a.sql_id,
         ROUND(SUM(a.cpu_time_delta)/1e6,2) cpu_s,
         ROUND(SUM(a.disk_reads_delta),0) disk_reads,
         SUM(a.executions_delta) execs
  FROM   dba_hist_sqlstat a
  JOIN   dba_hist_snapshot s ON s.snap_id = a.snap_id
  WHERE  s.begin_interval_time > SYSDATE - 1
  GROUP  BY a.sql_id
  ORDER  BY cpu_s DESC
) FETCH FIRST &top_n ROWS ONLY;

11. Top SQL at a Reported Hour

First, find the specific snapshot ID for the hour you want to investigate.

SELECT snap_id, TO_CHAR(begin_interval_time,'YYYY-MM-DD HH24:MI') begin_time,
       TO_CHAR(end_interval_time,'YYYY-MM-DD HH24:MI') end_time
FROM   dba_hist_snapshot
WHERE  end_interval_time BETWEEN TO_DATE('&&start_ts','YYYY-MM-DD HH24:MI')
                         AND TO_DATE('&&end_ts','YYYY-MM-DD HH24:MI')
ORDER  BY snap_id;

Then, define the snapid and run the query below to see the top SQL by disk reads for that snapshot.

-- After you pick a specific &snapid:
-- SELECT * FROM (
--   SELECT sql.sql_id,
--          sql.buffer_gets_delta  buf_gets,
--          sql.disk_reads_delta   disk_reads,
--          ROUND(sql.iowait_delta/1e6,2) io_wait_s,
--          ROUND(sql.cpu_time_delta/1e6,2) cpu_s,
--          ROUND(sql.elapsed_time_delta/1e6,2) ela_s
--   FROM   dba_hist_sqlstat sql
--   WHERE  sql.snap_id = &snapid
--   ORDER  BY disk_reads DESC
-- ) FETCH FIRST &top_n ROWS ONLY;

12. Trends for One SQL_ID (last day)

Tracks the performance metrics of a specific sql_id over the last day, snapshot by snapshot.

SELECT s.snap_id,
       TO_CHAR(s.begin_interval_time,'YYYY-MM-DD HH24:MI') AS begin_time,
       sql.executions_delta   AS execs,
       sql.buffer_gets_delta  AS buffer_gets,
       sql.disk_reads_delta   AS disk_reads,
       ROUND(sql.iowait_delta/1e6,2) AS io_wait_s,
       ROUND(sql.cpu_time_delta/1e6,2) AS cpu_s,
       ROUND(sql.elapsed_time_delta/1e6,2) AS ela_s
FROM   dba_hist_sqlstat sql
JOIN   dba_hist_snapshot s ON s.snap_id = sql.snap_id
WHERE  s.begin_interval_time > SYSDATE - 1
  AND  sql.sql_id = '&&sqlid'
ORDER  BY s.begin_interval_time;

13. Plans observed for SQL_ID

Checks if a specific SQL statement is "flipping" between different execution plans, which can cause performance instability.

SELECT sql_id,
       plan_hash_value,
       SUM(executions_delta) AS executions,
       SUM(rows_processed_delta) AS rows_processed,
       TRUNC(SUM(cpu_time_delta)/1e6/60) AS cpu_mins,
       TRUNC(SUM(elapsed_time_delta)/1e6/60) AS ela_mins
FROM   dba_hist_sqlstat
WHERE  sql_id = '&&sqlid'
GROUP  BY sql_id, plan_hash_value
ORDER  BY cpu_mins DESC;

14. Top 5 SQL by ADDM Benefit (last 7 days)

Lists the SQL statements that the Automatic Database Diagnostic Monitor (ADDM) has identified as having the most potential for improvement.

SELECT * FROM (
  SELECT b.attr1 AS sql_id, MAX(a.benefit) AS benefit
  FROM   dba_advisor_recommendations a
  JOIN   dba_advisor_objects b
    ON   a.task_id = b.task_id AND a.rec_id = b.object_id
  WHERE  a.task_id IN (
          SELECT DISTINCT t.task_id
          FROM   dba_advisor_tasks t
          JOIN   dba_advisor_log   l ON l.task_id = t.task_id AND l.status='COMPLETED'
          WHERE  t.advisor_name = 'ADDM'
            AND  t.created >= SYSDATE - 7
         )
    AND  LENGTH(b.attr4) > 1
  GROUP  BY b.attr1
  ORDER  BY MAX(a.benefit) DESC
) FETCH FIRST 5 ROWS ONLY;

Patch

SELECT bug_number, creation_date FROM ad_bugs WHERE bug_number IN ('', '', '', '') ORDER BY creation_date DESC; 









 SELECT a.bug_number, e.patch_name, TRUNC(c.end_date) AS end_date, b.applied_flag FROM ad_bugs a, ad_patch_run_bugs b, ad_patch_runs c, ad_patch_drivers d, ad_applied_patches e WHERE a.bug_id = b.bug_id AND b.patch_run_id = c.patch_run_id AND c.patch_driver_id = d.patch_driver_id AND d.applied_patch_id = e.applied_patch_id AND a.bug_number IN ('', '', '', '') ORDER BY c.end_date DESC;

Your Go-To SQL Queries for Monitoring Oracle EBS Concurrent Requests

Your Go-To SQL Queries for Monitoring Oracle EBS Concurrent Requests 🧑‍💻

Need to see what’s happening in your Oracle E-Business Suite instance right now? Instead of clicking through endless forms, you can use these simple, read-only SQL queries to get a real-time snapshot of your concurrent requests. These scripts work on EBS 12.1 and 12.2 and can be run as the APPS user or any user with select grants on the FND tables.

Let's dive in!


0) Quick Reference: Phase & Status Codes

Before we start, here’s a handy key for decoding the phase_code and status_code columns you'll see in the queries.

  • phase_code: P=Pending, R=Running, C=Completed
  • status_code: R=Running, T=Terminating, X=Terminated, C=Normal, E=Error, G=Warning, D=Cancelled, W=Wait, H=On Hold

1) What is running right now?

This is the most fundamental query. It shows all currently executing concurrent requests, ordered by how long they've been running.


SELECT
  r.request_id,
  p.concurrent_program_name              AS prog_short,
  p.user_concurrent_program_name         AS program,
  u.user_name                            AS requested_by,
  r.actual_start_date,
  ROUND( (SYSDATE - r.actual_start_date)*24*60, 1 ) AS mins_running,
  r.phase_code,
  r.status_code,
  r.argument_text
FROM apps.fnd_concurrent_requests r
JOIN apps.fnd_concurrent_programs_vl p
  ON p.concurrent_program_id = r.concurrent_program_id
 AND p.application_id        = r.program_application_id
JOIN apps.fnd_user u
  ON u.user_id = r.requested_by
WHERE r.phase_code = 'R'        -- Running now
ORDER BY mins_running DESC, r.request_id;

2) Running requests with manager/node details

This query extends the previous one to show which concurrent manager, on which server node, is responsible for running the request.


SELECT
  r.request_id,
  p.user_concurrent_program_name AS program,
  q.user_concurrent_queue_name   AS manager,
  q.concurrent_queue_name        AS manager_code,
  q.target_node                  AS node,
  cp.concurrent_process_id,
  cp.os_process_id,
  r.actual_start_date,
  ROUND((SYSDATE - r.actual_start_date)*24*60, 1) AS mins_running
FROM apps.fnd_concurrent_requests r
JOIN apps.fnd_concurrent_programs_vl p
  ON p.concurrent_program_id = r.concurrent_program_id
 AND p.application_id        = r.program_application_id
LEFT JOIN apps.fnd_concurrent_processes cp
  ON cp.concurrent_process_id = r.controlling_manager
LEFT JOIN apps.fnd_concurrent_queues_vl q
  ON q.concurrent_queue_id    = cp.concurrent_queue_id
 AND q.application_id         = cp.queue_application_id
WHERE r.phase_code = 'R'
ORDER BY mins_running DESC;

3) See the Database session, waits, and SQL_ID

This is crucial for performance tuning. It links a running request directly to its database session (SID), wait events, and active SQL_ID.


SELECT
  r.request_id,
  p.user_concurrent_program_name AS program,
  s.inst_id,
  s.sid, s.serial#,
  s.username                     AS db_user,
  s.status                       AS sess_status,
  s.sql_id,
  s.event                        AS wait_event,
  s.seconds_in_wait,
  r.actual_start_date,
  ROUND((SYSDATE - r.actual_start_date)*24*60, 1) AS mins_running
FROM apps.fnd_concurrent_requests r
JOIN apps.fnd_concurrent_programs_vl p
  ON p.concurrent_program_id = r.concurrent_program_id
 AND p.application_id        = r.program_application_id
JOIN gv$session s
  ON s.audsid = r.oracle_session_id
WHERE r.phase_code = 'R'
ORDER BY mins_running DESC;

Note: If a request doesn't show up here, it might be executing code within the manager itself and not running a specific SQL statement at this exact moment.


4) Get the full SQL text for a running request

When you have the SQL_ID from the query above, you can use this to fetch the complete SQL text. Use this one sparingly as it can be a heavy query.


SELECT
  r.request_id,
  p.user_concurrent_program_name AS program,
  s.inst_id, s.sid, s.serial#, s.sql_id,
  q.sql_text
FROM apps.fnd_concurrent_requests r
JOIN apps.fnd_concurrent_programs_vl p
  ON p.concurrent_program_id = r.concurrent_program_id
 AND p.application_id        = r.program_application_id
JOIN gv$session s
  ON s.audsid = r.oracle_session_id
JOIN gv$sql q
  ON q.sql_id = s.sql_id
 AND q.inst_id = s.inst_id
WHERE r.phase_code = 'R';

5) Find concurrency hot-spots

Is a specific report being run by many users at once? This query identifies programs that have multiple instances running simultaneously.


SELECT
  p.user_concurrent_program_name AS program,
  COUNT(*) AS running_count,
  MIN(r.actual_start_date) AS oldest_start,
  MAX(r.actual_start_date) AS newest_start
FROM apps.fnd_concurrent_requests r
JOIN apps.fnd_concurrent_programs_vl p
  ON p.concurrent_program_id = r.concurrent_program_id
 AND p.application_id        = r.program_application_id
WHERE r.phase_code = 'R'
GROUP BY p.user_concurrent_program_name
HAVING COUNT(*) > 1
ORDER BY running_count DESC, oldest_start;

6) Check the current manager load

This query provides a quick summary of how many requests each concurrent manager is currently handling.


SELECT
  q.user_concurrent_queue_name AS manager,
  q.target_node                AS node,
  COUNT(*)                     AS running_requests
FROM apps.fnd_concurrent_requests r
JOIN apps.fnd_concurrent_processes cp
  ON cp.concurrent_process_id = r.controlling_manager
JOIN apps.fnd_concurrent_queues_vl q
  ON q.concurrent_queue_id    = cp.concurrent_queue_id
 AND q.application_id         = cp.queue_application_id
WHERE r.phase_code = 'R'
GROUP BY q.user_concurrent_queue_name, q.target_node
ORDER BY running_requests DESC, manager;

7) Find long-running requests

Use this script to find all jobs that have been running longer than a specific threshold (e.g., more than 15 minutes).


-- Set :mins_threshold to your desired value, e.g., 15
SELECT
  r.request_id,
  p.user_concurrent_program_name AS program,
  u.user_name                     AS requested_by,
  r.actual_start_date,
  ROUND((SYSDATE - r.actual_start_date)*24*60, 1) AS mins_running
FROM apps.fnd_concurrent_requests r
JOIN apps.fnd_concurrent_programs_vl p
  ON p.concurrent_program_id = r.concurrent_program_id
 AND p.application_id        = r.program_application_id
JOIN apps.fnd_user u
  ON u.user_id = r.requested_by
WHERE r.phase_code = 'R'
  AND (SYSDATE - r.actual_start_date) * 24 * 60 >= :mins_threshold
ORDER BY mins_running DESC;

8) See what's in the pending queue

This shows you all the requests that are waiting to run, whether they are scheduled for the future or on hold.


SELECT
  r.request_id,
  p.user_concurrent_program_name AS program,
  u.user_name                     AS requested_by,
  r.requested_start_date,
  r.phase_code,
  r.status_code,
  r.hold_flag
FROM apps.fnd_concurrent_requests r
JOIN apps.fnd_concurrent_programs_vl p
  ON p.concurrent_program_id = r.concurrent_program_id
 AND p.application_id        = r.program_application_id
JOIN apps.fnd_user u
  ON u.user_id = r.requested_by
WHERE r.phase_code = 'P'   -- Pending
ORDER BY NVL(r.requested_start_date, SYSDATE), r.request_id;

9) Get log and output file locations

Quickly find the exact log and output file names and server locations for any running request.


SELECT
  r.request_id,
  p.user_concurrent_program_name AS program,
  r.logfile_name,
  r.logfile_node_name,
  r.outfile_name,
  r.outfile_node_name
FROM apps.fnd_concurrent_requests r
JOIN apps.fnd_concurrent_programs_vl p
  ON p.concurrent_program_id = r.concurrent_program_id
 AND p.application_id        = r.program_application_id
WHERE r.phase_code = 'R'
ORDER BY r.request_id;

10) See running children of a request set

When a request set is running, use this query to see the status of all its child requests.


-- Replace :parent_request_id with the parent request ID
SELECT
  r.parent_request_id,
  r.request_id,
  p.user_concurrent_program_name AS program,
  r.phase_code,
  r.status_code,
  r.actual_start_date,
  ROUND((SYSDATE - r.actual_start_date)*24*60, 1) AS mins_running
FROM apps.fnd_concurrent_requests r
JOIN apps.fnd_concurrent_programs_vl p
  ON p.concurrent_program_id = r.concurrent_program_id
 AND p.application_id        = r.program_application_id
WHERE (r.parent_request_id = :parent_request_id OR r.request_id = :parent_request_id)
  AND r.phase_code IN ('R','P')  -- running or pending
ORDER BY r.parent_request_id, r.request_id;

11) Filter by a specific program name

Quickly find all running instances of a specific program, searching by either its short name or its user-facing display name.


-- Bind either :prog_short (e.g., 'XX_REPORT') or :prog_name
SELECT
  r.request_id,
  p.concurrent_program_name      AS prog_short,
  p.user_concurrent_program_name AS program,
  r.phase_code,
  r.status_code,
  r.actual_start_date,
  ROUND((SYSDATE - r.actual_start_date)*24*60, 1) AS mins_running
FROM apps.fnd_concurrent_requests r
JOIN apps.fnd_concurrent_programs_vl p
  ON p.concurrent_program_id = r.concurrent_program_id
 AND p.application_id        = r.program_application_id
WHERE r.phase_code = 'R'
  AND (p.concurrent_program_name = :prog_short OR p.user_concurrent_program_name = :prog_name)
ORDER BY mins_running DESC;

12) See all long-running jobs started today

This is a handy end-of-day check to see which jobs kicked off today are still running.


SELECT
  r.request_id,
  p.user_concurrent_program_name AS program,
  r.actual_start_date,
  ROUND((SYSDATE - r.actual_start_date)*24*60, 1) AS mins_running
FROM apps.fnd_concurrent_requests r
JOIN apps.fnd_concurrent_programs_vl p
  ON p.concurrent_program_id = r.concurrent_program_id
 AND p.application_id        = r.program_application_id
WHERE r.phase_code = 'R'
  AND r.actual_start_date >= TRUNC(SYSDATE)
ORDER BY mins_running DESC;

✅ Key Takeaways & Gotchas

  • The most reliable flag for an executing job is phase_code='R'.
  • The join from r.oracle_session_id to gv$session.audsid is the definitive way to link a request to its database session.
  • If you are on a RAC database, using GV$ views (like gv$session) instead of V$ is critical to see sessions on all nodes.
  • For a higher-level view, you can also query APPS.FND_CONC_REQ_SUMMARY_V, but the queries above give you the raw, detailed data for deep-dive analysis.

Saturday, August 9, 2025

Your Guide to Locking in a Good Execution Plan 🚀

Taming the Oracle Optimizer: Your Guide to Locking in a Good Execution Plan 🚀

Ever had a critical SQL query that ran perfectly fast yesterday, but is crawling today? You haven't changed the code, so what gives? The culprit is often the Oracle Optimizer changing its mind about the execution plan—the internal "road map" it uses to fetch your data.

When performance is unpredictable, you need to take control. SQL Plan Management (SPM) is Oracle's built-in feature that lets you find a "golden" execution plan and tell the database to use it every time. This guide will walk you through how to capture and enforce a good plan using SQL Plan Baselines.


## Before You Start: Quick Checks & Privileges

Before diving in, make sure your environment is ready.

  1. Confirm SPM is enabled: The optimizer_use_sql_plan_baselines parameter must be set to TRUE. Run this check:

    SQL
    SHOW PARAMETER optimizer_use_sql_plan_baselines;
    
  2. Ensure you have the right privileges: You'll need specific permissions to manage baselines and query performance data. Your DBA can grant you these:

    • ADMINISTER SQL MANAGEMENT OBJECT: Required for using the DBMS_SPM package.

    • SELECT_CATALOG_ROLE: Required for querying the Automatic Workload Repository (AWR) views like dba_hist_sqlstat.


## Step 1: Identify the SQL and the "Good" Plan

First, you need to find the specific query and the high-performing execution plan you want to stabilize. Every query has a unique SQL_ID, and each of its execution plans has a PLAN_HASH_VALUE (PHV).

  • If the good plan ran recently, you can find it in the cursor cache:

    SQL
    -- Find a recent plan in the cursor cache
    SELECT
        sql_id,
        plan_hash_value,
        parsing_schema_name,
        executions
    FROM
        gv$sqlarea
    WHERE
        sql_id = '&SQL_ID';
    
  • If the good plan is older, you'll need to look in the AWR history:

    SQL
    -- Find a historical plan in AWR
    SELECT
        snap_id,
        sql_id,
        plan_hash_value,
        elapsed_time_delta,
        executions_delta
    FROM
        dba_hist_sqlstat
    WHERE
        sql_id = '&SQL_ID'
    ORDER BY
        elapsed_time_delta DESC; -- Find the fastest executions
    

Once you have the SQL_ID and the PLAN_HASH_VALUE of your desired plan, you're ready to create a baseline.


## Step 2: Create the SQL Plan Baseline

You can load a baseline from either the live cursor cache or the historical AWR data.

### Option A: From the Cursor Cache (The Quickest Method)

Use this method if the good plan is still in memory. It's the fastest way to create a baseline.

SQL
DECLARE
  l_loaded NUMBER;
BEGIN
  l_loaded := DBMS_SPM.LOAD_PLANS_FROM_CURSOR_CACHE(
                sql_id          => '&SQL_ID',
                plan_hash_value => &PLAN_HASH_VALUE,
                enabled         => 'YES'
              );

  DBMS_OUTPUT.PUT_LINE('Baselines loaded: ' || l_loaded);
END;
/

### Option B: From AWR (When the Plan is Not in Cache)

If the plan is no longer in the cache, you can pull it from AWR. First, identify a snapshot window when the good plan was running.

SQL
-- 1. Find the begin and end snapshot IDs
SELECT
    MIN(snap_id) begin_snap,
    MAX(snap_id) end_snap
FROM
    dba_hist_snapshot
WHERE
    end_interval_time BETWEEN TO_DATE('&START_TIME', 'YYYY-MM-DD HH24:MI')
                      AND     TO_DATE('&END_TIME', 'YYYY-MM-DD HH24:MI');

-- 2. Load the baseline from the AWR snapshot window
DECLARE
  l_loaded NUMBER;
BEGIN
  l_loaded := DBMS_SPM.LOAD_PLANS_FROM_AWR(
                begin_snap      => &BEGIN_SNAP,
                end_snap        => &END_SNAP,
                sql_id          => '&SQL_ID',
                plan_hash_value => &PLAN_HASH_VALUE,
                enabled         => 'YES'
              );

  DBMS_OUTPUT.PUT_LINE('Baselines loaded: ' || l_loaded);
END;
/

## Step 3: Verify the Baseline and (Optionally) "Fix" It

After loading the plan, verify that the baseline was created. You'll need the SQL_HANDLE for future actions.

SQL
SELECT
    sql_handle,
    plan_name,
    enabled,
    accepted,
    fixed,
    created
FROM
    dba_sql_plan_baselines
WHERE
    sql_text LIKE '%<unique fragment of the SQL>%';

By default, a new baseline is ENABLED and ACCEPTED. This means the optimizer will consider it. If you want to force the optimizer to only use this plan, you can set it to FIXED.

Best Practice: Avoid fixing a plan immediately. Let it run as ENABLED and ACCEPTED first. Only fix it once you are absolutely certain this plan is the best choice under all conditions.

SQL
-- Optionally "fix" the plan to pin it
DECLARE
  l_out PLS_INTEGER;
BEGIN
  l_out := DBMS_SPM.ALTER_SQL_PLAN_BASELINE(
             sql_handle      => '&SQL_HANDLE',
             plan_name       => '&PLAN_NAME',
             attribute_name  => 'fixed',
             attribute_value => 'YES'
           );
END;
/

## Step 4: Test That the Baseline Is Used ✅

Now for the final check! Run your query again and inspect the execution plan details.

SQL
-- Show the executed plan and check the notes
SELECT * FROM TABLE(DBMS_XPLAN.DISPLAY_CURSOR(NULL, NULL, 'BASIC +NOTE'));

In the output, you should see a "Note" section confirming that your baseline was used:

Note
-----
   - SQL plan baseline "PLAN_NAME_HERE" used for this statement

You can also query gv$sql to see which baseline is attached to your session's cursor.

SQL
SELECT sql_id, sql_plan_baseline, plan_hash_value FROM gv$sql WHERE sql_id = '&SQL_ID';

## Step 5 (Optional): Migrate Baselines Between Databases

If you've identified and tested a great plan in your UAT or test environment, you can easily migrate it to production using staging tables.

  1. In the source database, create a staging table and pack the baseline into it.

    SQL
    BEGIN
      -- Create the staging table
      DBMS_SPM.CREATE_STGTAB_BASELINE(table_name => 'SPM_STAGE', schema_name => 'APPS');
    
      -- Pack the desired baseline into the table
      DBMS_SPM.PACK_STGTAB_BASELINE(
        table_name   => 'SPM_STAGE',
        schema_name  => 'APPS',
        sql_handle   => '&SQL_HANDLE'
      );
    END;
    /
    
  2. Move the SPM_STAGE table to the target database (using Data Pump or another method).

  3. In the target database, unpack the baseline from the staging table.

    SQL
    BEGIN
      DBMS_SPM.UNPACK_STGTAB_BASELINE(
        table_name   => 'SPM_STAGE',
        schema_name  => 'APPS'
      );
    END;
    /
    

## Step 6: Maintenance - How to Back Out

If a baseline is no longer needed or is causing issues, you can either disable it (keeping it for reference) or drop it completely.

  • Disable a baseline:

    SQL
    DECLARE
      n PLS_INTEGER;
    BEGIN
      n := DBMS_SPM.ALTER_SQL_PLAN_BASELINE(
             sql_handle      => '&SQL_HANDLE',
             plan_name       => '&PLAN_NAME',
             attribute_name  => 'enabled',
             attribute_value => 'NO'
           );
    END;
    /
    
  • Drop a baseline permanently:

    SQL
    DECLARE
      n PLS_INTEGER;
    BEGIN
      n := DBMS_SPM.DROP_SQL_PLAN_BASELINE(
             sql_handle => '&SQL_HANDLE',
             plan_name  => '&PLAN_NAME'
           );
    END;
    /
    

## An Alternative: When to Use a SQL Profile

Sometimes, an application generates SQL with varying literals (e.g., WHERE id = 101 vs. WHERE id = 205). SPM requires an exact text match, so it may not work here. In these cases, a SQL Profile is a better choice.

A SQL Profile doesn't lock a plan; instead, it attaches a set of hints to a query to "nudge" the optimizer toward the right plan shape.

  1. Get the Outline Hints for the good plan:

    SQL
    SELECT * FROM TABLE(DBMS_XPLAN.DISPLAY_CURSOR('&SQL_ID', NULL, 'ADVANCED'));
    -- Copy the hints from the "Outline Data" section
    
  2. Import a SQL Profile with those hints:

    SQL
    DECLARE
      h SYS.SQLPROF_ATTR;
    BEGIN
      h := SYS.SQLPROF_ATTR(
        'USE_HASH_AGGREGATION(@SEL$1)',
        'LEADING(@"SEL$1" "T1"@"SEL$1" "T2"@"SEL$1")',
        'INDEX_RS_ASC("T1"@"SEL$1" "T1_IDX")'
        -- Paste all other outline hints here
      );
    
      DBMS_SQLTUNE.IMPORT_SQL_PROFILE(
        sql_text     => q'[ PASTE THE EXACT SQL TEXT HERE ]',
        profile      => h,
        name         => 'PROF_FIX_MY_QUERY',
        force_match  => TRUE, -- Set TRUE to match queries with different literals
        replace      => TRUE
      );
    END;
    /
    

While flexible, SQL Profiles offer less deterministic control than a fixed SPM baseline. Prefer SPM for strict plan stability.


## Practical Tips & Common Gotchas

  • Exact Text Match: SPM is picky about SQL text. If your application uses literals instead of bind variables, consider setting CURSOR_SHARING to FORCE or using a SQL Profile with force_match => TRUE.

  • Statistics Drift: A baseline forces a plan's shape (join order, access methods), but the optimizer's row count estimates can still change if statistics become stale. Keep your stats fresh!

  • Bind Peeking: A plan that's great for one set of bind variables might be terrible for another. If a single fixed plan isn't safe, avoid fixing it and explore adaptive features.

  • Troubleshooting: If your baseline isn't being used, double-check that optimizer_use_sql_plan_baselines is TRUE, the SQL text matches perfectly, and you don't have conflicting baselines.