Monday, August 4, 2025

Mastering Oracle EBS Concurrent Requests: Your Ultimate SQL Query Toolkit

 

Mastering Oracle EBS Concurrent Requests: Your Ultimate SQL Query Toolkit

As an Oracle E-Business Suite administrator or developer, managing concurrent requests is a fundamental part of your daily routine. From troubleshooting long-running programs to auditing user activity, having a powerful set of SQL queries can save you a significant amount of time and effort.

This post provides a comprehensive toolkit of essential SQL queries to help you monitor, troubleshoot, and analyze concurrent requests and other critical system data in your Oracle EBS environment.

1. Finding Concurrent Request Information

The queries below are your go-to tools for getting basic information about concurrent requests. You can use these to find out which manager ran a specific request, what program is associated with a request, or to see a summary of a request's status.

How to Determine Which Manager Ran a Specific Concurrent Request

This query helps you pinpoint the concurrent manager responsible for a particular request, which is useful for troubleshooting queue-related issues.

SQL
col USER_CONCURRENT_QUEUE_NAME for a100
select b.USER_CONCURRENT_QUEUE_NAME
from fnd_concurrent_processes a,
     fnd_concurrent_queues_vl b,
     fnd_concurrent_requests c
where a.CONCURRENT_QUEUE_ID = b.CONCURRENT_QUEUE_ID
and a.CONCURRENT_PROCESS_ID = c.controlling_manager
and c.request_id =&conc_reqid’;

To Find Concurrent Program Name, Phase, and Status for a Given Request ID

This is a quick way to get the essential details of a completed or running request, including the program name and its current phase and status.

SQL
SELECT request_id,
       user_concurrent_program_name,
       DECODE(phase_code, 'C', 'Completed', phase_code) phase_code,
       DECODE(status_code, 'D', 'Cancelled',
              'E', 'Error',
              'G', 'Warning',
              'H', 'On Hold',
              'T', 'Terminating',
              'M', 'No Manager',
              'X', 'Terminated',
              'C', 'Normal', status_code) status_code,
       to_char(actual_start_date, 'dd-mon-yy:hh24:mi:ss') Start_Date,
       to_char(actual_completion_date, 'dd-mon-yy:hh24:mi:ss'),
       completion_text
FROM apps.fnd_conc_req_summary_v
WHERE request_id = '&req_id'
ORDER BY 6 DESC;

To Find Child Requests

If a program spawns other requests (a parent-child relationship), this query helps you find all the requests submitted by a parent.

SQL
set lines 200
col USER_CONCURRENT_PROGRAM_NAME for a40
col PHASE_CODE for a10
col STATUS_CODE for a10
col COMPLETION_TEXT for a20
SELECT sum.request_id,
       req.PARENT_REQUEST_ID,
       sum.user_concurrent_program_name,
       DECODE (sum.phase_code, 'C', 'Completed', sum.phase_code) phase_code,
       DECODE(sum.status_code, 'D', 'Cancelled' ,
              'E', 'Error' ,
              'G', 'Warning',
              'H', 'On Hold' ,
              'T', 'Terminating',
              'M', 'No Manager' ,
              'X', 'Terminated',
              'C', 'Normal', sum.status_code) status_code,
       sum.actual_start_date,
       sum.actual_completion_date,
       sum.completion_text
FROM apps.fnd_conc_req_summary_v sum,
     apps.fnd_concurrent_requests req
where req.request_id=sum.request_id
and req.PARENT_REQUEST_ID = '&parent_concurrent_request_id';

2. Troubleshooting Long-Running and Errored Requests

These queries are essential for performance tuning and troubleshooting. They help you identify sessions that are taking too long, find the underlying SQL, and pinpoint requests that have failed.

Long-Running Concurrent Requests

This is a powerful query that provides a comprehensive overview of currently running requests, including session information (SPID), and their runtime duration.

SQL
SELECT a.request_id,
       a.oracle_process_id "SPID",
       frt.responsibility_name,
       c.concurrent_program_name || ': ' || ctl.user_concurrent_program_name,
       a.description,
       a.ARGUMENT_TEXT,
       b.node_name,
       b.db_instance,
       a.logfile_name,
       a.logfile_node_name,
       a.outfile_name,
       q.concurrent_queue_name,
       a.phase_code,
       a.status_code,
       a.completion_text,
       actual_start_date,
       actual_completion_date,
       fu.user_name,
       (nvl(actual_completion_date,sysdate)-actual_start_date)*1440 mins,
       (SELECT avg(nvl(a2.actual_completion_date-a2.actual_start_date,0))*1440 avg_run_time
        FROM APPLSYS.fnd_Concurrent_requests a2,
             APPLSYS.fnd_concurrent_programs c2
        WHERE c2.concurrent_program_id = c.concurrent_program_id
        AND a2.concurrent_program_id = c2.concurrent_program_id
        AND a2.program_application_id = c2.application_id
        AND a2.phase_code || '' = 'C') avg_mins,
       round((actual_completion_date - requested_start_date),2) * 24 duration_in_hours
FROM APPLSYS.fnd_Concurrent_requests a,
     APPLSYS.fnd_concurrent_processes b,
     applsys.fnd_concurrent_queues q,
     APPLSYS.fnd_concurrent_programs c,
     APPLSYS.fnd_concurrent_programs_tl ctl,
     apps.fnd_user fu,
     apps.FND_RESPONSIBILITY_TL frt
WHERE a.controlling_manager = b.concurrent_process_id
AND a.concurrent_program_id = c.concurrent_program_id
AND a.program_application_id = c.application_id
AND a.phase_code = 'R'
AND a.status_code = 'R'
AND b.queue_application_id = q.application_id
AND b.concurrent_queue_id = q.concurrent_queue_id
AND ctl.concurrent_program_id = c.concurrent_program_id
AND a.requested_by = fu.user_id
AND a.responsibility_id = frt.responsibility_id
ORDER BY a.actual_start_date DESC;

To Find the SQL Query for a Given Concurrent Request SID

Once you have the Session ID (SID) from the previous query, you can use this to find the specific SQL statement the request is currently executing.

SQL
select sid,sql_text
from gv$session ses, gv$sqlarea sql
where ses.sql_hash_value = sql.hash_value(+)
and ses.sql_address = sql.address(+)
and ses.sid='&oracle_sid'
/

Oracle Concurrent Request Error Script

This query quickly lists all concurrent requests that have errored out in the last two days, helping you identify and resolve issues.

SQL
SELECT a.request_id "Req Id",
       a.phase_code,
       a.status_code,
       actual_start_date,
       actual_completion_date,
       c.concurrent_program_name || ': ' || ctl.user_concurrent_program_name "program"
FROM APPLSYS.fnd_Concurrent_requests a,
     APPLSYS.fnd_concurrent_processes b,
     applsys.fnd_concurrent_queues q,
     APPLSYS.fnd_concurrent_programs c,
     APPLSYS.fnd_concurrent_programs_tl ctl
WHERE a.controlling_manager = b.concurrent_process_id
AND a.concurrent_program_id = c.concurrent_program_id
AND a.program_application_id = c.application_id
AND a.status_code = 'E'
AND a.phase_code = 'C'
AND actual_start_date > sysdate - 2
AND b.queue_application_id = q.application_id
AND b.concurrent_queue_id = q.concurrent_queue_id
AND ctl.concurrent_program_id = c.concurrent_program_id
AND ctl.LANGUAGE = 'US'
ORDER BY 5 DESC;

3. Monitoring and Performance Tuning

These queries help you monitor the overall health of your concurrent processing system by checking manager status, pending requests, and average run times.

Concurrent Manager Status for a Given SID

Use this to get detailed session information for a specific concurrent manager session.

SQL
col MODULE for a20
col OSUSER for a10
col USERNAME for a10
set num 10
col MACHINE for a20
set lines 200
col SCHEMANAME for a10
select s.sid,
       s.serial#,
       p.spid os_pid,
       s.status,
       s.osuser,
       s.username,
       s.MACHINE,
       s.MODULE,
       s.SCHEMANAME,
       s.action
from gv$session s, gv$process p
WHERE s.paddr = p.addr
and s.sid = '&oracle_sid';

Checking Which Manager Will Execute a Program

This query helps you understand the concurrent manager specialization rules by showing which manager is set to run a specific program.

SQL
SELECT user_concurrent_program_name,
       user_concurrent_queue_name
FROM apps.fnd_concurrent_programs_tl cp,
     apps.fnd_concurrent_queue_content cqc,
     apps.fnd_concurrent_queues_tl cq
WHERE cqc.type_application_id(+) = cp.application_id
AND cqc.type_id(+) = cp.concurrent_program_id
AND cqc.type_code(+) = 'P'
AND cqc.include_flag(+) = 'I'
AND cp.LANGUAGE = 'US'
AND cp.user_concurrent_program_name = '&USER_CONCURRENT_PROGRAM_NAME'
AND NVL (cqc.concurrent_queue_id, 0) = cq.concurrent_queue_id
AND NVL (cqc.queue_application_id, 0) = cq.application_id
AND cq.LANGUAGE = 'US'

To See All Pending / Running Requests per Manager

This gives you a quick snapshot of the workload for each concurrent manager.

SQL
SELECT request_id,
       phase_code,
       status_code,
       user_name,
       user_concurrent_queue_name
FROM apps.fnd_concurrent_worker_requests cwr,
     apps.fnd_concurrent_queues_tl cq,
     apps.fnd_user fu
WHERE (cwr.phase_code = 'P' OR cwr.phase_code = 'R')
AND cwr.hold_flag != 'Y'
AND cwr.requested_start_date <= SYSDATE
AND cwr.concurrent_queue_id = cq.concurrent_queue_id
AND cwr.queue_application_id = cq.application_id
AND cq.LANGUAGE = 'US'
AND cwr.requested_by = fu.user_id
ORDER BY 5

4. Administering Oracle EBS

These queries provide a broader view of your Oracle EBS environment, from user management to applied patches.

To Find Out If a Module is Installed

This query lists all installed Oracle EBS products along with their patch levels.

SQL
SELECT a.application_name,
       a.product_code,
       DECODE (b.status, 'I', 'Installed', 'S', 'Shared', 'N/A') status,
       patch_level
FROM apps.fnd_application_vl a,
     apps.fnd_product_installations b
WHERE a.application_id = b.application_id
and b.status='I'
order by product_code asc;

Script to Reset Oracle Apps Front-End User Password

This is an essential administrative script for resetting a user's password from the backend using PL/SQL.

SQL
DECLARE
  flag_value BOOLEAN;
BEGIN
  flag_value := fnd_user_pkg.changepassword(username=> 'NKUMAR2', newpassword => 'welcome1');
  IF flag_value THEN
    DBMS_OUTPUT.PUT_LINE('The password reset successfully');
  ELSE
    DBMS_OUTPUT.PUT_LINE('The password reset has failed');
  END IF;
END;
/
COMMIT;

How to Check the System Administrator Responsibility for Users

This query quickly identifies which users have been assigned the "System Administrator" responsibility and for what time frame.

SQL
set lines 2000
set pages 5000
col USER_NAME for a45
Select b.user_name,
       c.responsibility_name,
       a.START_DATE,
       a.END_DATE
from fnd_user_resp_groups_direct a,
     fnd_user b,
     fnd_responsibility_tl c
where a.user_id = b.user_id
and a.responsibility_id = c.responsibility_id
and c.responsibility_name='System Administrator';

This collection of queries should serve as a powerful resource for any Oracle EBS professional. Bookmark this page and refer to it whenever you need to dive into the details of your concurrent processing environment.

Unmasking Performance Bottlenecks: A DBA's Guide to Finding Slow Concurrent Programs

 

Unmasking Performance Bottlenecks: A DBA's Guide to Finding Slow Concurrent Programs

As an Oracle Apps DBA, one of the most critical tasks is ensuring the smooth and efficient operation of concurrent programs. A single long-running or poorly performing program can bring the entire system to a halt.

For proactive performance analysis, it's essential to regularly identify which concurrent programs are consuming the most time. This allows you to focus your tuning efforts where they will have the greatest impact.

Here is a powerful and straightforward SQL query that I use regularly to find the top 10 longest-running completed concurrent programs in Oracle E-Business Suite.

The Query

-- This query identifies the top 10 longest-running completed concurrent programs.
-- It joins two key tables to get the request details and program names.
select
    fcpt.user_concurrent_program_name as "Program Name",
    fcr.request_id as "Request ID",
    fcr.actual_start_date as "Start Time",
    fcr.actual_completion_date as "End Time",
    -- Calculate the run time in minutes
    round((fcr.actual_completion_date - fcr.actual_start_date) * 24 * 60, 2) as "Run Time (Minutes)"
from
    apps.fnd_concurrent_requests fcr,
    apps.fnd_concurrent_programs_tl fcpt
where
    -- Join condition to link requests to program names
    fcr.concurrent_program_id = fcpt.concurrent_program_id
    and fcr.program_application_id = fcpt.application_id
    and fcpt.language = userenv('LANG')
    -- Filter for completed requests with a start and end time
    and fcr.phase_code = 'C' -- 'C' for Complete
    and fcr.actual_start_date is not null
    and fcr.actual_completion_date is not null
order by
    "Run Time (Minutes)" desc
fetch first 10 rows only; -- Fetch only the top 10 longest-running programs

How It Works: An Expert's Breakdown

Let's dissect this query to understand its power:

  • Data Source: The query leverages two essential tables:

    • fnd_concurrent_requests: This is the master table for all concurrent requests. It contains the raw data, including the request_id, program_id, and, most importantly, the actual_start_date and actual_completion_date timestamps.

    • fnd_concurrent_programs_tl: This table provides the user-facing name of the program (user_concurrent_program_name), making the output immediately understandable without needing to cross-reference program IDs.

  • The Crucial Calculation: The heart of this query is the round((fcr.actual_completion_date - fcr.actual_start_date) * 24 * 60, 2) calculation. In Oracle, subtracting two DATE data types results in a value representing the number of days between them. By multiplying this by 24 (hours per day) and then 60 (minutes per hour), we get the total runtime in minutes.

  • Targeted Filtering: The WHERE clause is key to getting meaningful results.

    • fcr.phase_code = 'C': We only look at requests that have successfully completed.

    • fcr.actual_start_date is not null and fcr.actual_completion_date is not null: This ensures we're only calculating the runtime for programs that had a complete lifecycle recorded.

  • Ranking the Results: order by "Run Time (Minutes)" desc sorts the output to show the most time-consuming programs first. fetch first 10 rows only is an efficient, modern Oracle feature that limits the result set to the top 10, preventing an unwieldy output in a busy system.

Final Thoughts

This simple, yet effective, query is an invaluable tool for any Oracle Apps DBA. It helps to quickly pinpoint performance hogs, allowing you to prioritize your performance tuning efforts and keep your Oracle EBS environment running smoothly. Regularly running this query can be a game-changer for maintaining a healthy system.

How to Identify and Manage the Largest Objects in Your Oracle EBS Database

 

Title: How to Identify and Manage the Largest Objects in Your Oracle EBS Database

Introduction

Database size can be a critical factor for performance and stability, especially in a complex and high-volume environment like Oracle E-Business Suite (EBS). A growing database can lead to slower queries, longer backup times, and increased storage costs. One of the most effective ways to manage database growth is to identify which tables and objects are consuming the most space.

In this blog post, we'll walk you through how to use simple yet powerful SQL queries to find the largest objects in your Oracle EBS database. We'll start with a basic query and then move on to a more comprehensive one that provides a detailed breakdown of space usage, which is essential for effective database management.


Step 1: The Quick and Simple Approach - Find the Top 10 Largest Objects

If you just need a quick overview of the biggest segments in your entire database, this query is your best friend. It lists the top 10 largest objects (tables, indexes, LOBs, etc.) in terms of raw size.

SQL
SELECT *
FROM (
  SELECT
    SEGMENT_NAME,
    SEGMENT_TYPE,
    BYTES / 1024 / 1024 / 1024 AS GB,
    TABLESPACE_NAME
  FROM DBA_SEGMENTS
  ORDER BY BYTES DESC
)
WHERE ROWNUM <= 10;

What this query does:

  • It queries the DBA_SEGMENTS view, which contains information about all segments in the database.

  • It sorts all segments by size (BYTES) in descending order.

  • The ROWNUM <= 10 clause limits the output to just the top 10 rows.

  • The BYTES column is converted to gigabytes (GB) for easy reading.

This is a great starting point, but it doesn't tell you the whole story, especially in an EBS database where a single table might have multiple indexes and LOB segments.


Step 2: The Detailed Approach - A Comprehensive Breakdown for EBS

In an EBS environment, it's more useful to know which tables (including all their associated indexes and LOBs) are taking up the most space. This next query provides a more comprehensive view by grouping all related segments under their parent table.

SQL
WITH tab_size AS (
  SELECT
    owner,
    table_name,
    SUM(total_bytes) total_bytes,
    SUM(tab_bytes) tab_bytes,
    SUM(ind_bytes) ind_bytes,
    SUM(lob_bytes) lob_bytes,
    SUM(lobind_bytes) lobind_bytes
  FROM (
    -- Table segments
    SELECT
      owner,
      segment_name table_name,
      bytes total_bytes,
      bytes tab_bytes,
      0 ind_bytes,
      0 lob_bytes,
      0 lobind_bytes
    FROM dba_segments
    WHERE segment_type IN ('TABLE', 'TABLE PARTITION', 'TABLE SUBPARTITION')
    UNION ALL
    -- Index segments
    SELECT
      i.table_owner owner,
      i.table_name,
      s.bytes total_bytes,
      0 tab_bytes,
      s.bytes ind_bytes,
      0 lob_bytes,
      0 lobind_bytes
    FROM dba_indexes i, dba_segments s
    WHERE s.segment_name = i.index_name
    AND s.owner = i.owner
    AND s.segment_type IN ('INDEX', 'INDEX PARTITION', 'INDEX SUBPARTITION')
    UNION ALL
    -- LOB segments
    SELECT
      l.owner,
      l.table_name,
      s.bytes total_bytes,
      0 tab_bytes,
      0 ind_bytes,
      s.bytes lob_bytes,
      0 lobind_bytes
    FROM dba_lobs l, dba_segments s
    WHERE s.segment_name = l.segment_name
    AND s.owner = l.owner
    AND s.segment_type IN ('LOBSEGMENT', 'LOB PARTITION')
    UNION ALL
    -- LOB index segments
    SELECT
      l.owner,
      l.table_name,
      s.bytes total_bytes,
      0 tab_bytes,
      0 ind_bytes,
      0 lob_bytes,
      s.bytes lobind_bytes
    FROM dba_lobs l, dba_segments s
    WHERE s.segment_name = l.index_name
    AND s.owner = l.owner
    AND s.segment_type = 'LOBINDEX'
  )
  GROUP BY owner, table_name
)
SELECT
  t.owner,
  t.table_name,
  ROUND(t.total_bytes / (1024 * 1024), 2) AS Total_MB,
  ROUND(t.tab_bytes / (1024 * 1024), 2) AS Table_MB,
  ROUND(t.ind_bytes / (1024 * 1024), 2) AS Indexes_MB,
  ROUND(t.lob_bytes / (1024 * 1024), 2) AS LOB_MB,
  ROUND(t.lobind_bytes / (1024 * 1024), 2) AS LOBIndexes_MB
FROM tab_size t
ORDER BY t.total_bytes DESC;

SQL

WITH tab_size AS (
  SELECT
    owner,
    table_name,
    SUM(total_bytes) total_bytes,
    SUM(tab_bytes) tab_bytes,
    SUM(ind_bytes) ind_bytes,
    SUM(lob_bytes) lob_bytes,
    SUM(lobind_bytes) lobind_bytes
  FROM (
    -- Table segments
    SELECT
      owner,
      segment_name table_name,
      bytes total_bytes,
      bytes tab_bytes,
      0 ind_bytes,
      0 lob_bytes,
      0 lobind_bytes
    FROM dba_segments
    WHERE segment_type IN ('TABLE', 'TABLE PARTITION', 'TABLE SUBPARTITION')
    UNION ALL
    -- Index segments
    SELECT
      i.table_owner owner,
      i.table_name,
      s.bytes total_bytes,
      0 tab_bytes,
      s.bytes ind_bytes,
      0 lob_bytes,
      0 lobind_bytes
    FROM dba_indexes i, dba_segments s
    WHERE s.segment_name = i.index_name
    AND s.owner = i.owner
    AND s.segment_type IN ('INDEX', 'INDEX PARTITION', 'INDEX SUBPARTITION')
    UNION ALL
    -- LOB segments
    SELECT
      l.owner,
      l.table_name,
      s.bytes total_bytes,
      0 tab_bytes,
      0 ind_bytes,
      s.bytes lob_bytes,
      0 lobind_bytes
    FROM dba_lobs l, dba_segments s
    WHERE s.segment_name = l.segment_name
    AND s.owner = l.owner
    AND s.segment_type IN ('LOBSEGMENT', 'LOB PARTITION')
    UNION ALL
    -- LOB index segments
    SELECT
      l.owner,
      l.table_name,
      s.bytes total_bytes,
      0 tab_bytes,
      0 ind_bytes,
      0 lob_bytes,
      s.bytes lobind_bytes
    FROM dba_lobs l, dba_segments s
    WHERE s.segment_name = l.index_name
    AND s.owner = l.owner
    AND s.segment_type = 'LOBINDEX'
  )
  GROUP BY owner, table_name
)
SELECT
  t.owner,
  t.table_name,
  ROUND(t.total_bytes / (1024 * 1024 * 1024), 2) AS Total_GB,
  ROUND(t.tab_bytes / (1024 * 1024 * 1024), 2) AS Table_GB,
  ROUND(t.ind_bytes / (1024 * 1024 * 1024), 2) AS Indexes_GB,
  ROUND(t.lob_bytes / (1024 * 1024 * 1024), 2) AS LOB_GB,
  ROUND(t.lobind_bytes / (1024 * 1024 * 1024), 2) AS LOBIndexes_GB
FROM tab_size t
ORDER BY t.total_bytes DESC;

Why this query is so powerful for EBS:

  • It aggregates the size of a table and all its related indexes and LOB segments into a single, easy-to-read row.

  • You can see the breakdown of space usage (e.g., "This table is large because of its indexes, not the table data itself").

  • This helps you identify which schemas (e.g., APPLSYS, AP, GL) are growing the most.

Note: To run these queries, you must have the necessary privileges, typically as a DBA or a user with permissions to access the DBA_ views.


Conclusion and Next Steps

By regularly running these queries, you can stay on top of your Oracle EBS database growth. Pinpointing the largest objects is the first step toward effective database management. Once you've identified the culprits, you can investigate potential actions like:

  • Archiving old data: Many large tables contain historical data that can be moved to an archive tablespace or a separate database.

  • Purging data: Some tables, especially log or temporary tables, can have their data purged after a certain period.

  • Analyzing indexes: For large indexes, you might need to analyze them for fragmentation and consider rebuilding or reorganizing them.

  • Compressing tables/indexes: In some cases, Oracle's compression features can significantly reduce space usage.

Proactive monitoring and management of your database's largest objects will ensure your Oracle EBS system remains performant and stable for the long term.

long

 


set markup html on

set heading off

set linesize 1000

set pagesize 50000


-- Spool the output to an HTML file

spool long_running_requests.html


-- Add a title to the HTML file

SELECT '<h2>Monitoring Long-Running Concurrent Requests</h2>' FROM DUAL;


-- Add the HTML table header

SELECT

    '<table>' ||

    '<tr>' ||

    '<th>Concurrent Program</th>' ||

    '<th>Run Time (Minutes)</th>' ||

    '<th>Request ID</th>' ||

    '<th>Parent ID</th>' ||

    '<th>Request Date</th>' ||

    '<th>Start Date</th>' ||

    '<th>Completion Date</th>' ||

    '<th>End-to-End Mins</th>' ||

    '<th>Lag Time (Seconds)</th>' ||

    '<th>Requested By</th>' ||

    '<th>Phase</th>' ||

    '<th>Status</th>' ||

    '<th>Arguments</th>' ||

    '<th>Priority</th>' ||

    '</tr>' AS HTML_HEADER

FROM DUAL;


-- The actual query to generate the table rows

SELECT

    '<tr>' ||

    '<td>' || c.user_concurrent_program_name || '</td>' ||

    '<td>' || round(((sysdate-a.actual_start_date)*24*60),2) || '</td>' ||

    '<td>' || a.request_id || '</td>' ||

    '<td>' || NVL(TO_CHAR(a.parent_request_id), 'N/A') || '</td>' ||

    '<td>' || TO_CHAR(a.request_date, 'DD-MON-YYYY HH24:MI:SS') || '</td>' ||

    '<td>' || TO_CHAR(a.actual_start_date, 'DD-MON-YYYY HH24:MI:SS') || '</td>' ||

    '<td>' || TO_CHAR(a.actual_completion_date, 'DD-MON-YYYY HH24:MI:SS') || '</td>' ||

    '<td>' || round((a.actual_completion_date-a.request_date)*24*60, 2) || '</td>' ||

    '<td>' || round((a.actual_start_date-a.request_date)*24*60*60, 2) || '</td>' ||

    '<td>' || d.user_name || '</td>' ||

    '<td>' || a.phase_code || '</td>' ||

    '<td>' || a.status_code || '</td>' ||

    '<td>' || a.argument_text || '</td>' ||

    '<td>' || a.priority || '</td>' ||

    '</tr>'

FROM

    apps.fnd_concurrent_requests a,

    apps.fnd_concurrent_programs b,

    apps.FND_CONCURRENT_PROGRAMS_TL c,

    apps.fnd_user d

WHERE

    a.concurrent_program_id = b.concurrent_program_id

    AND b.concurrent_program_id = c.concurrent_program_id

    AND a.requested_by = d.user_id

    AND a.status_code = 'R'

ORDER BY

    run_time_minutes DESC;


-- Close the table and stop spooling

SELECT '</table>' FROM DUAL;


spool off

exit


#!/bin/bash


# Database connection string

DB_CONN="username/password@(DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=your_host)(PORT=your_port))(CONNECT_DATA=(SERVICE_NAME=your_service_name)))"


# The email recipient and subject

EMAIL_TO="recipient@example.com"

EMAIL_SUBJECT="Monitoring: Long-Running Concurrent Requests Report"


# Run the SQL script to generate the HTML file

sqlplus -s $DB_CONN @long_running_requests.sql


# Send the email with the HTML file as the body

cat long_running_requests.html | mailx -s "$EMAIL_SUBJECT" -a "Content-Type: text/html" "$EMAIL_TO"


# Clean up the generated file

rm long_running_requests.html


echo "Monitoring report sent successfully."






Saturday, May 17, 2025

Root Cause Analysis for High Server Load on Oracle 19c Gold Tier Database



## Introduction


This document provides a comprehensive guide for performing root cause analysis (RCA) on a Solaris-based server running Oracle 19c Gold Tier databases experiencing high server load. The environment includes 22 integrations, which may contribute to the performance issues. This guide includes SQL queries, Solaris commands, a step-by-step RCA process, and actionable recommendations to resolve the identified issues.


## 1. System-Level Diagnostics (Solaris Commands)


### 1.1 CPU Utilization


```bash

# Overall CPU utilization

mpstat 5 5

prstat -a -c -n 20 -s cpu 5 5


# Process-specific CPU usage (Oracle processes)

prstat -a -c -n 20 -s cpu -P 'oracle'


# Thread-level analysis

prstat -amLc -n 20 -s cpu 5 5


# Historical CPU data

sar -u 5 5

```


### 1.2 Memory Usage


```bash

# Overall memory statistics

vmstat 5 5

echo "::memstat" | mdb -k


# Process memory usage

prstat -a -c -n 20 -s size 5 5

pmap -x `pgrep -f ora_pmon`


# Swap usage

swap -s

swap -l

```


### 1.3 Disk I/O Performance


```bash

# Overall I/O statistics

iostat -xnz 5 5


# I/O by process

iotop -P 'oracle' 5 5


# File system usage

df -h

zpool status

zpool iostat -v 5 5


# I/O wait time

iostat -xnzc 5 5

```


### 1.4 Network Activity


```bash

# Network interface statistics

netstat -i 5

nicstat 5 5


# Connection statistics

netstat -an | grep ESTABLISHED | wc -l

netstat -an | grep "\.1521" | wc -l


# Network throughput

nicstat -i all 5 5

```


### 1.5 Process Activity


```bash

# Process listing

ps -ef | grep oracle

ps -o pid,pcpu,pmem,vsz,rss,time,args -p `pgrep -f ora_`


# Process tree

ptree `pgrep -f ora_pmon`


# Process resource usage over time

prstat -a -c -n 20 5 5

```


### 1.6 System Logs


```bash

# Check system logs for errors

tail -100 /var/adm/messages

dmesg | tail -100


# Check Oracle alert logs

tail -100 $ORACLE_BASE/diag/rdbms/$ORACLE_SID/$ORACLE_SID/trace/alert_$ORACLE_SID.log

```


## 2. Database-Level Diagnostics (Oracle SQL Queries)


### 2.1 Overall Database Health


```sql

-- Instance status and resource usage

SELECT instance_name, status, database_status, active_state, blocked

FROM v$instance;


-- Database resource usage summary

SELECT * FROM v$resource_limit

WHERE resource_name IN ('processes', 'sessions', 'transactions')

ORDER BY resource_name;


-- Database time model

SELECT stat_name, value, ROUND(value/1000000,2) as seconds

FROM v$sys_time_model

WHERE stat_name IN ('DB time', 'DB CPU', 'background cpu time', 'background elapsed time')

ORDER BY value DESC;

```


### 2.2 High-Load SQL Statements


```sql

-- Top SQL by CPU usage

SELECT sql_id, plan_hash_value, executions, 

       ROUND(elapsed_time/1000000,2) as elapsed_sec,

       ROUND(cpu_time/1000000,2) as cpu_sec,

       ROUND(buffer_gets/DECODE(executions,0,1,executions),2) as buffer_gets_per_exec,

       ROUND(disk_reads/DECODE(executions,0,1,executions),2) as disk_reads_per_exec,

       ROUND(rows_processed/DECODE(executions,0,1,executions),2) as rows_per_exec,

       parsing_schema_name

FROM v$sql

WHERE executions > 0

ORDER BY cpu_time DESC

FETCH FIRST 20 ROWS ONLY;


-- Top SQL by elapsed time

SELECT sql_id, plan_hash_value, executions, 

       ROUND(elapsed_time/1000000,2) as elapsed_sec,

       ROUND(cpu_time/1000000,2) as cpu_sec,

       ROUND(buffer_gets/DECODE(executions,0,1,executions),2) as buffer_gets_per_exec,

       ROUND(disk_reads/DECODE(executions,0,1,executions),2) as disk_reads_per_exec,

       ROUND(rows_processed/DECODE(executions,0,1,executions),2) as rows_per_exec,

       parsing_schema_name

FROM v$sql

WHERE executions > 0

ORDER BY elapsed_time DESC

FETCH FIRST 20 ROWS ONLY;


-- SQL text for high-load statements

SELECT sql_id, sql_text

FROM v$sqlarea

WHERE sql_id IN (

    SELECT sql_id FROM (

        SELECT sql_id FROM v$sql

        ORDER BY cpu_time DESC

        FETCH FIRST 20 ROWS ONLY

    )

);


-- SQL execution plans for high-load statements

SELECT * FROM table(DBMS_XPLAN.DISPLAY_CURSOR('&sql_id', NULL, 'ALLSTATS LAST'));

```


### 2.3 Active Session History (ASH) Analysis


```sql

-- Top wait events from ASH

SELECT event, COUNT(*) as wait_count

FROM v$active_session_history

WHERE sample_time > SYSDATE - INTERVAL '1' HOUR

GROUP BY event

ORDER BY wait_count DESC

FETCH FIRST 20 ROWS ONLY;


-- Top SQL from ASH

SELECT sql_id, COUNT(*) as activity_count

FROM v$active_session_history

WHERE sample_time > SYSDATE - INTERVAL '1' HOUR

  AND sql_id IS NOT NULL

GROUP BY sql_id

ORDER BY activity_count DESC

FETCH FIRST 20 ROWS ONLY;


-- Top sessions from ASH

SELECT session_id, session_serial#, user_id, COUNT(*) as activity_count

FROM v$active_session_history

WHERE sample_time > SYSDATE - INTERVAL '1' HOUR

GROUP BY session_id, session_serial#, user_id

ORDER BY activity_count DESC

FETCH FIRST 20 ROWS ONLY;


-- ASH by wait class

SELECT wait_class, COUNT(*) as wait_count

FROM v$active_session_history

WHERE sample_time > SYSDATE - INTERVAL '1' HOUR

GROUP BY wait_class

ORDER BY wait_count DESC;

```


### 2.4 Wait Events Analysis


```sql

-- System-wide wait events

SELECT event, total_waits, total_timeouts, 

       ROUND(time_waited/100,2) as time_waited_sec,

       ROUND(average_wait/100,2) as average_wait_ms

FROM v$system_event

WHERE wait_class != 'Idle'

ORDER BY time_waited DESC

FETCH FIRST 20 ROWS ONLY;


-- Session wait events

SELECT s.sid, s.serial#, s.username, s.status, s.machine, s.program,

       sw.event, sw.wait_class, sw.state, 

       ROUND(sw.seconds_in_wait,2) as wait_sec

FROM v$session s, v$session_wait sw

WHERE s.sid = sw.sid

  AND sw.wait_class != 'Idle'

  AND s.status = 'ACTIVE'

ORDER BY sw.seconds_in_wait DESC;


-- Historical wait events (AWR)

SELECT event_name, wait_class, 

       ROUND(time_waited_micro/1000000,2) as time_waited_sec

FROM dba_hist_system_event

WHERE snap_id BETWEEN &start_snap_id AND &end_snap_id

  AND wait_class != 'Idle'

ORDER BY time_waited_micro DESC

FETCH FIRST 20 ROWS ONLY;

```


### 2.5 Session Activity


```sql

-- Active sessions

SELECT s.sid, s.serial#, s.username, s.status, s.machine, s.program,

       s.last_call_et, s.logon_time, s.sql_id, s.prev_sql_id,

       p.spid as os_pid, s.blocking_session

FROM v$session s, v$process p

WHERE s.paddr = p.addr

  AND s.status = 'ACTIVE'

  AND s.username IS NOT NULL

ORDER BY s.last_call_et DESC;


-- Session resource usage

SELECT se.sid, se.serial#, se.username, se.status, se.machine, se.program,

       ss.value as logical_reads

FROM v$session se, v$sesstat ss, v$statname sn

WHERE se.sid = ss.sid

  AND ss.statistic# = sn.statistic#

  AND sn.name = 'session logical reads'

  AND se.username IS NOT NULL

ORDER BY ss.value DESC

FETCH FIRST 20 ROWS ONLY;


-- Long-running operations

SELECT sid, serial#, opname, target, sofar, totalwork, 

       ROUND(elapsed_seconds/60,2) as elapsed_min, 

       ROUND(time_remaining/60,2) as remaining_min,

       ROUND(sofar/totalwork*100,2) as pct_complete

FROM v$session_longops

WHERE sofar < totalwork

ORDER BY elapsed_seconds DESC;

```


### 2.6 Database Resource Consumption


```sql

-- Memory usage

SELECT * FROM v$sgastat

ORDER BY bytes DESC

FETCH FIRST 20 ROWS ONLY;


SELECT * FROM v$pgastat;


-- Undo usage

SELECT * FROM v$undostat

ORDER BY begin_time DESC

FETCH FIRST 10 ROWS ONLY;


-- Redo log activity

SELECT * FROM v$log;


SELECT thread#, sequence#, ROUND(bytes/1024/1024,2) as size_mb, 

       members, archived, status

FROM v$log

ORDER BY thread#, sequence#;


-- Tablespace usage

SELECT df.tablespace_name, df.file_name, df.bytes/1024/1024 as size_mb,

       ROUND((df.bytes - fs.bytes)/1024/1024,2) as used_mb,

       ROUND(fs.bytes/1024/1024,2) as free_mb,

       ROUND((df.bytes - fs.bytes) / df.bytes * 100, 2) as pct_used

FROM dba_data_files df,

     (SELECT file_id, SUM(bytes) as bytes

      FROM dba_free_space

      GROUP BY file_id) fs

WHERE df.file_id = fs.file_id

ORDER BY pct_used DESC;

```


### 2.7 Contended Objects and Locking Issues


```sql

-- Locked objects

SELECT l.session_id, l.oracle_username, l.os_user_name, 

       o.owner, o.object_name, o.object_type,

       l.locked_mode

FROM v$locked_object l, dba_objects o

WHERE l.object_id = o.object_id

ORDER BY l.session_id;


-- Blocking sessions

SELECT s1.username || '@' || s1.machine as blocker, 

       s2.username || '@' || s2.machine as blockee,

       l1.sid, l2.sid, l1.type,

       o.owner, o.object_name, o.object_type,

       l1.lmode, l2.request

FROM v$lock l1, v$lock l2, v$session s1, v$session s2, dba_objects o

WHERE l1.id1 = l2.id1

  AND l1.block = 1

  AND l2.request > 0

  AND l1.sid = s1.sid

  AND l2.sid = s2.sid

  AND l1.id1 = o.object_id;


-- Hot objects (frequently accessed)

SELECT o.owner, o.object_name, o.object_type, o.status,

       s.statistic_name, s.value

FROM v$segment_statistics s, dba_objects o

WHERE s.owner = o.owner

  AND s.object_name = o.object_name

  AND s.object_type = o.object_type

  AND s.statistic_name IN ('logical reads', 'physical reads', 'physical writes')

ORDER BY s.value DESC

FETCH FIRST 20 ROWS ONLY;

```


### 2.8 Integration-Specific Analysis


```sql

-- Sessions by module (to identify integrations)

SELECT module, COUNT(*) as session_count

FROM v$session

WHERE module IS NOT NULL

GROUP BY module

ORDER BY session_count DESC;


-- Resource usage by module

SELECT se.module, COUNT(DISTINCT se.sid) as session_count,

       SUM(ss.value) as total_logical_reads

FROM v$session se, v$sesstat ss, v$statname sn

WHERE se.sid = ss.sid

  AND ss.statistic# = sn.statistic#

  AND sn.name = 'session logical reads'

  AND se.module IS NOT NULL

GROUP BY se.module

ORDER BY total_logical_reads DESC;


-- SQL by module

SELECT se.module, sq.sql_id, COUNT(*) as execution_count

FROM v$session se, v$sql sq

WHERE se.sql_id = sq.sql_id

  AND se.module IS NOT NULL

GROUP BY se.module, sq.sql_id

ORDER BY execution_count DESC

FETCH FIRST 20 ROWS ONLY;

```


## 3. AWR and ADDM Analysis


### 3.1 Generate AWR Reports


```sql

-- List available snapshots

SELECT snap_id, begin_interval_time, end_interval_time

FROM dba_hist_snapshot

ORDER BY snap_id DESC

FETCH FIRST 20 ROWS ONLY;


-- Generate AWR report

SELECT output FROM TABLE(DBMS_WORKLOAD_REPOSITORY.AWR_REPORT_TEXT(

    l_dbid        => &database_id,

    l_inst_num    => &instance_number,

    l_bid         => &begin_snap_id,

    l_eid         => &end_snap_id,

    l_options     => 0

));


-- Generate AWR Global report for RAC

SELECT output FROM TABLE(DBMS_WORKLOAD_REPOSITORY.AWR_GLOBAL_REPORT_TEXT(

    l_dbid        => &database_id,

    l_inst_num    => &instance_number,

    l_bid         => &begin_snap_id,

    l_eid         => &end_snap_id,

    l_options     => 0

));

```


### 3.2 Generate ADDM Reports


```sql

-- Run ADDM analysis

DECLARE

  task_name VARCHAR2(30);

  task_desc VARCHAR2(256);

  task_id   NUMBER;

BEGIN

  task_name := 'ADDM_' || TO_CHAR(SYSDATE, 'YYYYMMDD_HH24MISS');

  task_desc := 'ADDM run for high load analysis';

  

  DBMS_ADVISOR.CREATE_TASK(

    advisor_name => 'ADDM',

    task_id      => task_id,

    task_name    => task_name,

    task_desc    => task_desc

  );

  

  DBMS_ADVISOR.SET_TASK_PARAMETER(

    task_name => task_name,

    parameter => 'START_SNAPSHOT',

    value     => &begin_snap_id

  );

  

  DBMS_ADVISOR.SET_TASK_PARAMETER(

    task_name => task_name,

    parameter => 'END_SNAPSHOT',

    value     => &end_snap_id

  );

  

  DBMS_ADVISOR.EXECUTE_TASK(task_name => task_name);

END;

/


-- View ADDM report

SELECT DBMS_ADVISOR.GET_TASK_REPORT('&task_name', 'TEXT', 'ALL') as report

FROM DUAL;

```


## 4. Step-by-Step RCA Process


### 4.1 Initial Assessment


1. **Collect baseline metrics**:

   - Run `mpstat`, `vmstat`, `iostat`, and `prstat` to establish current system load

   - Query v$instance, v$resource_limit, and v$system_time_model for database status


2. **Identify immediate symptoms**:

   - Check for CPU saturation using `mpstat` and `prstat`

   - Check for memory pressure using `vmstat` and `echo "::memstat" | mdb -k`

   - Check for I/O bottlenecks using `iostat -xnz`

   - Check for network issues using `netstat` and `nicstat`


3. **Review alert logs**:

   - Check system logs: `/var/adm/messages` and `dmesg`

   - Check Oracle alert logs: `$ORACLE_BASE/diag/rdbms/$ORACLE_SID/$ORACLE_SID/trace/alert_$ORACLE_SID.log`


### 4.2 Database Performance Analysis


1. **Generate and analyze AWR reports**:

   - Identify peak load periods from AWR snapshots

   - Generate AWR reports for those periods

   - Review Top 5 Timed Events section to identify main wait events


2. **Analyze active sessions**:

   - Query v$session for currently active sessions

   - Query v$session_wait for current wait events

   - Query v$active_session_history for historical session activity


3. **Identify high-load SQL statements**:

   - Query v$sql for top SQL by CPU and elapsed time

   - Get execution plans for high-load SQL statements

   - Analyze SQL statistics for inefficient execution


4. **Check for resource contention**:

   - Query v$locked_object for object locks

   - Check for blocking sessions

   - Identify hot objects from v$segment_statistics


### 4.3 Integration-Specific Analysis


1. **Identify active integrations**:

   - Query v$session by module to identify active integration connections

   - Group sessions by module to identify high-resource integrations


2. **Analyze integration patterns**:

   - Check connection frequency and duration

   - Identify SQL statements executed by each integration

   - Analyze resource consumption patterns by integration


3. **Correlation with system metrics**:

   - Match integration activity peaks with system load spikes

   - Identify integrations causing excessive I/O or CPU usage


### 4.4 Root Cause Identification


1. **Cross-reference all metrics**:

   - Correlate system-level bottlenecks with database wait events

   - Match high-load periods with specific SQL or integration activity

   - Identify resource contention patterns


2. **Categorize issues by impact**:

   - CPU-bound issues (high CPU usage, runaway processes)

   - Memory-bound issues (insufficient SGA/PGA, excessive sorting)

   - I/O-bound issues (disk bottlenecks, inefficient I/O patterns)

   - Concurrency issues (locks, latches, contention)

   - Application design issues (inefficient SQL, excessive connections)


3. **Validate findings**:

   - Confirm patterns across multiple data points

   - Verify timing correlation between symptoms and causes

   - Test hypotheses with targeted queries


## 5. Recommendations for Resolution


### 5.1 SQL Tuning Recommendations


1. **Optimize high-load SQL statements**:

   ```sql

   -- Create SQL profile for high-load statement

   DECLARE

     l_sql_text CLOB;

   BEGIN

     SELECT sql_fulltext INTO l_sql_text

     FROM v$sql

     WHERE sql_id = '&sql_id'

     AND rownum = 1;

     

     DBMS_SQLTUNE.CREATE_TUNING_TASK(

       sql_text    => l_sql_text,

       task_name   => 'TUNE_&sql_id',

       description => 'Tuning task for high-load SQL'

     );

     

     DBMS_SQLTUNE.EXECUTE_TUNING_TASK(task_name => 'TUNE_&sql_id');

   END;

   /

   

   -- View tuning recommendations

   SELECT DBMS_SQLTUNE.REPORT_TUNING_TASK('TUNE_&sql_id') AS recommendations

   FROM DUAL;

   

   -- Implement SQL profile if recommended

   DECLARE

     l_result VARCHAR2(4000);

   BEGIN

     l_result := DBMS_SQLTUNE.ACCEPT_SQL_PROFILE(

       task_name   => 'TUNE_&sql_id',

       name        => 'PROF_&sql_id',

       force_match => TRUE

     );

   END;

   /

   ```


2. **Create missing indexes**:

   ```sql

   -- Identify missing indexes

   SELECT recommendation, benefit

   FROM TABLE(DBMS_ADVISOR.GET_TASK_REPORT('TUNE_&sql_id', 'TEXT', 'ALL'))

   WHERE recommendation LIKE '%CREATE INDEX%';

   

   -- Create recommended index

   CREATE INDEX owner.index_name ON owner.table_name(column1, column2);

   ```


3. **Consider SQL plan baselines**:

   ```sql

   -- Create SQL plan baseline

   DECLARE

     l_sql_handle VARCHAR2(30);

     l_plan_name  VARCHAR2(30);

   BEGIN

     DBMS_SPM.LOAD_PLANS_FROM_CURSOR_CACHE(

       sql_id          => '&sql_id',

       plan_hash_value => &plan_hash_value,

       sql_handle      => l_sql_handle,

       plan_name       => l_plan_name

     );

   END;

   /

   ```


### 5.2 Database Configuration Changes


1. **Adjust memory parameters**:

   ```sql

   -- Check current memory allocation

   SHOW PARAMETER sga;

   SHOW PARAMETER pga;

   

   -- Adjust SGA and PGA sizes

   ALTER SYSTEM SET sga_target = 16G SCOPE=SPFILE;

   ALTER SYSTEM SET pga_aggregate_target = 8G SCOPE=SPFILE;

   

   -- Adjust shared pool size if needed

   ALTER SYSTEM SET shared_pool_size = 4G SCOPE=SPFILE;

   ```


2. **Optimize redo log configuration**:

   ```sql

   -- Check current redo log configuration

   SELECT group#, bytes/1024/1024 as size_mb, members, status FROM v$log;

   

   -- Add redo log groups or resize if needed

   ALTER DATABASE ADD LOGFILE GROUP 4 ('/path/to/redo04a.log', '/path/to/redo04b.log') SIZE 1G;

   ALTER DATABASE ADD LOGFILE GROUP 5 ('/path/to/redo05a.log', '/path/to/redo05b.log') SIZE 1G;

   ```


3. **Adjust database parameters**:

   ```sql

   -- Optimize parallel execution

   ALTER SYSTEM SET parallel_max_servers = 32 SCOPE=BOTH;

   ALTER SYSTEM SET parallel_min_servers = 8 SCOPE=BOTH;

   

   -- Optimize optimizer parameters

   ALTER SYSTEM SET optimizer_index_cost_adj = 50 SCOPE=BOTH;

   ALTER SYSTEM SET optimizer_index_caching = 75 SCOPE=BOTH;

   

   -- Adjust session limits for integrations

   ALTER SYSTEM SET sessions = 1000 SCOPE=SPFILE;

   ALTER SYSTEM SET processes = 800 SCOPE=SPFILE;

   ```


### 5.3 System-Level Optimizations


1. **Solaris kernel parameter tuning**:

   ```bash

   # Check current kernel parameters

   /usr/sbin/ndd -get /dev/tcp tcp_conn_req_max_q

   /usr/sbin/ndd -get /dev/tcp tcp_conn_req_max_q0

   /usr/sbin/ndd -get /dev/tcp tcp_xmit_hiwat

   /usr/sbin/ndd -get /dev/tcp tcp_recv_hiwat

   

   # Set optimized parameters

   echo "tcp_conn_req_max_q=1024" >> /etc/system

   echo "tcp_conn_req_max_q0=4096" >> /etc/system

   echo "tcp_xmit_hiwat=65536" >> /etc/system

   echo "tcp_recv_hiwat=65536" >> /etc/system

   ```


2. **I/O scheduling optimization**:

   ```bash

   # Check current I/O scheduler

   echo "::iosched" | mdb -k

   

   # Adjust I/O priority for Oracle processes

   priocntl -e -c RT -p 59 -i pid `pgrep -f ora_dbw`

   priocntl -e -c RT -p 59 -i pid `pgrep -f ora_lgwr`

   ```


3. **File system optimization**:

   ```bash

   # Check current mount options

   mount | grep oracle

   

   # Remount with optimized options

   mount -o remount,forcedirectio,nologging /oracle

   ```


### 5.4 Integration-Specific Recommendations


1. **Connection pooling optimization**:

   - Implement connection pooling for all integrations

   - Set appropriate pool sizes based on workload

   - Configure timeout and validation parameters


2. **Batch processing improvements**:

   - Convert real-time queries to batch processing where possible

   - Schedule integration jobs during off-peak hours

   - Implement incremental processing for large data sets


3. **Load balancing**:

   - Distribute integration connections across RAC nodes

   - Implement service-based routing for workload management

   - Configure services with appropriate runtime priorities


### 5.5 Monitoring and Alerting Setup


1. **Set up proactive monitoring**:

   - Configure OEM metrics and thresholds

   - Create custom metrics for integration-specific monitoring

   - Set up baseline deviation alerts


2. **Implement automated response**:

   - Create incident rules for common issues

   - Configure automatic trace collection for problematic sessions

   - Implement automatic SQL tuning for high-load statements


3. **Long-term trend analysis**:

   - Configure AWR retention for longer periods

   - Set up regular AWR comparison reports

   - Implement capacity planning based on growth trends


## 6. Implementation Plan


### 6.1 Immediate Actions (0-24 hours)


1. Identify and tune top 5 resource-intensive SQL statements

2. Resolve any blocking sessions or locking issues

3. Adjust memory parameters if insufficient

4. Implement connection pooling for high-volume integrations

5. Set up basic monitoring alerts for critical resources


### 6.2 Short-Term Actions (1-7 days)


1. Complete SQL tuning for all high-load statements

2. Implement recommended indexes and SQL profiles

3. Adjust database parameters based on workload analysis

4. Optimize Solaris kernel parameters

5. Implement batch processing improvements for integrations


### 6.3 Long-Term Actions (1-4 weeks)


1. Redesign inefficient application code or database structures

2. Implement comprehensive monitoring and alerting

3. Develop capacity planning based on growth projections

4. Consider hardware upgrades if necessary

5. Implement regular performance review process


## 7. Conclusion


This root cause analysis guide provides a comprehensive approach to diagnosing and resolving high server load issues on a Solaris-based Oracle 19c Gold Tier database environment with multiple integrations. By systematically collecting and analyzing both system and database metrics, you can identify the specific bottlenecks causing performance degradation and implement targeted solutions to address them.


The recommendations provided are designed to be implemented in a phased approach, starting with immediate actions to alleviate critical issues, followed by short-term and long-term improvements to ensure sustained performance. Regular monitoring and proactive tuning will help prevent similar issues from recurring in the future.


Thursday, April 24, 2025

longops.sql

 



rem LONGOPS.SQL

rem Long Running Statements

rem Helmut Pfau, Oracle Deutschland GmbH

set linesize 120

col opname format a20

col target format a15

col units format a10

col time_remaining format 99990 heading Remaining[s]

col bps format 9990.99 heading [Units/s]

col fertig format 90.99 heading "complete[%]"

select sid,

       opname,

       target,

       sofar,

       totalwork,

       units,

       (totalwork-sofar)/time_remaining bps,

       time_remaining,

       sofar/totalwork*100 fertig

from   v$session_longops

where  time_remaining > 0

/


Planning relatede document from oracle for troubleshooting

 



Script for Getting Correct Request IDs and their Corresponding Logs for Plan and Collection Run (Doc ID 262276.1)

REQUESTS.sql Script for Parent/Child Request IDs and Trace File IDs (Doc ID 280295.1)

Tips on Collecting Request Log Files for Logging SRs for VCP (Doc ID 1268131.1)

How To Find Trace File From Concurrent Request ID (Doc ID 452225.1)

Refresh Collection Snapshots Performance - Managing MLOG$ Tables and Snapshots for Data Collections (Doc ID 1063953.1)

RAC Configuration Setup For Running MRP Planning, APS Planning,Data Collection Processes and Distributed Multi Node Transactions (Doc ID 279156.1)

Concurrent Processing - CP Analyzer Sample SQL Statements Including Date Range Parameters from Doc 1411723.1- Concurrent Processing - CP Analyzer for E-Business Suite (Doc ID 1499538.1)

Thursday, April 10, 2025

csv

 Prompt Title,Category,Purpose,Prompt (for LLM)

Shift Handover Email,Email Communication,Summarize Oracle Apps DBA shift activity,"Write a professional Oracle Apps DBA shift handover email. Include ADOP cycles, patching status, concurrent request failures, and pending user requests."

Daily Status Email,Email Communication,Communicate daily progress,"Create a daily Oracle Apps DBA update email listing completed tasks (e.g., backup validation, patching, monitoring), open SRs, and pending approvals."

Patch Completion Notification,Email Communication,Notify team of patch cycle completion,"Write an internal email notifying completion of ADOP patching on EBS 12.2. Include downtime, steps performed, and post-patch checks."

Incident Summary Email,Email Communication,Report an Oracle EBS outage or incident,"Summarize a PROD EBS login failure incident, including timeline, root cause (e.g., WebLogic hang), resolution steps, and preventive action."

Pre-Clone Notification Email,Email Communication,Notify teams of an upcoming clone,"Draft an email informing stakeholders of a scheduled Oracle EBS cloning from PROD to TEST. Include timeline, impact, and credentials sharing plan."

Post-Clone Update Email,Email Communication,Inform completion of clone,"Write an email confirming successful clone of EBS PROD to DEV. Include system status, known issues, and post-clone action points."

SR Escalation Email,Email Communication,Escalate Oracle SRs,"Draft an escalation email for Oracle Support regarding a high-priority SR. Include SR number, impact on PROD, and requested action."

Change Request Approval Email,Email Communication,Formal approval request,"Write an email requesting approval for an EBS patch application in PROD. Include patch details, ADOP phase plan, and rollback strategy."

Concurrent Request Alert,Email Communication,Notify about failing concurrent programs,"Generate an alert email about repeated failure of a scheduled concurrent program in PROD. Include log snippet and suggested action."

Out-of-Office Reply,Email Communication,Auto-reply for Oracle Apps DBA role,"Write an out-of-office reply for an Oracle Apps DBA. Mention limited access to Oracle SRs and include alternate contact for urgent PROD issues."


Incident Timeline Generator,Incident Response,Create a detailed timeline of an incident,"Generate a detailed timeline of an Oracle EBS incident, including detection, impact (e.g. Forms not launching), DBA actions, and recovery completion time."

RCA Summary,Incident Response,Summarize root cause analysis,"Write an RCA for a PROD outage caused by Workflow Mailer failure. Include symptoms, root cause, resolution, and future prevention steps."

Post-Mortem Report,Incident Response,Document full incident analysis,"Draft a post-mortem report for an Oracle DB node eviction from a RAC cluster. Include timestamps, ASM/GRID impact, actions, and lessons learned."

Impact Assessment Email,Incident Response,Communicate business impact,"Write an email summarizing how the EBS outage affected users (e.g. PO entry delay), including total downtime and systems impacted."

Urgent SR Request Email,Incident Response,Request support escalation,"Generate an urgent escalation email for a P1 SR related to PROD EBS login issues. Include SR number, impact, and request for immediate response."

Incident Ticket Summary,Incident Response,Summarize ticket for team visibility,"Summarize this SR in bullet points: Issue, Affected Components (e.g. Concurrent Manager), RCA, Fix, and Follow-ups."

Severity Classification Prompt,Incident Response,Classify and justify severity,"Based on this issue (e.g. forms crashing in PROD), classify its severity (Sev1–Sev3) and justify the categorization."

Suggested Fix Generator,Incident Response,Generate fix ideas,"Suggest common fixes for Workflow Mailer not starting in Oracle EBS. Include log checks, listener status, and bounce steps."

Alert Log Analysis,Incident Response,Analyze repeating Oracle DB alerts,"Analyze repeated ORA-600 errors in alert log. Identify possible causes and recommend Oracle SR references if needed."

Executive Summary Email,Incident Response,Simplified leadership communication,"Write a one-paragraph incident summary suitable for leadership about PROD DB performance issues due to stats gathering delay."


Change Request Summary,Change & Release Management,Summarize patch or config change,"Generate a change request summary for applying Oracle Patch 32809584 (PRC Payables). Include why, when, risk, and rollback plan."

Stakeholder Notification Email,Change & Release Management,Notify users of a change,"Draft a user notification email for scheduled downtime for EBS patching. Include timeline, scope, and contact info."

Rollback Plan Generator,Change & Release Management,Prepare rollback plan,"Write a rollback plan for a failed EBS adop phase=apply. Include fs_clone steps, backup reference, and service restart."

Impact Analysis Prompt,Change & Release Management,Define application impact,"Describe how applying patch 32809584 impacts AP module workflows, concurrent programs, and APIs."

Internal Summary Brief,Change & Release Management,Internal DBA team communication,"Summarize the change plan for team: patch ID, expected downtime, patch stage, testing steps, and owner."

CAB Meeting Summary,Change & Release Management,Document CAB discussions,"Summarize the CAB meeting outcomes for EBS changes: approved patches, rejections, risks, and action items."

Pre-Change Checklist,Change & Release Management,Ensure readiness,"Generate a pre-change checklist for EBS PROD patching: fs_backup, adop cleanup, service checks, notification plan."

Approval Request Email,Change & Release Management,Formal approval ask,"Write an email to request manager approval for EBS patching in PROD. Include justification, risk level, and patch scope."

Post-Change Validation Summary,Change & Release Management,Confirm post-patch system state,"Write a summary after patching: modules tested, login success, concurrent manager status, and workflow checks."

Deployment Completion Email,Change & Release Management,Inform users of success,"Write an email announcing successful patching of Oracle EBS PROD. Include modules impacted and verification status."


SQL Script Generator,Task Automation & Scripts,Create SQL for DBA tasks,"Generate a SQL script to list all concurrent programs scheduled to run in the next 24 hours in Oracle EBS."

Backup Script Builder,Task Automation & Scripts,Generate backup script,"Write a shell script to perform RMAN backup of Oracle PROD DB and log output with timestamp."

Concurrent Monitor Script,Task Automation & Scripts,Monitor running jobs,"Create a SQL script to check long-running concurrent programs and their statuses."

Cronjob Example (Maintenance),Task Automation & Scripts,Automate EBS task via cron,"Generate a cron entry that runs an EBS concurrent request monitoring script every 15 mins."

Patch Analyzer Tool,Task Automation & Scripts,Analyze patch impact,"Write a PL/SQL script to check applied patches related to AP module in EBS."

Clone Precheck Script,Task Automation & Scripts,Verify clone readiness,"Generate a shell script to verify space, DB status, and app tier for upcoming PROD to TEST clone."

Alert Log Monitor Script,Task Automation & Scripts,Monitor for critical DB alerts,"Create a script that tails Oracle alert log and triggers email if ORA-00600 or ORA-07445 appears."

SQL Formatter,Task Automation & Scripts,Format SQL for readability,"Format and optimize this SQL for Oracle EBS reporting: [insert SQL]"

Password Expiry Checker,Task Automation & Scripts,Check for expiring users,"Write a SQL query to list Oracle DB users whose passwords expire in next 7 days."

Workflow Error Extractor,Task Automation & Scripts,Extract WF errors,"Create a script that pulls last 50 Workflow Mailer errors from WF_NOTIFICATIONS for troubleshooting."


Alert Summary Email,Monitoring & Alerts,Summarize current alerts,"Summarize today's Oracle DB alerts, including ORA errors, tablespace warnings, and listener status."

Anomaly Detection Prompt,Monitoring & Alerts,Explain deviation,"Explain why PROD EBS login response time is high compared to baseline. Use available AWR and system stats."

Alert Escalation Email,Monitoring & Alerts,Raise persistent issues,"Draft an email escalating high undo tablespace usage in PROD for 3+ hours."

Monitoring Rule Generator,Monitoring & Alerts,Define alert threshold,"Create an alert rule in OEM for failed concurrent requests > 10 in 1 hour in EBS PROD."

Standard Alert Template,Monitoring & Alerts,Template for alert emails,"Generate a standard alert email format: subject, alert ID, system, time, impact, and next step."

Performance Threshold Advisor,Monitoring & Alerts,Define safe limits,"Suggest Oracle DB alert thresholds for CPU > 90%, session waits > 100, and I/O over 50ms latency."

DB Health Check Summary,Monitoring & Alerts,Overall environment status,"Summarize daily DB health: archive logs, backups, listener, FRA usage, and services."

Weekly Metrics Comparison,Monitoring & Alerts,Compare usage trends,"Compare AWR metrics for last week vs this week: DB time, wait classes, top SQLs."

Alert Correlation Prompt,Monitoring & Alerts,Identify related issues,"Analyze alerts and correlate DB locks, long jobs, and listener issues from same time frame."

Non-Technical Status Update,Monitoring & Alerts,Write for business users,"Write a simplified DB health update email for functional users — 'All systems green' or mention known slowness."


Thursday, March 20, 2025

in

#!/bin/bash

# Set the environment
cd $HOME
source Ebsapps.env run

# Define variables
REPORT_FILE="objects_report_$(date +%Y%m%d_%H%M%S).html"
ORACLE_USER="apps"
ORACLE_PASS="mad15max"
ORACLE_SID=$(echo $TWO_TASK)  # Assumes TWO_TASK is set in Ebsapps.env

# SQL query to fetch data from DBA_OBJECTS and AD_OBJECTS
SQL_QUERY=$(cat << 'EOF'
SET HEADING OFF
SET FEEDBACK OFF
SET PAGESIZE 0
SET LINESIZE 500
SET ECHO OFF
SET MARKUP HTML ON TABLE "border='1' cellpadding='5' cellspacing='0' style='border-collapse: collapse; width: 80%; margin: 20px auto;'"
SPOOL dba_ad_objects.html

SELECT '<h2 style="color: #2E7D32; text-align: center;">Database Objects Report - ' || TO_CHAR(SYSDATE, 'DD-MON-YYYY HH24:MI:SS') || '</h2>' FROM dual;

SELECT '<table>' ||
       '<tr style="background-color: #4CAF50; color: white;">' ||
       '<th>Owner</th><th>Object Name</th><th>Object Type</th><th>Status</th><th>Created</th>' ||
       '</tr>'
FROM dual;

SELECT CASE
         WHEN status = 'VALID' THEN '<tr style="background-color: #E8F5E9;">'
         ELSE '<tr style="background-color: #FFEBEE;">'
       END ||
       '<td>' || owner || '</td>' ||
       '<td>' || object_name || '</td>' ||
       '<td>' || object_type || '</td>' ||
       '<td>' || status || '</td>' ||
       '<td>' || TO_CHAR(created, 'DD-MON-YYYY') || '</td>' ||
       '</tr>'
FROM dba_objects
WHERE owner NOT IN ('SYS', 'SYSTEM')
  AND rownum <= 50;  -- Limit to 50 rows for demo; adjust as needed

SELECT '<h2 style="color: #0277BD; text-align: center;">EBS AD Objects Report</h2>' FROM dual;

SELECT '<table>' ||
       '<tr style="background-color: #0288D1; color: white;">' ||
       '<th>Object Name</th><th>Object Type</th><th>Status</th><th>Last DDL Time</th>' ||
       '</tr>'
FROM dual;

SELECT CASE
         WHEN status = 'VALID' THEN '<tr style="background-color: #E3F2FD;">'
         ELSE '<tr style="background-color: #FFEBEE;">'
       END ||
       '<td>' || object_name || '</td>' ||
       '<td>' || object_type || '</td>' ||
       '<td>' || status || '</td>' ||
       '<td>' || TO_CHAR(last_ddl_time, 'DD-MON-YYYY HH24:MI:SS') || '</td>' ||
       '</tr>'
FROM ad_objects
WHERE rownum <= 50;  -- Limit to 50 rows for demo; adjust as needed

SELECT '</table>' FROM dual;

SELECT '<p style="text-align: center; color: #555;">Report generated on ' || TO_CHAR(SYSDATE, 'DD-MON-YYYY HH24:MI:SS') || ' by Grok 3 (xAI)</p>' FROM dual;

SPOOL OFF
EXIT
EOF
)

# Execute the SQL query and generate the HTML report
echo "$SQL_QUERY" | sqlplus -S $ORACLE_USER/$ORACLE_PASS@$ORACLE_SID

# Move and rename the output file
mv dba_ad_objects.html $REPORT_FILE

# Check if the report was generated successfully
if [ -f "$REPORT_FILE" ]; then
    echo "Report generated successfully: $REPORT_FILE"
else
    echo "Error: Failed to generate the report."
    exit 1
fi

exit 0