Pre Prod Analysis

 -- 1. Confirm concurrent request details

SELECT request_id,
       phase_code,
       status_code,
       actual_start_date,
       actual_completion_date,
       argument_text,
       logfile_name,
       outfile_name
FROM apps.fnd_concurrent_requests
WHERE request_id = 598460495;

-- 2. Get program/application details
SELECT r.request_id,
       p.concurrent_program_name,
       pt.user_concurrent_program_name,
       a.application_short_name,
       r.phase_code,
       r.status_code
FROM apps.fnd_concurrent_requests r,
     apps.fnd_concurrent_programs p,
     apps.fnd_concurrent_programs_tl pt,
     apps.fnd_application a
WHERE r.concurrent_program_id = p.concurrent_program_id
AND   p.concurrent_program_id = pt.concurrent_program_id
AND   p.application_id = a.application_id
AND   p.application_id = pt.application_id
AND   pt.language = 'US'
AND   r.request_id = 598460495;

-- 3. Validate current database session
SELECT s.inst_id,
       s.sid,
       s.serial#,
       s.username,
       s.status,
       s.sql_id,
       s.prev_sql_id,
       s.event,
       s.wait_class,
       s.seconds_in_wait,
       s.blocking_session,
       s.module,
       s.action,
       s.client_identifier,
       s.machine,
       s.program
FROM gv$session s
WHERE s.sid = 2581
AND   s.serial# = 24100;

-- 4. Check if it is really sleeping/retry loop
SELECT inst_id,
       sid,
       serial#,
       event,
       wait_class,
       state,
       seconds_in_wait,
       sql_id,
       prev_sql_id,
       plsql_entry_object_id,
       plsql_object_id,
       plsql_subprogram_id
FROM gv$session
WHERE sid = 2581
AND serial# = 24100;

-- 5. Identify PL/SQL object currently involved
SELECT s.sid,
       s.sql_id,
       s.event,
       o.owner,
       o.object_name,
       o.object_type
FROM gv$session s
LEFT JOIN dba_objects o
       ON s.plsql_object_id = o.object_id
WHERE s.sid = 2581
AND s.serial# = 24100;

-- 6. Get SQL text for SQL_ID
SELECT sql_id,
       sql_text
FROM gv$sql
WHERE sql_id = 'dxasruz3nkuvf';

-- 7. Check ASH history for this request/session
SELECT sample_time,
       session_id,
       session_serial#,
       sql_id,
       event,
       wait_class,
       module,
       action,
       blocking_session,
       current_obj#
FROM gv$active_session_history
WHERE session_id = 2581
AND session_serial# = 24100
ORDER BY sample_time DESC;

-- 8. If ASH aged out, check AWR ASH
SELECT sample_time,
       instance_number,
       session_id,
       session_serial#,
       sql_id,
       event,
       wait_class,
       module,
       action,
       blocking_session
FROM dba_hist_active_sess_history
WHERE session_id = 2581
AND session_serial# = 24100
AND sample_time >= SYSDATE - 1
ORDER BY sample_time DESC;

-- 9. Check if request is waiting for another request
SELECT request_id,
       parent_request_id,
       priority_request_id,
       phase_code,
       status_code,
       hold_flag,
       requested_start_date,
       actual_start_date
FROM apps.fnd_concurrent_requests
WHERE request_id = 598460495
   OR parent_request_id = 598460495;

-- 10. Check incompatibility/blocking at concurrent manager level
SELECT r.request_id,
       r.phase_code,
       r.status_code,
       r.hold_flag,
       r.requested_start_date,
       r.actual_start_date,
       r.controlling_manager,
       r.concurrent_program_id
FROM apps.fnd_concurrent_requests r
WHERE r.phase_code = 'R'
AND r.status_code = 'R'
ORDER BY r.actual_start_date;

Initial RCA direction:

The first request is not showing a normal I/O wait. It is waiting on PL/SQL lock timer, which commonly indicates the program is intentionally sleeping or polling inside package logic. Since it has already run for around 128 minutes, the DBA should verify whether the TPA Monitor program is stuck in a retry loop, waiting for dependent activity, waiting for another concurrent request, or controlled by application logic.

Do not kill immediately. First collect:

-- Evidence snapshot
SELECT SYSDATE evidence_time,
       inst_id,
       sid,
       serial#,
       sql_id,
       prev_sql_id,
       event,
       wait_class,
       seconds_in_wait,
       module,
       action,
       client_identifier
FROM gv$session
WHERE sid = 2581
AND serial# = 24100;

Then check the request log file from logfile_name. If the log is not moving and ASH repeatedly shows PL/SQL lock timer, then update business/application owner:

Suggested update:

The request  is currently active for ~128 minutes. Database session is not waiting on I/O or CPU bottleneck; it is mainly waiting on PL/SQL lock timer, which indicates the program is sleeping/polling inside application PL/SQL logic. DBA is validating the request log, PL/SQL package, dependency with other requests, and ASH history before taking any action. Recommended not to kill until application/business confirmation is received.

Deep Dive: Oracle EBS 12.2 OACore JVM OutOfMemoryError Troubleshooting Runbook

Oracle E-Business Suite 12.2 OACore JVM memory issues are common in production environments. One of the most critical errors Apps DBAs may see is:

java.lang.OutOfMemoryError: GC overhead limit exceeded

This error means the OACore JVM is spending too much time in Garbage Collection but is unable to release enough memory. As a result, users may experience application slowness, page hangs, login issues, or OAF screen failures.

---

1. Common Error Seen in OACore Logs

In the OACore server output/log file, you may see errors like:

<Error> <Socket> <BEA-000405> <Uncaught Throwable in processSockets>
java.lang.OutOfMemoryError: GC overhead limit exceeded

Other related errors may include:

java.lang.OutOfMemoryError: Java heap space
BEA-000337 ExecuteThread has been busy
BEA-000339 Thread has become unstuck
Unable to reserve connection
ThreadPool has stuck threads

---

2. What This Error Means

GC overhead limit exceeded means the JVM is almost out of heap memory.

The JVM is continuously trying to clean memory, but it is not able to recover enough memory.

In simple terms:

OACore JVM memory is exhausted.
Garbage Collection is running repeatedly.
Application threads become slow or stuck.
Users experience application slowness or hanging.

---

3. Typical User Symptoms

Users may report:

Application is slow
OAF pages are hanging
Login page is slow
Forms launch delay
Blank page after clicking responsibility
Submit button not responding
Intermittent page errors

---

4. First Step: Identify the Affected OACore

Login to the application tier and run:

ps -ef | grep -i oacore | grep -v grep

Example output:

oacore_server4  PID=4269

Note the affected OACore managed server and its PID.

---

5. Check OACore Logs

Go to the WebLogic server logs directory:

cd $EBS_DOMAIN_HOME/servers

Search for recent OACore errors:

find . -iname "*oacore*.out" -mtime -1
find . -iname "*oacore*.log" -mtime -1

Search inside logs:

grep -iE "OutOfMemory|GC overhead|Java heap space|BEA-000405|stuck thread|Unable to reserve connection" */logs/*

If you see:

java.lang.OutOfMemoryError: GC overhead limit exceeded

then the OACore JVM is under memory pressure.

---

6. Capture Evidence Before Restart

Before restarting OACore, capture diagnostics.

Create a directory:

mkdir -p /tmp/oacore_oom_$(date +%F_%H%M)
cd /tmp/oacore_oom_$(date +%F_%H%M)

Capture thread dumps:

jstack -l <PID> > jstack_1.txt
sleep 30
jstack -l <PID> > jstack_2.txt
sleep 30
jstack -l <PID> > jstack_3.txt

Capture heap summary:

jmap -heap <PID> > heap_summary.txt

Capture live heap histogram:

jmap -histo:live <PID> > heap_histo_live.txt

Capture GC utilization:

jstat -gcutil <PID> 5s 12 > gcutil.txt

---

7. How to Read GC Output

Run:

jstat -gcutil <PID> 5s 12

Focus on these columns:

Column	Meaning	Warning Sign
O	Old Generation usage	90% to 100% continuously
FGC	Full GC count	Increasing frequently
FGCT	Full GC time	Increasing continuously
YGC	Young GC count	Frequent is acceptable
GCT	Total GC time	Very high means GC pressure

Bad sign:

O = 98% or 99%
FGC increasing every few seconds
FGCT increasing continuously

This confirms GC thrashing.

---

8. Check Current JVM Heap Size

Run:

ps -ef | grep -i oacore | grep Xmx

Look for values like:

-Xms2048m -Xmx2048m

If the heap is too small for the workload, OACore may hit memory exhaustion.

---

9. Validate from Database Side

From the database, check JDBC sessions created by the OACore PID:

SELECT s.inst_id,
       s.sid,
       s.serial#,
       s.status,
       s.username,
       s.program,
       s.process client_pid,
       s.sql_id,
       s.prev_sql_id,
       s.event,
       s.wait_class,
       s.last_call_et
FROM gv$session s
WHERE s.program LIKE '%JDBC%'
AND s.process = '<OACORE_PID>'
ORDER BY s.last_call_et DESC;

If you see:

SQL_ID     = NULL
EVENT      = SQL*Net message from client
WAIT_CLASS = Idle

then the database is waiting for OACore to send the next request.

This confirms the database is not the bottleneck at that moment.

---

10. Find Abnormal JDBC Session Count

Use this query:

SELECT s.process client_pid,
       COUNT(*) total_sessions,
       SUM(CASE WHEN s.status='ACTIVE' THEN 1 ELSE 0 END) active_sessions,
       SUM(CASE WHEN s.status='INACTIVE' THEN 1 ELSE 0 END) inactive_sessions
FROM gv$session s
WHERE s.program LIKE '%JDBC%'
GROUP BY s.process
ORDER BY total_sessions DESC;

If one OACore PID has an unusually high number of sessions, check for:

Connection leak
Stuck requests
JDBC pool saturation
Unhealthy load balancing

---

11. Check WebLogic Stuck Threads

Search OACore logs:

grep -iE "stuck|hogging|ExecuteThread|ThreadPool|BEA-000337|BEA-000339" $EBS_DOMAIN_HOME/servers/oacore_server*/logs/*

Important messages:

BEA-000337 ExecuteThread has been busy
BEA-000339 Thread has become unstuck

Many stuck threads indicate OACore is not processing requests normally.

---

12. Check JDBC Pool Saturation

From WebLogic Console, check the OACore data source.

Review:

Active Connections Current Count
Active Connections High Count
Waiting For Connection Current Count
Leaked Connection Count
Failures To Reconnect Count

Danger signs:

Waiting For Connection > 0
Leaked Connection Count > 0
Active Connections near Max Capacity

---

13. Analyze Heap Histogram

Run:

jmap -histo:live <PID> | head -50

Look for top objects:

Object Pattern	Possible Meaning
byte[]	Large payloads, XML, files, attachments
char[] / String	Large text, session data, cached data
oracle.apps.fnd.framework	OAF object growth
oracle.jbo	BC4J/Application Module objects
weblogic.servlet	HTTP session objects
java.util.HashMap	Cache/session growth
XML classes	Large XML or BI Publisher payload

This helps identify whether memory is consumed by XML, OAF pages, sessions, attachments, or custom code.

---

14. Immediate Mitigation

If OACore is unhealthy, restart only the affected managed server after collecting diagnostics.

admanagedsrvctl.sh stop oacore_server4
admanagedsrvctl.sh start oacore_server4

If graceful stop is not working, take approval and then use OS-level kill carefully:

kill -3 <PID>
kill <PID>

Avoid:

kill -9

unless the process is completely hung and approved.

---

15. Post-Restart Validation

Check OACore process:

ps -ef | grep -i oacore | grep -v grep

Check logs:

tail -100f $EBS_DOMAIN_HOME/servers/oacore_server4/logs/oacore_server4.out

Check GC behavior:

jstat -gcutil <NEW_PID> 5s 10

Healthy signs:

Old Generation is not stuck at 99%
Full GC is not continuously increasing
Users can access OAF pages normally
No new OutOfMemoryError appears

---

16. Permanent Fix Approach

16.1 Increase OACore Heap

Example:

-Xms4096m
-Xmx4096m

Do not blindly increase heap. First validate server RAM and total number of JVMs.

---

16.2 Add More OACore Managed Servers

If user load is high, distribute traffic across more OACore JVMs.

---

16.3 Review Custom OAF Pages

Common causes:

VO query fetching too many rows
Large LOV
Attachment rendering
Personalization issue
Session objects not released
Custom code memory leak

---

16.4 Enable Heap Dump on OOM

Add JVM options:

-XX:+HeapDumpOnOutOfMemoryError
-XX:HeapDumpPath=/u01/heapdumps

Make sure the filesystem has enough space.

---

16.5 Review WebLogic JDBC Pool

Check:

Max Capacity
Initial Capacity
Connection Reserve Timeout
Inactive Connection Timeout
Leaked Connection Count

---

17. Final RCA Statement

Oracle EBS 12.2 OACore managed server became unhealthy due to JVM heap memory exhaustion.

OACore logs showed repeated java.lang.OutOfMemoryError: GC overhead limit exceeded along with BEA-000405 Uncaught Throwable in processSockets.

Database validation showed JDBC sessions mostly inactive and waiting on SQL*Net message from client, classified under Idle wait class. This confirmed that the database had completed prior SQL execution and was waiting for the application tier.

The root cause was isolated to OACore JVM memory pressure, where excessive garbage collection prevented the JVM from recovering sufficient heap memory.

Immediate mitigation was to capture thread dumps, heap summary, GC statistics, and restart the affected OACore managed server.

Permanent corrective actions include reviewing JVM heap sizing, GC behavior, stuck threads, JDBC pool utilization, custom OAF memory usage, and enabling heap dump generation for future OutOfMemoryError analysis.

---

18. Conclusion

When OACore is unhealthy and the database shows idle JDBC sessions, do not immediately blame SQL or the database.

A correct Apps DBA investigation should correlate:

OACore logs
JVM heap usage
GC behavior
Thread dumps
JDBC sessions
Database wait events
WebLogic health

In this case, the real issue is not SQL tuning.

The real issue is:

OACore JVM memory exhaustion causing GC overhead limit exceeded.

 

Oracle EBS R12.2 Database Upgrade to Oracle AI Database 26ai — Apps DBA Runbook

1. Introduction

As an Oracle Apps DBA, database upgrade activity in Oracle E-Business Suite is a critical production operation. This runbook explains the step-by-step approach to upgrade an Oracle EBS R12.2 single-node database to Oracle AI Database 26ai.

This runbook is written in a practical Apps DBA style so that it can be used during planning, execution, validation, and post-upgrade support.

---

2. Scope of This Runbook

This runbook is applicable for:

Item	Details
Application	Oracle E-Business Suite R12.2
Database Upgrade Target	Oracle AI Database 26ai
Architecture	Single Node
Environment	On-premises or OCI Compute
Role	Oracle Apps DBA / EBS DBA
Activity Type	Database upgrade with EBS interoperability steps

---

3. High-Level Upgrade Flow

The complete upgrade activity can be divided into the following phases:

Phase	Activity
Phase 1	Certification and prerequisite validation
Phase 2	Install new 26ai Oracle Home
Phase 3	Apply required database patches
Phase 4	Prepare EBS appsutil and TNS configuration
Phase 5	Execute pre-upgrade checks
Phase 6	Upgrade database using AutoUpgrade
Phase 7	Perform post-upgrade EBS tasks
Phase 8	Run AutoConfig, ETCC, EDBPC, and validations
Phase 9	Start application services and release system to users

---

4. Pre-Upgrade Planning Checklist

Before touching production, prepare the below checklist.

4.1 Confirm Certification

First confirm that your EBS release, database version, operating system, and platform are certified for Oracle AI Database 26ai.

Check:

Validation	Status
EBS R12.2 certified with Oracle AI Database 26ai	Pending / Completed
OS certified	Pending / Completed
Required database patches identified	Pending / Completed
EBS interoperability notes reviewed	Pending / Completed
Backup strategy confirmed	Pending / Completed
Downtime window approved	Pending / Completed
Rollback plan documented	Pending / Completed

---

4.2 Take Mandatory Backups

Before starting the upgrade, take valid backups.

Recommended backups:

# Database backup using RMAN
rman target /

RMAN> backup database plus archivelog;
RMAN> backup current controlfile;
RMAN> backup spfile;

Also back up:

$ORACLE_HOME
$CONTEXT_FILE
tnsnames.ora
listener.ora
sqlnet.ora
init.ora/spfile
Application context files
EBS DBC files

---

5. Install Oracle AI Database 26ai Software

5.1 Create a New Oracle Home

The 26ai Oracle Home must be installed in a different directory from the existing 19c Oracle Home.

Example:

Old 19c Home : /u01/install/APPS/19.0.0
New 26ai Home: /u01/install/APPS/23.0.0

Do not overwrite the old Oracle Home.

---

5.2 Install Software Only

Run the Oracle Database 26ai installer.

Select:

Set Up Software Only
Enterprise Edition

Do not create a new database during installation.

---

5.3 Set Environment Variables

After installing 26ai software, verify the environment.

export ORACLE_BASE=/u01/install/APPS
export ORACLE_HOME=/u01/install/APPS/23.0.0
export PATH=$ORACLE_HOME/bin:$ORACLE_HOME/perl/bin:$PATH
export LD_LIBRARY_PATH=$ORACLE_HOME/lib
export PERL5LIB=$ORACLE_HOME/perl/lib:$ORACLE_HOME/perl/lib/site_perl

Validate:

echo $ORACLE_HOME
echo $ORACLE_BASE
echo $PATH
echo $LD_LIBRARY_PATH
echo $PERL5LIB

---

6. Apply Required 26ai Database Patches

Before upgrade, apply all mandatory Oracle EBS certified database patches for 26ai.

Run OPatch validation:

$ORACLE_HOME/OPatch/opatch version
$ORACLE_HOME/OPatch/opatch lsinventory

Apply patches as per the certified patch list.

After patching, confirm:

$ORACLE_HOME/OPatch/opatch lsinventory | grep -i patch

---

7. Create 9idata Directory

Run the below command from the new 26ai Oracle Home:

perl $ORACLE_HOME/nls/data/old/cr9idata.pl

Then set:

export ORA_NLS10=$ORACLE_HOME/nls/data/9idata

Validate:

echo $ORA_NLS10
ls -ld $ORACLE_HOME/nls/data/9idata

---

8. Create appsutil.zip and Copy to DB Tier

From the application tier, create appsutil.zip and copy it to the database tier.

Typical approach:

cd $AD_TOP/bin
perl admkappsutil.pl

Copy the generated file to the database server and unzip under the new Oracle Home:

cd $ORACLE_HOME
unzip appsutil.zip

Validate:

ls -ld $ORACLE_HOME/appsutil
ls -l $ORACLE_HOME/appsutil/bin

---

9. Generate CDB TNS Files

On the database server, run:

cd $ORACLE_HOME/appsutil
. ./txkSetCfgCDB.env dboraclehome=$ORACLE_HOME

cd $ORACLE_HOME/appsutil/bin

perl txkGenCDBTnsAdmin.pl \
-dboraclehome=$ORACLE_HOME \
-cdbname=<CDB_SID> \
-cdbsid=<CDB_SID> \
-dbport=<DB_PORT> \
-outdir=$ORACLE_HOME/appsutil/log

Example:

perl txkGenCDBTnsAdmin.pl \
-dboraclehome=/u01/install/APPS/23.0.0 \
-cdbname=EBSCDB \
-cdbsid=EBSCDB \
-dbport=1521 \
-outdir=/u01/install/APPS/23.0.0/appsutil/log

Verify sqlnet.ora contains:

SQLNET.ALLOWED_LOGON_VERSION_SERVER=12

Command:

grep SQLNET.ALLOWED_LOGON_VERSION_SERVER $ORACLE_HOME/network/admin/sqlnet.ora

---

10. Update UTL_FILE_DIR Directory Object Values

Before the upgrade, review and update UTL_FILE_DIR related directory object values.

Check current value:

show parameter utl_file_dir

or:

select name, value
from v$parameter
where name = 'utl_file_dir';

Also identify EBS temporary directories required for application processing.

---

11. Shutdown Application Tier

Before database upgrade, stop all EBS application services.

On application tier:

cd $ADMIN_SCRIPTS_HOME
./adstpall.sh apps/<apps_password>

Validate:

ps -ef | grep FNDLIBR
ps -ef | grep oacore
ps -ef | grep forms
ps -ef | grep apache

Expected result: no active EBS application services.

---

12. Shutdown Old Database Listener

On database tier, using old 19c Oracle Home:

lsnrctl status
lsnrctl stop <listener_name>

Also check LOCAL_LISTENER:

show parameter local_listener

If it points to an old or invalid listener during upgrade, review and correct it before proceeding.

---

13. Prepare AutoUpgrade Configuration File

Create an AutoUpgrade config file.

Example file:

vi /home/oracle/ebs26ai_upgrade.cfg

Sample content:

global.autoupg_log_dir=/u01/install/APPS/23.0.0/cfgtoollogs/autoupgrade/ebscdb

autoupgrade1.log_dir=/u01/install/APPS/23.0.0/cfgtoollogs/autoupgrade/ebscdb
autoupgrade1.sid=EBSCDB
autoupgrade1.source_home=/u01/install/APPS/19.0.0
autoupgrade1.target_home=/u01/install/APPS/23.0.0
autoupgrade1.start_time=NOW
autoupgrade1.upgrade_node=<hostname>
autoupgrade1.pdbs=EBSPDB
autoupgrade1.run_utlrp=yes
autoupgrade1.timezone_upg=yes
autoupgrade1.target_version=23
autoupgrade1.restoration=no

autoupgrade1.add_during_upgrade_pfile=/home/oracle/add_during_upgrade_pfile_ebscdb.ora
autoupgrade1.add_after_upgrade_pfile=/home/oracle/add_after_upgrade_pfile_ebscdb.ora
autoupgrade1.del_during_upgrade_pfile=/home/oracle/del_during_upgrade_pfile_ebscdb.ora
autoupgrade1.del_after_upgrade_pfile=/home/oracle/del_after_upgrade_pfile_ebscdb.ora

---

14. Prepare Parameter Add/Delete Files

Create add and delete parameter files as required.

Example add file:

vi /home/oracle/add_during_upgrade_pfile_ebscdb.ora

Example content:

event='10946 trace name context forever, level 8388608'
permit_92_wrap_format=true

Example delete file:

vi /home/oracle/del_during_upgrade_pfile_ebscdb.ora

Example content:

db_cache_size
sec_case_sensitive_logon
undo_retention
user_dump_dest
utl_file_dir

Important Apps DBA note:

Do not blindly copy parameters. Validate them against your environment and Oracle EBS recommended database initialization parameter document.

---

15. Run AutoUpgrade Analyze Mode

Set new Oracle Home:

export ORACLE_HOME=/u01/install/APPS/23.0.0
export PATH=$ORACLE_HOME/bin:$ORACLE_HOME/jdk/bin:$PATH

Run analyze:

$ORACLE_HOME/jdk/bin/java -jar /home/oracle/autoupgrade.jar \
-config /home/oracle/ebs26ai_upgrade.cfg \
-mode analyze

Check logs:

cd /u01/install/APPS/23.0.0/cfgtoollogs/autoupgrade/ebscdb
ls -ltr

Review errors and warnings carefully.

---

16. Run AutoUpgrade Fixups

After analyze mode, run fixups:

$ORACLE_HOME/jdk/bin/java -jar /home/oracle/autoupgrade.jar \
-config /home/oracle/ebs26ai_upgrade.cfg \
-mode fixups

Validate fixup logs:

find /u01/install/APPS/23.0.0/cfgtoollogs/autoupgrade/ebscdb -name "*.log" -mtime -1

Do not proceed to deploy until critical issues are resolved.

---

17. Run AutoUpgrade Deploy Mode

Once analyze and fixups are clean, start upgrade:

$ORACLE_HOME/jdk/bin/java -jar /home/oracle/autoupgrade.jar \
-config /home/oracle/ebs26ai_upgrade.cfg \
-mode deploy

Monitor upgrade:

tail -f <autoupgrade_log_file>

Check database status:

select name, open_mode, database_role from v$database;

select con_id, name, open_mode
from v$pdbs;

---

18. Set Compatible Parameter After Testing

Oracle recommends changing compatible only after upgrade testing is completed.

Check current value:

show parameter compatible

Set only after approval:

alter system set compatible='<recommended_version>' scope=spfile;

Restart database:

shutdown immediate;
startup;

---

19. Start Listener from 26ai Oracle Home

Generate PDB TNS admin files:

cd $ORACLE_HOME/appsutil
. ./txkSetCfgCDB.env dboraclehome=$ORACLE_HOME

export ORACLE_SID=<CDB_SID>

cd $ORACLE_HOME/appsutil/bin

perl txkGenPDBTnsAdmin.pl \
-dboraclehome=$ORACLE_HOME \
-cdbname=<CDB_SID> \
-cdbsid=<CDB_SID> \
-pdbsid=<PDB_SID> \
-dbport=<DB_PORT> \
-outdir=$ORACLE_HOME/appsutil/log

Set PDB environment:

export ORACLE_PDB_SID=<PDB_SID>
export TNS_ADMIN=$ORACLE_HOME/network/admin/<PDB_SID>_<hostname>

Start listener:

lsnrctl start <CDB_SID>
lsnrctl status <CDB_SID>

---

20. Run adrevoke.sql and Grant EBS Privileges

Copy adrevoke.sql from application tier to database tier:

$AD_TOP/patch/115/sql/adrevoke.sql

Connect to PDB as SYSDBA:

sqlplus "/ as sysdba" @adrevoke.sql APPS

Then perform EBS database privilege grants as per Oracle EBS privilege management procedure.

---

21. Compile Invalid Objects

Run utlrp.sql using catcon.pl:

unset ORACLE_PDB_SID

cd $ORACLE_HOME/rdbms/admin

$ORACLE_HOME/perl/bin/perl catcon.pl --n 1 --e --b utlrp --d '''.''' utlrp.sql

Check invalid objects:

select owner, count(*)
from dba_objects
where status='INVALID'
group by owner
order by count(*) desc;

For APPS invalids:

select object_name, object_type, status
from dba_objects
where owner='APPS'
and status='INVALID'
order by object_type, object_name;

---

22. Gather SYS Schema Statistics

Copy adstats.sql from application tier:

$APPL_TOP/admin/adstats.sql

Run in restricted mode:

sqlplus "/ as sysdba"

alter system enable restricted session;

@adstats.sql

alter system disable restricted session;

exit;

Validate:

select owner, table_name, last_analyzed
from dba_tables
where owner='SYS'
and rownum <= 10;

---

23. Generate New Database Context File and Run AutoConfig

Run post PDB creation tasks:

cd $ORACLE_HOME/appsutil
. ./txkSetCfgCDB.env dboraclehome=$ORACLE_HOME

perl $ORACLE_HOME/appsutil/bin/txkPostPDBCreationTasks.pl \
-dboraclehome=$ORACLE_HOME \
-outdir=$ORACLE_HOME/appsutil/log \
-cdbsid=<CDB_SID> \
-pdbsid=<PDB_SID> \
-appsuser=apps \
-dbport=<DB_PORT> \
-servicetype=onpremise \
-dbuniquename=<DB_UNIQUE_NAME>

Check logs:

ls -ltr $ORACLE_HOME/appsutil/log

---

24. Update Application Tier Context File

On each application tier node, update these context variables:

Context Variable	Required Value
s_dbport	New database port
s_apps_jdbc_connect_descriptor	NULL / blank
s_applptmp	Valid directory defined for EBS temporary file usage

Check allowed temporary directory:

select value
from v$parameter
where name='utl_file_dir';

---

25. Validate TWO_TASK Entry

Check $TNS_ADMIN/tnsnames.ora on both run and patch file systems.

Expected format:

<TWO_TASK> =
 (DESCRIPTION =
  (ADDRESS = (PROTOCOL=tcp)(HOST=<hostname.domain>)(PORT=<port>))
  (CONNECT_DATA =
    (SERVICE_NAME=ebs_<PDB_SID>)
    (INSTANCE_NAME=<CDB_SID>)
  )
 )

Test connection:

tnsping <TWO_TASK>
sqlplus apps/<apps_password>@<TWO_TASK>

---

26. Run AutoConfig on Application Tier

Run AutoConfig on both run and patch file systems.

Source run file system:

. EBSapps.env run
cd $INST_TOP/admin/scripts
./adautocfg.sh

Source patch file system:

. EBSapps.env patch
cd $INST_TOP/admin/scripts
./adautocfg.sh

Apps DBA note:

Errors on patch file system AutoConfig may be ignorable in some cases, but always review the log before closing the task.

---

27. Run ETCC on Database Tier

Run ETCC to verify required database patches.

cd <ETCC_LOCATION>
./checkDBpatch.sh

Review output:

All required one-off bug fixes are present.

If missing patches are reported, apply them before releasing the environment.

---

28. Run EBS Database Parameter Checker

Download latest EDBPC utility patch and run the database parameter checker.

Purpose:

Check	Reason
Missing parameters	Identify required EBS parameters
Incorrect values	Correct non-compliant settings
Deprecated parameters	Remove unsupported values
Sizing parameters	Validate recommended configuration

Review the generated report and correct all mandatory findings.

---

29. Regenerate JAR Files

Run adadmin from the run file system.

. EBSapps.env run
adadmin

Select:

Generate Applications Files
Generate product JAR files

Then select:

Compile/Reload Applications Database Entities
Reload JAR files to database

---

30. Implement Network ACLs

For Oracle AI Database 26ai, implement required Network ACLs for EBS database network access.

Typical validation:

select host, lower_port, upper_port, acl
from dba_network_acls;

select acl, principal, privilege, is_grant
from dba_network_acl_privileges;

Confirm required EBS schemas have appropriate network privileges.

---

31. Conditional Post-Upgrade Tasks

Depending on your environment, perform these additional tasks.

Component	Required Action
Database Vault	Disable before upgrade, enable after upgrade if used
TLS	Reconfigure database tier TLS setup
ISG	Perform ISG post-upgrade tasks
OID/OUD	Update DIP provisioning profile
APEX	Upgrade APEX if installed
External Integrations	Reconfigure with updated DBC file
ECC	Reconfigure JNDI / DBC connectivity if applicable
Encrypted DB	Review wallet and sqlnet entries

---

32. Set INACTIVE_ACCOUNT_TIME to Unlimited

Connect to PDB as SYSDBA:

export ORACLE_PDB_SID=<PDB_SID>
sqlplus "/ as sysdba"

Run:

alter profile default limit INACTIVE_ACCOUNT_TIME unlimited;

Validate:

select profile, resource_name, limit
from dba_profiles
where resource_name='INACTIVE_ACCOUNT_TIME';

---

33. Start Application Services

Start EBS application tier services:

. EBSapps.env run

cd $ADMIN_SCRIPTS_HOME

./adstrtal.sh apps/<apps_password>

Validate services:

./adopmnctl.sh status

Check:

ps -ef | grep FNDLIBR
ps -ef | grep oacore
ps -ef | grep forms
ps -ef | grep apache

---

34. Post-Upgrade Validation Checklist

34.1 Database Validation

select name, open_mode from v$database;

select con_id, name, open_mode from v$pdbs;

select banner_full from v$version;

---

34.2 Invalid Object Validation

select owner, count(*)
from dba_objects
where status='INVALID'
group by owner
order by count(*) desc;

---

34.3 EBS Login Validation

Validate:

Test	Status
EBS login page opens	Pass / Fail
User login successful	Pass / Fail
Forms launch successful	Pass / Fail
OAF page opens	Pass / Fail
Concurrent manager starts	Pass / Fail
Sample concurrent request completes	Pass / Fail
Workflow mailer status checked	Pass / Fail
Integration connectivity validated	Pass / Fail

---

34.4 Concurrent Manager Validation

select concurrent_queue_name, running_processes, max_processes
from apps.fnd_concurrent_queues
where enabled_flag='Y';

Check active requests:

select request_id, phase_code, status_code, actual_start_date, actual_completion_date
from apps.fnd_concurrent_requests
where requested_start_date > sysdate - 1
order by request_id desc;

---

35. Common Issues and Troubleshooting

Issue 1: Application Login Fails After Upgrade

Check:

tnsping <TWO_TASK>
sqlplus apps/<password>@<TWO_TASK>

Validate:

echo $TNS_ADMIN
cat $TNS_ADMIN/tnsnames.ora

Possible causes:

Cause	Action
Wrong TNS entry	Correct tnsnames.ora
AutoConfig not run	Run AutoConfig
Listener not started	Start listener
Wrong service name	Correct service name as ebs_<PDB_SID>

---

Issue 2: Invalid Objects Remain

Run:

select owner, object_type, count(*)
from dba_objects
where status='INVALID'
group by owner, object_type
order by count(*) desc;

Recompile:

exec utl_recomp.recomp_parallel(8);

---

Issue 3: Concurrent Managers Not Starting

Check internal manager:

select concurrent_queue_name, running_processes, max_processes
from apps.fnd_concurrent_queues
where concurrent_queue_name='FNDICM';

Check logs:

cd $APPLCSF/$APPLLOG
ls -ltr

---

Issue 4: Forms Not Opening

Check:

echo $FORMS_WEB_CONFIG_FILE
echo $CONTEXT_FILE

Bounce services if required:

adstpall.sh apps/<password>
adstrtal.sh apps/<password>

---

36. Final Production Handover Checklist

Item	Status
Database upgraded successfully	Completed
Listener running from 26ai Oracle Home	Completed
PDB open read write	Completed
AutoConfig completed on DB tier	Completed
AutoConfig completed on app tier run FS	Completed
AutoConfig completed on app tier patch FS	Completed
ETCC completed	Completed
EDBPC completed	Completed
Invalid objects reviewed	Completed
SYS stats gathered	Completed
JAR files regenerated	Completed
EBS login tested	Completed
Concurrent manager tested	Completed
Business smoke testing completed	Completed
Monitoring enabled	Completed
Backup after upgrade completed	Completed

---

37. Apps DBA Best Practices

1. Never upgrade directly without certification validation.
2. Always install 26ai in a new Oracle Home.
3. Keep old Oracle Home untouched until upgrade is fully signed off.
4. Take complete RMAN and configuration backups.
5. Run AutoUpgrade analyze and fixups before deploy.
6. Review every warning in AutoUpgrade logs.
7. Run ETCC after upgrade.
8. Run EDBPC to validate database parameters.
9. Always test EBS login, Forms, OAF pages, concurrent managers, workflow, and integrations.
10. Keep rollback plan ready until business sign-off is received.

---

38. Conclusion

Upgrading Oracle EBS R12.2 database to Oracle AI Database 26ai is not only a database upgrade activity. It is a complete EBS technology stack activity involving database home installation, EBS appsutil configuration, TNS generation, AutoUpgrade execution, AutoConfig, ETCC, EDBPC, invalid object compilation, JAR regeneration, and full application validation.

For an Apps DBA, the success of this activity depends on careful planning, clean execution, strong validation, and proper communication with business and application teams