Apps DBA
EBS 12.2.x
Monitoring
Fresher + Experienced
🩺 Oracle EBS 12.2 — The Daily Morning Health Check
Every Apps DBA Must Run (Fresher to L4)
Environment: Oracle EBS 12.2.x · Oracle DB 19c · Solaris 11.4 SPARC · RAC · WebLogic 12c
Audience: Freshers joining Apps DBA teams + Experienced DBAs building a structured morning routine
Time to execute: 20–30 minutes · Do this before the first business user logs in
If you are a fresher stepping into an Oracle Apps DBA role for the first time, the first question you will face every morning is: "Is the environment healthy?" If you are an experienced DBA, you probably run a mental checklist already — but is it consistent, documented, and defensible if something goes wrong at 9 AM?
This post gives you a structured, repeatable morning health check — with the exact commands, SQL scripts, and warning signs — covering all critical layers of Oracle EBS 12.2: Database, Application tier, Concurrent Manager, WebLogic, RMAN, and OS-level checks on Solaris.
Bookmark it. Run it every morning. Your on-call stress will drop significantly.
📋 What This Post Covers
- Why a structured health check matters (and what happens without one)
- Layer 1 — Database tier: Alert log, tablespace, sessions
- Layer 2 — Application tier: EBS services, autoconfig, fs1/fs2
- Layer 3 — Concurrent Manager and batch jobs
- Layer 4 — WebLogic / AdminServer health
- Layer 5 — RMAN backup status
- Layer 6 — OS and Solaris checks
- The complete morning checklist (print-ready)
- Quick-reference scripts in one place
1. Why This Matters — The Cost of No Morning Check
Consider this scenario: a DBA starts their shift at 8 AM. No health check. At 10 AM, a user calls — EBS login is broken. Investigation reveals the OHS process crashed at 6:45 AM. The concurrent manager has been rejecting jobs since 7:00 AM. RMAN backup failed last night but nobody noticed.
A 20-minute morning health check at 8 AM catches all three of these before the business starts. It also gives you a baseline snapshot — so if something breaks at noon, you know exactly what changed since morning.
💡 For Freshers: Your manager will ask "did you check the environment this morning?" A documented health check means your answer is always "yes — and here is what I found." This single habit differentiates a junior DBA from a reliable one within your first 90 days.
⚠️ For Experienced DBAs: If your morning check is informal and in your head, it is not repeatable by your team across APAC / EMEA / Americas shifts. Document it, standardise it, share it. That is the difference between individual knowledge and team resilience.
2. Layer 1 — Database Tier
2.1 — Check the DB Alert Log (First Priority, Always)
The DB alert log is the single most important file in your EBS environment. Check it before anything else. Any ORA- error here is a potential incident.
🎓 Fresher Explanation
The alert log is a running diary that Oracle writes for every database event — startup, shutdown, errors, checkpoints, log switches. It lives on the DB server (not the app server). Your first job every morning is to read the last 200 lines and look for anything that starts with ORA-.
# Find the alert log path (Solaris)
find $ORACLE_BASE/diag/rdbms -name "alert_*.log" 2>/dev/null
# Tail the last 200 lines
tail -200 $ORACLE_BASE/diag/rdbms/<DBNAME>/<SID>/trace/alert_<SID>.log
# Filter for ORA- errors only (the quick check)
grep "ORA-" $ORACLE_BASE/diag/rdbms/<DBNAME>/<SID>/trace/alert_<SID>.log | tail -30
# Check for errors from last 24 hours only
awk '/'"$(date -d 'yesterday' '+%a %b')"'/,0' alert_<SID>.log | grep "ORA-"
| ORA- Error | What It Means | Priority |
ORA-00600 | Internal Oracle error — raise SR immediately | 🔴 P1 |
ORA-07445 | Exception encountered — core dump likely | 🔴 P1 |
ORA-04031 | Shared pool / SGA memory insufficient | 🔴 P1 |
ORA-01555 | Snapshot too old — UNDO retention issue | 🟡 P2 |
ORA-00060 | Deadlock detected — check application logic | 🟡 P2 |
ORA-01652 | TEMP tablespace full | 🟡 P2 |
ORA-00257 | Archiver error — archive log destination full | 🔴 P1 |
2.2 — Tablespace Usage
A tablespace hitting 90%+ before business hours starts is a disaster waiting to happen. EBS SYSTEM, APPS_TS_TX_DATA, APPS_TS_SEED, and UNDOTBS are the critical ones.
-- Tablespace usage — flag anything above 85%
SELECT
df.tablespace_name,
ROUND(df.total_mb) total_mb,
ROUND(df.total_mb - NVL(fs.free_mb, 0)) used_mb,
ROUND((1 - NVL(fs.free_mb, 0) / df.total_mb) * 100, 1) pct_used,
df.autoextend
FROM
(SELECT tablespace_name,
SUM(bytes) / 1048576 total_mb,
MAX(CASE autoextensible WHEN 'YES' THEN 'YES' ELSE 'NO' END) autoextend
FROM dba_data_files
GROUP BY tablespace_name) df,
(SELECT tablespace_name, SUM(bytes) / 1048576 free_mb
FROM dba_free_space
GROUP BY tablespace_name) fs
WHERE df.tablespace_name = fs.tablespace_name (+)
AND (1 - NVL(fs.free_mb, 0) / df.total_mb) * 100 > 75
ORDER BY pct_used DESC;
⚡ Experienced DBA Note
On EBS 12.2 with online patching, APPS_UNDOTS1 can spike dramatically during ADOP apply phase. If you see it above 80% on a patch day morning — proactively extend before the patch window opens, not during it.
2.3 — Active Session Count and Long-Running Queries
-- Session overview: active vs inactive
SELECT inst_id, status, COUNT(*) sessions
FROM gv$session
WHERE type = 'USER'
GROUP BY inst_id, status
ORDER BY inst_id, status;
-- Long-running queries (over 10 minutes)
SELECT
s.inst_id, s.sid, s.serial#, s.username,
s.module, s.status,
ROUND(s.last_call_et / 60, 1) mins_running,
SUBSTR(q.sql_text, 1, 60) sql_text
FROM gv$session s, gv$sql q
WHERE s.sql_id = q.sql_id
AND s.status = 'ACTIVE'
AND s.last_call_et > 600
AND s.type = 'USER'
ORDER BY s.last_call_et DESC;
3. Layer 2 — Application Tier: EBS Services
3.1 — Are All EBS Services Running?
EBS 12.2 has a specific service start order. If any service is down when users arrive, you will face immediate calls. Check this from the application tier server as the applmgr user.
🎓 Fresher Explanation
Oracle EBS 12.2 runs as multiple services layered on top of each other: OHS (web entry point) → WebLogic (Java tier) → Database. If OHS is down, users see a browser error. If WebLogic is down, they see an EBS error page. If the Concurrent Manager is down, batch jobs stop silently. You need to check all three every morning.
# Check all EBS services status (run as applmgr)
source /path/to/<CONTEXT_NAME>.env
$ADMIN_SCRIPTS_HOME/adstatus.sh
# Check individual service status
$ADMIN_SCRIPTS_HOME/adopmnctl.sh status # OHS / Oracle HTTP Server
$ADMIN_SCRIPTS_HOME/adadminsrvctl.sh status # WebLogic AdminServer
$ADMIN_SCRIPTS_HOME/admanagedsrvctl.sh status # WLS Managed Servers
$ADMIN_SCRIPTS_HOME/adcmctl.sh status # Concurrent Manager
# Quick process check on Solaris
ps -ef | grep -E "httpd|java|FNDLIBR|FNDSM" | grep -v grep
3.2 — Verify the Correct Filesystem is Active (fs1 / fs2)
This is EBS 12.2 specific and catches more DBAs off guard than anything else. After ADOP patching, the run edition switches between fs1 and fs2. Always confirm which is active.
# Check which filesystem is the current run edition
cat $EBS_DOMAIN_HOME/applications/EBSapps_instance/controlfile/adinfo_*.xml | \
grep -i "run_edition_home"
# Or directly from the context file
grep "s_current_base" $CONTEXT_FILE
⚠️ Watch out for this: If ADOP cleanup did not complete after the last patch cycle, both fs1 and fs2 may show as partially configured. Run adop phase=status to confirm the current ADOP state before assuming the environment is clean.
4. Layer 3 — Concurrent Manager (CM)
The Concurrent Manager is the heartbeat of EBS batch processing. If it is not running, or if requests are piling up in Pending/Standby status, business processes stop silently — and users notice hours later.
4.1 — CM Manager Status
-- Check all manager statuses
SELECT
c.concurrent_queue_name manager_name,
c.user_concurrent_queue_name display_name,
c.enabled_flag,
r.running_processes,
r.max_processes,
r.node_name
FROM fnd_concurrent_queues_vl c,
fnd_concurrent_queue_size r
WHERE c.concurrent_queue_id = r.concurrent_queue_id (+)
AND c.application_id = r.application_id (+)
ORDER BY c.concurrent_queue_name;
4.2 — Pending and Stuck Requests (The Most Common Morning Issue)
-- Count of requests by phase and status
SELECT
phase_code phase,
status_code status,
COUNT(*) cnt
FROM fnd_concurrent_requests
WHERE phase_code IN ('P', 'R') -- Pending and Running
GROUP BY phase_code, status_code
ORDER BY phase_code, status_code;
-- Requests stuck Pending for more than 30 minutes
SELECT
r.request_id,
r.concurrent_program_name program,
p.user_concurrent_program_name display_name,
r.requested_start_date,
ROUND((SYSDATE - r.requested_start_date) * 1440) mins_pending,
r.phase_code, r.status_code
FROM fnd_concurrent_requests r,
fnd_concurrent_programs_vl p
WHERE r.concurrent_program_id = p.concurrent_program_id (+)
AND r.phase_code = 'P'
AND r.status_code = 'I' -- Normal (Pending/Normal = stuck)
AND (SYSDATE - r.requested_start_date) * 1440 > 30
ORDER BY r.requested_start_date;
⚡ Experienced DBA Note
If you see a large number of Pending/Standby requests with no active ICM (Internal Concurrent Manager) process, the Conflict Resolution Manager has likely stalled. Check the ICM log at $APPLCSF/$APPLLOG/ for FNDLIBR errors before bouncing the CM. A blind bounce without reading the ICM log will not fix the root cause.
5. Layer 4 — WebLogic / AdminServer Health
WebLogic AdminServer memory exhaustion is one of the most common EBS 12.2 production issues — especially in environments where the heap size was not tuned after initial installation. A quick morning check takes 60 seconds and prevents emergency 2 AM restarts.
5.1 — AdminServer Process and Memory
# Check AdminServer JVM heap usage (Solaris)
ps -ef | grep AdminServer | grep -v grep | \
awk '{print "PID:", $2, "RSS(MB):", $6/1024}'
# Check WLS process and port (AdminServer default: 7001)
netstat -an | grep 7001
# Tail the AdminServer log for recent errors
tail -100 $DOMAIN_HOME/servers/AdminServer/logs/AdminServer.log | \
grep -iE "error|exception|OutOfMemory|BEA-"
5.2 — JDBC Connection Pool Check (via WLST)
# Quick WLST snippet to check JDBC pool health
cd $FMW_HOME/wlserver/common/bin
./wlst.sh <<EOF
connect('weblogic','<password>','t3://<host>:7001')
serverRuntime()
cd('JDBCServiceRuntime/<server_name>/JDBCDataSourceRuntimeMBeans')
ls()
EOF
💡 If AdminServer RSS memory (from ps) is above 3 GB and climbing steadily every morning, your heap settings need tuning. Standard recommendation for EBS 12.2: set -Xms2048m -Xmx4096m in setDomainEnv.sh and schedule a monthly AdminServer restart in a low-traffic window.
6. Layer 5 — RMAN Backup Status
If last night's backup failed and you did not catch it this morning, you are running production EBS without a current backup. That is an unacceptable risk. This check takes 2 minutes.
-- Check last backup status (last 24 hours)
SELECT
session_key,
input_type,
status,
TO_CHAR(start_time, 'DD-MON-YYYY HH24:MI') start_time,
TO_CHAR(end_time, 'DD-MON-YYYY HH24:MI') end_time,
ROUND(elapsed_seconds / 60) elapsed_mins,
ROUND(output_bytes / 1073741824, 2) output_gb
FROM v$rman_backup_job_details
WHERE start_time > SYSDATE - 1
ORDER BY start_time DESC;
-- Archive log gap check (critical for standby or recovery)
SELECT
sequence#, first_time, next_time, applied, deleted
FROM v$archived_log
WHERE first_time > SYSDATE - 1
AND standby_dest = 'NO'
ORDER BY sequence#;
| RMAN Status | Meaning | Action |
COMPLETED | Backup successful | No action needed ✅ |
COMPLETED WITH WARNINGS | Backup ran but some files skipped | Check output — usually safe, but investigate |
FAILED | Backup did not complete | Check RMAN log immediately, re-run or escalate |
| No rows returned | Backup never ran or scheduler issue | Check RMAN script cron job — treat as FAILED |
7. Layer 6 — OS-Level Checks (Solaris 11.4 SPARC)
7.1 — Disk and Mount Point Health
# Disk usage — flag anything above 85%
df -h | awk 'NR==1 || $5+0 > 85'
# Check NFS mounts are still alive (critical in EBS)
mount | grep nfs
df -k | grep -v Filesystem | awk '$6 ~ /nfs/'
# Check for zombie or stuck processes
ps -ef | awk '$8 == "Z"'
# Archive log destination — must not be above 80%
df -h /u03/archivelogs # adjust path to your arch dest
7.2 — System Load and Memory
# Current system load average (1, 5, 15 min)
uptime
# Memory usage on Solaris
prtconf | grep "Memory size"
vmstat 1 3 # watch 'si/so' for swapping — must be 0 on healthy system
# Check for any core dumps from last 24 hours
find /var/core -name core -newer /tmp -ls 2>/dev/null
⚠️ Solaris-specific: On SPARC systems, vmstat columns differ from Linux. Watch the sr (scan rate) column — if it is consistently above 200 and pi/po (page in/out) is non-zero, the system is under memory pressure. For EBS, this will show up as slow Forms loading and WebLogic response delays before it becomes obvious to users.
8. The Complete Morning Checklist (Print-Ready)
Use this as your daily sign-off. Document your results in your shift handover notes every morning.
- 📋 DB Alert Log — No new ORA- errors since last check
- 💾 Tablespace Usage — All critical tablespaces below 85%
- 🔒 UNDO / TEMP — Not exhausted, autoextend functioning
- 🔗 Archive Log Destination — Below 80%, no gaps in sequence
- 👥 Active Sessions — No unexpected spike; no long-runners (>10 min)
- 🌐 EBS Services — OHS, AdminServer, Managed Servers all UP
- 📂 fs1/fs2 Edition — Run edition confirmed; no dangling ADOP phase
- ⚙️ Concurrent Manager — ICM running; no Pending/Standby queue buildup
- ☕ WebLogic AdminServer — No OutOfMemoryError in last 24 hrs
- 💿 RMAN Backup — Last backup status: COMPLETED
- 🖥️ Disk Space (OS) — No filesystem above 85%
- 🔗 NFS Mounts — All EBS-related NFS mounts alive
- 📊 System Load — Load average within normal range; no swapping
✅ You are done. If all 13 checks are green, the environment is healthy and you can start your shift with confidence. Document your check in the shift handover — even two lines is enough: "Health check complete 08:15. All services up. RMAN backup completed. No ORA- errors. One tablespace at 83% — monitoring."
9. One Shell Script to Run Everything
Here is a lightweight shell script that combines the OS-level checks into a single morning summary. Run it as applmgr from the app tier. Add DB checks to cron separately (requires DB connectivity).
#!/bin/ksh
# ebs_morning_check.sh
# Run as: applmgr user on application tier
# Oracle EBS 12.2.x — Morning Health Summary
source /path/to/<CONTEXT_NAME>.env
echo "═══════════════════════════════════════"
echo " EBS MORNING HEALTH CHECK — $(date '+%d-%b-%Y %H:%M')"
echo "═══════════════════════════════════════"
echo "\n📂 DISK USAGE (>70% shown):"
df -h | awk 'NR==1 || ($5+0) > 70 {print}'
echo "\n🔗 NFS MOUNTS:"
mount | grep nfs | awk '{print " ✓", $1, "→", $3}'
echo "\n⚙️ EBS SERVICES:"
$ADMIN_SCRIPTS_HOME/adstatus.sh 2>/dev/null | grep -iE "running|stopped|dead"
echo "\n📊 SYSTEM LOAD:"
uptime
echo "\n🔍 ALERT LOG - Last ORA- errors:"
grep "ORA-" $ORACLE_BASE/diag/rdbms/*/*/trace/alert_*.log 2>/dev/null | \
tail -5
echo "\n═══════════════════════════════════════"
echo " Check complete. Run SQL checks from DB server."
echo "═══════════════════════════════════════"
10. Summary
A structured morning health check is not optional in a 24×7 EBS production support role — it is professional table stakes. For freshers, it gives you a daily framework that builds environmental awareness faster than anything else. For experienced DBAs, documenting it makes your knowledge reproducible across shifts and geography.
Run it consistently. Document your findings. Your 9 AM incidents will become 8 AM catches — and your manager will notice the difference before any appraisal conversation starts.
💡 Next in this series: Oracle EBS 12.2 ADOP Phase Failures — How to Diagnose and Recover Without a Full Rollback · Oracle Apps DBA — Freshers' Guide to Your First Clone Operation
Tags:
Oracle EBS
Apps DBA
EBS 12.2.x
Health Check
Monitoring
Concurrent Manager
WebLogic
RMAN
Solaris
Fresher Guide
Shell Script
SQL
ADOP
RAC