Apps DBA EBS 12.2.x Monitoring Fresher + Experienced
š©ŗ Oracle EBS 12.2 — The Daily Morning Health Check
Every Apps DBA Must Run (Fresher to L4)
If you are a fresher stepping into an Oracle Apps DBA role for the first time, the first question you will face every morning is: "Is the environment healthy?" If you are an experienced DBA, you probably run a mental checklist already — but is it consistent, documented, and defensible if something goes wrong at 9 AM?
This post gives you a structured, repeatable morning health check — with the exact commands, SQL scripts, and warning signs — covering all critical layers of Oracle EBS 12.2: Database, Application tier, Concurrent Manager, WebLogic, RMAN, and OS-level checks on Solaris.
Bookmark it. Run it every morning. Your on-call stress will drop significantly.
- Why a structured health check matters (and what happens without one)
- Layer 1 — Database tier: Alert log, tablespace, sessions
- Layer 2 — Application tier: EBS services, autoconfig, fs1/fs2
- Layer 3 — Concurrent Manager and batch jobs
- Layer 4 — WebLogic / AdminServer health
- Layer 5 — RMAN backup status
- Layer 6 — OS and Solaris checks
- The complete morning checklist (print-ready)
- Quick-reference scripts in one place
1. Why This Matters — The Cost of No Morning Check
Consider this scenario: a DBA starts their shift at 8 AM. No health check. At 10 AM, a user calls — EBS login is broken. Investigation reveals the OHS process crashed at 6:45 AM. The concurrent manager has been rejecting jobs since 7:00 AM. RMAN backup failed last night but nobody noticed.
A 20-minute morning health check at 8 AM catches all three of these before the business starts. It also gives you a baseline snapshot — so if something breaks at noon, you know exactly what changed since morning.
2. Layer 1 — Database Tier
2.1 — Check the DB Alert Log (First Priority, Always)
The DB alert log is the single most important file in your EBS environment. Check it before anything else. Any ORA- error here is a potential incident.
The alert log is a running diary that Oracle writes for every database event — startup, shutdown, errors, checkpoints, log switches. It lives on the DB server (not the app server). Your first job every morning is to read the last 200 lines and look for anything that starts with ORA-.
# Find the alert log path (Solaris) find $ORACLE_BASE/diag/rdbms -name "alert_*.log" 2>/dev/null # Tail the last 200 lines tail -200 $ORACLE_BASE/diag/rdbms/<DBNAME>/<SID>/trace/alert_<SID>.log # Filter for ORA- errors only (the quick check) grep "ORA-" $ORACLE_BASE/diag/rdbms/<DBNAME>/<SID>/trace/alert_<SID>.log | tail -30 # Check for errors from last 24 hours only awk '/'"$(date -d 'yesterday' '+%a %b')"'/,0' alert_<SID>.log | grep "ORA-"
| ORA- Error | What It Means | Priority |
|---|---|---|
ORA-00600 | Internal Oracle error — raise SR immediately | š“ P1 |
ORA-07445 | Exception encountered — core dump likely | š“ P1 |
ORA-04031 | Shared pool / SGA memory insufficient | š“ P1 |
ORA-01555 | Snapshot too old — UNDO retention issue | š” P2 |
ORA-00060 | Deadlock detected — check application logic | š” P2 |
ORA-01652 | TEMP tablespace full | š” P2 |
ORA-00257 | Archiver error — archive log destination full | š“ P1 |
2.2 — Tablespace Usage
A tablespace hitting 90%+ before business hours starts is a disaster waiting to happen. EBS SYSTEM, APPS_TS_TX_DATA, APPS_TS_SEED, and UNDOTBS are the critical ones.
-- Tablespace usage — flag anything above 85% SELECT df.tablespace_name, ROUND(df.total_mb) total_mb, ROUND(df.total_mb - NVL(fs.free_mb, 0)) used_mb, ROUND((1 - NVL(fs.free_mb, 0) / df.total_mb) * 100, 1) pct_used, df.autoextend FROM (SELECT tablespace_name, SUM(bytes) / 1048576 total_mb, MAX(CASE autoextensible WHEN 'YES' THEN 'YES' ELSE 'NO' END) autoextend FROM dba_data_files GROUP BY tablespace_name) df, (SELECT tablespace_name, SUM(bytes) / 1048576 free_mb FROM dba_free_space GROUP BY tablespace_name) fs WHERE df.tablespace_name = fs.tablespace_name (+) AND (1 - NVL(fs.free_mb, 0) / df.total_mb) * 100 > 75 ORDER BY pct_used DESC;
On EBS 12.2 with online patching, APPS_UNDOTS1 can spike dramatically during ADOP apply phase. If you see it above 80% on a patch day morning — proactively extend before the patch window opens, not during it.
2.3 — Active Session Count and Long-Running Queries
-- Session overview: active vs inactive SELECT inst_id, status, COUNT(*) sessions FROM gv$session WHERE type = 'USER' GROUP BY inst_id, status ORDER BY inst_id, status; -- Long-running queries (over 10 minutes) SELECT s.inst_id, s.sid, s.serial#, s.username, s.module, s.status, ROUND(s.last_call_et / 60, 1) mins_running, SUBSTR(q.sql_text, 1, 60) sql_text FROM gv$session s, gv$sql q WHERE s.sql_id = q.sql_id AND s.status = 'ACTIVE' AND s.last_call_et > 600 AND s.type = 'USER' ORDER BY s.last_call_et DESC;
3. Layer 2 — Application Tier: EBS Services
3.1 — Are All EBS Services Running?
EBS 12.2 has a specific service start order. If any service is down when users arrive, you will face immediate calls. Check this from the application tier server as the applmgr user.
Oracle EBS 12.2 runs as multiple services layered on top of each other: OHS (web entry point) → WebLogic (Java tier) → Database. If OHS is down, users see a browser error. If WebLogic is down, they see an EBS error page. If the Concurrent Manager is down, batch jobs stop silently. You need to check all three every morning.
# Check all EBS services status (run as applmgr) source /path/to/<CONTEXT_NAME>.env $ADMIN_SCRIPTS_HOME/adstatus.sh # Check individual service status $ADMIN_SCRIPTS_HOME/adopmnctl.sh status # OHS / Oracle HTTP Server $ADMIN_SCRIPTS_HOME/adadminsrvctl.sh status # WebLogic AdminServer $ADMIN_SCRIPTS_HOME/admanagedsrvctl.sh status # WLS Managed Servers $ADMIN_SCRIPTS_HOME/adcmctl.sh status # Concurrent Manager # Quick process check on Solaris ps -ef | grep -E "httpd|java|FNDLIBR|FNDSM" | grep -v grep
3.2 — Verify the Correct Filesystem is Active (fs1 / fs2)
This is EBS 12.2 specific and catches more DBAs off guard than anything else. After ADOP patching, the run edition switches between fs1 and fs2. Always confirm which is active.
# Check which filesystem is the current run edition cat $EBS_DOMAIN_HOME/applications/EBSapps_instance/controlfile/adinfo_*.xml | \ grep -i "run_edition_home" # Or directly from the context file grep "s_current_base" $CONTEXT_FILE
adop phase=status to confirm the current ADOP state before assuming the environment is clean.4. Layer 3 — Concurrent Manager (CM)
The Concurrent Manager is the heartbeat of EBS batch processing. If it is not running, or if requests are piling up in Pending/Standby status, business processes stop silently — and users notice hours later.
4.1 — CM Manager Status
-- Check all manager statuses SELECT c.concurrent_queue_name manager_name, c.user_concurrent_queue_name display_name, c.enabled_flag, r.running_processes, r.max_processes, r.node_name FROM fnd_concurrent_queues_vl c, fnd_concurrent_queue_size r WHERE c.concurrent_queue_id = r.concurrent_queue_id (+) AND c.application_id = r.application_id (+) ORDER BY c.concurrent_queue_name;
4.2 — Pending and Stuck Requests (The Most Common Morning Issue)
-- Count of requests by phase and status SELECT phase_code phase, status_code status, COUNT(*) cnt FROM fnd_concurrent_requests WHERE phase_code IN ('P', 'R') -- Pending and Running GROUP BY phase_code, status_code ORDER BY phase_code, status_code; -- Requests stuck Pending for more than 30 minutes SELECT r.request_id, r.concurrent_program_name program, p.user_concurrent_program_name display_name, r.requested_start_date, ROUND((SYSDATE - r.requested_start_date) * 1440) mins_pending, r.phase_code, r.status_code FROM fnd_concurrent_requests r, fnd_concurrent_programs_vl p WHERE r.concurrent_program_id = p.concurrent_program_id (+) AND r.phase_code = 'P' AND r.status_code = 'I' -- Normal (Pending/Normal = stuck) AND (SYSDATE - r.requested_start_date) * 1440 > 30 ORDER BY r.requested_start_date;
If you see a large number of Pending/Standby requests with no active ICM (Internal Concurrent Manager) process, the Conflict Resolution Manager has likely stalled. Check the ICM log at $APPLCSF/$APPLLOG/ for FNDLIBR errors before bouncing the CM. A blind bounce without reading the ICM log will not fix the root cause.
5. Layer 4 — WebLogic / AdminServer Health
WebLogic AdminServer memory exhaustion is one of the most common EBS 12.2 production issues — especially in environments where the heap size was not tuned after initial installation. A quick morning check takes 60 seconds and prevents emergency 2 AM restarts.
5.1 — AdminServer Process and Memory
# Check AdminServer JVM heap usage (Solaris) ps -ef | grep AdminServer | grep -v grep | \ awk '{print "PID:", $2, "RSS(MB):", $6/1024}' # Check WLS process and port (AdminServer default: 7001) netstat -an | grep 7001 # Tail the AdminServer log for recent errors tail -100 $DOMAIN_HOME/servers/AdminServer/logs/AdminServer.log | \ grep -iE "error|exception|OutOfMemory|BEA-"
5.2 — JDBC Connection Pool Check (via WLST)
# Quick WLST snippet to check JDBC pool health cd $FMW_HOME/wlserver/common/bin ./wlst.sh <<EOF connect('weblogic','<password>','t3://<host>:7001') serverRuntime() cd('JDBCServiceRuntime/<server_name>/JDBCDataSourceRuntimeMBeans') ls() EOF
ps) is above 3 GB and climbing steadily every morning, your heap settings need tuning. Standard recommendation for EBS 12.2: set -Xms2048m -Xmx4096m in setDomainEnv.sh and schedule a monthly AdminServer restart in a low-traffic window.6. Layer 5 — RMAN Backup Status
If last night's backup failed and you did not catch it this morning, you are running production EBS without a current backup. That is an unacceptable risk. This check takes 2 minutes.
-- Check last backup status (last 24 hours) SELECT session_key, input_type, status, TO_CHAR(start_time, 'DD-MON-YYYY HH24:MI') start_time, TO_CHAR(end_time, 'DD-MON-YYYY HH24:MI') end_time, ROUND(elapsed_seconds / 60) elapsed_mins, ROUND(output_bytes / 1073741824, 2) output_gb FROM v$rman_backup_job_details WHERE start_time > SYSDATE - 1 ORDER BY start_time DESC; -- Archive log gap check (critical for standby or recovery) SELECT sequence#, first_time, next_time, applied, deleted FROM v$archived_log WHERE first_time > SYSDATE - 1 AND standby_dest = 'NO' ORDER BY sequence#;
| RMAN Status | Meaning | Action |
|---|---|---|
COMPLETED | Backup successful | No action needed ✅ |
COMPLETED WITH WARNINGS | Backup ran but some files skipped | Check output — usually safe, but investigate |
FAILED | Backup did not complete | Check RMAN log immediately, re-run or escalate |
| No rows returned | Backup never ran or scheduler issue | Check RMAN script cron job — treat as FAILED |
7. Layer 6 — OS-Level Checks (Solaris 11.4 SPARC)
7.1 — Disk and Mount Point Health
# Disk usage — flag anything above 85% df -h | awk 'NR==1 || $5+0 > 85' # Check NFS mounts are still alive (critical in EBS) mount | grep nfs df -k | grep -v Filesystem | awk '$6 ~ /nfs/' # Check for zombie or stuck processes ps -ef | awk '$8 == "Z"' # Archive log destination — must not be above 80% df -h /u03/archivelogs # adjust path to your arch dest
7.2 — System Load and Memory
# Current system load average (1, 5, 15 min) uptime # Memory usage on Solaris prtconf | grep "Memory size" vmstat 1 3 # watch 'si/so' for swapping — must be 0 on healthy system # Check for any core dumps from last 24 hours find /var/core -name core -newer /tmp -ls 2>/dev/null
vmstat columns differ from Linux. Watch the sr (scan rate) column — if it is consistently above 200 and pi/po (page in/out) is non-zero, the system is under memory pressure. For EBS, this will show up as slow Forms loading and WebLogic response delays before it becomes obvious to users.8. The Complete Morning Checklist (Print-Ready)
Use this as your daily sign-off. Document your results in your shift handover notes every morning.
- š DB Alert Log — No new ORA- errors since last check
- š¾ Tablespace Usage — All critical tablespaces below 85%
- š UNDO / TEMP — Not exhausted, autoextend functioning
- š Archive Log Destination — Below 80%, no gaps in sequence
- š„ Active Sessions — No unexpected spike; no long-runners (>10 min)
- š EBS Services — OHS, AdminServer, Managed Servers all UP
- š fs1/fs2 Edition — Run edition confirmed; no dangling ADOP phase
- ⚙️ Concurrent Manager — ICM running; no Pending/Standby queue buildup
- ☕ WebLogic AdminServer — No OutOfMemoryError in last 24 hrs
- šæ RMAN Backup — Last backup status: COMPLETED
- š„️ Disk Space (OS) — No filesystem above 85%
- š NFS Mounts — All EBS-related NFS mounts alive
- š System Load — Load average within normal range; no swapping
9. One Shell Script to Run Everything
Here is a lightweight shell script that combines the OS-level checks into a single morning summary. Run it as applmgr from the app tier. Add DB checks to cron separately (requires DB connectivity).
#!/bin/ksh # ebs_morning_check.sh # Run as: applmgr user on application tier # Oracle EBS 12.2.x — Morning Health Summary source /path/to/<CONTEXT_NAME>.env echo "═══════════════════════════════════════" echo " EBS MORNING HEALTH CHECK — $(date '+%d-%b-%Y %H:%M')" echo "═══════════════════════════════════════" echo "\nš DISK USAGE (>70% shown):" df -h | awk 'NR==1 || ($5+0) > 70 {print}' echo "\nš NFS MOUNTS:" mount | grep nfs | awk '{print " ✓", $1, "→", $3}' echo "\n⚙️ EBS SERVICES:" $ADMIN_SCRIPTS_HOME/adstatus.sh 2>/dev/null | grep -iE "running|stopped|dead" echo "\nš SYSTEM LOAD:" uptime echo "\nš ALERT LOG - Last ORA- errors:" grep "ORA-" $ORACLE_BASE/diag/rdbms/*/*/trace/alert_*.log 2>/dev/null | \ tail -5 echo "\n═══════════════════════════════════════" echo " Check complete. Run SQL checks from DB server." echo "═══════════════════════════════════════"
10. Summary
A structured morning health check is not optional in a 24×7 EBS production support role — it is professional table stakes. For freshers, it gives you a daily framework that builds environmental awareness faster than anything else. For experienced DBAs, documenting it makes your knowledge reproducible across shifts and geography.
Run it consistently. Document your findings. Your 9 AM incidents will become 8 AM catches — and your manager will notice the difference before any appraisal conversation starts.
No comments:
Post a Comment