Saturday, April 18, 2026

Oracle EBS 12.2 — Morning Health Check Checklist

Apps DBA EBS 12.2.x Monitoring Fresher + Experienced

🩺 Oracle EBS 12.2 — The Daily Morning Health Check
Every Apps DBA Must Run (Fresher to L4)

Environment: Oracle EBS 12.2.x · Oracle DB 19c · Solaris 11.4 SPARC · RAC · WebLogic 12c
Audience: Freshers joining Apps DBA teams + Experienced DBAs building a structured morning routine
Time to execute: 20–30 minutes · Do this before the first business user logs in

If you are a fresher stepping into an Oracle Apps DBA role for the first time, the first question you will face every morning is: "Is the environment healthy?" If you are an experienced DBA, you probably run a mental checklist already — but is it consistent, documented, and defensible if something goes wrong at 9 AM?

This post gives you a structured, repeatable morning health check — with the exact commands, SQL scripts, and warning signs — covering all critical layers of Oracle EBS 12.2: Database, Application tier, Concurrent Manager, WebLogic, RMAN, and OS-level checks on Solaris.

Bookmark it. Run it every morning. Your on-call stress will drop significantly.

📋 What This Post Covers

Why a structured health check matters (and what happens without one)
Layer 1 — Database tier: Alert log, tablespace, sessions
Layer 2 — Application tier: EBS services, autoconfig, fs1/fs2
Layer 3 — Concurrent Manager and batch jobs
Layer 4 — WebLogic / AdminServer health
Layer 5 — RMAN backup status
Layer 6 — OS and Solaris checks
The complete morning checklist (print-ready)
Quick-reference scripts in one place

1. Why This Matters — The Cost of No Morning Check

Consider this scenario: a DBA starts their shift at 8 AM. No health check. At 10 AM, a user calls — EBS login is broken. Investigation reveals the OHS process crashed at 6:45 AM. The concurrent manager has been rejecting jobs since 7:00 AM. RMAN backup failed last night but nobody noticed.

A 20-minute morning health check at 8 AM catches all three of these before the business starts. It also gives you a baseline snapshot — so if something breaks at noon, you know exactly what changed since morning.

💡 For Freshers: Your manager will ask "did you check the environment this morning?" A documented health check means your answer is always "yes — and here is what I found." This single habit differentiates a junior DBA from a reliable one within your first 90 days.

⚠️ For Experienced DBAs: If your morning check is informal and in your head, it is not repeatable by your team across APAC / EMEA / Americas shifts. Document it, standardise it, share it. That is the difference between individual knowledge and team resilience.

2. Layer 1 — Database Tier

2.1 — Check the DB Alert Log (First Priority, Always)

The DB alert log is the single most important file in your EBS environment. Check it before anything else. Any ORA- error here is a potential incident.

🎓 Fresher Explanation

The alert log is a running diary that Oracle writes for every database event — startup, shutdown, errors, checkpoints, log switches. It lives on the DB server (not the app server). Your first job every morning is to read the last 200 lines and look for anything that starts with ORA-.

# Find the alert log path (Solaris)
find $ORACLE_BASE/diag/rdbms -name "alert_*.log" 2>/dev/null

# Tail the last 200 lines
tail -200 $ORACLE_BASE/diag/rdbms/<DBNAME>/<SID>/trace/alert_<SID>.log

# Filter for ORA- errors only (the quick check)
grep "ORA-" $ORACLE_BASE/diag/rdbms/<DBNAME>/<SID>/trace/alert_<SID>.log | tail -30

# Check for errors from last 24 hours only
awk '/'"$(date -d 'yesterday' '+%a %b')"'/,0' alert_<SID>.log | grep "ORA-"

ORA- Error	What It Means	Priority
`ORA-00600`	Internal Oracle error — raise SR immediately	🔴 P1
`ORA-07445`	Exception encountered — core dump likely	🔴 P1
`ORA-04031`	Shared pool / SGA memory insufficient	🔴 P1
`ORA-01555`	Snapshot too old — UNDO retention issue	🟡 P2
`ORA-00060`	Deadlock detected — check application logic	🟡 P2
`ORA-01652`	TEMP tablespace full	🟡 P2
`ORA-00257`	Archiver error — archive log destination full	🔴 P1

2.2 — Tablespace Usage

A tablespace hitting 90%+ before business hours starts is a disaster waiting to happen. EBS SYSTEM, APPS_TS_TX_DATA, APPS_TS_SEED, and UNDOTBS are the critical ones.

-- Tablespace usage — flag anything above 85%
SELECT
    df.tablespace_name,
    ROUND(df.total_mb) total_mb,
    ROUND(df.total_mb - NVL(fs.free_mb, 0)) used_mb,
    ROUND((1 - NVL(fs.free_mb, 0) / df.total_mb) * 100, 1) pct_used,
    df.autoextend
FROM
    (SELECT tablespace_name,
            SUM(bytes) / 1048576 total_mb,
            MAX(CASE autoextensible WHEN 'YES' THEN 'YES' ELSE 'NO' END) autoextend
     FROM   dba_data_files
     GROUP  BY tablespace_name) df,
    (SELECT tablespace_name, SUM(bytes) / 1048576 free_mb
     FROM   dba_free_space
     GROUP  BY tablespace_name) fs
WHERE  df.tablespace_name = fs.tablespace_name (+)
AND   (1 - NVL(fs.free_mb, 0) / df.total_mb) * 100 > 75
ORDER  BY pct_used DESC;

⚡ Experienced DBA Note

On EBS 12.2 with online patching, APPS_UNDOTS1 can spike dramatically during ADOP apply phase. If you see it above 80% on a patch day morning — proactively extend before the patch window opens, not during it.

2.3 — Active Session Count and Long-Running Queries

-- Session overview: active vs inactive
SELECT inst_id, status, COUNT(*) sessions
FROM   gv$session
WHERE  type = 'USER'
GROUP  BY inst_id, status
ORDER  BY inst_id, status;

-- Long-running queries (over 10 minutes)
SELECT
    s.inst_id, s.sid, s.serial#, s.username,
    s.module, s.status,
    ROUND(s.last_call_et / 60, 1) mins_running,
    SUBSTR(q.sql_text, 1, 60) sql_text
FROM   gv$session s, gv$sql q
WHERE  s.sql_id = q.sql_id
AND    s.status = 'ACTIVE'
AND    s.last_call_et > 600
AND    s.type = 'USER'
ORDER  BY s.last_call_et DESC;

3. Layer 2 — Application Tier: EBS Services

3.1 — Are All EBS Services Running?

EBS 12.2 has a specific service start order. If any service is down when users arrive, you will face immediate calls. Check this from the application tier server as the applmgr user.

🎓 Fresher Explanation

Oracle EBS 12.2 runs as multiple services layered on top of each other: OHS (web entry point) → WebLogic (Java tier) → Database. If OHS is down, users see a browser error. If WebLogic is down, they see an EBS error page. If the Concurrent Manager is down, batch jobs stop silently. You need to check all three every morning.

# Check all EBS services status (run as applmgr)
source /path/to/<CONTEXT_NAME>.env
$ADMIN_SCRIPTS_HOME/adstatus.sh

# Check individual service status
$ADMIN_SCRIPTS_HOME/adopmnctl.sh status   # OHS / Oracle HTTP Server
$ADMIN_SCRIPTS_HOME/adadminsrvctl.sh status  # WebLogic AdminServer
$ADMIN_SCRIPTS_HOME/admanagedsrvctl.sh status  # WLS Managed Servers
$ADMIN_SCRIPTS_HOME/adcmctl.sh status    # Concurrent Manager

# Quick process check on Solaris
ps -ef | grep -E "httpd|java|FNDLIBR|FNDSM" | grep -v grep

3.2 — Verify the Correct Filesystem is Active (fs1 / fs2)

This is EBS 12.2 specific and catches more DBAs off guard than anything else. After ADOP patching, the run edition switches between fs1 and fs2. Always confirm which is active.

# Check which filesystem is the current run edition
cat $EBS_DOMAIN_HOME/applications/EBSapps_instance/controlfile/adinfo_*.xml | \
    grep -i "run_edition_home"

# Or directly from the context file
grep "s_current_base" $CONTEXT_FILE

⚠️ Watch out for this: If ADOP cleanup did not complete after the last patch cycle, both fs1 and fs2 may show as partially configured. Run adop phase=status to confirm the current ADOP state before assuming the environment is clean.

4. Layer 3 — Concurrent Manager (CM)

The Concurrent Manager is the heartbeat of EBS batch processing. If it is not running, or if requests are piling up in Pending/Standby status, business processes stop silently — and users notice hours later.

4.1 — CM Manager Status

-- Check all manager statuses
SELECT
    c.concurrent_queue_name       manager_name,
    c.user_concurrent_queue_name  display_name,
    c.enabled_flag,
    r.running_processes,
    r.max_processes,
    r.node_name
FROM   fnd_concurrent_queues_vl c,
       fnd_concurrent_queue_size r
WHERE  c.concurrent_queue_id = r.concurrent_queue_id (+)
AND    c.application_id      = r.application_id (+)
ORDER  BY c.concurrent_queue_name;

4.2 — Pending and Stuck Requests (The Most Common Morning Issue)

-- Count of requests by phase and status
SELECT
    phase_code  phase,
    status_code status,
    COUNT(*)    cnt
FROM   fnd_concurrent_requests
WHERE  phase_code IN ('P', 'R')   -- Pending and Running
GROUP  BY phase_code, status_code
ORDER  BY phase_code, status_code;

-- Requests stuck Pending for more than 30 minutes
SELECT
    r.request_id,
    r.concurrent_program_name  program,
    p.user_concurrent_program_name display_name,
    r.requested_start_date,
    ROUND((SYSDATE - r.requested_start_date) * 1440) mins_pending,
    r.phase_code, r.status_code
FROM   fnd_concurrent_requests r,
       fnd_concurrent_programs_vl p
WHERE  r.concurrent_program_id = p.concurrent_program_id (+)
AND    r.phase_code  = 'P'
AND    r.status_code = 'I'  -- Normal (Pending/Normal = stuck)
AND    (SYSDATE - r.requested_start_date) * 1440 > 30
ORDER  BY r.requested_start_date;

⚡ Experienced DBA Note

If you see a large number of Pending/Standby requests with no active ICM (Internal Concurrent Manager) process, the Conflict Resolution Manager has likely stalled. Check the ICM log at $APPLCSF/$APPLLOG/ for FNDLIBR errors before bouncing the CM. A blind bounce without reading the ICM log will not fix the root cause.

5. Layer 4 — WebLogic / AdminServer Health

WebLogic AdminServer memory exhaustion is one of the most common EBS 12.2 production issues — especially in environments where the heap size was not tuned after initial installation. A quick morning check takes 60 seconds and prevents emergency 2 AM restarts.

5.1 — AdminServer Process and Memory

# Check AdminServer JVM heap usage (Solaris)
ps -ef | grep AdminServer | grep -v grep | \
    awk '{print "PID:", $2, "RSS(MB):", $6/1024}'

# Check WLS process and port (AdminServer default: 7001)
netstat -an | grep 7001

# Tail the AdminServer log for recent errors
tail -100 $DOMAIN_HOME/servers/AdminServer/logs/AdminServer.log | \
    grep -iE "error|exception|OutOfMemory|BEA-"

5.2 — JDBC Connection Pool Check (via WLST)

# Quick WLST snippet to check JDBC pool health
cd $FMW_HOME/wlserver/common/bin
./wlst.sh <<EOF
connect('weblogic','<password>','t3://<host>:7001')
serverRuntime()
cd('JDBCServiceRuntime/<server_name>/JDBCDataSourceRuntimeMBeans')
ls()
EOF

💡 If AdminServer RSS memory (from ps) is above 3 GB and climbing steadily every morning, your heap settings need tuning. Standard recommendation for EBS 12.2: set -Xms2048m -Xmx4096m in setDomainEnv.sh and schedule a monthly AdminServer restart in a low-traffic window.

6. Layer 5 — RMAN Backup Status

If last night's backup failed and you did not catch it this morning, you are running production EBS without a current backup. That is an unacceptable risk. This check takes 2 minutes.

-- Check last backup status (last 24 hours)
SELECT
    session_key,
    input_type,
    status,
    TO_CHAR(start_time, 'DD-MON-YYYY HH24:MI') start_time,
    TO_CHAR(end_time,   'DD-MON-YYYY HH24:MI') end_time,
    ROUND(elapsed_seconds / 60) elapsed_mins,
    ROUND(output_bytes / 1073741824, 2) output_gb
FROM   v$rman_backup_job_details
WHERE  start_time > SYSDATE - 1
ORDER  BY start_time DESC;

-- Archive log gap check (critical for standby or recovery)
SELECT
    sequence#, first_time, next_time, applied, deleted
FROM   v$archived_log
WHERE  first_time > SYSDATE - 1
AND    standby_dest = 'NO'
ORDER  BY sequence#;

RMAN Status	Meaning	Action
`COMPLETED`	Backup successful	No action needed ✅
`COMPLETED WITH WARNINGS`	Backup ran but some files skipped	Check output — usually safe, but investigate
`FAILED`	Backup did not complete	Check RMAN log immediately, re-run or escalate
No rows returned	Backup never ran or scheduler issue	Check RMAN script cron job — treat as FAILED

7. Layer 6 — OS-Level Checks (Solaris 11.4 SPARC)

7.1 — Disk and Mount Point Health

# Disk usage — flag anything above 85%
df -h | awk 'NR==1 || $5+0 > 85'

# Check NFS mounts are still alive (critical in EBS)
mount | grep nfs
df -k | grep -v Filesystem | awk '$6 ~ /nfs/'

# Check for zombie or stuck processes
ps -ef | awk '$8 == "Z"'

# Archive log destination — must not be above 80%
df -h /u03/archivelogs  # adjust path to your arch dest

7.2 — System Load and Memory

# Current system load average (1, 5, 15 min)
uptime

# Memory usage on Solaris
prtconf | grep "Memory size"
vmstat 1 3   # watch 'si/so' for swapping — must be 0 on healthy system

# Check for any core dumps from last 24 hours
find /var/core -name core -newer /tmp -ls 2>/dev/null

⚠️ Solaris-specific: On SPARC systems, vmstat columns differ from Linux. Watch the sr (scan rate) column — if it is consistently above 200 and pi/po (page in/out) is non-zero, the system is under memory pressure. For EBS, this will show up as slow Forms loading and WebLogic response delays before it becomes obvious to users.

8. The Complete Morning Checklist (Print-Ready)

Use this as your daily sign-off. Document your results in your shift handover notes every morning.

📋 DB Alert Log — No new ORA- errors since last check
💾 Tablespace Usage — All critical tablespaces below 85%
🔒 UNDO / TEMP — Not exhausted, autoextend functioning
🔗 Archive Log Destination — Below 80%, no gaps in sequence
👥 Active Sessions — No unexpected spike; no long-runners (>10 min)
🌐 EBS Services — OHS, AdminServer, Managed Servers all UP
📂 fs1/fs2 Edition — Run edition confirmed; no dangling ADOP phase
⚙️ Concurrent Manager — ICM running; no Pending/Standby queue buildup
☕ WebLogic AdminServer — No OutOfMemoryError in last 24 hrs
💿 RMAN Backup — Last backup status: COMPLETED
🖥️ Disk Space (OS) — No filesystem above 85%
🔗 NFS Mounts — All EBS-related NFS mounts alive
📊 System Load — Load average within normal range; no swapping

✅ You are done. If all 13 checks are green, the environment is healthy and you can start your shift with confidence. Document your check in the shift handover — even two lines is enough: "Health check complete 08:15. All services up. RMAN backup completed. No ORA- errors. One tablespace at 83% — monitoring."

9. One Shell Script to Run Everything

Here is a lightweight shell script that combines the OS-level checks into a single morning summary. Run it as applmgr from the app tier. Add DB checks to cron separately (requires DB connectivity).

#!/bin/ksh
# ebs_morning_check.sh
# Run as: applmgr user on application tier
# Oracle EBS 12.2.x — Morning Health Summary

source /path/to/<CONTEXT_NAME>.env

echo "═══════════════════════════════════════"
echo "  EBS MORNING HEALTH CHECK — $(date '+%d-%b-%Y %H:%M')"
echo "═══════════════════════════════════════"

echo "\n📂 DISK USAGE (>70% shown):"
df -h | awk 'NR==1 || ($5+0) > 70 {print}'

echo "\n🔗 NFS MOUNTS:"
mount | grep nfs | awk '{print " ✓", $1, "→", $3}'

echo "\n⚙️  EBS SERVICES:"
$ADMIN_SCRIPTS_HOME/adstatus.sh 2>/dev/null | grep -iE "running|stopped|dead"

echo "\n📊 SYSTEM LOAD:"
uptime

echo "\n🔍 ALERT LOG - Last ORA- errors:"
grep "ORA-" $ORACLE_BASE/diag/rdbms/*/*/trace/alert_*.log 2>/dev/null | \
    tail -5

echo "\n═══════════════════════════════════════"
echo "  Check complete. Run SQL checks from DB server."
echo "═══════════════════════════════════════"

10. Summary

A structured morning health check is not optional in a 24×7 EBS production support role — it is professional table stakes. For freshers, it gives you a daily framework that builds environmental awareness faster than anything else. For experienced DBAs, documenting it makes your knowledge reproducible across shifts and geography.

Run it consistently. Document your findings. Your 9 AM incidents will become 8 AM catches — and your manager will notice the difference before any appraisal conversation starts.

💡 Next in this series: Oracle EBS 12.2 ADOP Phase Failures — How to Diagnose and Recover Without a Full Rollback · Oracle Apps DBA — Freshers' Guide to Your First Clone Operation

Tags: Oracle EBS Apps DBA EBS 12.2.x Health Check Monitoring Concurrent Manager WebLogic RMAN Solaris Fresher Guide Shell Script SQL ADOP RAC

Friday, April 17, 2026

OACORE

🔍 What Is Each OACORE Node Doing? — Oracle EBS JDBC Session Investigation

Oracle EBS 12.2 · Apps DBA · Performance Tuning · GV$SESSION

In Oracle EBS 12.2 environments, OACORE (Oracle Application Core) is the WebLogic managed server responsible for serving the core EBS application tier. Each OACORE node maintains a pool of JDBC connections into the database. When performance degrades or sessions spike, DBAs need to quickly identify what each OACORE node is actually doing — not just how many connections exist.

This post covers three focused SQL scripts to investigate OACORE JDBC sessions across all RAC instances using GV$SESSION and GV$SQL.

📌 Query 1 — Active SQL Being Executed Per OACORE Right Now

This is your first-response query. It joins GV$SESSION with GV$SQL to show the actual SQL text being executed by each active OACORE session at this moment.

set lines 200 pages 500
col machine   for a30
col module    for a40
col sql_text  for a60
col status    for a10
col username  for a15

select s.machine,
       s.module,
       s.status,
       s.username,
       s.sid,
       s.serial#,
       s.last_call_et,
       substr(q.sql_text,1,60) sql_text
from   gv$session s,
       gv$sql     q
where  s.program            like '%JDBC%'
and    s.sql_id             = q.sql_id
and    s.sql_child_number   = q.child_number
and    s.status             = 'ACTIVE'
order by s.machine, s.last_call_et desc;

📋 What to look for: Sessions with high last_call_et (seconds since last DB call) indicate long-running transactions. The machine column tells you which OACORE node the session originated from.

📌 Query 2 — Session Count Breakdown Per OACORE Node and Module

This improved aggregation query gives you a breakdown by instance, OACORE node, module, and status — showing exactly how many sessions are active vs. waiting per node. Much more useful than a plain count(*).

select s.inst_id,
       s.machine,
       s.module,
       s.status,
       count(*)                                                  sessions,
       sum(case when s.state  = 'WAITING' then 1 else 0 end)    waiting,
       sum(case when s.status = 'ACTIVE'  then 1 else 0 end)    active
from   gv$session s
where  s.program like '%JDBC%'
group  by s.inst_id, s.machine, s.module, s.status
order  by s.machine, sessions desc;

📋 What to look for: If one OACORE machine shows a disproportionately high session count compared to others, that node may be overloaded or stuck. Check if WebLogic connection pool on that node needs recycling.

📌 Query 3 — Wait Events Per OACORE Node

Knowing what sessions are waiting on is critical for diagnosing whether the bottleneck is I/O, locking, network, or CPU. This query filters out SQL*Net waits (normal idle waits) and surfaces the real bottlenecks per OACORE node.

select s.machine,
       s.event,
       count(*) cnt
from   gv$session s
where  s.program like '%JDBC%'
and    s.state   = 'WAITING'
and    s.event   not like 'SQL*Net%'
group  by s.machine, s.event
order  by s.machine, cnt desc;

Common Wait Events and Their Meaning

Wait Event	What It Means	Action
`enq: TX - row lock contention`	Session blocked by another session's uncommitted DML	Find blocking session, consider killing or committing
`db file sequential read`	Single block I/O — index scans, possible FTS	Review execution plans, check for missing indexes
`db file scattered read`	Full table / index scans reading multiple blocks	Tune SQL, consider parallel or partitioning
`latch: shared pool`	Hard parsing pressure on shared pool	Enable cursor sharing, review literal SQL
`library cache lock`	DDL or recompilation blocking parse operations	Find DDL session and evaluate impact
`gc buffer busy acquire`	RAC interconnect contention — cross-instance block transfer	Review service affinity, check interconnect health

⚠️ Bonus: Module Reference — What the Module Column Tells You

Module Value	Meaning
`FNDCPGSC`	Concurrent Manager connections — Generic Service Controller
`Oracle Applications`	Standard EBS UI user sessions via Forms or OAF
`e-Business Suite`	Self-service / OAF framework sessions
`JDBC Thin Client`	Generic JDBC — third-party integrations, custom code
`null / blank`	Background or unregistered connection — worth investigating

💡 Side Note — result_cache_max_size
If your current result_cache_max_size is around 184 MB (193298432 bytes) but Oracle 19c recommends 600M for EBS 12.2 environments, that is a tuning gap. You can update this dynamically:

alter system set result_cache_max_size = 600M scope=both;

Verify against your SGA allocation and Oracle Support Note for EBS 12.2 on 19c.

✅ Quick Investigation Workflow

Run Query 2 first — get the count/status snapshot per OACORE node
Run Query 3 — identify dominant wait events per node
Run Query 1 — drill into the actual SQL on ACTIVE sessions of the problem node
Cross-reference with module names to isolate whether the load is from UI users, concurrent programs, or integrations

Environment: Oracle EBS 12.2.x · Oracle DB 19c · RAC · Solaris 11.4 SPARC
Views Used: GV$SESSION · GV$SQL
Tags: Oracle EBS, Apps DBA, OACORE, JDBC, Performance Tuning, WebLogic, GV$SESSION

Thursday, March 26, 2026

spluk

Oracle EBS 12.2 · SIEM Engineering

Complete Guide: Capturing Oracle EBS 12.2 Logs in Splunk

A production-grade reference covering all 8 tiers — from the DB alert log to DMZ extranet servers, WAF, SiteMinder, OAM WebGate, and custom automation feeds. 80+ log sources. No log left behind.

· Oracle Apps DBA Lead· Oracle Solaris 11.4 SPARC· EBS 12.2.x Production· Splunk Universal Forwarder

8+

Log layers

80+

Log sources

5

DMZ hops

24/7

Coverage

Table of Contents

01. Why Splunk for Oracle EBS?
02. Application Tier Logs
03. WebLogic Server (WLS) Logs
04. Database Tier Logs
05. OS / Solaris 11.4 Logs
06. Security & Threat Detection Logs
07. Monitoring & Automation Feeds
08. Infrastructure / Storage / Network
09. Patching / ADOP / Change Logs
10. DMZ & Extranet EBS — Full Log Strategy
11. Splunk inputs.conf Reference
12. SPL Correlation Queries

Why Splunk for Oracle EBS?

Oracle E-Business Suite 12.2 generates logs across a sprawling multi-tier stack — OHS, WebLogic, Oracle DB, Solaris OS, OAM, SiteMinder, ADOP, and custom automation scripts. Without centralised log aggregation, an incident that spans even two layers can take hours to diagnose.

Splunk bridges this gap by ingesting all these sources into a single searchable platform, enabling real-time alerting, cross-layer correlation, and forensic investigation.

Post-CL0P Ransomware Context: The CL0P ransomware campaign specifically targeted Oracle EBS environments via unpatched JSP endpoints. Complete Splunk coverage — especially JSP endpoint monitoring on OHS access logs and DB audit trails — is no longer optional.

Application Tier Logs

The application tier covers the full request lifecycle — OHS web entry point, OAF/Forms, Concurrent Manager, and ADOP patching artefacts.

Log	Path	Source Type	Priority
Apache/OHS access	$INST_TOP/logs/ora/10.1.3/Apache/access_log	oracle:ebs:ohs:access	P1
Apache/OHS error	$INST_TOP/logs/ora/10.1.3/Apache/error_log	oracle:ebs:ohs:error	P1
OA Framework / JSP errors	$INST_TOP/logs/ora/10.1.3/j2ee/	oracle:ebs:oaf	P1
OPMN process log	$INST_TOP/logs/ora/10.1.3/opmn/	oracle:ebs:opmn	P2
Forms server	$INST_TOP/logs/ora/10.1.3/forms/	oracle:ebs:forms	P2
Concurrent Manager startup	$APPLCSF/$APPLLOG/	oracle:ebs:cm:manager	P1
Concurrent request logs	$APPLCSF/$APPLLOG/*.req	oracle:ebs:cm:request:log	P2
Concurrent request output	$APPLCSF/$APPLOUT/*.out	oracle:ebs:cm:request:out	P3
ADOP patch logs	$NE_BASE/EBSapps/patch/	oracle:ebs:adop	P2
AutoConfig logs	$INST_TOP/admin/log/	oracle:ebs:autoconfig	P3

WebLogic Server (WLS) Logs

WLS is the critical Java EE container underpinning EBS 12.2. These logs are the first place to look for SSO login failures, 500 errors on the external portal, OAM redirect loops, and JDBC pool exhaustion.

Real-world case: A production CORP EBS SSO login performance issue traced to a CRP Test OAM server being referenced instead of production was first visible in the WLS managed server log — not the OHS access log.

Log	Path	Source Type	Priority
AdminServer log	$DOMAIN_HOME/servers/AdminServer/logs/AdminServer.log	oracle:wls:admin	P1
Managed server log	$DOMAIN_HOME/servers/EBS_managed/logs/.log	oracle:wls:managed	P1
WLS access log	$DOMAIN_HOME/servers/*/logs/access.log	oracle:wls:access	P1
OAM/SSO integration log	$DOMAIN_HOME/servers/*/logs/	oracle:wls:oam	P1
GC / JVM heap log	$FMW_HOME/../domain/logs/*.log	oracle:wls:jvm	P2
WLS Node Manager log	$WL_HOME/common/nodemanager/*.log	oracle:wls:nodemanager	P2
WLS JDBC datasource log	$DOMAIN_HOME/servers/*/logs/	oracle:wls:jdbc	P2
WLS deployment log	$DOMAIN_HOME/servers/*/logs/	oracle:wls:deploy	P3

Database Tier Logs

The DB alert log is the single highest-priority log in any EBS environment. It surfaces ORA- errors, startup/shutdown events, redo switches, deadlocks, and block corruption — all in one place.

Log	Path	Source Type	Priority
DB alert log	$ORACLE_BASE/diag/rdbms/<db>/<SID>/trace/alert_<SID>.log	oracle:db:alert	P1
Listener log (XML)	$ORACLE_BASE/diag/tnslsnr/<host>/listener/alert/log.xml	oracle:db:listener	P1
DB audit trail (AUD$)	$ORACLE_BASE/admin/<SID>/adump/*.aud	oracle:db:audit	P1
FGA audit (FGA_LOG$)	DB view → flat file extract	oracle:db:fga	P1
Trace files	$ORACLE_BASE/diag/rdbms/<db>/<SID>/trace/*.trc	oracle:db:trace	P1
RMAN backup log	$ORACLE_BASE/admin/<SID>/log/	oracle:db:rman	P2
Data Pump log	$DATA_PUMP_DIR/*.log	oracle:db:datapump	P3
FND_LOGINS (app audit)	DB extract → flat file	oracle:ebs:fnd:logins	P1
FND_UNSUCCESSFUL_LOGINS	DB extract → flat file	oracle:ebs:fnd:auth_fail	P1
AD_PATCH_HIST	DB extract → flat file	oracle:ebs:patch:hist	P2

OS / Solaris 11.4 Logs

Oracle Solaris 11.4 SPARC has a distinct log layout from Linux. Syslog lives in /var/adm/messages, auth in /var/log/authlog, and C2/BSM audit in /var/audit/. The SMF service log is Solaris-specific and frequently missed in Splunk deployments.

Log	Path	Source Type	Priority
Syslog / messages	/var/adm/messages	solaris:syslog	P1
Auth log	/var/log/authlog	solaris:auth	P1
Cron log	/var/cron/log	solaris:cron	P2
Audit log (BSM/C2)	/var/audit/	solaris:bsm	P2
ZFS / Volume manager	/var/adm/messages (zpool events)	solaris:zfs	P2
NFS / mount events	/var/adm/messages	solaris:nfs	P1
SMF service log	/var/svc/log/*.log	solaris:smf	P2
Disk / SCSI errors	/var/adm/messages	solaris:disk	P1
Core dump events	/var/core/	solaris:coredump	P2
Network interface errors	kstat / snoop logs	solaris:network	P2

Security & Threat Detection Logs

Security-focused logs deserve their own tier. Several are DB extracts that require a scheduled export script to make them Splunk-consumable as flat files. These feed SOC dashboards, incident response playbooks, and compliance reports.

Log	Source	Source Type	Priority
OHS access — JSP endpoints	access_log (filtered)	oracle:ebs:ohs:access	P1
EBS FND security events	FND_EVENTS_Q extract	oracle:ebs:fnd:security	P1
SiteMinder / FCC log	$OAM_HOME/../logs/	oracle:oam:siteminder	P1
OAM access server log	$OAM_HOME/../logs/	oracle:oam:access	P1
OS sudo / privilege log	/var/log/authlog	solaris:sudo	P1
File integrity events	AIDE / Solaris BART	security:fim	P1
Network IDS alerts	Snort/Suricata/Sourcefire	security:ids	P1
Oracle AVDF / DB Vault	AVDF export	oracle:avdf	P2
Patch compliance gaps	ADOP / OEM feed	oracle:ebs:patch:compliance	P2

Monitoring & Automation Feeds

Custom automation scripts encode your team's institutional knowledge about what "healthy" looks like. Treat them as first-class Splunk sources, not afterthoughts. These feeds are especially valuable for trend analysis and proactive alerting.

Feed	Source	Source Type	Priority
OEM metric alerts	OEM → syslog bridge	oracle:oem:alert	P1
PagerDuty incident feed	PagerDuty REST → HEC	pagerduty:incident	P2
Datadog APM spans	Datadog → HEC	datadog:apm	P2
FlexDeploy deploy log	FlexDeploy log export	flexdeploy:deploy	P2
ServiceNow change records	SNOW REST feed	snow:change	P2
mount_monitor.sh output	/var/log/mount_monitor.log	custom:mount_monitor	P1
Batch consolidation report (72 programs)	HTML email + log file	custom:batch_monitor	P1
copy_clonebkp.sh exit codes	syslog or log file	custom:clone_pipeline	P2
CORP_QCC_RELINK output	Log file	custom:relink	P2
EBS URL extractor report	HTML email + log	custom:url_extractor	P3

Infrastructure / Storage / Network

NTP is often overlooked: Clock drift on any EBS tier silently breaks Kerberos token validity and OAM session handling. Monitor NTP sync on ALL hosts — internal and DMZ — as a P1 operational alert.

Log	Source	Source Type	Priority
Storage array log	NetApp/EMC syslog	storage:array	P1
SAN switch log	Brocade/Cisco FC syslog	storage:san	P1
Load balancer log	F5/Oracle LBR syslog	network:lb	P1
Firewall / ACL log	Firewall syslog	network:firewall	P1
DNS resolution log	BIND / Unbound log	network:dns	P2
IPMI / iLO / ILOM hardware	IPMI syslog	hardware:ipmi	P1
Backup agent log	Veritas/Commvault agent	backup:agent	P2
NTP sync log	ntpd / chrony log	infra:ntp	P2

Patching / ADOP / Change Logs

ADOP introduced online patching for EBS 12.2. Each phase generates distinct log artefacts. Capturing these phase-by-phase enables automatic change window validation and unauthorized patch detection — including out-of-window DFF recompilation events.

Log	Path	Source Type	Priority
ADOP prepare phase	$NE_BASE/EBSapps/patch//log/adop_.log	oracle:ebs:adop:prepare	P1
ADOP apply phase	$NE_BASE/EBSapps/patch//log/adop_.log	oracle:ebs:adop:apply	P1
ADOP finalize phase	$NE_BASE/EBSapps/patch//log/adop_.log	oracle:ebs:adop:finalize	P1
ADOP cleanup phase	$NE_BASE/EBSapps/patch//log/adop_.log	oracle:ebs:adop:cleanup	P2
ADOP worker log	$NE_BASE/EBSapps/patch//log/worker.log	oracle:ebs:adop:worker	P2
AD_PATCH_HIST extract	DB extract → flat file	oracle:ebs:patch:hist	P1
AutoPatch log	$APPL_TOP/admin/log/	oracle:ebs:autopatch	P2
DFF / flex compilation log	Concurrent req log + AD worker	oracle:ebs:dff:compile	P3

DMZ & Extranet EBS — Full Log Strategy

This is the section most teams get wrong. The DMZ/extranet layer is not just "another OHS server" — it is a separate attack surface with distinct authentication infrastructure, network controls, and external-facing modules. A Splunk deployment that treats DMZ hosts the same as internal hosts will have critical blind spots.

DMZ Traffic Flow — Log Capture at Every Hop

External user

→

FW1 ①

→

WAF / F5 LBR

→

DMZ OHS (SSL term.)

→

OAM WebGate / SiteMinder

→

FW2 ②

→

Internal WLS / OAM

→

Oracle DB

Critical distinction: The DMZ OHS access log captures real external client IPs. The internal OHS access log shows only the DMZ proxy IP. Both logs are mandatory for end-to-end IP correlation during incident investigation. Tag them with different host values in inputs.conf.

Layer 1 — Perimeter / WAF / External Load Balancer

Log	Source	Source Type	Priority
F5 BIG-IP access log	F5 syslog → Splunk	network:lb:access	P1
F5 BIG-IP SSL log	F5 syslog	network:lb:ssl	P1
WAF alert log	F5 ASM / ModSecurity	network:waf:alert	P1
WAF traffic log	F5 ASM / ModSecurity	network:waf:traffic	P1
External firewall (FW1)	Firewall syslog	network:fw:external	P1

Layer 2 — DMZ OHS / Reverse Proxy (SSL Termination)

Log	Path	Source Type	Priority
DMZ OHS access log	$INST_TOP/logs/ora/10.1.3/Apache/access_log (DMZ host)	oracle:ebs:dmz:ohs:access	P1
DMZ OHS error log	$INST_TOP/logs/ora/10.1.3/Apache/error_log (DMZ host)	oracle:ebs:dmz:ohs:error	P1
SSL/TLS error log	$INST_TOP/logs/ora/10.1.3/Apache/ssl_error_log	oracle:ebs:dmz:ssl	P1
mod_proxy / mod_rewrite log	Apache error log (rewrite debug)	oracle:ebs:dmz:proxy	P2
OHS OPMN (DMZ)	$INST_TOP/logs/ora/10.1.3/opmn/ (DMZ)	oracle:ebs:dmz:opmn	P2

Layer 3 — Authentication: SiteMinder + OAM WebGate

Log	Path	Source Type	Priority
SiteMinder Web Agent log	$NETE_WA_ROOT/webagent.log	oracle:siteminder:webagent	P1
SiteMinder Policy Server log	$SMPS_HOME/log/smps.log	oracle:siteminder:policy	P1
SiteMinder Audit log	$SMPS_HOME/log/smaccess.log	oracle:siteminder:audit	P1
FCC (Forms Credential Collector) log	SiteMinder Web Agent log	oracle:siteminder:fcc	P1
SiteMinder session store log	LDAP / SQL session DB	oracle:siteminder:session	P2
OAM WebGate log (DMZ)	$WEBGATE_HOME/oblix/log/	oracle:oam:webgate:dmz	P1
OAM Access Server log	$OAM_HOME/oblix/log/	oracle:oam:access	P1
OAM Audit log	$OAM_HOME/oblix/log/obaudit.log	oracle:oam:audit	P1
OID / LDAP access log	$ORACLE_HOME/ldap/log/	oracle:oid:access	P2

Layer 4 — External-Facing EBS Modules

Module	Filter Pattern	Source Type	Priority
iSupplier Portal	/OA_HTML/OA.jsp?OAFunc=ISUPPLIER*	oracle:ebs:isupplier:access	P1
XML Gateway / B2B	WLS log + $INST_TOP/logs	oracle:ebs:xmlgw	P1
Guest / anonymous sessions	FND_LOGINS (GUEST user extract)	oracle:ebs:fnd:guest	P1
iRecruitment	OHS access log (filtered by function)	oracle:ebs:irecruitment:access	P2
Self-Service HR (SSHR)	OHS access log (filtered)	oracle:ebs:sshr:access	P2
iStore / Quoting	OHS access log (filtered)	oracle:ebs:istore:access	P2

XML Gateway is high-risk: B2B/EDI inbound payloads can arrive unauthenticated in some configurations. Monitor for abnormal payload sizes, unexpected source IPs, and calls outside business hours.

Layer 5 — DMZ Network & Infrastructure

Log	Source	Source Type	Priority
Internal firewall (FW2)	FW2 syslog (DMZ → internal)	network:fw:internal	P1
IDS / IPS alerts (DMZ)	Snort/Suricata/Sourcefire	security:ids:dmz	P1
SSL certificate expiry alerts	Cert manager / cron check	security:cert:expiry	P1
DMZ switch log	Cisco/Juniper syslog	network:switch:dmz	P2
Reverse DNS failure log	DNS server log	network:dns:dmz	P2
NTP sync (DMZ hosts)	chrony/ntpd log	infra:ntp:dmz	P1

Layer 6 — DMZ OS & Bastion Host

Log	Path	Source Type	Priority
DMZ host syslog	/var/adm/messages (DMZ Solaris)	solaris:syslog:dmz	P1
DMZ auth log	/var/log/authlog (DMZ Solaris)	solaris:auth:dmz	P1
Bastion / jump host auth	/var/log/authlog (bastion)	security:bastion:auth	P1
Bastion session recording	/var/log/bastion/sessions/ (CyberArk/Teleport)	security:bastion:session	P1
DMZ cron log	/var/cron/log (DMZ)	solaris:cron:dmz	P2
DMZ BSM audit	/var/audit/ (DMZ)	solaris:bsm:dmz	P2

Splunk inputs.conf Reference

Representative inputs.conf snippets for both internal and DMZ Universal Forwarder deployments on Solaris 11.4.

inputs.conf — Internal Tier (Solaris UF)

# DB Alert Log
[monitor://$ORACLE_BASE/diag/rdbms/*/*/trace/alert_*.log]
index = oracle_db
sourcetype = oracle:db:alert
host = EBSPROD_DB01

# OHS Access Log
[monitor://$INST_TOP/logs/ora/10.1.3/Apache/access_log]
index = ebs_app
sourcetype = oracle:ebs:ohs:access
host = EBSPROD_APP01

# WLS Managed Server
[monitor://$DOMAIN_HOME/servers/*/logs/*.log]
index = ebs_app
sourcetype = oracle:wls:managed

# Concurrent Manager
[monitor://$APPLCSF/$APPLLOG/*]
index = ebs_batch
sourcetype = oracle:ebs:cm:manager

# ADOP Patch Logs
[monitor://$NE_BASE/EBSapps/patch/*/log/*.log]
index = ebs_change
sourcetype = oracle:ebs:adop

# Solaris OS Logs
[monitor:///var/adm/messages]
index = os_internal
sourcetype = solaris:syslog

[monitor:///var/log/authlog]
index = os_security
sourcetype = solaris:auth

# Custom Automation
[monitor:///var/log/mount_monitor.log]
index = custom_ops
sourcetype = custom:mount_monitor

inputs.conf — DMZ Tier (Solaris UF)

# DMZ OHS — separate index from internal OHS
[monitor://$INST_TOP/logs/ora/10.1.3/Apache/access_log]
index = ebs_dmz
sourcetype = oracle:ebs:dmz:ohs:access
host = EBSDMZ_OHS01

# SiteMinder Web Agent
[monitor://$NETE_WA_ROOT/webagent.log]
index = ebs_security
sourcetype = oracle:siteminder:webagent

# SiteMinder Audit
[monitor://$SMPS_HOME/log/smaccess.log]
index = ebs_security
sourcetype = oracle:siteminder:audit

# OAM WebGate (DMZ)
[monitor://$WEBGATE_HOME/oblix/log/*]
index = ebs_security
sourcetype = oracle:oam:webgate:dmz

# DMZ OS Logs
[monitor:///var/adm/messages]
index = os_dmz
sourcetype = solaris:syslog:dmz
host = EBSDMZ_HOST01

[monitor:///var/log/authlog]
index = os_dmz
sourcetype = solaris:auth:dmz

SPL Correlation Queries

Production-ready SPL searches for the most common cross-layer alert scenarios covering internal, DMZ, and security tiers.

Detect external IP bypassing WAF

index=ebs_dmz sourcetype=oracle:ebs:dmz:ohs:access
| where NOT src_ip IN ("<<WAF_IP_LIST>>")
| stats count by src_ip, uri
| where count > 5

FCC 500 error detection (SiteMinder)

index=ebs_security sourcetype=oracle:siteminder:webagent
  (fcc OR redirect) (error OR fail OR 500)
| timechart span=5m count by host

Brute force on extranet login

index=ebs_security sourcetype=oracle:siteminder:audit action=REJECT
| bucket _time span=5m
| stats count by src_ip, _time
| where count > 10

ORA- errors in DB alert log (P1 alert)

index=oracle_db sourcetype=oracle:db:alert
  (ORA-600 OR ORA-7445 OR ORA-4031 OR ORA-1555)
| rex field=_raw "(?P<ora_error>ORA-\d+)"
| stats count by ora_error, host
| sort -count

SSL certificate expiry < 30 days

index=infra sourcetype=security:cert:expiry
| where days_to_expiry < 30
| table cn, expiry_date, host, days_to_expiry
| sort days_to_expiry

ADOP patch applied outside change window

index=ebs_change sourcetype=oracle:ebs:adop:apply
| eval hour=strftime(_time,"%H")
| where hour < 22 AND hour > 6
| stats count by host, _time, patch_id

CM batch job failure rate (QCT production)

index=ebs_batch sourcetype=oracle:ebs:cm:request:log
  completion_status=ERROR
| timechart span=1h count as failures
| where failures > 5