Friday, April 17, 2026

OACORE

🔍 What Is Each OACORE Node Doing? — Oracle EBS JDBC Session Investigation

Oracle EBS 12.2 · Apps DBA · Performance Tuning · GV$SESSION

In Oracle EBS 12.2 environments, OACORE (Oracle Application Core) is the WebLogic managed server responsible for serving the core EBS application tier. Each OACORE node maintains a pool of JDBC connections into the database. When performance degrades or sessions spike, DBAs need to quickly identify what each OACORE node is actually doing — not just how many connections exist.

This post covers three focused SQL scripts to investigate OACORE JDBC sessions across all RAC instances using GV$SESSION and GV$SQL.


📌 Query 1 — Active SQL Being Executed Per OACORE Right Now

This is your first-response query. It joins GV$SESSION with GV$SQL to show the actual SQL text being executed by each active OACORE session at this moment.

set lines 200 pages 500
col machine   for a30
col module    for a40
col sql_text  for a60
col status    for a10
col username  for a15

select s.machine,
       s.module,
       s.status,
       s.username,
       s.sid,
       s.serial#,
       s.last_call_et,
       substr(q.sql_text,1,60) sql_text
from   gv$session s,
       gv$sql     q
where  s.program            like '%JDBC%'
and    s.sql_id             = q.sql_id
and    s.sql_child_number   = q.child_number
and    s.status             = 'ACTIVE'
order by s.machine, s.last_call_et desc;
📋 What to look for: Sessions with high last_call_et (seconds since last DB call) indicate long-running transactions. The machine column tells you which OACORE node the session originated from.

📌 Query 2 — Session Count Breakdown Per OACORE Node and Module

This improved aggregation query gives you a breakdown by instance, OACORE node, module, and status — showing exactly how many sessions are active vs. waiting per node. Much more useful than a plain count(*).

select s.inst_id,
       s.machine,
       s.module,
       s.status,
       count(*)                                                  sessions,
       sum(case when s.state  = 'WAITING' then 1 else 0 end)    waiting,
       sum(case when s.status = 'ACTIVE'  then 1 else 0 end)    active
from   gv$session s
where  s.program like '%JDBC%'
group  by s.inst_id, s.machine, s.module, s.status
order  by s.machine, sessions desc;
📋 What to look for: If one OACORE machine shows a disproportionately high session count compared to others, that node may be overloaded or stuck. Check if WebLogic connection pool on that node needs recycling.

📌 Query 3 — Wait Events Per OACORE Node

Knowing what sessions are waiting on is critical for diagnosing whether the bottleneck is I/O, locking, network, or CPU. This query filters out SQL*Net waits (normal idle waits) and surfaces the real bottlenecks per OACORE node.

select s.machine,
       s.event,
       count(*) cnt
from   gv$session s
where  s.program like '%JDBC%'
and    s.state   = 'WAITING'
and    s.event   not like 'SQL*Net%'
group  by s.machine, s.event
order  by s.machine, cnt desc;

Common Wait Events and Their Meaning

Wait Event What It Means Action
enq: TX - row lock contention Session blocked by another session's uncommitted DML Find blocking session, consider killing or committing
db file sequential read Single block I/O — index scans, possible FTS Review execution plans, check for missing indexes
db file scattered read Full table / index scans reading multiple blocks Tune SQL, consider parallel or partitioning
latch: shared pool Hard parsing pressure on shared pool Enable cursor sharing, review literal SQL
library cache lock DDL or recompilation blocking parse operations Find DDL session and evaluate impact
gc buffer busy acquire RAC interconnect contention — cross-instance block transfer Review service affinity, check interconnect health

⚠️ Bonus: Module Reference — What the Module Column Tells You

Module Value Meaning
FNDCPGSC Concurrent Manager connections — Generic Service Controller
Oracle Applications Standard EBS UI user sessions via Forms or OAF
e-Business Suite Self-service / OAF framework sessions
JDBC Thin Client Generic JDBC — third-party integrations, custom code
null / blank Background or unregistered connection — worth investigating

💡 Side Note — result_cache_max_size
If your current result_cache_max_size is around 184 MB (193298432 bytes) but Oracle 19c recommends 600M for EBS 12.2 environments, that is a tuning gap. You can update this dynamically:
alter system set result_cache_max_size = 600M scope=both;
Verify against your SGA allocation and Oracle Support Note for EBS 12.2 on 19c.

✅ Quick Investigation Workflow

  1. Run Query 2 first — get the count/status snapshot per OACORE node
  2. Run Query 3 — identify dominant wait events per node
  3. Run Query 1 — drill into the actual SQL on ACTIVE sessions of the problem node
  4. Cross-reference with module names to isolate whether the load is from UI users, concurrent programs, or integrations
Environment: Oracle EBS 12.2.x · Oracle DB 19c · RAC · Solaris 11.4 SPARC
Views Used: GV$SESSION · GV$SQL
Tags: Oracle EBS, Apps DBA, OACORE, JDBC, Performance Tuning, WebLogic, GV$SESSION

Thursday, March 26, 2026

spluk

 Oracle EBS 12.2  ·  SIEM Engineering

Complete Guide: Capturing Oracle EBS 12.2 Logs in Splunk

A production-grade reference covering all 8 tiers — from the DB alert log to DMZ extranet servers, WAF, SiteMinder, OAM WebGate, and custom automation feeds. 80+ log sources. No log left behind.

· Oracle Apps DBA Lead· Oracle Solaris 11.4 SPARC· EBS 12.2.x Production· Splunk Universal Forwarder
8+
Log layers
80+
Log sources
5
DMZ hops
24/7
Coverage
Table of Contents
  1. 01. Why Splunk for Oracle EBS?
  2. 02. Application Tier Logs
  3. 03. WebLogic Server (WLS) Logs
  4. 04. Database Tier Logs
  5. 05. OS / Solaris 11.4 Logs
  6. 06. Security & Threat Detection Logs
  7. 07. Monitoring & Automation Feeds
  8. 08. Infrastructure / Storage / Network
  9. 09. Patching / ADOP / Change Logs
  10. 10. DMZ & Extranet EBS — Full Log Strategy
  11. 11. Splunk inputs.conf Reference
  12. 12. SPL Correlation Queries
01

Why Splunk for Oracle EBS?

Oracle E-Business Suite 12.2 generates logs across a sprawling multi-tier stack — OHS, WebLogic, Oracle DB, Solaris OS, OAM, SiteMinder, ADOP, and custom automation scripts. Without centralised log aggregation, an incident that spans even two layers can take hours to diagnose.

Splunk bridges this gap by ingesting all these sources into a single searchable platform, enabling real-time alerting, cross-layer correlation, and forensic investigation.

Post-CL0P Ransomware Context: The CL0P ransomware campaign specifically targeted Oracle EBS environments via unpatched JSP endpoints. Complete Splunk coverage — especially JSP endpoint monitoring on OHS access logs and DB audit trails — is no longer optional.
02

Application Tier Logs

The application tier covers the full request lifecycle — OHS web entry point, OAF/Forms, Concurrent Manager, and ADOP patching artefacts.

LogPathSource TypePriority
Apache/OHS access$INST_TOP/logs/ora/10.1.3/Apache/access_logoracle:ebs:ohs:accessP1
Apache/OHS error$INST_TOP/logs/ora/10.1.3/Apache/error_logoracle:ebs:ohs:errorP1
OA Framework / JSP errors$INST_TOP/logs/ora/10.1.3/j2ee/oracle:ebs:oafP1
OPMN process log$INST_TOP/logs/ora/10.1.3/opmn/oracle:ebs:opmnP2
Forms server$INST_TOP/logs/ora/10.1.3/forms/oracle:ebs:formsP2
Concurrent Manager startup$APPLCSF/$APPLLOG/oracle:ebs:cm:managerP1
Concurrent request logs$APPLCSF/$APPLLOG/*.reqoracle:ebs:cm:request:logP2
Concurrent request output$APPLCSF/$APPLOUT/*.outoracle:ebs:cm:request:outP3
ADOP patch logs$NE_BASE/EBSapps/patch/oracle:ebs:adopP2
AutoConfig logs$INST_TOP/admin/log/oracle:ebs:autoconfigP3
03

WebLogic Server (WLS) Logs

WLS is the critical Java EE container underpinning EBS 12.2. These logs are the first place to look for SSO login failures, 500 errors on the external portal, OAM redirect loops, and JDBC pool exhaustion.

Real-world case: A production CORP EBS SSO login performance issue traced to a CRP Test OAM server being referenced instead of production was first visible in the WLS managed server log — not the OHS access log.
LogPathSource TypePriority
AdminServer log$DOMAIN_HOME/servers/AdminServer/logs/AdminServer.logoracle:wls:adminP1
Managed server log$DOMAIN_HOME/servers/EBS_managed*/logs/*.logoracle:wls:managedP1
WLS access log$DOMAIN_HOME/servers/*/logs/access.logoracle:wls:accessP1
OAM/SSO integration log$DOMAIN_HOME/servers/*/logs/oracle:wls:oamP1
GC / JVM heap log$FMW_HOME/../domain/logs/*.logoracle:wls:jvmP2
WLS Node Manager log$WL_HOME/common/nodemanager/*.logoracle:wls:nodemanagerP2
WLS JDBC datasource log$DOMAIN_HOME/servers/*/logs/oracle:wls:jdbcP2
WLS deployment log$DOMAIN_HOME/servers/*/logs/oracle:wls:deployP3
04

Database Tier Logs

The DB alert log is the single highest-priority log in any EBS environment. It surfaces ORA- errors, startup/shutdown events, redo switches, deadlocks, and block corruption — all in one place.

LogPathSource TypePriority
DB alert log$ORACLE_BASE/diag/rdbms/<db>/<SID>/trace/alert_<SID>.logoracle:db:alertP1
Listener log (XML)$ORACLE_BASE/diag/tnslsnr/<host>/listener/alert/log.xmloracle:db:listenerP1
DB audit trail (AUD$)$ORACLE_BASE/admin/<SID>/adump/*.audoracle:db:auditP1
FGA audit (FGA_LOG$)DB view → flat file extractoracle:db:fgaP1
Trace files$ORACLE_BASE/diag/rdbms/<db>/<SID>/trace/*.trcoracle:db:traceP1
RMAN backup log$ORACLE_BASE/admin/<SID>/log/oracle:db:rmanP2
Data Pump log$DATA_PUMP_DIR/*.logoracle:db:datapumpP3
FND_LOGINS (app audit)DB extract → flat fileoracle:ebs:fnd:loginsP1
FND_UNSUCCESSFUL_LOGINSDB extract → flat fileoracle:ebs:fnd:auth_failP1
AD_PATCH_HISTDB extract → flat fileoracle:ebs:patch:histP2
05

OS / Solaris 11.4 Logs

Oracle Solaris 11.4 SPARC has a distinct log layout from Linux. Syslog lives in /var/adm/messages, auth in /var/log/authlog, and C2/BSM audit in /var/audit/. The SMF service log is Solaris-specific and frequently missed in Splunk deployments.

LogPathSource TypePriority
Syslog / messages/var/adm/messagessolaris:syslogP1
Auth log/var/log/authlogsolaris:authP1
Cron log/var/cron/logsolaris:cronP2
Audit log (BSM/C2)/var/audit/solaris:bsmP2
ZFS / Volume manager/var/adm/messages (zpool events)solaris:zfsP2
NFS / mount events/var/adm/messagessolaris:nfsP1
SMF service log/var/svc/log/*.logsolaris:smfP2
Disk / SCSI errors/var/adm/messagessolaris:diskP1
Core dump events/var/core/solaris:coredumpP2
Network interface errorskstat / snoop logssolaris:networkP2
06

Security & Threat Detection Logs

Security-focused logs deserve their own tier. Several are DB extracts that require a scheduled export script to make them Splunk-consumable as flat files. These feed SOC dashboards, incident response playbooks, and compliance reports.

LogSourceSource TypePriority
OHS access — JSP endpointsaccess_log (filtered)oracle:ebs:ohs:accessP1
EBS FND security eventsFND_EVENTS_Q extractoracle:ebs:fnd:securityP1
SiteMinder / FCC log$OAM_HOME/../logs/oracle:oam:siteminderP1
OAM access server log$OAM_HOME/../logs/oracle:oam:accessP1
OS sudo / privilege log/var/log/authlogsolaris:sudoP1
File integrity eventsAIDE / Solaris BARTsecurity:fimP1
Network IDS alertsSnort/Suricata/Sourcefiresecurity:idsP1
Oracle AVDF / DB VaultAVDF exportoracle:avdfP2
Patch compliance gapsADOP / OEM feedoracle:ebs:patch:complianceP2
07

Monitoring & Automation Feeds

Custom automation scripts encode your team's institutional knowledge about what "healthy" looks like. Treat them as first-class Splunk sources, not afterthoughts. These feeds are especially valuable for trend analysis and proactive alerting.

FeedSourceSource TypePriority
OEM metric alertsOEM → syslog bridgeoracle:oem:alertP1
PagerDuty incident feedPagerDuty REST → HECpagerduty:incidentP2
Datadog APM spansDatadog → HECdatadog:apmP2
FlexDeploy deploy logFlexDeploy log exportflexdeploy:deployP2
ServiceNow change recordsSNOW REST feedsnow:changeP2
mount_monitor.sh output/var/log/mount_monitor.logcustom:mount_monitorP1
Batch consolidation report (72 programs)HTML email + log filecustom:batch_monitorP1
copy_clonebkp.sh exit codessyslog or log filecustom:clone_pipelineP2
CORP_QCC_RELINK outputLog filecustom:relinkP2
EBS URL extractor reportHTML email + logcustom:url_extractorP3
08

Infrastructure / Storage / Network

NTP is often overlooked: Clock drift on any EBS tier silently breaks Kerberos token validity and OAM session handling. Monitor NTP sync on ALL hosts — internal and DMZ — as a P1 operational alert.
LogSourceSource TypePriority
Storage array logNetApp/EMC syslogstorage:arrayP1
SAN switch logBrocade/Cisco FC syslogstorage:sanP1
Load balancer logF5/Oracle LBR syslognetwork:lbP1
Firewall / ACL logFirewall syslognetwork:firewallP1
DNS resolution logBIND / Unbound lognetwork:dnsP2
IPMI / iLO / ILOM hardwareIPMI sysloghardware:ipmiP1
Backup agent logVeritas/Commvault agentbackup:agentP2
NTP sync logntpd / chrony loginfra:ntpP2
09

Patching / ADOP / Change Logs

ADOP introduced online patching for EBS 12.2. Each phase generates distinct log artefacts. Capturing these phase-by-phase enables automatic change window validation and unauthorized patch detection — including out-of-window DFF recompilation events.

LogPathSource TypePriority
ADOP prepare phase$NE_BASE/EBSapps/patch/*/log/adop_*.logoracle:ebs:adop:prepareP1
ADOP apply phase$NE_BASE/EBSapps/patch/*/log/adop_*.logoracle:ebs:adop:applyP1
ADOP finalize phase$NE_BASE/EBSapps/patch/*/log/adop_*.logoracle:ebs:adop:finalizeP1
ADOP cleanup phase$NE_BASE/EBSapps/patch/*/log/adop_*.logoracle:ebs:adop:cleanupP2
ADOP worker log$NE_BASE/EBSapps/patch/*/log/worker*.logoracle:ebs:adop:workerP2
AD_PATCH_HIST extractDB extract → flat fileoracle:ebs:patch:histP1
AutoPatch log$APPL_TOP/admin/log/oracle:ebs:autopatchP2
DFF / flex compilation logConcurrent req log + AD workeroracle:ebs:dff:compileP3

10

DMZ & Extranet EBS — Full Log Strategy

This is the section most teams get wrong. The DMZ/extranet layer is not just "another OHS server" — it is a separate attack surface with distinct authentication infrastructure, network controls, and external-facing modules. A Splunk deployment that treats DMZ hosts the same as internal hosts will have critical blind spots.

DMZ Traffic Flow — Log Capture at Every Hop
External user
FW1 ①
WAF / F5 LBR
DMZ OHS (SSL term.)
OAM WebGate / SiteMinder
FW2 ②
Internal WLS / OAM
Oracle DB
Critical distinction: The DMZ OHS access log captures real external client IPs. The internal OHS access log shows only the DMZ proxy IP. Both logs are mandatory for end-to-end IP correlation during incident investigation. Tag them with different host values in inputs.conf.

Layer 1 — Perimeter / WAF / External Load Balancer

LogSourceSource TypePriority
F5 BIG-IP access logF5 syslog → Splunknetwork:lb:accessP1
F5 BIG-IP SSL logF5 syslognetwork:lb:sslP1
WAF alert logF5 ASM / ModSecuritynetwork:waf:alertP1
WAF traffic logF5 ASM / ModSecuritynetwork:waf:trafficP1
External firewall (FW1)Firewall syslognetwork:fw:externalP1

Layer 2 — DMZ OHS / Reverse Proxy (SSL Termination)

LogPathSource TypePriority
DMZ OHS access log$INST_TOP/logs/ora/10.1.3/Apache/access_log (DMZ host)oracle:ebs:dmz:ohs:accessP1
DMZ OHS error log$INST_TOP/logs/ora/10.1.3/Apache/error_log (DMZ host)oracle:ebs:dmz:ohs:errorP1
SSL/TLS error log$INST_TOP/logs/ora/10.1.3/Apache/ssl_error_logoracle:ebs:dmz:sslP1
mod_proxy / mod_rewrite logApache error log (rewrite debug)oracle:ebs:dmz:proxyP2
OHS OPMN (DMZ)$INST_TOP/logs/ora/10.1.3/opmn/ (DMZ)oracle:ebs:dmz:opmnP2

Layer 3 — Authentication: SiteMinder + OAM WebGate

LogPathSource TypePriority
SiteMinder Web Agent log$NETE_WA_ROOT/webagent.logoracle:siteminder:webagentP1
SiteMinder Policy Server log$SMPS_HOME/log/smps.logoracle:siteminder:policyP1
SiteMinder Audit log$SMPS_HOME/log/smaccess.logoracle:siteminder:auditP1
FCC (Forms Credential Collector) logSiteMinder Web Agent logoracle:siteminder:fccP1
SiteMinder session store logLDAP / SQL session DBoracle:siteminder:sessionP2
OAM WebGate log (DMZ)$WEBGATE_HOME/oblix/log/oracle:oam:webgate:dmzP1
OAM Access Server log$OAM_HOME/oblix/log/oracle:oam:accessP1
OAM Audit log$OAM_HOME/oblix/log/obaudit.logoracle:oam:auditP1
OID / LDAP access log$ORACLE_HOME/ldap/log/oracle:oid:accessP2

Layer 4 — External-Facing EBS Modules

ModuleFilter PatternSource TypePriority
iSupplier Portal/OA_HTML/OA.jsp?OAFunc=ISUPPLIER*oracle:ebs:isupplier:accessP1
XML Gateway / B2BWLS log + $INST_TOP/logsoracle:ebs:xmlgwP1
Guest / anonymous sessionsFND_LOGINS (GUEST user extract)oracle:ebs:fnd:guestP1
iRecruitmentOHS access log (filtered by function)oracle:ebs:irecruitment:accessP2
Self-Service HR (SSHR)OHS access log (filtered)oracle:ebs:sshr:accessP2
iStore / QuotingOHS access log (filtered)oracle:ebs:istore:accessP2
XML Gateway is high-risk: B2B/EDI inbound payloads can arrive unauthenticated in some configurations. Monitor for abnormal payload sizes, unexpected source IPs, and calls outside business hours.

Layer 5 — DMZ Network & Infrastructure

LogSourceSource TypePriority
Internal firewall (FW2)FW2 syslog (DMZ → internal)network:fw:internalP1
IDS / IPS alerts (DMZ)Snort/Suricata/Sourcefiresecurity:ids:dmzP1
SSL certificate expiry alertsCert manager / cron checksecurity:cert:expiryP1
DMZ switch logCisco/Juniper syslognetwork:switch:dmzP2
Reverse DNS failure logDNS server lognetwork:dns:dmzP2
NTP sync (DMZ hosts)chrony/ntpd loginfra:ntp:dmzP1

Layer 6 — DMZ OS & Bastion Host

LogPathSource TypePriority
DMZ host syslog/var/adm/messages (DMZ Solaris)solaris:syslog:dmzP1
DMZ auth log/var/log/authlog (DMZ Solaris)solaris:auth:dmzP1
Bastion / jump host auth/var/log/authlog (bastion)security:bastion:authP1
Bastion session recording/var/log/bastion/sessions/ (CyberArk/Teleport)security:bastion:sessionP1
DMZ cron log/var/cron/log (DMZ)solaris:cron:dmzP2
DMZ BSM audit/var/audit/ (DMZ)solaris:bsm:dmzP2

11

Splunk inputs.conf Reference

Representative inputs.conf snippets for both internal and DMZ Universal Forwarder deployments on Solaris 11.4.

inputs.conf — Internal Tier (Solaris UF)
# DB Alert Log
[monitor://$ORACLE_BASE/diag/rdbms/*/*/trace/alert_*.log]
index = oracle_db
sourcetype = oracle:db:alert
host = EBSPROD_DB01

# OHS Access Log
[monitor://$INST_TOP/logs/ora/10.1.3/Apache/access_log]
index = ebs_app
sourcetype = oracle:ebs:ohs:access
host = EBSPROD_APP01

# WLS Managed Server
[monitor://$DOMAIN_HOME/servers/*/logs/*.log]
index = ebs_app
sourcetype = oracle:wls:managed

# Concurrent Manager
[monitor://$APPLCSF/$APPLLOG/*]
index = ebs_batch
sourcetype = oracle:ebs:cm:manager

# ADOP Patch Logs
[monitor://$NE_BASE/EBSapps/patch/*/log/*.log]
index = ebs_change
sourcetype = oracle:ebs:adop

# Solaris OS Logs
[monitor:///var/adm/messages]
index = os_internal
sourcetype = solaris:syslog

[monitor:///var/log/authlog]
index = os_security
sourcetype = solaris:auth

# Custom Automation
[monitor:///var/log/mount_monitor.log]
index = custom_ops
sourcetype = custom:mount_monitor
inputs.conf — DMZ Tier (Solaris UF)
# DMZ OHS — separate index from internal OHS
[monitor://$INST_TOP/logs/ora/10.1.3/Apache/access_log]
index = ebs_dmz
sourcetype = oracle:ebs:dmz:ohs:access
host = EBSDMZ_OHS01

# SiteMinder Web Agent
[monitor://$NETE_WA_ROOT/webagent.log]
index = ebs_security
sourcetype = oracle:siteminder:webagent

# SiteMinder Audit
[monitor://$SMPS_HOME/log/smaccess.log]
index = ebs_security
sourcetype = oracle:siteminder:audit

# OAM WebGate (DMZ)
[monitor://$WEBGATE_HOME/oblix/log/*]
index = ebs_security
sourcetype = oracle:oam:webgate:dmz

# DMZ OS Logs
[monitor:///var/adm/messages]
index = os_dmz
sourcetype = solaris:syslog:dmz
host = EBSDMZ_HOST01

[monitor:///var/log/authlog]
index = os_dmz
sourcetype = solaris:auth:dmz
12

SPL Correlation Queries

Production-ready SPL searches for the most common cross-layer alert scenarios covering internal, DMZ, and security tiers.

Detect external IP bypassing WAF
index=ebs_dmz sourcetype=oracle:ebs:dmz:ohs:access
| where NOT src_ip IN ("<<WAF_IP_LIST>>")
| stats count by src_ip, uri
| where count > 5
FCC 500 error detection (SiteMinder)
index=ebs_security sourcetype=oracle:siteminder:webagent
  (fcc OR redirect) (error OR fail OR 500)
| timechart span=5m count by host
Brute force on extranet login
index=ebs_security sourcetype=oracle:siteminder:audit action=REJECT
| bucket _time span=5m
| stats count by src_ip, _time
| where count > 10
ORA- errors in DB alert log (P1 alert)
index=oracle_db sourcetype=oracle:db:alert
  (ORA-600 OR ORA-7445 OR ORA-4031 OR ORA-1555)
| rex field=_raw "(?P<ora_error>ORA-\d+)"
| stats count by ora_error, host
| sort -count
SSL certificate expiry < 30 days
index=infra sourcetype=security:cert:expiry
| where days_to_expiry < 30
| table cn, expiry_date, host, days_to_expiry
| sort days_to_expiry
ADOP patch applied outside change window
index=ebs_change sourcetype=oracle:ebs:adop:apply
| eval hour=strftime(_time,"%H")
| where hour < 22 AND hour > 6
| stats count by host, _time, patch_id
CM batch job failure rate (QCT production)
index=ebs_batch sourcetype=oracle:ebs:cm:request:log
  completion_status=ERROR
| timechart span=1h count as failures
| where failures > 5