Complete Guide: Capturing Oracle EBS 12.2 Logs in Splunk
A production-grade reference covering all 8 tiers — from the DB alert log to DMZ extranet servers, WAF, SiteMinder, OAM WebGate, and custom automation feeds. 80+ log sources. No log left behind.
· Oracle Apps DBA Lead· Oracle Solaris 11.4 SPARC· EBS 12.2.x Production· Splunk Universal Forwarder
Oracle E-Business Suite 12.2 generates logs across a sprawling multi-tier stack — OHS, WebLogic, Oracle DB, Solaris OS, OAM, SiteMinder, ADOP, and custom automation scripts. Without centralised log aggregation, an incident that spans even two layers can take hours to diagnose.
Splunk bridges this gap by ingesting all these sources into a single searchable platform, enabling real-time alerting, cross-layer correlation, and forensic investigation.
Post-CL0P Ransomware Context: The CL0P ransomware campaign specifically targeted Oracle EBS environments via unpatched JSP endpoints. Complete Splunk coverage — especially JSP endpoint monitoring on OHS access logs and DB audit trails — is no longer optional.
The application tier covers the full request lifecycle — OHS web entry point, OAF/Forms, Concurrent Manager, and ADOP patching artefacts.
| Log | Path | Source Type | Priority |
|---|
| Apache/OHS access | $INST_TOP/logs/ora/10.1.3/Apache/access_log | oracle:ebs:ohs:access | P1 |
| Apache/OHS error | $INST_TOP/logs/ora/10.1.3/Apache/error_log | oracle:ebs:ohs:error | P1 |
| OA Framework / JSP errors | $INST_TOP/logs/ora/10.1.3/j2ee/ | oracle:ebs:oaf | P1 |
| OPMN process log | $INST_TOP/logs/ora/10.1.3/opmn/ | oracle:ebs:opmn | P2 |
| Forms server | $INST_TOP/logs/ora/10.1.3/forms/ | oracle:ebs:forms | P2 |
| Concurrent Manager startup | $APPLCSF/$APPLLOG/ | oracle:ebs:cm:manager | P1 |
| Concurrent request logs | $APPLCSF/$APPLLOG/*.req | oracle:ebs:cm:request:log | P2 |
| Concurrent request output | $APPLCSF/$APPLOUT/*.out | oracle:ebs:cm:request:out | P3 |
| ADOP patch logs | $NE_BASE/EBSapps/patch/ | oracle:ebs:adop | P2 |
| AutoConfig logs | $INST_TOP/admin/log/ | oracle:ebs:autoconfig | P3 |
WLS is the critical Java EE container underpinning EBS 12.2. These logs are the first place to look for SSO login failures, 500 errors on the external portal, OAM redirect loops, and JDBC pool exhaustion.
Real-world case: A production CORP EBS SSO login performance issue traced to a CRP Test OAM server being referenced instead of production was first visible in the WLS managed server log — not the OHS access log.
| Log | Path | Source Type | Priority |
|---|
| AdminServer log | $DOMAIN_HOME/servers/AdminServer/logs/AdminServer.log | oracle:wls:admin | P1 |
| Managed server log | $DOMAIN_HOME/servers/EBS_managed*/logs/*.log | oracle:wls:managed | P1 |
| WLS access log | $DOMAIN_HOME/servers/*/logs/access.log | oracle:wls:access | P1 |
| OAM/SSO integration log | $DOMAIN_HOME/servers/*/logs/ | oracle:wls:oam | P1 |
| GC / JVM heap log | $FMW_HOME/../domain/logs/*.log | oracle:wls:jvm | P2 |
| WLS Node Manager log | $WL_HOME/common/nodemanager/*.log | oracle:wls:nodemanager | P2 |
| WLS JDBC datasource log | $DOMAIN_HOME/servers/*/logs/ | oracle:wls:jdbc | P2 |
| WLS deployment log | $DOMAIN_HOME/servers/*/logs/ | oracle:wls:deploy | P3 |
The DB alert log is the single highest-priority log in any EBS environment. It surfaces ORA- errors, startup/shutdown events, redo switches, deadlocks, and block corruption — all in one place.
| Log | Path | Source Type | Priority |
|---|
| DB alert log | $ORACLE_BASE/diag/rdbms/<db>/<SID>/trace/alert_<SID>.log | oracle:db:alert | P1 |
| Listener log (XML) | $ORACLE_BASE/diag/tnslsnr/<host>/listener/alert/log.xml | oracle:db:listener | P1 |
| DB audit trail (AUD$) | $ORACLE_BASE/admin/<SID>/adump/*.aud | oracle:db:audit | P1 |
| FGA audit (FGA_LOG$) | DB view → flat file extract | oracle:db:fga | P1 |
| Trace files | $ORACLE_BASE/diag/rdbms/<db>/<SID>/trace/*.trc | oracle:db:trace | P1 |
| RMAN backup log | $ORACLE_BASE/admin/<SID>/log/ | oracle:db:rman | P2 |
| Data Pump log | $DATA_PUMP_DIR/*.log | oracle:db:datapump | P3 |
| FND_LOGINS (app audit) | DB extract → flat file | oracle:ebs:fnd:logins | P1 |
| FND_UNSUCCESSFUL_LOGINS | DB extract → flat file | oracle:ebs:fnd:auth_fail | P1 |
| AD_PATCH_HIST | DB extract → flat file | oracle:ebs:patch:hist | P2 |
Oracle Solaris 11.4 SPARC has a distinct log layout from Linux. Syslog lives in /var/adm/messages, auth in /var/log/authlog, and C2/BSM audit in /var/audit/. The SMF service log is Solaris-specific and frequently missed in Splunk deployments.
| Log | Path | Source Type | Priority |
|---|
| Syslog / messages | /var/adm/messages | solaris:syslog | P1 |
| Auth log | /var/log/authlog | solaris:auth | P1 |
| Cron log | /var/cron/log | solaris:cron | P2 |
| Audit log (BSM/C2) | /var/audit/ | solaris:bsm | P2 |
| ZFS / Volume manager | /var/adm/messages (zpool events) | solaris:zfs | P2 |
| NFS / mount events | /var/adm/messages | solaris:nfs | P1 |
| SMF service log | /var/svc/log/*.log | solaris:smf | P2 |
| Disk / SCSI errors | /var/adm/messages | solaris:disk | P1 |
| Core dump events | /var/core/ | solaris:coredump | P2 |
| Network interface errors | kstat / snoop logs | solaris:network | P2 |
Security-focused logs deserve their own tier. Several are DB extracts that require a scheduled export script to make them Splunk-consumable as flat files. These feed SOC dashboards, incident response playbooks, and compliance reports.
| Log | Source | Source Type | Priority |
|---|
| OHS access — JSP endpoints | access_log (filtered) | oracle:ebs:ohs:access | P1 |
| EBS FND security events | FND_EVENTS_Q extract | oracle:ebs:fnd:security | P1 |
| SiteMinder / FCC log | $OAM_HOME/../logs/ | oracle:oam:siteminder | P1 |
| OAM access server log | $OAM_HOME/../logs/ | oracle:oam:access | P1 |
| OS sudo / privilege log | /var/log/authlog | solaris:sudo | P1 |
| File integrity events | AIDE / Solaris BART | security:fim | P1 |
| Network IDS alerts | Snort/Suricata/Sourcefire | security:ids | P1 |
| Oracle AVDF / DB Vault | AVDF export | oracle:avdf | P2 |
| Patch compliance gaps | ADOP / OEM feed | oracle:ebs:patch:compliance | P2 |
Custom automation scripts encode your team's institutional knowledge about what "healthy" looks like. Treat them as first-class Splunk sources, not afterthoughts. These feeds are especially valuable for trend analysis and proactive alerting.
| Feed | Source | Source Type | Priority |
|---|
| OEM metric alerts | OEM → syslog bridge | oracle:oem:alert | P1 |
| PagerDuty incident feed | PagerDuty REST → HEC | pagerduty:incident | P2 |
| Datadog APM spans | Datadog → HEC | datadog:apm | P2 |
| FlexDeploy deploy log | FlexDeploy log export | flexdeploy:deploy | P2 |
| ServiceNow change records | SNOW REST feed | snow:change | P2 |
| mount_monitor.sh output | /var/log/mount_monitor.log | custom:mount_monitor | P1 |
| Batch consolidation report (72 programs) | HTML email + log file | custom:batch_monitor | P1 |
| copy_clonebkp.sh exit codes | syslog or log file | custom:clone_pipeline | P2 |
| CORP_QCC_RELINK output | Log file | custom:relink | P2 |
| EBS URL extractor report | HTML email + log | custom:url_extractor | P3 |
NTP is often overlooked: Clock drift on any EBS tier silently breaks Kerberos token validity and OAM session handling. Monitor NTP sync on ALL hosts — internal and DMZ — as a P1 operational alert.
| Log | Source | Source Type | Priority |
|---|
| Storage array log | NetApp/EMC syslog | storage:array | P1 |
| SAN switch log | Brocade/Cisco FC syslog | storage:san | P1 |
| Load balancer log | F5/Oracle LBR syslog | network:lb | P1 |
| Firewall / ACL log | Firewall syslog | network:firewall | P1 |
| DNS resolution log | BIND / Unbound log | network:dns | P2 |
| IPMI / iLO / ILOM hardware | IPMI syslog | hardware:ipmi | P1 |
| Backup agent log | Veritas/Commvault agent | backup:agent | P2 |
| NTP sync log | ntpd / chrony log | infra:ntp | P2 |
ADOP introduced online patching for EBS 12.2. Each phase generates distinct log artefacts. Capturing these phase-by-phase enables automatic change window validation and unauthorized patch detection — including out-of-window DFF recompilation events.
| Log | Path | Source Type | Priority |
|---|
| ADOP prepare phase | $NE_BASE/EBSapps/patch/*/log/adop_*.log | oracle:ebs:adop:prepare | P1 |
| ADOP apply phase | $NE_BASE/EBSapps/patch/*/log/adop_*.log | oracle:ebs:adop:apply | P1 |
| ADOP finalize phase | $NE_BASE/EBSapps/patch/*/log/adop_*.log | oracle:ebs:adop:finalize | P1 |
| ADOP cleanup phase | $NE_BASE/EBSapps/patch/*/log/adop_*.log | oracle:ebs:adop:cleanup | P2 |
| ADOP worker log | $NE_BASE/EBSapps/patch/*/log/worker*.log | oracle:ebs:adop:worker | P2 |
| AD_PATCH_HIST extract | DB extract → flat file | oracle:ebs:patch:hist | P1 |
| AutoPatch log | $APPL_TOP/admin/log/ | oracle:ebs:autopatch | P2 |
| DFF / flex compilation log | Concurrent req log + AD worker | oracle:ebs:dff:compile | P3 |
This is the section most teams get wrong. The DMZ/extranet layer is not just "another OHS server" — it is a separate attack surface with distinct authentication infrastructure, network controls, and external-facing modules. A Splunk deployment that treats DMZ hosts the same as internal hosts will have critical blind spots.
DMZ Traffic Flow — Log Capture at Every Hop
External user
→FW1 ①
→WAF / F5 LBR
→DMZ OHS (SSL term.)
→OAM WebGate / SiteMinder
→FW2 ②
→Internal WLS / OAM
→Oracle DB
Critical distinction: The DMZ OHS access log captures real external client IPs. The internal OHS access log shows only the DMZ proxy IP. Both logs are mandatory for end-to-end IP correlation during incident investigation. Tag them with different host values in inputs.conf.
Layer 1 — Perimeter / WAF / External Load Balancer
| Log | Source | Source Type | Priority |
|---|
| F5 BIG-IP access log | F5 syslog → Splunk | network:lb:access | P1 |
| F5 BIG-IP SSL log | F5 syslog | network:lb:ssl | P1 |
| WAF alert log | F5 ASM / ModSecurity | network:waf:alert | P1 |
| WAF traffic log | F5 ASM / ModSecurity | network:waf:traffic | P1 |
| External firewall (FW1) | Firewall syslog | network:fw:external | P1 |
Layer 2 — DMZ OHS / Reverse Proxy (SSL Termination)
| Log | Path | Source Type | Priority |
|---|
| DMZ OHS access log | $INST_TOP/logs/ora/10.1.3/Apache/access_log (DMZ host) | oracle:ebs:dmz:ohs:access | P1 |
| DMZ OHS error log | $INST_TOP/logs/ora/10.1.3/Apache/error_log (DMZ host) | oracle:ebs:dmz:ohs:error | P1 |
| SSL/TLS error log | $INST_TOP/logs/ora/10.1.3/Apache/ssl_error_log | oracle:ebs:dmz:ssl | P1 |
| mod_proxy / mod_rewrite log | Apache error log (rewrite debug) | oracle:ebs:dmz:proxy | P2 |
| OHS OPMN (DMZ) | $INST_TOP/logs/ora/10.1.3/opmn/ (DMZ) | oracle:ebs:dmz:opmn | P2 |
Layer 3 — Authentication: SiteMinder + OAM WebGate
| Log | Path | Source Type | Priority |
|---|
| SiteMinder Web Agent log | $NETE_WA_ROOT/webagent.log | oracle:siteminder:webagent | P1 |
| SiteMinder Policy Server log | $SMPS_HOME/log/smps.log | oracle:siteminder:policy | P1 |
| SiteMinder Audit log | $SMPS_HOME/log/smaccess.log | oracle:siteminder:audit | P1 |
| FCC (Forms Credential Collector) log | SiteMinder Web Agent log | oracle:siteminder:fcc | P1 |
| SiteMinder session store log | LDAP / SQL session DB | oracle:siteminder:session | P2 |
| OAM WebGate log (DMZ) | $WEBGATE_HOME/oblix/log/ | oracle:oam:webgate:dmz | P1 |
| OAM Access Server log | $OAM_HOME/oblix/log/ | oracle:oam:access | P1 |
| OAM Audit log | $OAM_HOME/oblix/log/obaudit.log | oracle:oam:audit | P1 |
| OID / LDAP access log | $ORACLE_HOME/ldap/log/ | oracle:oid:access | P2 |
Layer 4 — External-Facing EBS Modules
| Module | Filter Pattern | Source Type | Priority |
|---|
| iSupplier Portal | /OA_HTML/OA.jsp?OAFunc=ISUPPLIER* | oracle:ebs:isupplier:access | P1 |
| XML Gateway / B2B | WLS log + $INST_TOP/logs | oracle:ebs:xmlgw | P1 |
| Guest / anonymous sessions | FND_LOGINS (GUEST user extract) | oracle:ebs:fnd:guest | P1 |
| iRecruitment | OHS access log (filtered by function) | oracle:ebs:irecruitment:access | P2 |
| Self-Service HR (SSHR) | OHS access log (filtered) | oracle:ebs:sshr:access | P2 |
| iStore / Quoting | OHS access log (filtered) | oracle:ebs:istore:access | P2 |
XML Gateway is high-risk: B2B/EDI inbound payloads can arrive unauthenticated in some configurations. Monitor for abnormal payload sizes, unexpected source IPs, and calls outside business hours.
Layer 5 — DMZ Network & Infrastructure
| Log | Source | Source Type | Priority |
|---|
| Internal firewall (FW2) | FW2 syslog (DMZ → internal) | network:fw:internal | P1 |
| IDS / IPS alerts (DMZ) | Snort/Suricata/Sourcefire | security:ids:dmz | P1 |
| SSL certificate expiry alerts | Cert manager / cron check | security:cert:expiry | P1 |
| DMZ switch log | Cisco/Juniper syslog | network:switch:dmz | P2 |
| Reverse DNS failure log | DNS server log | network:dns:dmz | P2 |
| NTP sync (DMZ hosts) | chrony/ntpd log | infra:ntp:dmz | P1 |
Layer 6 — DMZ OS & Bastion Host
| Log | Path | Source Type | Priority |
|---|
| DMZ host syslog | /var/adm/messages (DMZ Solaris) | solaris:syslog:dmz | P1 |
| DMZ auth log | /var/log/authlog (DMZ Solaris) | solaris:auth:dmz | P1 |
| Bastion / jump host auth | /var/log/authlog (bastion) | security:bastion:auth | P1 |
| Bastion session recording | /var/log/bastion/sessions/ (CyberArk/Teleport) | security:bastion:session | P1 |
| DMZ cron log | /var/cron/log (DMZ) | solaris:cron:dmz | P2 |
| DMZ BSM audit | /var/audit/ (DMZ) | solaris:bsm:dmz | P2 |
Representative inputs.conf snippets for both internal and DMZ Universal Forwarder deployments on Solaris 11.4.
inputs.conf — Internal Tier (Solaris UF)# DB Alert Log
[monitor://$ORACLE_BASE/diag/rdbms/*/*/trace/alert_*.log]
index = oracle_db
sourcetype = oracle:db:alert
host = EBSPROD_DB01
# OHS Access Log
[monitor://$INST_TOP/logs/ora/10.1.3/Apache/access_log]
index = ebs_app
sourcetype = oracle:ebs:ohs:access
host = EBSPROD_APP01
# WLS Managed Server
[monitor://$DOMAIN_HOME/servers/*/logs/*.log]
index = ebs_app
sourcetype = oracle:wls:managed
# Concurrent Manager
[monitor://$APPLCSF/$APPLLOG/*]
index = ebs_batch
sourcetype = oracle:ebs:cm:manager
# ADOP Patch Logs
[monitor://$NE_BASE/EBSapps/patch/*/log/*.log]
index = ebs_change
sourcetype = oracle:ebs:adop
# Solaris OS Logs
[monitor:///var/adm/messages]
index = os_internal
sourcetype = solaris:syslog
[monitor:///var/log/authlog]
index = os_security
sourcetype = solaris:auth
# Custom Automation
[monitor:///var/log/mount_monitor.log]
index = custom_ops
sourcetype = custom:mount_monitor
inputs.conf — DMZ Tier (Solaris UF)# DMZ OHS — separate index from internal OHS
[monitor://$INST_TOP/logs/ora/10.1.3/Apache/access_log]
index = ebs_dmz
sourcetype = oracle:ebs:dmz:ohs:access
host = EBSDMZ_OHS01
# SiteMinder Web Agent
[monitor://$NETE_WA_ROOT/webagent.log]
index = ebs_security
sourcetype = oracle:siteminder:webagent
# SiteMinder Audit
[monitor://$SMPS_HOME/log/smaccess.log]
index = ebs_security
sourcetype = oracle:siteminder:audit
# OAM WebGate (DMZ)
[monitor://$WEBGATE_HOME/oblix/log/*]
index = ebs_security
sourcetype = oracle:oam:webgate:dmz
# DMZ OS Logs
[monitor:///var/adm/messages]
index = os_dmz
sourcetype = solaris:syslog:dmz
host = EBSDMZ_HOST01
[monitor:///var/log/authlog]
index = os_dmz
sourcetype = solaris:auth:dmz
Production-ready SPL searches for the most common cross-layer alert scenarios covering internal, DMZ, and security tiers.
Detect external IP bypassing WAFindex=ebs_dmz sourcetype=oracle:ebs:dmz:ohs:access
| where NOT src_ip IN ("<<WAF_IP_LIST>>")
| stats count by src_ip, uri
| where count > 5 FCC 500 error detection (SiteMinder)index=ebs_security sourcetype=oracle:siteminder:webagent
(fcc OR redirect) (error OR fail OR 500)
| timechart span=5m count by host
Brute force on extranet loginindex=ebs_security sourcetype=oracle:siteminder:audit action=REJECT
| bucket _time span=5m
| stats count by src_ip, _time
| where count > 10
ORA- errors in DB alert log (P1 alert)index=oracle_db sourcetype=oracle:db:alert
(ORA-600 OR ORA-7445 OR ORA-4031 OR ORA-1555)
| rex field=_raw "(?P<ora_error>ORA-\d+)"
| stats count by ora_error, host
| sort -count
SSL certificate expiry < 30 daysindex=infra sourcetype=security:cert:expiry
| where days_to_expiry < 30
| table cn, expiry_date, host, days_to_expiry
| sort days_to_expiry
ADOP patch applied outside change windowindex=ebs_change sourcetype=oracle:ebs:adop:apply
| eval hour=strftime(_time,"%H")
| where hour < 22 AND hour > 6
| stats count by host, _time, patch_id
CM batch job failure rate (QCT production)index=ebs_batch sourcetype=oracle:ebs:cm:request:log
completion_status=ERROR
| timechart span=1h count as failures
| where failures > 5
No comments:
Post a Comment