Oracle EBS R12.2 Workflow Notification Mailer Down — A Step-by-Step Root Cause Analysis Guide

A production-safe, read-only-first troubleshooting methodology for Oracle E-Business Suite R12.2 Workflow Notification Mailer failures. No blind restarts — find the real root cause first.

---

One of the most common production incidents an Oracle Apps DBA faces is:

“Workflow Notification Mailer is DOWN / Stopped with Error — users are not receiving workflow emails.”

The typical (wrong) reaction is to restart the mailer immediately. It may come back up for a few minutes and stop again — because the real root cause (an expired password, a renewed SSL certificate, a firewall change) was never fixed. This post walks through a structured RCA approach: collect evidence first, classify the failure, fix the cause, and only then restart.

Environment assumed: Oracle EBS R12.2, Oracle Database 19c, Linux/UNIX application tier, Workflow Mailer managed through Oracle Applications Manager (OAM) / Generic Service Management (GSM).

---

The Notification Flow (Know What You Are Debugging)

Business Event
   → WF_DEFERRED queue
      → Workflow Agent Listener
         → WF_NOTIFICATION_OUT queue
            → Workflow Notification Mailer
               → SMTP Server / Mail Relay
                  → User Mailbox

A failure at any stage produces “no emails” — but each stage leaves different evidence. Our job is to find where the chain is broken before touching anything.

---

Phase 1 — Establish Current State (Read-Only)

1.1 Status of all Workflow service components

Run as APPS:

SET LINESIZE 200 PAGESIZE 100
COL component_name FORMAT A45
COL component_status FORMAT A18

SELECT component_id,
       component_name,
       component_status,
       component_status_info,
       startup_mode
FROM   fnd_svc_components
ORDER  BY component_id;

Expected: Notification Mailer and all Agent Listeners = RUNNING, startup mode AUTOMATIC.

Abnormal: STOPPED_ERROR, DEACTIVATED_SYSTEM, or stuck in STARTING. Note the mailer's COMPONENT_ID, and read COMPONENT_STATUS_INFO carefully — it often contains the actual Java exception text.

1.2 Container status and hosting node (multi-node aware)

SELECT cq.concurrent_queue_name,
       cq.target_node,
       DECODE(cp.process_status_code,'A','ACTIVE','R','RUNNING',
              'T','TERMINATING','K','TERMINATED','S','STOPPED',
              cp.process_status_code) status,
       cp.concurrent_process_id,
       cp.logfile_name
FROM   fnd_concurrent_queues cq,
       fnd_concurrent_processes cp
WHERE  cp.concurrent_queue_id (+) = cq.concurrent_queue_id
AND    cp.process_status_code (+) = 'A'
AND    cq.concurrent_queue_name IN ('WFMLRSVC','WFALSNRSVC');

WFMLRSVC = Mailer Service container, WFALSNRSVC = Agent Listener container. If there is no active process for the container, the problem is at the Java/GSM level — not the mail server.

1.3 Identify the exact current Mailer log (never guess the filename)

SELECT fcp.concurrent_process_id,
       fcp.logfile_name,
       fcp.node_name,
       fcp.process_start_date
FROM   fnd_concurrent_processes fcp,
       fnd_concurrent_queues fcq
WHERE  fcp.concurrent_queue_id = fcq.concurrent_queue_id
AND    fcq.concurrent_queue_name = 'WFMLRSVC'
ORDER  BY fcp.process_start_date DESC
FETCH FIRST 3 ROWS ONLY;

Then confirm on the OS (as the applmgr user, on the node returned above):

cd $APPLCSF/$APPLLOG
ls -ltr FNDCPGSC*.txt | tail -5

The most recently modified FNDCPGSC*.txt matching the process ID is your live mailer log. You can also cross-check in OAM → Workflow Manager → Notification Mailer → View Log.

---

Phase 2 — Extract the Failure Signature

On the application node, sweep the log for classic failure patterns using a portable shell loop:

LOGF=FNDCPGSC<nnnnn>.txt   # from Phase 1.3

tail -200 $LOGF

for p in "ERROR" "Exception" "SMTP" "IMAP" \
         "AuthenticationFailedException" "Connection refused" \
         "Connection timed out" "UnknownHostException" \
         "SSLHandshakeException" "PKIX" \
         "unable to find valid certification path" \
         "MessagingException" "javax.mail"
do
  echo "===== $p ====="
  grep -n "$p" $LOGF | tail -10
done

To see the context around a hit at line N:

sed -n '<N-20>,<N+20>p' $LOGF

---

Phase 3 — Database Evidence: Configuration, Backlog, Queues

3.1 Mailer configuration parameters

SELECT p.parameter_name,
       DECODE(p.parameter_name,'MAILPASSWORD','*****',
              'INBOUND_PASSWORD','*****', v.parameter_value) parameter_value
FROM   fnd_svc_comp_param_vals v,
       fnd_svc_comp_params_b  p,
       fnd_svc_components     c
WHERE  c.component_type = 'WF_MAILER'
AND    v.component_id   = c.component_id
AND    v.parameter_id   = p.parameter_id
AND    p.parameter_name IN
       ('OUTBOUND_SERVER','SMTP_OUT_PORT','OUTBOUND_SSL_ENABLED',
        'SSL_TRUSTSTORE','INBOUND_SERVER','INBOUND_PORT',
        'INBOUND_SSL_ENABLED','ACCOUNT','FROM','REPLYTO',
        'NODENAME','TEST_ADDRESS','MAX_ERROR_COUNT')
ORDER  BY p.parameter_name;

Tip: If TEST_ADDRESS is populated, all notifications are silently diverted to that single address — a classic “mailer is RUNNING but nobody gets email” cause (typically left over from a clone).

3.2 Notification backlog

SELECT mail_status, status, COUNT(*)
FROM   wf_notifications
GROUP  BY mail_status, status
ORDER  BY 3 DESC;

Result	Meaning
Large, growing MAIL_STATUS = 'MAIL'	Notifications queued; mailer not draining them
MAIL_STATUS = 'FAILED' rows	Delivery attempted and rejected by the mail server
MAIL_STATUS NULL for affected users	User notification preference issue — not a mailer fault

3.3 Queue depths across the flow

SELECT 'WF_DEFERRED' q, msg_state, COUNT(*)
FROM applsys.aq$wf_deferred GROUP BY msg_state
UNION ALL
SELECT 'WF_NOTIFICATION_OUT', msg_state, COUNT(*)
FROM applsys.aq$wf_notification_out GROUP BY msg_state
UNION ALL
SELECT 'WF_ERROR', msg_state, COUNT(*)
FROM applsys.aq$wf_error GROUP BY msg_state;

Pattern	Where the chain is broken
WF_DEFERRED READY huge, WF_NOTIFICATION_OUT small	Agent Listener stage (upstream of mailer)
WF_NOTIFICATION_OUT READY growing	Mailer stage — consistent with STOPPED_ERROR
Queues draining but no mail arrives	Mail server / relay / quarantine (external)

Also verify no queue was accidentally disabled:

SELECT name, enqueue_enabled, dequeue_enabled
FROM   dba_queues
WHERE  owner = 'APPLSYS'
AND    name IN ('WF_DEFERRED','WF_NOTIFICATION_OUT','WF_NOTIFICATION_IN','WF_ERROR');

All should be YES / YES.

---

Phase 4 — Network and Mail Server Tests (From the App Node)

Use the exact host and port parameters retrieved in Phase 3.1:

# DNS resolution
host <smtp_server>
nslookup <smtp_server>

# TCP reachability
telnet <smtp_server> 25        # or 587 / 465

# Manual SMTP handshake — proves relay permission
telnet <smtp_server> 25
EHLO <appnode_fqdn>
MAIL FROM:<workflow_account@domain>
RCPT TO:<your_address@domain>
QUIT

# SSL/TLS certificate validation
openssl s_client -connect <smtp_server>:465
openssl s_client -connect <smtp_server>:587 -starttls smtp

# IMAP (if inbound processing is enabled)
telnet <imap_server> 143
openssl s_client -connect <imap_server>:993

Test result	Meaning
DNS lookup fails	DNS issue (check resolv.conf / name servers)
Connection refused	Wrong port, or SMTP service down
Telnet hangs, then times out	Firewall blocking (common after security policy changes)
550 relay denied on RCPT TO	App server IP/name not whitelisted on the mail relay
openssl verify error / incomplete chain	Certificate ↔ truststore problem
protocol_version / handshake alert	TLS version mismatch (JDK runtime vs mail server profile)

---

Phase 5 — Root Cause Classification Matrix

Match the dominant log signature to the problem area:

Log signature	Root cause area	Typical real-world trigger
javax.mail.AuthenticationFailedException, 535 5.7.x	Mail account / auth	Password rotated or expired; basic auth disabled on Exchange Online
Connection refused	SMTP service/port	Relay down, port changed, wrong OUTBOUND_SERVER
Connection timed out	Firewall / network	New firewall rule; relay IP migrated
UnknownHostException	DNS	Hostname decommissioned; DNS server change
SSLHandshakeException, PKIX path building failed	SSL cert ↔ Java truststore	Mail server certificate renewed with a new CA chain
handshake_failure, protocol_version	TLS/cipher mismatch	Mail infra enforcing TLS 1.2+; older JDK on app tier
ORA-25xxx, dequeue errors	Workflow AQ queues	Queue disabled or subscriber issue (often post-clone)
Container won't start, OutOfMemoryError	Java / GSM container	JVM heap sizing; Service Manager (FNDSM) down on the node
Mailer RUNNING, backlog grows, no errors	Mailer configuration	TEST_ADDRESS set; wrong NODENAME
Mail accepted (250 OK) but never arrives	External mail infra	Anti-spam, SPF/DKIM policy, quarantine, journaling
WF_DEFERRED growing, listeners STOPPED_ERROR	Workflow / Agent Listener	Failing event subscription hitting max error count

Field observation: The two most frequent production root causes for a mailer that “worked fine for months and suddenly stopped” are:

1. A renewed SSL certificate on the corporate mail relay causing SSLHandshakeException / PKIX errors.
2. A rotated or expired mail account password causing AuthenticationFailedException.

---

Phase 6 — Safest Recovery Sequence

Fix the cause first, then recover with the smallest possible scope:

1. Remediate the root cause:
   • Certificate/truststore: Import the new CA chain into the truststore the mailer uses (the SSL_TRUSTSTORE parameter location, otherwise the JDK cacerts on the application tier) using keytool -import. In R12.2, perform this on the run file system and remember to propagate or sync to the patch file system during the next fs_clone.
   • Password: Update it directly through OAM → Notification Mailer → Edit — never update FND tables manually.
   • Firewall/DNS/relay: Have the network or mail team fix it, then re-run the Phase 4 tests to confirm connectivity before restarting services.
2. Restart only the Notification Mailer component via OAM Workflow Manager.
3. If the component will not start cleanly, restart the WFMLRSVC container only.
4. Only if GSM itself is implicated, bounce Concurrent Managers (adcmctl.sh) during an approved window.
5. Queue rebuild scripts (e.g., the notification queue rebuild procedure) are a last resort — run only under Oracle Support guidance matching your exact symptoms. Never run destructive queue cleanup casually, and never update Workflow tables directly.

---

Phase 7 — Post-Recovery Verification (End to End)

-- 1. Mailer is RUNNING
SELECT component_name, component_status
FROM   fnd_svc_components
WHERE  component_type = 'WF_MAILER';

-- 2. Backlog is draining (run twice, 10 minutes apart — count must decrease)
SELECT COUNT(*) FROM wf_notifications WHERE mail_status = 'MAIL';

-- 3. Outbound queue is draining
SELECT msg_state, COUNT(*)
FROM   applsys.aq$wf_notification_out
GROUP  BY msg_state;

Then run a controlled end-to-end verification:

1. OAM → Workflow Manager → Notification Mailer → Test Mailer (or trigger any standard workflow notification).
2. Confirm the notification's MAIL_STATUS changes from MAIL to SENT.
3. Confirm the email arrives in the destination inbox (check junk and quarantine folders too).
4. Monitor the live log for 15–30 minutes: tail -f $APPLCSF/$APPLLOG/FNDCPGSC<new>.txt — verify there are no exceptions and only periodic processing messages appear.
5. Confirm Agent Listeners remain RUNNING and WF_DEFERRED is not accumulating.

---

Key Takeaways

• Never restart first. A restart destroys nothing but often hides critical runtime evidence and wastes outage windows.
• The mailer log filename must be identified from the database, not guessed.
• The queue-depth pattern tells you which stage of the flow is broken before you read a single log line.
• 80% of sudden mailer failures trace back to changes outside EBS: passwords, certificates, firewalls, DNS, or mail platform migrations.
• All fixes go through OAM / supported procedures — never direct table updates, never unsupported queue purges.

If this helped you resolve a production Workflow Mailer outage, share your failure signature in the comments — the classification matrix above grows with every real-world case.