Oracle EBS R12.2 Workflow Notification Mailer Stuck in Starting – Troubleshooting and RCA Guide
In Oracle E-Business Suite R12.2, the Workflow Notification Mailer is responsible for sending and receiving workflow email notifications.
A common issue is that the mailer remains in:
Starting
instead of moving to:
Running
If the status remains in Starting for several minutes, the mailer has usually encountered an initialization problem.
This article provides a generic troubleshooting approach to identify the root cause before making configuration changes.
1. Check Workflow Mailer Status
Navigate to:
Oracle Applications Manager
↓
Workflow Manager
↓
Service Components
Filter using:
Type (Internal) = WF_MAILER
Check the status of:
Workflow Notification Mailer
Typical statuses include:
Running
Stopped
Starting
Stopping
Error
If the mailer remains in Starting, continue with the following checks.
2. Check View Log
Select:
Workflow Notification Mailer
and click:
View Log
Look at the log entries corresponding to the exact time when the mailer was started.
Search for errors such as:
ERROR
Exception
Failed
Authentication
Connection
Timeout
SSL
PKIX
UnknownHost
OutOfMemory
The first exception generated during startup is usually more useful for RCA than the final generic error.
3. Check View Event History
From the same Service Components page, select the Workflow Notification Mailer and click:
View Event History
Review events such as:
Start requested
Component starting
Component stopped
Component error
Unexpected termination
Event history helps determine whether the component:
never started
started and immediately failed
repeatedly restarted
was manually stopped
was terminated by the service container
4. Locate Workflow Mailer Log Files
Source the Oracle EBS RUN filesystem environment.
For example:
. EBSapps.env run
Check the log location:
echo $APPLCSF
echo $APPLLOG
Workflow Java/GSM component logs commonly appear as:
$APPLCSF/$APPLLOG/FNDCPGSC*.txt
List the latest files:
ls -ltr $APPLCSF/$APPLLOG/FNDCPGSC*.txt | tail
Check the latest log:
tail -300 <latest_FNDCPGSC_file>
Search for errors:
egrep -i "error|exception|failed|authentication|ssl|connect|timeout|unknownhost|outofmemory|pkix" \
<latest_FNDCPGSC_file>
5. Check Workflow Mailer Component Status from Database
Connect as the APPS user.
set lines 200
col component_name format a40
col component_status format a20
col startup_mode format a15
select component_id,
component_name,
component_status,
startup_mode
from fnd_svc_components
where component_type = 'WF_MAILER';
Example:
COMPONENT_ID COMPONENT_NAME COMPONENT_STATUS
------------ ------------------------------ ----------------
10006 Workflow Notification Mailer STARTING
Record the:
COMPONENT_ID
for additional investigation.
6. Check Mailer Configuration
Review the mailer configuration from:
Workflow Manager
↓
Service Components
↓
Workflow Notification Mailer
↓
Edit
Verify parameters related to:
SMTP Server
SMTP Port
IMAP Server
IMAP Port
Mailbox Username
Reply-To Address
Inbound Processing
Outbound Processing
SSL/TLS configuration
Be particularly careful after:
Clone
Refresh
Migration
DR activation
Environment build
Network migration
Mail-server migration
A cloned environment may still contain configuration inherited from another environment.
7. Verify Non-Encrypted Parameters from Database
The following query can be used to review component parameters that are not encrypted.
Replace the component ID with the value obtained earlier.
set lines 220
col parameter_name format a40
col parameter_display_name format a45
col parameter_value format a60
select p.parameter_name,
v.parameter_display_name,
v.parameter_value
from fnd_svc_comp_param_vals v,
fnd_svc_comp_params_b p
where v.parameter_id = p.parameter_id
and v.component_id = <COMPONENT_ID>
and nvl(p.encrypted_flag,'N') = 'N'
order by p.parameter_name;
Check whether the configuration matches the intended environment.
8. Verify Concurrent Processing and GSM
Workflow Notification Mailer is managed through Oracle EBS Generic Service Management.
Check the relevant operating-system processes:
ps -ef | grep FNDLIBR | grep -v grep
Check Service Manager:
ps -ef | grep FNDSM | grep -v grep
The following components should also be verified from Oracle EBS:
Internal Concurrent Manager
Service Manager
Workflow Mailer Service
Workflow Agent Listener Service
Navigate to:
System Administrator
↓
Concurrent
↓
Manager
↓
Administer
If the Service Manager or GSM infrastructure is unavailable, the Workflow Mailer may not start correctly.
9. Check DNS Resolution
From the application server where the mailer is running, verify that the mail servers resolve correctly.
host <smtp_server>
and:
host <imap_server>
Alternatively:
nslookup <smtp_server>
nslookup <imap_server>
If the hostname cannot be resolved, investigate:
DNS
/etc/hosts
Network configuration
Incorrect mail server hostname
10. Test SMTP Connectivity
For traditional SMTP:
telnet <smtp_server> 25
For SMTP using STARTTLS:
openssl s_client -starttls smtp \
-connect <smtp_server>:587 \
-servername <smtp_server>
A successful connection indicates that the application server can reach the SMTP service.
11. Test IMAP Connectivity
For IMAPS:
openssl s_client \
-connect <imap_server>:993 \
-servername <imap_server>
For standard IMAP:
telnet <imap_server> 143
If the connection times out, investigate the network before changing Workflow Mailer configuration.
12. Understand Common Errors
Authentication Failure
Example:
AuthenticationFailedException
or:
535 Authentication failed
Possible causes:
Incorrect mailbox password
Password expired
Mailbox account locked
SMTP authentication disabled
Authentication policy changed
Connection Timeout
Example:
Connection timed out
Possible causes:
Firewall blocking the port
Network routing issue
Incorrect hostname
Incorrect port
Mail server unreachable
Connection Refused
Example:
Connection refused
Possible causes:
Incorrect port
SMTP/IMAP service stopped
Mail server not listening on the configured port
Load balancer issue
Unknown Host
Example:
UnknownHostException
Possible causes:
DNS resolution failure
Incorrect hostname
Missing DNS entry
Incorrect /etc/hosts entry
SSL Handshake Failure
Example:
SSLHandshakeException
Possible causes:
Certificate problem
Unsupported TLS protocol
Cipher mismatch
Expired certificate
Missing certificate chain
PKIX Error
Example:
PKIX path building failed
This normally indicates that Java cannot establish trust with the certificate presented by the mail server.
Check:
Mail server certificate
Intermediate certificates
Root certificate
Java trust store
Oracle EBS certificate configuration
JVM Memory Problem
Example:
java.lang.OutOfMemoryError
Investigate:
Mailer JVM memory
Large email attachments
Very large notification messages
Abnormal mail queue
JVM configuration
13. Check Workflow Agent Listener
The Workflow Agent Listener should also be checked because Workflow Mailer processing depends on Workflow event processing.
Query service components:
set lines 200
select component_id,
component_name,
component_status,
startup_mode
from fnd_svc_components
order by component_name;
Look for components related to:
Workflow Mailer
Workflow Agent Listener
Verify that required Workflow services are operational.
14. Check Workflow Queues
Check whether Workflow queues are accumulating messages.
Example:
select count(*)
from wf_deferred;
Check notification status:
select status,
mail_status,
count(*)
from wf_notifications
group by status,
mail_status
order by status,
mail_status;
A large number of pending notifications can help identify whether the problem is:
Mailer startup
Outbound email processing
Workflow queue processing
SMTP delivery
15. Check Pending Notifications
For a more detailed view:
select notification_id,
message_type,
message_name,
recipient_role,
status,
mail_status,
begin_date
from wf_notifications
where mail_status = 'MAIL'
order by begin_date;
Avoid changing Workflow notification data directly unless instructed by Oracle Support or supported documentation.
16. Restart Only After Identifying the Error
Repeatedly restarting the Workflow Mailer without checking the logs normally does not resolve the underlying issue.
Use the supported Workflow Manager interface:
Workflow Manager
↓
Service Components
↓
Workflow Notification Mailer
Perform:
Stop
Wait until the component reaches:
Stopped
Then:
Start
Monitor:
View Log
View Event History
FNDCPGSC log
17. Do Not Force the Status from Database
Avoid directly updating tables such as:
FND_SVC_COMPONENTS
to artificially change:
STARTING
to:
RUNNING
Changing the database status does not start the underlying Java component and can make troubleshooting more difficult.
Use supported Workflow Service Component controls.
18. Run Workflow Mailer Diagnostic Tests
Oracle EBS provides Workflow diagnostic utilities.
Run the relevant Workflow Mailer diagnostic and validate:
Mailer configuration
SMTP configuration
IMAP configuration
Service component configuration
Workflow services
Diagnostics are particularly useful when the configuration looks correct but the component still fails during initialization.
19. Send a Test Notification
Once the mailer reaches:
Running
send a test notification.
From Workflow Manager use the available mailer test functionality.
Validate:
Notification generated
Notification dequeued
SMTP connection established
Mail delivered
Recipient received email
For inbound processing, also reply to the notification and validate the inbound mail path.
Recommended Troubleshooting Sequence
The following sequence normally provides the fastest RCA:
1. Check Mailer status
↓
2. View Log
↓
3. View Event History
↓
4. Find latest FNDCPGSC log
↓
5. Identify first Java exception
↓
6. Check GSM / Service Manager
↓
7. Verify SMTP/IMAP configuration
↓
8. Verify DNS
↓
9. Test SMTP/IMAP connectivity
↓
10. Check SSL/TLS certificates
↓
11. Check authentication
↓
12. Check Workflow queues
↓
13. Correct root cause
↓
14. Restart Workflow Mailer
↓
15. Send test notification
Quick RCA Matrix
| Symptom / Error | Likely Root Cause | Area to Check |
|---|---|---|
| Mailer remains Starting | Initialization failure | FNDCPGSC log |
| AuthenticationFailedException | Invalid mailbox credentials | Mail account |
| 535 Authentication Failed | SMTP authentication | SMTP server/account |
| Connection timed out | Firewall/network | Network |
| Connection refused | Service/port unavailable | Mail server |
| UnknownHostException | DNS problem | DNS |
| SSLHandshakeException | TLS/certificate issue | Certificates |
| PKIX path building failed | Certificate not trusted | Java trust store |
| OutOfMemoryError | Mailer JVM memory | JVM |
| No mailer Java process | GSM/Service Manager | Concurrent Processing |
| Mailer Running but no emails | Queue/SMTP issue | WF queues/SMTP |
| Outbound works, inbound fails | IMAP configuration | IMAP/mailbox |
| Problem starts after clone | Source configuration retained | Post-clone configuration |
Useful Commands
Check latest GSM logs
ls -ltr $APPLCSF/$APPLLOG/FNDCPGSC*.txt | tail
Check errors
egrep -i "error|exception|failed|authentication|ssl|connect|timeout|unknownhost|outofmemory|pkix" \
$APPLCSF/$APPLLOG/FNDCPGSC*.txt | tail -100
Check FNDLIBR
ps -ef | grep FNDLIBR | grep -v grep
Check Service Manager
ps -ef | grep FNDSM | grep -v grep
Check SMTP
openssl s_client -starttls smtp \
-connect <smtp_server>:587 \
-servername <smtp_server>
Check IMAPS
openssl s_client \
-connect <imap_server>:993 \
-servername <imap_server>
RCA Example
A proper RCA should identify the complete failure chain.
For example:
Issue:
Workflow Notification Mailer remained in STARTING status.
Observation:
FNDCPGSC log reported connection timeout while establishing
connection to the configured SMTP server.
Root Cause:
The application server was unable to establish connectivity to
the SMTP server on the configured port because the network
firewall rule was missing.
Resolution:
The required network connectivity was enabled between the
Oracle EBS application server and SMTP server.
Validation:
SMTP connectivity was successfully tested from the application
server. Workflow Notification Mailer was restarted and changed
to RUNNING status. A test notification was successfully delivered.
Preventive Action:
SMTP and IMAP connectivity checks were added to the
post-clone/environment validation checklist.
Conclusion
When an Oracle EBS Workflow Notification Mailer is stuck in Starting, avoid immediately restarting services or modifying component tables.
The most effective troubleshooting approach is:
FNDCPGSC log
↓
First exception
↓
GSM status
↓
SMTP/IMAP configuration
↓
Network/DNS
↓
Authentication
↓
SSL/TLS
↓
Workflow queues
In most cases, the first meaningful exception in the Workflow Mailer log provides the direction for the RCA.
This method can be used across Oracle E-Business Suite R12.2 environments including Development, Test, UAT, Production, DR and post-clone environments.
No comments:
Post a Comment