ADOP Failure Investigation in Oracle EBS 12.2: A Production Troubleshooting Runbook
Online patching failures in Oracle E-Business Suite Release 12.2 are rarely resolved by searching for the word “ERROR” in one log file. An ADOP session coordinates database editions, application-tier file systems, patch workers, WebLogic components and multiple application nodes.
A reliable investigation must answer four questions:
- Which ADOP phase failed?
- What was the first actionable error?
- Is the problem at the database, file-system, patch-worker or node level?
- Can the current session be restarted safely, or must it be aborted?
This article presents a structured Apps DBA workflow for investigating ADOP failures without immediately jumping to destructive recovery actions.
1. Understand the ADOP Failure Boundary
An online patching cycle normally contains the following phases:
prepare → apply → finalize → cutover → cleanupEach phase has a different failure profile.
| ADOP phase | Common problem areas |
|---|---|
| Prepare | Run/patch file-system synchronization, context mismatch, patch edition creation, database connectivity |
| Apply | Patch driver failure, failed workers, invalid objects, missing files, prerequisite patches |
| Finalize | Invalid objects, editioned-object validation, compilation or readiness checks |
| Cutover | Service shutdown/startup, WebLogic failure, database edition switch, insufficient cutover time |
| Cleanup | Old editions, obsolete objects, database space, blocking sessions |
| fs_clone | File-system space, permissions, context mismatch, node connectivity, incomplete previous synchronization |
Always identify the failing phase before attempting recovery.
2. Confirm the Correct EBS Environment
Before running any diagnostic command, confirm that the environment points to the expected instance and file-system edition.
echo $CONTEXT_FILE
echo $FILE_EDITION
echo $RUN_BASE
echo $PATCH_BASE
echo $APPL_TOP
echo $AD_TOPExpected considerations:
- Normal ADOP phases are initiated from the run file system.
- Patch-side investigation may require sourcing the patch environment.
- The database SID, context file and application node must belong to the same environment.
- In a multi-node environment, verify the environment on every participating node.
A surprising number of troubleshooting mistakes occur because the DBA investigates the wrong edition or sources an environment left over from another instance.
3. Check the Current ADOP Session
Start with the session status:
adop -statusThis identifies whether:
- An online patching cycle is active
- A phase is complete, running or failed
- A previous session requires attention
- The system is ready for a new patching cycle
Do not start another prepare phase merely because no ADOP process is visible at the operating-system level. The database may still record an active or incomplete patching session.
For a quick configuration and online-patching health check, run:
adop -validateValidation can expose configuration, edition or synchronization problems that may not be obvious from the failed patch log alone.
4. Scan the Logs Before Opening Individual Files
ADOP creates logs across multiple directories, phases, nodes and workers. Instead of manually searching hundreds of files, begin with adopscanlog.
Scan the latest session:
adopscanlogDisplay error-level messages:
adopscanlog loglevel=errorScan all available sessions:
adopscanlog session_id=0Depending on the EBS code level, supported arguments and output can vary. Check the local help before building automation around a specific syntax:
adopscanlog -helpThe objective is not to collect every warning. It is to locate the first meaningful failure that triggered the later cascade of errors.
For example, messages such as the following may be secondary symptoms:
Worker failed
Phase failed
ADOP exiting with status 1The real cause may appear earlier:
ORA-01652: unable to extend temp segment
ORA-04021: timeout occurred while waiting to lock object
Permission denied
No space left on device
Patch prerequisite is missing
Invalid username/passwordAlways investigate upward from the final failure message.
5. Locate the ADOP Log Directory
The non-editioned file system stores ADOP logs beneath:
$NE_BASE/EBSapps/log/adopList the most recently updated files:
find $NE_BASE/EBSapps/log/adop -type f -exec ls -lt {} \; | headA typical structure separates logs by:
- ADOP session
- Timestamp
- Phase
- Application node
- Patch number
- Worker
When a multi-node session fails, do not inspect only the primary node. The controlling ADOP log may report that another node failed while the actual Java, WebLogic, file-copy or patch error exists only in the remote-node log.
Useful searches include:
grep -i "error" logfile
grep -i "failed" logfile
grep -i "ORA-" logfile
grep -i "adop exit status" logfile
grep -i "not found" logfile
grep -i "permission denied" logfileAvoid assuming every line containing “error” is fatal. Some patch logs include expected exceptions or informational error counters. Correlate the message with the phase result and timestamp.
6. Monitor a Running Session with ADOPMON
For long-running phases, use:
adopmonadopmon provides a continuously refreshed view of important online-patching actions. It is useful during:
- Large patch applications
- Finalize
- Cutover
- Multi-node patching
- Sessions that appear to be hanging
Treat “no visible progress” carefully. An ADOP operation may be waiting on:
- A database lock
- A long SQL statement
- Object compilation
- A remote application node
- A WebLogic operation
- A patch worker
- File-system synchronization
Before stopping any process, confirm whether work is still active at the database and operating-system levels.
7. Investigate Failed Patch Workers
An apply-phase failure frequently originates from one or more AD workers.
Check the worker status using the AD Controller utility:
adctrlTypical actions include:
- Display worker status
- Review the failed worker
- Restart a worker after correcting the cause
- Skip a failed job only when explicitly permitted by the patch documentation or Oracle Support
Review the worker log before restarting it. Common worker-level failures include:
- SQL compilation errors
- Object locks
- Missing grants or synonyms
- Tablespace exhaustion
- Invalid custom objects
- Failed form or report generation
- File permission problems
- Incorrect product-top configuration
Do not repeatedly restart a worker without resolving the underlying error. Repeated attempts usually add noise while leaving the actual condition unchanged.
8. Query AD_ZD_LOGS for Database-Side Evidence
Online patching also stores diagnostic information in database tables. AD_ZD_LOGS can help when application-tier logs are incomplete or when the error originated inside the database.
Connect using an approved secure method:
sqlplus /nologCONNECT appsReview recent entries:
SET LINESIZE 220
SET PAGESIZE 100
SET LONG 100000
SET LONGCHUNKSIZE 100000
SELECT log_sequence,
TO_CHAR(log_timestamp, 'YYYY-MM-DD HH24:MI:SS') log_time,
message_text
FROM ad_zd_logs
ORDER BY log_sequence DESC
FETCH FIRST 100 ROWS ONLY;Search for likely failure messages:
SELECT log_sequence,
TO_CHAR(log_timestamp, 'YYYY-MM-DD HH24:MI:SS') log_time,
message_text
FROM ad_zd_logs
WHERE UPPER(message_text) LIKE '%ERROR%'
OR UPPER(message_text) LIKE '%FAILED%'
OR UPPER(message_text) LIKE '%ORA-%'
ORDER BY log_sequence DESC;Capturing the latest sequence before reproducing an issue makes it easier to isolate new messages:
SELECT MAX(log_sequence)
FROM ad_zd_logs;After reproducing the failure:
SELECT log_sequence,
TO_CHAR(log_timestamp, 'YYYY-MM-DD HH24:MI:SS') log_time,
message_text
FROM ad_zd_logs
WHERE log_sequence > :starting_sequence
ORDER BY log_sequence;Use a sequence captured from the same environment and incident window. An arbitrary historical sequence value can produce misleading output.
9. Use ADZDSHOWLOG.sql
Oracle supplies a SQL script for displaying online-patching log details:
sqlplus /nologCONNECT apps
@$AD_TOP/sql/ADZDSHOWLOG.sqlThis is particularly helpful when:
- The application-tier message is generic
- A database editioning operation failed
- The log hierarchy is difficult to navigate
- Oracle Support requests database-side online-patching evidence
Run the script from the correct application environment so that $AD_TOP resolves to the intended EBS instance.
10. Generate an Online-Patching Diagnostic Report
The adzdreport.pl utility can collect detailed information about online-patching and editioning status.
Check the local usage first:
$AD_TOP/bin/adzdreport.pl -helpRun the utility according to the syntax supported by your EBS code level.
Avoid putting the APPS password directly in a shared shell script, command history, email or incident ticket. If a utility requires credentials, enter them interactively or use the secure method approved by your organization.
The report is useful for identifying:
- Editioning inconsistencies
- AD_ZD component problems
- Run and patch edition status
- Online-patching object issues
- Conditions that require deeper Oracle Support analysis
Because the output may contain hostnames, paths and configuration details, review it before sharing outside the DBA or support team.
11. Perform Database Health Checks
An ADOP failure may be a database-capacity or concurrency problem rather than a patching-tool defect.
Check invalid objects
SELECT owner,
object_type,
COUNT(*) invalid_count
FROM dba_objects
WHERE status = 'INVALID'
GROUP BY owner, object_type
ORDER BY owner, object_type;Do not treat every invalid object as an ADOP failure. Compare the list with the pre-patching baseline and focus on newly invalid objects related to the failing patch.
Check tablespace usage
SELECT tablespace_name,
ROUND(used_percent, 2) used_percent
FROM dba_tablespace_usage_metrics
ORDER BY used_percent DESC;Also verify:
- TEMP space
- UNDO space and retention
- Archive destination availability
- FRA usage, where applicable
- Datafile autoextend limits
- Operating-system file-system usage
Check blocking sessions
SELECT sid,
serial#,
username,
status,
event,
blocking_session,
seconds_in_wait
FROM v$session
WHERE blocking_session IS NOT NULL
ORDER BY seconds_in_wait DESC;Do not kill a blocking session solely because it appears in this query. Identify the owner, business operation and transaction impact before taking action.
12. Check File-System and Node Health
On every application node, verify:
df -gOn Linux, use:
df -hAlso verify:
hostname
date
ulimit -aCheck the ownership and permissions of:
$RUN_BASE$PATCH_BASE$NE_BASE$APPL_TOP$COMMON_TOP$FMW_HOME- Patch staging directories
- Temporary directories
In multi-node systems, confirm:
- All required nodes are reachable
- Passwordless SSH is working where required
- Context files contain the correct node information
- Patch files exist in the same
$PATCH_TOPlocation on non-shared file systems - Clocks are synchronized
- Shared mounts are available
- No node has a full local file system
A primary-node log may only say that a remote operation failed. The actual error can be in the remote node’s log or operating-system event log.
13. Build a Failure Timeline
A disciplined incident timeline is more useful than a large ZIP file containing every log.
Capture:
ADOP session ID
Patch number
Failed phase
Failure timestamp
Application node
Failed worker number
First actionable error
Database alert-log timestamp
Corrective action
Restart timestamp
Final validation resultCorrelate events across:
- Main ADOP log
- Patch driver log
- Worker log
- AD_ZD_LOGS
- Database alert log
- Listener log
- WebLogic logs
- OHS logs
- Operating-system logs
Use timestamps to distinguish the root cause from the cleanup messages generated after the failure.
14. Restart or Abort: Make the Correct Decision
Restart the existing session when:
- The root cause is understood
- The condition is reversible
- Disk space, permission, lock or connectivity issues have been corrected
- The patch documentation supports restart
- The ADOP session remains recoverable
Restart the failed phase using the appropriate ADOP command. ADOP maintains restart information and can resume supported operations after the underlying problem is corrected.
Consider abort only when:
- The cycle has not entered cutover
- The failure cannot be resolved within the maintenance plan
- The patch edition must be abandoned
- The decision has been reviewed by the Apps DBA lead and change owner
- The recovery sequence is understood
Oracle documents that abort is available before cutover. A normal recovery sequence is:
adop phase=abort
adop phase=cleanup cleanup_mode=full
adop phase=fs_cloneAbort and cleanup may also be combined where appropriate:
adop phase=abort,cleanup cleanup_mode=fullAfter an abort, full cleanup is required. If patch application was attempted, synchronize the patch file system using fs_clone before beginning another online-patching cycle.
Do not run abort, cleanup or forced fs_clone merely as a generic response to an error. These are recovery operations, not diagnostic commands.
15. Special Caution After Cutover Starts
Once cutover begins, the operational situation changes significantly.
Do not assume that adop phase=abort can roll back a cutover. Oracle documents that abort is available only before cutover is initiated.
If cutover fails:
- Preserve all logs.
- Determine whether the database edition switch occurred.
- Check the status of application services.
- Review the cutover logs on every node.
- Confirm the active run and patch file systems.
- Follow the applicable patch README and Oracle Support guidance.
- Avoid manually changing edition metadata or ADOP tables.
A failed cutover requires controlled recovery because application availability and file-system edition state may already have changed.
16. Post-Recovery Validation
After the issue is corrected, do not stop at “ADOP completed successfully.”
Run:
adop -status
adop -validateThen verify:
- Required ADOP phase completed
- No failed workers remain
- Application services are running
- Login page is available
- Forms and OAF pages open
- Concurrent Managers are operating
- Workflow Mailer and other critical services are healthy
- No unexpected invalid objects remain
- Database and application logs contain no new critical errors
- Run and patch file systems are synchronized as required
- Patch level matches the intended change
For cutover incidents, also perform a business smoke test with the application team.
17. Production-Safe Command Checklist
echo $CONTEXT_FILE
echo $FILE_EDITION
echo $APPL_TOP
echo $AD_TOP
echo $NE_BASE
adop -status
adop -validate
adopscanlog
adopscanlog loglevel=error
adopmon
df -g
ulimit -aDatabase-side tools:
@$AD_TOP/sql/ADZDSHOWLOG.sqlDiagnostic utility:
$AD_TOP/bin/adzdreport.pl -helpRecovery commands—use only after diagnosis and approval:
adop phase=abort
adop phase=cleanup cleanup_mode=full
adop phase=fs_clone18. Common Troubleshooting Mistakes
Avoid the following practices:
- Starting another prepare phase without checking
adop -status - Searching only the final ADOP log
- Ignoring remote-node logs
- Restarting failed workers without correcting the cause
- Killing database sessions without identifying the transaction owner
- Using
skipsyncerror=yeswithout proving that a subsequent patch will resolve the synchronization failure - Running abort or forced
fs_cloneas a first response - Updating ADOP or AD_ZD tables manually
- Supplying APPS passwords on the command line or saving them in scripts
- Deleting patching logs before completing the RCA
- Treating every invalid object as patch-related
- Declaring success without application-level validation
Conclusion
Effective ADOP troubleshooting is not about knowing one special command. It is about collecting evidence in the correct sequence.
A strong Apps DBA workflow is:
Confirm environment
↓
Check ADOP session and phase
↓
Scan consolidated logs
↓
Locate the first actionable error
↓
Correlate worker, database and node evidence
↓
Correct the underlying condition
↓
Restart when recoverable
↓
Abort only through a controlled decision
↓
Validate the complete EBS serviceTools such as adop -status, adop -validate, adopscanlog, adopmon, ADZDSHOWLOG.sql, AD_ZD_LOGS and adzdreport.pl become most valuable when used together as part of a repeatable incident runbook.
