72 Golden Rules Every Oracle DBA and Enterprise Architect Should Live By

Battle-tested operational principles for Oracle E-Business Suite, Core DBA, RAC, ASM, Data Guard, RMAN, performance forensics, incident response, and engineering leadership.

Production systems rarely fail because somebody forgot a command. They fail because a technically valid command was executed at the wrong time, against the wrong target, without sufficient evidence or without a safe rollback path.

These golden rules are built around the most common operational anti-pattern: taking a premature, drastic, or misdirected action before establishing the actual cause. They are deliberately punchy, but each one reflects a serious lesson from running mission-critical Oracle environments.

The governing principle: Evidence first. Hypothesis second. Controlled action third. Validation always.

1. Apps DBA — Oracle E-Business Suite R12/12.2

This section covers Concurrent Managers, online patching, AutoConfig, WebLogic, application-tier services, and cloning.

1. We don’t bounce the entire EBS application tier until we identify which service, managed server, or node is actually unhealthy! 🎯

2. We don’t restart Concurrent Managers until we inspect FND_CONCURRENT_REQUESTS, FND_CONCURRENT_QUEUES, FND_CONCURRENT_PROCESSES, and the Internal Manager log! 🔍

3. We don’t blame the Internal Concurrent Manager for pending requests until incompatibilities, specialization rules, work shifts, target processes, and node assignments are validated! 🧭

4. We don’t terminate a long-running concurrent request until its database session, SQL ID, wait event, blocking chain, and business criticality are confirmed! 🛑

5. We don’t delete rows directly from FND_CONCURRENT_REQUESTS when Oracle provides supported purge programs and retention controls! 🧹

6. We don’t run adop phase=abort until the current session, failed phase, worker logs, AD_ADOP_SESSIONS, and recovery options are understood! 🚧

7. We don’t execute adop cleanup_mode=full as a magic broom until the failed patching cycle and filesystem synchronization state are documented! 🪄

8. We don’t apply an EBS patch because its README looks friendly until prerequisites, supersedence, ETCC results, code levels, and interoperability notes are checked! 📚

9. We don’t use adpatch casually in an online-patching-enabled EBS 12.2 environment when the patch belongs inside an adop cycle! ⚠️

10. We don’t run AutoConfig everywhere until the correct context file, RUN filesystem, node role, shared filesystem design, and pending configuration changes are verified! 🧩

11. We don’t edit generated files under $INST_TOP, OHS, WebLogic, or application configuration directories until we know whether AutoConfig will overwrite them! ✍️

12. We don’t declare an EBS clone successful when the login page opens until Concurrent Managers, Workflow Mailer, OPP, forms, integrations, printers, profiles, database links, and scheduled jobs are validated! 🧪

2. Core DBA and Architecture

These rules address RAC, ASM, storage, RMAN, Data Guard, disaster recovery, and platform architecture.

13. We don’t restart a RAC database until crsctl stat res -t, srvctl status database, instance health, services, and cluster interconnect symptoms identify the failing layer! 🧠

14. We don’t relocate a RAC service during an incident until connection pools, transaction affinity, FAN/TAF behavior, and surviving-instance capacity are confirmed! 🔀

15. We don’t evict a RAC node manually until CSS, voting-disk, interconnect, and OS evidence explains why the cluster is threatening to do it for us! 🗳️

16. We don’t add ASM disks until failure groups, allocation-unit size, rebalance power, usable capacity, and storage-path redundancy are validated! 💿

17. We don’t drop an ASM disk until V$ASM_OPERATION, rebalance status, redundancy, and partner-disk health prove the data is safely redistributed! 🧨

18. We don’t blame ASM for latency until iostat, sar, V$ASM_DISK_IOSTAT, database wait events, multipathing, and storage-array metrics tell the same story! 📊

19. We don’t increase DB_FILE_MULTIBLOCK_READ_COUNT to cure slow storage until execution plans and actual I/O latency prove that multiblock reads are the problem! 🐢

20. We don’t activate a Data Guard standby until the failover decision, redo gap, data-loss exposure, application fencing, and no-return point are formally accepted! 🚨

21. We don’t restart managed recovery until V$ARCHIVE_GAP, V$DATAGUARD_STATUS, transport errors, standby redo logs, and broker state are inspected! 📡

22. We don’t call a backup successful because RMAN returned exit code zero until logs, backup-piece availability, retention, control-file protection, and restore validation are checked! 🎭

23. We don’t trust an untested backup when RESTORE VALIDATE, VALIDATE DATABASE, block checks, and a recovery rehearsal have never met it! 🧯

24. We don’t design DR around an RPO/RTO PowerPoint until bandwidth, redo rate, restore throughput, dependency sequencing, DNS, certificates, and application recovery are measured! ⏱️

3. Performance Tuning and Forensics

Performance engineering begins with workload evidence—not parameter roulette.

25. We don’t tune CPU when the database is waiting on I/O, locks, commits, network responses, or application think time! 🩺

26. We don’t tune a SQL statement from elapsed time alone until DB time, CPU time, wait profile, executions, rows processed, and business context are known! 🔬

27. We don’t add an index until predicates, selectivity, clustering factor, DML overhead, plan alternatives, and existing index coverage are evaluated! 🗂️

28. We don’t drop an allegedly unused index until DBA_HIST_SQL_PLAN, index-monitoring limitations, reporting cycles, and emergency workloads have been considered! 🪦

29. We don’t flush the shared pool when one SQL statement misbehaves until child cursors, mutex waits, invalidations, bind behavior, and dependency churn are understood! 🚿

30. We don’t purge a SQL plan from the cursor cache until we have captured its SQL ID, plan hash value, outline, bind information, and reproducible evidence! 📸

31. We don’t gather schema-wide statistics during peak hours until stale objects, sampling strategy, histograms, incremental statistics, and plan-change risk are assessed! 🎲

32. We don’t delete histograms because bind peeking looks suspicious until V$SQL_SHARED_CURSOR, adaptive cursor sharing, column skew, and workload diversity are verified! 📐

33. We don’t increase SGA_TARGET, PGA_AGGREGATE_TARGET, or HugePages until swapping, paging, NUMA placement, PGA spills, and OS memory headroom are measured! 🧠

34. We don’t treat db file sequential read as automatically bad until latency, call volume, access path, storage tier, and rows returned per execution are correlated! 🧵

35. We don’t blame log file sync entirely on storage until commit frequency, log file parallel write, redo allocation, LGWR CPU scheduling, and application commit design are compared! 🧾

36. We don’t use one AWR snapshot pair to explain an intermittent incident until ASH, baselines, peak intervals, time models, and OS telemetry reconstruct the actual timeline! 🕵️

4. Troubleshooting and Root Cause Analysis

Troubleshooting is the disciplined elimination of possibilities—not a contest to see who can restart something first.

37. We don’t change a parameter until we can state the hypothesis, expected metric movement, validation window, and rollback command! 🧪

38. We don’t troubleshoot five layers simultaneously until the failure is isolated across client, load balancer, OHS, WebLogic, database, network, and OS boundaries! 🧅

39. We don’t blame the database because the application says “database error” until listener logs, JDBC errors, connection-pool state, SQL*Net evidence, and database alert logs agree! 🙃

40. We don’t blame the network because ping is slow until TCP throughput, packet loss, MTU, retransmissions, routing, firewall inspection, and application-port tests are measured! 🌐

41. We don’t accept “the server is slow” until CPU utilization, run queue, paging, filesystem latency, network errors, and top processes are timestamped! 🖥️

42. We don’t kill a blocker until V$SESSION, V$LOCK, DBA_BLOCKERS, DBA_WAITERS, transaction age, object ownership, and rollback cost are reviewed! 🔒

43. We don’t kill an OS process with kill -9 until graceful database or application termination has failed and process identity is proven beyond PID coincidence! ☠️

44. We don’t enable event 10046 or broad SQL tracing in production until scope, level, duration, trace-file growth, and performance overhead are controlled! 🔦

45. We don’t enable system-wide debug logging until targeted component logging has failed to capture the evidence and filesystem capacity can survive the experiment! 🪵

46. We don’t declare corruption from one ORA-01578 until DBV, RMAN validation, V$DATABASE_BLOCK_CORRUPTION, object mapping, and storage evidence confirm its scope! 🧱

47. We don’t call an issue intermittent when timestamps, time zones, request IDs, SQL IDs, hostnames, session identifiers, and correlation IDs were never captured! ⌚

48. We don’t close an incident because the symptom disappeared until the trigger, failure mechanism, corrective action, and recurrence-detection method are documented! 🧾

5. Crisis Management and War Rooms

During a critical incident, uncontrolled technical activity can become a second outage layered on top of the first.

49. We don’t start a SEV1 bridge without one incident commander, one technical lead, one communications owner, and one timestamped action log! 🎖️

50. We don’t let ten engineers execute ten ideas until hypotheses are ranked, owners are assigned, and mutually conflicting actions are stopped! 🚦

51. We don’t restart everything because executives joined the bridge until the failing component and expected recovery mechanism are identified! 🎪

52. We don’t make a production change during a crisis until the exact command, target, blast radius, success signal, and rollback path are read back! 📣

53. We don’t combine multiple fixes in one emergency change until each action’s effect can still be isolated and reversed! 🧬

54. We don’t fail over to DR until the primary is fenced and split-brain, data-loss, DNS, routing, integration, and reconciliation risks are accepted! ⚔️

55. We don’t promote a standby because replication is “almost caught up” until the business signs off on the exact recoverable SCN or timestamp! 🕰️

56. We don’t restore a database over the suspected source until forensic evidence, logs, control files, and recovery artifacts are safely preserved! 🧊

57. We don’t announce recovery when the homepage responds until transactions, batch processing, integrations, authentication, monitoring, and data consistency pass validation! 🟢

58. We don’t allow stakeholder pressure to redefine technical truth until metrics demonstrate stability across an agreed observation window! 🌡️

59. We don’t end the bridge until temporary workarounds, monitoring thresholds, ownership, next update, and customer-impact statements are recorded! 📋

60. We don’t call rollback a failure when rollback was the engineered control that prevented a larger outage! 🪂

6. Engineering Leadership and Operational Culture

Reliable platforms require more than technical skill. They require disciplined automation, meaningful ownership, useful documentation, and a culture that learns from failure.

61. We don’t automate a broken process until its inputs, ownership, exception paths, and desired outcome are understood! 🤖

62. We don’t automate destructive commands until dry-run mode, target validation, logging, idempotency, failure handling, and recovery controls exist! 🛡️

63. We don’t replace human toil with silent cron failures until alerting, exit-code handling, lock files, retention, and operational ownership are built in! ⏰

64. We don’t call a script production-ready until it handles spaces, nulls, timeouts, concurrent execution, partial failure, credentials, and reruns safely! 🧰

65. We don’t accept a dashboard full of green boxes until every metric has a source, threshold, timestamp, owner, and actionable response! 🚥

66. We don’t write an RCA around who clicked the button until we explain why one click could bypass review, safeguards, testing, or rollback! 🫵

67. We don’t conduct a blameless retrospective without evidence, because “blameless” does not mean “factless”! 🔎

68. We don’t close a problem record with “human error” until process design, access control, automation gaps, training, and workload conditions are examined! 🧑‍⚖️

69. We don’t carry a manual operational step forever when its frequency, error rate, recovery cost, and automation value justify eliminating the toil! 🏗️

70. We don’t postpone technical debt indefinitely until the accumulated outage risk, security exposure, delivery drag, and support cost are visible to decision-makers! 💳

71. We don’t approve architecture by diagram beauty until failure modes, capacity limits, observability, operability, upgrade paths, and ownership are demonstrated! 🏛️

72. We don’t call knowledge transfer complete until another engineer can diagnose, operate, recover, and safely change the platform without calling its original builder! 🧑‍🚀

Final Takeaway

Production does not reward confidence without evidence.
It rewards engineers who preserve evidence, control the blast radius, understand dependencies, prepare rollback options, and validate the complete business service—not merely the component they restarted.

The best Oracle DBA is not the person who remembers the most commands. It is the person who knows which command should be executed, why it should be executed, what could go wrong, how to reverse it, and how to prove that the system is genuinely healthy afterward.

Before touching production, ask four questions:

• What evidence supports the proposed action?
• What is the expected technical outcome?
• What is the blast radius if the hypothesis is wrong?
• How will we roll back and validate recovery?

If those questions cannot be answered clearly, the next command should probably be a diagnostic command.

---

Topics: Oracle DBA, Oracle E-Business Suite, Apps DBA, RAC, ASM, Data Guard, RMAN, Performance Tuning, Incident Management, AWR, ASH and Enterprise Architecture