Thursday, September 8, 2011

How to Perform a Health Check on the Database


How to Perform a Health Check on the Database [ID 122669.1]

Modified 24-MAR-2011 Type BULLETIN Status PUBLISHED

Applies to:

Oracle Server - Enterprise Edition - Version: 7.3.4.0 to 11.2.0.1 - Release: 7.3.4 to 11.2
Information in this document applies to any platform.

Purpose

This article explains how to perform a Basic Health Check on the database verifying
several configuration issues. General guidelines are given on what areas to investigate
to get a better overview on how the database is working and evolving. These guidelines
will reveal common issues regarding configuration as well as problems that may occur in the future.
For a more in depth health check to check Database structure and data dictionary integrity,
please follow the appropriate links in chapter 11.
The areas investigated here are mostly based on scripts and are brought to you without
any warranty, these scripts may need to be adapted for next database releases and features.
This article will probably need to be extended to serve specific application need0s/checks.
Although some performance areas are discussed in this article, it is not the intention
of this article to give a full detailed explanation of optimizing the database performance.

Scope and Application

1. Parameter file
2. Controlfiles
3. Redolog files
4. Archiving
5. Datafiles
5.1 Autoextend
5.2 Location
6. Tablespaces
6.1 SYSTEM Tablespace
6.2 SYSAUX Tablespace (10g Release and above)
6.3 Locally vs Dictionary Managed Tablespaces
6.4 Temporary Tablespace
6.5 Tablespace Fragmentation
7. Objects
7.1 Number of Extents
7.2 Next extent
7.3 Indexes
8. AUTO vs MANUAL undo
8.1 AUTO UNDO
8.2 MANUAL UNDO
9. Memory Management
9.1 Pre Oracle 9i
9.2 Oracle 9i
9.3 Oracle 10g
9.4 Oracle 11g
10. Logging & Tracing
10.1 Alert File
10.2 Max_dump_file_size
10.3 User and core dump size parameters
10.4 Audit files
11. Advanced Health Checking

How to Perform a Health Check on the Database

1. Parameter file

The parameter file can exists in 2 forms. First of all we have the text-based
version, commonly referred to as init.ora or pfile, and a binary-based file,
commonly referred to as spfile. The pfile can be adjusted using a standard Operating
System editor, while the spfile needs to be managed through the instance itself.
It is important to realize that the spfile takes presedence above the pfile, meaning
whenever there is an spfile available this will be automatically taken unless
specified otherwise.

NOTE: Getting an RDA report after making changes to the database configuration is
also a recommendation. Keeping historical RDA reports will ensure you have
an overview of the database configuration as the database evolves.

Reference:
Note 249664.1 Pfile vs SPfile

2. Controlfiles

It is highly recommended to have at least two copies of the controlfile. This can
be done by mirroring the controlfile, strongly recommended on different physical
disks. If a controlfile is lost, due to a disk crash for example, then you can
use the mirrored file to startup the database. In this way fast and easy recovery
from controlfile loss is obtained.

connect as sysdba
SQL> select status, name from v$controlfile;

STATUS NAME
------- ---------------------------------
/u01/oradata/L102/control01.ctl
/u02/oradata/L102/control02.ctl


The location and the number of controlfiles can be controlled by the 'control_files'
initialization parameter.

3. Redolog files

The Oracle server maintains online redo log files to minimize loss of data in the
database. Redo log files are used in a situation such as instance failure to recover
commited data that has not yet been written to the data files. Mirroring the
redo log files, strongly recommended on different physical disks, makes recovery more
easy in case one of the redo log files is lost due to a disk crash, user delete, etc.

connect as sysdba
SQL> select * from v$logfile;

GROUP# STATUS TYPE MEMBER
--------- ------- ------ -----------------------------------
1 ONLINE /u01/oradata/L102/redo01_A.log
1 ONLINE /u02/oradata/L102/redo01_B.log

2 ONLINE /u01/oradata/L102/redo02_A.log
2 ONLINE /u02/oradata/L102/redo02_B.log

3 ONLINE /u01/oradata/L102/redo03_A.log
3 ONLINE /u02/oradata/L102/redo03_B.log


At least two redo log groups are required, although it is advisable to have at least
three redo log groups when archiving is enabled (see the following chapter). It is
common, in environments where there are intensive log switches, to see the ARCHiver
background process fall behind of the LGWR background process. In this case the LGWR
process needs to wait for the ARCH process to complete archiving the redo log file.

References:
Note 102995.1 Maintenance of Online Redo Log Groups and Members

4. Archiving

Archiving provides the mechanism needed to backup the changes of the database.
The archive files are essential in providing the necessary information to recover the
database. It is advisable to run the database in archive log mode, although you may
have reasons for not doing this, for example in case of a TEST environment where you
accept to loose the changes made between the current time and the last backup.
You may ignore this chapter when the database doesn't run in archive log mode.

There are several ways of checking the archive configuration, below is one of them:

connect as sysdba
SQL> archive log list

Database log mode No Archive Mode --OR-- Archive Mode
Automatic archival Disabled --OR-- Enabled
Archive destination --OR-- USE_DB_RECOVERY_FILE_DEST
Oldest online log sequence seq. no
Current log sequence seq. no


Pre-10g, if the database is running in archive log mode but the automatic archiver
process is disabled, then you were required to manually archive the redolog files.
If this is not done in time then the database is frozen and any activity is prevented.
Therefore you should enable automatic archiving when the database is running in archive
log mode. This can be done by setting the 'log_archive_start' parameter to true in
the parameter file.
Starting from 10g, this parameter became obsolete and is no longer required to be set
explicitly. It is important that there is enough free space on the dedicated disk(s)
for the archive files, otherwise the ARCHiver process can't write and a crash is inevitable.

References:
Note 69739.1 How to Turn Archiving ON and OFF
Note 122555.1 Determine how many disk space is needed for the archive files

5. Datafiles

5.1 Autoextend

The autoextend command option enables or disables the automatic extension of
data files. If the given datafile is unable to allocate the space needed, it
can increase the size of the datafile to make space for objects to grow.

A standard Oracle datafile can have, at most, 4194303 Oracle datablocks.
So this also implies that the maximum size is dependant on the Oracle Block size used.

DB_BLOCK_SIZE Max Mb value to use in any command
~~~~~~~~~~~~~ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
2048 8191 M
4096 16383 M
8192 32767 M
16384 65535 M


Starting from Oracle 10g, we have a new functionality called BIGFILE, which
allows for bigger files to be created. Please also consider that every Operating
System has its limits, therefore you should make sure that the maximum size of
a datafile cannot be extended past the Operating System allowed limit.

To determine if a datafile and thus, a tablespace, has AUTOEXTEND capabilities:

SQL> select file_id, tablespace_name, bytes, maxbytes, maxblocks, increment_by, file_name
from dba_data_files
where autoextensible = 'YES';


References:
Note 112011.1 ALERT: RESIZE or AUTOEXTEND can "Over-size" Datafiles and Corrupt the Dictionary
Note 262472.1 10g BIGFILE Type Tablespaces Versus SMALLFILE Type

5.2 Location

Verify the location of your datafiles. Overtime a database will grow and datafiles
may be added to the database. Avoid placing datafiles on a 'wherever there is space'
basis as this will complicate backup strategies and maintenance.

Below is an example of bad usage:

SQL> select * from v$dbfile;

FILE# NAME
--------- --------------------------------------------------
1 D:\DATABASE\SYS1D806.DBF
2 D:\DATABASE\D806\RBS1D806.DBF
3 D:\DATABASE\D806\TMP1D806.DBF
5 D:\DATABASE\D806\USR1D806.DBF
6 D:\USR2D806.DBF
7 F:\ORACLE\USR3D806.DBF


References:
Note 115424.1 How to Rename or Move Datafiles and Logfiles

6. Tablespaces

6.1 SYSTEM Tablespace

User objects should not be created in the system tablespace. Doing so can lead
to unnecessary fragmentation and preventing system tables of growing. The following query
returns a list of objects that are created in the system tablespace but not owned
by SYS or SYSTEM.

SQL> select owner, segment_name, segment_type
from dba_segments
where tablespace_name = 'SYSTEM'
and owner not in ('SYS','SYSTEM');

6.2 SYSAUX Tablespace (10g Release and above)

The SYSAUX tablespace was automatically installed as an auxiliary tablespace to
the SYSTEM tablespace when you created or upgraded the database. Some database
components that formerly created and used separate tablespaces now occupy the
SYSAUX tablespace.

If the SYSAUX tablespace becomes unavailable, core database functionality will
remain operational. The database features that use the SYSAUX tablespace could
fail, or function with limited capability.

The amount of data stored in this tablespace can be significant and may grow
over time to unmanageble sizes if not configured properly. There are a few
components that need special attention.

To check which components are occupying space:

SQL> select space_usage_kbytes, occupant_name, occupant_desc
from v$sysaux_occupants
order by 1 desc;


Reference:
Note 329984.1 Usage and Storage Management of SYSAUX tablespace occupants SM/AWR, SM/ADVISOR, SM/OPTSTAT and SM/OTHER

6.3 Locally vs Dictionary Managed Tablespaces

Locally Managed Tablespaces are available since Oracle 8i, however they became
the default starting from Oracle 9i. Locally Managed Tablespaces, also referred to
as LMT, have some advantage over Data Dictionary managed tablespaces.

To verify which tablespace is Locally Managed or Dictionary Managed, you can run
the following query:

SQL> select tablespace_name, extent_management
from dba_tablespaces;


References:
Note 93771.1 Introduction to Locally-Managed Tablespaces
Note 105120.1Advantages of Using Locally Managed vs Dictionary Managed Tablespaces

6.4 Temporary Tablespace

* Locally Managed Tablespaces use tempfiles to serve the temporary tablespace,
whereas Dictionary Managed Tablespaces use a tablespace of the type temporary.
When you are running an older version (pre Oracle 9i), then it is important to
check the type of tablespace used to store the temporary segments. By default,
all tablespaces are created as PERMANENT, therefore you should make sure that
the tablespace dedicated for temporary segments is of the type TEMPORARY.

SQL> select tablespace_name, contents
from dba_tablespaces;

TABLESPACE_NAME CONTENTS
------------------------------ ---------
SYSTEM PERMANENT
USER_DATA PERMANENT
ROLLBACK_DATA PERMANENT
TEMPORARY_DATA TEMPORARY



* Make sure that the users on the database are assigned a tablespace of the
type temporary. The following query lists all the users that have a permanent
tablespace specified as their default temporary tablespace.

SQL> select u.username, t.tablespace_name
from dba_users u, dba_tablespaces t
where u.temporary_tablespace = t.tablespace_name
and t.contents <> 'TEMPORARY';


Note: User SYS and SYSTEM will show the SYSTEM tablespace as there default
temporary tablespace. This value can be altered as well to prevent fragmentation
in the SYSTEM tablespace.

SQL> alter user SYSTEM temporary tablespace TEMP



*The space allocated in the temporary tablespace is reused. This is done for
performance reasons to avoid the bottleneck of constant allocating and de-allocating
of extents and segments. Therefore when looking at the free space in the temporary
tablespace, this may appear as full all the time. The following are a few queries
that can be used to list more meaningful information about the temporary segment usage:

This will give the size of the temporary tablespace:

SQL> select tablespace_name, sum(bytes)/1024/1024 mb
from dba_temp_files
group by tablespace_name;


This will give the "high water mark" of that temporary tablespace (= max used at one time):

SQL> select tablespace_name, sum(bytes_cached)/1024/1024 mb
from v$temp_extent_pool
group by tablespace_name;


This will give current usage:

SQL> select ss.tablespace_name,
sum((ss.used_blocks*ts.blocksize))/1024/1024 mb
from gv$sort_segment ss, sys.ts$ ts
where ss.tablespace_name = ts.name
group by ss.tablespace_name;


6.5 Tablespace Fragmentation

Heavly fragmented tablespaces can have an impact on the performance, especially
when a lot of Full Table Scans are occurring on the system. Another disadvantage
of fragmentation is that you can get out-of-space errors while the total sum of
all free space is much more then you had requested.

The only way to resolve fragmentation is recreate the object. As of Oracle8i you
can use the 'alter table .. move' command. Prior to Oracle8i you could use export/import.

If you need to defragment your system tablespace, you must rebuild the whole database since
it is NOT possible to drop the system tablespace.

References:
Note 1020182.6 SCRIPT to detect tablespace fragmentation
Note 1012431.6 Common causes of Fragmentation
Note 158162.1 How To Move All Tables From One User To Another Tablespace

7. Objects

7.1 Number of Extents

While the performance hit on over extended objects is not significant, the
aggregate effect on many over extended objects does impact performance. The
following query will list all the objects that have allocated more extents
than a specified minimum. Change the <--minext--> value by an actual number,
in general objects allocating more then 100 a 200 extents can be recreated
with larger extent sizes:

SQL> select owner, segment_type, segment_name, tablespace_name,
count(blocks), SUM(bytes/1024) "BYTES K", SUM(blocks)
from dba_extents
where owner NOT IN ('SYS','SYSTEM')
group by owner, segment_type, segment_name, tablespace_name
having count(*) > <--minext-->>
order by segment_type, segment_name;


7.2 Next extent

It is important that segments can grow and therefore allocate their next extent
when needed. If there is not enough free space in the tablespace then the next
extent can not be allocated and the object will fail to grow.
The following query returns all the segments that are unable to allocate their
next extent :

select s.owner, s.segment_name, s.segment_type,
s.tablespace_name, s.next_extent
from dba_segments s
where s.next_extent > (select MAX(f.bytes)
from dba_free_space f
where f.tablespace_name = s.tablespace_name);



Note that if there is a lot of fragmentation in the tablespace, then this query
may give you objects that still are able to grow. The above query is based on
the largest free chunk in the tablespace available. If there are a lot of
'small' free chunks after each other, then Oracle will coalesce these to serve
the extent allocation.

Therefore it can be interesting to adapt the script in Note 1020182.6 'SCRIPT
to detect tablespace fragmentation' to compare the next extent for each object
with the 'contiguous' bytes (table space_temp) in the tablespace.

7.3 Indexes

The need to rebuild an index is very rare and often the coalescing the index is
a better option. Please see the following article for a full explanation:

Reference:
Note 122008.1 Script Lists All Indexes that Benefit from a Rebuild

8. AUTO vs MANUAL undo

Starting from Oracle 9i we introduced a new way of managing the before-images.
Previously this was achieved through the RollBack Segments or also referred to as
manual undo. Automatic undo is used when the UNDO_MANAGEMENT parameter is set to
AUTO. When not set or set to MANUAL then we use the 'old' rollback segment mechanism.
Although both versions are still available in current release, automatic undo is preferred.

8.1 AUTO UNDO

There is little to no configuration involved to AUM (Automatic Undo Management).
You basically define the amount of time the before image needs to be kept available.
This is controlled through the parameter UNDO_RETENTION, defined in seconds. So a
value of 900 indicates 15 minutes.

It is important to realize that this value is not honored when we are under space
pressure in the undo tablespace.

Therefore the following formula can be used to calculate the optimal undo tablespace size:

Note 262066.1: How To Size UNDO Tablespace For Automatic Undo Management

Starting from Oracle 10g, you may choose to use the GUARANTEE option, to make sure
the undo information does not get overwritten before the defined undo_retention time.

Note 311615.1: Oracle 10G new feature - Automatic Undo Retention Tuning

8.2 MANUAL UNDO

* Damaged rollback segments will prevent the instance to open the database. Only if names of rollback segments are known, corrective action can be taken. Therefore specify all the rollback segments in the 'rollback_segments' parameter in the init.ora

* Too small or not enough rollback segments can have serious impact on the behavior of your database. Therefore several issues must be taken into account. The following query will show you if there are not enough rollback segments online or if the rollback segments are too small.

SQL> select d.segment_name, d.tablespace_name, s.waits, s.shrinks,
s.wraps, s.status
from v$rollstat s, dba_rollback_segs d
where s.usn = d.segment_id
order by 1;

SEGMENT_NAME TABLESPACE_NAME WAITS SHRINKS WRAPS STATUS
--------------- ------------------ ----- --------- --------- --------
RB1 ROLLBACK_DATA 1 0 160 ONLINE
RB2 ROLLBACK_DATA 31 1 149 ONLINE
SYSTEM SYSTEM 0 0 0 ONLINE


The WAITS indicates which rollback segment headers had waits for them. Typically
you would want to reduce such contention by adding rollback segments.

If SHRINKS is non zero then the OPTIMAL parameter is set for that particular
rollback segment, or a DBA explicitly issued a shrink on the rollback segment.
The number of shrinks indicates the number of times a rollback segment shrinked
because a transaction has extended it beyond the OPTIMAL size. If this value is
too high then the value of the OPTIMAL size should be increased as well as the
overall size of the rollback segment (the value of minextents can be increased
or the extent size itself, this depends mostly on the indications of the WRAPS
column).

The WRAPS column indicate the number of times the rollback segment wrapped to
another extent to serve the transaction. If this number is significant then you
need to increase the extent size of the rollback segment.

Reference:
Note 62005.1 Creating, Optimizing, and Understanding Rollback Segments

9. Memory Management

This chapter is very version driven. Depending on which version you are running the
option available will be different. Overtime Oracle has invested a great deal of
time and effort in managing the memory more efficiently and transparently for the end-user.
Therefore it is advisable to use the automation features as much as possible.

9.1 Pre Oracle 9i

The different memory components (SGA & PGA) needed to be defined at the startup of the
database. These values were static. So if one of the memory components was too low the
database needed to be restarted to make the changes effective.
How to determine the optimal or best value for the different memory components is not
covered in this note, since this would lead us too far. However a parameter that was
often misused in these versions is the sort_area_size.

The 'sort_area_size' parameter in the init.ora defines the amount of memory that can be
used for sorting. This value should be chosen carefully since this is part of the
User Global Area (UGA) and therefore is allocated for each user individually.
If there are a lot of concurrent users performing large sort operation on the database
then the system can run out of memory.

E.g.: You have a sort_area_size of 1Mb, with 200 concurrent users on the database.
Although this memory is allocated dynamically, it can allocate up to 200Mb
and therefore can cause extensive swapping on the system.

9.2 Oracle 9i

Starting from Oracle 9i we introduced the parameters:

workarea_size_policy = [AUTO | MANUAL]
pga_aggregate_target =

This allows you define 1 pool for the PGA memory, which will be shared across sessions.
When you often receive ORA-4030 errors, then this can be an indication that this value is
specified too low.

9.3 Oracle 10g

Automatic Shared Memory Management (ASMM) was introduced in 10g. The automatic shared memory
management feature is enabled by setting the SGA_TARGET parameter to a non-zero value.

This feature has the advantage that you can share memory resources among the different components.
Resources will be allocated and deallocated as needed by Oracle automatically.

Automatic PGA Memory management is still available through the 'workarea_size_policy' and
'pga_aggregate_target' parameters.

9.4 Oracle 11g

Automatic Memory Management (AMM) is being introduced in 11g. This enables automatic tuning
of PGA and SGA with use of two new parameters named MEMORY_MAX_TARGET and MEMORY_TARGET.


Reference:
Note 443746.1 Automatic Memory Management(AMM) on 11g

10. Logging & Tracing

10.1 Alert File

The alert log file of the database is written chronologically. Data is always
appended and therefore this file can grow to an enormous size. It should be
cleared or truncated on a regular basis, as a large alert file occupies
unnecessary disk space and can slow down OS write performance to the file.


Pre-11g:

SQL> show parameter background_dump_dest

NAME TYPE VALUE
------------------------------ ------- ----------------------------------
background_dump_dest string D:\Oradata\Admin\PROD\Trace\BDump


11g and above:

SQL> show parameter diagnostic_dest

NAME TYPE VALUE
------------------------------ ------- ----------------------------------
diagnostic_dest string /oracle/admin/L111

10.2 Max_dump_file_size

Oracle Server processes generate trace files for certain errors or conflicts.
These trace files are of use for further analyzing the problem. The init.ora
parameter 'max_dump_file_size' limits the size of these trace files. The value
of this parameter should be specified in Operating System blocks.
Make sure the disk space can handle the maximum size specified, if not then
this value should be changed.

SQL> show parameter max_dump_file_size

NAME TYPE VALUE
---------------------------------- ------- ---------------------
max_dump_file_size integer 10240

10.3 User and core dump size parameters

The parameters 'user_dump_dest' and 'core_dump_dest' can contain a lot of trace information.
It is important to clear this directory at regular times as this can take up a significant
amount of space.

Note: starting from Oracle 11g, this location is controlled by the 'diagnostic_dest' parameter

Reference:
Note 564989.1 How To Truncate a Background Trace File Without Bouncing the Database

10.4 Audit files

By default, every connection as SYS or SYSDBA is logged in an operating system file.
The location is controlled through the parameter 'audit_file_dest'. If this parameter is
not set then the location defaults to $ORACLE_HOME/rdbms/audit.
Overtime this directory may contain a lot of auditing information and can take up a significant amount of space.

11. Advanced Health Checking

The previous chapters have been outlining the basic items to check to prevent common

database cavehats. In this section you will find references to several articles explaining
how a more in depth analyses and monitoring can be achieved. These article mainly focus
on Data Dictionary Integrity and DataBase structure verification.

Oracle Pre-11g:

Note 456468.1 Identify Data Dictionary Inconsistency
NOTE 136697.1 "hcheck8i.sql" script to check for known problems in Oracle8i,Oracle9i, and Oracle10g


Oracle 11g

Note 466920.1: 11g New Feature Health monitor




Show Related Information Related


Products
  • Oracle Database Products > Oracle Database > Oracle Database > Oracle Server - Enterprise Edition
Keywords
HEALTHCHECK; INTEGRITY
Errors
ORA-4030; 4030 ERROR

Back to topBack to top

Merge Patches using admrgpch

Merge Patches using admrgpch

Admrgph utility is used to merge two or more patches in oracle apps. The advantage of merging patches is that it reduces downtime as the repetitive task of compiling invalid database objects, generating forms and reports,jar files etc.

How to use Admrgpch to merge patches

Download the patches in /patch directory. Now create 2 subdirectory in /patch say mergesource -which contains the unzipped patches to be merged and mergedest -which contains the merged patch. Please note that both mergesource and mergedest should be created as immediate child of same parent directory say /patch. Now you can execute the following command to merge the patches.

admrgpch -s -d -merge_name

For example->

admrgpch -s -d -merge_name

Please make sure the the merge path log file "admrgpch.log" does not contain any error. If the above command to merge patches completes successfully then it displays the following->

Executing the merge of the patch drivers
-- Processing patch: /patch/mergesource/5708576
-- Done processing patch: /patch/mergesource/5708576

-- Processing patch: /patch/mergesource/4428060
-- Done processing patch: /patch/mergesource/4428060

Copying files...

5% complete. Copied 269 files of 5373...
10% complete. Copied 538 files of 5373...
15% complete. Copied 806 files of 5373...
20% complete. Copied 1075 files of 5373...
25% complete. Copied 1344 files of 5373...
30% complete. Copied 1612 files of 5373...
35% complete. Copied 1881 files of 5373...
40% complete. Copied 2150 files of 5373...
45% complete. Copied 2418 files of 5373...
50% complete. Copied 2687 files of 5373...
55% complete. Copied 2956 files of 5373...
60% complete. Copied 3224 files of 5373...
65% complete. Copied 3493 files of 5373...
70% complete. Copied 3762 files of 5373...
75% complete. Copied 4030 files of 5373...
80% complete. Copied 4299 files of 5373...
85% complete. Copied 4568 files of 5373...
90% complete. Copied 4836 files of 5373...
95% complete. Copied 5105 files of 5373...
100% complete. Copied 5373 files of 5373...

Character-set converting files...

2 unified drivers merged.

Patch merge completed successfully

Please check the log file at ./admrgpch.log

Now go to the destination merge patch directory say "mergedest". You can see that the admrgpch already created a driver with name "u_.drv" say u_amebrup2.drv. Now apply the merged patch as a single patch using adpatch. So you have to give this driver name u_.drv" when prompted.

Restrictions of admrgpch ->

It will not merge patches of different releases,platform,different parallel modes. Also do not use admrgpch to merge AD and Non-AD patches ad AD patches will change the patch utility itself.

Tuesday, August 30, 2011

After Clone Concurrent Managers not coming up

After Clone Concurrent Managers not coming up

When cloned my instance last time, I have an issue with concurrent manager not coming up using
Finally decided to stop the concurrent manager using
$COMMON_TOP/admin/scripts/HOST_NAME
. adcmctl.sh stop apps/password

When I have checked the ICM logfile there is an error entry as

Could not initialize the Service Manager FNDSM_HOSTNAME_SID. Verify that HOSTNAME has been registered for concurrent processing.

The cause of this issue was FND_NODES table having incorrect node entries (old node from the Source Instance) and that no Service Manager was found for the existing node.

To solve this problem, need to perform the below steps

1. Stop all middle tier services including the concurrent managers.
2. Stop the database.
3. Start the database.
4. Connect SQLPLUS as APPS user and run the following :

EXEC FND_CONC_CLONE.SETUP_CLEAN;
COMMIT;
EXIT;

5. Run AutoConfig on all tiers, firstly on the DB tier and then the APPS tiers and webtiers to repopulate the required system tables.
6. Connect to SQLPLUS as APPS user and run the following statement :

select CONCURRENT_QUEUE_NAME from FND_CONCURRENT_QUEUES where CONCURRENT_QUEUE_NAME like 'FNDSM%';

If the above SQL does not returning any value then please do the following:

Go to $FND_TOP/patch/115/sql

Connect SQLPLUS as APPS user and run the following script :

SQL> @afdcm037.sql;

This script will create libraries for FNDSM and create managers for preexisting nodes.

Check again that FNDSM entries now exist:

select CONCURRENT_QUEUE_NAME from FND_CONCURRENT_QUEUES where CONCURRENT_QUEUE_NAME like 'FNDSM%';

7. Go to cd $FND_TOP/bin
$ adrelink.sh force=y "fnd FNDLIBR"
$ adrelink.sh force=y "fnd FNDSM"
$ adrelink.sh force=y "fnd FNDFS"
$ adrelink.sh force=y "fnd FNDCRM"

8. Run the CMCLEAN.SQL script from the referenced note below (don't forget to commit).

9. Start the middle tier services including your concurrent manager.

Now cloned instance concurrent managers became active. This will applicable to both 11i and R12 instances.

Monday, August 29, 2011

Concurrent Processing - Product Information Center



Concurrent Processing - Product Information Center (PIC) [ID 1304305.1]

Modified 17-AUG-2011 Type BULLETIN Status PUBLISHED
E-Business Concurrent Processing Information Center


  • Concurent Processing provides a batch processing job engine and service controller to handle program execution, scheduling, and service process management. This document is intended as a customer landing pad for Concurrent Processsing.

  • Please visit the Core Concurrent Processing Community to and post your question or issue for a quick response from Oracle and Industry experts.

  • ATG Advisor Webcasts Visit the ATG Advisor Webcasts page for live webcast listings and a complete listing of previously recorded presentations.

  • ATG ATG Newsletter Archive Visit the ATG Newsletter Archive for the latest announcements and news.

  • E-Business Community Join our growing EBS Community and learn from your peers and Oracle about how to leverage EBS Best Practices.

Notes designated with a video icon Video Icon include embedded video clips as part of the document.

Concurrent Processing Resources

Document Icon

Concurrent Processing - Creating Concurrent Manager Diagnostic and Debug files (troubleshooting)

Document Icon

E-Business Suite Diagnostics Running Shell Tests in Command Mode

Document Icon

Best Practices for Performance for Concurrent Managers in E-Business Suite

Document Icon

REQCHECK.sql - Diagnostic Script for Concurrent Requests

Document Icon

Concurrent Manager Questions and Answers Relating to Generic Platform

Document Icon

Enhancing Oracle Applications Concurrent Processing

Document Icon

Potential Output Post Processor Failures Due to a Multi-Node Setup

Document Icon

Concurrent Processing Tables and Purge Concurrent Request and/or Manager Data Program (FNDCPPUR)

Document Icon

Concurrent Report Access Level

Document Icon

Concurrent Manager Not Running Concurrent Manager Status Column Shows 'System Hold Fix Manager before resetting counters'

Document Icon

R12 RBAC Unable to View Output of Requests Submitted by other Users

Document Icon

How To Run the Purge Concurrent Request FNDCPPUR, Which Tables Are Purged, And Known Issues Like Files Are Not Deleted From File System or Slow Performance

Document Icon

Concurrent Processing (CP) / APPS Reporting Scripts

Document Icon

Virtual Hostnames With Concurrent Managers

Document Icon

R12.1.2 Documentation Library

Document Icon

Complete Library - All Products & Releases

Document Icon

Oracle eBusiness Suite Electronic Technical Reference Manual - eTRM

Document Icon

E-Business Courses From Oracle Education (Search on "Concurrent Processing")


News & Hot Topics

ICM log file shows 'CONC-SM TNS FAIL', 'Call to PingProcess failed', and/or 'Call to StopProcess failed' for FNDCPGSC/FNDOPP

New document detailing the steps to handle ICM errors related to FNDCPGSC and FNDOPP.

Concurrent Processing - One Concurrent Manager SQL statement shows excessive executions

Latest details on a common CP performance issue

After Doing a Clone All the Concurrent Managers Do Not Start

This document discusses concurrent manager related cloning issues.

All Concurrent Reports Are Failing With Error REP-0004,REP-0082 and REP-0104

Latest details on CP requests and Report (REP) related errors

Starting Workflow Service Gives ORA-24067: Exceeded Maximum Number of Subscribers for Queue APPLSYS.FND_CP_GSM_IPC_AQ

New document detailing the steps to handle subscriber errors for Queue APPLSYS.FND_CP_GSM_IPC_AQ.

Concurrent Request Fails Due to Timeout of the Post Processing Action

Latest Output Post Processor issues related to timeouts and request failures

FDPSTP Failed Due to ORA-20100: File o0017164.tmp Creation for FND_FILE Failed

Concurrent Request Completes with Errors ORA-00164 and APP-FND-01564

Apache and Mod PL/SQL Listener Cannot be Managed by GSM at the Same Time

An error Occurred While Attempting to Establish an Applications File Server Connection with Node FNDFS Error

Post ATG RUP7 XML Publisher German Reports Give FRM-41072: Cannot create Group req_layout Error



Diagnostics and Debugging

How to Debug Concurrent Request Transaction Managers from Oracle Applications Manager (OAM)(Note 1308185.1)

ANALYZEREQ.SQL - Detailed Analysis of One Concurrent Request (Release 11 and up)(Note 134035.1)

CCM.sql Diagnostic Script for Concurrent Manager(Note 171855.1)This script will collect relevant Concurrent Manager Diagnostic information

How To Trace a Concurrent Request And Generate TKPROF File (Note453527.1) Steps for generating Concurrent Request debug logs and traces

11i : Oracle Application Object Library Concurrent Processing Data Collection Test 11i Concurrent Processing Data Collection Diagnostic Test

11i : Oracle Application Object Library Concurrent Manager Setup Test11i Concurrent Processing Setup Diagnostic Test

Complete Diagnostic Catalogs for Each Release: Release 11i (note 179661.1, Release 12.0.4 (note 469721.1), Release 12.0.6+ (note 741601.1), Release 12.1 (note 783319.1), Release 12.1 (note 942527.1), Find transaction based diagnostics in these catalogs for the product/feature where the tax problem is occurring.

FAQ's & Business Requirements

CMCLEAN.SQL - Non Destructive Script to Clean Concurrent Manager Tables (Note 134007.1)

Steps To Clean Nonexistent Nodes or IP Addresses From FND_NODES (Note 260887.1)

APP-FND-00222 Error When Submitting a Custom Concurrent Request Registered With a Custom Application (Note 150993.1)

Default Layout Format for XML Publisher Concurrent Request Is Always PDF (Note 401328.1)

ORA-00060: Deadlock Detected - UPDATE FND_CONCURRENT_QUEUES(Note 866298.1)


Service Manager Processes Are Not Starting After A Restart - usdsop cannot redirect standard output (Note 733901.1)

Concurrent Report Access Level (Note 736547.1)

Java Concurrent Program FAQ (Note 827575.1)

Concurrent Manager Not Running Concurrent Manager Status Column Shows 'System Hold Fix Manager before resetting counters'(Note 1209773.1)

CONC-SM TNS FAIL In Internal Concurrent Manager (ICM) Log File With Virtual Hosts Configuration(Note 961216.1)

How to Register a Host Concurrent Program in Applications (Note 156636.1)

How to Cancel a Concurrent Request Stuck in the Queue? (Note 749748.1)

Creating a Conflict Domain for a Concurrent Program (Note 267167.1)

How to Setup Resource Profiles for Concurrent Managers to Limit CPU Usage?(Note 394562.1)

Concurrent Manager Fails with ORA-00604 & ORA-00018 (Note 110202.1)

APP-FND-01270 ERROR GENERATING JULIAN SYSDATE in Forms and REP-1416:'C_bal_segment_promptformula' in Reports (Note 565307.1)


(More)

Concurrent Processing on RAC Information Center

How to run a concurrent program against a specific RAC instance with PCP/RAC setup? (Note 1129203.1)

How to Setup Parallel Concurrent Processing using Shared APPL_TOP for RAC Environment (Note 278816.1)

How To Ensure Load Balancing Of Concurrent Manager Processes In PCP-RAC Configuration(Note 762024.1)

How to Setup and Test Failover of PCP on Non-RAC Environments. (Note 743716.1)

Parallel Concurrent Processing Failover/Failback Expectations (Note 271090.1)

RAC Configuration Setup For Running MRP Planning, APS Planning, and Data Collection Processes (Note 279156.1)

Concurrent Managers Will Only Start on One Node in PCP/RAC Environment (Note 264296.1)

Concurrent managers not running/failing over to correct node in PCP/RAC environment (Note 790624.1)

Concurrent Manager Setup and Configuration Requirements in an 11i RAC Environment (Note 241370.1)



(More)

Concurrent Processing Features (Setup and Troubleshooting)

How to Activate Parallel Concurrent Processing - Background Facts and Setup Steps (Note 602899.1)

Generic Service Management (GSM) in Oracle Applications 11i (Note 210062.1)

Virtual Hostnames With Concurrent Managers (Note 734506.1)

How To Create a Java Concurrent Program? (Note 827563.1)

How to Register a Host Concurrent Program in Applications (Note 156636.1)

Using the Output Post Processor (OPP) in Oracle Applications Concurrent Processing (Note 291792.1)

How to start the Output Post Processor (OPP)?(Note 762454.1)

Troubleshooting ORA-20100 on Concurrent Processing (Note 261693.1)

How to Troubleshoot Concurrent Manager Start Up Issue Related to Service Manager (Note 735148.1)

Troubleshooting Concurrent Manager (Unix specific) (Note 104452.1)

Troubleshooting FNDFS and FNDWRR Issues on the Oracle E-Business Suite (Note 847844.1)

Internal Manager Log File is having Messages like "Could Not Initialize the Service Manager FNDSM__" for Non Concurrent Processing Nodes (Note 754698.1)

Concurrent Managers Fail to Start, No Service Manager Process Spawned (Note 470100.1)


Concurrent Processing Integration with EBS Modules

Sample Code to Call Customer Interface RACUST Program using FND_SUBMIT Api in 11i and R12 (Note 429278.1 )

How to Add Concurrent Programs Upload, Import and Validate Accounting Definition to Receivables? (Note 808675.1)

How to add 'Transfer Journal Entries to General Ledger' Concurrent Program to Receivables Reports Group(Note 734495.1)

How do you find/check storage of report parameters? (Note 74962.1)

Subledger Accounting Balances update concurrent program wrong balances (Note 876017.1 )

Order Management Diagnostic Concurrent Programs(Note 201231.1)

Order Management Exception Management: Retry Activities in Error Concurrent Program(Note 376040.1)

How to run a Payables Concurrent Program in Debug Mode (Note 198799.1)

the Concurrent Programs "Payables Transfer To General Ledger" And "Payables Transfer to GL" Fail In R12 (Note 1179363.1)

APXIIMPT: Payables Open Interface Import Concurrent Program Shows No List of Values for Parameter Operating Unit (Note 947303.1)

R12 Payables Suppliers: Supplier Merge silently fails to submit when HR is not installed. Suppliers Not Merged (Note 1304936.1)

R12: 'Online Accounting Could Not Be Created For This Transaction' On Create Accouting (Note 745232.1)

R12: Problems with the creation process and accounting transfer (AP X GL) - JLARAPAC (Note 1054066.1)

E-Business Suite: Oracle Payables Information Center (Note 1196096.1)

Using the Concurrent Manager for Troubleshooting Purchasing Documents Stuck at Deferred Activities (Note 1249175.1)

Possible Causes Resulting In The PO Approval Failing At "Wait for Concurrent Program" (Note 1288874.1)



Show Related Information Related


Products
  • Oracle E-Business Suite > Applications Technology > Application Object Library > Oracle Concurrent Processing
  • Oracle E-Business Suite > Applications Technology > Application Object Library > Oracle Application Object Library
  • Oracle E-Business Suite > Applications Technology > Application Object Library > Oracle Application Object Library
Keywords
CONC-SM TNS FAIL; CONCURRENT MANAGER; CONCURRENT PROCESSING; FNDCPPUR; FNDFS; FNDSM; FND_CONCURRENT_QUEUES; INFORMATION CENTER
Errors
REP-1416; REP-3000; FND-1564; FND-1270; ORA-60; ORA-24067; ORA-20100; ORA-604; ORA-18; ORA-164

Back to topBack to top