ORA-00450, ORA-00443, background process "CJQ0" did not start On : 11.g

ORA-00450, ORA-00443, background process "CJQ0" did not start On : 11.2.0.3 version, RDBMS

When attempting to shutdown the database the following error occurs.

ORA-450: background process 'CJQ0' did not start
ORA-443: background process "CJQ0" did not start


The issue can be reproduced at will by shutdown the instance.

Cause


This problem is as described in
unpublished Bug 6865966 PMON TRYING TO START CJQ0 DURING DATABASE SHUTDOWN-ORA-443 & ORA-450
where development explained that the errors are misleading but are unharmful.


Solution

This is caused by scheduler autostart feature trying to start up the  coordinator at the same time as database is shutting down. So you can safely ignore the errors.

The unpublished Bug 6865966 is still under investigation and you can apply the patch, once the fix is made available.

The Autostart feature will not be available in 12c and hence this issue wont be present in 12c version.


References

@ BUG:6865966 - PMON TRYING TO START CJQ0 DURING DATABASE SHUTDOWN-ORA-443 & ORA-450

ORA-00450, ORA-00443, background process "CJQ0" did not start On : 11.g

ORA-00450, ORA-00443, background process "CJQ0" did not start On : 11.2.0.3 version, RDBMS

When attempting to shutdown the database the following error occurs.

ORA-450: background process 'CJQ0' did not start
ORA-443: background process "CJQ0" did not start


The issue can be reproduced at will by shutdown the instance.

Cause


This problem is as described in
unpublished Bug 6865966 PMON TRYING TO START CJQ0 DURING DATABASE SHUTDOWN-ORA-443 & ORA-450
where development explained that the errors are misleading but are unharmful.


Solution

This is caused by scheduler autostart feature trying to start up the  coordinator at the same time as database is shutting down. So you can safely ignore the errors.

The unpublished Bug 6865966 is still under investigation and you can apply the patch, once the fix is made available.

The Autostart feature will not be available in 12c and hence this issue wont be present in 12c version.


References

@ BUG:6865966 - PMON TRYING TO START CJQ0 DURING DATABASE SHUTDOWN-ORA-443 & ORA-450

What is the database initialization parameter that is associated to an ORA-32004 error ?

What is the database initialization parameter that is associated to an ORA-32004 error ?


Oracle Server - Enterprise Edition - Version: 9.2.0.1 to 11.2.0.3 - Release: 9.2 to 11.2
Information in this document applies to any platform.
***Checked for relevance on 24-Jan-2012***
Goal

How to know which parameter is obsolete and/or deprecated when the only message on screen when
attempting to start a database is "ORA-32004: obsolete and/or deprecated parameter(s) specified" ?

Solution

1.

Look at the DBA's alert log of the associated instance as newer versions of the RDBMS software will identify the parameters that are obsolete and/or deprecated within the alert log at the time that an instance startup is initiated.

2.

If the alert log is not available or does not identify the obsolete or deprecated parameter and the database is accessible then check to see that none of the parameters in the parameter initialization file match a parameter name stored in the V$OBSOLETE_PARAMETER view of that same database.
Especially review any parameter where V$OBSOLETE_PARAMETER.ISSPECIFIED='TRUE'.

If there is a match, then it is an obsolete parameter.

3.

If the alert log is not available or does not identify the obsolete or deprecated parameter then check to see that none of the parameters in the parameter initialization file match a parameter name listed within the migration guide's list of deprecated parameters or it's list of obsolete parameters.

Vi: Search and Replace

Vi: Search and Replace

Change to normal mode with .

Search (Wraped around at end of file):

  Search STRING forward :   / STRING.
  Search STRING backward:   ? STRING.

  Repeat search:   n
  Repeat search in opposite direction:  N  (SHIFT-n)

Replace: Same as with sed, Replace OLD with NEW:

 
 First occurrence on current line:      :s/OLD/NEW
  
 Globally (all) on current line:        :s/OLD/NEW/g 

 Between two lines #,#:                 :#,#s/OLD/NEW/g
  
 Every occurrence in file:              :%s/OLD/NEW/g 

R12: Contact Center: Package CSC_ACTION_ASSEMBLER_PVT_W Invalid

Solution

To compile the CSC_ACTION_ASSEMBLER_PVT_W package and body manually,
run these statements in SQLplus:

SQL> alter session set "_disable_fast_validate"=TRUE;
SQL> alter package CSC_ACTION_ASSEMBLER_PVT compile;
SQL> alter package CSC_ACTION_ASSEMBLER_PVT compile body;

SQL> alter package CSC_ACTION_ASSEMBLER_PVT_W compile;
SQL> alter package CSC_ACTION_ASSEMBLER_PVT_W compile body;



The "_disable_fast_validate" parameter should be set to true during maintenance operations.
See this Note for further explanation:

Note.1058763.1 Ext/Pub Interoperability Notes EBS R12 with Database 11gR2

How To Resolve RLM Invalids R12.1.3 upgrade

          How To Resolve RLM Invalids

On 12.1.3 need to resolve the following invalid objects:
RLM_EXTINTERFACE_SV
RLM_RD_SV
RLM_MANAGE_DEMAND_SV

Fix

Please run the following for the 3 RLM invalid objects:
SQL>
RLM_MANAGE_DEMAND_SV PACKAGE
RLM_RD_SV PACKAG BODY
RLM_EXTINTERFACE_SV PACKAGE BODY

select object_name,object_type from from dba_objects
where object_name ='RLM_MANAGE_DEMAND_SV';

alter package RLM_MANAGE_DEMAND_SV compile;

select object_name,object_type from from dba_objects
where object_name ='RLM_RD_SV PACKAG';

alter package RLM_RD_SV;

select object_name,object_type from from dba_objects
where object_name ='RLM_EXTINTERFACE_SV PACKAGE';

alter package RLM_EXTINTERFACE_SV PACKAGE compile;

ASO object in my 12.1.3 environment

How can I successfully compile the invalid ASO object in my 12.1.3 environment? ASO_OPP_QTE_PUB ASO_QUOTE_HEADERS_PVT  ASO_QUOTE_PUB_W  ASO_SECURITY_INT  Please do the following: 1. Compile each of the objects using the package source files. First compile the package SPEC asoisecs.pls, then the package BODY: For example: @asoisecs.pls @asoisecb.pls @asovqw1s.pls @asovqw1b.pls @asopopqs.pls @asopopqb.pls @asovqhds.pls @asovqhdb.pls 2. Confirm these ASO packages have compiled successfully. select object_name, object_type, status from dba_objects where object_type like '%PACKAGE%' and object_name like '%ASO%'; 3. If any ASO invalids remain, please upload the appscheck as follows from Order Management Log into your Order Management responsibility View->Requests->Submit a request->Single Request->Diagnostics Appscheck ----> Application Parameters Quoting (QOT) Order Capture (ASO)

R12.1.1 Invalid Objects After Patching - FRM-18108: Failed to load the following objects

R12.1.1 Invalid Objects After Patching - FRM-18108: Failed to load the following objects

When attempting to prepare a new instance on R12.1.1  for Order Management, receive invalid objects that will not regenerate.


ERROR
-----------------------
FRM-18108: Failed to load the following objects.

Source Module:OEXOEREL.fmb
Source Object: RELATED_ITEMS_PROMPT
Source Module:OEXOEREL.fmb
Source Object: RELATED_ITEMS
Source Module:OEXOEREL.fmb
Source Object: RELATED_ITEMS_CONTROL
Source Module:OEXOEREL.fmb
Source Object: RELATED_ITEMS_FOLDER
Source Module:OEXOEREL.fmb
Source Object: RELATED_ITEMS_FIXED
Source Module:OEXOEREL.fmb

In file aderrchk.lst
See the following error:
JAVA CLASS 0
oracle/apps/fnd/common/ias/JservA
dminHelper
ORA-29552: verification warning:
java.lang.UnsupportedClassVersionError:
oracle/apps/fnd/common/ias/JservAdminHelper (Unsupported
major.minor version 49.0)
(repeated for more JAVA messages)
...
PACKAGE BODY 0
BPEL_UPDATEORDEREBS11I10TOCOMS
PLS-00907: cannot load library unit APPS.OE_ACKNOWLEDGMENT_PUB
(referenced by APPS.OE_OUTBOUND_INT)

ran adadmin for ONT applications; below objects where not  compiled:
 au resource OESCHORG.pll
 au resource OEXOEFRM.pll
 au resource OEXOELIB.pll
 au resource OEXOELIN.pll
 ont forms/US OESCHORG.fmx
 ont forms/US OEXOEORD.fmx
 ont forms/US OEXOETEL.fmx
 ont forms/US OEXOERLI.fmx
 ont forms/US OEXOEVER.fmx

STEPS
-----------------------
The issue can be reproduced at will with the following steps:
1.Refresh from production ( 0 invalid objects)
2. Apply about 50 patches
3. Error in OM files

BUSINESS IMPACT
-----------------------
The issue has the following business impact:
Due to this issue, users cannot access forms to complete setup and create sales orders.

Changes

Create a new R12.1.1 instance.
Cause

Many invalid OM objects after creating and patching a new instance.

The following bug was created for this issue:
OM Bug  9655621 - R12.1.1 - OE INVALID OBJECTS AFTER APPLING 51 PATCHES (NON OM)


Solution


To implement the solution, please execute the following steps:
1.
a) restart the database
b) run utlirp/utlrp as sys as instructed in the base bug.
    $ORACLE_HOME/rdbms/admin/utlirp.sql
    $ORACLE_HOME/rdbms/admin/utlrp.sql
c) if apps objects are still invalid after this then run the relevant
Apps script(s) to recompile.
d) once all objects in the database are valid run the following select
to ensure there are no timestamp mismatches remaining:

alter session set nls_date_format='dd-mon-yy hh24:mi:ss';

select do.name dname, po.name pname, d.p_timestamp, po.stime p_stime
from sys.obj$ do, sys.dependency$ d, sys.obj$ po
where d.P_OBJ#=po.obj#(+)
and d.D_OBJ#=do.obj#
and do.status=1 /*dependent is valid*/
and po.status=1 /*parent is valid*/
and po.stime!=d.p_timestamp /*parent timestamp not match*/
and do.type# not in (28,29,30) /*dependent type is not java*/
and po.type# not in (28,29,30) /*parent type is not java*/
order by 2,1;

If issues performing step 1, create a new service request with DataBase product,
component PL/SQL.

2. After following the above instruction, if all the Database packages are valid,
then try recompiling the pll and forms

How to Manually Kill the Correct Database Session For a Hanging Forms Runtime Process

How to Manually Kill the Correct Database Session For a Hanging Forms Runtime Process


Problem Description
-------------------

You are trying to kill an operating system process ID (PID) for a specific
Oracle job, but you are unsure of the exact process number to kill.  You asked
how to find the desired PID number.

Solution Description
--------------------

Run the following select statement to find the desired process ID (PID) and
username:

   select spid, sid, a.serial#, b.username
   from v$session a, v$process b
   where a.paddr = b.addr;

Following is an example of what the query returns:

  SPID    SID     SERIAL#     USERNAME
  ----    ---     -------     --------
  11526     1        1        absmith
  11547     3        1        absmith
  11551     6        1        absmith
 

Explanation
-----------

The SPID is the actual operating system process identifier.

The SID is Oracle's session identifier.

The SERIAL# is Oracle's session serial number used to identify a session's
objects.

After the operating system PID has been killed, type the following (it is
faster than waiting for PMON to clean up the terminated process):

     alter system kill session 'sid, serial#';

orms Process (FRMWEB) Consumes 100% of CPU in Oracle Applications R12

Forms Process (FRMWEB) Consumes 100% of CPU in Oracle Applications R12

On Oracle Applications R12 when checking the top processes on the OS level for the middle tier, you find that forms process (frmweb) almost consumes 100% of the CPU.
Cause

The root cause of the issue is that returning rows from LOVs in core forms causes the forms process to grow up into memory depending on the number of rows returned.

When an end user login to forms and start working with LOV within core forms sometimes and according to the search criteria that the user will provide to filter the results in LOV, it may fetch huge numbers of records in which causes the frmweb process to grow very large, and in extreme cases this can even lock up the current process or even the whole machine.

So when executing a LOV query, every row is fetched into memory on the middle tier, the frmweb process can get extremely large, and the larger it gets the more likely it is to start paging.
Eventually it starts consuming excessive CPU just paging the process in and out of memory, which is probably what you can see here in this case as the amount of memory consumed when the LOV records are fetched into memory obviously depends on the amount of data in each record.

This has been mentioned in the following bug:
Bug 6519700 - ESC: CSE: R12SIP: 6513826 FRMWEB RUNAWAY PROCESS CONSUMING 100% CPU-MIDDLE TIER

Solution

To implement the solution, please execute the following steps:

1. Stop all services on the middle tier.

2. Set following forms environment variables:

FORMS_RECORD_GROUP_MAX to 10000 or if that proves too restrictive, increase it to 20000 or 30000.
FORMS_CATCHTERM=0

In order to set the above forms variables so next time autoconfig run does not override those values, do the following steps :

1- For Forms Variable "FORMS_CATCHTERM" the context vairable name is: "s_forms_catchterm" and you can update the context file located in ($INST_TOP/appl/admin/) 

2- For other forms variable "FORMS_RECORD_GROUP_MAX" there is no variable defined in Autoconfig for that one and have to customize the autoconfig for the forms variables to set that environment as following:
a- Go to the autoconfig Template folder:
$cd $AD_TOP/admin/template
b- Create new directory named (custom)
$ mkdir custom
c- Make sure that new directory has same file permissions as ($AD_TOP/admin/template)
d- Copy the following autoconfig template to the new custom directory:
$cp $AD_TOP/admin/template/APPLSYS_ux.env $AD_TOP/admin/template/custom/APPLSYS_ux.env
e- Edit the file copied file under custom directory and add the following 2 lines at the end of section:

####################################
# Oracle Forms environment variables
####################################

FORMS_RECORD_GROUP_MAX=10000
export FORMS_RECORD_GROUP_MAX

f- Save and exit from the file.
g- Next time autoconfig run, it will read the custom directory and check for any customizations there.
3. Run Autoconfig on the middle tier and make sure it is completed successfully.

4. Startup all services.

5. Monitor the forms process to see its CPU usage, and you will see that form process usage is reduced and not causing any more CPU consumption up to 100% as before.

6. Migrate the solution as appropriate to other environments.

IMPORTANT NOTE: If you find that setting (FORMS_RECORD_GROUP_MAX=30000) still too restrictive for certain LOV's then Oracle product team needs to consider redesigning the LOV's in their form and at that time raise a new SR with the Oracle Support team so they can raise a new bug with development for that specific LOV to get it redesigned but this may happen ONLY in RARE cases.

How To Select Number of Workers Based on Number of CPUs When Running ADPATCH

How To Select Number of Workers Based on Number of CPUs When Running ADPATCH


Application Install - Version: 11.5.10.2 to 11.5.10.2 - Release: 11.5 to 11.5
Generic UNIX
Checked for relevance on 11-MAY-2010
Goal

How to select number of workers based on number of processors when running adpatch

Solution

Select 2x the number of processors (CPUs) on the instance to be patched.

Ex. Machine has 4 processors, select up to 8 workers when running adpatch.

Finding alert.log file in 11g

  Finding alert.log file in 11g [ID 438148.1]


Beginning with Release 11g of Oracle Database, the alert log is written as both an XML-formatted file and as a text file, as in earlier releases.Both these log files are stored inside the ADR home.The ADR root directory is known as ADR BASE.The Automatic Diagnostic Repository (ADR) is a directory structure that is stored outside of the database.This parameter is set by DIAGNOSTIC_DEST initialization parameter.

If this parameter is omitted or left null, the database sets DIAGNOSTIC_DEST upon startup as follows:

If environment variable ORACLE_BASE is set, DIAGNOSTIC_DEST is set to the directory designated by ORACLE_BASE.
If environment variable ORACLE_BASE is not set, DIAGNOSTIC_DEST is set to ORACLE_HOME/log.
for e.g

SQL> show parameter diagno

NAME                          TYPE          VALUE
--------------------------- ----------- ------------------------------
diagnostic_dest             string      /u01/oracle/product/ora11g/log
The location of an ADR home is given by the following path, which starts at the ADR base directory:

/diag/// 

For example,

 for a database with a SID and database name both equal to ora11g, the ADR home would be in the following location:

/diag/rdbms/ora11g/ora11g/

Within the ADR home directory are subdirectories where the database instance stores diagnostic data.

Subdirectory Name Contents
alert The XML-formatted alert log
trace Background and server process trace files and SQL trace files and text alert.log file
cdump Core files


XML formatted alert.log
-------------------------
The alert log is named log.xml and is stored in the alert subdirectory of ADR home.

To get the log.xml path

ADR_BASE/diag/product_type/product_id/instance_id/alert

from sqlplus

SQL> select value from v$diag_info where name ='Diag Alert';

ADRCI utility to view a text version of the alert log (with XML tags stripped)

Text formatted alert.log
-----------------------

The alert.log is named alertSID.log and is stored in the trace subdirectory of ADR home.


To view the text only alert.log file

/diag////trace 

from sqlplus

SQL> select value from v$diag_info where name ='Diag Trace';
or
SQL>show parameter background_dump_dest

Open file alert_SID.log with a text editor

You can also use the ADR Client Interface called 'adrci' to view the alert.log

% adrci
ADRCI> show alert
       show alert -tail 50

The alert log of a database is a chronological log of messages and errors, including the following:

Error while running adbldxml.pl

Error while running adbldxml.pl after upgrading database to 11.1.0.7 from 10.2.0.2

The following error was encountered while running adbldxml.pl on the database tier to create the context file. 


====================================================================
Could not Connect to the Database : ORA-01034: ORACLE not available
ORA-27101: shared memory realm does not exist
Linux Error: 2: No such file or directory

Connecting to the VISN11 database instance...

Connection paramaters values: 
Database server hostname ==> NGLINUX05.NEWGEN.COM
Database listener port ==> 1542
Database SID ==> VISN11
Database schema name ==> apps

Could not Connect to the Database : ORA-01034: ORACLE not available
ORA-27101: shared memory realm does not exist
Linux Error: 2: No such file or directory


AC-40000: Error: Exception - java.sql.SQLException: ORA-01034: ORACLE not available
ORA-27101: shared memory realm does not exist
====================================================================


The problem was that the database was not registered with the listener.




SQL> show parameter local_listener;

NAME TYPE VALUE
------------------------------------ ----------- ------------------------------
local_listener string




The value for LOCAL_LISTENER is missing, so automatic instance registration is not taking place
We can correct this by setting the LOCAL_LISTENER parameter:


SQL> alter system set local_listener='(ADDRESS =(PROTOCOL=TCP)(HOST=host.domain.com)(PORT=port_number)(SID=sid_name)';

System altered.

SQL> alter system register;

System altered.


SQL> exit

Now adbldxml runs successfully.

FNDLOAD Download / Upload Concurrent Programs Fails To Load Parameters

How to load parameter values when using FNDLOAD to move a concurrent program from one instance to another? There have been changes since Enhancement Request << BUG 6934421>>.


Solution

At the source instance run FNDLOAD DOWNLOAD using the control file afcpprog.lct and the parameter P_VSET_DOWNLOAD_CHILDREN="Y".

For example:
FNDLOAD apps/{pwd} 0 Y DOWNLOAD $FND_TOP/patch/115/import/afcpprog.lct /export/home/applmgr/XXINTF_INV_ITEM_OUTBOUND_PKG.ldt PROGRAM APPLICATION_SHORT_NAME="XXINTF" CONCURRENT_PROGRAM_NAME="XXINTF_INV_ITEM_OUTBOUND_PKG" P_VSET_DOWNLOAD_CHILDREN="Y"

Then at the target instance run FNDLOAD UPLOAD using the data file you just generated.

For example:
FNDLOAD apps/{pwd} 0 Y UPLOAD $FND_TOP/patch/115/import/afcpprog.lct /export/home/applmgr/XXINTF_INV_ITEM_OUTBOUND_PKG.ldt

If you are replacing an existing custom program, add the parameter - CUSTOM_MODE=FORCE.