How To Resolve TNS Operation Timed Out Errors

How To Resolve TNS Operation Timed Out Errors

While examining logs I have found lots of “TNS Operation Timeout Errors” in sqlnet.log file

Fatal NI connect error 12170.

  VERSION INFORMATION:
        TNS for Linux: Version 10.2.0.4.0 - Production
        Oracle Bequeath NT Protocol Adapter for Linux: Version 10.2.0.4.0 - Production
        TCP/IP NT Protocol Adapter for Linux: Version 10.2.0.4.0 - Production
  Time: 06-MAY-2009 18:54:41
  Tracing not turned on.
  Tns error struct:
    ns main err code: 12535
    TNS-12535: TNS:operation timed out
    ns secondary err code: 12560
    nt main err code: 505
    TNS-00505: Operation timed out
    nt secondary err code: 110
    nt OS err code: 0
  Client address: (ADDRESS=(PROTOCOL=tcp)(HOST=*.*.*.*)(PORT=3448))


The same error message was repeating during whole day almost for every application server.

I’ve finally found what was causing the problem. Our database is behind a firewall. Firewall has an “idle session timeout” value. If a connection remains idle for more than the “idle session timeout” value it drops the connections.

Application developers usually configure their connection pools to remain alive for a long time, which is more than the “idle session timeout” value of the firewall. Not all the connections in the pool are used. Some stays idle. After sometime the firewall drops them and I get those operation timed out problems.

Solution:

Add the following line to the sqlnet.ora file on the server.
SQLNET.EXPIRE_TIME=10


In this configuration database will probe the application servers every 10 minutes to see if they are up. In fact this is a mechanism to determine stale connections by database. Because it sends packets to application servers every ten minutes, the connections are detected as active by firewalls and they are not broken.

Reset the WebLogic EPMSystem Domain admin epm_admin Password Without Knowing the Password

Reset the WebLogic EPMSystem Domain admin epm_admin Password Without Knowing the Password [ID 1096427.1]
		Modified 08-JUL-2011     Type REFERENCE     Status PUBLISHED

In this Document
  Purpose
  Scope
  Reset the WebLogic EPMSystem Domain admin epm_admin Password Without Knowing the Password

---

Applies to:

Hyperion Planning - Version: 11.1.2.0.00 to 11.1.2.0.00 - Release: 11.1 to 11.1
Generic UNIX

Purpose

The purpose of this article is to allow you "the Hyperion EPM 11.1.2 admin" a way to reset the WebLogic epm_admin user's password without having a way to login to the WebLogic EPMSystem domain as an administrator. 

Scope

This article is intended for Hyperion EPM 11.1.2 system administrators with:

• A strong understanding of the Hyperion EPM 11.1.2 system architecture
• A basic understanding of WebLogic
• A basic administrators understanding of Unix operating systems

Reset the WebLogic EPMSystem Domain admin epm_admin Password Without Knowing the Password

You just installed EPM 11.1.2 and you did not write down the Weblogic Domain admin password, or you've had it installed for awhile and can't remember what the password is to get into the WebLogic Admin console to make a change. This article is going to help you do that so you can get on with your business.

In order to perform this task you'll need to know what the DOMAIN_HOME location is. Within these steps we'll refer to the DOMAIN_HOME as a variable. Unix Variable references begin with a '$' dollar sign.

Unix Systems
The DOMAIN_HOME on unix servers by default will exist under the installing users $HOME directory. In this example the user hypadmin has it's $HOME directory here /export/home/hypadmin

Example
DOMAIN_HOME=/export/home/hypadmin/Oracle/Middleware/user_projects/domains/EPMSystem

1. Open a shell to the Unix server where EPM 11.1.2 is installed and set the DOMAIN_HOME variable

DOMAIN_HOME=$HOME/Oracle/Middleware/user_projects/domains/EPMSystem

2. Change directory to $DOMAIN_HOME/bin

cd $DOMAIN_HOME/bin

3. Source the setDomainEnv.sh script so we can get all the WebLogic variables set correctly

. setDomainEnv.sh

4. Now change directory to $DOMAIN_HOME/security

cd $DOMAIN_HOME/security

5. Move the DefaultAuthenticatorInit.ldift to a backup file

mv DefaultAuthenticatorInit.ldift DefaultAuthenticatorInit_old.ldift

6. Run the following java command to generate a new DefaultAuthenticatorInit.ldift within the security directory

java weblogic.security.utils.AdminAccount newuser newpassword .

This will create a new user with the assigned password. 

Note: replace newuser and newpassword with a user and password of your choosing. Ensure the password is a minimum of 8 characters and contains at least one numeric character.

7. Change directory to $DOMAIN_HOME/servers/AdminServer/security

cd $DOMAIN_HOME/servers/AdminServer/security/

8. Edit the boot.properties file using a file editor of your choosing. Replace newpassword and newusername with the user and password you defined in step 6.

password=newpassword
username=newusername

The boot.properties will get encrypted once the admin server starts up

9. Rename or remove directory $DOMAIN_HOME/servers/AdminServer/data/ldap as it will get recreated once the WebLogic Admin server starts.

rename $DOMAIN_HOME/servers/AdminServer/data/ldap $DOMAIN_HOME/servers/AdminServer/data/ldap_old

10. Change directory to $DOMAIN_HOME/bin and start the Weblogic Admin Server

startWebLogic.sh

11. Open a browser and navigate to the WebLogic Domain console for the EPMSystem domain

http://wl_admin_server_hostname:7001/console/login/LoginForm.jsp

Login with the newuser and newpassword you defined in step 6

12. In the left hand window under Domain Structure click the Security Realms link


13. Select the myrealm link



14. Select the Users and Groups tab 


15. Click the epm_admin link to edit the epm_admin users password


16. Click the password tab


17. Enter the new password for the epm_admin user. Ensure that the password is a minimum of 8 characters and has at least one numeric character. Once finished click the save button to make the changes permanent. Logout of the WebLogic console and login as the epm_admin user with the new password to ensure the change has been made.


18. At this point you've modified the user to start the WebLogic console and have reset the epm_admin password. If you decide to remove the user you used to get into WebLogic be warned that you won't be able to start the Weblogic console until you perform the steps back in step 8. Replace the user with 'epm_admin' and the password with the epm_admin password.

APP-FND-00362: Routine afpbep cannot execute request &REQUEST for program &PROGRAM

APP-FND-00362: Routine afpbep cannot execute request &REQUEST for program &PROGRAM
Error:
=====

APP-FND-00362: Routine afpbep cannot execute request &REQUEST for program &PROGRAM, because the environment variable &BASEPATH is not set for the application
to which the concurrent program executable &EXECUTABLE belongs.
Shut down the concurrent managers. Set the basepath environment variable for the application. Restart the concurrent managers. (VARIABLE=)Routine AFPEOT cannot construct the name of an executable file for your concurrent request 6945444.
Check that the file name components are correct and valid on your system. Check that the environment for the person who started the concurrent manager

+Executing request completion options... -- 1) PUBLISH -------------+Beginning post-processing of request 6945444 on node TESTSERVER at 19-JUL-2011 15:23:24.Post-processing of request 6988838 failed at 19-JUL-2011 15:33:24 with the error message:The Output Post-processor is running but has not picked up this request. No further attempts will be made to post-process this request, and the request will be marked with Warning status.Setting the profile option Concurrent: OPP Response Timeout to a higher value may be necessary.+--------------------------------------+

Scenario:
=======

There is a Custom Top registered and that top is not set in the ENV while starting the conc.


Request Name :

XXTEST Daily Stock of PRODUCTS

Query to Check whether the Env is set for the Particular request :
===============================================

select variable_name,value from fnd_env_context e ,fnd_concurrent_processes p ,fnd_concurrent_requests r where p.concurrent_process_id =
e.concurrent_process_id and p.concurrent_process_id = r.controlling_manager and e.variable_name like '%CUSTOM_TOP%' and r.request_id = &request_id;

Solution :
=======

Add the respective Top in env file

ie) APPL_TOP/admin/adovars.env

so that while running autoconfig it wont remove every time.

XXTEST_TOP="/u01/apps/apps_st/appl/XXTEST/1.0.0" export XXTEST_TOP

APP-FND-00362: Routine afpbep cannot execute request &REQUEST for program &PROGRAM

APP-FND-00362: Routine afpbep cannot execute request &REQUEST for program &PROGRAM
Error:
=====

APP-FND-00362: Routine afpbep cannot execute request &REQUEST for program &PROGRAM, because the environment variable &BASEPATH is not set for the application
to which the concurrent program executable &EXECUTABLE belongs.
Shut down the concurrent managers. Set the basepath environment variable for the application. Restart the concurrent managers. (VARIABLE=)Routine AFPEOT cannot construct the name of an executable file for your concurrent request 6945444.
Check that the file name components are correct and valid on your system. Check that the environment for the person who started the concurrent manager

+Executing request completion options... -- 1) PUBLISH -------------+Beginning post-processing of request 6945444 on node TESTSERVER at 19-JUL-2011 15:23:24.Post-processing of request 6988838 failed at 19-JUL-2011 15:33:24 with the error message:The Output Post-processor is running but has not picked up this request. No further attempts will be made to post-process this request, and the request will be marked with Warning status.Setting the profile option Concurrent: OPP Response Timeout to a higher value may be necessary.+--------------------------------------+

Scenario:
=======

There is a Custom Top registered and that top is not set in the ENV while starting the conc.


Request Name :

XXTEST Daily Stock of PRODUCTS

Query to Check whether the Env is set for the Particular request :
===============================================

select variable_name,value from fnd_env_context e ,fnd_concurrent_processes p ,fnd_concurrent_requests r where p.concurrent_process_id =
e.concurrent_process_id and p.concurrent_process_id = r.controlling_manager and e.variable_name like '%CUSTOM_TOP%' and r.request_id = &request_id;

Solution :
=======

Add the respective Top in env file

ie) APPL_TOP/admin/adovars.env

so that while running autoconfig it wont remove every time.

XXTEST_TOP="/u01/apps/apps_st/appl/XXTEST/1.0.0" export XXTEST_TOP

OracleAS 10g (9.0.4) Installation Requirements for Linux Red Hat 4.0 [ID 315832.1]

OracleAS 10g (9.0.4) Installation Requirements for Linux Red Hat 4.0 [ID 315832.1]
		Modified 12-JUN-2009     Type BULLETIN     Status PUBLISHED

In this Document
  Purpose
  Scope and Application
  OracleAS 10g (9.0.4) Installation Requirements for Linux Red Hat 4.0
     Step 1: Software Requirements for Red Hat 4.0 Systems
     Step 2: Environment Setup
     Step 3:  Installation of Oracle Application Server 10g (9.0.4)
     Step 4: Oracle Application Server Middle Tier - Business Intelligence and Forms:
     Known Issues (Post-Installation)

---

Applies to:

Oracle Application Server 10g Enterprise Edition - Version: 9.0.4.0.0
Linux x86
Checked for relevance on 12-May-2009

Purpose

This document describes requirements and steps needed for installing Oracle Application Server 10g (9.0.4) on Red Hat Enterprise Linux (RHEL) 4.0.   Its main purpose is to list the certification requirements, over and beyond what the released Linux Installation Guide and Release Notes states, after Oracle's testing of this installation. This is what allows Oracle Application Server 10g (9.0.4) on Red Hat Enterprise Linux (RHEL) 4.0 to be "certified" within Metalink's certification status.

Scope and Application

• This is intended for any Application Server Administrator responsible for installing Oracle Application Server 10g (9.0.4) on Red Hat Enterprise Linux (RHEL) 4.0.
• The requirements and steps in this document override what is mentioned Oracle Application Server 10g (9.0.4) Quick Installation and Upgrade Guide, Oracle Application Server 10g(9.0.4) Installation Guide, and Upgrading to 10g (9.0.4) Guide. However, these must be used in conjunction with this Note.
• The requirements and steps described in this document are applicable for all the Oracle Application Server 10g (9.0.4) installation types.

OracleAS 10g (9.0.4) Installation Requirements for Linux Red Hat 4.0

Step 1: Software Requirements for Red Hat 4.0 Systems

Complete the following steps before installing Oracle Application Server on Red Hat 4.0 systems:

   1. Log in as the root user.
   2. Check that Red Hat Enterprise Linux AS/ES 4.0 is installed:

# cat /etc/issue

Red Hat Enterprise Linux AS release 4 (Nahant)

The minimum supported kernel and glibc version are:

          * kernel-smp-2.6.9-11.EL
          * kernel-2.6.9-11.EL
          * glibc-2.3.4-2.9

   3. Check that the following software packages, or higher versions, are installed:

          * gcc-3.4.3-22.1
          * gcc-c++-3.4.3-22.1
          * setarch-1.6-1
          * pdksh-5.2.14-30
          * openmotif21-2.1.30-11
          * gnome-libs-1.4.1.2.90-44.1
          * compat-libstdc++-296-2.96-132.7.2
          * compat-libgcc-296-2.96-132.7.2
          * compat-gcc-32-c++-3.2.3-47.3
          * compat-gcc-32-3.2.3-47.3
          * compat-libcwait-2.0-2
          * compat-oracle-rhel4-1.0-5
          * sysstat-5.0.5-1
          * gnome-libs-1.4.1.2.90-44.1
          * ORBit2-2.12.0-3
          * ORBit-0.5.17-14
          * tcp_wrappers-7.6-37.2
          * esound-0.2.35-2
          * audiofile-0.2.6-1

To determine whether a package is installed, enter a command similar to the following:

    # rpm -q package_name

If a package is missing, download it and install it using the following command:

    # rpm -i package_name

    The following packages can be obtained on Metalink in the form of a patch:

    compat-libcwait-2.0-2
    compat-oracle-rhel4-1.0-5

    These can be obtained from the following location:
    https://updates.oracle.com/download/4198954.html
    (Patch 4198954)

In addition to the packages listed, compat-oracle-rhel4-1.0-5.i386 requires two additional packages:

     libaio-0.3.103-3.i386.rpm
     libaio-devel-0.3.103-3.i386.rpm

compat-libcwait-2.0-2.i386.rpm and compat-oracle-rhel4-1.0-5.i386.rpm are Oracle packages, and not available from RHN. They provide backwards compatibility with prior distribution versions to allow Oracle software to install, relink, or run properly.

Package   : compat-libcwait-2.0-2.i386.rpm 
Summary  : Compatibility files for running Oracle

Description :
This package installs /usr/lib/libcwait.so to provide a missing symbol needed by some Oracle products.  This is also added to /etc/ld.so.preload.

Package   : compat-oracle-rhel4-1.0-5.i386.rpm
Summary  : Compatibility files for running Oracle on RHEL 4

Description :
This package provides gcc and g++ scripts that use libgcc.a 2.96 when linking to provide compatibility for certain Oracle products.  Since this package changes the behavior of gcc and g++, it should be installed when necessary and removed once it is no longer needed. NOTE: If upgrading gcc/g++, you must first uninstall this package,upgrade gcc/g++, and then reinstall this package if necessary.  See http://oss.oracle.com/projects/compat-oracle/ for more information.


Recommended Procedure to Apply Packages
1) After installing the base operating system, apply the latest update from RedHat

2) Apply all the remaining patches listed in step 1 (except for compat-libcwait-2.0-2.i386.rpm and compat-oracle-rhel4-1.0-5.i386.rpm). Some of these are prerequisites for compat-oracle-rhel4-1.0-5.i386.rpm.

3) Apply these packages (which are also pre-requisites for compat-oracle-rhel4-1.0-5.i386.rpm):

• libaio-0.3.103-3.i386.rpm
• libaio-devel-0.3.103-3.i386.rpm
• These will be available from RHN

4) Apply compat-libcwait-2.0-2.i386.rpm and compat-oracle-rhel4-1.0-5.i386.rpm, available as Patch 4198954.

Step 2: Environment Setup

The following environment variable needs to be set, in addition to what is mentioned in the Oracle AS10g Quick Installation and Upgrade Guide, Installation Guide, and Upgrade Guide, before invoking the installer.
Define and export the environment variable:

For ksh, bash:
export LD_ASSUME_KERNEL=2.4.19

For csh:
setenv LD_ASSUME_KERNEL=2.4.19

Step 3:  Installation of Oracle Application Server 10g (9.0.4)

1. Follow the instructions from Oracle Application Server 10g(9.0.4) documentation Mounting your CD-ROM or DVD to mount Oracle Application Server 10g(9.0.4) CD-ROM or DVD.
2. Before the installation, login as Oracle Application Server 10g(9.0.4) installation user and download and install the Patch 4467191 in a temporary area.
   
3. Invoke the runInstaller with –paramFile option pointing to the oraparam.ini file. Run the command as following to start up the Oracle Universal: /Disk1/runInstaller –paramFile /oraparam.ini

Step 4: Oracle Application Server Middle Tier - Business Intelligence and Forms:

1. Complete the Environment Setup in Step 2
2. Start the runInstaller following Step 3.
3. Continue the installation by following the steps documented in the Installation and Configuration Guide.
4. You will be prompted by the following, but before running the configuration script root.sh, a patch needs to be applied.

Warning:*** Alert: A configuration script needs to be run as root before installation can proceed. Leaving this window open, open another window and run '/ee/ias/in/ias904/biinst /root.sh' as root, then return to this window and click OK to continue. ***

User Selected: Yes/OK

• Apply the Patch 4217997 for Discoverer
  • Note:  Because of the need to apply at this stage of installation, its recommended to apply this patch whether or not Discoverer usage is planned.
• Proceed to run the root.sh as root.
• Continue with the installation, as documented 

Known Issues (Post-Installation)

   1. Enterprise Manager performance issues with consumption of CPU Usage:
          * Apply Patch 3856399 for the Enterprise Manager.
   2. Discoverer URL does not come up on RH 4.0 system:
          * Apply Patch 4421372 for Discoverer.
      

---

Related Products Middleware > Application Servers > Oracle Application Server > Oracle Fusion Middleware

Back to top

 Rate this document 

Relink fails for Oracle 9.2 patchset with undefined reference to dl_iterate_phdr@GLIBC_2.2.4

Relink fails for Oracle 9.2 patchset with undefined reference to dl_iterate_phdr@GLIBC_2.2.4 [ID 335515.1]
		Modified 21-JUL-2011     Type PROBLEM     Status PUBLISHED

In this Document
  Symptoms
  Cause
  Solution
  References

---

Applies to:

Oracle Server - Enterprise Edition - Version: 9.2.0.4 to 9.2.0.8 - Release: 9.2 to 9.2
Linux x86
Red Hat Advanced Server
Checked for relevance on 05-Feb-2010

Symptoms

While installing the 9.2.0.x.0 patchset on the linux x86 RedHat Server following error were encountered during the relink phase.
$ORACLE_HOME/install/make.log
------------------------------------------

/usr/lib/gcc/i386-redhat-linux/3.4.3/libgcc_s.so: undefined reference to `dl_iterate_phdr@GLIBC_2.2.4'
collect2: ld returned 1 exit status
make: *** [trcroute] Error 1

On checking:

Note.303859.1 Requirements for Installing Oracle 9iR2 on RHEL 4

We find all the required packages has been already installed.

# rpm -qa compat-oracle-rhel4

compat-oracle-rhel4-1.0-5

Cause

The System Administrator has installed the latest O/S updates from the RedHat.  This has caused the Oracle patch "compat-oracle-rhel4-1.0-5" to be overwritten.

# ls -l /usr/bin/gcc*

-rwxr-xr-x  2 root root 93120 May 17 01:36 /usr/bin/gcc
-rwxr-xr-x  2 root root 86364 Dec  1  2004 /usr/bin/gcc32
-rwxr-xr-x  1 root root 94784 May 17 01:36 /usr/bin/gcc.orig

# rpm -qs compat-oracle-rhel4

normal        /usr/bin/g++.tmp_1
normal        /usr/bin/gcc.tmp_1

Solution

Reinstall "compat-oracle-rhel4-1.0-5"

• Redownload the patch:4198954 from Oracle Metalink.
• Follow the instruction from the README to reapply the patch.

This time when you reapply the patch you will receive the following error.

# rpm -Uvh compat-oracle-rhel4-1.0-5.i386.rpm

Preparing...               ########################################### [100%]
        package compat-oracle-rhel4-1.0-5 is already installed

Use the following command to reapply the patch.

# rpm -Uvh --force compat-oracle-rhel4-1.0-5.i386.rpm

Preparing...               ########################################### [100%]
   1:compat-oracle-rhel4    ########################################### [100%]

# ls -l /usr/bin/gcc*

-rwxr-xr-x  1 root root   914 Feb 23  2005 /usr/bin/gcc
-rwxr-xr-x  2 root root 84780 Dec  1  2004 /usr/bin/gcc32
-rwxr-xr-x  1 root root 93120 Sep 26 10:28 /usr/bin/gcc.orig

References

NOTE:303859.1 - Requirements for Installing Oracle 9iR2 on RHEL 4

Related Products Oracle Database Products > Oracle Database > Oracle Database > Oracle Server - Enterprise Edition Errors ERROR 1

Back to top

 Rate this document 

AC-50480: Internal error occurred: java.lang.Exception: Error while generating listener.ora.

AC-50480: Internal error occurred: java.lang.Exception: Error while generating listener.ora. [ID 1324667.1]
		Modified 26-MAY-2011     Type PROBLEM     Status PUBLISHED

In this Document
  Symptoms
  Cause
  Solution

---

Applies to:

Oracle Application Install - Version: 12.1.3 and later   [Release: 12.1 and later ]
Information in this document applies to any platform.

Symptoms

When running autoconfig on the application Tier,
the following error occurs.

Error
##########################################################################
Generate Tns Names
##########################################################################

Updating s_tnsmode to 'generateTNS'
UpdateContext exited with status: 0
AC-50480: Internal error occurred: java.lang.Exception: Error while generating listener.ora.
Error generating tnsnames.ora from the database, temporary tnsnames.ora will be generated using templates
Instantiating Tools tnsnames.ora
Tools tnsnames.ora instantiated
Web tnsnames.ora instantiated

adgentns.pl exiting with status 2
ERRORCODE = 2 ERRORCODE_END
.end std out.

Cause

There is no information in the table FND_NODES for the database node.
This means that autoconfig has not been executed in the database tier or there was a problem
when it was executed in the database tier
.
So you need to run autoconfig on the database tier to gather the information.


There is no database node declared in the FND_NODES table

Solution

To implement the solution, please execute the following steps:

1. Ensure that you have taken a backup of your system before applying the recommended solution.

2. Clean the Data Topology Model
Sqlplus apps/...
SQL> EXEC FND_CONC_CLONE.SETUP_CLEAN;
COMMIT;
EXIT;

3. Run AutoConfig on all tiers, firstly on the DB tier and then the APPS tiers, to repopulate the required system tables

How to get a clean Autoconfig Environment ?

How to get a clean Autoconfig Environment ? [ID 391406.1]
		Modified 27-SEP-2011     Type HOWTO     Status PUBLISHED

In this Document
  Goal
  Solution
  References

---

Applies to:

Oracle Applications Technology Stack - Version: 11.5.10.2 to 12.1.3 - Release: 11.5.10 to 12.1
Information in this document applies to any platform.
***Checked for relevance on 30-Jun-2011***

Goal

The goal is to get a cleaned up Autoconfig Environment.

Solution

1. Clear the net services topology data model used by Autoconfig  - see Note 218089.1 - 'Autoconfig FAQ'

1.1 Question :
How do I deregister a APPLICATION-Tier from the Net Services Topology Data Model ?

1.1 Answer :
To deregister the current application tier from the Net Services Topology Data Model, invoke the following command :

perl /bin/adgentns.pl appspass= contextfile= -removeserver

1.2 Question :
How do I deregister a DATABASE-Tier from the Net Services Topology Data Model ?

1.2 Answer :
To deregister the current database tier from the Net Services Topology Data Model, invoke the following command :

perl /appsutil/bin/adgentns.pl appspass=< APPSpwd> contextfile= - removeserver

2. Clear the FND_NODES table as described in Note 260887.1 - 'Steps to Clean Nonexistent Nodes or IP Addresses from FND_NODES'

Execute following commands in SQLPlus :

EXEC FND_CONC_CLONE.SETUP_CLEAN;
COMMIT;
EXIT;

3. Run AutoConfig on all nodes - first on the DB-Tier and then on the APPS-Tier-node(s), to repopulate the required system tables.

Forms Process (FRMWEB) Consumes 100% of CPU in Oracle Applications R12

Forms Process (FRMWEB) Consumes 100% of CPU in Oracle Applications R12 [ID 745711.1]
		Modified 14-OCT-2009     Type PROBLEM     Status PUBLISHED

In this Document
  Symptoms
  Cause
  Solution
  References

---

Applies to:

Oracle Application Object Library - Version: 12.0 to 12.1
This problem can occur on any platform.

Symptoms

On Oracle Applications R12 when checking the top processes on the OS level for the middle tier, you find that forms process (frmweb) almost consumes 100% of the CPU.

Cause

The root cause of the issue is that returning rows from LOVs in core forms causes the forms process to grow up into memory depending on the number of rows returned.

When an end user login to forms and start working with LOV within core forms sometimes and according to the search criteria that the user will provide to filter the results in LOV, it may fetch huge numbers of records in which causes the frmweb process to grow very large, and in extreme cases this can even lock up the current process or even the whole machine.

So when executing a LOV query, every row is fetched into memory on the middle tier, the frmweb process can get extremely large, and the larger it gets the more likely it is to start paging.
Eventually it starts consuming excessive CPU just paging the process in and out of memory, which is probably what you can see here in this case as the amount of memory consumed when the LOV records are fetched into memory obviously depends on the amount of data in each record.

This has been mentioned in the following bug:
Bug 6519700 - ESC: CSE: R12SIP: 6513826 FRMWEB RUNAWAY PROCESS CONSUMING 100% CPU-MIDDLE TIER

Solution

To implement the solution, please execute the following steps:

1. Stop all services on the middle tier.

2. Set following forms environment variables:

FORMS_RECORD_GROUP_MAX to 10000 or if that proves too restrictive, increase it to 20000 or 30000.
FORMS_CATCHTERM=0

In order to set the above forms variables so next time autoconfig run does not override those values, do the following steps :

1- For Forms Variable "FORMS_CATCHTERM" the context vairable name is: "s_forms_catchterm" and you can update the context file located in ($INST_TOP/appl/admin/)

2- For other forms variable "FORMS_RECORD_GROUP_MAX" there is no variable defined in Autoconfig for that one and have to customize the autoconfig for the forms variables to set that environment as following:
a- Go to the autoconfig Template folder:
$cd $AD_TOP/admin/template
b- Create new directory named (custom)
$ mkdir custom
c- Make sure that new directory has same file permissions as ($AD_TOP/admin/template)
d- Copy the following autoconfig template to the new custom directory:
$cp $AD_TOP/admin/template/APPLSYS_ux.env $AD_TOP/admin/template/custom/APPLSYS_ux.env
e- Edit the file copied file under custom directory and add the following 2 lines at the end of section:

####################################
# Oracle Forms environment variables
####################################

FORMS_RECORD_GROUP_MAX=10000
export FORMS_RECORD_GROUP_MAX

f- Save and exit from the file.
g- Next time autoconfig run, it will read the custom directory and check for any customizations there. 

3. Run Autoconfig on the middle tier and make sure it is completed successfully.

4. Startup all services.

5. Monitor the forms process to see its CPU usage, and you will see that form process usage is reduced and not causing any more CPU consumption up to 100% as before.

6. Migrate the solution as appropriate to other environments.

IMPORTANT NOTE: If you find that setting (FORMS_RECORD_GROUP_MAX=30000) still too restrictive for certain LOV's then Oracle product team needs to consider redesigning the LOV's in their form and at that time raise a new SR with the Oracle Support team so they can raise a new bug with development for that specific LOV to get it redesigned but this may happen ONLY in RARE cases.

References

BUG:6519700 - ESC: CSE: R12SIP: 6513826 FRMWEB RUNAWAY PROCESS CONSUMING 100% CPU-MIDDLE TIER

---

Where To Locate The Oracle RDBMS Installation Guides, Upgrade Guides and Release Notes

Where To Locate The Oracle RDBMS Installation Guides, Upgrade Guides and Release Notes [ID 605251.1]
		Modified 28-SEP-2010     Type BULLETIN     Status PUBLISHED

In this Document
  Purpose
  Scope and Application
  Where To Locate The Oracle RDBMS Installation Guides, Upgrade Guides and Release Notes
     Oracle9i Release 2 (9.2) 
     Oracle10g Release 1 (10.1)
     Oracle10g Release 2 (10.2)
     Oracle11g Release 1 (11.1)
     Oracle11g Release 2 (11.2)

---

Applies to:

Oracle Server - Enterprise Edition - Version: 9.2.0.1 to 11.2.0.1.0 - Release: 9.2 to 11.2
Oracle Server - Enterprise Edition - Version: 9.2.0.1 to 11.2.0.1.0   [Release: 9.2 to 11.2]
Information in this document applies to any platform.

Purpose

The purpose of this bulletin is to inform customers where to locate the following on-line documentation for Oracle9i Release 2 (9.2) and higher:

• Oracle RDBMS Installation Guides,
• Upgrade Guides, and
• Release Notes

Scope and Application

This bulletin is only for customers who are trying to locate the on-line Oracle RDBMS Installation Guides, Upgrade Guides and Release Notes. It is not intended to help customers locate the equivalent guides for other Oracle products.

Where To Locate The Oracle RDBMS Installation Guides, Upgrade Guides and Release Notes

All of the following documentation is located on Oracle Technology Network (OTN). A username and password is required to access this site.

Oracle9i Release 2 (9.2)

http://www.oracle.com/pls/db92/db92.homepage

Installation Guide:

• HTML format
• PDF format

Migration Guide:

• HTML format
• PDF format

Release Notes for each platform are available from this page:

http://www.oracle.com/pls/db92/db92.docindex?remark=homepage#index-REL

Oracle10g Release 1 (10.1)

Installation Guides (for the Database, Client and Companion CD) for each platform are available from this page:

http://www.oracle.com/pls/db10g/db10g.homepage

Upgrade Guide:

• HTML format
• PDF format

Release Notes for each platform are available from this page:

http://www.oracle.com/technology/documentation/database10g.html

Oracle10g Release 2 (10.2)

Installation Guides and Release Notes for each platform are available from this page:

http://www.oracle.com/pls/db102/homepage

The following page provides links to the above plus links to the Client and Companion CD Installation Guides:

http://www.oracle.com/pls/db102/portal.portal_db?selected=11

Upgrade Guide:

• HTML format
• PDF format

Oracle11g Release 1 (11.1)

Installation Guides (for the Database, Client and Examples) and Release Notes for each platform are available from this page (in the "Installing and Upgrading" section of the documentation):

http://www.oracle.com/pls/db111/homepage

Upgrade Guide:

• HTML format
• PDF format

Oracle11g Release 2 (11.2)

Installation Guides (for the Database, Client and Examples) and Release Notes for each platform are available from this page (in the "Installing and Upgrading" section of the documentation):

http://www.oracle.com/pls/db112/homepage

Upgrade Guide:

• HTML format
• PDF format