In this Document
Abstract
History
Details
Setting up the Virtually External 11i Node
Step 1: Create a Home for the Virtually External 11i Tier
Step 2: Create a new Context File for the Virtually External 11i Tier
Step 3: Verify and Edit the Newly Created Context File
Step 4: Instantiate the New Configuration Files Based on the New Context File
Step 5: Finishing Steps
Touring the New Environment
Outstanding Issues
Summary
References
APPLIES TO:
Oracle Applications Technology Stack - Version 11.5.9 to 11.5.10.0 [Release 11.5]
Information in this document applies to any platform.
Checked for relevance on 28-SEP-2013
ABSTRACT
In the spirit of Note:364439.1, this whitepaper covers an example of an actual implementation of the complex "Reverse Proxy with No External Web Tier" configuration as described in Note:287176.1, sections 2.4 and 5.5.3.
Recommended prerequisites:
Note:287176.1 provides a technical overview of supported DMZ topologies
Note:364439.1 provides troubleshooting scripts, tips, and technical background information
HISTORY
Author : Dan Collier
Create Date 08-MAY-2007
Update Date 16-JUL-2008
Expire Date distant future
DETAILS
Setting up the Virtually External 11i Node
Upgrading from a simple single-tier or even multiple tier environment to a DMZ or other advanced topology is extremely difficult without some sort of preconceived plan in writing. When requesting support for an advanced configuration, customers should expect that a request for a topology diagram will be made before any advice beyond the generic can be offered. The "Reverse Proxy Alone in the DMZ" topology is no exception.
As always, the drawing should show each machine that is part of the instance, to include each server, firewall, and hardware load balancer (if any) with each described by their name, alias (if any), IP address, and the type of node installed (database, apache, forms, etc.) with appropriate version information. In general, the drawing should look something like those featured in Note:287176.1 and the top of 364439.1 with appropriate labels specific to the customer configuration. The problems most often faced are configuration related and it is not possible to know from the traces and configuration files where the functionality goes awry without a previous understanding of what it should look like. When troubleshooting, we start with the drawing and confirm that the various configuration parameters match the drawing.
In the above conceptual drawing, the classic DMZ configuration is split. Within the DMZ there is only the reverse proxy server, while within the internal network a single server makes use of the shared APPL_TOP and shared ORACLE_HOME technology to host and effectively perform the functions of both the internal as well as the external web tier.
The internal web tier should already have an entry in the fnd_nodes table with the IP address of the server itself. The virtually external web tier will be registered in the fnd_nodes table with the IP address of the reverse proxy server by using the new autoconfig variable s_server_ip_address. In this case, the term "virtually external" is used to denote the fact that the external web node, in common DMZ nomenclature, is physically located on the internal server within the internal network while appearing to Internet users to be a standalone, external server in the DMZ.
For background information on the use of shared ORACLE_HOME and shared APPL_TOP technology beyond this special case, see the following notes:
Note 233428.1 - Sharing the Application Tier File System in Oracle E-Business Suite 11i
Note 243880.1 - Frequently Asked Questions Sharing the Application Tier File System in Oracle Applications 11i
The virtually external 11i tier will still be run as the same applmgr user as the internal tier on this same server. Attempting to do this as another operating system user is difficult. Normally a shared APPL_TOP gets its permissions via an NFS share rule, but we won't be sharing via NFS in this case; the shared architecture of Note:287176.1/Section 2.4 is a clever derivation of this. For this virtually external 11i tier I created a config directory ahead of time. In a normal, shared environment this would be on a separate server, but in this case, it is on this same server. Second, my "reverse proxy" is generically named "external" for this exercise.
Step 1: Create a Home for the Virtually External 11i Tier
When using a shared filesystem, as covered by the referenced notes, not everything is shared. For example, the new node will need to have its own autoconfig context file, environment files, and configuration files. Where you choose to put these is a matter of personal choice; just keep it tidy. For example, when using Rapid Install to install an application node with a shared APPL_TOP, the node specific files install into $COMMON_TOP/conf by default. In my case, the config directory I created and specified is: /space/virtualext
Step 2: Create a new Context File for the Virtually External 11i Tier
Using the cloning script adclonectx.pl and my example of a config directory, the command syntax is simply (all on one line):
perl adclonectx.pl contextfile=/space/v115cu2/viscappl/admin/VISC_dcollierpc8.xml outfile=/space/virtualext/VISC_external.xml
In my case, the key questions from adclonectx.pl and my answers were:
Provide the values required for creation of the new APPL_TOP Context file.
Do you want to use a virtual hostname for the target node (y/n) [n] ?:y
Target hostname [dcollierpc8]:external
I took the default for essentially all of the other questions. This is important to do since this is not a normal clone. In this case, you definitely do want to use the same port pool as prompted by the adclonectx.pl script. Unlike a typical clone, you are not going to clone the database. You will be using the same database as the existing web tier.
When adclonectx.pl finished, it said:
The new APPL_TOP context file has been created :
/space/virtualext/VISC_external.xml
Step 3: Verify and Edit the Newly Created Context File
In my case, I found that all of the s_isXXX variables were NO, and therefore changed them appropriately:
For example:
YES
YES
These variables indicate that this new tier will be a Web node. Optionally, I could also specify "YES" for "s_isForms" and "s_isFormsDev" if I wanted this to be a forms node as well.
Then, in the section, pick new ports that don't conflict with the existing web server and specify them with the following variables:
s_webport
s_webport_pls
s_oprocmgr_port
Having to change these variables manually is a side effect of keeping the port pool in step 2. If you had chosen a different port pool, then you'd have to change close to a dozen variables to point back to the existing database.
Please note that the Java Object Cache Port value must remain same on all application context files.
Then, much like 287176.1 says, set:
3a. s_webentryhost - to the reverse proxy server hostname
3b. s_webentrydomain - to the domain name of the reverse proxy server
3c. s_active_webport - to the reverse proxy listener port
3d. s_webentryurlprotocol - to the reverse proxy's protocol e.g. "http" or "https"
3e. s_login_page - to the values you specified above for
://.:
For example: https://external.somecompany.com:443
3f. s_fnd_secure - confirm that this does not point to the same directory as the existing web node. This variable specifies the location of the node's DBC file.
For example: /space/v115cu2/viscappl/fnd/11/5/0/secure/VISC_external
3g. s_server_ip_address - to the IP address of the reverse proxy server. This is critical because this will be used to register this virtually external 11i tier in the fnd_nodes table. You need this node to be in the fnd_nodes table so that you can later select it when specifying node_trust_level in the remainder of the DMZ configuration.
Step 4: Instantiate the New Configuration Files Based on the New Context File
Execute the shared oracle home configuration script ( txkSOHM.pl ) to instantiate the required http server configuration files into the configuration home directory you created in step 1:
cd $FND_TOP/patch/115/bin
perl -I $AU_TOP/perl txkSOHM.pl
The "8.0.6 Shared Oracle Home" is just "$ORACLE_HOME" of the applmgr environment. Similarly, the "iAS Shared Oracle Home" is just "$IAS_ORACLE_HOME". Here's how I answered the questions txkSOHM.pl asked:
Absolute path of Application's Context XML file : /space/virtualext/VISC_external.xml
Type of Instance [primary/secondary] : secondary
Absolute path of 8.0.6 Shared Oracle Home : /space/v115cu2/viscora/8.0.6
Absolute path of iAS Shared Oracle Home : /space/v115cu2/viscora/iAS
Absolute path of config top : /space/virtualext
Step 5: Finishing Steps
The script txkSOHM.pl ends with a run of autoconfig. This creates a new scripts directory under the $COMMON_TOP such as the following:
appv512:dcpc8:/space/v115cu2/visccomn/admin/scripts> ls -l
drwxr-xr-x 2 appv512 dba 4096 Mar 30 18:39 VISC_dcollierpc8
drwxr-xr-x 2 appv512 dba 4096 Mar 30 18:35 VISC_external
In this example, VISC_dcollierpc8 is from my original, internal tier. VISC_external is the newly created directory for the virtually external 11i tier. You can start this new tier with the included adapcctl.sh script and it will listen on the port you told it to listen on. This port should be the same port that your reverse proxy server proxies to.
Touring the New Environment
In the existing, now shared, APPL_TOP/admin we have new environment files (Jun30):
appv512:dcpc8:/space/v115cu2/viscappl> cd $APPL_TOP
appv512:dcpc8:/space/v115cu2/viscappl> ls -lrt *.env
-rw-r--r-- 1 appv512 dba 1025 Jun 16 14:03 APPSVISC_dcollierpc8.env
-rw-r--r-- 1 appv512 dba 10674 Jun 16 14:03 VISC_dcollierpc8.env
-rw-r--r-- 1 appv512 dba 1008 Jun 30 13:53 APPSVISC_external.env
-rw-r--r-- 1 appv512 dba 1008 Jun 30 13:53 APPSORA.env
-rw-r--r-- 1 appv512 dba 10614 Jun 30 13:53 VISC_external.env
-rw-r--r-- 1 appv512 dba 10614 Jun 30 13:53 VISC.env
Sourcing the sid_server env file sets the environment for the virtually external tier so that we can wander around. This is otherwise unnecessary since the new scripts in the new scripts directory source them automatically.
appv512:dcpc8:/space/v115cu2/viscappl> . ./VISC_external.env
After this, a simple grep of the environment shows a number of useful environment variables (edited for content):
appv512:dcpc8:/space/v115cu2/viscappl> env | grep ext
APPLOUT=out/VISC_external
APPLLOG=log/VISC_external
APPLRGF=/space/v115cu2/visccomn/rgf/VISC_external
IAS_CONFIG_HOME=/space/virtualext/iAS
TOOLS_CONFIG_HOME=/space/virtualext/8.0.6
FND_SECURE=/space/v115cu2/viscappl/fnd/11.5.0/secure/VISC_external
CONTEXT_NAME=VISC_external
FORMS60_RTI_DIR=/space/v115cu2/visccomn/admin/log/VISC_external
APPLFENV=VISC_external.env
CONTEXT_FILE=/space/virtualext/VISC_external.xml
In the new configuration directory we indeed have the directories indicated by IAS_CONFIG_HOME and TOOLS_CONFIG_HOME:
appv512:dcpc8:/space/virtualext> ls -l
total 364
drwxr-xr-x 7 appv512 dba 4096 Jun 8 19:10 8.0.6
drwxr-xr-x 6 appv512 dba 4096 Jun 8 19:10 iAS
-rw-r--r-- 1 appv512 dba 82812 Jun 30 13:58 VISC_external.xml
Beneath the iAS directory (aka IAS_CONFIG_HOME) are the Apache and Jserv log files of the now running virtually external tier. For example:
appv512:dcpc8:.../visccomn/admin/scripts/VISC_external> cd $IAS_CONFIG_HOME
appv512:dcpc8:/space/virtualext/iAS> cd Apache/Apache/logs
appv512:dcpc8:/space/virtualext/iAS/Apache/Apache/logs> ls -l
-rw-r--r-- 1 appv512 dba 657 Jul 5 18:25 access_log.1183680000
-rw-r--r-- 1 appv512 dba 205 Jul 5 18:25 error_log
-rw-r--r-- 1 appv512 dba 205 Jul 5 18:25 error_log_pls
-rw-r--r-- 1 appv512 dba 5 Jul 5 18:25 httpd.pid
-rw-r--r-- 1 appv512 dba 5 Jul 5 18:25 httpd_pls.pid
-rwxr-xr-x 1 appv512 dba 0 Jul 5 18:25 opm_sys_mtx.pid.8606
-rw-r--r-- 1 appv512 dba 0 Jul 5 18:25 rewrite.lock
-rw-r--r-- 1 appv512 dba 0 Jul 5 18:25 rewrite.log
-rw------- 1 appv512 dba 0 Jul 5 18:25 sec_audit.log
-rw------- 1 appv512 dba 0 Jul 5 18:25 sec_debug.log
For more information on troubleshooting Apache/Jserv, see Note 230688.1.
Outstanding Issues
1. ORA-1422 During Login and Certain Other Navigations Due to Duplicate Webhost Entries
While most customers will not have this problem, I have personally encountered three instances where the login (in two cases) or the navigation to a particular responsibility (in one case) results in an error message that is best detailed in the access_log entry like the following:
10.1.2.3 - - [31/Oct/2007:12:09:07 -0400]
"GET /OA_HTML/AppsLocalLogin.jsp?
requestUrl=APPSHOMEPAGE
&cancelUrl=http%3A%2F%2Fsomeserver.domain.com
%3A8000%2Foa_servlets
%2Foracle.apps.fnd.sso.AppsLogin
&langCode=US
&errText=Please+contact+the+Web+Master+or+System+Administrator.
+ORA-01422%3A+exact+fetch+returns+more+than+requested+number+of+rows%0A
&username=SYSADMIN
&s1=....
HTTP/1.1" 200 6189
In this case, the ORA-1422 of the errText is thrown because the fnd_nodes.webhost column has the same value for both the internal tier and the virtually external tier. The immediate workaround is to update the fnd_nodes.webhost column to equal the fnd_nodes.node_name + the fnd_nodes.domain.
The fnd_nodes query from Note 364439.1 illustrates this clearly, below. Notice that the external server (literally named EXTERNAL) has the same webhost value as the internal server (DCOLLIERPC8):
NODE_NAME DOMAIN WEBHOST
--------------- ------------- ---------------
DCOLLIER10G oracle.com dcollier10g.oracle.com
AUTHENTICATION SECURE *
DCOLLIERPC8 oracle.com dcollierpc8.oracle.com
EXTERNAL oracle.com dcollierpc8.oracle.com
The current workaround is to update the table, as in the following example, from sqlplus as apps:
update fnd_nodes
set webhost = 'external.oracle.com'
where upper(node_name) = 'EXTERNAL';
2. The OACoreGroup JVM Fails to Bind to Port xxx Because it is on Another Server
This too is a very rare occurance, but some customers are reporting that their s_webhost AutoConfig variable, after running adclonectx.pl, has the name of their reverse proxy server when it should have the proper hostname of the internal web node. Left alone, this leads to the following problem in the jserv.log when starting apache and prevents logins by preventing the JVMs from starting:
Excerpt of jserv.log with debug enabled as described in Note 230688.1:
[05/10/2007 17:44:59:619 PKT] ApacheJServ/1.1.2 is starting...
[05/10/2007 17:44:59:619 PKT] using confFile /apps11i/external/iAS/Apache/Jserv/etc/jserv.properties
[05/10/2007 17:44:59:745 PKT] Connection authentication enabled
[05/10/2007 17:44:59:775 PKT] Connection allowed from /127.0.0.1
[05/10/2007 17:44:59:778 PKT] Connection allowed from someserver.domain.com/10.1.2.3
[05/10/2007 17:44:59:779 PKT] port 16000 is specified in properties file
[05/10/2007 17:44:59:780 PKT] port 16001 is specified in properties file
...
[05/10/2007 17:44:59:786 PKT] try to start on port 16000
[05/10/2007 17:44:59:791 PKT] cannot create socket on 16000
[05/10/2007 17:44:59:792 PKT] try to start on port 16001
[05/10/2007 17:44:59:795 PKT] cannot create socket on 16001
...
[05/10/2007 17:44:59:807 PKT]
ApacheJServ/1.1.2: Failed to bind to port(s) specified in /apps11i/external/iAS/Apache/Jserv/etc/jserv.properties. Please check /apps11i/external/iAS/Apache/Jserv/etc/jserv.properties and jserv.conf file, and make sure number of JServ process specified in jserv.conf is less than number of ports specified in /apps11i/external/iAS/Apache/Jserv/etc/jserv.properties. and the ports are not used by other processes.
The typical solution for this type of problem is to use tools such as lsof and netstat to determine if the ports are free, but in this case the problem is that the autoconfig variable s_webhost refers to the reverse proxy server and this causes the BindAddress specified in the jserv.properties to also indicate the name of the reverse proxy server. This is clearly wrong because it results in the Apache mod_jserv on the web tier to try and bind to ports on the reverse proxy machine and quickly fail.
The solution is simply to set the AutoConfig variable s_webhost, in the context file for the virtually external tier, to the same s_webhost used in the internal web node's context file on this same machine. Then, run AutoConfig for the virtually external tier.
SUMMARY
This note is the second of a currently three part series of DMZ troubleshooting notes and provides a crisp walkthrough of an actual implementation of the complex "Reverse Proxy with No External Web Tier" configuration as described in Note 287176.1, sections 2.4 and 5.5.3.
Abstract
History
Details
Setting up the Virtually External 11i Node
Step 1: Create a Home for the Virtually External 11i Tier
Step 2: Create a new Context File for the Virtually External 11i Tier
Step 3: Verify and Edit the Newly Created Context File
Step 4: Instantiate the New Configuration Files Based on the New Context File
Step 5: Finishing Steps
Touring the New Environment
Outstanding Issues
Summary
References
APPLIES TO:
Oracle Applications Technology Stack - Version 11.5.9 to 11.5.10.0 [Release 11.5]
Information in this document applies to any platform.
Checked for relevance on 28-SEP-2013
ABSTRACT
In the spirit of Note:364439.1, this whitepaper covers an example of an actual implementation of the complex "Reverse Proxy with No External Web Tier" configuration as described in Note:287176.1, sections 2.4 and 5.5.3.
Recommended prerequisites:
Note:287176.1 provides a technical overview of supported DMZ topologies
Note:364439.1 provides troubleshooting scripts, tips, and technical background information
HISTORY
Author : Dan Collier
Create Date 08-MAY-2007
Update Date 16-JUL-2008
Expire Date distant future
DETAILS
Setting up the Virtually External 11i Node
Upgrading from a simple single-tier or even multiple tier environment to a DMZ or other advanced topology is extremely difficult without some sort of preconceived plan in writing. When requesting support for an advanced configuration, customers should expect that a request for a topology diagram will be made before any advice beyond the generic can be offered. The "Reverse Proxy Alone in the DMZ" topology is no exception.
As always, the drawing should show each machine that is part of the instance, to include each server, firewall, and hardware load balancer (if any) with each described by their name, alias (if any), IP address, and the type of node installed (database, apache, forms, etc.) with appropriate version information. In general, the drawing should look something like those featured in Note:287176.1 and the top of 364439.1 with appropriate labels specific to the customer configuration. The problems most often faced are configuration related and it is not possible to know from the traces and configuration files where the functionality goes awry without a previous understanding of what it should look like. When troubleshooting, we start with the drawing and confirm that the various configuration parameters match the drawing.
In the above conceptual drawing, the classic DMZ configuration is split. Within the DMZ there is only the reverse proxy server, while within the internal network a single server makes use of the shared APPL_TOP and shared ORACLE_HOME technology to host and effectively perform the functions of both the internal as well as the external web tier.
The internal web tier should already have an entry in the fnd_nodes table with the IP address of the server itself. The virtually external web tier will be registered in the fnd_nodes table with the IP address of the reverse proxy server by using the new autoconfig variable s_server_ip_address. In this case, the term "virtually external" is used to denote the fact that the external web node, in common DMZ nomenclature, is physically located on the internal server within the internal network while appearing to Internet users to be a standalone, external server in the DMZ.
For background information on the use of shared ORACLE_HOME and shared APPL_TOP technology beyond this special case, see the following notes:
Note 233428.1 - Sharing the Application Tier File System in Oracle E-Business Suite 11i
Note 243880.1 - Frequently Asked Questions Sharing the Application Tier File System in Oracle Applications 11i
The virtually external 11i tier will still be run as the same applmgr user as the internal tier on this same server. Attempting to do this as another operating system user is difficult. Normally a shared APPL_TOP gets its permissions via an NFS share rule, but we won't be sharing via NFS in this case; the shared architecture of Note:287176.1/Section 2.4 is a clever derivation of this. For this virtually external 11i tier I created a config directory ahead of time. In a normal, shared environment this would be on a separate server, but in this case, it is on this same server. Second, my "reverse proxy" is generically named "external" for this exercise.
Step 1: Create a Home for the Virtually External 11i Tier
When using a shared filesystem, as covered by the referenced notes, not everything is shared. For example, the new node will need to have its own autoconfig context file, environment files, and configuration files. Where you choose to put these is a matter of personal choice; just keep it tidy. For example, when using Rapid Install to install an application node with a shared APPL_TOP, the node specific files install into $COMMON_TOP/conf by default. In my case, the config directory I created and specified is: /space/virtualext
Step 2: Create a new Context File for the Virtually External 11i Tier
Using the cloning script adclonectx.pl and my example of a config directory, the command syntax is simply (all on one line):
perl adclonectx.pl contextfile=/space/v115cu2/viscappl/admin/VISC_dcollierpc8.xml outfile=/space/virtualext/VISC_external.xml
In my case, the key questions from adclonectx.pl and my answers were:
Provide the values required for creation of the new APPL_TOP Context file.
Do you want to use a virtual hostname for the target node (y/n) [n] ?:y
Target hostname [dcollierpc8]:external
I took the default for essentially all of the other questions. This is important to do since this is not a normal clone. In this case, you definitely do want to use the same port pool as prompted by the adclonectx.pl script. Unlike a typical clone, you are not going to clone the database. You will be using the same database as the existing web tier.
When adclonectx.pl finished, it said:
The new APPL_TOP context file has been created :
/space/virtualext/VISC_external.xml
Step 3: Verify and Edit the Newly Created Context File
In my case, I found that all of the s_isXXX variables were NO, and therefore changed them appropriately:
For example:
These variables indicate that this new tier will be a Web node. Optionally, I could also specify "YES" for "s_isForms" and "s_isFormsDev" if I wanted this to be a forms node as well.
Then, in the
s_webport
s_webport_pls
s_oprocmgr_port
Having to change these variables manually is a side effect of keeping the port pool in step 2. If you had chosen a different port pool, then you'd have to change close to a dozen variables to point back to the existing database.
Please note that the Java Object Cache Port value must remain same on all application context files.
Then, much like 287176.1 says, set:
3a. s_webentryhost - to the reverse proxy server hostname
3b. s_webentrydomain - to the domain name of the reverse proxy server
3c. s_active_webport - to the reverse proxy listener port
3d. s_webentryurlprotocol - to the reverse proxy's protocol e.g. "http" or "https"
3e. s_login_page - to the values you specified above for
For example: https://external.somecompany.com:443
3f. s_fnd_secure - confirm that this does not point to the same directory as the existing web node. This variable specifies the location of the node's DBC file.
For example: /space/v115cu2/viscappl/fnd/11/5/0/secure/VISC_external
3g. s_server_ip_address - to the IP address of the reverse proxy server. This is critical because this will be used to register this virtually external 11i tier in the fnd_nodes table. You need this node to be in the fnd_nodes table so that you can later select it when specifying node_trust_level in the remainder of the DMZ configuration.
Step 4: Instantiate the New Configuration Files Based on the New Context File
Execute the shared oracle home configuration script ( txkSOHM.pl ) to instantiate the required http server configuration files into the configuration home directory you created in step 1:
cd $FND_TOP/patch/115/bin
perl -I $AU_TOP/perl txkSOHM.pl
The "8.0.6 Shared Oracle Home" is just "$ORACLE_HOME" of the applmgr environment. Similarly, the "iAS Shared Oracle Home" is just "$IAS_ORACLE_HOME". Here's how I answered the questions txkSOHM.pl asked:
Absolute path of Application's Context XML file : /space/virtualext/VISC_external.xml
Type of Instance [primary/secondary] : secondary
Absolute path of 8.0.6 Shared Oracle Home : /space/v115cu2/viscora/8.0.6
Absolute path of iAS Shared Oracle Home : /space/v115cu2/viscora/iAS
Absolute path of config top : /space/virtualext
Step 5: Finishing Steps
The script txkSOHM.pl ends with a run of autoconfig. This creates a new scripts directory under the $COMMON_TOP such as the following:
appv512:dcpc8:/space/v115cu2/visccomn/admin/scripts> ls -l
drwxr-xr-x 2 appv512 dba 4096 Mar 30 18:39 VISC_dcollierpc8
drwxr-xr-x 2 appv512 dba 4096 Mar 30 18:35 VISC_external
In this example, VISC_dcollierpc8 is from my original, internal tier. VISC_external is the newly created directory for the virtually external 11i tier. You can start this new tier with the included adapcctl.sh script and it will listen on the port you told it to listen on. This port should be the same port that your reverse proxy server proxies to.
Touring the New Environment
In the existing, now shared, APPL_TOP/admin we have new environment files (Jun30):
appv512:dcpc8:/space/v115cu2/viscappl> cd $APPL_TOP
appv512:dcpc8:/space/v115cu2/viscappl> ls -lrt *.env
-rw-r--r-- 1 appv512 dba 1025 Jun 16 14:03 APPSVISC_dcollierpc8.env
-rw-r--r-- 1 appv512 dba 10674 Jun 16 14:03 VISC_dcollierpc8.env
-rw-r--r-- 1 appv512 dba 1008 Jun 30 13:53 APPSVISC_external.env
-rw-r--r-- 1 appv512 dba 1008 Jun 30 13:53 APPSORA.env
-rw-r--r-- 1 appv512 dba 10614 Jun 30 13:53 VISC_external.env
-rw-r--r-- 1 appv512 dba 10614 Jun 30 13:53 VISC.env
Sourcing the sid_server env file sets the environment for the virtually external tier so that we can wander around. This is otherwise unnecessary since the new scripts in the new scripts directory source them automatically.
appv512:dcpc8:/space/v115cu2/viscappl> . ./VISC_external.env
After this, a simple grep of the environment shows a number of useful environment variables (edited for content):
appv512:dcpc8:/space/v115cu2/viscappl> env | grep ext
APPLOUT=out/VISC_external
APPLLOG=log/VISC_external
APPLRGF=/space/v115cu2/visccomn/rgf/VISC_external
IAS_CONFIG_HOME=/space/virtualext/iAS
TOOLS_CONFIG_HOME=/space/virtualext/8.0.6
FND_SECURE=/space/v115cu2/viscappl/fnd/11.5.0/secure/VISC_external
CONTEXT_NAME=VISC_external
FORMS60_RTI_DIR=/space/v115cu2/visccomn/admin/log/VISC_external
APPLFENV=VISC_external.env
CONTEXT_FILE=/space/virtualext/VISC_external.xml
In the new configuration directory we indeed have the directories indicated by IAS_CONFIG_HOME and TOOLS_CONFIG_HOME:
appv512:dcpc8:/space/virtualext> ls -l
total 364
drwxr-xr-x 7 appv512 dba 4096 Jun 8 19:10 8.0.6
drwxr-xr-x 6 appv512 dba 4096 Jun 8 19:10 iAS
-rw-r--r-- 1 appv512 dba 82812 Jun 30 13:58 VISC_external.xml
Beneath the iAS directory (aka IAS_CONFIG_HOME) are the Apache and Jserv log files of the now running virtually external tier. For example:
appv512:dcpc8:.../visccomn/admin/scripts/VISC_external> cd $IAS_CONFIG_HOME
appv512:dcpc8:/space/virtualext/iAS> cd Apache/Apache/logs
appv512:dcpc8:/space/virtualext/iAS/Apache/Apache/logs> ls -l
-rw-r--r-- 1 appv512 dba 657 Jul 5 18:25 access_log.1183680000
-rw-r--r-- 1 appv512 dba 205 Jul 5 18:25 error_log
-rw-r--r-- 1 appv512 dba 205 Jul 5 18:25 error_log_pls
-rw-r--r-- 1 appv512 dba 5 Jul 5 18:25 httpd.pid
-rw-r--r-- 1 appv512 dba 5 Jul 5 18:25 httpd_pls.pid
-rwxr-xr-x 1 appv512 dba 0 Jul 5 18:25 opm_sys_mtx.pid.8606
-rw-r--r-- 1 appv512 dba 0 Jul 5 18:25 rewrite.lock
-rw-r--r-- 1 appv512 dba 0 Jul 5 18:25 rewrite.log
-rw------- 1 appv512 dba 0 Jul 5 18:25 sec_audit.log
-rw------- 1 appv512 dba 0 Jul 5 18:25 sec_debug.log
For more information on troubleshooting Apache/Jserv, see Note 230688.1.
Outstanding Issues
1. ORA-1422 During Login and Certain Other Navigations Due to Duplicate Webhost Entries
While most customers will not have this problem, I have personally encountered three instances where the login (in two cases) or the navigation to a particular responsibility (in one case) results in an error message that is best detailed in the access_log entry like the following:
10.1.2.3 - - [31/Oct/2007:12:09:07 -0400]
"GET /OA_HTML/AppsLocalLogin.jsp?
requestUrl=APPSHOMEPAGE
&cancelUrl=http%3A%2F%2Fsomeserver.domain.com
%3A8000%2Foa_servlets
%2Foracle.apps.fnd.sso.AppsLogin
&langCode=US
&errText=Please+contact+the+Web+Master+or+System+Administrator.
+ORA-01422%3A+exact+fetch+returns+more+than+requested+number+of+rows%0A
&username=SYSADMIN
&s1=....
HTTP/1.1" 200 6189
In this case, the ORA-1422 of the errText is thrown because the fnd_nodes.webhost column has the same value for both the internal tier and the virtually external tier. The immediate workaround is to update the fnd_nodes.webhost column to equal the fnd_nodes.node_name + the fnd_nodes.domain.
The fnd_nodes query from Note 364439.1 illustrates this clearly, below. Notice that the external server (literally named EXTERNAL) has the same webhost value as the internal server (DCOLLIERPC8):
NODE_NAME DOMAIN WEBHOST
--------------- ------------- ---------------
DCOLLIER10G oracle.com dcollier10g.oracle.com
AUTHENTICATION SECURE *
DCOLLIERPC8 oracle.com dcollierpc8.oracle.com
EXTERNAL oracle.com dcollierpc8.oracle.com
The current workaround is to update the table, as in the following example, from sqlplus as apps:
update fnd_nodes
set webhost = 'external.oracle.com'
where upper(node_name) = 'EXTERNAL';
2. The OACoreGroup JVM Fails to Bind to Port xxx Because it is on Another Server
This too is a very rare occurance, but some customers are reporting that their s_webhost AutoConfig variable, after running adclonectx.pl, has the name of their reverse proxy server when it should have the proper hostname of the internal web node. Left alone, this leads to the following problem in the jserv.log when starting apache and prevents logins by preventing the JVMs from starting:
Excerpt of jserv.log with debug enabled as described in Note 230688.1:
[05/10/2007 17:44:59:619 PKT]
[05/10/2007 17:44:59:619 PKT]
[05/10/2007 17:44:59:745 PKT]
[05/10/2007 17:44:59:775 PKT]
[05/10/2007 17:44:59:778 PKT]
[05/10/2007 17:44:59:779 PKT]
[05/10/2007 17:44:59:780 PKT]
...
[05/10/2007 17:44:59:786 PKT]
[05/10/2007 17:44:59:791 PKT]
[05/10/2007 17:44:59:792 PKT]
[05/10/2007 17:44:59:795 PKT]
...
[05/10/2007 17:44:59:807 PKT]
ApacheJServ/1.1.2: Failed to bind to port(s) specified in /apps11i/external/iAS/Apache/Jserv/etc/jserv.properties. Please check /apps11i/external/iAS/Apache/Jserv/etc/jserv.properties and jserv.conf file, and make sure number of JServ process specified in jserv.conf is less than number of ports specified in /apps11i/external/iAS/Apache/Jserv/etc/jserv.properties. and the ports are not used by other processes.
The typical solution for this type of problem is to use tools such as lsof and netstat to determine if the ports are free, but in this case the problem is that the autoconfig variable s_webhost refers to the reverse proxy server and this causes the BindAddress specified in the jserv.properties to also indicate the name of the reverse proxy server. This is clearly wrong because it results in the Apache mod_jserv on the web tier to try and bind to ports on the reverse proxy machine and quickly fail.
The solution is simply to set the AutoConfig variable s_webhost, in the context file for the virtually external tier, to the same s_webhost used in the internal web node's context file on this same machine. Then, run AutoConfig for the virtually external tier.
SUMMARY
This note is the second of a currently three part series of DMZ troubleshooting notes and provides a crisp walkthrough of an actual implementation of the complex "Reverse Proxy with No External Web Tier" configuration as described in Note 287176.1, sections 2.4 and 5.5.3.
This comment has been removed by a blog administrator.
ReplyDelete