ORA-27300: OS System Dependent Operation:fork Failed With Status: 11
APPLIES TO:
Oracle Database - Enterprise Edition - Version 11.2.0.3 and later
Linux x86-64
SYMPTOMS
After a System restart getting errors in the Database alert logs:
...
ORA-27300: OS system dependent operation:fork failed with status: 11
ORA-27301: OS failure message: Resource temporarily unavailable
ORA-27302: failure occurred at: skgpspawn3
Process PZ98 died, see its trace file
...
- This is a 2 node RAC environment with 6 Databases, and only some are having these errors.
- Top shows no concerns with memory or CPU.
- You have restarted all components in the Cluster (Listeners, Database, etc.) and the problem continues.
CHANGES
Increase in user processes on the server.
CAUSE
Low setting for the OS kernel parameter pid_max.
Root Cause Due to an increased process list average size, the kernel fails to allocate a new PID NUMBER, because its assignable range for PID numbers is temporarily exhausted; The fork(2) system call eventually returns -EAGAIN (11) when it fails to alloc a pid number, as can be verified at the following kernel/fork.c excerpt:1467 long do_fork(unsigned long clone_flags,[...]1475 struct pid *pid = alloc_pid();1476 long nr;1477 1478 if (!pid)1479 return -EAGAIN;[...]
SOLUTION
1. Increase sysctl tunable kernel.pid_max, in the same proportion as the allowed number of user processes (ulimit -u) was increased, plus some extra value for Operating System stability (approx. 8192), when configuring the system to fit the database sizing recommendations.
2. Verifying the actual limits set in /etc/security/limits.conf:[root@localhost ~]# cat /etc/security/limits.conf[...]oracle soft nproc 131072 oracle hard nproc 131072[...]
3. Setting the value found to 'nproc' limits as kernel.pid_max plus extra 8192 buffer (in order to make this change permanent, write it to the /etc/sysctl.conf file):[root@localhost ~]# sysctl -w kernel.pid_max=139264 kernel.pid_max = 139264
APPLIES TO:
Oracle Database - Enterprise Edition - Version 11.2.0.3 and later
Linux x86-64
SYMPTOMS
After a System restart getting errors in the Database alert logs:
...
ORA-27300: OS system dependent operation:fork failed with status: 11
ORA-27301: OS failure message: Resource temporarily unavailable
ORA-27302: failure occurred at: skgpspawn3
Process PZ98 died, see its trace file
...
- This is a 2 node RAC environment with 6 Databases, and only some are having these errors.
- Top shows no concerns with memory or CPU.
- You have restarted all components in the Cluster (Listeners, Database, etc.) and the problem continues.
CHANGES
Increase in user processes on the server.
CAUSE
Low setting for the OS kernel parameter pid_max.
Root Cause Due to an increased process list average size, the kernel fails to allocate a new PID NUMBER, because its assignable range for PID numbers is temporarily exhausted; The fork(2) system call eventually returns -EAGAIN (11) when it fails to alloc a pid number, as can be verified at the following kernel/fork.c excerpt:1467 long do_fork(unsigned long clone_flags,[...]1475 struct pid *pid = alloc_pid();1476 long nr;1477 1478 if (!pid)1479 return -EAGAIN;[...]
SOLUTION
1. Increase sysctl tunable kernel.pid_max, in the same proportion as the allowed number of user processes (ulimit -u) was increased, plus some extra value for Operating System stability (approx. 8192), when configuring the system to fit the database sizing recommendations.
2. Verifying the actual limits set in /etc/security/limits.conf:[root@localhost ~]# cat /etc/security/limits.conf[...]oracle soft nproc 131072 oracle hard nproc 131072[...]
3. Setting the value found to 'nproc' limits as kernel.pid_max plus extra 8192 buffer (in order to make this change permanent, write it to the /etc/sysctl.conf file):[root@localhost ~]# sysctl -w kernel.pid_max=139264 kernel.pid_max = 139264
This comment has been removed by a blog administrator.
ReplyDelete