Problem:
Recently
our test database instance crashed, and
we had the following ORA error in alert log:-
Process
startup failed, error stack:
ORA-27300:
OS system dependent operation:fork failed with status: 11
ORA-27301:
OS failure message: Resource temporarily unavailable
ORA-27302:
failure occurred at: skgpspawn5
Cause:
The
possible reasons could be
- Low memory on the server.
- Maximum number of PROCESSES allowed per user may be too low.
- User processes are not set correctly on the server.
- Insufficient SWAP space on the server -- (RAM:SWAP -- 1:2 recommended in pre requisite)
Solution:
Basically
we need to understand what does " operation:fork
failed " means
So
first question which comes to mind is What is fork?
fork()
creates a new process by duplicating the calling process. The new process,
referred to as the child, is an exact duplicate of the calling process,
referred to as the parent.
Errors
EAGAIN
fork()
cannot allocate sufficient memory to copy the parent's page tables and allocate
a task structure for the child.
It
looks we are running too low on the system parameter, so let's check the
installation document and see if all installation requirements for our OS are
met properly
I
see, the value for kernel parameter pid_max is too low, due to this the kernel
fails to allocate a new PID NUMBER as the max limit is already reached.
That's
the reason we see this error in OS syslog :- "The fork(2) system call
eventually returns -EAGAIN (11) when it fails to alloc a pid number"
So,
we had to increase the sysctl tunable kernel.pid_max to be the same as the
allowed number of user processes (ulimit -u)
plus add 8192 for Operating System stability.
- On Linux, you can find the maximum PID value for your system with this command:
$ cat /proc/sys/kernel/pid_max
Currently,
the values of kernel.pic_max in my env is set to:
cat
/proc/sys/kernel/pid_max
32768
ulimit
-u
63936
So
I set the value of pid_max to 72128.
In
order to make this change permanent, write it to the /etc/sysctl.conf file)
#
sysctl -w kernel.pid_max=72128
HTH
Thanks
for reading.
