ORA-27300: OS system dependent operation:fork failed with status: 11

Problem:

Recently our test database instance crashed, and  we had the following ORA error in alert log:-

Process startup failed, error stack:

ORA-27300: OS system dependent operation:fork failed with status: 11

ORA-27301: OS failure message: Resource temporarily unavailable

ORA-27302: failure occurred at: skgpspawn5

Cause:

The possible reasons could be

• Low memory on the server. 
• Maximum number of PROCESSES allowed per user may be too low. 
• User processes are not set correctly on the server. 
• Insufficient SWAP space on the server -- (RAM:SWAP -- 1:2 recommended in pre requisite)

Solution:

Basically we need to understand what does " operation:fork failed " means

So first question which comes to mind is What is fork?

fork() creates a new process by duplicating the calling process. The new process, referred to as the child, is an exact duplicate of the calling process, referred to as the parent.

Errors

EAGAIN

fork() cannot allocate sufficient memory to copy the parent's page tables and allocate a task structure for the child.

It looks we are running too low on the system parameter, so let's check the installation document and see if all installation requirements for our OS are met properly

I see, the value for kernel parameter pid_max is too low, due to this the kernel fails to allocate a new PID NUMBER as the max limit is already reached.

That's the reason we see this error in OS syslog :- "The fork(2) system call eventually returns -EAGAIN (11) when it fails to alloc a pid number"

So, we had to increase the sysctl tunable kernel.pid_max to be the same as the allowed number of user processes (ulimit -u)  plus add 8192 for Operating System stability.

• On Linux, you can find the maximum PID value for your system with this command:

            $ cat /proc/sys/kernel/pid_max

Currently, the values of kernel.pic_max in my env is set to:

cat /proc/sys/kernel/pid_max

32768

ulimit -u

63936

So I set the value of pid_max to 72128.

In order to make this change permanent, write it to the /etc/sysctl.conf file)

# sysctl -w kernel.pid_max=72128

HTH

Thanks for reading.