Search Blog Post

Wednesday, February 12, 2014

ORA-00221: Error On Write To Control File

Problem Description:
Recently I was unable to connect to em12c console and after looking at the status of oms, I came to know that my database instance has crashed.

./emctl status oms
Oracle Enterprise Manager Cloud Control 12c Release 2
Copyright (c) 1996, 2012 Oracle Corporation.  All rights reserved.
WebTier is Up
Oracle Management Server is not functioning because of the following reason:
Failed to connect to repository database. OMS will be automatically restarted once it identifies that database and listener are up.



I logged in and verified the alert log which had below I/O errors on the controlfile. I was afraid afraid as my control files were not multiplexed but I do have a valid backup of my controlfile.

Errors in file /u01/app/oracle/diag/rdbms/trcoem/TRCOEM/trace/TRCOEM_lgwr_25655.trc:
ORA-00345: redo log write error block 39818 count 15
ORA-00312: online log 1 thread 1: '/u01/app/oracle/oradata/TRCOEM/redo01.log'
ORA-27072: File I/O error
Linux-x86_64 Error: 5: Input/output error
Additional information: 4
Additional information: 39818
Additional information: -1
Tue Jan 28 11:34:57 2014
Errors in file /u01/app/oracle/diag/rdbms/trcoem/TRCOEM/trace/TRCOEM_ckpt_25659.trc:
ORA-00206: error in writing (block 3, # blocks 1) of control file
ORA-00202: control file: '/u01/app/oracle/oradata/TRCOEM/control01.ctl'
ORA-27072: File I/O error
Linux-x86_64 Error: 5: Input/output error
Additional information: 4
Additional information: 3
Additional information: -1
Errors in file /u01/app/oracle/diag/rdbms/trcoem/TRCOEM/trace/TRCOEM_ckpt_25659.trc:
ORA-00221: error on write to control file
ORA-00206: error in writing (block 3, # blocks 1) of control file
ORA-00202: control file: '/u01/app/oracle/oradata/TRCOEM/control01.ctl'
ORA-27072: File I/O error
Linux-x86_64 Error: 5: Input/output error
Additional information: 4
Additional information: 3
Additional information: -1
Tue Jan 28 11:34:57 2014
CKPT (ospid: 25659): terminating the instance due to error 221


Cause:
It can be a issue with filesystem, verify it and make sure the filesystems are mounted in read-write mode, not in read-only mode. Oracle has to be able to write to the files as well as read them.

Also, check the syslog for any I/O errors.
Syslog does not show any I/O errors.

Solution:
I followed the below steps to fix the issue.
1] Shutdown the database
2] Restore the control01.dbf from the backup location.
3] Mount the database
4] Recover database until cancel
5] Alter database open resetlogs.

Also, took a complete RMAN cold backup of the database and multiplexed control file.

1 comment: