ORA-00221: Error On Write To Control File

Problem Description:

Recently I was unable to connect to em12c console and after looking at the status of oms, I came to know that my database instance has crashed.

./emctl status oms Oracle Enterprise Manager Cloud Control 12c Release 2 Copyright (c) 1996, 2012 Oracle Corporation.  All rights reserved. WebTier is Up Oracle Management Server is not functioning because of the following reason: Failed to connect to repository database. OMS will be automatically restarted once it identifies that database and listener are up.

I logged in and verified the alert log which had below I/O errors on the controlfile. I was afraid afraid as my control files were not multiplexed but I do have a valid backup of my controlfile.

Errors in file /u01/app/oracle/diag/rdbms/trcoem/TRCOEM/trace/TRCOEM_lgwr_25655.trc: ORA-00345: redo log write error block 39818 count 15 ORA-00312: online log 1 thread 1: '/u01/app/oracle/oradata/TRCOEM/redo01.log' ORA-27072: File I/O error Linux-x86_64 Error: 5: Input/output error Additional information: 4 Additional information: 39818 Additional information: -1 Tue Jan 28 11:34:57 2014 Errors in file /u01/app/oracle/diag/rdbms/trcoem/TRCOEM/trace/TRCOEM_ckpt_25659.trc: ORA-00206: error in writing (block 3, # blocks 1) of control file ORA-00202: control file: '/u01/app/oracle/oradata/TRCOEM/control01.ctl' ORA-27072: File I/O error Linux-x86_64 Error: 5: Input/output error Additional information: 4 Additional information: 3 Additional information: -1 Errors in file /u01/app/oracle/diag/rdbms/trcoem/TRCOEM/trace/TRCOEM_ckpt_25659.trc: ORA-00221: error on write to control file ORA-00206: error in writing (block 3, # blocks 1) of control file ORA-00202: control file: '/u01/app/oracle/oradata/TRCOEM/control01.ctl' ORA-27072: File I/O error Linux-x86_64 Error: 5: Input/output error Additional information: 4 Additional information: 3 Additional information: -1 Tue Jan 28 11:34:57 2014 CKPT (ospid: 25659): terminating the instance due to error 221

Cause:

It can be a issue with filesystem, verify it and make sure the filesystems are mounted in read-write mode, not in read-only mode. Oracle has to be able to write to the files as well as read them.

Also, check the syslog for any I/O errors.

Syslog does not show any I/O errors.

Solution:
I followed the below steps to fix the issue.
1] Shutdown the database
2] Restore the control01.dbf from the backup location.
3] Mount the database
4] Recover database until cancel
5] Alter database open resetlogs.

Also, took a complete RMAN cold backup of the database and multiplexed control file.