Blog Post: Oracle 12c (12.1.0.2) : Enhancement made to the alternate archive dest[ination] handling

In one of my earlier post , I had discussed about the option of setting up a alternate archive destination in Oracle database. We have seen that, how we can leverage the alternate archive destination in the event of any failure (space , hardware, etc.) in the primary archive destination. However, there was an issue with that setup. We had to manually reset the archive configuration, once the issue with primary archive destination was fixed and the archive destination was made available again. Oracle was not able to automatically failback to the primary archive destination. Oracle has answered this concern with release 12.1.0.2. Oracle is now capable of AUTOMATIC failback from alternate to the primary archive destination once the primary destination is made available. For the AUTOMATIC failback to work, we need to set the alternate log archive destination as the ALTERNATE of primary log archive destination and the primary log archive destination as the ALTERNATE of alternate log archive destination . In this way, when the primary destination fails, Oracle will continue archiving the logs to the alternate destination and the moment when primary destination is restored Oracle will automatically failback to the primary archive destination without any manual intervention. Let’s go through a quick demonstration to understand how it works. In the following demonstration, the primary archive destination is set to /arch1/primary_dest location, which is currently 99% utilized. ---// ---// checking primary archive destination //--- ---// SQL> select version from v$instance; VERSION ----------------- 12.1.0.2.0 SQL> select DEST_NAME,TARGET,DESTINATION,VALID_ROLE,STATUS from v$archive_dest where status!='INACTIVE'; DEST_NAME TARGET DESTINATION VALID_ROLE STATUS ------------------------------ ---------------- ------------------------------ ------------ --------- LOG_ARCHIVE_DEST_1 PRIMARY /arch1/primary_dest ALL_ROLES VALID ---// ---// primary archive dest is 99% utilized //--- ---// SQL> !df -h /arch1/primary_dest Filesystem Size Used Avail Use% Mounted on /dev/sdf1 1004M 940M 14M 99% /arch1 Let’s add an alternate archive destination /arch2/alternate_dest , which can be utilized in the event of primary destination being unavailable. ---// ---// adding alternate archive dest (LOG_ARCHIVE_DEST_2) //--- ---// SQL> !df -h /arch2/alternate_dest Filesystem Size Used Avail Use% Mounted on /dev/sdg1 1004M 18M 936M 2% /arch2 SQL> alter system set log_archive_dest_2='LOCATION=/arch2/alternate_dest' scope=both; System altered. SQL> alter system set log_archive_dest_state_2=ALTERNATE scope=both; System altered. SQL> select DEST_NAME,TARGET,DESTINATION,VALID_ROLE,STATUS,ALTERNATE from v$archive_dest where status!='INACTIVE'; DEST_NAME TARGET DESTINATION VALID_ROLE STATUS ALTERNATE ------------------------------ ---------------- ------------------------------ ------------ --------- ------------------------------ LOG_ARCHIVE_DEST_1 PRIMARY /arch1/primary_dest ALL_ROLES VALID NONE LOG_ARCHIVE_DEST_2 PRIMARY /arch2/alternate_dest ALL_ROLES ALTERNATE NONE At this point, we have added an alternate archive destination (LOG_ARCHIVE_DEST_2). However, we haven’t yet linked (ALTERNATE column is showing NONE) this alternate destination to a primary archive destination. Let’s link the alternate archive destination (LOG_ARCHIVE_DEST_2) to the primary archive destination (LOG_ARCHIVE_DEST_1) as shown below. ---// ---// linking alternate archive dest to the respective primary archive dest //--- ---// SQL> alter system set log_archive_dest_1='LOCATION=/arch1/primary_dest NOREOPEN ALTERNATE=LOG_ARCHIVE_DEST_2' scope=both; System altered. SQL> select DEST_NAME,TARGET,DESTINATION,VALID_ROLE,STATUS,ALTERNATE from v$archive_dest where status!='INACTIVE'; DEST_NAME TARGET DESTINATION VALID_ROLE STATUS ALTERNATE ------------------------------ ---------------- ------------------------------ ------------ --------- ------------------------------ LOG_ARCHIVE_DEST_1 PRIMARY /arch1/primary_dest ALL_ROLES VALID LOG_ARCHIVE_DEST_2 LOG_ARCHIVE_DEST_2 PRIMARY /arch2/alternate_dest ALL_ROLES ALTERNATE NONE Once we link the primary and alternate archive destinations (ALTERNATE column is now pointing to LOG_ARCHIVE_DEST_2 for primary archive dest LOG_ARCHIVE_DEST_1), Oracle will be able to failover to alternate archive destination at the event of primary destination failure. However, Oracle will not be able to failback to primary destination once it becomes available. For that to work, we need to set the primary archive destination LOG_ARCHIVE_DEST_1 as the ALTERNATE for alternate archive destination LOG_ARCHIVE_DEST_2 as shown below. Note: The rules about REOPEN and MAX_FAILURE properties remain the same as stated in my previous post . ---// ---// inter link primary and alternate archive dest //--- ---// SQL> alter system set log_archive_dest_2='LOCATION=/arch2/alternate_dest NOREOPEN ALTERNATE=LOG_ARCHIVE_DEST_1' scope=both; System altered. SQL> select DEST_NAME,TARGET,DESTINATION,VALID_ROLE,STATUS,ALTERNATE from v$archive_dest where status!='INACTIVE'; DEST_NAME TARGET DESTINATION VALID_ROLE STATUS ALTERNATE ------------------------------ ---------------- ------------------------------ ------------ --------- ------------------------------ LOG_ARCHIVE_DEST_1 PRIMARY /arch1/primary_dest ALL_ROLES VALID LOG_ARCHIVE_DEST_2 LOG_ARCHIVE_DEST_2 PRIMARY /arch2/alternate_dest ALL_ROLES ALTERNATE LOG_ARCHIVE_DEST_1 As we can see, now the primary and alternate archive destinations are inter linked. Oracle should be now able to failback to primary once the primary is made available. Now, since the primary archive dest was 99% utilized, after few DML operations, Oracle failed over to the alternate archive destination LOG_ARCHIVE_DEST_2 as found from the alert log file (Sequence# 965 and 966 were archived to dest 2) . ##--- ##--- alternate archive dest activated ---## ##--- Wed Jun 08 17:13:40 2016 ARC3: Encountered disk I/O error 19502 Wed Jun 08 17:13:40 2016 ARC3: Closing local archive destination LOG_ARCHIVE_DEST_1: '/arch1/primary_dest/1_965_903715118.dbf' (error 19502) (orpcdb1) Wed Jun 08 17:13:40 2016 Errors in file /app/oracle/diag/rdbms/orpcdb1/orpcdb1/trace/orpcdb1_arc3_8293.trc: ORA-27072: File I/O error Additional information: 4 Additional information: 14336 Additional information: 188416 ORA-19502: write error on file "/arch1/primary_dest/1_965_903715118.dbf", block number 14336 (block size=512) Wed Jun 08 17:13:40 2016 Errors in file /app/oracle/diag/rdbms/orpcdb1/orpcdb1/trace/orpcdb1_arc3_8293.trc: ORA-19502: write error on file "/arch1/primary_dest/1_965_903715118.dbf", block number 14336 (block size=512) ORA-27072: File I/O error Additional information: 4 Additional information: 14336 Additional information: 188416 ORA-19502: write error on file "/arch1/primary_dest/1_965_903715118.dbf", block number 14336 (block size=512) ARC3: I/O error 19502 archiving log 2 to '/arch1/primary_dest/1_965_903715118.dbf' Wed Jun 08 17:13:42 2016 Archived Log entry 205 added for thread 1 sequence 965 ID 0xaeb9a9ee dest 2: Wed Jun 08 17:13:45 2016 Thread 1 advanced to log sequence 967 (LGWR switch) Current log# 1 seq# 967 mem# 0: /data/oracle/orpcdb1/redo_01.log Wed Jun 08 17:13:47 2016 Archived Log entry 206 added for thread 1 sequence 966 ID 0xaeb9a9ee dest 2: When the alternate archive destination gets activated (due to primary destination failure), the respective primary destination acts as the alternate destination as found from the following query. ---// ---// primary becomes alternate when alternate dest is activated //--- ---// SQL> select DEST_NAME,TARGET,DESTINATION,VALID_ROLE,STATUS,ALTERNATE from v$archive_dest where status!='INACTIVE'; DEST_NAME TARGET DESTINATION VALID_ROLE STATUS ALTERNATE ------------------------------ ---------------- ------------------------------ ------------ --------- ------------------------------ LOG_ARCHIVE_DEST_1 PRIMARY /arch1/primary_dest ALL_ROLES ALTERNATE LOG_ARCHIVE_DEST_2 LOG_ARCHIVE_DEST_2 PRIMARY /arch2/alternate_dest ALL_ROLES VALID LOG_ARCHIVE_DEST_1 However unlike pre 12c database (where primary destination gets marked as DISABLED upon failure), here the role reversal is not permanent. In 12c, when both primary and alternate archive destination are inter linked, Oracle always tries to archive the logs to the primary destination. If primary destination is not available, it fails over to the alternate destination. However, it doesn’t continue to archive all the subsequent logs to the alternate destination, rather it always first attempts to archive to the primary destination. This is evident from the alert log file. If we observe the alert log file, we could see that even after failing over to the alternate destination, Oracle tried to archive the next set of logs (sequence# 968 and 969) to the primary destination as a first attempt, which in turn failed (due to space issue) and failed over to alternate destination (dest 2) to archive sequence# 968 and 969 . ##--- ##--- Oracle always attempts to archive to primary dest ---## ##--- Wed Jun 08 17:17:08 2016 Thread 1 advanced to log sequence 969 (LGWR switch) Current log# 3 seq# 969 mem# 0: /data/oracle/orpcdb1/redo_03.log Wed Jun 08 17:17:08 2016 ARC0: Encountered disk I/O error 19502 Wed Jun 08 17:17:08 2016 ARC0: Closing local archive destination LOG_ARCHIVE_DEST_1: '/arch1/primary_dest/1_968_903715118.dbf' (error 19502) (orpcdb1) Wed Jun 08 17:17:08 2016 Errors in file /app/oracle/diag/rdbms/orpcdb1/orpcdb1/trace/orpcdb1_arc0_8277.trc: ORA-27072: File I/O error Additional information: 4 Additional information: 14336 Additional information: 126976 ORA-19502: write error on file "/arch1/primary_dest/1_968_903715118.dbf", block number 14336 (block size=512) Wed Jun 08 17:17:08 2016 Errors in file /app/oracle/diag/rdbms/orpcdb1/orpcdb1/trace/orpcdb1_arc0_8277.trc: ORA-19502: write error on file "/arch1/primary_dest/1_968_903715118.dbf", block number 14336 (block size=512) ORA-27072: File I/O error Additional information: 4 Additional information: 14336 Additional information: 126976 ORA-19502: write error on file "/arch1/primary_dest/1_968_903715118.dbf", block number 14336 (block size=512) ARC0: I/O error 19502 archiving log 2 to '/arch1/primary_dest/1_968_903715118.dbf' Wed Jun 08 17:17:10 2016 Archived Log entry 208 added for thread 1 sequence 968 ID 0xaeb9a9ee dest 2: Wed Jun 08 17:17:13 2016 Thread 1 advanced to log sequence 970 (LGWR switch) Current log# 1 seq# 970 mem# 0: /data/oracle/orpcdb1/redo_01.log Wed Jun 08 17:17:15 2016 Archived Log entry 209 added for thread 1 sequence 969 ID 0xaeb9a9ee dest 2: Since Oracle always tries to archive to primary destination, the automatic failback to primary destination works in 12c. Technically speaking, it’s not actually a automatic failback. Oracle always tries to archive the redo logs to primary destination as a first attempt and if that fails it fails over to respective alternate archive destination. We can also confirm this behaviour by querying the v$archive_dest view. After failing over to alternate destination, once the redo streams are archived and the instance is sitting idle (not archiving redo logs), we could see the archive destination is pointing back to the primary destination by querying the v$archive_dest view. This means that the next set of redo logs would be archived to primary destination by default and if that fails, Oracle will try to archive the logs to alternate destination. ---// ---// archive dest is pointing back to primary after the completion of redo archiving to alternate dest //--- ---// SQL> select DEST_NAME,TARGET,DESTINATION,VALID_ROLE,STATUS,ALTERNATE from v$archive_dest where status!='INACTIVE'; DEST_NAME TARGET DESTINATION VALID_ROLE STATUS ALTERNATE ------------------------------ ---------------- ------------------------------ ------------ --------- ------------------------------ LOG_ARCHIVE_DEST_1 PRIMARY /arch1/primary_dest ALL_ROLES VALID LOG_ARCHIVE_DEST_2 LOG_ARCHIVE_DEST_2 PRIMARY /arch2/alternate_dest ALL_ROLES ALTERNATE LOG_ARCHIVE_DEST_1 Let’s clean up the primary archive destination (/arch1/primary_dest) to allow Oracle continue archiving to primary destination. ---// ---// Cleaning up primary archive destination //--- ---// SQL> !df -h /arch* Filesystem Size Used Avail Use% Mounted on /dev/sdf1 1004M 946M 7.1M 100% /arch1 /dev/sdg1 1004M 491M 463M 52% /arch2 SQL> !rm /arch1/primary_dest/* SQL> !df -h /arch* Filesystem Size Used Avail Use% Mounted on /dev/sdf1 1004M 18M 936M 2% /arch1 /dev/sdg1 1004M 491M 463M 52% /arch2 The moment, we clean up the primary destination and make it available for archiving, Oracle continues to archiving the redo logs to primary destination without reporting any errors or failback message in the alert log file as found below (Sequence# 970, 971 and 972 are archived to primary destination dest 1) . ---// ---// Oracle is archiving to primary destination after space cleanup //--- ---// Wed Jun 08 17:19:05 2016 Thread 1 advanced to log sequence 971 (LGWR switch) Current log# 2 seq# 971 mem# 0: /data/oracle/orpcdb1/redo_02.log Wed Jun 08 17:19:07 2016 Archived Log entry 210 added for thread 1 sequence 970 ID 0xaeb9a9ee dest 1: Wed Jun 08 17:19:08 2016 Thread 1 advanced to log sequence 972 (LGWR switch) Current log# 3 seq# 972 mem# 0: /data/oracle/orpcdb1/redo_03.log Wed Jun 08 17:19:10 2016 Archived Log entry 211 added for thread 1 sequence 971 ID 0xaeb9a9ee dest 1: Wed Jun 08 17:19:12 2016 Thread 1 cannot allocate new log, sequence 973 Checkpoint not complete Current log# 3 seq# 972 mem# 0: /data/oracle/orpcdb1/redo_03.log Wed Jun 08 17:19:14 2016 Thread 1 advanced to log sequence 973 (LGWR switch) Current log# 1 seq# 973 mem# 0: /data/oracle/orpcdb1/redo_01.log Wed Jun 08 17:19:15 2016 Archived Log entry 212 added for thread 1 sequence 972 ID 0xaeb9a9ee dest 1: If we query the v$archived_log view, we could see Oracle jumped between primary and alternate destination matching the archiving entries reported alert log file. ---// ---// validating archive destination switching //--- ---// SQL> select thread#,sequence#,name from v$archived_log where sequence# > 964; THREAD# SEQUENCE# NAME ---------- ---------- ------------------------------------------------------------ 1 965 /arch2/alternate_dest/1_965_903715118.dbf 1 966 /arch2/alternate_dest/1_966_903715118.dbf 1 967 /arch2/alternate_dest/1_967_903715118.dbf 1 968 /arch2/alternate_dest/1_968_903715118.dbf 1 969 /arch2/alternate_dest/1_969_903715118.dbf 1 970 /arch1/primary_dest/1_970_903715118.dbf 1 971 /arch1/primary_dest/1_971_903715118.dbf 1 972 /arch1/primary_dest/1_972_903715118.dbf 8 rows selected. Oracle has made a significant improvement in the automatic management of archive destination switching between the primary and alternate destination. However, Oracle will intermittently log error messages in the alert log file about the primary destination failure (as it first attempts to archive to primary destination) until the primary archive destination is made available again. This intermittent errors are useful in terms of getting continuous alert from the database about the primary destination failure and will help in not to overlook the errors.