DG环境恢复同步遇到报错ORA-00353ORA-00334以及ORA-00600[2619], [47745]
问题说明
客户环境主库4节点RAC11.2.0.4,单实例DG环境,DG由于空间不足,导致同步中断,由于DG备库未应用的归档主库都再,本次恢复的方式,是开启dg mrp进程,自动同步追上主库。
以下遇到的问题,都在此过程中,遇到的报错。
一 dg环境遇到
ORA-00353 ORA-00334 如果主库还存在归档日志,则可以mv ,rm删除备库有问题的归档,让dg产生gap,主库会再次像备库传输归档日志
另一种方法,就是如果主库无法自动像备库传输日志,可以从主库scp归档至备库dg,注册归档后,解决问题。
其次,本次通过主库,dg alert排除存在异常现象,也就是说再本次问题前,alert日志中并未出现日志块损坏现象!
ORA-354 ORA-353 Corrupt Redo Log Block Header against online redolog using EMC/Cisco SANTap Service (Doc ID 779711.1)
ORA-354 ORA-353 and ORA-312 on Redo Log Group members (Doc ID 332672.1)
hu Nov 21 22:13:04 2019 Media Recovery Log /oradata/newora901dg/archivelog/NEWORA901DG/archivelog/2019_11_21/o1_mf_4_31416_gxf6v6p0_.arc Incomplete read from log member '/oradata/newora901dg/archivelog/NEWORA901DG/archivelog/2019_11_21/o1_mf_1_46650_gxc023sj_.arc'.
Trying next member. Errors in file /u01/app/oracle/diag/rdbms/newora901dg/ora901/trace/ora901_pr00_9153.trc (incident=192300): ORA-00353: log corruption near block 40 change 33381243833 time 11/19/2019 13:34:10 ORA-00334: archived log: '/oradata/newora901dg/archivelog/NEWORA901DG/archivelog/2019_11_21/o1_mf_1_46650_gxc023sj_.arc' Incident details in: /u01/app/oracle/diag/rdbms/newora901dg/ora901/incident/incdir_192300/ora901_pr00_9153_i192300.trc Errors with log /oradata/newora901dg/archivelog/NEWORA901DG/archivelog/2019_11_21/o1_mf_4_31416_gxf6v6p0_.arc MRP0: Background Media Recovery terminated with error 354 Errors in file /u01/app/oracle/diag/rdbms/newora901dg/ora901/trace/ora901_pr00_9153.trc: ORA-00354: corrupt redo log block header ORA-00353: log corruption near block 40 change 33381243833 time 11/19/2019 13:34:10 ORA-00334: archived log: '/oradata/newora901dg/archivelog/NEWORA901DG/archivelog/2019_11_21/o1_mf_1_46650_gxc023sj_.arc' Thu Nov 21 22:13:05 2019 Dumping diagnostic data in directory=[cdmp_20191121221305], requested by (instance=1, osid=9153 (PR00)), summary=[incident=192300]. Recovery interrupted! Recovered data files to a consistent state at change 33381242204 Thu Nov 21 22:13:06 2019 MRP0: Background Media Recovery process shutdown (ora901) Thu Nov 21 22:13:08 2019 Sweep [inc][192300]: completed Sweep [inc][192299]: completed Sweep [inc2][192300]: completed Sweep [inc2][192299]: completed MRP0: Background Media Recovery process shutdown (ora901) 操作:
dg环境
mv 有问题的归档 xxx.bak 后,dg备库启动Mrp进程
SQL> recover managed standby database disconnect from session;
Thu Nov 21 22:21:38 2019 ALTER DATABASE RECOVER managed standby database disconnect from session Attempt to start background Managed Standby Recovery process (ora901) Thu Nov 21 22:21:38 2019 MRP0 started with pid=37, OS id=9721 MRP0: Background Managed Standby Recovery process started (ora901) started logmerger process Thu Nov 21 22:21:43 2019 Managed Standby Recovery not using Real Time Apply Parallel Media Recovery started with 64 slaves Waiting for all non-current ORLs to be archived... All non-current ORLs have been archived. Media Recovery Log /oradata/newora901dg/archivelog/NEWORA901DG/archivelog/2019_11_21/o1_mf_2_47491_gxf6t3v7_.arc Media Recovery Log /oradata/newora901dg/archivelog/NEWORA901DG/archivelog/2019_11_21/o1_mf_1_46650_gxc023sj_.arc Error opening /oradata/newora901dg/archivelog/NEWORA901DG/archivelog/2019_11_21/o1_mf_1_46650_gxc023sj_.arc Attempting refetch Media Recovery Waiting for thread 1 sequence 46650 Fetching gap sequence in thread 1, gap sequence 46650-46650 Thu Nov 21 22:21:46 2019 RFS[23]: Allowing overwrite of partial archivelog for thread 1 sequence 46650 RFS[23]: Opened log for thread 1 sequence 46650 dbid -1930502573 branch 988229931 Archived Log entry 160947 added for thread 1 sequence 46650 rlc 988229931 ID 0x948808ac dest 2: Completed: ALTER DATABASE RECOVER managed standby database disconnect from session Thu Nov 21 22:21:56 2019 Media Recovery Log /oradata/newora901dg/archivelog/NEWORA901DG/archivelog/2019_11_21/o1_mf_1_46650_gxf7ctdd_.arc Media Recovery Log /oradata/newora901dg/archivelog/NEWORA901DG/archivelog/2019_11_21/o1_mf_3_31565_gxbzzfxk_.arc Media Recovery Log /oradata/newora901dg/archivelog/NEWORA901DG/archivelog/2019_11_21/o1_mf_4_31416_gxf6v6p0_.arc Media Recovery Log /oradata/newora901dg/archivelog/NEWORA901DG/archivelog/2019_11_21/o1_mf_4_31417_gxbzvr69_.arc Media Recovery Log /oradata/newora901dg/archivelog/NEWORA901DG/archivelog/2019_11_21/o1_mf_1_46651_gxc023sz_.arc
#备选方案,如果是10G环境,备库产生gap,但是主库很可能并不会再次将归档传输到备库,因此需要手工scp归档日志传输至备库。
SQL> select name,thread# from v$archived_log where sequence#=46650;
NAME
------------------------------------------------------------------------
+FRA/newora901/archivelog/2019_11_19/thread_1_seq_46650.4973.1024753747
ASMCMD> cp thread_1_seq_46650.4973.1024753747 /home/grid/thread_1_seq_46650.arc
copying +FRA/newora901/archivelog/2019_11_19/thread_1_seq_46650.4973.1024753747 -> /home/grid/thread_1_seq_46650.arc
scp 传输归档到备库
SQL> ALTER DATABASE REGISTER LOGFILE ‘/zz/xxx.arc’;
二 DG同步遇到如下ORA 600报错,MOS并未找到相关文档
本次通过将报错的上一个归档日志,进行mv 移除后,再次开启mrp进程,dg产生gap,主库重传报错归档解决问题。
Waiting for all non-current ORLs to be archived... All non-current ORLs have been archived. Media Recovery Log /oradata/newora901dg/archivelog/NEWORA901DG/archivelog/2019_11_21/o1_mf_2_47745_gxc193wj_.arc Errors in file /u01/app/oracle/diag/rdbms/newora901dg/ora901/trace/ora901_pr00_17446.trc (incident=208045): ORA-00600: internal error code, arguments: [2619], [47745], [], [], [], [], [], [], [], [], [], [] Incident details in: /u01/app/oracle/diag/rdbms/newora901dg/ora901/incident/incdir_208045/ora901_pr00_17446_i208045.trc Completed: ALTER DATABASE RECOVER managed standby database disconnect from session Thu Nov 21 23:14:30 2019 Dumping diagnostic data in directory=[cdmp_20191121231430], requested by (instance=1, osid=17446 (PR00)),
summary=[incident=208045]. Use ADRCI or Support Workbench to package the incident. See Note 411.1 at My Oracle Support for error and packaging details. Errors with log /oradata/newora901dg/archivelog/NEWORA901DG/archivelog/2019_11_21/o1_mf_2_47745_gxc193wj_.arc MRP0: Background Media Recovery terminated with error 600 Errors in file /u01/app/oracle/diag/rdbms/newora901dg/ora901/trace/ora901_pr00_17446.trc: ORA-00600: internal error code, arguments: [2619], [47745], [], [], [], [], [], [], [], [], [], [] Recovery interrupted! Errors in file /u01/app/oracle/diag/rdbms/newora901dg/ora901/trace/ora901_pr00_17446.trc: ORA-00600: internal error code, arguments: [2619], [47745], [], [], [], [], [], [], [], [], [], [] MRP0: Background Media Recovery process shutdown (ora901) SQL> host mv /oradata/newora901dg/archivelog/NEWORA901DG/archivelog/2019_11_21/o1_mf_2_47745_gxc193wj_.arc
/oradata/newora901dg/archivelog/NEWORA901DG/archivelog/2019_11_21/o1_mf_2_47745_gxc193wj_.arc_20191121.bak