Hi All,
We have Upgraded the Oracle 9i to 10.2.0.4 for sap 4.6C production machine. After the upgrade the sap was succesfully able to start. even i can see the version was reflected in the sap system status. in the Opatch lsinventory all the bug fixes are covered. However when ever the brtools starts like taking the backup, or dbcheck, or archivelog backup. we are getting terminated with errors and the system system hangs. here is the initSID.log as follows.
**************************************************************************************************
Wed Nov 18 12:52:02 2009
Errors in file i:\oracle\cpa\saptrace\usertrace\cpa_ora_5888.trc:
ORA-07445: exception encountered: core dump [ACCESS_VIOLATION] [_kghstack_err+101] [PC:0x3FB10E9] [ADDR:0x4E202024] [UNABLE_TO_READ] []
ORA-00607: Internal error occurred while making a change to a data block
ORA-00602: internal programming exception
ORA-07445: exception encountered: core dump [ACCESS_VIOLATION] [_kghstack_err+101] [PC:0x3FB10E9] [ADDR:0x4E202024] [UNABLE_TO_READ] []
ORA-00607: Internal error occurred while making a change to a data block
ORA-00602: internal programming exception
ORA-07445: exception encountered: core dump [ACCESS_VIOLATION] [_kghstack_err+101] [PC:0x3FB10E9] [ADDR:0x4E202024] [UNABLE_TO_READ] []
ORA-00607: Internal error occurred while making a change to a data block
ORA-00602: internal programming exception
ORA-07445: exception encountered: core dump [ACCESS_VIOLATION] [_kghstack_err+101] [PC:0x3FB10E9] [ADDR:0x4E202024] [UNABLE_TO_READ] []
ORA-00607: Internal error occurred while making a change to a data block
ORA-00602: internal programming exception
ORA-07445: excepti
Wed Nov 18 12:52:04 2009
Doing block recovery for file 28 block 68766
Block recovery from logseq 13424, block 2157 to scn 186123675
Wed Nov 18 12:52:04 2009
Recovery of Online Redo Log: Thread 1 Group 11 Seq 13424 Reading mem 0
Mem# 0: M:\ORACLE\CPA\ORIGLOGA\LOG_G11M1.DBF
Mem# 1: M:\ORACLE\CPA\MIRRLOGA\LOG_G11M2.DBF
Block recovery completed at rba 13424.2423.16, scn 0.186123678
Doing block recovery for file 27 block 1547
Doing block recovery for file 27 block 1547
Block recovery from logseq 13424, block 2157 to scn 186123679
Wed Nov 18 12:52:04 2009
Recovery of Online Redo Log: Thread 1 Group 11 Seq 13424 Reading mem 0
Mem# 0: M:\ORACLE\CPA\ORIGLOGA\LOG_G11M1.DBF
Mem# 1: M:\ORACLE\CPA\MIRRLOGA\LOG_G11M2.DBF
Block recovery stopped at EOT rba 13424.2426.16
Block recovery completed at rba 13424.2426.16, scn 0.186123679
Wed Nov 18 13:35:44 2009
Starting ORACLE instance (normal)
Wed Nov 18 13:59:16 2009
Thread 1 advanced to log sequence 13425 (LGWR switch)
Current log# 13 seq# 13425 mem# 0: M:\ORACLE\CPA\ORIGLOGA\LOG_G13M1.DBF
Current log# 13 seq# 13425 mem# 1: M:\ORACLE\CPA\MIRRLOGA\LOG_G13M2.DBF
Wed Nov 18 15:45:52 2009
Shutting down instance: further logons disabled
Wed Nov 18 15:45:54 2009
Stopping background process QMNC
Wed Nov 18 15:45:55 2009
Stopping background process CJQ0
Wed Nov 18 15:45:56 2009
Stopping background process MMNL
Wed Nov 18 15:45:57 2009
Stopping background process MMON
Wed Nov 18 15:45:58 2009
Shutting down instance (immediate)
License high water mark = 27
Wed Nov 18 15:45:58 2009
Stopping Job queue slave processes, flags = 7
Wed Nov 18 15:45:58 2009
Job queue slave processes stopped
Wed Nov 18 15:46:01 2009
ALTER DATABASE CLOSE NORMAL
Wed Nov 18 15:46:01 2009
SMON: disabling tx recovery
SMON: disabling cache recovery
Wed Nov 18 15:46:01 2009
Shutting down archive processes
Archiving is disabled
Wed Nov 18 15:46:06 2009
ARCH shutting down
ARC1: Archival stopped
Wed Nov 18 15:46:11 2009
ARCH shutting down
ARC0: Archival stopped
Wed Nov 18 15:46:12 2009
Thread 1 closed at log sequence 13425
Successful close of redo thread 1
Wed Nov 18 15:46:12 2009
Completed: ALTER DATABASE CLOSE NORMAL
Wed Nov 18 15:47:10 2009
ALTER DATABASE DISMOUNT
Wed Nov 18 15:47:10 2009
Completed: ALTER DATABASE DISMOUNT
ARCH: Archival disabled due to shutdown: 1089
Shutting down archive processes
Archiving is disabled
Archive process shutdown avoided: 0 active
ARCH: Archival disabled due to shutdown: 1089
Shutting down archive processes
Archiving is disabled
Archive process shutdown avoided: 0 active
Wed Nov 18 15:51:52 2009
Starting ORACLE instance (normal)
Wed Nov 18 15:51:52 2009
System parameter db_block_buffers enabled without VLM on.
LICENSE_MAX_SESSION = 0
LICENSE_SESSIONS_WARNING = 0
Picked latch-free SCN scheme 2
Autotune of undo retention is turned on.
IMODE=BR
ILAT =12
LICENSE_MAX_USERS = 0
SYS auditing is disabled
ksdpec: called for event 13740 prior to event group initialization
Starting up ORACLE RDBMS Version: 10.2.0.4.0.
System parameters with non-default values:
processes = 100
sessions = 115
timed_statistics = TRUE
event = 10411 trace name context forever, level 1
shared_pool_size = 377487360
large_pool_size = 8388608
java_pool_size = 67108864
streams_pool_size = 50331648
shared_pool_reserved_size= 37614518
firstspare_parameter = 1
control_files = M:\ORACLE\CPA\SAPDATA1\CNTRL\CNTRLCPA.DBF, M:\ORACLE\CPA\MIRRLOGA\CNTRL\CNTRLCPA.DBF, M:\ORACLE\CPA\ORIGLOGA\CNTRL\CNTRLCPA.DBF
control_file_record_keep_time= 30
db_block_buffers = 68874
db_block_size = 8192
compatible = 10.2.0
log_archive_dest = H:\oracle\CPA\saparch\CPAarch
log_buffer = 2850816
log_checkpoint_interval = 0
log_checkpoint_timeout = 0
db_files = 254
db_file_multiblock_read_count= 8
undo_management = AUTO
undo_tablespace = PSAPUNDO
undo_retention = 43200
recyclebin = off
remote_login_passwordfile= EXCLUSIVE
local_listener = LISTENER_CPA
cursor_space_for_time = FALSE
job_queue_processes = 1
create_bitmap_area_size = 33554432
bitmap_merge_area_size = 33554432
parallel_max_servers = 4
parallel_execution_message_size= 8192
hash_area_size = 4194304
background_dump_dest = I:\ORACLE\CPA\SAPTRACE\BACKGROUND
user_dump_dest = I:\ORACLE\CPA\SAPTRACE\USERTRACE
max_dump_file_size = 20000
session_max_open_files = 20
optimizer_features_enable= 9.2.0
sort_area_size = 2097152
sort_area_retained_size = 0
db_name = CPA
open_cursors = 800
optimizer_mode = CHOOSE
star_transformation_enabled= true
optimizer_index_cost_adj = 10
pga_aggregate_target = 25165824
PMON started with pid=2, OS id=2880
PSP0 started with pid=3, OS id=5268
MMAN started with pid=4, OS id=5768
DBW0 started with pid=5, OS id=5076
LGWR started with pid=6, OS id=6600
CKPT started with pid=7, OS id=812
SMON started with pid=8, OS id=3052
RECO started with pid=9, OS id=6132
CJQ0 started with pid=10, OS id=3648
MMON started with pid=11, OS id=4872
MMNL started with pid=12, OS id=3656
Wed Nov 18 15:51:53 2009
ALTER DATABASE MOUNT
Wed Nov 18 15:51:57 2009
Setting recovery target incarnation to 1
Wed Nov 18 15:51:57 2009
Successful mount of redo thread 1, with mount id 4087130713
Wed Nov 18 15:51:57 2009
Database mounted in Exclusive Mode
Completed: ALTER DATABASE MOUNT
Wed Nov 18 15:51:58 2009
ALTER DATABASE OPEN
Wed Nov 18 15:51:58 2009
LGWR: STARTING ARCH PROCESSES
ARC0 started with pid=14, OS id=5620
Wed Nov 18 15:51:58 2009
ARC0: Archival started
Wed Nov 18 15:51:58 2009
ARC1: Archival started
LGWR: STARTING ARCH PROCESSES COMPLETE
Thread 1 opened at log sequence 13425
Current log# 13 seq# 13425 mem# 0: M:\ORACLE\CPA\ORIGLOGA\LOG_G13M1.DBF
Current log# 13 seq# 13425 mem# 1: M:\ORACLE\CPA\MIRRLOGA\LOG_G13M2.DBF
Successful open of redo thread 1
Wed Nov 18 15:51:58 2009
MTTR advisory is disabled because FAST_START_MTTR_TARGET is not set
ARC1 started with pid=15, OS id=3344
Wed Nov 18 15:51:59 2009
ARC0: Becoming the 'no FAL' ARCH
ARC0: Becoming the 'no SRL' ARCH
Wed Nov 18 15:51:59 2009
ARC1: Becoming the heartbeat ARCH
Wed Nov 18 15:51:59 2009
SMON: enabling cache recovery
Wed Nov 18 15:52:00 2009
Successfully onlined Undo Tablespace 31.
Wed Nov 18 15:52:00 2009
SMON: enabling tx recovery
Wed Nov 18 15:52:00 2009
Database Characterset is US7ASCII
Opening with internal Resource Manager plan
where NUMA PG = 1, CPUs = 4
replication_dependency_tracking turned off (no async multimaster replication found)
Starting background process QMNC
QMNC started with pid=16, OS id=5172
Wed Nov 18 15:52:04 2009
Completed: ALTER DATABASE OPEN
Wed Nov 18 15:53:00 2009
Thread 1 advanced to log sequence 13426 (LGWR switch)
Current log# 14 seq# 13426 mem# 0: M:\ORACLE\CPA\ORIGLOGB\LOG_G14M1.DBF
Current log# 14 seq# 13426 mem# 1: M:\ORACLE\CPA\MIRRLOGB\LOG_G14M2.DBF
Wed Nov 18 16:02:30 2009
Errors in file i:\oracle\cpa\saptrace\usertrace\cpa_ora_6540.trc:
ORA-00600: internal error code, arguments: [qertbFetchByRowID], [], [], [], [], [], [], []
Wed Nov 18 16:07:23 2009
Hex dump of (file 15, block 157170) in trace file i:\oracle\cpa\saptrace\background\cpa_dbw0_5076.trc
Corrupt block relative dba: 0x03c265f2 (file 15, block 157170)
Bad header found during preparing block for write
Data in bad block:
type: 48 format: 0 rdba: 0x30303030
last change scn: 0x5043.50415330 seq: 0x41 flg: 0x41
spare1: 0x30 spare2: 0x30 spare3: 0x5a31
consistency value in tail: 0x8b15a015
check value in block header: 0x4730
block checksum disabled
Wed Nov 18 16:07:24 2009
Errors in file i:\oracle\cpa\saptrace\background\cpa_dbw0_5076.trc:
ORA-00600: internal error code, arguments: [kcbzpbuf_1], [4], [1], [], [], [], [], []
Wed Nov 18 16:07:26 2009
Errors in file i:\oracle\cpa\saptrace\background\cpa_dbw0_5076.trc:
ORA-00600: internal error code, arguments: [kcbzpbuf_1], [4], [1], [], [], [], [], []
DBW0: terminating instance due to error 471
Wed Nov 18 16:07:26 2009
Errors in file i:\oracle\cpa\saptrace\background\cpa_pmon_2880.trc:
ORA-00471: DBWR process terminated with error
Wed Nov 18 16:07:26 2009
Errors in file i:\oracle\cpa\saptrace\background\cpa_ckpt_812.trc:
ORA-00471: DBWR process terminated with error
Wed Nov 18 16:07:28 2009
Errors in file i:\oracle\cpa\saptrace\background\cpa_q001_1980.trc:
ORA-00471: DBWR process terminated with error
Wed Nov 18 16:07:28 2009
Errors in file i:\oracle\cpa\saptrace\background\cpa_j000_5552.trc:
ORA-00471: DBWR process terminated with error
Wed Nov 18 16:07:29 2009
Errors in file i:\oracle\cpa\saptrace\background\cpa_lgwr_6600.trc:
ORA-00471: DBWR process terminated with error
Wed Nov 18 16:12:29 2009
Errors in file i:\oracle\cpa\saptrace\background\cpa_mman_5768.trc:
ORA-00471: DBWR process terminated with error
Wed Nov 18 16:12:29 2009
Errors in file i:\oracle\cpa\saptrace\background\cpa_psp0_5268.trc:
ORA-00471: DBWR process terminated with error
Wed Nov 18 16:12:34 2009
Errors in file i:\oracle\cpa\saptrace\background\cpa_dbw0_5076.trc:
ORA-27300: OS system dependent operation:spdde:9262:4210 failed with status: 0
ORA-27301: OS failure message: The operation completed successfully.
ORA-27302: failure occurred at: skgpkill4
ORA-27144: attempt to kill process failed
ORA-00600: internal error code, arguments: [kcbzpbuf_1], [4], [1], [], [], [], [], []
Wed Nov 18 16:12:39 2009
Errors in file i:\oracle\cpa\saptrace\background\cpa_smon_3052.trc:
ORA-00471: DBWR process terminated with error
***********************************************************************************************************************
can any one please help me on this . as this is production we are facing this problem from past 3 weeks. no one was able to analyze whats giong wrong. so because of this we even stopped the backup's of archivelog and online backup. Please some one help me asap. Your help is really appreciated.
Hemanth