Skip to Content
0
Former Member
Dec 01, 2009 at 11:17 AM

SAP 4.6C hangs when Brtools run after oracle upgrade 10.2.0.4

220 Views

Hi All,

We have Upgraded the Oracle 9i to 10.2.0.4 for sap 4.6C production machine. After the upgrade the sap was succesfully able to start. even i can see the version was reflected in the sap system status. in the Opatch lsinventory all the bug fixes are covered. However when ever the brtools starts like taking the backup, or dbcheck, or archivelog backup. we are getting terminated with errors and the system system hangs. here is the initSID.log as follows.

**************************************************************************************************

Wed Nov 18 12:52:02 2009

Errors in file i:\oracle\cpa\saptrace\usertrace\cpa_ora_5888.trc:

ORA-07445: exception encountered: core dump [ACCESS_VIOLATION] [_kghstack_err+101] [PC:0x3FB10E9] [ADDR:0x4E202024] [UNABLE_TO_READ] []

ORA-00607: Internal error occurred while making a change to a data block

ORA-00602: internal programming exception

ORA-07445: exception encountered: core dump [ACCESS_VIOLATION] [_kghstack_err+101] [PC:0x3FB10E9] [ADDR:0x4E202024] [UNABLE_TO_READ] []

ORA-00607: Internal error occurred while making a change to a data block

ORA-00602: internal programming exception

ORA-07445: exception encountered: core dump [ACCESS_VIOLATION] [_kghstack_err+101] [PC:0x3FB10E9] [ADDR:0x4E202024] [UNABLE_TO_READ] []

ORA-00607: Internal error occurred while making a change to a data block

ORA-00602: internal programming exception

ORA-07445: exception encountered: core dump [ACCESS_VIOLATION] [_kghstack_err+101] [PC:0x3FB10E9] [ADDR:0x4E202024] [UNABLE_TO_READ] []

ORA-00607: Internal error occurred while making a change to a data block

ORA-00602: internal programming exception

ORA-07445: excepti

Wed Nov 18 12:52:04 2009

Doing block recovery for file 28 block 68766

Block recovery from logseq 13424, block 2157 to scn 186123675

Wed Nov 18 12:52:04 2009

Recovery of Online Redo Log: Thread 1 Group 11 Seq 13424 Reading mem 0

Mem# 0: M:\ORACLE\CPA\ORIGLOGA\LOG_G11M1.DBF

Mem# 1: M:\ORACLE\CPA\MIRRLOGA\LOG_G11M2.DBF

Block recovery completed at rba 13424.2423.16, scn 0.186123678

Doing block recovery for file 27 block 1547

Doing block recovery for file 27 block 1547

Block recovery from logseq 13424, block 2157 to scn 186123679

Wed Nov 18 12:52:04 2009

Recovery of Online Redo Log: Thread 1 Group 11 Seq 13424 Reading mem 0

Mem# 0: M:\ORACLE\CPA\ORIGLOGA\LOG_G11M1.DBF

Mem# 1: M:\ORACLE\CPA\MIRRLOGA\LOG_G11M2.DBF

Block recovery stopped at EOT rba 13424.2426.16

Block recovery completed at rba 13424.2426.16, scn 0.186123679

Wed Nov 18 13:35:44 2009

Starting ORACLE instance (normal)

Wed Nov 18 13:59:16 2009

Thread 1 advanced to log sequence 13425 (LGWR switch)

Current log# 13 seq# 13425 mem# 0: M:\ORACLE\CPA\ORIGLOGA\LOG_G13M1.DBF

Current log# 13 seq# 13425 mem# 1: M:\ORACLE\CPA\MIRRLOGA\LOG_G13M2.DBF

Wed Nov 18 15:45:52 2009

Shutting down instance: further logons disabled

Wed Nov 18 15:45:54 2009

Stopping background process QMNC

Wed Nov 18 15:45:55 2009

Stopping background process CJQ0

Wed Nov 18 15:45:56 2009

Stopping background process MMNL

Wed Nov 18 15:45:57 2009

Stopping background process MMON

Wed Nov 18 15:45:58 2009

Shutting down instance (immediate)

License high water mark = 27

Wed Nov 18 15:45:58 2009

Stopping Job queue slave processes, flags = 7

Wed Nov 18 15:45:58 2009

Job queue slave processes stopped

Wed Nov 18 15:46:01 2009

ALTER DATABASE CLOSE NORMAL

Wed Nov 18 15:46:01 2009

SMON: disabling tx recovery

SMON: disabling cache recovery

Wed Nov 18 15:46:01 2009

Shutting down archive processes

Archiving is disabled

Wed Nov 18 15:46:06 2009

ARCH shutting down

ARC1: Archival stopped

Wed Nov 18 15:46:11 2009

ARCH shutting down

ARC0: Archival stopped

Wed Nov 18 15:46:12 2009

Thread 1 closed at log sequence 13425

Successful close of redo thread 1

Wed Nov 18 15:46:12 2009

Completed: ALTER DATABASE CLOSE NORMAL

Wed Nov 18 15:47:10 2009

ALTER DATABASE DISMOUNT

Wed Nov 18 15:47:10 2009

Completed: ALTER DATABASE DISMOUNT

ARCH: Archival disabled due to shutdown: 1089

Shutting down archive processes

Archiving is disabled

Archive process shutdown avoided: 0 active

ARCH: Archival disabled due to shutdown: 1089

Shutting down archive processes

Archiving is disabled

Archive process shutdown avoided: 0 active

Wed Nov 18 15:51:52 2009

Starting ORACLE instance (normal)

Wed Nov 18 15:51:52 2009

System parameter db_block_buffers enabled without VLM on.

LICENSE_MAX_SESSION = 0

LICENSE_SESSIONS_WARNING = 0

Picked latch-free SCN scheme 2

Autotune of undo retention is turned on.

IMODE=BR

ILAT =12

LICENSE_MAX_USERS = 0

SYS auditing is disabled

ksdpec: called for event 13740 prior to event group initialization

Starting up ORACLE RDBMS Version: 10.2.0.4.0.

System parameters with non-default values:

processes = 100

sessions = 115

timed_statistics = TRUE

event = 10411 trace name context forever, level 1

shared_pool_size = 377487360

large_pool_size = 8388608

java_pool_size = 67108864

streams_pool_size = 50331648

shared_pool_reserved_size= 37614518

firstspare_parameter = 1

control_files = M:\ORACLE\CPA\SAPDATA1\CNTRL\CNTRLCPA.DBF, M:\ORACLE\CPA\MIRRLOGA\CNTRL\CNTRLCPA.DBF, M:\ORACLE\CPA\ORIGLOGA\CNTRL\CNTRLCPA.DBF

control_file_record_keep_time= 30

db_block_buffers = 68874

db_block_size = 8192

compatible = 10.2.0

log_archive_dest = H:\oracle\CPA\saparch\CPAarch

log_buffer = 2850816

log_checkpoint_interval = 0

log_checkpoint_timeout = 0

db_files = 254

db_file_multiblock_read_count= 8

undo_management = AUTO

undo_tablespace = PSAPUNDO

undo_retention = 43200

recyclebin = off

remote_login_passwordfile= EXCLUSIVE

local_listener = LISTENER_CPA

cursor_space_for_time = FALSE

job_queue_processes = 1

create_bitmap_area_size = 33554432

bitmap_merge_area_size = 33554432

parallel_max_servers = 4

parallel_execution_message_size= 8192

hash_area_size = 4194304

background_dump_dest = I:\ORACLE\CPA\SAPTRACE\BACKGROUND

user_dump_dest = I:\ORACLE\CPA\SAPTRACE\USERTRACE

max_dump_file_size = 20000

session_max_open_files = 20

optimizer_features_enable= 9.2.0

sort_area_size = 2097152

sort_area_retained_size = 0

db_name = CPA

open_cursors = 800

optimizer_mode = CHOOSE

star_transformation_enabled= true

optimizer_index_cost_adj = 10

pga_aggregate_target = 25165824

PMON started with pid=2, OS id=2880

PSP0 started with pid=3, OS id=5268

MMAN started with pid=4, OS id=5768

DBW0 started with pid=5, OS id=5076

LGWR started with pid=6, OS id=6600

CKPT started with pid=7, OS id=812

SMON started with pid=8, OS id=3052

RECO started with pid=9, OS id=6132

CJQ0 started with pid=10, OS id=3648

MMON started with pid=11, OS id=4872

MMNL started with pid=12, OS id=3656

Wed Nov 18 15:51:53 2009

ALTER DATABASE MOUNT

Wed Nov 18 15:51:57 2009

Setting recovery target incarnation to 1

Wed Nov 18 15:51:57 2009

Successful mount of redo thread 1, with mount id 4087130713

Wed Nov 18 15:51:57 2009

Database mounted in Exclusive Mode

Completed: ALTER DATABASE MOUNT

Wed Nov 18 15:51:58 2009

ALTER DATABASE OPEN

Wed Nov 18 15:51:58 2009

LGWR: STARTING ARCH PROCESSES

ARC0 started with pid=14, OS id=5620

Wed Nov 18 15:51:58 2009

ARC0: Archival started

Wed Nov 18 15:51:58 2009

ARC1: Archival started

LGWR: STARTING ARCH PROCESSES COMPLETE

Thread 1 opened at log sequence 13425

Current log# 13 seq# 13425 mem# 0: M:\ORACLE\CPA\ORIGLOGA\LOG_G13M1.DBF

Current log# 13 seq# 13425 mem# 1: M:\ORACLE\CPA\MIRRLOGA\LOG_G13M2.DBF

Successful open of redo thread 1

Wed Nov 18 15:51:58 2009

MTTR advisory is disabled because FAST_START_MTTR_TARGET is not set

ARC1 started with pid=15, OS id=3344

Wed Nov 18 15:51:59 2009

ARC0: Becoming the 'no FAL' ARCH

ARC0: Becoming the 'no SRL' ARCH

Wed Nov 18 15:51:59 2009

ARC1: Becoming the heartbeat ARCH

Wed Nov 18 15:51:59 2009

SMON: enabling cache recovery

Wed Nov 18 15:52:00 2009

Successfully onlined Undo Tablespace 31.

Wed Nov 18 15:52:00 2009

SMON: enabling tx recovery

Wed Nov 18 15:52:00 2009

Database Characterset is US7ASCII

Opening with internal Resource Manager plan

where NUMA PG = 1, CPUs = 4

replication_dependency_tracking turned off (no async multimaster replication found)

Starting background process QMNC

QMNC started with pid=16, OS id=5172

Wed Nov 18 15:52:04 2009

Completed: ALTER DATABASE OPEN

Wed Nov 18 15:53:00 2009

Thread 1 advanced to log sequence 13426 (LGWR switch)

Current log# 14 seq# 13426 mem# 0: M:\ORACLE\CPA\ORIGLOGB\LOG_G14M1.DBF

Current log# 14 seq# 13426 mem# 1: M:\ORACLE\CPA\MIRRLOGB\LOG_G14M2.DBF

Wed Nov 18 16:02:30 2009

Errors in file i:\oracle\cpa\saptrace\usertrace\cpa_ora_6540.trc:

ORA-00600: internal error code, arguments: [qertbFetchByRowID], [], [], [], [], [], [], []

Wed Nov 18 16:07:23 2009

Hex dump of (file 15, block 157170) in trace file i:\oracle\cpa\saptrace\background\cpa_dbw0_5076.trc

Corrupt block relative dba: 0x03c265f2 (file 15, block 157170)

Bad header found during preparing block for write

Data in bad block:

type: 48 format: 0 rdba: 0x30303030

last change scn: 0x5043.50415330 seq: 0x41 flg: 0x41

spare1: 0x30 spare2: 0x30 spare3: 0x5a31

consistency value in tail: 0x8b15a015

check value in block header: 0x4730

block checksum disabled

Wed Nov 18 16:07:24 2009

Errors in file i:\oracle\cpa\saptrace\background\cpa_dbw0_5076.trc:

ORA-00600: internal error code, arguments: [kcbzpbuf_1], [4], [1], [], [], [], [], []

Wed Nov 18 16:07:26 2009

Errors in file i:\oracle\cpa\saptrace\background\cpa_dbw0_5076.trc:

ORA-00600: internal error code, arguments: [kcbzpbuf_1], [4], [1], [], [], [], [], []

DBW0: terminating instance due to error 471

Wed Nov 18 16:07:26 2009

Errors in file i:\oracle\cpa\saptrace\background\cpa_pmon_2880.trc:

ORA-00471: DBWR process terminated with error

Wed Nov 18 16:07:26 2009

Errors in file i:\oracle\cpa\saptrace\background\cpa_ckpt_812.trc:

ORA-00471: DBWR process terminated with error

Wed Nov 18 16:07:28 2009

Errors in file i:\oracle\cpa\saptrace\background\cpa_q001_1980.trc:

ORA-00471: DBWR process terminated with error

Wed Nov 18 16:07:28 2009

Errors in file i:\oracle\cpa\saptrace\background\cpa_j000_5552.trc:

ORA-00471: DBWR process terminated with error

Wed Nov 18 16:07:29 2009

Errors in file i:\oracle\cpa\saptrace\background\cpa_lgwr_6600.trc:

ORA-00471: DBWR process terminated with error

Wed Nov 18 16:12:29 2009

Errors in file i:\oracle\cpa\saptrace\background\cpa_mman_5768.trc:

ORA-00471: DBWR process terminated with error

Wed Nov 18 16:12:29 2009

Errors in file i:\oracle\cpa\saptrace\background\cpa_psp0_5268.trc:

ORA-00471: DBWR process terminated with error

Wed Nov 18 16:12:34 2009

Errors in file i:\oracle\cpa\saptrace\background\cpa_dbw0_5076.trc:

ORA-27300: OS system dependent operation:spdde:9262:4210 failed with status: 0

ORA-27301: OS failure message: The operation completed successfully.

ORA-27302: failure occurred at: skgpkill4

ORA-27144: attempt to kill process failed

ORA-00600: internal error code, arguments: [kcbzpbuf_1], [4], [1], [], [], [], [], []

Wed Nov 18 16:12:39 2009

Errors in file i:\oracle\cpa\saptrace\background\cpa_smon_3052.trc:

ORA-00471: DBWR process terminated with error

***********************************************************************************************************************

can any one please help me on this . as this is production we are facing this problem from past 3 weeks. no one was able to analyze whats giong wrong. so because of this we even stopped the backup's of archivelog and online backup. Please some one help me asap. Your help is really appreciated.

Hemanth