Hi,
tonight our database crashed again due to a bad index problem and cannot be brought into admin state. I can't do a check and/or recovery either, because it's not in state admin.
Any suggestions???
Thanks,
Arne
This is the KNLDIAG.ERR extract:
2007-12-11 05:32:14 151 ERR 51080 SYSERROR -9049 BD Primkey from inv in primtree no
2007-12-11 05:32:14 151 ERR 53419 B*TREE BD600: predsep.k > sep.k: 0
2007-12-11 05:32:14 151 ERR 53000 B*TREE 07010000000000018EBE000000000000
2007-12-11 05:32:14 151 ERR 53000 B*TREE Index Root 834277
2007-12-11 05:32:14 151 ERR 53000 B*TREE bd600BuildSeparator: lef: 383217
2007-12-11 05:32:14 151 ERR 53000 B*TREE 07010000000000018EBE000000000000
2007-12-11 05:32:14 151 ERR 53000 B*TREE Index Root 834277
2007-12-11 05:32:14 151 ERR 53000 B*TREE bd600BuildSeparator: rig: 153800
2007-12-11 05:32:14 151 ERR 53250 INDEX Bad Index 834277 (Root)
2007-12-11 05:32:14 151 ERR 53250 INDEX Reason "System error: BD Invalid leaves"
2007-12-11 05:32:14 151 ERR 51080 SYSERROR -9049 BD Primkey from inv in primtree no
2007-12-11 05:33:29 158 ERR 53093 DATACACH Occup PNO 1245190 PTR C88162B72A000000
2007-12-11 05:33:29 158 ERR 53000 B*TREE 07040000000000018AFA000000000000
2007-12-11 05:33:29 158 ERR 53000 B*TREE Index Root 48642
2007-12-11 05:33:29 158 ERR 3 Admin Kernel_Administration.cpp:606
2007-12-11 05:33:29 158 ERR 3 Admin 2007-12-11 05:33:29 Admin Error 3
2007-12-11 05:33:29 158 ERR 3 Admin Database state: OFFLINE
2007-12-11 05:33:29 158 ERR 3 Admin + vgg999.cpp:153
2007-12-11 05:33:29 158 ERR 3 Admin + 2007-12-11 05:33:29 Admin Error 2
2007-12-11 05:33:29 158 ERR 3 Admin - A fatal error caused EMERGENCY SHUTDOWN. core is written
2007-12-11 05:33:29 158 ERR 11196 DBCRASH vabort:Emergency Shutdown, Kernel_Administration.cpp: 606
2007-12-11 05:33:29 158 ERR 11599 BTRACE -
> Emergency Stack Back Trace <----
2007-12-11 05:33:29 158 ERR 11599 BTRACE (0):0xddcc95
2007-12-11 05:33:29 158 ERR 11599 BTRACE (1):0xddccf9
2007-12-11 05:33:29 158 ERR 11599 BTRACE (2):0xddce42
2007-12-11 05:33:29 158 ERR 11599 BTRACE (3):0xddce77
2007-12-11 05:33:29 158 ERR 11599 BTRACE (4):0xdfeed7
2007-12-11 05:33:29 158 ERR 11599 BTRACE (5):0xd1bbdd
2007-12-11 05:33:29 158 ERR 11599 BTRACE (6):0xce28ab
2007-12-11 05:33:29 158 ERR 11599 BTRACE (7):0xa25e4f
2007-12-11 05:33:29 158 ERR 11599 BTRACE (8):0x91343b
2007-12-11 05:33:29 158 ERR 11599 BTRACE (9):0x914814
2007-12-11 05:33:29 158 ERR 11599 BTRACE (10):0x94d8ba
2007-12-11 05:33:29 158 ERR 11599 BTRACE (11):0x9d81a4
2007-12-11 05:33:29 158 ERR 11599 BTRACE (12):0x9b88e0
2007-12-11 05:33:29 158 ERR 11599 BTRACE (13):0x93c360
2007-12-11 05:33:29 158 ERR 11599 BTRACE (14):0x84e68f
2007-12-11 05:33:29 158 ERR 11599 BTRACE (15):0x8439a4
2007-12-11 05:33:29 158 ERR 11599 BTRACE (16):0x844455
2007-12-11 05:33:29 158 ERR 11599 BTRACE (17):0x84c43d
2007-12-11 05:33:29 158 ERR 11599 BTRACE (18):0x84af19
2007-12-11 05:33:29 158 ERR 11599 BTRACE (19):0x8350fb
2007-12-11 05:33:29 158 ERR 11599 BTRACE (20):0x80ba7f
2007-12-11 05:33:29 158 ERR 11599 BTRACE (21):0x80e28f
2007-12-11 05:33:29 158 ERR 11599 BTRACE (22):0x74195f
2007-12-11 05:33:29 158 ERR 11599 BTRACE (23):0x742ac0
2007-12-11 05:33:29 158 ERR 11599 BTRACE (24):0x737886
2007-12-11 05:33:29 158 ERR 11599 BTRACE (25):0x738750
2007-12-11 05:33:29 158 ERR 11599 BTRACE (26):0x7f150c
2007-12-11 05:33:29 158 ERR 11599 BTRACE (27):0x7f200d
2007-12-11 05:33:29 158 ERR 11599 BTRACE (28):0x7f3047
2007-12-11 05:33:29 158 ERR 11599 BTRACE (29):0x7f5d88
2007-12-11 05:33:29 158 ERR 11599 BTRACE (30):0xcf39a4
2007-12-11 05:33:29 158 ERR 11599 BTRACE (31):0xcf3b83
2007-12-11 05:33:29 158 ERR 11599 BTRACE (32):0xd7a03f
2007-12-11 05:33:29 158 ERR 11599 BTRACE (33):0xe23dec
2007-12-11 05:33:29 158 ERR 11599 BTRACE (34):0xe2405a
2007-12-11 05:33:29 158 ERR 11599 BTRACE (35):0xdd0b21
2007-12-11 05:33:29 158 ERR 11599 BTRACE (36):0x2a95673aff
2007-12-11 05:33:29 158 ERR 11599 BTRACE (37):0x2a95e884b3
2007-12-11 05:33:29 158 ERR 11599 BTRACE -
> Module List <----
2007-12-11 05:33:29 158 ERR 11599 BTRACE |.text Start |.text End | Module File Name
2007-12-11 05:33:29 158 ERR 11599 BTRACE |0x0000000000400000|0x0000000000fc4000| /opt/sdb/7600/pgm/kernel
2007-12-11 05:33:29 158 ERR 11599 BTRACE |0x0000002a95556000|0x0000002a9556b000| /lib64/ld-2.3.3.so
2007-12-11 05:33:29 158 ERR 11599 BTRACE |0x0000002a9566d000|0x0000002a9567b000| /lib64/tls/libpthread.so.0
2007-12-11 05:33:29 158 ERR 11599 BTRACE |0x0000002a95782000|0x0000002a95784000| /lib64/libdl.so.2
2007-12-11 05:33:29 158 ERR 11599 BTRACE |0x0000002a95885000|0x0000002a9588c000| /lib64/tls/librt.so.1
2007-12-11 05:33:29 158 ERR 11599 BTRACE |0x0000002a9598d000|0x0000002a95a3b000| /usr/lib64/libstdc++.so.5.0.6
2007-12-11 05:33:29 158 ERR 11599 BTRACE |0x0000002a95b6b000|0x0000002a95bbf000| /lib64/tls/libm.so.6
2007-12-11 05:33:29 158 ERR 11599 BTRACE |0x0000002a95cc3000|0x0000002a95ccd000| /lib64/libgcc_s.so.1
2007-12-11 05:33:29 158 ERR 11599 BTRACE |0x0000002a95dce000|0x0000002a95ed9000| /lib64/tls/libc.so.6
2007-12-11 05:33:29 158 ERR 11599 BTRACE |0x0000002a9610e000|0x0000002a96117000| /lib64/libnss_files.so.2
2007-12-11 05:33:29 158 ERR 11599 BTRACE |0x0000002ab96da000|0x0000002ab9904000| /opt/sdb/7600/lib/liboms.so
2007-12-11 05:33:29 158 ERR 11599 BTRACE
2007-12-11 05:33:29 158 ERR 11599 BTRACE -
> Symbolic Stack Back Trace <----
2007-12-11 05:33:29 158 ERR 11599 BTRACE 0: 0x0000000000ddc32d eo670_UnixTraceStack +0x01ad
2007-12-11 05:33:29 158 ERR 11599 BTRACE /opt/sdb/7600/pgm/kernel
2007-12-11 05:33:29 158 ERR 11599 BTRACE Frameinfo [0x0]
2007-12-11 05:33:29 158 ERR 11599 BTRACE 1: 0x0000000000ddccf9 eo670_CTraceContextStackOCB +0x0009
2007-12-11 05:33:29 158 ERR 11599 BTRACE /opt/sdb/7600/pgm/kernel
2007-12-11 05:33:29 158 ERR 11599 BTRACE Frameinfo [0x0]
2007-12-11 05:33:29 158 ERR 11599 BTRACE 2: 0x0000000000ddce42 eo670_CTraceStackOCB +0x0032
2007-12-11 05:33:29 158 ERR 11599 BTRACE /opt/sdb/7600/pgm/kernel
2007-12-11 05:33:29 158 ERR 11599 BTRACE Frameinfo [0x0]
2007-12-11 05:33:29 158 ERR 11599 BTRACE 3: 0x0000000000ddce77 eo670_CTraceStack +0x0017
2007-12-11 05:33:29 158 ERR 11599 BTRACE /opt/sdb/7600/pgm/kernel
2007-12-11 05:33:29 158 ERR 11599 BTRACE Frameinfo [0x0]
2007-12-11 05:33:29 158 ERR 11599 BTRACE 4: 0x0000000000dfeed7 vabort +0x0027
2007-12-11 05:33:29 158 ERR 11599 BTRACE /opt/sdb/7600/pgm/kernel
2007-12-11 05:33:29 158 ERR 11599 BTRACE Frameinfo [0x0]
2007-12-11 05:33:29 158 ERR 11599 BTRACE 5: 0x0000000000d1bbdd Z9RTECrashRK20SAPDBErr_MessageList +0x008d
2007-12-11 05:33:29 158 ERR 11599 BTRACE /opt/sdb/7600/pgm/kernel
2007-12-11 05:33:29 158 ERR 11599 BTRACE Frameinfo [0x0]
2007-12-11 05:33:29 158 ERR 11599 BTRACE 6: 0x0000000000ce28ab ZN21KernelAdministration7OfflineERK20SAPDBErr_MessageListbb +0x034b
2007-12-11 05:33:29 158 ERR 11599 BTRACE /opt/sdb/7600/pgm/kernel
2007-12-11 05:33:29 158 ERR 11599 BTRACE Frameinfo [0x0]
2007-12-11 05:33:29 158 ERR 11599 BTRACE 7: 0x0000000000a25e4f gg999Abort +0x031f
2007-12-11 05:33:29 158 ERR 11599 BTRACE /opt/sdb/7600/pgm/kernel
2007-12-11 05:33:29 158 ERR 11599 BTRACE Frameinfo [0x0]
2007-12-11 05:33:29 158 ERR 11599 BTRACE 8: 0x000000000091343b bd20_Abort +0x003b
2007-12-11 05:33:29 158 ERR 11599 BTRACE /opt/sdb/7600/pgm/kernel
2007-12-11 05:33:29 158 ERR 11599 BTRACE Frameinfo [0x0]
2007-12-11 05:33:29 158 ERR 11599 BTRACE 9: 0x0000000000914814 bd20RReleasePage +0x0124
2007-12-11 05:33:29 158 ERR 11599 BTRACE /opt/sdb/7600/pgm/kernel
2007-12-11 05:33:29 158 ERR 11599 BTRACE Frameinfo [0x0]
2007-12-11 05:33:29 158 ERR 11599 BTRACE 10: 0x000000000094d8ba b13r_release_node +0x00ba
2007-12-11 05:33:29 158 ERR 11599 BTRACE /opt/sdb/7600/pgm/kernel
2007-12-11 05:33:29 158 ERR 11599 BTRACE Frameinfo [0x0]
2007-12-11 05:33:29 158 ERR 11599 BTRACE 11: 0x00000000009d81a4 ZN11cbd600Node12bd600ReleaseEb +0x0044
2007-12-11 05:33:29 158 ERR 11599 BTRACE /opt/sdb/7600/pgm/kernel
2007-12-11 05:33:29 158 ERR 11599 BTRACE Frameinfo [0x0]
2007-12-11 05:33:29 158 ERR 11599 BTRACE 12: 0x00000000009b88e0 Z35bd402SearchIndexForQualifiedRecordsR17cbd300InvCurrentR12tgg00_FileIdPhRsS3_iS3_iS3_i
2007-12-11 05:33:29 158 ERR 11599 BTRACE S3_S4_iP10tsp00_VoidR23tgg00_SelectFieldsParamR15tgg00_StackDescR10tsp00_EnumI22tgg00_Lock
2007-12-11 05:33:29 158 ERR 11599 BTRACE ReqMode_EnumLi7EhERb +0x0770
2007-12-11 05:33:29 158 ERR 11599 BTRACE /opt/sdb/7600/pgm/kernel
2007-12-11 05:33:29 158 ERR 11599 BTRACE Frameinfo [0x0]
2007-12-11 05:33:29 158 ERR 11599 BTRACE 13: 0x000000000093c360 b03select_invrec +0x0510
2007-12-11 05:33:29 158 ERR 11599 BTRACE /opt/sdb/7600/pgm/kernel
2007-12-11 05:33:29 158 ERR 11599 BTRACE Frameinfo [0x0]
2007-12-11 05:33:29 158 ERR 11599 BTRACE 14: 0x000000000084e68f ZN22JoinInvAccessOperator13SelectRecordsEv +0x011f
2007-12-11 05:33:29 158 ERR 11599 BTRACE /opt/sdb/7600/pgm/kernel
2007-12-11 05:33:29 158 ERR 11599 BTRACE Frameinfo [0x0]
2007-12-11 05:33:29 158 ERR 11599 BTRACE 15: 0x00000000008439a4 ZN19JoinAccessOperator4NextERP9tgg00_Rec +0x00a4
2007-12-11 05:33:29 158 ERR 11599 BTRACE /opt/sdb/7600/pgm/kernel
2007-12-11 05:33:29 158 ERR 11599 BTRACE Frameinfo [0x0]
2007-12-11 05:33:29 158 ERR 11599 BTRACE 16: 0x0000000000844455 ZN20JoinIAccessOperator4nextEPP9tgg00_Rec +0x0025
2007-12-11 05:33:29 158 ERR 11599 BTRACE /opt/sdb/7600/pgm/kernel
2007-12-11 05:33:29 158 ERR 11599 BTRACE Frameinfo [0x0]
2007-12-11 05:33:29 158 ERR 11599 BTRACE 17: 0x000000000084c43d ZN18JoinJoinOperator24nextEPP9tgg00_Rec +0x00fd
2007-12-11 05:33:29 158 ERR 11599 BTRACE /opt/sdb/7600/pgm/kernel
2007-12-11 05:33:29 158 ERR 11599 BTRACE Frameinfo [0x0]
2007-12-11 05:33:29 158 ERR 11599 BTRACE 18: 0x000000000084af19 ZN18JoinJoinOperator24NextERP9tgg00_Rec +0x0059
2007-12-11 05:33:29 158 ERR 11599 BTRACE /opt/sdb/7600/pgm/kernel
2007-12-11 05:33:29 158 ERR 11599 BTRACE Frameinfo [0x0]
2007-12-11 05:33:29 158 ERR 11599 BTRACE 19: 0x00000000008350fb a686_create_join_result +0x073b
2007-12-11 05:33:29 158 ERR 11599 BTRACE /opt/sdb/7600/pgm/kernel
2007-12-11 05:33:29 158 ERR 11599 BTRACE Frameinfo [0x0]
2007-12-11 05:33:29 158 ERR 11599 BTRACE 20: 0x000000000080ba7f a682_execute_join_operator +0x036f
2007-12-11 05:33:29 158 ERR 11599 BTRACE /opt/sdb/7600/pgm/kernel
2007-12-11 05:33:29 158 ERR 11599 BTRACE Frameinfo [0x0]
2007-12-11 05:33:29 158 ERR 11599 BTRACE 21: 0x000000000080e28f a682_only_ex_join +0x020f
2007-12-11 05:33:29 158 ERR 11599 BTRACE /opt/sdb/7600/pgm/kernel
2007-12-11 05:33:29 158 ERR 11599 BTRACE Frameinfo [0x0]
2007-12-11 05:33:29 158 ERR 11599 BTRACE 22: 0x000000000074195f a505most_execute +0x0bff
2007-12-11 05:33:29 158 ERR 11599 BTRACE /opt/sdb/7600/pgm/kernel
2007-12-11 05:33:29 158 ERR 11599 BTRACE Frameinfo [0x0]
2007-12-11 05:33:29 158 ERR 11599 BTRACE 23: 0x0000000000742ac0 a505loop_most_execute +0x0360
2007-12-11 05:33:29 158 ERR 11599 BTRACE /opt/sdb/7600/pgm/kernel
2007-12-11 05:33:29 158 ERR 11599 BTRACE Frameinfo [0x0]
2007-12-11 05:33:29 158 ERR 11599 BTRACE 24: 0x0000000000737886 a501exec_with_change_rec +0x05b6
2007-12-11 05:33:29 158 ERR 11599 BTRACE /opt/sdb/7600/pgm/kernel
2007-12-11 05:33:29 158 ERR 11599 BTRACE Frameinfo [0x0]
2007-12-11 05:33:29 158 ERR 11599 BTRACE 25: 0x0000000000738750 a501execute +0x0510
2007-12-11 05:33:29 158 ERR 11599 BTRACE /opt/sdb/7600/pgm/kernel
2007-12-11 05:33:29 158 ERR 11599 BTRACE Frameinfo [0x0]
2007-12-11 05:33:29 158 ERR 11599 BTRACE 26: 0x00000000007f150c ak92analyze_messagetype +0x042c
2007-12-11 05:33:29 158 ERR 11599 BTRACE /opt/sdb/7600/pgm/kernel
2007-12-11 05:33:29 158 ERR 11599 BTRACE Frameinfo [0x0]
2007-12-11 05:33:29 158 ERR 11599 BTRACE 27: 0x00000000007f200d a92_mode_analyzer +0x019d
2007-12-11 05:33:29 158 ERR 11599 BTRACE /opt/sdb/7600/pgm/kernel
2007-12-11 05:33:29 158 ERR 11599 BTRACE Frameinfo [0x0]
2007-12-11 05:33:29 158 ERR 11599 BTRACE 28: 0x00000000007f3047 ak93one_command +0x0477
2007-12-11 05:33:29 158 ERR 11599 BTRACE /opt/sdb/7600/pgm/kernel
2007-12-11 05:33:29 158 ERR 11599 BTRACE Frameinfo [0x0]
2007-12-11 05:33:29 158 ERR 11599 BTRACE 29: 0x00000000007f5d88 a93_user_commands +0x0528
2007-12-11 05:33:29 158 ERR 11599 BTRACE /opt/sdb/7600/pgm/kernel
2007-12-11 05:33:29 158 ERR 11599 BTRACE Frameinfo [0x0]
2007-12-11 05:33:29 158 ERR 11599 BTRACE 30: 0x0000000000cf39a4 Z7SQLTaskR20takall_command_globRhb +0x0074
2007-12-11 05:33:29 158 ERR 11599 BTRACE /opt/sdb/7600/pgm/kernel
2007-12-11 05:33:29 158 ERR 11599 BTRACE Frameinfo [0x0]
2007-12-11 05:33:29 158 ERR 11599 BTRACE 31: 0x0000000000cf3b83 Z11KernelMainR13RTETask_ITask +0x01b3
2007-12-11 05:33:29 158 ERR 11599 BTRACE /opt/sdb/7600/pgm/kernel
2007-12-11 05:33:29 158 ERR 11599 BTRACE Frameinfo [0x0]
2007-12-11 05:33:29 158 ERR 11599 BTRACE 32: 0x0000000000d7a03f RTETask_TaskMain +0x005f
2007-12-11 05:33:29 158 ERR 11599 BTRACE /opt/sdb/7600/pgm/kernel
2007-12-11 05:33:29 158 ERR 11599 BTRACE Frameinfo [0x0]
2007-12-11 05:33:29 158 ERR 11599 BTRACE 33: 0x0000000000e23dec Z23en88CallKernelTaskMainP9TASK_TYPE +0x009c
2007-12-11 05:33:29 158 ERR 11599 BTRACE /opt/sdb/7600/pgm/kernel
2007-12-11 05:33:29 158 ERR 11599 BTRACE Frameinfo [0x0]
2007-12-11 05:33:29 158 ERR 11599 BTRACE 34: 0x0000000000e2405a en88_RunPthreadTask +0x003a
2007-12-11 05:33:29 158 ERR 11599 BTRACE /opt/sdb/7600/pgm/kernel
2007-12-11 05:33:29 158 ERR 11599 BTRACE Frameinfo [0x0]
2007-12-11 05:33:29 158 ERR 11599 BTRACE -
> End of Stack Back Trace <----
2007-12-11 05:33:30 158 ERR 7 Messages Msg_List.cpp:3617
2007-12-11 05:33:30 158 ERR 7 Messages 2007-12-11 05:33:30 Messages Error 7
2007-12-11 05:33:30 158 ERR 7 Messages Begin of dump of registered messages
2007-12-11 05:33:30 158 ERR 2 Admin vgg999.cpp:153
2007-12-11 05:33:30 158 ERR 2 Admin 2007-12-11 05:33:29 Admin Error 2
2007-12-11 05:33:30 158 ERR 2 Admin A fatal error caused EMERGENCY SHUTDOWN. core is written
2007-12-11 05:33:30 158 ERR 2 Admin vgg999.cpp:153
2007-12-11 05:33:30 158 ERR 2 Admin 2007-12-11 05:33:29 Admin Error 2
2007-12-11 05:33:30 158 ERR 2 Admin A fatal error caused EMERGENCY SHUTDOWN. core is written
2007-12-11 05:33:30 158 ERR 8 Messages Msg_List.cpp:3636
2007-12-11 05:33:30 158 ERR 8 Messages 2007-12-11 05:33:30 Messages Error 8
2007-12-11 05:33:30 158 ERR 8 Messages End of the message list registry dump
2007-12-11 05:33:30 158 ERR 11196 DBCRASH vabort:Emergency Shutdown, Task: 141
2007-12-11 05:33:37 0 ERR 12006 DBCRASH Kernel exited without core and exit status 0x200
2007-12-11 05:33:37 0 ERR 12010 DBCRASH Kernel exited exit code 2
2007-12-11 05:33:38 ___ Stopping GMT 2007-12-11 04:33:38 7.6.00 Build 016-123-109-428