Skip to Content
avatar image
Former Member

Environment PRD froze

Hi,

Two days ago (08/06/2012), the environment PRD froze. I have checked the OS and there was high CPU spikes. I had to run stop / start in SAP and returned to normal operation. In SAP only found dumps (attachments), but I could not be sure the real reason for this freeze. Any ideas?

Windows Server 2003 En.terprise x64 Sp 2

SQL Server 2008

64GB Ram

SWAp conf       free                  swap sise

107.383.544    32.226.980       41.926.656

ztta/roll_area   = 6500000

ztta/roll_extension   = 2000000000

abap/heap_area_total = 38654705664

abap/heap_area_dia = 2000000000

abap/heap_area_nondia = 0

em/initial_size_MB= 36864

abap/heaplimit = 40000000

tks

Add comment
10|10000 characters needed characters exceeded

  • Get RSS Feed

2 Answers

  • avatar image
    Former Member
    Aug 09, 2012 at 11:55 PM

    Hi,

    What else did you observe when the PRD was in frozen state ?

    When the CPU was spiking did you check SM66/SM50 what was it looking like ? Also did you check killing some of the spiking processes from SM50 as  a workaround to stop/start SAP to restore normalcy ?

    You can as well check SM21 and dev trace files for any errors there.

    Regards,

    Samik

    Add comment
    10|10000 characters needed characters exceeded

    • Former Member Nicholas Chang

      I checked the file dev_w0.old, but I could not understand the real cause of the problem:

      Mon Aug 06 15:15:45 2012

      A  ***SUBPOOL*** generating subroutine pool %_T02A9Q for user CALCARFE (226b0302).

      A                MainProg=SAPLSBAL_DISPLAY_BASE, Incl=LSBAL_DISPLAY_BASEF01, Line=316

      M Mon Aug 06 15:17:33 2012

      M  *****************************************************************************

      M  *

      M  *  LOCATION    SAP-Gateway on host ERPTB706P.erpbtg.com / sapgw00

      M  *  ERROR       internal error

      M  *

      M  *  TIME        Mon Aug 06 15:17:33 2012

      M  *  RELEASE     701

      M  *  COMPONENT   NI (network interface)

      M  *  VERSION     38

      M  *  RC          -1

      M  *  MODULE      ninti.c

      M  *  LINE        742

      M  *  DETAIL      NiPGetHostByAddr3: '???' not found

      M  *  SYSTEM CALL getnameinfo

      M  *  ERRNO       10022

      M  *  ERRNO TEXT  WSAEINVAL: Invalid argument

      M  *  COUNTER     983632

      M  *

      M  *****************************************************************************

      A  RFC 2322  CONVID 86716184

      A   * CMRC=20 DATA=0 STATUS=0 SAPRC=236 ThSAPCMINIT

      A  RFC> ABAP Programm: /BDL/SAPLBDL11 (Transaction: )

      A  RFC> User: BASIS (Client: 300)

      A  RFC> Destination: SM_SMPCLNT100_BACK (handle: 3, , {F8F2DFE1-D475-F10C-BB82-00265533596C})

      A  RFC> Called function module: RFC_GET_FUNCTION_INTERFACE_P

      A  RFC SERVER> RFC Server Session (handle: 1, 86690075, {EFF2DFE1-5974-F193-BB82-00265533596C})

      A  RFC SERVER> Caller host:

      A  RFC SERVER> Caller transaction code:  (Caller Program: /BDL/TASK_SCHEDULER)

      A  RFC SERVER> Called function module: /BDL/_SESSION_REFRESH_TASK

      A  TH VERBOSE LEVEL FULL

      A  ** RABAX: end RX_GET_MESSAGE

      A Mon Aug 06 15:22:08 2012

      A  ***SUBPOOL*** generating subroutine pool %_T02AA3 for user AFONSOFE (24d40001).

      A                MainProg=SAPLSBAL_DISPLAY_BASE, Incl=LSBAL_DISPLAY_BASEF01, Line=316

      A Mon Aug 06 15:23:17 2012

      A  ***SUBPOOL*** generating subroutine pool %_T02AA6 for user FERREIFA (24320202).

      A                MainProg=SAPLSKBH, Incl=LSKBHF06, Line=549

      A  ***SUBPOOL*** generating subroutine pool %_T02AA7 for user FERREIFA (24320202).

      A                MainProg=SAPLSKBH, Incl=LSKBHF06, Line=549

      A  ***SUBPOOL*** generating subroutine pool %_T02AA8 for user FERREIFA (24320202).

      A                MainProg=SAPLSKBH, Incl=LSKBHF06, Line=549

      A  ***SUBPOOL*** generating subroutine pool %_T02AA9 for user FERREIFA (24320202).

      A                MainProg=SAPLSKBH, Incl=LSKBHF06, Line=549

      A  ***SUBPOOL*** generating subroutine pool %_T02AAA for user FERREIFA (24320202).

      A                MainProg=SAPLSKBH, Incl=LSKBHF06, Line=549

      A Mon Aug 06 15:23:51 2012

      A  ***SUBPOOL*** generating subroutine pool %_T02AAB for user RODRIGHA (20f90101).

      A                MainProg=SAPLSBAL_DISPLAY_BASE, Incl=LSBAL_DISPLAY_BASEF01, Line=316

      A Mon Aug 06 15:26:36 2012

      A  ***SUBPOOL*** generating subroutine pool %_T02AAH for user GRECHIGI (22b50001).

      A                MainProg=SAPLSBAL_DISPLAY_BASE, Incl=LSBAL_DISPLAY_BASEF01, Line=316

      A Mon Aug 06 15:27:42 2012

      A  ***SUBPOOL*** generating subroutine pool %_T02AAI for user CAETANDA (29b20102).

      A                MainProg=SAPLSKBH, Incl=LSKBHF06, Line=549

      A  ***SUBPOOL*** generating subroutine pool %_T02AAJ for user CAETANDA (29b20102).

      A                MainProg=SAPLSKBH, Incl=LSKBHF06, Line=549

      A  ***SUBPOOL*** generating subroutine pool %_T02AAK for user CAETANDA (29b20102).

      A                MainProg=SAPLSKBH, Incl=LSKBHF06, Line=549

      A  ***SUBPOOL*** generating subroutine pool %_T02AAL for user CAETANDA (29b20102).

      A                MainProg=SAPLSKBH, Incl=LSKBHF06, Line=549

      A  ***SUBPOOL*** generating subroutine pool %_T02AAM for user CAETANDA (29b20102).

      A                MainProg=SAPLSKBH, Incl=LSKBHF06, Line=549

      C Mon Aug 06 15:33:56 2012

      C  Thread ID:14396

      C  Thank You for using the SLODBC-interface

      C  Using dynamic link library 'F:\usr\sap\PRD\DVEBMGS00\exe\dbmssslib.dll'

      C  dbmssslib.dll patch info

      C    SAP patchlevel  0

      C    SAP patchno  137

      C    Last MSSQL DBSL patchlevel 0

      C    Last MSSQL DBSL patchno         134

      C    Last MSSQL DBSL patchcomment UPSERT retry corrected (1566676)

      C  lpc:(local) connection used on ERPTB706P

      C  lpc:(local) connection used on ERPTB706P

      C  Driver: sqlncli10.dll Driver release: 10.00.2531

      C  GetDbRelease: 10.00.2531

      C  GetDbRelease: Got DB release numbers (10,0,2531)

      B  Connection 4 opened (DBSL handle 2)

      A Mon Aug 06 15:34:32 2012

      A  TH VERBOSE LEVEL FULL

      A  ** RABAX: level LEV_RX_PXA_RELEASE_MTX entered.

      A  ** RABAX: level LEV_RX_PXA_RELEASE_MTX completed.

      A  ** RABAX: level LEV_RX_COVERAGE_ANALYSER entered.

      A  ** RABAX: level LEV_RX_COVERAGE_ANALYSER completed.

      A  ** RABAX: level LEV_RX_SAVE_SHMLOCKS entered.

      A  ** RABAX: level LEV_RX_SAVE_SHMLOCKS completed.

      A  ** RABAX: level LEV_RX_RESET_SHMLOCKS entered.

      A  ** RABAX: level LEV_RX_RESET_SHMLOCKS completed.

      A  ** RABAX: level LEV_RX_ROLLBACK entered.

      A  ** RABAX: level LEV_RX_ROLLBACK completed.

      A  ** RABAX: level LEV_RX_DB_ALIVE entered.

      A  ** RABAX: level LEV_RX_DB_ALIVE completed.

      A  ** RABAX: level LEV_RX_HOOKS entered.

      A  ** RABAX: level LEV_RX_HOOKS completed.

      A  ** RABAX: level LEV_RX_STANDARD entered.

      A Mon Aug 06 15:34:33 2012

      A  ** RABAX: level LEV_RX_STANDARD completed.

      A  ** RABAX: level LEV_RX_STOR_VALUES entered.

      A  ** RABAX: level LEV_RX_STOR_VALUES completed.

      A  ** RABAX: level LEV_RX_C_STACK entered.

      A  ** RABAX: level LEV_RX_C_STACK completed.

      A  ** RABAX: level LEV_RX_MEMO_CHECK entered.

      A  ** RABAX: level LEV_RX_MEMO_CHECK completed.

      A  ** RABAX: level LEV_RX_AFTER_MEMO_CHECK entered.

      A  ** RABAX: level LEV_RX_AFTER_MEMO_CHECK completed.

      A  ** RABAX: level LEV_RX_INTERFACES entered.

      A  ** RABAX: level LEV_RX_INTERFACES completed.

      A  ** RABAX: level LEV_RX_GET_MESS entered.

      A  ** RABAX: level LEV_RX_GET_MESS completed.

      A  ** RABAX: level LEV_RX_INIT_SNAP entered.

      A  ** RABAX: level LEV_RX_INIT_SNAP completed.

      A  ** RABAX: level LEV_RX_WRITE_SYSLOG entered.

      A  ** RABAX: level LEV_RX_WRITE_SYSLOG completed.

      A  ** RABAX: level LEV_RX_WRITE_SNAP_BEG entered.

      A  ** RABAX: level LEV_RX_WRITE_SNAP_BEG completed.

      A  ** RABAX: level LEV_RX_WRITE_SNAP entered.

      A Mon Aug 06 15:34:38 2012

      A  ** RABAX: level LEV_SN_END completed.

      A  ** RABAX: level LEV_RX_WRITE_SNAP_END entered.

      A  ** RABAX: level LEV_RX_WRITE_SNAP_END completed.

      A  ** RABAX: level LEV_RX_SET_ALERT entered.

      A  ** RABAX: level LEV_RX_SET_ALERT completed.

      A  ** RABAX: level LEV_RX_COMMIT entered.

      A  ** RABAX: level LEV_RX_COMMIT completed.

      A  ** RABAX: level LEV_RX_SNAP_SYSLOG entered.

      A  ** RABAX: level LEV_RX_SNAP_SYSLOG completed.

      A  ** RABAX: level LEV_RX_RESET_PROGS entered.

      A  ** RABAX: level LEV_RX_RESET_PROGS completed.

      A  ** RABAX: level LEV_RX_STDERR entered.

      A  Mon Aug 06 15:34:38 2012

      A  ABAP Program SAPLKMS0                                .

      A  Source LKMS0F03                                 Line 480.

      A  Error Code SYSTEM_NO_TASK_STORAGE.

      A  Module  $Id: //bas/701_REL/src/krn/runt/abtsv2.c#4 $ SAP.

      A  Function ArraySort Line 1318.

      A  ** RABAX: level LEV_RX_STDERR completed.

      A  ** RABAX: level LEV_RX_RFC_ERROR entered.

      A  ** RABAX: level LEV_RX_RFC_ERROR completed.

      A  ** RABAX: level LEV_RX_RFC_CLOSE entered.

      A  ** RABAX: level LEV_RX_RFC_CLOSE completed.

      A  ** RABAX: level LEV_RX_IMC_ERROR entered.

      A  ** RABAX: level LEV_RX_IMC_ERROR completed.

      A  ** RABAX: level LEV_RX_DATASET_CLOSE entered.

      A  ** RABAX: level LEV_RX_DATASET_CLOSE completed.

      A  ** RABAX: level LEV_RX_ERROR_SAVE entered.

      A  ** RABAX: level LEV_RX_ERROR_SAVE completed.

      A  ** RABAX: level LEV_RX_ERROR_TPDA entered.

      A  ** RABAX: level LEV_RX_ERROR_TPDA completed.

      A  ** RABAX: level LEV_RX_PXA_RELEASE_RUDI entered.

      A  ** RABAX: level LEV_RX_PXA_RELEASE_RUDI completed.

      A  ** RABAX: level LEV_RX_LIVE_CACHE_CLEANUP entered.

      A  ** RABAX: level LEV_RX_LIVE_CACHE_CLEANUP completed.

      A  ** RABAX: level LEV_RX_END entered.

      A  ** RABAX: level LEV_RX_END completed.

      A  ** RABAX: end no http/smtp

      A  ** RABAX: end RX_GOTO_RABAX

      A  Unable to fulfil request for 7883584 bytes of storage space..

      A Mon Aug 06 15:36:16 2012

      A  ***SUBPOOL*** generating subroutine pool %_T02AAY for user RODRIGHA (20f90202).

      A                MainProg=SAPLSBAL_DISPLAY_BASE, Incl=LSBAL_DISPLAY_BASEF01, Line=316

      M Mon Aug 06 15:52:41 2012

      M  in_ThErrHandle: 1

      M  ThIErrHandle: new stat of W0 is WP_SHUTDOWN   

      M  ThIErrHandle: I'm during shutdown

      M  PfStatDisconnect: disconnect statistics

      M  Entering ThSetStatError

      C  SlBreak

      B  db_sqlbreak() = 15

      M  *** WARNING => sql break failed (16388)

      M  ThIErrHandle: don't try rollback again

      M  ThShutDownServer: shutdown server

      M  PfWriteIntoFile: shared buffer is empty.

      M  ThIErrHandle: do not call ThrCoreInfo (no_core_info=0, in_dynp_env=0)

      M  Entering ThReadDetachMode

      M  call ThrShutDown (1)...

      B  Disconnecting from ALL connections:

      B  Wp  Hdl ConName                        ConId     ConState     TX  HC  PRM RCT FRC TIM MAX OPT Date     Time   DBHost                       

      B  000 000 R/3                            000000000 INACTIVE     NO  NO  YES NO  NO  000 255 255 20120804 171703 ERPTB706P                    

      B  000 001 ++DBO++0020                    000084952 INACTIVE     NO  NO  NO  NO  NO  004 255 255 20120804 172237 ERPTB706P                    

      B  000 002 R/3*INACT_PACK                 000084907 DISCONNECTED NO  NO  NO  NO  NO  000 255 255 20120806 141743 ERPTB706P                    

      B  000 003 ++DBO++0020                    000084562 DISCONNECTED NO  NO  NO  NO  NO  000 255 255 20120806 044242 ERPTB706P                    

      B  000 004 R/3*EXPRESSPOPUP               000084954 INACTIVE     NO  NO  NO  NO  NO  004 255 255 20120806 153356 ERPTB706P                    

      B  Disconnected from connection 0

      B  Disconnected from connection 1

      B Mon Aug 06 15:52:42 2012

      B  Disconnected from connection 4

      B  statistics db_con_commit (com_total=5918337, com_tx=46876)

      B  statistics db_con_rollback (roll_total=8049, roll_tx=3)

      M  ***LOG Q02=> wp_halt, WPStop (Workproc 0 14636) [dpnttool.c   334]

  • Aug 10, 2012 at 09:18 AM

    Hi Emerson,

    It seems you have a 64 bit system which is good but from the short dumps your memory config needs to be addjusted. I see you have heap non-dia set to 0 which is not as per

    note 146289.

    You can also tune the memory usage to decrease the swap occupation, so
    please check and tune the memory parameter following the note:

    146289 - Parameter Recommendations for 64-Bit SAP Kernel

    especially increase the following parameters

    abap/heap_area_dia      2000000000 (2GB)

    abap/heap_area_nondia   2000000000 (2GB)

    abap/heap_area_total    2000000000 (2GB)

    Check SAP Note 

    88416 - Zero administration memory management as of 4.0A/Windows

    Check what is set for PHYS_MEMSIZE and how much pyhsical RAM is on the machine.

    Please, evaluate according to the note if that PHYS_MEMSIZE value is convenient for your
    system (this value is the key for the application of note 88416).

       110172 - Windows: Transactions with large memory requirements
       552209 - Maximum memory utilization for processes on NT/Win2000
       313347 - Windows editions and memory support

    Hope this helps.

    Regards,

    Paul

    Add comment
    10|10000 characters needed characters exceeded

    • Former Member

      Hello Paul

      I checked note 146289, 88416. The note 88416 states that the  parameter abap / heap_area_nondia is 0 when using the parameter PHYS_MEMSIZE (Current PHYS_MEMSIZE = 36864).

      Tks